Grouper Users - Open Discussion List

[Shilen Patel <>]

Can you run the following query?

select source_id, count(*) from grouper_pit_members where active='T' group by source_id having count(*)>  1;

Then for each result, run this query:

 -  select * from grouper_pit_members where source_id='STRING_FROM_ABOVE'

 -  You'll get multiple rows back.  Assuming they are identical (except different ID values), delete all but one row.  If one of the rows has foreign keys, delete the others.

There are some API and index changes in 2.1.3 that will prevent this from happening.  There's also a script in 2.1.3 that will attempt at removing these duplicates if you want to try that instead — you shouldn't have to upgrade to run it.

Thanks!

-- Shilen

From: Rahul Doshi <>
Date: Mon, 18 Mar 2013 18:08:47 +0000
To: Shilen Patel <>
Cc: "" <>
Subject: Re: [grouper-users] grouper PSP errors

Yes, we are still having the problem.  I just went back in the logs to find the first occurrence of the problem so that it might help to know better what triggered the issue.  Current errors are below.

2013-03-18 14:05:50,061: [DefaultQuartzScheduler_Worker-5] ERROR GrouperLoaderJob.runJob(406) -  - Error on job: CHANGE_LOG_changeLogTempToChangeLog

edu.internet2.middleware.grouper.internal.dao.GrouperDAOException: Problem in HibernateSession: HibernateSession: isNew: false, isReadonly: false, grouperTransactionType: READ_WRITE_NEW,

Exception in uniqueResult: (class edu.internet2.middleware.grouper.pit.PITMember), ByHqlStatic, query: 'select pitMember from PITMember as pitMember where pitMember.sourceId = :id and activeDb = 'T'', cacheable: true, cacheRegion: edu.internet2.middleware.grouper.internal.dao.hib3.Hib3PITMemberDAO.FindBySourceIdActive, tx type: null, tx type: nullBind var[0]: 'Param (class java.lang.String): 'id'->'07058251fa4b46cba6cab0ecbe9a6d0d', ,

Problem in HibernateSession: HibernateSession: isNew: true, isReadonly: false, grouperTransactionType: READ_WRITE_NEW

        at edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch(HibernateSession.java:495)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:581)

        at edu.internet2.middleware.grouper.hibernate.ByHqlStatic.uniqueResult(ByHqlStatic.java:323)

        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3PITMemberDAO.findBySourceIdActive(Hib3PITMemberDAO.java:70)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity.processMembershipAdd(ChangeLogTempToEntity.java:708)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity.access$1700(ChangeLogTempToEntity.java:59)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity$1.callback(ChangeLogTempToEntity.java:135)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:571)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity.convertRecords(ChangeLogTempToEntity.java:91)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$4.runJob(GrouperLoaderType.java:550)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.runJob(GrouperLoaderJob.java:398)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.execute(GrouperLoaderJob.java:300)

        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)

Caused by: org.hibernate.NonUniqueResultException: query did not return a unique result: 3

        at org.hibernate.impl.AbstractQueryImpl.uniqueElement(AbstractQueryImpl.java:899)

        at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:890)

        at edu.internet2.middleware.grouper.hibernate.ByHql.uniqueResult(ByHql.java:257)

        at edu.internet2.middleware.grouper.hibernate.ByHqlStatic$1.callback(ByHqlStatic.java:332)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:571)

        ... 12 more

2013-03-18 14:05:50,066: [DefaultQuartzScheduler_Worker-5] ERROR GrouperLoaderJob.execute(303) -  - Error running up job

edu.internet2.middleware.grouper.internal.dao.GrouperDAOException: Problem in HibernateSession: HibernateSession: isNew: false, isReadonly: false, grouperTransactionType: READ_WRITE_NEW,

Exception in uniqueResult: (class edu.internet2.middleware.grouper.pit.PITMember), ByHqlStatic, query: 'select pitMember from PITMember as pitMember where pitMember.sourceId = :id and activeDb = 'T'', cacheable: true, cacheRegion: edu.internet2.middleware.grouper.internal.dao.hib3.Hib3PITMemberDAO.FindBySourceIdActive, tx type: null, tx type: nullBind var[0]: 'Param (class java.lang.String): 'id'->'07058251fa4b46cba6cab0ecbe9a6d0d', ,

Problem in HibernateSession: HibernateSession: isNew: true, isReadonly: false, grouperTransactionType: READ_WRITE_NEW

        at edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch(HibernateSession.java:495)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:581)

        at edu.internet2.middleware.grouper.hibernate.ByHqlStatic.uniqueResult(ByHqlStatic.java:323)

        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3PITMemberDAO.findBySourceIdActive(Hib3PITMemberDAO.java:70)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity.processMembershipAdd(ChangeLogTempToEntity.java:708)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity.access$1700(ChangeLogTempToEntity.java:59)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity$1.callback(ChangeLogTempToEntity.java:135)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:571)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity.convertRecords(ChangeLogTempToEntity.java:91)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$4.runJob(GrouperLoaderType.java:550)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.runJob(GrouperLoaderJob.java:398)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.execute(GrouperLoaderJob.java:300)

        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)

Caused by: org.hibernate.NonUniqueResultException: query did not return a unique result: 3

        at org.hibernate.impl.AbstractQueryImpl.uniqueElement(AbstractQueryImpl.java:899)

        at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:890)

        at edu.internet2.middleware.grouper.hibernate.ByHql.uniqueResult(ByHql.java:257)

        at edu.internet2.middleware.grouper.hibernate.ByHqlStatic$1.callback(ByHqlStatic.java:332)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:571)

        ... 12 more

Thanks,

Rahul

From: Shilen Patel <>
Date: Monday, March 18, 2013 2:05 PM
To: Rahul Doshi <>
Cc: "" <>
Subject: Re: [grouper-users] grouper PSP errors

The time in the logs you gave are from last week.  Are you continuing to see the exact same errors appear every minute?  Can you send your latest logs?  You may have some duplicate point in time entries that need to be deleted.  The issue was fixed in 2.1.3.

Thanks!

-- Shilen

From: Rahul Doshi <>
Date: Mon, 18 Mar 2013 17:52:52 +0000
To: Shilen Patel <>
Cc: "" <>
Subject: Re: [grouper-users] grouper PSP errors

We are running grouper 2.1.2 and I am quite sure we don't have multiple grouper daemons running.

Thanks,

Rahul

From: Shilen Patel <>
Date: Monday, March 18, 2013 1:38 PM
To: Rahul Doshi <>
Cc: "" <>
Subject: Re: [grouper-users] grouper PSP errors

Do you have multiple instances of the Grouper daemon running?  What version of Grouper do you have?

Thanks!

-- Shilen

From: Rahul Doshi <>
Date: Mon, 18 Mar 2013 17:32:34 +0000
To: "" <>
Subject: [grouper-users] grouper PSP errors

Hello,

We are having some database errors which is causing PSP to not update the groups correctly. Does anyone has any suggestions how I can avoid this error in the future and what is the best way to recover from this?

2013-03-11 09:09:54,094: [DefaultQuartzScheduler_Worker-9] ERROR JDBCExceptionReporter.logExceptions(234) -  - Duplicate entry '947053' for key 'PRIMARY'

2013-03-11 09:09:54,096: [DefaultQuartzScheduler_Worker-9] ERROR GrouperLoaderJob.runJob(406) -  - Error on job: CHANGE_LOG_changeLogTempToChangeLog

edu.internet2.middleware.grouper.internal.dao.GrouperDAOException: Problem in HibernateSession: HibernateSession: isNew: false, isReadonly: false, grouperTransactionType: READ_WRITE_NEW,

Exception in save: edu.internet2.middleware.grouper.changeLog.ChangeLogEntry, ByObjectStatic, query: ', cacheable: null, cacheRegion: null, entityName: ChangeLogEntryEntity, tx type: null,

Problem in HibernateSession: HibernateSession: isNew: true, isReadonly: false, grouperTransactionType: READ_WRITE_NEW

        at edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch(HibernateSession.java:495)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:581)

        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic.save(ByObjectStatic.java:468)

        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3ChangeLogEntryDAO.save(Hib3ChangeLogEntryDAO.java:47)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogEntry.save(ChangeLogEntry.java:356)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity$1.callback(ChangeLogTempToEntity.java:196)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:571)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity.convertRecords(ChangeLogTempToEntity.java:91)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$4.runJob(GrouperLoaderType.java:550)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.runJob(GrouperLoaderJob.java:398)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.execute(GrouperLoaderJob.java:300)

        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)

Caused by: org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update,

Exception in save: edu.internet2.middleware.grouper.changeLog.ChangeLogEntry, edu.internet2.middleware.grouper.hibernate.ByObject@257f8345

        at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:96)

        at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)

        at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:275)

        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:268)

        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:184)

        at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)

        at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)

        at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)

        at edu.internet2.middleware.grouper.hibernate.ByObject.save(ByObject.java:214)

        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic$7.callback(ByObjectStatic.java:481)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:571)

        ... 11 more

Caused by: java.sql.BatchUpdateException: Duplicate entry '947053' for key 'PRIMARY'

        at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2024)

        at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1449)

        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1723)

        at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)

        at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)

        ... 19 more

Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '947053' for key 'PRIMARY'

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:532)

        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)

        at com.mysql.jdbc.Util.getInstance(Util.java:386)

        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1039)

        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)

        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)

        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)

        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)

        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)

        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)

        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)

        at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1976)

        ... 23 more

2013-03-11 09:10:00,043: [DefaultQuartzScheduler_Worker-1] ERROR GrouperLoaderType$4.runJob(596) -  - Problem with change log consumer: psp

edu.internet2.middleware.grouper.exception.GrouperStaleObjectStateException: Problem in HibernateSession: HibernateSession: isNew: true, isReadonly: false, grouperTransactionType: READ_WRITE_NEW

        at edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch(HibernateSession.java:491)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:581)

        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic.saveOrUpdate(ByObjectStatic.java:354)

        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3ChangeLogConsumerDAO.saveOrUpdate(Hib3ChangeLogConsumerDAO.java:41)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogHelper.processRecords(ChangeLogHelper.java:258)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$4.runJob(GrouperLoaderType.java:589)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.runJob(GrouperLoaderJob.java:398)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.execute(GrouperLoaderJob.java:300)

        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)

Caused by: org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [edu.internet2.middleware.grouper.changeLog.ChangeLogConsumer#8be8a00baea240eaae513780a4325886],

Exception in save: edu.internet2.middleware.grouper.changeLog.ChangeLogConsumer, edu.internet2.middleware.grouper.hibernate.ByObject@63e4a96f: [edu.internet2.middleware.grouper.changeLog.ChangeLogConsumer#8be8a00baea240eaae513780a4325886]

        at org.hibernate.persister.entity.AbstractEntityPersister.check(AbstractEntityPersister.java:1950)

        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2594)

        at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2494)

        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2821)

        at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:113)

        at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273)

        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265)

        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185)

        at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)

        at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)

        at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)

        at edu.internet2.middleware.grouper.hibernate.ByObject.saveOrUpdate(ByObject.java:377)

        at edu.internet2.middleware.grouper.hibernate.ByObjectStatic$5.callback(ByObjectStatic.java:367)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:571)

        ... 8 more

2013-03-11 09:10:50,047: [DefaultQuartzScheduler_Worker-6] ERROR GrouperLoaderJob.runJob(406) -  - Error on job: CHANGE_LOG_changeLogTempToChangeLog

edu.internet2.middleware.grouper.internal.dao.GrouperDAOException: Problem in HibernateSession: HibernateSession: isNew: false, isReadonly: false, grouperTransactionType: READ_WRITE_NEW,

Exception in uniqueResult: (class edu.internet2.middleware.grouper.pit.PITMember), ByHqlStatic, query: 'select pitMember from PITMember as pitMember where pitMember.sourceId = :id and activeDb = 'T'', cacheable: true, cacheRegion: edu.internet2.middleware.grouper.internal.dao.hib3.Hib3PITMemberDAO.FindBySourceIdActive, tx type: null, tx type: nullBind var[0]: 

'Param (class java.lang.String): 'id'->'07058251fa4b46cba6cab0ecbe9a6d0d', ,

Problem in HibernateSession: HibernateSession: isNew: true, isReadonly: false, grouperTransactionType: READ_WRITE_NEW

        at edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch(HibernateSession.java:495)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:581)

        at edu.internet2.middleware.grouper.hibernate.ByHqlStatic.uniqueResult(ByHqlStatic.java:323)

        at edu.internet2.middleware.grouper.internal.dao.hib3.Hib3PITMemberDAO.findBySourceIdActive(Hib3PITMemberDAO.java:70)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity.processMembershipAdd(ChangeLogTempToEntity.java:708)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity.access$1700(ChangeLogTempToEntity.java:59)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity$1.callback(ChangeLogTempToEntity.java:135)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:571)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity.convertRecords(ChangeLogTempToEntity.java:91)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$4.runJob(GrouperLoaderType.java:550)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.runJob(GrouperLoaderJob.java:398)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.execute(GrouperLoaderJob.java:300)

        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)

Caused by: org.hibernate.NonUniqueResultException: query did not return a unique result: 3

        at org.hibernate.impl.AbstractQueryImpl.uniqueElement(AbstractQueryImpl.java:899)

        at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:890)

        at edu.internet2.middleware.grouper.hibernate.ByHql.uniqueResult(ByHql.java:257)

        at edu.internet2.middleware.grouper.hibernate.ByHqlStatic$1.callback(ByHqlStatic.java:332)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:571)

        ... 12 more

Thanks,

Rahul