Grouper Users - Open Discussion List

[Colin Hudler <>]

Thanks, that fixed it quite neatly!  We are testing to try and identify 
the cause.

On 12/4/10 10:40 AM, Shilen Patel wrote:
> So the owner_id column in the view comes from the grouper_group_set
> table. If you query this table for owner_id=null and look at the
> create_time, were the rows created about the same time or were they
> created over several days or weeks or more? Were they created during
> your upgrade from 1.5 or sometime after? Did you run any special load
> job during the time? I'm just trying to get an idea of how the issue
> occurred.
>
> In any case, to fix the owner_id column....
>
> update grouper_group_set set owner_id = owner_group_id where owner_id is
> null and owner_group_id is not null;
> update grouper_group_set set owner_id = owner_stem_id where owner_id is
> null and owner_stem_id is not null;
> update grouper_group_set set owner_id = owner_attr_def_id where owner_id
> is null and owner_attr_def_id is not null;
>
> Then set a not null constraint on owner_id and if anything tries to add
> a null value, you'll get an error before bad data gets added.
>
> Thanks!
>
> -- Shilen
>
>
> On 12/4/10 11:00 AM, Colin Hudler wrote:
> > I ran that query. The owner_id does match the others, but they are
> > null :-/
> >
> > There's others that are null also. I think you are right about the db
> > inconsistency.
> >
> > SQL> select
> > owner_id,owner_group_id,owner_attr_def_id,owner_group_id,owner_stem_id
> > from grouper_memberships_all_v where
> > immediate_membership_id='00554d01d4eb451e93305b898f81adf3' and
> > group_set_id='55d3a30ada5040febdd7ac5ddc504ae5';
> >
> > OWNER_ID OWNER_GROUP_ID
> > OWNER_ATTR_DEF_ID OWNER_GROUP_ID
> > OWNER_STEM_ID
> > ----------------------------------------
> > ----------------------------------------
> > ----------------------------------------
> > ----------------------------------------
> > ----------------------------------------
> >
> > 865e18c2d5eb4314a7fe9ea2fb493bc2
> > 865e18c2d5eb4314a7fe9ea2fb493bc2
> >
> > SQL> select count(*) from grouper_memberships_all_v where owner_id is
> > null;
> >
> > COUNT(*)
> > ----------
> > 303
> >
> >
> > On 12/4/10 8:44 AM, Shilen Patel wrote:
> > > I think you can get that error if the database has inconsistent data. If
> > > the full stack trace shows a call to the hibernate flush(), that's
> > > possibly the issue. Can you run the following query?
> > >
> > > select * from grouper_memberships_all_v where
> > > immediate_membership_id='00554d01d4eb451e93305b898f81adf3' and
> > > group_set_id='55d3a30ada5040febdd7ac5ddc504ae5'
> > >
> > > Does the owner_id column match either owner_attr_def_id, owner_group_id,
> > > or owner_stem_id? If not, then that's the problem. Otherwise, can you
> > > send me the result of that query so I can see what else might be wrong?
> > >
> > > Thanks!
> > >
> > > -- Shilen
> > >
> > > On 12/4/10 8:05 AM, Shilen Patel wrote:
> > > > Hi Colin,
> > > >
> > > > For the second issue, can you send me the complete stack trace?
> > > >
> > > > Thanks!
> > > >
> > > > -- Shilen
> > > >
> > > > On 12/3/10 2:44 PM, Colin Hudler wrote:
> > > > > Greetings,
> > > > >
> > > > > Today I am troubleshooting a multi-symptom problem with grouper 1.6.2
> > > > > running on oracle 11g. I know of nothing that happened or changed in
> > > > > the last two days to cause this (we've been running smoothly until
> > > > > now).
> > > > > Sorry for vomiting these stack traces here; I hope it helps someone to
> > > > > see the direction of the problem. I also hope that the problems are
> > > > > related, because they all showed up around the same time. When
> > > > > browsing
> > > > > to a particular group-of-groups in the UI this error is logged:
> > > > >
> > > > > 2010-12-03 13:21:17,695: [TP-Processor23] ERROR
> > > > > GrouperCapableAction.execute(?) -< chudler
> > > > > 485E91D737D48700834238215E9B8B65-0110 840a2034d00a41608b9ec2436cd5c142
> > > > > 21882177X ucmcdb> - java.lang.RuntimeException: Too many results:
> > > > > 42672
> > > > > at
> > > > > edu.internet2.middleware.grouper.internal.dao.hib3.Hib3MembershipDAO.findAllByGroupOwnerOptions(Hib3MembershipDAO.java:630)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.MembershipFinder.findMemberships(MembershipFinder.java:81)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.ui.actions.PopulateGroupMembersAction.grouperExecute(Unknown
> > > > >
> > > > >
> > > > > Source)
> > > > > at
> > > > > edu.internet2.middleware.grouper.ui.actions.GrouperCapableAction$1.callback(Unknown
> > > > >
> > > > >
> > > > > Source)
> > > > > at
> > > > > <snip>
> > > > > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
> > > > >
> > > > >
> > > > > at java.lang.Thread.run(Thread.java:619)
> > > > >
> > > > > The web interface simply shows the generic error page. Additionally,
> > > > > our daily report from grouper-loader generates a different stack trace
> > > > > (an oracle error!) while it is syncing flat memberships table:
> > > > >
> > > > > edu.internet2.middleware.grouper.misc.GrouperReportException: Problem
> > > > > generating daily report
> > > > > at
> > > > > edu.internet2.middleware.grouper.misc.GrouperReport.runReport(GrouperReport.java:400)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$2.runJob(GrouperLoaderType.java:215)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.runJob(GrouperLoaderJob.java:344)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.execute(GrouperLoaderJob.java:247)
> > > > >
> > > > >
> > > > > at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> > > > > at
> > > > > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
> > > > >
> > > > >
> > > > > Caused by:
> > > > > edu.internet2.middleware.grouper.internal.dao.GrouperDAOException:
> > > > > Problem in HibernateSession: HibernateSession: isNew: true,
> > > > > isReadonly:
> > > > > true, grouperTransactionType: READONLY_NEW,
> > > > > Exception in list: (class
> > > > > edu.internet2.middleware.grouper.Membership),
> > > > > ByHqlStatic, query: 'select ms from MembershipEntry ms where
> > > > > ms.enabledDb='T' and not exists (select 1 from FlatMembership
> > > > > flatMship
> > > > > where flatMship.ownerId=ms.ownerId and
> > > > > flatMship.memberId=ms.memberUuid and flatMship.fieldId=ms.fieldId) and
> > > > > not exists (select 1 from ChangeLogEntryTemp temp, ChangeLogType type
> > > > > where temp.string06 = ms.ownerGroupId and temp.string09=ms.fieldId
> > > > > and type.actionName='addMembership' and
> > > > > type.changeLogCategory='membership' and type.id=temp.changeLogTypeId)
> > > > > and not exists (select 1 from ChangeLogEntryTemp temp, ChangeLogType
> > > > > type where temp.string07 = ms.ownerId and temp.string10=ms.fieldId
> > > > > and type.actionName='addPrivilege' and
> > > > > type.changeLogCategory='privilege' and
> > > > > type.id=temp.changeLogTypeId) and
> > > > > not exists (select 1 from ChangeLogEntryTemp temp, ChangeLogType type,
> > > > > Member m, MembershipEntry ms2 where temp.string09=:fieldId and
> > > > > type.actionName='addMembership' and
> > > > > type.changeLogCategory='membership'
> > > > > and type.id=temp.changeLogTypeId and m.subjectIdDb=temp.string06 and
> > > > > m.subjectSourceIdDb='g:gsa' and ms2.memberUuid = m.uuid and
> > > > > ms.ownerId=ms2.ownerId and ms.fieldId = ms2.fieldId and
> > > > > ms2.enabledDb='T') order by ms.immediateMembershipId, ms.groupSetId',
> > > > > cacheable: false, cacheRegion:
> > > > > edu.internet2.middleware.grouper.internal.dao.hib3.Hib3FlatMembershipDAO.FindMissingFlatMemberships,
> > > > >
> > > > >
> > > > > tx type: null, options: QueryOptions: queryPaging: pageSize: 100000,
> > > > > pageNumberOnIndexed: 1, , tx type: nullBind var[0]: 'Param (class
> > > > > java.lang.String): 'fieldId'->'8007cdc65bc24e928c13ab7a9ea3fae2',
> > > > > at
> > > > > edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch(HibernateSession.java:424)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:510)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.hibernate.ByHqlStatic.list(ByHqlStatic.java:346)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.hibernate.ByHqlStatic.listSet(ByHqlStatic.java:395)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.internal.dao.hib3.Hib3FlatMembershipDAO.findMissingFlatMemberships(Hib3FlatMembershipDAO.java:288)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.misc.SyncFlatTables.addMissingFlatMemberships(SyncFlatTables.java:341)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.misc.SyncFlatTables.syncAllFlatTables(SyncFlatTables.java:158)
> > > > >
> > > > >
> > > > > at
> > > > > edu.internet2.middleware.grouper.misc.GrouperReport.runReport(GrouperReport.java:195)
> > > > >
> > > > >
> > > > > ... 5 more
> > > > > Caused by: org.hibernate.exception.SQLGrammarException: could not
> > > > > update:
> > > > > [MembershipEntry#component[uuid,immediateMembershipId,groupSetId]{groupSetId=55d3a30ada5040febdd7ac5ddc504ae5,
> > > > >
> > > > >
> > > > > immediateMembershipId=00554d01d4eb451e93305b898f81adf3,
> > > > > uuid=00554d01d4eb451e93305b898f81adf3:55d3a30ada5040febdd7ac5ddc504ae5}]
> > > > >
> > > > > at
> > > > > org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:67)
> > > > >
> > > > >
> > > > > at
> > > > > org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
> > > > >
> > > > >
> > > > > at
> > > > > org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2430)
> > > > >
> > > > >
> > > > > at
> > > > > org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2312)
> > > > >
> > > > >
> > > > > at
> > > > > org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2612)
> > > > >
> > > > >
> > > > > at
> > > > > org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:96)
> > > > >
> > > > >
> > > > > <snip>
> > > > > edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:500)
> > > > >
> > > > >
> > > > > ... 11 more
> > > > > Caused by: java.sql.SQLException: ORA-01779: cannot modify a column
> > > > > which maps to a non key-preserved table
> > > > >
> > > > > at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:439)
> > > > >
> > > > >
> > > > > I can easily duplicate this with gsh: "new
> > > > > SyncFlatTables().sendNotifications(false).syncAllFlatTables();", but
> > > > > debugging logs haven't revealed much of use. Finally, "gsh
> > > > > -findbadmemberships -all" fails with:
> > > > >
> > > > > org/hibernate/engine/TwoPhaseLoad.java:104:in `initializeEntity':
> > > > > org.hibernate.AssertionFailure: possible non-threadsafe access to the
> > > > > session, (NativeException)
> > > > > Problem in HibernateSession: HibernateSession: isNew: false,
> > > > > isReadonly:
> > > > > false, grouperTransactionType: READ_WRITE_NEW,
> > > > > Exception in list: (class [Ljava.lang.Object;), ByHqlStatic, query:
> > > > > 'select distinct ms, m from MembershipEntry as ms, Member as m,
> > > > > Field as
> > > > > field where ms.ownerGroupId = :owner and
> > > > > ms.memberUuid = :member and ms.fieldId = :fuuid and
> > > > > ms.memberUuid = m.uuid and ms.enabledDb = 'T'', cacheable: false,
> > > > > cacheRegion:
> > > > > edu.internet2.middleware.grouper.internal.dao.hib3.Hib3MembershipDAO.FindAllByGroupOwnerAndMemberAndField,
> > > > >
> > > > >
> > > > > tx type: null, tx type: nullBind var[0]: 'Param (class
> > > > > java.lang.String): 'owner'->'7a246c38-b60e-4b11-96e6-67683fe82237',
> > > > > Bind
> > > > > var[1]: 'Param (class java.lang.String):
> > > > > 'member'->'89fc30292ed14ea091d5e792c24038e8'Bind var[2]: 'Param (class
> > > > > java.lang.String): 'fuuid'->'8007cdc65bc24e928c13ab7a9ea3fae2', ,
> > > > > Exception in delete: edu.internet2.middleware.grouper.Membership,
> > > > > edu.internet2.middleware.grouper.hibernate.ByObject@6f65bb15,
> > > > > Problem in HibernateSession: HibernateSession: isNew: false,
> > > > > isReadonly:
> > > > > false, grouperTransactionType: READ_WRITE_NEW,
> > > > > Exception in delete: edu.internet2.middleware.grouper.Membership,
> > > > > ByObjectStatic, query: ', cacheable: null, cacheRegion: null,
> > > > > entityName: ImmediateMembershipEntry, tx type: null,
> > > > > Problem in HibernateSession: HibernateSession: isNew: false,
> > > > > isReadonly:
> > > > > false, grouperTransactionType: READ_WRITE_NEW,
> > > > > Problem in HibernateSession: HibernateSession: isNew: true,
> > > > > isReadonly:
> > > > > false, grouperTransactionType: READ_WRITE_NEW
> > > > > from org/hibernate/loader/Loader.java:854:in
> > > > > `initializeEntitiesAndCollections'
> > > > > from org/hibernate/loader/Loader.java:729:in `doQuery'
> > > > > from org/hibernate/loader/Loader.java:236:in
> > > > > `doQueryAndInitializeNonLazyCollections'
> > > > > from org/hibernate/loader/Loader.java:2213:in `doList'
> > > > > from org/hibernate/loader/Loader.java:2104:in `listIgnoreQueryCache'
> > > > > from org/hibernate/loader/Loader.java:2099:in `list'
> > > > > from org/hibernate/loader/hql/QueryLoader.java:378:in `list'
> > > > > from org/hibernate/hql/ast/QueryTranslatorImpl.java:338:in `list'
> > > > > ... 35 levels...
> > > > > from run.rb:50
> > > > > from run.rb:40:in `each'
> > > > > from run.rb:40
> > > > >
> > > > > Several of our own loader jobs and scripts (using the API) generate
> > > > > the
> > > > > same error.