Grouper Users - Open Discussion List

["Black, Carey M." <>]

I was doing some "maintenance checks" and was surprised to find that when I 
ran " new 
edu.internet2.middleware.grouper.misc.SyncPITTables().syncAllPITTables();" 
Things were found to be missing.

Some of the items appeared to be corrected up to a point during the run of 
the syncAllPITTables(). 

However, the process bombs out:
"
Searching for missing active point in time memberships
Found 592949 missing active point in time memberships
ERROR edu.internet2.middleware.grouper.internal.dao.GrouperDAOException:
Problem in HibernateSession: HibernateSession (6df09e64): notNew, 
notReadonly, READ_WRITE_NEW, activeTransaction, session (186514ee),
Exception in save: edu.internet2.middleware.grouper.changeLog.ChangeLogEntry, 
ByObjectStatic, query: ', cacheable: null, cacheRegion: null, entityName: 
ChangeLogEntryEntity, tx type: null,
Exception in save: edu.internet2.middleware.grouper.pit.PITMembership, 
edu.internet2.middleware.grouper.hibernate.ByObject@7adb263a,
Problem in HibernateSession: HibernateSession (57da5aba): new, notReadonly, 
READ_WRITE_NEW, notActiveTransaction, session (186514ee),
Exception in saveOrUpdate: 
edu.internet2.middleware.grouper.pit.PITMembership, ByObjectStatic, query: ', 
cacheable: null, cacheRegion: null, entityName: null, tx type: null
        at 
edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch
 (HibernateSession.java:588)
        at 
edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession
 (HibernateSession.java:710)
        at 
edu.internet2.middleware.grouper.hibernate.ByObjectStatic.saveBatch 
(ByObjectStatic.java:535)
        at 
edu.internet2.middleware.grouper.internal.dao.hib3.Hib3ChangeLogEntryDAO.saveBatch
 (Hib3ChangeLogEntryDAO.java:61)
        at edu.internet2.middleware.grouper.pit.PITMembership.onPreSave 
(PITMembership.java:517)
        at edu.internet2.middleware.grouper.hibernate.ByObject.saveOrUpdate 
(ByObject.java:375)
        at 
edu.internet2.middleware.grouper.hibernate.ByObjectStatic$5.callback 
(ByObjectStatic.java:376)
        at 
edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession
 (HibernateSession.java:700)
        at 
edu.internet2.middleware.grouper.hibernate.ByObjectStatic.saveOrUpdate 
(ByObjectStatic.java:363)
        at 
edu.internet2.middleware.grouper.internal.dao.hib3.Hib3PITMembershipDAO.saveOrUpdate
 (Hib3PITMembershipDAO.java:41)
        at edu.internet2.middleware.grouper.pit.PITMembership.save 
(PITMembership.java:267)
        at 
edu.internet2.middleware.grouper.misc.SyncPITTables.processMissingActivePITMemberships
 (SyncPITTables.java:328)
        at 
edu.internet2.middleware.grouper.misc.SyncPITTables.syncAllPITTables 
(SyncPITTables.java:236)
        at 
edu.internet2.middleware.grouper.misc.SyncPITTables$syncAllPITTables.call 
(Unknown Source)
"

OK... 592k is a fairly large set....

I tracked the stacktrace back to lines 280(+ ish) of 
edu.internet2.middleware.grouper.misc.SyncPITTables.java .

And since I see the log output of " Found 592949 missing active point in time 
memberships"... I appear to be able to execute through line 286...
Which is just before a for loop. :)

So... to try to figure out what is going on I dropped into gsh to see what I 
could see with a manual iteration of the code. :)

// root sesson
rsess = GrouperSession.startRootSession();
// find the missing memberships
mships = 
GrouperDAOFactory.getFactory().getPITMembership().findMissingActivePITMemberships();
///.... A long time passes.....
/// And a lot of output happens. (I am guessing 592949 worth of a 
"Set<Membership>" :) ) 
///     OK... :)
mships.size()
// ===> 590930
//
//    hum.... not exactly the same count as when it failed... 
//    maybe it fixed some before it failed?
//
// Get the iterator... for the for loop and manually try to do the first one 
returned....
Iter = mships.iterator();
// make sure it has something
iter.hasNext();
//===> true
// get the first member
mship = iter.next();
//===> 
Membership[createTime=1522953142643,creatorUuid=10e2f04137854100be0a0b6688200b9a,depth=0,listName=members,listType=list,memberUuid=d4e035ab57514286928ec105e34c4edb,groupId=8a9982154ecc432089b2a5d3ade41a36,type=immediate]
// get the pitField
pitField = 
GrouperDAOFactory.getFactory().getPITField().findBySourceIdActive(mship.getFieldId(),
 true)
// get the pitMember
pitMember = 
GrouperDAOFactory.getFactory().getPITMember().findBySourceIdActive(mship.getMemberUuid(),
 true);
// set a new UUID for the membership
pitMembership.setId(edu.internet2.middleware.grouper.internal.util.GrouperUuid.getUuid());
//===> null
///  ^^^^^^ null ??? Is that right????
pitMembership.getId();
//===> 17bd3c3339d44bb6bf87aa947ad99e1e
// oh.. it has one.. it just did not get echoed... ok... moving on...
pitMembership.setMemberId(pitMember.getId());
// ===> null
pitMembership.getMemberId();
// ===> 0bb0d58dee9249268a7ac31962ab6c73
pitMembership.setFieldId(pitField.getId());
// ===> null
pitMembership.getFieldId();
// ===> f5a893ef1a764c039f8c8286a47e8488
pitMembership.setActiveDb("T");
// ===> null
pitMembership.getActiveDb();
//===> T
pitMembership.setStartTimeDb(System.currentTimeMillis() * 1000);
// ===> null
pitMembership.getStartTimeDb();
// ===> 1523657695155000
pitMembership.setOwnerGroupId(GrouperDAOFactory.getFactory().getPITGroup().findBySourceIdActive(mship.getOwnerGroupId(),
 true).getId());
// ===> null
pitMembership.getOwnerGroupId();
// ===> 8e465160bb8a4541b4b466da3bc3788b
pitMembership.setOwnerGroupId(GrouperDAOFactory.getFactory().getPITGroup().findBySourceIdActive(mship.getOwnerGroupId(),
 true).getId());
//===> null
mship.getContextId()
//===> 88d48142c2574bb9b01c5f61eb9b9876
!GrouperUtil.isEmpty(mship.getContextId())
//===> true
pitMembership.setContextId(mship.getContextId());
//===> null
pitMembership.getContextId();
//===> 88d48142c2574bb9b01c5f61eb9b9876
includeFlattenedMemberships = 
GrouperLoaderConfig.retrieveConfig().propertyValueBoolean("changeLog.includeFlattenedMemberships",
 true);
//===> true
includeFlattenedPrivileges = 
GrouperLoaderConfig.retrieveConfig().propertyValueBoolean("changeLog.includeFlattenedPrivileges",
 true);
// ===> true
pitMembership.setFlatMembershipNotificationsOnSaveOrUpdate(includeFlattenedMemberships);
// ===> null
pitMembership.setFlatPrivilegeNotificationsOnSaveOrUpdate(includeFlattenedPrivileges);
// ===> null
includeRolesWithPermissionChanges = 
GrouperLoaderConfig.retrieveConfig().propertyValueBoolean("changeLog.includeRolesWithPermissionChanges",
 false);
//===> false
groovy:000> 
pitMembership.setNotificationsForRolesWithPermissionChangesOnSaveOrUpdate(includeRolesWithPermissionChanges);
//===> null
pitMembership.save();

// and boom .. .on the first one out of the set .   :)
"
ERROR edu.internet2.middleware.grouper.internal.dao.GrouperDAOException:
Problem in HibernateSession: HibernateSession (75413bc6): new, notReadonly, 
READ_WRITE_NEW, notActiveTransaction, session (77b76a1b),
Exception in saveOrUpdate: 
edu.internet2.middleware.grouper.pit.PITMembership, ByObjectStatic, query: ', 
cacheable: null, cacheRegion: null, entityName: null, tx type: null
        at 
edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch
 (HibernateSession.java:588)
        at 
edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession
 (HibernateSession.java:710)
        at 
edu.internet2.middleware.grouper.hibernate.ByObjectStatic.saveOrUpdate 
(ByObjectStatic.java:363)
        at 
edu.internet2.middleware.grouper.internal.dao.hib3.Hib3PITMembershipDAO.saveOrUpdate
 (Hib3PITMembershipDAO.java:41)
        at edu.internet2.middleware.grouper.pit.PITMembership.save 
(PITMembership.java:267)
        at edu.internet2.middleware.grouper.pit.PITMembership$save$12.call 
(Unknown Source)
"


I suspect that the root source of all of these are related to some large 
groups being loaded by the Loader process and things not "always working out".
However, the "fix it" process (AKA: syncAllPITTables()) fails.

Any ideas? 
Is this addressed in a later patch? Than I am on? ( 
grouper_v2_3_0_api_patch_79.state = applied  ... Yea.. I have a few to 
install yet....)

Thanks in advance.

-- 
Carey Matthew