Grouper Users - Open Discussion List

[Nathan Weyenberg <>]

​I overestimated a bit, its been 7 hours for processing the fifth large group add.

--Nathan Weyenberg

UW-Madison

---

From: <> on behalf of Nathan Weyenberg <>
Sent: Thursday, October 09, 2014 11:20 AM
To: Shilen Patel
Cc: Mike Roszkowski;
Subject: Re: [grouper-users] ChangeLogTempToChangeLog processing questions

 

Hey Shilen thank you for the help thus far!

Tables have been analyzed, oracle seems to be humming along as best we can tell since we applied the patch.Since restarting the grouper loader the CHANGE_LOG_changeLogTempToChangeLog job has been processing the permission changes very quickly.  

We also have some membership adds and deletes of large groups queued up to be processed. We made it through 4/5 of the large group adds and deletes, but we have been processing the fifth one for approximately 11 hours. The first four similarly sized group adds took substantially less time. Would restarting the grouper loader be advised?  If so, would we lose any records that had been processed from the temp change log but not made it into the grouper_change_log_entry?

--Nathan Weyenberg

UW-Madison

---

From: Shilen Patel <>
Sent: Tuesday, October 07, 2014 9:25 PM
To: Nathan Weyenberg
Cc: Mike Roszkowski;
Subject: Re: [grouper-users] ChangeLogTempToChangeLog processing questions

 

Hey,

A few thoughts.

1.  Have you recently analyzed your tables?  If not, try that and see if that improves the rate.

2.  Are you using Oracle?  Can your DBAs tell if there's a query that's performing slowly?

3.  If it's still a problem, can you see if this patch helps?  (Modify source, recompile jar, restart daemon)

---------------------------------------

Index: src/grouper/edu/internet2/middleware/grouper/changeLog/ChangeLogTempToEntity.java

===================================================================

--- src/grouper/edu/internet2/middleware/grouper/changeLog/ChangeLogTempToEntity.java

+++ src/grouper/edu/internet2/middleware/grouper/changeLog/ChangeLogTempToEntity.java

@@ -921,7 +921,7 @@

     boolean includeFlattenedPrivileges = GrouperLoaderConfig.getPropertyBoolean("changeLog.includeFlattenedPrivileges", true);

     boolean includeRolesWithPermissionChanges = GrouperLoaderConfig.getPropertyBoolean("changeLog.includeRolesWithPermissionChanges", false);

     pitMembership.setNotificationsForRolesWithPermissionChangesOnSaveOrUpdate(includeRolesWithPermissionChanges);

-    pitMembership.setFlatMembershipNotificationsOnSaveOrUpdate(includeFlattenedMemberships);

+    pitMembership.setFlatMembershipNotificationsOnSaveOrUpdate(false);

     pitMembership.setFlatPrivilegeNotificationsOnSaveOrUpdate(includeFlattenedPrivileges);

     

     pitMembership.save();

@@ -958,7 +958,7 @@

     boolean includeFlattenedPrivileges = GrouperLoaderConfig.getPropertyBoolean("changeLog.includeFlattenedPrivileges", true);

     boolean includeRolesWithPermissionChanges = GrouperLoaderConfig.getPropertyBoolean("changeLog.includeRolesWithPermissionChanges", false);

     pitMembership.setNotificationsForRolesWithPermissionChangesOnSaveOrUpdate(includeRolesWithPermissionChanges);

-    pitMembership.setFlatMembershipNotificationsOnSaveOrUpdate(includeFlattenedMemberships);

+    pitMembership.setFlatMembershipNotificationsOnSaveOrUpdate(false);

     pitMembership.setFlatPrivilegeNotificationsOnSaveOrUpdate(includeFlattenedPrivileges);

     

     pitMembership.update();

---------------------------------------

Thanks!

-- Shilen

From: Nathan Weyenberg <>
Date: Tuesday, October 7, 2014 6:47 PM
To: Shilen Patel <>
Cc: Mike Roszkowski <>, "" <>
Subject: Re: [grouper-users] ChangeLogTempToChangeLog processing questions

​ Hey Shilen,

We did restart the daemon after making the config change. While the temp change log is processing it does not appear to be inserting records into the table. The change_log_entry_temp is getting smaller (in terms of records) however.

—Nathan

​

P.S. sorry for a potential double e-mail, I had to resubscribe with my O365 e-mail address. 

---

From: Nathan Weyenberg
Sent: Tuesday, October 07, 2014 5:39 PM
To: Shilen Patel
Cc: Mike Roszkowski;
Subject: Re: [grouper-users] ChangeLogTempToChangeLog processing questions

 

​ Hey Shilen,

We did restart the daemon after making the config change and while the temp change log is processing it does not appear to be inserting records into the table.

—Nathan

On Oct 7, 2014, at 5:11 PM, Shilen Patel <> wrote:

Hi,

Just to be sure, you restarted the daemon after making the
includeFlattenedPrivileges config change?  Also, while the temp change log
is processing (at the 15 records per minute rate), are records being
inserted into grouper_change_log_entry?  If so, which categories/actions
are being inserted there?

Thanks!

-- Shilen

On 10/7/14 5:11 PM, "Mike Roszkowski" <> wrote:

Hi,

(Sorry if this is a duplicate message, I tried to send to the list and
forgot
that
my email address had changed due to an Office 365 migration.)

After upgrading to 2.1.5, we made some sweeping permission changes to our
groups and as a result, we have about 115,000 records in
grouper_change_log_entry_temp waiting to be processed into
grouper_change_log_entry.

We have a pair of VMs running grouper, with one machine running the
grouper loader.

We¹re seeing performance of about 15 records per minute going from the
temp log to the change log, which means we¹re looking at 127 hours to get
through the changes already queued (and there are more changes that are
going to be made to group permissions which will increase the backlog of
records even further).

The majority of changes (> 100000) are permission changes, which we don¹t
even end up consuming.

So, a few questions:

- Any guidance on what we might be able to do to speed up processing of
change log changes? For example, would increasing the threads in the
thread pool help? More memory for the grouper loader process (we¹re
running with -Xms1024m and -Xmx2048m now)?

- Since we don¹t really care about consuming the privilege changes, would
it be safe to shut down and remove all the records in
grouper_change_log_entry_temp with change_log_category='privilege¹?

The (I think) relevant settings in grouper-loader.properties are:

loader.use.transactions = false
loader.thread.pool.size=10
changeLog.changeLogTempToChangeLog.enable = true
changeLog.includeFlattenedMemberships = true
changeLog.includeFlattenedPrivileges = false
changeLog.includeRolesWithPermissionChanges = false
changeLog.includeNonFlattenedMemberships = false
changeLog.includeNonFlattenedPrivileges = false

We changed includeFlattenedPrivileges to false after the 115000 changes
were queued, thinking that might help reduce the number of priv changes
for our next set of changes.

Thanks for any advice you can offer.
Mike Roszkowski
DoIT Middleware, University of Wisconsin-Madison