Grouper Users - Open Discussion List

[Mike Roszkowski <>]

Hi,

(Sorry if this is a duplicate message, I tried to send to the list and forgot
that
my email address had changed due to an Office 365 migration.)

After upgrading to 2.1.5, we made some sweeping permission changes to our
groups and as a result, we have about 115,000 records in
grouper_change_log_entry_temp waiting to be processed into 
grouper_change_log_entry.

We have a pair of VMs running grouper, with one machine running the
grouper loader.

We¹re seeing performance of about 15 records per minute going from the
temp log to the change log, which means we¹re looking at 127 hours to get
through the changes already queued (and there are more changes that are
going to be made to group permissions which will increase the backlog of
records even further).

The majority of changes (> 100000) are permission changes, which we don¹t
even end up consuming.

So, a few questions:

- Any guidance on what we might be able to do to speed up processing of
change log changes? For example, would increasing the threads in the
thread pool help? More memory for the grouper loader process (we¹re
running with -Xms1024m and -Xmx2048m now)?

- Since we don¹t really care about consuming the privilege changes, would
it be safe to shut down and remove all the records in
grouper_change_log_entry_temp with change_log_category='privilege¹?

The (I think) relevant settings in grouper-loader.properties are:

loader.use.transactions = false
loader.thread.pool.size=10
changeLog.changeLogTempToChangeLog.enable = true
changeLog.includeFlattenedMemberships = true
changeLog.includeFlattenedPrivileges = false
changeLog.includeRolesWithPermissionChanges = false
changeLog.includeNonFlattenedMemberships = false
changeLog.includeNonFlattenedPrivileges = false

We changed includeFlattenedPrivileges to false after the 115000 changes
were queued, thinking that might help reduce the number of priv changes
for our next set of changes.

Thanks for any advice you can offer.
Mike Roszkowski
DoIT Middleware, University of Wisconsin-Madison