Grouper Users - Open Discussion List

[Scott Koranda <>]

Hi Mike,

Have you considered changing how often the change log temp to change
log daemon runs? The default is 50 seconds after every minute:

changeLog.changeLogTempToChangeLog.quartz.cron = 50 * * * * ?

I have a production deployment that is roughly the same as yours (from
what I can gather) and we run it every 5 seconds without problems:

changeLog.changeLogTempToChangeLog.quartz.cron = 0/5 * * * * ?

You could be less aggressive and only run it every 10 or 15 seconds,
but still pick up a factor of 4-6.

Note that we do run with 20 for the pool size rather than 10.

HTH,

Scott K


On Tue, Oct 7, 2014 at 4:11 PM, Mike Roszkowski
<>
 wrote:
> Hi,
>
> (Sorry if this is a duplicate message, I tried to send to the list and 
> forgot
> that
> my email address had changed due to an Office 365 migration.)
>
> After upgrading to 2.1.5, we made some sweeping permission changes to our
> groups and as a result, we have about 115,000 records in
> grouper_change_log_entry_temp waiting to be processed into
> grouper_change_log_entry.
>
> We have a pair of VMs running grouper, with one machine running the
> grouper loader.
>
> We¹re seeing performance of about 15 records per minute going from the
> temp log to the change log, which means we¹re looking at 127 hours to get
> through the changes already queued (and there are more changes that are
> going to be made to group permissions which will increase the backlog of
> records even further).
>
> The majority of changes (> 100000) are permission changes, which we don¹t
> even end up consuming.
>
> So, a few questions:
>
> - Any guidance on what we might be able to do to speed up processing of
> change log changes? For example, would increasing the threads in the
> thread pool help? More memory for the grouper loader process (we¹re
> running with -Xms1024m and -Xmx2048m now)?
>
> - Since we don¹t really care about consuming the privilege changes, would
> it be safe to shut down and remove all the records in
> grouper_change_log_entry_temp with change_log_category='privilege¹?
>
> The (I think) relevant settings in grouper-loader.properties are:
>
> loader.use.transactions = false
> loader.thread.pool.size=10
> changeLog.changeLogTempToChangeLog.enable = true
> changeLog.includeFlattenedMemberships = true
> changeLog.includeFlattenedPrivileges = false
> changeLog.includeRolesWithPermissionChanges = false
> changeLog.includeNonFlattenedMemberships = false
> changeLog.includeNonFlattenedPrivileges = false
>
> We changed includeFlattenedPrivileges to false after the 115000 changes
> were queued, thinking that might help reduce the number of priv changes
> for our next set of changes.
>
> Thanks for any advice you can offer.
> Mike Roszkowski
> DoIT Middleware, University of Wisconsin-Madison