Grouper Users - Open Discussion List

[Michael R Gettes <>]

right now, my concern is performance.  We have an issue in production we are 
trying
to work through (seeing 7.4 changes per second) but our acceptance environment
is getting 25.4 per second.  I have been calculating the change rates based 
on total_count
and number of seconds to run the changelog temp job.  We get backed up 1 or 2 
million
changes and it can take some time to burn down the changes to get them in the 
main
change log.  i just want to make sure Grouper is not the bottleneck in the 
future.

Thanks for explaining things - much appreciated.

/mrg

> On Feb 14, 2017, at 11:18 AM, Shilen Patel 
> <>
>  wrote:
> 
> Maybe.  This was created back when the loader was single threaded and PSP
> was the main provisioner.  Both of which were often the bottlenecks.  Now
> with the loader being multi-threaded and PSPNG being much faster, the
> change log increasingly becomes an issue when you're loading a lot of data.
> 
> The records are processed one at a time and in a single thread mainly to
> make sure that flattened membership and privilege changes are represented
> correctly in the change log.  E.g. If you're added to a group in two
> different ways, there's only one flattened membership add in the change
> log.  And it also ensures that changes are committed sequentially and in
> the right order.
> 
> Also, from what I've seen in the past, the most expensive part of the
> process is determining the flattened changes.  If you're not interested in
> that, you can actually disable it.  That would improve performance and you
> can still get immediate membership changes.
> 
> We can take another look to see what improvements may be possible.  In the
> past, I think there were other thoughts floated around like using hooks to
> speed up provisioning.  In any case, do you have more info on what
> specifically you're seeing?
> 
> Thanks!
> 
> - Shilen
> 
> 
> 
> On 2/13/17, 3:14 PM, "Michael R Gettes" 
> <>
>  wrote:
> 
>> From what I can tell, moving data from the temp changelog to the main
>> changelog is handled by
>> 
>> select from temp, insert to main, delete from temp
>> 
>> loop - one record at a time.
>> 
>> Can this be handled more efficiently?  This seems to be the slowest part
>> of processing with Grouper.  If there are good reasons for it being the
>> way it is - education is appreciated.
>> 
>> Thanks!
>> 
>> /mrg
>> 
>