Grouper Users - Open Discussion List

[Michael R Gettes <>]

And, in case anyone is interested, here is what I use to do the calculation:

select 
  to_char ( sum (TOTAL_COUNT) / (sum (MILLIS)/1000), '9999.99') as 
AVERAGE_PER_SECOND,
  to_char ( sum (TOTAL_COUNT), '9999999') as TOTAL_CHANGES,
  to_char ( count ( MILLIS ), '999999') as TOTAL_JOBS,
  to_char ( sum (MILLIS) / 1000, '999999.99' ) as TOTAL_SECONDS
  from grouper_loader_log
  where ( job_name like '%CHANGE_LOG_ch%' ) and ENDED_TIME is not null 
    and (STARTED_TIME >= trunc(sysdate-1) -- today and yesterday
  )
  and TOTAL_COUNT > 0 ;

I am curious as to the performance seen by others.  We are on Oracle - but I 
am interested in other DB product numbers as well.

/mrg

> On Feb 14, 2017, at 11:30 AM, Michael R Gettes 
> <>
>  wrote:
> 
> right now, my concern is performance.  We have an issue in production we 
> are trying
> to work through (seeing 7.4 changes per second) but our acceptance 
> environment
> is getting 25.4 per second.  I have been calculating the change rates based 
> on total_count
> and number of seconds to run the changelog temp job.  We get backed up 1 or 
> 2 million
> changes and it can take some time to burn down the changes to get them in 
> the main
> change log.  i just want to make sure Grouper is not the bottleneck in the 
> future.
> 
> Thanks for explaining things - much appreciated.
> 
> /mrg
> 
>> On Feb 14, 2017, at 11:18 AM, Shilen Patel 
>> <>
>>  wrote:
>> 
>> Maybe.  This was created back when the loader was single threaded and PSP
>> was the main provisioner.  Both of which were often the bottlenecks.  Now
>> with the loader being multi-threaded and PSPNG being much faster, the
>> change log increasingly becomes an issue when you're loading a lot of data.
>> 
>> The records are processed one at a time and in a single thread mainly to
>> make sure that flattened membership and privilege changes are represented
>> correctly in the change log.  E.g. If you're added to a group in two
>> different ways, there's only one flattened membership add in the change
>> log.  And it also ensures that changes are committed sequentially and in
>> the right order.
>> 
>> Also, from what I've seen in the past, the most expensive part of the
>> process is determining the flattened changes.  If you're not interested in
>> that, you can actually disable it.  That would improve performance and you
>> can still get immediate membership changes.
>> 
>> We can take another look to see what improvements may be possible.  In the
>> past, I think there were other thoughts floated around like using hooks to
>> speed up provisioning.  In any case, do you have more info on what
>> specifically you're seeing?
>> 
>> Thanks!
>> 
>> - Shilen
>> 
>> 
>> 
>> On 2/13/17, 3:14 PM, "Michael R Gettes" 
>> <>
>>  wrote:
>> 
>>> From what I can tell, moving data from the temp changelog to the main
>>> changelog is handled by
>>> 
>>> select from temp, insert to main, delete from temp
>>> 
>>> loop - one record at a time.
>>> 
>>> Can this be handled more efficiently?  This seems to be the slowest part
>>> of processing with Grouper.  If there are good reasons for it being the
>>> way it is - education is appreciated.
>>> 
>>> Thanks!
>>> 
>>> /mrg
>>> 
>> 
>