Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] changelogTemp to changelog main processing

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] changelogTemp to changelog main processing

Chronological Thread 
  • From: Michael R Gettes <>
  • To: Shilen Patel <>
  • Cc: grouper-users <>
  • Subject: Re: [grouper-users] changelogTemp to changelog main processing
  • Date: Tue, 14 Feb 2017 13:39:36 -0500
  • Ironport-phdr: 9a23:zZ7/UhLBjDntExUJYtmcpTZWNBhigK39O0sv0rFitYgRLP/xwZ3uMQTl6Ol3ixeRBMOAuq8C07Kd4viocFdDyK7JiGoFfp1IWk1NouQttCtkPvS4D1bmJuXhdS0wEZcKflZk+3amLRodQ56mNBXdrXKo8DEdBAj0OxZrKeTpAI7SiNm82/yv95HJbQhFgDqwbaluIBmqogncstUaipZ+J6gszRfEvmFGcPlMy2NyIlKTkRf85sOu85Nm7i9dpfEv+dNeXKvjZ6g3QqBWAzogM2Au+c3krgLDQheV5nsdSWoZjBxFCBXY4R7gX5fxtiz6tvdh2CSfIMb7Q6w4VSik4qx2UxLjljsJOCAl/2HWksxwjbxUoBS9pxxk3oXYZJiZOOdicq/BeN8XQ3dKUMRMWCxbGo6zc4QAAfcBM+laoYfzqFgArRWgCwerH+7v1iZIhnrq0a06z+gtDwfL1xEgEdIUt3TUqc34OqATUe+pzKnH1yvMb/dL0jnh9YPGcw4uoe2QXb1uasra1E4iFwbfjlWfp4HpIyiY1vwWs2iG9eZvS/+gi3M+pgx3vzOhxd8sh5HUio4LyV3I7zh1zJs2KNGiVUJ2bt+pHIFNuyyUNoZ7RN4pTXtytyYg0LIGvIa2fCgUx5QjwB7Sc+CHc5KW7h79UuadPCp0i2h/dL2lnBa971SgxvfkVsmvzVZGtDRKncTRtnwV1hzT7NaISudl80u8xzqC1Brf5vxZLU03j6bXNpEsz78qmpYOv0nOHzf6mEDsg6+XckUk9PKo6+PiYrj+o5+cLYB0hw/iPqk1gsy/Gvk4PhIQUGeG5OSwzKfj8lHhQLVWkv02lbHUsJ/AJcQcu665GxFa0pw65xaiFDepzs4YnWIcIVJBeRKHlJTpO0rQLPziDPe/hUisnylxx/DAILLhHovBImLdn7j8YLYuo3JbnTEzzNRS4dpsA60MJLqnQU/wsNHeJhMkMAHyzur6XoZTzIQbDEeOCa+QPaeaixen6/4zLvLEMIoPt22lA+A++rjjgWJvygxVRrWgwZZCMCPwJf9hOUjMJCO02to=

And, in case anyone is interested, here is what I use to do the calculation:

to_char ( sum (TOTAL_COUNT) / (sum (MILLIS)/1000), '9999.99') as
to_char ( sum (TOTAL_COUNT), '9999999') as TOTAL_CHANGES,
to_char ( count ( MILLIS ), '999999') as TOTAL_JOBS,
to_char ( sum (MILLIS) / 1000, '999999.99' ) as TOTAL_SECONDS
from grouper_loader_log
where ( job_name like '%CHANGE_LOG_ch%' ) and ENDED_TIME is not null
and (STARTED_TIME >= trunc(sysdate-1) -- today and yesterday
and TOTAL_COUNT > 0 ;

I am curious as to the performance seen by others. We are on Oracle - but I
am interested in other DB product numbers as well.


> On Feb 14, 2017, at 11:30 AM, Michael R Gettes
> <>
> wrote:
> right now, my concern is performance. We have an issue in production we
> are trying
> to work through (seeing 7.4 changes per second) but our acceptance
> environment
> is getting 25.4 per second. I have been calculating the change rates based
> on total_count
> and number of seconds to run the changelog temp job. We get backed up 1 or
> 2 million
> changes and it can take some time to burn down the changes to get them in
> the main
> change log. i just want to make sure Grouper is not the bottleneck in the
> future.
> Thanks for explaining things - much appreciated.
> /mrg
>> On Feb 14, 2017, at 11:18 AM, Shilen Patel
>> <>
>> wrote:
>> Maybe. This was created back when the loader was single threaded and PSP
>> was the main provisioner. Both of which were often the bottlenecks. Now
>> with the loader being multi-threaded and PSPNG being much faster, the
>> change log increasingly becomes an issue when you're loading a lot of data.
>> The records are processed one at a time and in a single thread mainly to
>> make sure that flattened membership and privilege changes are represented
>> correctly in the change log. E.g. If you're added to a group in two
>> different ways, there's only one flattened membership add in the change
>> log. And it also ensures that changes are committed sequentially and in
>> the right order.
>> Also, from what I've seen in the past, the most expensive part of the
>> process is determining the flattened changes. If you're not interested in
>> that, you can actually disable it. That would improve performance and you
>> can still get immediate membership changes.
>> We can take another look to see what improvements may be possible. In the
>> past, I think there were other thoughts floated around like using hooks to
>> speed up provisioning. In any case, do you have more info on what
>> specifically you're seeing?
>> Thanks!
>> - Shilen
>> On 2/13/17, 3:14 PM, "Michael R Gettes"
>> <>
>> wrote:
>>> From what I can tell, moving data from the temp changelog to the main
>>> changelog is handled by
>>> select from temp, insert to main, delete from temp
>>> loop - one record at a time.
>>> Can this be handled more efficiently? This seems to be the slowest part
>>> of processing with Grouper. If there are good reasons for it being the
>>> way it is - education is appreciated.
>>> Thanks!
>>> /mrg

Archive powered by MHonArc 2.6.19.

Top of Page