Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] changelogTemp to changelog main processing

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] changelogTemp to changelog main processing

Chronological Thread 
  • From: Michael R Gettes <>
  • To: Shilen Patel <>
  • Cc: grouper-users <>
  • Subject: Re: [grouper-users] changelogTemp to changelog main processing
  • Date: Tue, 14 Feb 2017 11:30:05 -0500
  • Ironport-phdr: 9a23:25/rDh+iqgehyv9uRHKM819IXTAuvvDOBiVQ1KB41ekcTK2v8tzYMVDF4r011RmSDNids6oP1beempujcFRI2YyGvnEGfc4EfD4+ouJSoTYdBtWYA1bwNv/gYn9yNs1DUFh44yPzahANS47xaFLIv3K98yMZFAnhOgppPOT1HZPZg9iq2+yo9ZDeZwVFiCChbb9uIxm7rAXcvdQKjIV/Lao81gHHqWZSdeRMwmNoK1OTnxLi6cq14ZVu7Sdete8/+sBZSan1cLg2QrJeDDQ9LmA6/9brugXZTQuO/XQTTGMbmQdVDgff7RH6WpDxsjbmtud4xSKXM9H6QawyVD+/9KpgVgPmhzkbOD446GHXi9J/jKRHoBK6uhdzx5fYbJyJOPZie6/Qe84RS2hcUcZLTyFOHoyzYZYPAeUDM+hWoIrzp1UQoxW5HgSsGPrvyjpUin/2waE2zeIsGhzG0gw6GNIOtWzZotL0NKgOUeC61rfHzTHeZP1KxDzz6ZbHcgw9of6SRrJ7bM3cyUw1FwPKgFictZfoPyuO1uQQqWSU8fdvVf+2hmMhtgp/oSCvy98xhoXXhY8Z0E3I+Th6zYovONG1R092bcS5HJZetiyXMZZ9TNk4TGFyoik6z6ULuZ6lcygOz5Qq3wLfa+aZf4SW5BLvSfudLi1iiH1/Y7KwmQqy/VK4yu3nS8m4ykhFoTdYktXUt3AN0QLc6tSfR/dg4Eus2iyD2x3O5uxHO0w4iKXWJp87zrItmJcesFzPHirsl0X3iK+WeF8k+u+t6+n/Z7XmvJCcOoFohgzlKqQugdG/Df4mPQcTQmiX4f6826H7/U3lXLVKieU7kqbDsJDdOMQbvrC2AxVM3oY+8BawES2m0M8DkHkDLVJFYw6Hj5P3N13UIfD4C+u/jEq2kDdt2f/GIqPtDo/TIXfejbeyNYp6vnVcyQ4+y5hn7o5ZDvlVO/LyXkL3nNDFDRJ/PgCplbXJEtJ4g6YfUmKGD6vRCuv9vEOU6/lnd+yWa9RNkC7mNr4o6+O43ixxokMUYaT8hchfU3u/BPkzZhzBOXc=

right now, my concern is performance. We have an issue in production we are
to work through (seeing 7.4 changes per second) but our acceptance environment
is getting 25.4 per second. I have been calculating the change rates based
on total_count
and number of seconds to run the changelog temp job. We get backed up 1 or 2
changes and it can take some time to burn down the changes to get them in the
change log. i just want to make sure Grouper is not the bottleneck in the

Thanks for explaining things - much appreciated.


> On Feb 14, 2017, at 11:18 AM, Shilen Patel
> <>
> wrote:
> Maybe. This was created back when the loader was single threaded and PSP
> was the main provisioner. Both of which were often the bottlenecks. Now
> with the loader being multi-threaded and PSPNG being much faster, the
> change log increasingly becomes an issue when you're loading a lot of data.
> The records are processed one at a time and in a single thread mainly to
> make sure that flattened membership and privilege changes are represented
> correctly in the change log. E.g. If you're added to a group in two
> different ways, there's only one flattened membership add in the change
> log. And it also ensures that changes are committed sequentially and in
> the right order.
> Also, from what I've seen in the past, the most expensive part of the
> process is determining the flattened changes. If you're not interested in
> that, you can actually disable it. That would improve performance and you
> can still get immediate membership changes.
> We can take another look to see what improvements may be possible. In the
> past, I think there were other thoughts floated around like using hooks to
> speed up provisioning. In any case, do you have more info on what
> specifically you're seeing?
> Thanks!
> - Shilen
> On 2/13/17, 3:14 PM, "Michael R Gettes"
> <>
> wrote:
>> From what I can tell, moving data from the temp changelog to the main
>> changelog is handled by
>> select from temp, insert to main, delete from temp
>> loop - one record at a time.
>> Can this be handled more efficiently? This seems to be the slowest part
>> of processing with Grouper. If there are good reasons for it being the
>> way it is - education is appreciated.
>> Thanks!
>> /mrg

Archive powered by MHonArc 2.6.19.

Top of Page