Subject: Grouper Users - Open Discussion List
Re: [grouper-users] Decoupling PIT table updates and change log consumers
- From: Shilen Patel <>
- To: Peter DiCamillo <>
- Cc: "" <>
- Subject: Re: [grouper-users] Decoupling PIT table updates and change log consumers
- Date: Tue, 18 Aug 2015 17:12:15 +0000
- Accept-language: en-US
- Authentication-results: spf=none (sender IP is ) ;
- Spamdiagnosticmetadata: NSPM
- Spamdiagnosticoutput: 1:23
So the two are coupled together because the point in time data is used to
determine the flattened membership, privilege, and permission changes.
Usually it¹s that part (determining the flattened changes) that takes that
longest. And if there are a lot of flattened changes, there would be a
lot of inserts into the change log table.
But I¹ll take a look at the code to see what can be done. I may ping you
about getting more information about your environment so I can replicate
exactly what you¹re seeing.
On 8/18/15, 10:32 AM, "Peter DiCamillo"
>We use a change log consumer in Grouper for real-time provisioning, and
>it's key part of our provisioning process. Ordinarily that works very
>well. However, we've had problems recently and in the past where events
>are not sent to the change log consumer for minutes, or even hours
>after they occurred. My understanding is that this is caused by the
>Grouper loader doing at least two things with each change before going
>on to the next one: updating the Point-In-Time (PIT) tables, and sending
>the change to the change log consumer. The delays happen when a change
>requires extensive updates to the PIT tables. That can happen for us,
>since we have extensive nesting of groups, a very large number of
>groups, and some groups with a very large number of memberships.
>In our case, it's not a problem for us if it takes hours for the PIT
>tables to be updated. Our main use of them is checking the audit logs
>when debugging provisioning problems. However, as we become more
>dependent on real-time provisioning, delays in it can cause significant
>problems for us and for our users.
>It occurred to me that perhaps it's not necessary for Grouper to couple
>the PIT table updating to sending changes to a change log consumer. As a
>future enhancement, could those functions be separated? Maybe maintain
>separate queues of changes for PIT table updates and for the change log
>consumer, instead of one shared table? That would allow the change log
>consumer process events as fast as possible. Also, we could ensure that
>we don't lose any change log consumer events, even if there's some
>problem with the PIT table updating.
- [grouper-users] Decoupling PIT table updates and change log consumers, Peter DiCamillo, 08/18/2015
- Re: [grouper-users] Decoupling PIT table updates and change log consumers, Shilen Patel, 08/18/2015
Archive powered by MHonArc 2.6.16.