Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] Decoupling PIT table updates and change log consumers

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] Decoupling PIT table updates and change log consumers


Chronological Thread 
  • From: Shilen Patel <>
  • To: Peter DiCamillo <>
  • Cc: "" <>
  • Subject: Re: [grouper-users] Decoupling PIT table updates and change log consumers
  • Date: Tue, 18 Aug 2015 17:12:15 +0000
  • Accept-language: en-US
  • Authentication-results: spf=none (sender IP is ) ;
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:23

Hi Peter,

So the two are coupled together because the point in time data is used to
determine the flattened membership, privilege, and permission changes.

Usually it¹s that part (determining the flattened changes) that takes that
longest. And if there are a lot of flattened changes, there would be a
lot of inserts into the change log table.

But I¹ll take a look at the code to see what can be done. I may ping you
about getting more information about your environment so I can replicate
exactly what you¹re seeing.

Thanks!

- Shilen


On 8/18/15, 10:32 AM, "Peter DiCamillo"
<>
wrote:

>We use a change log consumer in Grouper for real-time provisioning, and
>it's key part of our provisioning process. Ordinarily that works very
>well. However, we've had problems recently and in the past where events
>are not sent to the change log consumer for minutes, or even hours
>after they occurred. My understanding is that this is caused by the
>Grouper loader doing at least two things with each change before going
>on to the next one: updating the Point-In-Time (PIT) tables, and sending
>the change to the change log consumer. The delays happen when a change
>requires extensive updates to the PIT tables. That can happen for us,
>since we have extensive nesting of groups, a very large number of
>groups, and some groups with a very large number of memberships.
>
>In our case, it's not a problem for us if it takes hours for the PIT
>tables to be updated. Our main use of them is checking the audit logs
>when debugging provisioning problems. However, as we become more
>dependent on real-time provisioning, delays in it can cause significant
>problems for us and for our users.
>
>It occurred to me that perhaps it's not necessary for Grouper to couple
>the PIT table updating to sending changes to a change log consumer. As a
>future enhancement, could those functions be separated? Maybe maintain
>separate queues of changes for PIT table updates and for the change log
>consumer, instead of one shared table? That would allow the change log
>consumer process events as fast as possible. Also, we could ensure that
>we don't lose any change log consumer events, even if there's some
>problem with the PIT table updating.
>
>Peter




Archive powered by MHonArc 2.6.16.

Top of Page