Grouper Developers Forum

[Chris Hyzer <>]

Note that the incremental should be reading the static with a method, and 
nothing outside of the full sync should be able to edit it, right?  :)

Thanks,
chris

-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Tom Zeller
Sent: Thursday, October 20, 2011 10:41 AM
To: Chris Hyzer
Cc: Grouper Dev
Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
clobbering avoidance ?

Yeah. I need a static "lock" variable and a static instance of the
provisioning class.

These statics will exist just in the loader jvm, right ?

I was avoiding statics because I was not sure what all might share the
instance of the provisioning class, like any other process controlled
by the loader.

I will give it a try.

On Thu, Oct 20, 2011 at 8:16 AM, Chris Hyzer 
<>
 wrote:
> Can you just set a static variable in a try/finally block which says the 
> full sync is running, and if so, just return the first index which means it 
> didn't make any progress.  I think (hope) that doesn't log an error...  
> alternately, the incremental could just sleep for 5 seconds in a loop until 
> the full is done (by static variable), no more incrementals will start 
> while the one didn't finish...
>
> Thanks,
> Chris
>
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Tom Zeller
> Sent: Wednesday, October 19, 2011 4:26 PM
> To: Chris Hyzer
> Cc: Grouper Dev
> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
> clobbering avoidance ?
>
>> Nothing except the change log process should be inserting stuff into the 
>> change log table.
>> I don't think inserting records into the change log table will solve the 
>> problem, you will always have race conditions
>>
>> What about this part of my previous email:
>>
>>> the full sync decides to add a member to a group, but the incremental 
>>> also wants to do that.
>>> Each process will need to ignore it if the member if already in the 
>>> group, right?
>>
>> Can it be idempotent?  So if it is already done it doesn't fail?
>
> Yup, for membership adds and deletes we have to search first, then
> perform necessary modifications. In other words :
>
> 1 calculate correct provisioning
> 2 lookup current provisioning
> 3 determine the difference between correct and current provisioning
> 4 apply any necessary modifications
>
> Here is a use case :
>
>  0 changelog : "add member to group"
>  1 incremental sync : "add member to group" - lookup membership
>  2 incremental sync : "add member to group" - add member to group
>
>  3 full sync : calculate correct provisioning of group
>  4 changelog : "delete member from group"
>
> If we perform the incremental sync while the full sync is running :
>
>  5a incremental sync : "delete member from group" : lookup membership
>  6a incremental sync : "delete member to group" - delete member from group
>  7a full sync : lookup current provisioning of group
>  8a full sync : diff - will add member to group
>  9a full sync : modify - add member to group
>
> Performing the incremental sync during the full sync will incorrectly
> provision the group.
>
> If we wait to perform the incremental sync until the full sync is
> finished running :
>
>  5b incremental sync : begin wait
>  6b full sync : lookup current provisioning of group
>  7b full sync : diff
>  8b full sync : modify
>  9b incremental sync : end wait
> 10b incremental sync : "delete member from group" : lookup membership
> 11b incremental sync : "delete member to group" - delete member from group
>
> Performing the incremental sync after the full sync completes will
> correctly provision the group.
>
> Here is a slightly different use case demonstrating the need for 
> idempotence :
>
>  0 changelog : "add member to group"
>  1 incremental sync : "add member to group" - lookup membership
>  2 incremental sync : "add member to group" - add member to group
>
>  3 changelog : "delete member from group"
>  4 full sync : calculate correct provisioning of group
>
>  5c incremental sync : begin wait
>  6c full sync : lookup current provisioning of group
>  7c full sync : diff - will delete member from group
>  8c full sync : modify - delete member from group
>  9c incremental sync : end wait
> 10b incremental sync : "delete member from group" : lookup membership
> 11b incremental sync : "delete member to group" - no changes necessary
>
> So, I guess the ldappcng change log consumer will need a cron entry in
> grouper-loader.properties to know when to trigger the full sync, and
> it will need to know that a full sync is running to delay provisioning
> of *any* change log entries until the full sync is complete.
>
> I had originally thought that we could delay provisioning per
> identifier, but it is simpler to just delay the whole incremental job.
> So, we can say we have real-time provisioning except when the full
> sync is running, which should be configurable and can default to once
> nightly.
>
> Are loader jobs Quartz Scheduler jobs in any way ? Can I extend
> StatefulJob somehow in the change log consumer class to prevent
> concurrency ? That would be a nice one-liner.
>