Grouper Developers Forum

[Chris Hyzer <>]

Nothing except the change log process should be inserting stuff into the 
change log table.
I don't think inserting records into the change log table will solve the 
problem, you will always have race conditions

What about this part of my previous email:

> the full sync decides to add a member to a group, but the incremental also 
> wants to do that.  
> Each process will need to ignore it if the member if already in the group, 
> right?  

Can it be idempotent?  So if it is already done it doesn't fail?

Thanks,
Chris

-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Tom Zeller
Sent: Wednesday, October 19, 2011 12:57 PM
To: Chris Hyzer
Cc: Grouper Dev
Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
clobbering avoidance ?

> Do you mean inserting records into the change log table, or just making 
> change log entry beans and processing them as you would off the change log? 
>  For the former, you shouldn't do it since other things read that table.  
> For the latter, if you aren't sacrificing performance due to already 
> knowing some of the answers that the incremental has to figure out but the 
> full sync might already know, then go for it I guess.
> Note: if the incremental is running, and you need to do a full sync in the 
> middle of the night, there will be some overlap.  i.e. maybe the full sync 
> decides to add a member to a group, but the incremental also wants to do 
> that.  Each process will need to ignore it if the member if already in the 
> group, right?  Are we stopping the incremental while the full sync if 
> running or will they run at the same time for a certain period of time?

Even if we stop the incremental job, once it starts again it may do
the wrong thing. For example, if during a full sync (after the correct
provisioning has been calculated from grouper) a member is deleted,
when the incremental job processes the member deletion that will be
the wrong thing.

I was thinking of inserting records into the change log table, to
preserve order of operations and to avoid overlap.

So ... is it possible to insert records in the change log table so
that they are only read by a particular consumer ? There could be lots
of records, one per every object in grouper, including memberships.

If not, and we go the bean approach, I am not sure how to avoid
overlap. We could think of some ways, timestamps, locks, ugh. The idea
I was floating was to store the timestamp of every provisioned object
upon full synchronization, and check each change log entry before
processing it to make sure that the timestamp of the change log entry
is after the full sync timestamp.