Grouper Developers Forum

[Chris Hyzer <>]

Do you mean inserting records into the change log table, or just making 
change log entry beans and processing them as you would off the change log?  
For the former, you shouldn't do it since other things read that table.  For 
the latter, if you aren't sacrificing performance due to already knowing some 
of the answers that the incremental has to figure out but the full sync might 
already know, then go for it I guess.

Note: if the incremental is running, and you need to do a full sync in the 
middle of the night, there will be some overlap.  i.e. maybe the full sync 
decides to add a member to a group, but the incremental also wants to do 
that.  Each process will need to ignore it if the member if already in the 
group, right?  Are we stopping the incremental while the full sync if running 
or will they run at the same time for a certain period of time?

Thanks,
Chris


-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Tom Zeller
Sent: Wednesday, October 19, 2011 12:02 PM
To: Grouper Dev
Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
clobbering avoidance ?

After taking a refactoring detour, I am getting back to figuring out
how to integrate (1) provisioning based on the changelog as well as
(2) optional scheduled full synchronizations (the current style of
operation).

The issue I have is that to take advantage of caching I would like the
changelog and full-sync jobs to use the same instance of the
provisioning service provider (PSP.java).

So...it seems old-fashioned but my thought is to insert changelog
entries to emulate the current full-sync style jobs. In other words,
at some scheduled time (e.g. via a grouper-loader.properties cron
statement), for every group, stem, membership, etc insert new
GROUP_UPDATE, STEM_UPDATE, MEMBERSHIP_UPDATE changelog entries even
though nothing has actually changed. This process would also be used
to bootstrap a new provisioned target.

What do you think ?

On Wed, Sep 28, 2011 at 10:34 AM, Jim Fox 
<>
 wrote:
>
> We provision our ldap directly off the change log (a variant of it) and use
> ActiveMQ to provision everything else.  We run an off line process that
> periodically compares grouper's registry with ldap and reports differences.
> So the only automatic updates to ldap are naturally in proper time order.
>
> Jim
>
>
> On Wed, 28 Sep 2011, McDermott, Michael wrote:
>
>> Date: Wed, 28 Sep 2011 02:06:24 -0700
>> From: "McDermott, Michael" 
>> <>
>> To: Tom Zeller 
>> <>
>> Cc: Grouper Dev 
>> <>
>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and
>>    clobbering avoidance ?
>>
>> Locally we are looking at solving this problem with the change log and a
>> queue for exactly this reason.  We're using a an independent, remote
>> queue, but as I read the features of ActiveMQ (Apache's JMS offering), you
>> can embed it and use it as part of an application.  To paraphrase a
>> quip about regular expressions:  'You come across a problem where it seems
>> like you can setup a simple queue in a database; now you have two
>> problems.'
>>
>>
>>
>> On Tue, Sep 27, 2011 at 10:48 PM, Tom Zeller 
>> <>
>>  wrote:
>>      How do we want to schedule real-time and batch provisioning ?
>>
>>      Currently, ldappcng has
>>
>>       -interval <seconds>
>>
>>      and
>>
>>       -intervalFullSync <seconds>
>>
>>      command line options to control how often provisioning runs are
>> performed.
>>
>>      The -interval <seconds> option looks at lastModifyTime to determine
>> if
>>      objects should be updated, and I think we will discontinue this in
>>      lieu of the changelog.
>>
>>      So, I assume, every few (configurable) seconds ldappcng will
>> provision
>>      (incrementally) based on the changelog.
>>
>>      And, I assume, deployers will want to schedule batch / full sync
>>      provisioning runs at night (or maybe more often) to make sure
>>      everything is properly provisioned.
>>
>>      However, let us assume a situation where a full sync of a large group
>>      takes a long time (e.g. 1 minute). During the full sync of the large
>>      group, membership changes might occur, which will be queued in the
>>      changelog. After the full sync of the large group is finished and we
>>      begin processing the changelog, we should not process changelog
>>      entries that were queued before the group was fully synchronized but
>>      after the full sync was started. So, I think we will need to store
>> the
>>      last update time of provisioned objects to not clobber ourselves.
>>
>>      We may also want to (optionally) schedule a full sync of a
>> provisioned
>>      object if provisioning incrementally based on the changelog
>>      experiences an error.
>>
>>      What do you think ?