Skip to Content.
Sympa Menu

grouper-dev - RE: [grouper-dev] [ldappcng] real-time & batch scheduling and clobbering avoidance ?

Subject: Grouper Developers Forum

List archive

RE: [grouper-dev] [ldappcng] real-time & batch scheduling and clobbering avoidance ?


Chronological Thread 
  • From: Chris Hyzer <>
  • To: Tom Zeller <>, Grouper Dev <>
  • Subject: RE: [grouper-dev] [ldappcng] real-time & batch scheduling and clobbering avoidance ?
  • Date: Wed, 19 Oct 2011 16:17:06 +0000
  • Accept-language: en-US

Do you mean inserting records into the change log table, or just making
change log entry beans and processing them as you would off the change log?
For the former, you shouldn't do it since other things read that table. For
the latter, if you aren't sacrificing performance due to already knowing some
of the answers that the incremental has to figure out but the full sync might
already know, then go for it I guess.

Note: if the incremental is running, and you need to do a full sync in the
middle of the night, there will be some overlap. i.e. maybe the full sync
decides to add a member to a group, but the incremental also wants to do
that. Each process will need to ignore it if the member if already in the
group, right? Are we stopping the incremental while the full sync if running
or will they run at the same time for a certain period of time?

Thanks,
Chris


-----Original Message-----
From:


[mailto:]
On Behalf Of Tom Zeller
Sent: Wednesday, October 19, 2011 12:02 PM
To: Grouper Dev
Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and
clobbering avoidance ?

After taking a refactoring detour, I am getting back to figuring out
how to integrate (1) provisioning based on the changelog as well as
(2) optional scheduled full synchronizations (the current style of
operation).

The issue I have is that to take advantage of caching I would like the
changelog and full-sync jobs to use the same instance of the
provisioning service provider (PSP.java).

So...it seems old-fashioned but my thought is to insert changelog
entries to emulate the current full-sync style jobs. In other words,
at some scheduled time (e.g. via a grouper-loader.properties cron
statement), for every group, stem, membership, etc insert new
GROUP_UPDATE, STEM_UPDATE, MEMBERSHIP_UPDATE changelog entries even
though nothing has actually changed. This process would also be used
to bootstrap a new provisioned target.

What do you think ?

On Wed, Sep 28, 2011 at 10:34 AM, Jim Fox
<>
wrote:
>
> We provision our ldap directly off the change log (a variant of it) and use
> ActiveMQ to provision everything else.  We run an off line process that
> periodically compares grouper's registry with ldap and reports differences.
> So the only automatic updates to ldap are naturally in proper time order.
>
> Jim
>
>
> On Wed, 28 Sep 2011, McDermott, Michael wrote:
>
>> Date: Wed, 28 Sep 2011 02:06:24 -0700
>> From: "McDermott, Michael"
>> <>
>> To: Tom Zeller
>> <>
>> Cc: Grouper Dev
>> <>
>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and
>>    clobbering avoidance ?
>>
>> Locally we are looking at solving this problem with the change log and a
>> queue for exactly this reason.  We're using a an independent, remote
>> queue, but as I read the features of ActiveMQ (Apache's JMS offering), you
>> can embed it and use it as part of an application.  To paraphrase a
>> quip about regular expressions:  'You come across a problem where it seems
>> like you can setup a simple queue in a database; now you have two
>> problems.'
>>
>>
>>
>> On Tue, Sep 27, 2011 at 10:48 PM, Tom Zeller
>> <>
>> wrote:
>>      How do we want to schedule real-time and batch provisioning ?
>>
>>      Currently, ldappcng has
>>
>>       -interval <seconds>
>>
>>      and
>>
>>       -intervalFullSync <seconds>
>>
>>      command line options to control how often provisioning runs are
>> performed.
>>
>>      The -interval <seconds> option looks at lastModifyTime to determine
>> if
>>      objects should be updated, and I think we will discontinue this in
>>      lieu of the changelog.
>>
>>      So, I assume, every few (configurable) seconds ldappcng will
>> provision
>>      (incrementally) based on the changelog.
>>
>>      And, I assume, deployers will want to schedule batch / full sync
>>      provisioning runs at night (or maybe more often) to make sure
>>      everything is properly provisioned.
>>
>>      However, let us assume a situation where a full sync of a large group
>>      takes a long time (e.g. 1 minute). During the full sync of the large
>>      group, membership changes might occur, which will be queued in the
>>      changelog. After the full sync of the large group is finished and we
>>      begin processing the changelog, we should not process changelog
>>      entries that were queued before the group was fully synchronized but
>>      after the full sync was started. So, I think we will need to store
>> the
>>      last update time of provisioned objects to not clobber ourselves.
>>
>>      We may also want to (optionally) schedule a full sync of a
>> provisioned
>>      object if provisioning incrementally based on the changelog
>>      experiences an error.
>>
>>      What do you think ?



Archive powered by MHonArc 2.6.16.

Top of Page