Grouper Developers Forum

[Tom Zeller <>]

Kind of horrible but here is what I have so far to schedule
incremental and full syncs without overlapping :

To schedule full syncs via grouper-loader.properties, a class which
provides a "public void fullSync()" method is cron'd :

 changeLog.consumer.ldappc.class =
edu.internet2.middleware.grouper.changeLog.LdappcngConsumer
 changeLog.consumer.ldappc.quartzCron = 0/5 * * * * ?

 # To run full synchronizations, specify the class which provides a
'public void fullSync()' method.
 changeLog.ldappcng.fullsync.class =
edu.internet2.middleware.grouper.changeLog.LdappcngConsumer
 changeLog.ldappcng.fullsync.quartzCron = 0 0/5 * * * ?

The fullSync() method is straightforward

public LdappcngConsumer {

 private static boolean fullSyncIsRunning;

 public synchronized void fullSync() {
  LOG.info("Ldappcng Consumer '{}' - Starting full sync", name);

   fullSyncIsRunning = true;

   psp.execute(new BulkSyncRequest());

   LOG.info("Ldappcng Consumer '{}' - Finished full sync", name);

   fullSyncIsRunning = false;
  }

and the processChangeLogEntries() method has :

for (ChangeLogEntry changeLogEntry : changeLogEntryList) {

 currentId = changeLogEntry.getSequenceNumber();

 if (fullSyncIsRunning) {
   LOG.debug("Ldappcng Consumer '{}' - Full sync is running, returning
sequence number '{}'", name, currentId - 1);
   return currentId - 1;
  }

  // process the change log entry
  processChangeLogEntry(changeLogEntry)
}

and here is the GrouperLoaderTypeEnum :

 public void runJob(LoaderJobBean loaderJobBean) {

   GrouperContext.createNewDefaultContext(GrouperEngineBuiltin.LOADER,
false, true);

    String theClassName =
GrouperLoaderConfig.getPropertyString("changeLog.ldappcng.fullsync.class");
    Class<?> theClass = GrouperUtil.forName(theClassName);
    Object theClassInstance = GrouperUtil.newInstance(theClass);

     try {
       theClass.getMethod("fullSync", new Class[]
{}).invoke(theClassInstance, new Object[] {});
      } catch (SecurityException e) {
       throw new RuntimeException(e);
      } catch (NoSuchMethodException e) {
        throw new RuntimeException(e);
      } catch (IllegalArgumentException e) {
        throw new RuntimeException(e);
      } catch (IllegalAccessException e) {
        throw new RuntimeException(e);
      } catch (InvocationTargetException e) {
        throw new RuntimeException(e);
      }
   }

I am open to suggestions.

Thanks,
TomZ

On Fri, Oct 28, 2011 at 4:11 PM, Chris Hyzer 
<>
 wrote:
> I think you need to run yours like you would one of the others from GSH (or 
> java main) and see whats missing...ok?
>
> Thanks,
> Chris
>
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Tom Zeller
> Sent: Friday, October 28, 2011 1:51 PM
> To: Chris Hyzer
> Cc: Grouper Dev
> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
> clobbering avoidance ?
>
> I guess I need help adding the ldappcng full sync job type to the loader.
>
> I added this to the GrouperLoaderType enum :
>
>    /**      * Run an ldappcng full sync.     */    LDAPPCNG_FULL_SYNC {
>       public boolean attributeRequired(String attributeName) {
>      return false;      }
>       public boolean attributeOptional(String attributeName) {
>      return false;      }
>       public void runJob(LoaderJobBean loaderJobBean) {
> GrouperContext.createNewDefaultContext(GrouperEngineBuiltin.LOADER,
> false, true);
>         Hib3GrouperLoaderLog hib3GrouploaderLog =
> loaderJobBean.getHib3GrouploaderLogOverall();                String
> theClassName = 
> GrouperLoaderConfig.getPropertyString("changeLog.ldappcng.fullsync.class");
>        Class<?> theClass = GrouperUtil.forName(theClassName);
>               GrouperLoaderJob job = (GrouperLoaderJob)
> GrouperUtil.newInstance(theClass);
> job.runJob(hib3GrouploaderLog, (Group)null,
> loaderJobBean.getGrouperSession());                }        };
>
> I copied the change log type enum, which instantiates a class defined
> in grouper-loader.properties, which here I assume extends Job.
>
> Is this correct ? I am not sure what to do.
>
> Thanks,
> TomZ
> On Fri, Oct 21, 2011 at 10:35 AM, Chris Hyzer 
> <>
>  wrote:
>> Sure, either that or make a new type for ldapppncg full sync... I have 
>> been globbing in the minor stuff into MAINTENANCE, either way :)  Ignore 
>> the required and optional attributes...  that more for the loader
>>
>> Thanks,
>> Chris
>>
>> -----Original Message-----
>> From: 
>> 
>>  
>> [mailto:]
>>  On Behalf Of Tom Zeller
>> Sent: Friday, October 21, 2011 11:29 AM
>> To: Chris Hyzer
>> Cc: Grouper Dev
>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
>> clobbering avoidance ?
>>
>> Right. But I am confused. Loader jobs are StatefulJobs :
>>
>>  public class GrouperLoaderJob implements Job, StatefulJob {
>>
>> so can I just add another Job to the loader to do the full-sync ?
>>
>> I see you added LDAP_SIMPLE, LDAP_GROUP_LIST, etc. Can I re-use
>> GrouperLoaderType.MAINTENANCE somehow ?
>>
>> On Thu, Oct 20, 2011 at 10:04 AM, Chris Hyzer 
>> <>
>>  wrote:
>>> Note that the incremental should be reading the static with a method, and 
>>> nothing outside of the full sync should be able to edit it, right?  :)
>>>
>>> Thanks,
>>> chris
>>>
>>> -----Original Message-----
>>> From: 
>>> 
>>>  
>>> [mailto:]
>>>  On Behalf Of Tom Zeller
>>> Sent: Thursday, October 20, 2011 10:41 AM
>>> To: Chris Hyzer
>>> Cc: Grouper Dev
>>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
>>> clobbering avoidance ?
>>>
>>> Yeah. I need a static "lock" variable and a static instance of the
>>> provisioning class.
>>>
>>> These statics will exist just in the loader jvm, right ?
>>>
>>> I was avoiding statics because I was not sure what all might share the
>>> instance of the provisioning class, like any other process controlled
>>> by the loader.
>>>
>>> I will give it a try.
>>>
>>> On Thu, Oct 20, 2011 at 8:16 AM, Chris Hyzer 
>>> <>
>>>  wrote:
>>>> Can you just set a static variable in a try/finally block which says the 
>>>> full sync is running, and if so, just return the first index which means 
>>>> it didn't make any progress.  I think (hope) that doesn't log an 
>>>> error...  alternately, the incremental could just sleep for 5 seconds in 
>>>> a loop until the full is done (by static variable), no more incrementals 
>>>> will start while the one didn't finish...
>>>>
>>>> Thanks,
>>>> Chris
>>>>
>>>> -----Original Message-----
>>>> From: 
>>>> 
>>>>  
>>>> [mailto:]
>>>>  On Behalf Of Tom Zeller
>>>> Sent: Wednesday, October 19, 2011 4:26 PM
>>>> To: Chris Hyzer
>>>> Cc: Grouper Dev
>>>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
>>>> clobbering avoidance ?
>>>>
>>>>> Nothing except the change log process should be inserting stuff into 
>>>>> the change log table.
>>>>> I don't think inserting records into the change log table will solve 
>>>>> the problem, you will always have race conditions
>>>>>
>>>>> What about this part of my previous email:
>>>>>
>>>>>> the full sync decides to add a member to a group, but the incremental 
>>>>>> also wants to do that.
>>>>>> Each process will need to ignore it if the member if already in the 
>>>>>> group, right?
>>>>>
>>>>> Can it be idempotent?  So if it is already done it doesn't fail?
>>>>
>>>> Yup, for membership adds and deletes we have to search first, then
>>>> perform necessary modifications. In other words :
>>>>
>>>> 1 calculate correct provisioning
>>>> 2 lookup current provisioning
>>>> 3 determine the difference between correct and current provisioning
>>>> 4 apply any necessary modifications
>>>>
>>>> Here is a use case :
>>>>
>>>>  0 changelog : "add member to group"
>>>>  1 incremental sync : "add member to group" - lookup membership
>>>>  2 incremental sync : "add member to group" - add member to group
>>>>
>>>>  3 full sync : calculate correct provisioning of group
>>>>  4 changelog : "delete member from group"
>>>>
>>>> If we perform the incremental sync while the full sync is running :
>>>>
>>>>  5a incremental sync : "delete member from group" : lookup membership
>>>>  6a incremental sync : "delete member to group" - delete member from 
>>>> group
>>>>  7a full sync : lookup current provisioning of group
>>>>  8a full sync : diff - will add member to group
>>>>  9a full sync : modify - add member to group
>>>>
>>>> Performing the incremental sync during the full sync will incorrectly
>>>> provision the group.
>>>>
>>>> If we wait to perform the incremental sync until the full sync is
>>>> finished running :
>>>>
>>>>  5b incremental sync : begin wait
>>>>  6b full sync : lookup current provisioning of group
>>>>  7b full sync : diff
>>>>  8b full sync : modify
>>>>  9b incremental sync : end wait
>>>> 10b incremental sync : "delete member from group" : lookup membership
>>>> 11b incremental sync : "delete member to group" - delete member from 
>>>> group
>>>>
>>>> Performing the incremental sync after the full sync completes will
>>>> correctly provision the group.
>>>>
>>>> Here is a slightly different use case demonstrating the need for 
>>>> idempotence :
>>>>
>>>>  0 changelog : "add member to group"
>>>>  1 incremental sync : "add member to group" - lookup membership
>>>>  2 incremental sync : "add member to group" - add member to group
>>>>
>>>>  3 changelog : "delete member from group"
>>>>  4 full sync : calculate correct provisioning of group
>>>>
>>>>  5c incremental sync : begin wait
>>>>  6c full sync : lookup current provisioning of group
>>>>  7c full sync : diff - will delete member from group
>>>>  8c full sync : modify - delete member from group
>>>>  9c incremental sync : end wait
>>>> 10b incremental sync : "delete member from group" : lookup membership
>>>> 11b incremental sync : "delete member to group" - no changes necessary
>>>>
>>>> So, I guess the ldappcng change log consumer will need a cron entry in
>>>> grouper-loader.properties to know when to trigger the full sync, and
>>>> it will need to know that a full sync is running to delay provisioning
>>>> of *any* change log entries until the full sync is complete.
>>>>
>>>> I had originally thought that we could delay provisioning per
>>>> identifier, but it is simpler to just delay the whole incremental job.
>>>> So, we can say we have real-time provisioning except when the full
>>>> sync is running, which should be configurable and can default to once
>>>> nightly.
>>>>
>>>> Are loader jobs Quartz Scheduler jobs in any way ? Can I extend
>>>> StatefulJob somehow in the change log consumer class to prevent
>>>> concurrency ? That would be a nice one-liner.
>>>>
>>>
>>
>