Grouper Developers Forum

[Tom Zeller <>]

Thanks.

> Looks good, though your cron strings are for testing, not the actual 
> defaults, right?

Right.

> Also, there are library methods to do easy reflection, though you could 
> make the full sync implementor implement an interface, then you don't need 
> reflection.  If you want reflection, you can just call this:
>
> GrouperUtil.callMethod(instance, "methodName");

OK.

TomZ

> It will do all the error handling for you...
>
> Thanks,
> Chris
>
>
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Tom Zeller
> Sent: Friday, November 04, 2011 8:21 PM
> To: Chris Hyzer
> Cc: Grouper Dev
> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
> clobbering avoidance ?
>
> Kind of horrible but here is what I have so far to schedule
> incremental and full syncs without overlapping :
>
> To schedule full syncs via grouper-loader.properties, a class which
> provides a "public void fullSync()" method is cron'd :
>
>  changeLog.consumer.ldappc.class =
> edu.internet2.middleware.grouper.changeLog.LdappcngConsumer
>  changeLog.consumer.ldappc.quartzCron = 0/5 * * * * ?
>
>  # To run full synchronizations, specify the class which provides a
> 'public void fullSync()' method.
>  changeLog.ldappcng.fullsync.class =
> edu.internet2.middleware.grouper.changeLog.LdappcngConsumer
>  changeLog.ldappcng.fullsync.quartzCron = 0 0/5 * * * ?
>
> The fullSync() method is straightforward
>
> public LdappcngConsumer {
>
>  private static boolean fullSyncIsRunning;
>
>  public synchronized void fullSync() {
>  LOG.info("Ldappcng Consumer '{}' - Starting full sync", name);
>
>   fullSyncIsRunning = true;
>
>   psp.execute(new BulkSyncRequest());
>
>   LOG.info("Ldappcng Consumer '{}' - Finished full sync", name);
>
>   fullSyncIsRunning = false;
>  }
>
> and the processChangeLogEntries() method has :
>
> for (ChangeLogEntry changeLogEntry : changeLogEntryList) {
>
>  currentId = changeLogEntry.getSequenceNumber();
>
>  if (fullSyncIsRunning) {
>   LOG.debug("Ldappcng Consumer '{}' - Full sync is running, returning
> sequence number '{}'", name, currentId - 1);
>   return currentId - 1;
>  }
>
>  // process the change log entry
>  processChangeLogEntry(changeLogEntry)
> }
>
> and here is the GrouperLoaderTypeEnum :
>
>  public void runJob(LoaderJobBean loaderJobBean) {
>
>   GrouperContext.createNewDefaultContext(GrouperEngineBuiltin.LOADER,
> false, true);
>
>    String theClassName =
> GrouperLoaderConfig.getPropertyString("changeLog.ldappcng.fullsync.class");
>    Class<?> theClass = GrouperUtil.forName(theClassName);
>    Object theClassInstance = GrouperUtil.newInstance(theClass);
>
>     try {
>       theClass.getMethod("fullSync", new Class[]
> {}).invoke(theClassInstance, new Object[] {});
>      } catch (SecurityException e) {
>       throw new RuntimeException(e);
>      } catch (NoSuchMethodException e) {
>        throw new RuntimeException(e);
>      } catch (IllegalArgumentException e) {
>        throw new RuntimeException(e);
>      } catch (IllegalAccessException e) {
>        throw new RuntimeException(e);
>      } catch (InvocationTargetException e) {
>        throw new RuntimeException(e);
>      }
>   }
>
> I am open to suggestions.
>
> Thanks,
> TomZ
>
> On Fri, Oct 28, 2011 at 4:11 PM, Chris Hyzer 
> <>
>  wrote:
>> I think you need to run yours like you would one of the others from GSH 
>> (or java main) and see whats missing...ok?
>>
>> Thanks,
>> Chris
>>
>> -----Original Message-----
>> From: 
>> 
>>  
>> [mailto:]
>>  On Behalf Of Tom Zeller
>> Sent: Friday, October 28, 2011 1:51 PM
>> To: Chris Hyzer
>> Cc: Grouper Dev
>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
>> clobbering avoidance ?
>>
>> I guess I need help adding the ldappcng full sync job type to the loader.
>>
>> I added this to the GrouperLoaderType enum :
>>
>>    /**      * Run an ldappcng full sync.     */    LDAPPCNG_FULL_SYNC {
>>       public boolean attributeRequired(String attributeName) {
>>      return false;      }
>>       public boolean attributeOptional(String attributeName) {
>>      return false;      }
>>       public void runJob(LoaderJobBean loaderJobBean) {
>> GrouperContext.createNewDefaultContext(GrouperEngineBuiltin.LOADER,
>> false, true);
>>         Hib3GrouperLoaderLog hib3GrouploaderLog =
>> loaderJobBean.getHib3GrouploaderLogOverall();                String
>> theClassName = 
>> GrouperLoaderConfig.getPropertyString("changeLog.ldappcng.fullsync.class");
>>        Class<?> theClass = GrouperUtil.forName(theClassName);
>>               GrouperLoaderJob job = (GrouperLoaderJob)
>> GrouperUtil.newInstance(theClass);
>> job.runJob(hib3GrouploaderLog, (Group)null,
>> loaderJobBean.getGrouperSession());                }        };
>>
>> I copied the change log type enum, which instantiates a class defined
>> in grouper-loader.properties, which here I assume extends Job.
>>
>> Is this correct ? I am not sure what to do.
>>
>> Thanks,
>> TomZ
>> On Fri, Oct 21, 2011 at 10:35 AM, Chris Hyzer 
>> <>
>>  wrote:
>>> Sure, either that or make a new type for ldapppncg full sync... I have 
>>> been globbing in the minor stuff into MAINTENANCE, either way :)  Ignore 
>>> the required and optional attributes...  that more for the loader
>>>
>>> Thanks,
>>> Chris
>>>
>>> -----Original Message-----
>>> From: 
>>> 
>>>  
>>> [mailto:]
>>>  On Behalf Of Tom Zeller
>>> Sent: Friday, October 21, 2011 11:29 AM
>>> To: Chris Hyzer
>>> Cc: Grouper Dev
>>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
>>> clobbering avoidance ?
>>>
>>> Right. But I am confused. Loader jobs are StatefulJobs :
>>>
>>>  public class GrouperLoaderJob implements Job, StatefulJob {
>>>
>>> so can I just add another Job to the loader to do the full-sync ?
>>>
>>> I see you added LDAP_SIMPLE, LDAP_GROUP_LIST, etc. Can I re-use
>>> GrouperLoaderType.MAINTENANCE somehow ?
>>>
>>> On Thu, Oct 20, 2011 at 10:04 AM, Chris Hyzer 
>>> <>
>>>  wrote:
>>>> Note that the incremental should be reading the static with a method, 
>>>> and nothing outside of the full sync should be able to edit it, right?  
>>>> :)
>>>>
>>>> Thanks,
>>>> chris
>>>>
>>>> -----Original Message-----
>>>> From: 
>>>> 
>>>>  
>>>> [mailto:]
>>>>  On Behalf Of Tom Zeller
>>>> Sent: Thursday, October 20, 2011 10:41 AM
>>>> To: Chris Hyzer
>>>> Cc: Grouper Dev
>>>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
>>>> clobbering avoidance ?
>>>>
>>>> Yeah. I need a static "lock" variable and a static instance of the
>>>> provisioning class.
>>>>
>>>> These statics will exist just in the loader jvm, right ?
>>>>
>>>> I was avoiding statics because I was not sure what all might share the
>>>> instance of the provisioning class, like any other process controlled
>>>> by the loader.
>>>>
>>>> I will give it a try.
>>>>
>>>> On Thu, Oct 20, 2011 at 8:16 AM, Chris Hyzer 
>>>> <>
>>>>  wrote:
>>>>> Can you just set a static variable in a try/finally block which says 
>>>>> the full sync is running, and if so, just return the first index which 
>>>>> means it didn't make any progress.  I think (hope) that doesn't log an 
>>>>> error...  alternately, the incremental could just sleep for 5 seconds 
>>>>> in a loop until the full is done (by static variable), no more 
>>>>> incrementals will start while the one didn't finish...
>>>>>
>>>>> Thanks,
>>>>> Chris
>>>>>
>>>>> -----Original Message-----
>>>>> From: 
>>>>> 
>>>>>  
>>>>> [mailto:]
>>>>>  On Behalf Of Tom Zeller
>>>>> Sent: Wednesday, October 19, 2011 4:26 PM
>>>>> To: Chris Hyzer
>>>>> Cc: Grouper Dev
>>>>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
>>>>> clobbering avoidance ?
>>>>>
>>>>>> Nothing except the change log process should be inserting stuff into 
>>>>>> the change log table.
>>>>>> I don't think inserting records into the change log table will solve 
>>>>>> the problem, you will always have race conditions
>>>>>>
>>>>>> What about this part of my previous email:
>>>>>>
>>>>>>> the full sync decides to add a member to a group, but the incremental 
>>>>>>> also wants to do that.
>>>>>>> Each process will need to ignore it if the member if already in the 
>>>>>>> group, right?
>>>>>>
>>>>>> Can it be idempotent?  So if it is already done it doesn't fail?
>>>>>
>>>>> Yup, for membership adds and deletes we have to search first, then
>>>>> perform necessary modifications. In other words :
>>>>>
>>>>> 1 calculate correct provisioning
>>>>> 2 lookup current provisioning
>>>>> 3 determine the difference between correct and current provisioning
>>>>> 4 apply any necessary modifications
>>>>>
>>>>> Here is a use case :
>>>>>
>>>>>  0 changelog : "add member to group"
>>>>>  1 incremental sync : "add member to group" - lookup membership
>>>>>  2 incremental sync : "add member to group" - add member to group
>>>>>
>>>>>  3 full sync : calculate correct provisioning of group
>>>>>  4 changelog : "delete member from group"
>>>>>
>>>>> If we perform the incremental sync while the full sync is running :
>>>>>
>>>>>  5a incremental sync : "delete member from group" : lookup membership
>>>>>  6a incremental sync : "delete member to group" - delete member from 
>>>>> group
>>>>>  7a full sync : lookup current provisioning of group
>>>>>  8a full sync : diff - will add member to group
>>>>>  9a full sync : modify - add member to group
>>>>>
>>>>> Performing the incremental sync during the full sync will incorrectly
>>>>> provision the group.
>>>>>
>>>>> If we wait to perform the incremental sync until the full sync is
>>>>> finished running :
>>>>>
>>>>>  5b incremental sync : begin wait
>>>>>  6b full sync : lookup current provisioning of group
>>>>>  7b full sync : diff
>>>>>  8b full sync : modify
>>>>>  9b incremental sync : end wait
>>>>> 10b incremental sync : "delete member from group" : lookup membership
>>>>> 11b incremental sync : "delete member to group" - delete member from 
>>>>> group
>>>>>
>>>>> Performing the incremental sync after the full sync completes will
>>>>> correctly provision the group.
>>>>>
>>>>> Here is a slightly different use case demonstrating the need for 
>>>>> idempotence :
>>>>>
>>>>>  0 changelog : "add member to group"
>>>>>  1 incremental sync : "add member to group" - lookup membership
>>>>>  2 incremental sync : "add member to group" - add member to group
>>>>>
>>>>>  3 changelog : "delete member from group"
>>>>>  4 full sync : calculate correct provisioning of group
>>>>>
>>>>>  5c incremental sync : begin wait
>>>>>  6c full sync : lookup current provisioning of group
>>>>>  7c full sync : diff - will delete member from group
>>>>>  8c full sync : modify - delete member from group
>>>>>  9c incremental sync : end wait
>>>>> 10b incremental sync : "delete member from group" : lookup membership
>>>>> 11b incremental sync : "delete member to group" - no changes necessary
>>>>>
>>>>> So, I guess the ldappcng change log consumer will need a cron entry in
>>>>> grouper-loader.properties to know when to trigger the full sync, and
>>>>> it will need to know that a full sync is running to delay provisioning
>>>>> of *any* change log entries until the full sync is complete.
>>>>>
>>>>> I had originally thought that we could delay provisioning per
>>>>> identifier, but it is simpler to just delay the whole incremental job.
>>>>> So, we can say we have real-time provisioning except when the full
>>>>> sync is running, which should be configurable and can default to once
>>>>> nightly.
>>>>>
>>>>> Are loader jobs Quartz Scheduler jobs in any way ? Can I extend
>>>>> StatefulJob somehow in the change log consumer class to prevent
>>>>> concurrency ? That would be a nice one-liner.
>>>>>
>>>>
>>>
>>
>