Grouper Developers Forum

[Chris Hyzer <>]

Looks good, though your cron strings are for testing, not the actual 
defaults, right?

Also, there are library methods to do easy reflection, though you could make 
the full sync implementor implement an interface, then you don't need 
reflection.  If you want reflection, you can just call this:

GrouperUtil.callMethod(instance, "methodName");

It will do all the error handling for you...

Thanks,
Chris


-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Tom Zeller
Sent: Friday, November 04, 2011 8:21 PM
To: Chris Hyzer
Cc: Grouper Dev
Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
clobbering avoidance ?

Kind of horrible but here is what I have so far to schedule
incremental and full syncs without overlapping :

To schedule full syncs via grouper-loader.properties, a class which
provides a "public void fullSync()" method is cron'd :

 changeLog.consumer.ldappc.class =
edu.internet2.middleware.grouper.changeLog.LdappcngConsumer
 changeLog.consumer.ldappc.quartzCron = 0/5 * * * * ?

 # To run full synchronizations, specify the class which provides a
'public void fullSync()' method.
 changeLog.ldappcng.fullsync.class =
edu.internet2.middleware.grouper.changeLog.LdappcngConsumer
 changeLog.ldappcng.fullsync.quartzCron = 0 0/5 * * * ?

The fullSync() method is straightforward

public LdappcngConsumer {

 private static boolean fullSyncIsRunning;

 public synchronized void fullSync() {
  LOG.info("Ldappcng Consumer '{}' - Starting full sync", name);

   fullSyncIsRunning = true;

   psp.execute(new BulkSyncRequest());

   LOG.info("Ldappcng Consumer '{}' - Finished full sync", name);

   fullSyncIsRunning = false;
  }

and the processChangeLogEntries() method has :

for (ChangeLogEntry changeLogEntry : changeLogEntryList) {

 currentId = changeLogEntry.getSequenceNumber();

 if (fullSyncIsRunning) {
   LOG.debug("Ldappcng Consumer '{}' - Full sync is running, returning
sequence number '{}'", name, currentId - 1);
   return currentId - 1;
  }

  // process the change log entry
  processChangeLogEntry(changeLogEntry)
}

and here is the GrouperLoaderTypeEnum :

 public void runJob(LoaderJobBean loaderJobBean) {

   GrouperContext.createNewDefaultContext(GrouperEngineBuiltin.LOADER,
false, true);

    String theClassName =
GrouperLoaderConfig.getPropertyString("changeLog.ldappcng.fullsync.class");
    Class<?> theClass = GrouperUtil.forName(theClassName);
    Object theClassInstance = GrouperUtil.newInstance(theClass);

     try {
       theClass.getMethod("fullSync", new Class[]
{}).invoke(theClassInstance, new Object[] {});
      } catch (SecurityException e) {
       throw new RuntimeException(e);
      } catch (NoSuchMethodException e) {
        throw new RuntimeException(e);
      } catch (IllegalArgumentException e) {
        throw new RuntimeException(e);
      } catch (IllegalAccessException e) {
        throw new RuntimeException(e);
      } catch (InvocationTargetException e) {
        throw new RuntimeException(e);
      }
   }

I am open to suggestions.

Thanks,
TomZ

On Fri, Oct 28, 2011 at 4:11 PM, Chris Hyzer 
<>
 wrote:
> I think you need to run yours like you would one of the others from GSH (or 
> java main) and see whats missing...ok?
>
> Thanks,
> Chris
>
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Tom Zeller
> Sent: Friday, October 28, 2011 1:51 PM
> To: Chris Hyzer
> Cc: Grouper Dev
> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
> clobbering avoidance ?
>
> I guess I need help adding the ldappcng full sync job type to the loader.
>
> I added this to the GrouperLoaderType enum :
>
>    /**      * Run an ldappcng full sync.     */    LDAPPCNG_FULL_SYNC {
>       public boolean attributeRequired(String attributeName) {
>      return false;      }
>       public boolean attributeOptional(String attributeName) {
>      return false;      }
>       public void runJob(LoaderJobBean loaderJobBean) {
> GrouperContext.createNewDefaultContext(GrouperEngineBuiltin.LOADER,
> false, true);
>         Hib3GrouperLoaderLog hib3GrouploaderLog =
> loaderJobBean.getHib3GrouploaderLogOverall();                String
> theClassName = 
> GrouperLoaderConfig.getPropertyString("changeLog.ldappcng.fullsync.class");
>        Class<?> theClass = GrouperUtil.forName(theClassName);
>               GrouperLoaderJob job = (GrouperLoaderJob)
> GrouperUtil.newInstance(theClass);
> job.runJob(hib3GrouploaderLog, (Group)null,
> loaderJobBean.getGrouperSession());                }        };
>
> I copied the change log type enum, which instantiates a class defined
> in grouper-loader.properties, which here I assume extends Job.
>
> Is this correct ? I am not sure what to do.
>
> Thanks,
> TomZ
> On Fri, Oct 21, 2011 at 10:35 AM, Chris Hyzer 
> <>
>  wrote:
>> Sure, either that or make a new type for ldapppncg full sync... I have 
>> been globbing in the minor stuff into MAINTENANCE, either way :)  Ignore 
>> the required and optional attributes...  that more for the loader
>>
>> Thanks,
>> Chris
>>
>> -----Original Message-----
>> From: 
>> 
>>  
>> [mailto:]
>>  On Behalf Of Tom Zeller
>> Sent: Friday, October 21, 2011 11:29 AM
>> To: Chris Hyzer
>> Cc: Grouper Dev
>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
>> clobbering avoidance ?
>>
>> Right. But I am confused. Loader jobs are StatefulJobs :
>>
>>  public class GrouperLoaderJob implements Job, StatefulJob {
>>
>> so can I just add another Job to the loader to do the full-sync ?
>>
>> I see you added LDAP_SIMPLE, LDAP_GROUP_LIST, etc. Can I re-use
>> GrouperLoaderType.MAINTENANCE somehow ?
>>
>> On Thu, Oct 20, 2011 at 10:04 AM, Chris Hyzer 
>> <>
>>  wrote:
>>> Note that the incremental should be reading the static with a method, and 
>>> nothing outside of the full sync should be able to edit it, right?  :)
>>>
>>> Thanks,
>>> chris
>>>
>>> -----Original Message-----
>>> From: 
>>> 
>>>  
>>> [mailto:]
>>>  On Behalf Of Tom Zeller
>>> Sent: Thursday, October 20, 2011 10:41 AM
>>> To: Chris Hyzer
>>> Cc: Grouper Dev
>>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
>>> clobbering avoidance ?
>>>
>>> Yeah. I need a static "lock" variable and a static instance of the
>>> provisioning class.
>>>
>>> These statics will exist just in the loader jvm, right ?
>>>
>>> I was avoiding statics because I was not sure what all might share the
>>> instance of the provisioning class, like any other process controlled
>>> by the loader.
>>>
>>> I will give it a try.
>>>
>>> On Thu, Oct 20, 2011 at 8:16 AM, Chris Hyzer 
>>> <>
>>>  wrote:
>>>> Can you just set a static variable in a try/finally block which says the 
>>>> full sync is running, and if so, just return the first index which means 
>>>> it didn't make any progress.  I think (hope) that doesn't log an 
>>>> error...  alternately, the incremental could just sleep for 5 seconds in 
>>>> a loop until the full is done (by static variable), no more incrementals 
>>>> will start while the one didn't finish...
>>>>
>>>> Thanks,
>>>> Chris
>>>>
>>>> -----Original Message-----
>>>> From: 
>>>> 
>>>>  
>>>> [mailto:]
>>>>  On Behalf Of Tom Zeller
>>>> Sent: Wednesday, October 19, 2011 4:26 PM
>>>> To: Chris Hyzer
>>>> Cc: Grouper Dev
>>>> Subject: Re: [grouper-dev] [ldappcng] real-time & batch scheduling and 
>>>> clobbering avoidance ?
>>>>
>>>>> Nothing except the change log process should be inserting stuff into 
>>>>> the change log table.
>>>>> I don't think inserting records into the change log table will solve 
>>>>> the problem, you will always have race conditions
>>>>>
>>>>> What about this part of my previous email:
>>>>>
>>>>>> the full sync decides to add a member to a group, but the incremental 
>>>>>> also wants to do that.
>>>>>> Each process will need to ignore it if the member if already in the 
>>>>>> group, right?
>>>>>
>>>>> Can it be idempotent?  So if it is already done it doesn't fail?
>>>>
>>>> Yup, for membership adds and deletes we have to search first, then
>>>> perform necessary modifications. In other words :
>>>>
>>>> 1 calculate correct provisioning
>>>> 2 lookup current provisioning
>>>> 3 determine the difference between correct and current provisioning
>>>> 4 apply any necessary modifications
>>>>
>>>> Here is a use case :
>>>>
>>>>  0 changelog : "add member to group"
>>>>  1 incremental sync : "add member to group" - lookup membership
>>>>  2 incremental sync : "add member to group" - add member to group
>>>>
>>>>  3 full sync : calculate correct provisioning of group
>>>>  4 changelog : "delete member from group"
>>>>
>>>> If we perform the incremental sync while the full sync is running :
>>>>
>>>>  5a incremental sync : "delete member from group" : lookup membership
>>>>  6a incremental sync : "delete member to group" - delete member from 
>>>> group
>>>>  7a full sync : lookup current provisioning of group
>>>>  8a full sync : diff - will add member to group
>>>>  9a full sync : modify - add member to group
>>>>
>>>> Performing the incremental sync during the full sync will incorrectly
>>>> provision the group.
>>>>
>>>> If we wait to perform the incremental sync until the full sync is
>>>> finished running :
>>>>
>>>>  5b incremental sync : begin wait
>>>>  6b full sync : lookup current provisioning of group
>>>>  7b full sync : diff
>>>>  8b full sync : modify
>>>>  9b incremental sync : end wait
>>>> 10b incremental sync : "delete member from group" : lookup membership
>>>> 11b incremental sync : "delete member to group" - delete member from 
>>>> group
>>>>
>>>> Performing the incremental sync after the full sync completes will
>>>> correctly provision the group.
>>>>
>>>> Here is a slightly different use case demonstrating the need for 
>>>> idempotence :
>>>>
>>>>  0 changelog : "add member to group"
>>>>  1 incremental sync : "add member to group" - lookup membership
>>>>  2 incremental sync : "add member to group" - add member to group
>>>>
>>>>  3 changelog : "delete member from group"
>>>>  4 full sync : calculate correct provisioning of group
>>>>
>>>>  5c incremental sync : begin wait
>>>>  6c full sync : lookup current provisioning of group
>>>>  7c full sync : diff - will delete member from group
>>>>  8c full sync : modify - delete member from group
>>>>  9c incremental sync : end wait
>>>> 10b incremental sync : "delete member from group" : lookup membership
>>>> 11b incremental sync : "delete member to group" - no changes necessary
>>>>
>>>> So, I guess the ldappcng change log consumer will need a cron entry in
>>>> grouper-loader.properties to know when to trigger the full sync, and
>>>> it will need to know that a full sync is running to delay provisioning
>>>> of *any* change log entries until the full sync is complete.
>>>>
>>>> I had originally thought that we could delay provisioning per
>>>> identifier, but it is simpler to just delay the whole incremental job.
>>>> So, we can say we have real-time provisioning except when the full
>>>> sync is running, which should be configurable and can default to once
>>>> nightly.
>>>>
>>>> Are loader jobs Quartz Scheduler jobs in any way ? Can I extend
>>>> StatefulJob somehow in the change log consumer class to prevent
>>>> concurrency ? That would be a nice one-liner.
>>>>
>>>
>>
>