Grouper Developers Forum

[Tom Zeller <>]

Reviewing the work I had previously done regarding incremental
provisioning, I am curious if there are opinions on the attribute
resolver configuration I mocked up.

Is it ok ? horrible ?

https://spaces.internet2.edu/display/Grouper/Incremental+Provisioning+Development

On Fri, Sep 16, 2011 at 3:15 PM, LYNN GARRISON 
<>
 wrote:
> Chris, Tom,
>     The first scenario would work for Penn State for incremental 
> provisioning.  Our goal is to have ldap sourced only from grouper.   We 
> typically create  our course groups at the beginning of every semester and 
> post incremental changes daily. So our massive group deletes/add don't 
> occur that often.   Generally they are scheduled for the weekend so I think 
> we could deal with lots of transactions in the queue.
>
>
>
> Lynn Garrison
>
> ----- Original Message -----
> From: "Chris Hyzer" 
> <>
> To: "Tom Zeller" 
> <>,
>  "JAMES VUCCOLO" 
> <>
> Cc: "Grouper Dev" 
> <>
> Sent: Friday, September 16, 2011 3:05:35 PM
> Subject: RE: [grouper-dev] ldapp incremental
>
> Are we tuning a full sync, or looking into incremental?
>
> Ie. For incremental, it could be as easy as:
>
> 1. member is added to a group in grouper.  (this could be manually or via 
> loader, or rule, or WS, or whatever).  Note the member might be in 100 
> groups, and the group might have 100k members
>
> 2. the change log decides if this is a flattened change.  Ie. If the member 
> was in the group already e.g. via another path, then it will not send an 
> event.  If the user was not previously in the group by any path, and is 
> now, then the change log event is sent.  Note, this is already how it 
> works, no code needed here.
>
> 3. if ldap is sourced only from grouper, then you know the member was not 
> already in the group, just do the ldap operations to resolve the user and 
> group, add the user to the hasMembers of the group object, and add the 
> group to the memberOf of the user object.
>
> Or, if you want it to be like the iamonline one, it gets to #3, and:
>
> 3. <alternate> get the change from the change log notification, then check 
> a grouper hasMember to see if the user is a member, do an LDAP query to see 
> if the group is in the memberOf and/or see if the user is in the 
> hasMembers, and adjust if not consistent, but not by replacing the full 
> member list, just by adding an attribute value or removing.
>
> In neither of these approaches are we getting or replacing the full multi 
> value attributes...
>
> Note, that this approach is not good when you add a huge group to another 
> group, or if you change memberships - delete the group - recreate the group 
> - etc all in a matter of seconds.  Also, if there are a ton of changes in 
> the queue, which will happen from time to time but should be infrequent or 
> off hours, then there will be more delay than 1 minute.  But as we said, it 
> is not perfect, but it handles the steady state requirements...  right?  
> Maybe down the line we can make it more efficient for these types of 
> things...
>
> The iamonline logic is very simple, seen here:
>
> http://anonsvn.internet2.edu/cgi-bin/viewvc.cgi/i2mi/trunk/grouper-misc/poc_secureUserData/src/edu/internet2/middleware/poc_secureUserData/SudRealTime.java?revision=7553&view=co
>
> Note, you would be using the grouper api, not WS, but same principle:
>
> /**
>   * sync a group and user in incremental fashion
>   * @param groupExtension
>   * @param subjectId
>   */
>  public static void syncGroupAndUser(String groupExtension, String 
> subjectId) {
>    //get the memberships from Grouper
>    String rowGroupsFolder = 
> GrouperClientUtils.propertiesValue("sud.rootFolder", true) + ":rowGroups";
>
>    //is there a member?
>    WsHasMemberResults wsHasMemberResults = new 
> GcHasMember().addSubjectLookup(new WsSubjectLookup(subjectId, "jdbc", null))
>      .assignGroupName(rowGroupsFolder + ":" + groupExtension).execute();
>
>    boolean hasGrouperMembership = StringUtils.equals("IS_MEMBER",
>        
> wsHasMemberResults.getResults()[0].getResultMetadata().getResultCode());
>
>    //add all from DB
>    Set<String> groupExtensionsFromDb = new 
> TreeSet<String>(GcDbUtils.listSelect(String.class,
>        "select distinct group_extension from secureuserdata_memberships 
> where personid = ? and group_extension = ?",
>        GrouperClientUtils.toList(DbType.STRING), 
> GrouperClientUtils.toList((Object)subjectId, groupExtension)));
>
>    boolean hasDbMembership = groupExtensionsFromDb.contains(groupExtension);
>
>    if (hasGrouperMembership && !hasDbMembership) {
>
>      SudMembership sudMembership = new SudMembership();
>      sudMembership.setGroupExtension(groupExtension);
>      sudMembership.setPersonid(subjectId);
>      sudMembership.store();
>
>      if (LOG.isInfoEnabled()) {
>        LOG.info("Add mship for group: "
>            + groupExtension + ", personid: " + subjectId);
>      }
>
>    } else if (!hasGrouperMembership && hasDbMembership) {
>
>      //delete by query in case multiple
>      int rows = GcDbUtils.executeUpdate("delete from 
> secureuserdata_memberships where group_extension = ? and personid = ?",
>          GrouperClientUtils.toList((Object)groupExtension, subjectId));
>
>      if (LOG.isInfoEnabled()) {
>        LOG.info("Del " + rows + " mships of group: "
>            + groupExtension + ", personid: " + subjectId);
>      }
>
>    }
>  }
>
> Thanks,
> Chris
>
>
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Tom Zeller
> Sent: Friday, September 16, 2011 2:20 PM
> To: JAMES VUCCOLO
> Cc: Grouper Dev
> Subject: Re: [grouper-dev] ldapp incremental
>
> By profiling, I confirmed to myself that the bottleneck when
> provisioning a group's memberships via ldappcng is typically IO, e.g.
> ldap searches.
>
> Before revisiting incremental provisioning from where I left off
> before I changed jobs, I thought I should take a fresh look at
> reducing the number of ldap searches.
>
> On Wednesday, I realized that in a typical grouper installation with
> an ldap subject source, we were searching ldap twice for every
> membership. One search was performed by the grouper subject finder and
> a second by the ldappcng ldap target. So, I added a shibboleth data
> connector which wraps the grouper subject finder and the number of
> ldap searches were reduced by half, great.
>
> Tuning the grouper subject cache large enough to hold all subjects
> resulted in some nice times to provision large groups after the first
> run. For a group with 50k members in my test environment, the first
> provisioning required ~30s, most of which was ldap searches.
> Subsequent provisionings, because of caching, were ~5s.
>
> So, with appropriate ehcache settings and by merging the subject ldap
> adapter and target ldap adapter, member identifier resolution
> performance is improved. Not exactly RTP (real-time provisioning), but
> desirable regardless.
>
> Next on my list is to look at caching target state, meaning, cache the
> representation of a group as it is represented on a target, e.g. cache
> ldap group entries. This is the other kind of ldap search bottleneck
> after member identifier resolution.
>
> Sizing questions for Penn State : are your groups currently in ldap ?
> if so, what is the size of your DIT ? if you exported ou=people and
> ou=groups to an ldif file, what is that file size (uncompressed) ?
>
> Hopefully RTP won't be my SLO :-)
>
> On Thu, Sep 15, 2011 at 1:56 PM, JAMES VUCCOLO 
> <>
>  wrote:
>>
>>
>> ----- Original Message -----
>>> From: "Chris Hyzer" 
>>> <>
>>> To: "Grouper Dev" 
>>> <>
>>> Sent: Thursday, September 15, 2011 2:00:25 PM
>>> Subject: [grouper-dev] ldapp incremental
>>>
>>>
>>>
>>>
>>> TomZ,
>>>
>>>
>>>
>>> Do you mind adding sections to the grouper incremental provisioning
>>> doc from 4 months ago:
>>>
>>>
>>>
>>> https://spaces.internet2.edu/display/Grouper/Incremental+Provisioning+Development
>>>
>>>
>>>
>>> Including the use cases supported, the use cases not supported (since
>>> we don't want it to be "perfect"), and how the changes will
>>> propagate from the change log to ldap (i.e. which type of operation
>>> will occur, what is being cached, etc). Then Penn State and others
>>> can review.
>>>
>>
>> We are very interested in hearing about this too, definitely what 
>> operations will happen in the directory.  Listening in on the PACCMAN call 
>> today, I am not sure the caching described will solve our problems.  Our 
>> users expect that when they make a change to a group in a small period of 
>> time (did not want to use the word real-time), will see that change in 
>> LDAP.  Most if not all of our applications today go against the directory 
>> and probably in the future will continue to do so.  Our hope is for the 
>> incremental provisioning to LDAP as opposed to what exists in LDAPPCNG.  
>> We have a large number of groups > 40K and a number of them have 
>> memberships in the 30K vicinity.
>>
>> Thanks JimmyV
>>
>>
>>>
>>>
>>> Thanks,
>>>
>>> Chris
>>>
>>>
>>>
>>>
>>
>> --
>> James "Jimmy" Vuccolo, 
>> 
>> Technical Manager, Identity and Access Management
>> The Pennsylvania State University
>> 215B Computer Building, University Park, PA 16802
>> Office: 814-865-5635
>> http://www.personal.psu.edu/jvuccolo/
>>
>