Grouper Developers Forum

[LYNN GARRISON <>]

Chris, Tom,
     The first scenario would work for Penn State for incremental 
provisioning.  Our goal is to have ldap sourced only from grouper.   We 
typically create  our course groups at the beginning of every semester and 
post incremental changes daily. So our massive group deletes/add don't occur 
that often.   Generally they are scheduled for the weekend so I think we 
could deal with lots of transactions in the queue.



Lynn Garrison

----- Original Message -----
From: "Chris Hyzer" 
<>
To: "Tom Zeller" 
<>,
 "JAMES VUCCOLO" 
<>
Cc: "Grouper Dev" 
<>
Sent: Friday, September 16, 2011 3:05:35 PM
Subject: RE: [grouper-dev] ldapp incremental

Are we tuning a full sync, or looking into incremental?

Ie. For incremental, it could be as easy as:

1. member is added to a group in grouper.  (this could be manually or via 
loader, or rule, or WS, or whatever).  Note the member might be in 100 
groups, and the group might have 100k members

2. the change log decides if this is a flattened change.  Ie. If the member 
was in the group already e.g. via another path, then it will not send an 
event.  If the user was not previously in the group by any path, and is now, 
then the change log event is sent.  Note, this is already how it works, no 
code needed here.

3. if ldap is sourced only from grouper, then you know the member was not 
already in the group, just do the ldap operations to resolve the user and 
group, add the user to the hasMembers of the group object, and add the group 
to the memberOf of the user object.

Or, if you want it to be like the iamonline one, it gets to #3, and:

3. <alternate> get the change from the change log notification, then check a 
grouper hasMember to see if the user is a member, do an LDAP query to see if 
the group is in the memberOf and/or see if the user is in the hasMembers, and 
adjust if not consistent, but not by replacing the full member list, just by 
adding an attribute value or removing.

In neither of these approaches are we getting or replacing the full multi 
value attributes...

Note, that this approach is not good when you add a huge group to another 
group, or if you change memberships - delete the group - recreate the group - 
etc all in a matter of seconds.  Also, if there are a ton of changes in the 
queue, which will happen from time to time but should be infrequent or off 
hours, then there will be more delay than 1 minute.  But as we said, it is 
not perfect, but it handles the steady state requirements...  right?  Maybe 
down the line we can make it more efficient for these types of things...

The iamonline logic is very simple, seen here:

http://anonsvn.internet2.edu/cgi-bin/viewvc.cgi/i2mi/trunk/grouper-misc/poc_secureUserData/src/edu/internet2/middleware/poc_secureUserData/SudRealTime.java?revision=7553&view=co

Note, you would be using the grouper api, not WS, but same principle:

/**
   * sync a group and user in incremental fashion
   * @param groupExtension
   * @param subjectId
   */
  public static void syncGroupAndUser(String groupExtension, String 
subjectId) {
    //get the memberships from Grouper
    String rowGroupsFolder = 
GrouperClientUtils.propertiesValue("sud.rootFolder", true) + ":rowGroups";

    //is there a member?
    WsHasMemberResults wsHasMemberResults = new 
GcHasMember().addSubjectLookup(new WsSubjectLookup(subjectId, "jdbc", null))
      .assignGroupName(rowGroupsFolder + ":" + groupExtension).execute();

    boolean hasGrouperMembership = StringUtils.equals("IS_MEMBER",
        
wsHasMemberResults.getResults()[0].getResultMetadata().getResultCode());

    //add all from DB
    Set<String> groupExtensionsFromDb = new 
TreeSet<String>(GcDbUtils.listSelect(String.class,
        "select distinct group_extension from secureuserdata_memberships 
where personid = ? and group_extension = ?",
        GrouperClientUtils.toList(DbType.STRING), 
GrouperClientUtils.toList((Object)subjectId, groupExtension)));

    boolean hasDbMembership = groupExtensionsFromDb.contains(groupExtension);

    if (hasGrouperMembership && !hasDbMembership) {

      SudMembership sudMembership = new SudMembership();
      sudMembership.setGroupExtension(groupExtension);
      sudMembership.setPersonid(subjectId);
      sudMembership.store();

      if (LOG.isInfoEnabled()) {
        LOG.info("Add mship for group: "
            + groupExtension + ", personid: " + subjectId);
      }

    } else if (!hasGrouperMembership && hasDbMembership) {

      //delete by query in case multiple
      int rows = GcDbUtils.executeUpdate("delete from 
secureuserdata_memberships where group_extension = ? and personid = ?",
          GrouperClientUtils.toList((Object)groupExtension, subjectId));

      if (LOG.isInfoEnabled()) {
        LOG.info("Del " + rows + " mships of group: "
            + groupExtension + ", personid: " + subjectId);
      }

    }
  }

Thanks,
Chris


-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Tom Zeller
Sent: Friday, September 16, 2011 2:20 PM
To: JAMES VUCCOLO
Cc: Grouper Dev
Subject: Re: [grouper-dev] ldapp incremental

By profiling, I confirmed to myself that the bottleneck when
provisioning a group's memberships via ldappcng is typically IO, e.g.
ldap searches.

Before revisiting incremental provisioning from where I left off
before I changed jobs, I thought I should take a fresh look at
reducing the number of ldap searches.

On Wednesday, I realized that in a typical grouper installation with
an ldap subject source, we were searching ldap twice for every
membership. One search was performed by the grouper subject finder and
a second by the ldappcng ldap target. So, I added a shibboleth data
connector which wraps the grouper subject finder and the number of
ldap searches were reduced by half, great.

Tuning the grouper subject cache large enough to hold all subjects
resulted in some nice times to provision large groups after the first
run. For a group with 50k members in my test environment, the first
provisioning required ~30s, most of which was ldap searches.
Subsequent provisionings, because of caching, were ~5s.

So, with appropriate ehcache settings and by merging the subject ldap
adapter and target ldap adapter, member identifier resolution
performance is improved. Not exactly RTP (real-time provisioning), but
desirable regardless.

Next on my list is to look at caching target state, meaning, cache the
representation of a group as it is represented on a target, e.g. cache
ldap group entries. This is the other kind of ldap search bottleneck
after member identifier resolution.

Sizing questions for Penn State : are your groups currently in ldap ?
if so, what is the size of your DIT ? if you exported ou=people and
ou=groups to an ldif file, what is that file size (uncompressed) ?

Hopefully RTP won't be my SLO :-)

On Thu, Sep 15, 2011 at 1:56 PM, JAMES VUCCOLO 
<>
 wrote:
>
>
> ----- Original Message -----
>> From: "Chris Hyzer" 
>> <>
>> To: "Grouper Dev" 
>> <>
>> Sent: Thursday, September 15, 2011 2:00:25 PM
>> Subject: [grouper-dev] ldapp incremental
>>
>>
>>
>>
>> TomZ,
>>
>>
>>
>> Do you mind adding sections to the grouper incremental provisioning
>> doc from 4 months ago:
>>
>>
>>
>> https://spaces.internet2.edu/display/Grouper/Incremental+Provisioning+Development
>>
>>
>>
>> Including the use cases supported, the use cases not supported (since
>> we don't want it to be "perfect"), and how the changes will
>> propagate from the change log to ldap (i.e. which type of operation
>> will occur, what is being cached, etc). Then Penn State and others
>> can review.
>>
>
> We are very interested in hearing about this too, definitely what 
> operations will happen in the directory.  Listening in on the PACCMAN call 
> today, I am not sure the caching described will solve our problems.  Our 
> users expect that when they make a change to a group in a small period of 
> time (did not want to use the word real-time), will see that change in 
> LDAP.  Most if not all of our applications today go against the directory 
> and probably in the future will continue to do so.  Our hope is for the 
> incremental provisioning to LDAP as opposed to what exists in LDAPPCNG.  We 
> have a large number of groups > 40K and a number of them have memberships 
> in the 30K vicinity.
>
> Thanks JimmyV
>
>
>>
>>
>> Thanks,
>>
>> Chris
>>
>>
>>
>>
>
> --
> James "Jimmy" Vuccolo, 
> 
> Technical Manager, Identity and Access Management
> The Pennsylvania State University
> 215B Computer Building, University Park, PA 16802
> Office: 814-865-5635
> http://www.personal.psu.edu/jvuccolo/
>