Grouper Developers Forum

[Tom Zeller <>]

By profiling, I confirmed to myself that the bottleneck when
provisioning a group's memberships via ldappcng is typically IO, e.g.
ldap searches.

Before revisiting incremental provisioning from where I left off
before I changed jobs, I thought I should take a fresh look at
reducing the number of ldap searches.

On Wednesday, I realized that in a typical grouper installation with
an ldap subject source, we were searching ldap twice for every
membership. One search was performed by the grouper subject finder and
a second by the ldappcng ldap target. So, I added a shibboleth data
connector which wraps the grouper subject finder and the number of
ldap searches were reduced by half, great.

Tuning the grouper subject cache large enough to hold all subjects
resulted in some nice times to provision large groups after the first
run. For a group with 50k members in my test environment, the first
provisioning required ~30s, most of which was ldap searches.
Subsequent provisionings, because of caching, were ~5s.

So, with appropriate ehcache settings and by merging the subject ldap
adapter and target ldap adapter, member identifier resolution
performance is improved. Not exactly RTP (real-time provisioning), but
desirable regardless.

Next on my list is to look at caching target state, meaning, cache the
representation of a group as it is represented on a target, e.g. cache
ldap group entries. This is the other kind of ldap search bottleneck
after member identifier resolution.

Sizing questions for Penn State : are your groups currently in ldap ?
if so, what is the size of your DIT ? if you exported ou=people and
ou=groups to an ldif file, what is that file size (uncompressed) ?

Hopefully RTP won't be my SLO :-)

On Thu, Sep 15, 2011 at 1:56 PM, JAMES VUCCOLO 
<>
 wrote:
>
>
> ----- Original Message -----
>> From: "Chris Hyzer" 
>> <>
>> To: "Grouper Dev" 
>> <>
>> Sent: Thursday, September 15, 2011 2:00:25 PM
>> Subject: [grouper-dev] ldapp incremental
>>
>>
>>
>>
>> TomZ,
>>
>>
>>
>> Do you mind adding sections to the grouper incremental provisioning
>> doc from 4 months ago:
>>
>>
>>
>> https://spaces.internet2.edu/display/Grouper/Incremental+Provisioning+Development
>>
>>
>>
>> Including the use cases supported, the use cases not supported (since
>> we don’t want it to be “perfect”), and how the changes will
>> propagate from the change log to ldap (i.e. which type of operation
>> will occur, what is being cached, etc). Then Penn State and others
>> can review…
>>
>
> We are very interested in hearing about this too, definitely what 
> operations will happen in the directory.  Listening in on the PACCMAN call 
> today, I am not sure the caching described will solve our problems.  Our 
> users expect that when they make a change to a group in a small period of 
> time (did not want to use the word real-time), will see that change in 
> LDAP.  Most if not all of our applications today go against the directory 
> and probably in the future will continue to do so.  Our hope is for the 
> incremental provisioning to LDAP as opposed to what exists in LDAPPCNG.  We 
> have a large number of groups > 40K and a number of them have memberships 
> in the 30K vicinity.
>
> Thanks JimmyV
>
>
>>
>>
>> Thanks,
>>
>> Chris
>>
>>
>>
>>
>
> --
> James "Jimmy" Vuccolo, 
> 
> Technical Manager, Identity and Access Management
> The Pennsylvania State University
> 215B Computer Building, University Park, PA 16802
> Office: 814-865-5635
> http://www.personal.psu.edu/jvuccolo/
>