Grouper Developers Forum

[JAMES VUCCOLO <>]

Tom, my answers are below.

Jimmy.

----- Original Message -----
> From: "Tom Zeller" 
> <>
> To: "JAMES VUCCOLO" 
> <>
> Cc: "Grouper Dev" 
> <>
> Sent: Friday, September 16, 2011 2:19:54 PM
> Subject: Re: [grouper-dev] ldapp incremental
>
> By profiling, I confirmed to myself that the bottleneck when
> provisioning a group's memberships via ldappcng is typically IO, e.g.
> ldap searches.
>
> Before revisiting incremental provisioning from where I left off
> before I changed jobs, I thought I should take a fresh look at
> reducing the number of ldap searches.
>
> On Wednesday, I realized that in a typical grouper installation with
> an ldap subject source, we were searching ldap twice for every
> membership. One search was performed by the grouper subject finder
> and
> a second by the ldappcng ldap target. So, I added a shibboleth data
> connector which wraps the grouper subject finder and the number of
> ldap searches were reduced by half, great.
>
> Tuning the grouper subject cache large enough to hold all subjects
> resulted in some nice times to provision large groups after the first
> run. For a group with 50k members in my test environment, the first
> provisioning required ~30s, most of which was ldap searches.
> Subsequent provisionings, because of caching, were ~5s.
>
> So, with appropriate ehcache settings and by merging the subject ldap
> adapter and target ldap adapter, member identifier resolution
> performance is improved. Not exactly RTP (real-time provisioning),
> but
> desirable regardless.
>
> Next on my list is to look at caching target state, meaning, cache
> the
> representation of a group as it is represented on a target, e.g.
> cache
> ldap group entries. This is the other kind of ldap search bottleneck
> after member identifier resolution.
>
> Sizing questions for Penn State : are your groups currently in ldap ?

Yes.

> if so, what is the size of your DIT ?

412,560

> if you exported ou=people and
> ou=groups to an ldif file, what is that file size (uncompressed) ?

510 MB.

>
> Hopefully RTP won't be my SLO :-)
>
> On Thu, Sep 15, 2011 at 1:56 PM, JAMES VUCCOLO 
> <>
> wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Chris Hyzer" 
> >> <>
> >> To: "Grouper Dev" 
> >> <>
> >> Sent: Thursday, September 15, 2011 2:00:25 PM
> >> Subject: [grouper-dev] ldapp incremental
> >>
> >>
> >>
> >>
> >> TomZ,
> >>
> >>
> >>
> >> Do you mind adding sections to the grouper incremental
> >> provisioning
> >> doc from 4 months ago:
> >>
> >>
> >>
> >> https://spaces.internet2.edu/display/Grouper/Incremental+Provisioning+Development
> >>
> >>
> >>
> >> Including the use cases supported, the use cases not supported
> >> (since
> >> we don’t want it to be “perfect”), and how the changes will
> >> propagate from the change log to ldap (i.e. which type of
> >> operation
> >> will occur, what is being cached, etc). Then Penn State and others
> >> can review…
> >>
> >
> > We are very interested in hearing about this too, definitely what
> > operations will happen in the directory.  Listening in on the
> > PACCMAN call today, I am not sure the caching described will solve
> > our problems.  Our users expect that when they make a change to a
> > group in a small period of time (did not want to use the word
> > real-time), will see that change in LDAP.  Most if not all of our
> > applications today go against the directory and probably in the
> > future will continue to do so.  Our hope is for the incremental
> > provisioning to LDAP as opposed to what exists in LDAPPCNG.  We
> > have a large number of groups > 40K and a number of them have
> > memberships in the 30K vicinity.
> >
> > Thanks JimmyV
> >
> >
> >>
> >>
> >> Thanks,
> >>
> >> Chris
> >>
> >>
> >>
> >>
> >
> > --
> > James "Jimmy" Vuccolo, 
> > 
> > Technical Manager, Identity and Access Management
> > The Pennsylvania State University
> > 215B Computer Building, University Park, PA 16802
> > Office: 814-865-5635
> > http://www.personal.psu.edu/jvuccolo/
> >
>

--
James "Jimmy" Vuccolo, 

Technical Manager, Identity and Access Management
The Pennsylvania State University
215B Computer Building, University Park, PA 16802
Office: 814-865-5635
http://www.personal.psu.edu/jvuccolo/