Grouper Developers Forum

[Chris Hyzer <>]

Is there a compromise where you can get batches of 100 at once?  (in sources, 
grouper attributes, etc)

Chris

-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Michael R. Gettes
Sent: Thursday, May 17, 2012 8:15 PM
To: Tom Zeller
Cc: Grouper Dev
Subject: Re: [grouper-dev] speedup by caching in the ldap source adapter ?

Hi Tom,

I am very concerned about this - allowing to do uid=* is killer on the cache 
for the LDAP server.  It's a really nasty thing to do.  Bottom-line, it's not 
just a caching problem for grouper or a provisioning system, it's a caching 
problem for the components you're querying, like LDAP.  I should also note, 
as you know, there are many parameters on LDAP servers to govern limits on 
searching and such and SOME ldap servers offer limits on a DN basis while 
others do not.  Maybe a "half-way" approach would be to offer an optional 
"priming search" at start-up where those who can do as you are suggesting 
would be able to do so.

I have 389 and because of my situation at CMU I have to search every member 
of every group before making any modifications to the group.  A group with 
22K members takes about 8 minutes to process after the first pass on the 
group when the set of member DNs gets loaded into the LDAP cache.  The first 
time through can take 2-3 times longer.  Additionally, I have the problem 
that uid=* would return a great deal more than the 22K needed for my largest 
group, I have about 400K objects and about 50% have uid attributes.  We are 
not configured to pull the full 400K objects into cache for LDAP.

I hope this helps.

/mrg

On May 17, 2012, at 19:43, Tom Zeller wrote:

> When provisioning an ldap target, it seems that querying for members
> takes the most time, especially when provisioning the "everyone"
> group. Even if the source resolver has a large cache, at least one
> ldap search is performed for every grouper membership.
> 
> To avoid the performance penalty and scaling issues with
> one-ldap-search-per-membership, I added a simple cache to the ldap
> source adapter, and this cache is warmed by slurping the target ldap
> directory at startup.
> 
> In other words, rather than n "(uid=...)" searches, just do one
> "(uid=*)" search.
> 
> I hesitate to give numbers, but on my local box, provisioning a group
> with 100k members takes approximately 25 minutes (not hours !) without
> the warmed-up cache. With the warmed-up cache, provisioning a group
> with 100k members takes more like 5 minutes.
> 
> Another option is to make sure that all subject attributes needed for
> provisioning are written to the grouper_members table, and avoid ldap
> member searches entirely.
> 
> What do you think ?
> 
> TomZ