Grouper Developers Forum

["Michael R. Gettes" <>]

yeah, well, this is a bit of a challenge in the LDAP world.  There's no real 
easy way to do this.  Let me stew on this a bit.  I presume doing as I now do 
which is the first interaction takes the longest and let normal operations 
(caching) provide the perf from that point forward isn't sufficient?  I know, 
PSU scale schools can be a bit of a problem.  If it takes me 5-8 minutes 
(actually, i was looking at my logs last evening after I sent my response and 
I see it's now between 3-5 min - and this is where the client is running on a 
2006 Solaris 2.8 SPARC - so on modern hardware it would be even faster), so 
let's say 5 min per 20K when things are working properly then for PSU scale 
(100K) it's 25 min - isn't this acceptable based on Tom's original note?  It 
could be an hour for the first time query.  Yes, all this presumes a proper 
performing environment but I think that's a reasonable expectation.  (sorry 
for the rambling - I hope all this makes sense).

/mrg

On May 18, 2012, at 9:45, Chris Hyzer wrote:

> Is there a compromise where you can get batches of 100 at once?  (in 
> sources, grouper attributes, etc)
> 
> Chris
> 
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Michael R. Gettes
> Sent: Thursday, May 17, 2012 8:15 PM
> To: Tom Zeller
> Cc: Grouper Dev
> Subject: Re: [grouper-dev] speedup by caching in the ldap source adapter ?
> 
> Hi Tom,
> 
> I am very concerned about this - allowing to do uid=* is killer on the 
> cache for the LDAP server.  It's a really nasty thing to do.  Bottom-line, 
> it's not just a caching problem for grouper or a provisioning system, it's 
> a caching problem for the components you're querying, like LDAP.  I should 
> also note, as you know, there are many parameters on LDAP servers to govern 
> limits on searching and such and SOME ldap servers offer limits on a DN 
> basis while others do not.  Maybe a "half-way" approach would be to offer 
> an optional "priming search" at start-up where those who can do as you are 
> suggesting would be able to do so.
> 
> I have 389 and because of my situation at CMU I have to search every member 
> of every group before making any modifications to the group.  A group with 
> 22K members takes about 8 minutes to process after the first pass on the 
> group when the set of member DNs gets loaded into the LDAP cache.  The 
> first time through can take 2-3 times longer.  Additionally, I have the 
> problem that uid=* would return a great deal more than the 22K needed for 
> my largest group, I have about 400K objects and about 50% have uid 
> attributes.  We are not configured to pull the full 400K objects into cache 
> for LDAP.
> 
> I hope this helps.
> 
> /mrg
> 
> On May 17, 2012, at 19:43, Tom Zeller wrote:
> 
>> When provisioning an ldap target, it seems that querying for members
>> takes the most time, especially when provisioning the "everyone"
>> group. Even if the source resolver has a large cache, at least one
>> ldap search is performed for every grouper membership.
>> 
>> To avoid the performance penalty and scaling issues with
>> one-ldap-search-per-membership, I added a simple cache to the ldap
>> source adapter, and this cache is warmed by slurping the target ldap
>> directory at startup.
>> 
>> In other words, rather than n "(uid=...)" searches, just do one
>> "(uid=*)" search.
>> 
>> I hesitate to give numbers, but on my local box, provisioning a group
>> with 100k members takes approximately 25 minutes (not hours !) without
>> the warmed-up cache. With the warmed-up cache, provisioning a group
>> with 100k members takes more like 5 minutes.
>> 
>> Another option is to make sure that all subject attributes needed for
>> provisioning are written to the grouper_members table, and avoid ldap
>> member searches entirely.
>> 
>> What do you think ?
>> 
>> TomZ
>