Grouper Developers Forum

["Michael R. Gettes" <>]

are you saying batch depending on the type of source or add batching as a 
config option on the DB source?  if so, that's reasonable.

/mrg

On May 18, 2012, at 10:33, Chris Hyzer wrote:

> Well, if batching from ldap isn't ideal, batching from the DB does improve 
> performance (member queries, attributes, etc), so maybe that would help...
> 
> Thanks,
> Chris
> 
> -----Original Message-----
> From: Michael R. Gettes 
> [mailto:]
>  
> Sent: Friday, May 18, 2012 10:01 AM
> To: Chris Hyzer
> Cc: Tom Zeller; Grouper Dev
> Subject: Re: [grouper-dev] speedup by caching in the ldap source adapter ?
> 
> yeah, well, this is a bit of a challenge in the LDAP world.  There's no 
> real easy way to do this.  Let me stew on this a bit.  I presume doing as I 
> now do which is the first interaction takes the longest and let normal 
> operations (caching) provide the perf from that point forward isn't 
> sufficient?  I know, PSU scale schools can be a bit of a problem.  If it 
> takes me 5-8 minutes (actually, i was looking at my logs last evening after 
> I sent my response and I see it's now between 3-5 min - and this is where 
> the client is running on a 2006 Solaris 2.8 SPARC - so on modern hardware 
> it would be even faster), so let's say 5 min per 20K when things are 
> working properly then for PSU scale (100K) it's 25 min - isn't this 
> acceptable based on Tom's original note?  It could be an hour for the first 
> time query.  Yes, all this presumes a proper performing environment but I 
> think that's a reasonable expectation.  (sorry for the rambling - I hope 
> all this makes sense).
> 
> /mrg
> 
> On May 18, 2012, at 9:45, Chris Hyzer wrote:
> 
>> Is there a compromise where you can get batches of 100 at once?  (in 
>> sources, grouper attributes, etc)
>> 
>> Chris
>> 
>> -----Original Message-----
>> From: 
>> 
>>  
>> [mailto:]
>>  On Behalf Of Michael R. Gettes
>> Sent: Thursday, May 17, 2012 8:15 PM
>> To: Tom Zeller
>> Cc: Grouper Dev
>> Subject: Re: [grouper-dev] speedup by caching in the ldap source adapter ?
>> 
>> Hi Tom,
>> 
>> I am very concerned about this - allowing to do uid=* is killer on the 
>> cache for the LDAP server.  It's a really nasty thing to do.  Bottom-line, 
>> it's not just a caching problem for grouper or a provisioning system, it's 
>> a caching problem for the components you're querying, like LDAP.  I should 
>> also note, as you know, there are many parameters on LDAP servers to 
>> govern limits on searching and such and SOME ldap servers offer limits on 
>> a DN basis while others do not.  Maybe a "half-way" approach would be to 
>> offer an optional "priming search" at start-up where those who can do as 
>> you are suggesting would be able to do so.
>> 
>> I have 389 and because of my situation at CMU I have to search every 
>> member of every group before making any modifications to the group.  A 
>> group with 22K members takes about 8 minutes to process after the first 
>> pass on the group when the set of member DNs gets loaded into the LDAP 
>> cache.  The first time through can take 2-3 times longer.  Additionally, I 
>> have the problem that uid=* would return a great deal more than the 22K 
>> needed for my largest group, I have about 400K objects and about 50% have 
>> uid attributes.  We are not configured to pull the full 400K objects into 
>> cache for LDAP.
>> 
>> I hope this helps.
>> 
>> /mrg
>> 
>> On May 17, 2012, at 19:43, Tom Zeller wrote:
>> 
>>> When provisioning an ldap target, it seems that querying for members
>>> takes the most time, especially when provisioning the "everyone"
>>> group. Even if the source resolver has a large cache, at least one
>>> ldap search is performed for every grouper membership.
>>> 
>>> To avoid the performance penalty and scaling issues with
>>> one-ldap-search-per-membership, I added a simple cache to the ldap
>>> source adapter, and this cache is warmed by slurping the target ldap
>>> directory at startup.
>>> 
>>> In other words, rather than n "(uid=...)" searches, just do one
>>> "(uid=*)" search.
>>> 
>>> I hesitate to give numbers, but on my local box, provisioning a group
>>> with 100k members takes approximately 25 minutes (not hours !) without
>>> the warmed-up cache. With the warmed-up cache, provisioning a group
>>> with 100k members takes more like 5 minutes.
>>> 
>>> Another option is to make sure that all subject attributes needed for
>>> provisioning are written to the grouper_members table, and avoid ldap
>>> member searches entirely.
>>> 
>>> What do you think ?
>>> 
>>> TomZ
>> 
>