Skip to Content.
Sympa Menu

grouper-dev - Re: [grouper-dev] speedup by caching in the ldap source adapter ?

Subject: Grouper Developers Forum

List archive

Re: [grouper-dev] speedup by caching in the ldap source adapter ?


Chronological Thread 
  • From: "Michael R. Gettes" <>
  • To: Chris Hyzer <>
  • Cc: Tom Zeller <>, Grouper Dev <>
  • Subject: Re: [grouper-dev] speedup by caching in the ldap source adapter ?
  • Date: Fri, 18 May 2012 14:35:44 +0000
  • Accept-language: en-US

are you saying batch depending on the type of source or add batching as a
config option on the DB source? if so, that's reasonable.

/mrg

On May 18, 2012, at 10:33, Chris Hyzer wrote:

> Well, if batching from ldap isn't ideal, batching from the DB does improve
> performance (member queries, attributes, etc), so maybe that would help...
>
> Thanks,
> Chris
>
> -----Original Message-----
> From: Michael R. Gettes
> [mailto:]
>
> Sent: Friday, May 18, 2012 10:01 AM
> To: Chris Hyzer
> Cc: Tom Zeller; Grouper Dev
> Subject: Re: [grouper-dev] speedup by caching in the ldap source adapter ?
>
> yeah, well, this is a bit of a challenge in the LDAP world. There's no
> real easy way to do this. Let me stew on this a bit. I presume doing as I
> now do which is the first interaction takes the longest and let normal
> operations (caching) provide the perf from that point forward isn't
> sufficient? I know, PSU scale schools can be a bit of a problem. If it
> takes me 5-8 minutes (actually, i was looking at my logs last evening after
> I sent my response and I see it's now between 3-5 min - and this is where
> the client is running on a 2006 Solaris 2.8 SPARC - so on modern hardware
> it would be even faster), so let's say 5 min per 20K when things are
> working properly then for PSU scale (100K) it's 25 min - isn't this
> acceptable based on Tom's original note? It could be an hour for the first
> time query. Yes, all this presumes a proper performing environment but I
> think that's a reasonable expectation. (sorry for the rambling - I hope
> all this makes sense).
>
> /mrg
>
> On May 18, 2012, at 9:45, Chris Hyzer wrote:
>
>> Is there a compromise where you can get batches of 100 at once? (in
>> sources, grouper attributes, etc)
>>
>> Chris
>>
>> -----Original Message-----
>> From:
>>
>>
>> [mailto:]
>> On Behalf Of Michael R. Gettes
>> Sent: Thursday, May 17, 2012 8:15 PM
>> To: Tom Zeller
>> Cc: Grouper Dev
>> Subject: Re: [grouper-dev] speedup by caching in the ldap source adapter ?
>>
>> Hi Tom,
>>
>> I am very concerned about this - allowing to do uid=* is killer on the
>> cache for the LDAP server. It's a really nasty thing to do. Bottom-line,
>> it's not just a caching problem for grouper or a provisioning system, it's
>> a caching problem for the components you're querying, like LDAP. I should
>> also note, as you know, there are many parameters on LDAP servers to
>> govern limits on searching and such and SOME ldap servers offer limits on
>> a DN basis while others do not. Maybe a "half-way" approach would be to
>> offer an optional "priming search" at start-up where those who can do as
>> you are suggesting would be able to do so.
>>
>> I have 389 and because of my situation at CMU I have to search every
>> member of every group before making any modifications to the group. A
>> group with 22K members takes about 8 minutes to process after the first
>> pass on the group when the set of member DNs gets loaded into the LDAP
>> cache. The first time through can take 2-3 times longer. Additionally, I
>> have the problem that uid=* would return a great deal more than the 22K
>> needed for my largest group, I have about 400K objects and about 50% have
>> uid attributes. We are not configured to pull the full 400K objects into
>> cache for LDAP.
>>
>> I hope this helps.
>>
>> /mrg
>>
>> On May 17, 2012, at 19:43, Tom Zeller wrote:
>>
>>> When provisioning an ldap target, it seems that querying for members
>>> takes the most time, especially when provisioning the "everyone"
>>> group. Even if the source resolver has a large cache, at least one
>>> ldap search is performed for every grouper membership.
>>>
>>> To avoid the performance penalty and scaling issues with
>>> one-ldap-search-per-membership, I added a simple cache to the ldap
>>> source adapter, and this cache is warmed by slurping the target ldap
>>> directory at startup.
>>>
>>> In other words, rather than n "(uid=...)" searches, just do one
>>> "(uid=*)" search.
>>>
>>> I hesitate to give numbers, but on my local box, provisioning a group
>>> with 100k members takes approximately 25 minutes (not hours !) without
>>> the warmed-up cache. With the warmed-up cache, provisioning a group
>>> with 100k members takes more like 5 minutes.
>>>
>>> Another option is to make sure that all subject attributes needed for
>>> provisioning are written to the grouper_members table, and avoid ldap
>>> member searches entirely.
>>>
>>> What do you think ?
>>>
>>> TomZ
>>
>




Archive powered by MHonArc 2.6.16.

Top of Page