Skip to Content.
Sympa Menu

grouper-dev - Re: [grouper-dev] speedup by caching in the ldap source adapter ?

Subject: Grouper Developers Forum

List archive

Re: [grouper-dev] speedup by caching in the ldap source adapter ?


Chronological Thread 
  • From: Tom Zeller <>
  • To: "Michael R. Gettes" <>
  • Cc: Grouper Dev <>
  • Subject: Re: [grouper-dev] speedup by caching in the ldap source adapter ?
  • Date: Fri, 18 May 2012 12:14:17 -0500

Actually, your half-way approach is what I was proposing : an
optional search for all subjects performed once at startup to
prime/warm the subject cache. Here's a potential configuration snippet
(sources.xml, with xml tags removed) :

searchSubject
filter : (&amp;(uid=%TERM%)(objectclass=person))
scope : subtree_scope
base : ou=people,dc=example,dc=edu

<!-- Set to true to cache subjects once at startup. -->
cache : true
<!-- The initial capacity should be slightly greater than the number
of subjects. -->
cacheInitialCapacity : 250000
<!-- The value used to replace %TERM% in the search filter. -->
cacheSearchValue : *

For what it's worth, I spent some time looking at the OpenIDM product,
and it too relies on _query_all_ids for provisioning.

TomZ

On Thu, May 17, 2012 at 7:15 PM, Michael R. Gettes
<>
wrote:
> Hi Tom,
>
> I am very concerned about this - allowing to do uid=* is killer on the
> cache for the LDAP server.  It's a really nasty thing to do.  Bottom-line,
> it's not just a caching problem for grouper or a provisioning system, it's
> a caching problem for the components you're querying, like LDAP.  I should
> also note, as you know, there are many parameters on LDAP servers to govern
> limits on searching and such and SOME ldap servers offer limits on a DN
> basis while others do not.  Maybe a "half-way" approach would be to offer
> an optional "priming search" at start-up where those who can do as you are
> suggesting would be able to do so.
>
> I have 389 and because of my situation at CMU I have to search every member
> of every group before making any modifications to the group.  A group with
> 22K members takes about 8 minutes to process after the first pass on the
> group when the set of member DNs gets loaded into the LDAP cache.  The
> first time through can take 2-3 times longer.  Additionally, I have the
> problem that uid=* would return a great deal more than the 22K needed for
> my largest group, I have about 400K objects and about 50% have uid
> attributes.  We are not configured to pull the full 400K objects into cache
> for LDAP.
>
> I hope this helps.
>
> /mrg
>
> On May 17, 2012, at 19:43, Tom Zeller wrote:
>
>> When provisioning an ldap target, it seems that querying for members
>> takes the most time, especially when provisioning the "everyone"
>> group. Even if the source resolver has a large cache, at least one
>> ldap search is performed for every grouper membership.
>>
>> To avoid the performance penalty and scaling issues with
>> one-ldap-search-per-membership, I added a simple cache to the ldap
>> source adapter, and this cache is warmed by slurping the target ldap
>> directory at startup.
>>
>> In other words, rather than n "(uid=...)" searches, just do one
>> "(uid=*)" search.
>>
>> I hesitate to give numbers, but on my local box, provisioning a group
>> with 100k members takes approximately 25 minutes (not hours !) without
>> the warmed-up cache. With the warmed-up cache, provisioning a group
>> with 100k members takes more like 5 minutes.
>>
>> Another option is to make sure that all subject attributes needed for
>> provisioning are written to the grouper_members table, and avoid ldap
>> member searches entirely.
>>
>> What do you think ?
>>
>> TomZ
>



Archive powered by MHonArc 2.6.16.

Top of Page