Grouper Developers Forum

[Tom Zeller <>]

Actually, your half-way approach is what I was proposing :  an
optional search for all subjects performed once at startup to
prime/warm the subject cache. Here's a potential configuration snippet
(sources.xml, with xml tags removed) :

searchSubject
filter : (&(uid=%TERM%)(objectclass=person))
scope : subtree_scope
base : ou=people,dc=example,dc=edu

<!-- Set to true to cache subjects once at startup. -->
cache : true
<!-- The initial capacity should be slightly greater than the number
of subjects. -->
cacheInitialCapacity : 250000
<!-- The value used to replace %TERM% in the search filter. -->
cacheSearchValue : *

For what it's worth, I spent some time looking at the OpenIDM product,
and it too relies on _query_all_ids for provisioning.

TomZ

On Thu, May 17, 2012 at 7:15 PM, Michael R. Gettes 
<>
 wrote:
> Hi Tom,
>
> I am very concerned about this - allowing to do uid=* is killer on the 
> cache for the LDAP server.  It's a really nasty thing to do.  Bottom-line, 
> it's not just a caching problem for grouper or a provisioning system, it's 
> a caching problem for the components you're querying, like LDAP.  I should 
> also note, as you know, there are many parameters on LDAP servers to govern 
> limits on searching and such and SOME ldap servers offer limits on a DN 
> basis while others do not.  Maybe a "half-way" approach would be to offer 
> an optional "priming search" at start-up where those who can do as you are 
> suggesting would be able to do so.
>
> I have 389 and because of my situation at CMU I have to search every member 
> of every group before making any modifications to the group.  A group with 
> 22K members takes about 8 minutes to process after the first pass on the 
> group when the set of member DNs gets loaded into the LDAP cache.  The 
> first time through can take 2-3 times longer.  Additionally, I have the 
> problem that uid=* would return a great deal more than the 22K needed for 
> my largest group, I have about 400K objects and about 50% have uid 
> attributes.  We are not configured to pull the full 400K objects into cache 
> for LDAP.
>
> I hope this helps.
>
> /mrg
>
> On May 17, 2012, at 19:43, Tom Zeller wrote:
>
>> When provisioning an ldap target, it seems that querying for members
>> takes the most time, especially when provisioning the "everyone"
>> group. Even if the source resolver has a large cache, at least one
>> ldap search is performed for every grouper membership.
>>
>> To avoid the performance penalty and scaling issues with
>> one-ldap-search-per-membership, I added a simple cache to the ldap
>> source adapter, and this cache is warmed by slurping the target ldap
>> directory at startup.
>>
>> In other words, rather than n "(uid=...)" searches, just do one
>> "(uid=*)" search.
>>
>> I hesitate to give numbers, but on my local box, provisioning a group
>> with 100k members takes approximately 25 minutes (not hours !) without
>> the warmed-up cache. With the warmed-up cache, provisioning a group
>> with 100k members takes more like 5 minutes.
>>
>> Another option is to make sure that all subject attributes needed for
>> provisioning are written to the grouper_members table, and avoid ldap
>> member searches entirely.
>>
>> What do you think ?
>>
>> TomZ
>