Grouper Developers Forum

[Gary Chapman <>]

Although not a "solution", what we've always done to meet the need for an occasional (uid=*), is point such a query at an LDAP replica that tends not to be used for normal read operations -- using some replicas for LDAP reads, and others for writes.  We front the LDAP directories with an F5 switch and have two VIPs (one for read traffic and one for write traffic) that point to different pairs of backend servers.

- Gary

On Thu, May 17, 2012 at 8:15 PM, Michael R. Gettes <> wrote:

Hi Tom,

I am very concerned about this - allowing to do uid=* is killer on the cache for the LDAP server.  It's a really nasty thing to do.  Bottom-line, it's not just a caching problem for grouper or a provisioning system, it's a caching problem for the components you're querying, like LDAP.  I should also note, as you know, there are many parameters on LDAP servers to govern limits on searching and such and SOME ldap servers offer limits on a DN basis while others do not.  Maybe a "half-way" approach would be to offer an optional "priming search" at start-up where those who can do as you are suggesting would be able to do so.

I have 389 and because of my situation at CMU I have to search every member of every group before making any modifications to the group.  A group with 22K members takes about 8 minutes to process after the first pass on the group when the set of member DNs gets loaded into the LDAP cache.  The first time through can take 2-3 times longer.  Additionally, I have the problem that uid=* would return a great deal more than the 22K needed for my largest group, I have about 400K objects and about 50% have uid attributes.  We are not configured to pull the full 400K objects into cache for LDAP.

I hope this helps.

/mrg

On May 17, 2012, at 19:43, Tom Zeller wrote:

> When provisioning an ldap target, it seems that querying for members
> takes the most time, especially when provisioning the "everyone"
> group. Even if the source resolver has a large cache, at least one
> ldap search is performed for every grouper membership.
>
> To avoid the performance penalty and scaling issues with
> one-ldap-search-per-membership, I added a simple cache to the ldap
> source adapter, and this cache is warmed by slurping the target ldap
> directory at startup.
>
> In other words, rather than n "(uid=...)" searches, just do one
> "(uid=*)" search.
>
> I hesitate to give numbers, but on my local box, provisioning a group
> with 100k members takes approximately 25 minutes (not hours !) without
> the warmed-up cache. With the warmed-up cache, provisioning a group
> with 100k members takes more like 5 minutes.
>
> Another option is to make sure that all subject attributes needed for
> provisioning are written to the grouper_members table, and avoid ldap
> member searches entirely.
>
> What do you think ?
>
> TomZ