Grouper Users - Open Discussion List

[Tom Zeller <>]

When I first heard that provisioning a group with 25k members took 25
hours, along with grumblings of garbage collection, I thought : "oh
dear, I must have made a mistake". Indeed I did, sort of.

The bottleneck was not provisioning the group, but rather provisioning
the group's 25k members. In the bug reporter's environment, the time
to provision each member was 3 to 4 seconds each, hence 25+ hours.

Why so slow ? Well, for every group that every member is a member of,
the group data connector retrieves all members of every group by
calling getMembers().

It turns out that getMembers() is not cached in grouper, which makes
sense when not provisioning. Turning on caching in grouper improved
performance by half, which is great, but 1.5 to 2 seconds times 25k is
still too long.

The easy fix is to call getMembers() only when necessary, which is
during group provisioning - not member provisioning. This can be
implemented by adding a group data connector to the attribute resolver
configuration which does not call getMembers(), and this memberless
data connector should be the only dependency of attributes resolving
to group identifiers. In other words, make identifier resolution
faster.

The downside to duplicating group data connectors is that the psp will
call GroupFinder.findByName() twice instead of once for every group
under certain circumstances. This should be ameliorated by grouper's
built-in caching.

Another potential fix is, of course, caching in the psp. The psp could
cache either by request scope or by TTL via ehcache. I should take a
look at how much work this is.

Another potential fix is to write identifiers to grouper tables as attributes.

But first, I have published a new 2.1.1-snapshot, which includes the
necessary changes to psp-resolver.xml in the various example
configuration projects.

Thanks to Penn State for reporting this bug in 2.1.0 which is
scheduled to be fixed in 2.1.1.

TomZ

https://bugs.internet2.edu/jira/browse/GRP-805

The latest psp snapshot is available at

 http://www.internet2.edu/grouper/release/2.1.1/grouper.psp-2.1.1.tar.gz

which is the same as

 psp-distribution-for-grouper-2.1.1-20120612.190301-3.tar.gz

from

 
https://oss.sonatype.org/content/repositories/snapshots/edu/internet2/middleware/