Grouper Users - Open Discussion List

[Jim Fox <>]

> > So my question is: What is the consequence of running without the
> > change log, ignoring the flattened memberships table altogether?
> > Are there any core functions of grouper that are going to depend
> > on this table?
>
> As Tom mentioned, the notifications are more efficient when
> the external system doesn't care about why someone is a member,
> just that they are a member.  Also, that table is more efficient
> to query.  For instance, if someone is an admin of many groups,
> and that person goes to the UI to in "Manage groups" mode, then
> Grouper needs to calculate which groups the person can admin.
> From the normal registry this has been reported too slow, but I
> believe the flattened table can do it quickly (we still need to
> work this part out).

I've not experienced this problem in our testing, and I'm a direct
member of about 100 of our 40K+ groups.  Many of those are in turn
members of other groups.

I have several objections to flattened memberships:

1) They don't scale.

   It is abhorrent to me that a few users, adding a couple of group
   members to their groups, might cause a million database records to
   be created.  In addition this new table will slow down the addition
   and deletion of members.  And this is exactly where, for us, grouper
   has its greatest performance issues.

2) They aren't necessary.

   We export our group memberships to three places: openldap directories;
   an active directory; and (coming soon) google groups.  Each of these 
supports
   nested group memberships. For a good reason I believe.

3) They can't be implemented in a federated world.

   Our clients know that members of groups can be of several
   types:  our own uwnetids; ePPNs from other sites; dns domains,
   i.e. certificate CNs; Active Directory computer names; and other
   groups in the registry.  I can see a federated future where we
   have groups with members that are groups from a remote registry,
   say, UPenn.  That is not so far-fetched as you might think,
   and it shows just how 20th century are flattened memberships.

> Another case if you have a lot of web service hits and the data
> doesn't need to be up to date (flattened should be a few minutes
> behind), then we will put a mode in where the request can go against
> the flattened tables, and this will help the DB load.

This is what the subordinate directories are for.

We find that about 90% of hits on our service are two types of queries:

  - find groups a user is an effective member of

  - find all members af a group

These are also the two queries that require the fastest response
time and the highest reliability.  Our Shibboleth IdP, for example,
uses the first query.  To provide the best support for these requests
we implement an alternate url that accesses our LDAP directories,
of which there are always at least two, rather than the registry.
Even with the web service resolving nested memberships at runtime,
the groups-for-user for me and my 100 or so groups, takes 50-70 msec.
Accessing the registry directly takes 500-1000 mesc.

A few minutes is an eternity around here.  Our LDAP directories 
are up to date in about 5 sec.

> Anyways, bottom line is, sorry this means you need more storage
> space in the DB, but it would be nice to assume that people use
> this feature.  If you strongly disagree (and perhaps either way)
> maybe we can discuss at a dev call some time.

For reasons just described I do disagree. We find the hook mechanism
much more efficient for event notification that the event log, we
already offload the most common queries to the ldap directories,
and have no other use for the flattened tables.  I don't want the
overhead of maintaining them, or of running the event log.

Jim