Grouper Users - Open Discussion List

[Jim Fox <>]

> > I've not experienced this problem in our testing, and I'm a direct member of 
> > about 100 of
> > our 40K+ groups.  Many of those are in turn members of other groups.
>
> Just curious, are you a wheel group member?  That one is a lot faster when 
> checking privileges.

I'm using the grouper API, so I make a combination of
getImmediateMemberships and getEffectiveMemberships for the field 'admin'.
I expect that's similar to what you do.

> > 1) They don't scale.
> >
> >     It is abhorrent to me that a few users, adding a couple of group
> >     members to their groups, might cause a million database records to
> >     be created.  In addition this new table will slow down the addition
> >     and deletion of members.  And this is exactly where, for us, grouper
> >     has its greatest performance issues.

> FYI this is done in a daemon process in the background, and the
> performance is tuned to leverage the logic in the unflattened table,
> so it is definitely faster than it was in 1.4.  I think it would
> only cause a million database records if you have a million groups
> and add someone to all (or someone to have privileges on all).
> I'd be interested to know if you have a user added to an admin
> group which is an admin of 10's of thousand of groups, or even
> hundreds of thousands, how long that would take to propagate to
> the flattened table.

The flattened table is for members of any field, isn't it?  If I
have a group, say, u:fox:myfriends, and I add as a member the group,
uw:affiliation:alumni (which has 150K members) do I not add 150K
records to the flattened table -- one record for each effective
member of my group?  If my group's membership was not world-read
and I wanted all myfriends to see all the others I might add the
group as a reader.  Another 150K records.

> > 3) They can't be implemented in a federated world.
> >
> >     Our clients know that members of groups can be of several
> >     types:  our own uwnetids; ePPNs from other sites; dns domains,
> >     i.e. certificate CNs; Active Directory computer names; and other
> >     groups in the registry.  I can see a federated future where we
> >     have groups with members that are groups from a remote registry,
> >     say, UPenn.  That is not so far-fetched as you might think,
> >     and it shows just how 20th century are flattened memberships.

> Im not sure how federated groups affect flattened memberships.
> The flattened table is just built on a query from a view in
> grouper... if the rows aren't in the view, then they wont work
> with the API in unflattened mode either.  If we need a resolver in
>  grouper to resolve external groups or dynamic groups, then other
> logic will change in grouper and it wont be any different if we
> use flattened or not.

Yes.  My point, not so well made, is that in a federated world
membership is best calculated at the time it is required.
Anything else is likely to be out of date.  I know that doesn't
apply to the flat membership table.  It's just a general rule.
Maybe just my rule.

> > > Another case if you have a lot of web service hits and the data
> > > doesn't need to be up to date (flattened should be a few minutes
> > > behind), then we will put a mode in where the request can go against
> > > the flattened tables, and this will help the DB load.
> >
> > This is what the subordinate directories are for.

> Well, not all queries can go against subordinate
> directories... when centralized permissions are used it might be
> too complex of a data model for directories I believe.  With groups,
> getting the members of a group is easy enough.  Some queries against
> directories right now require web services depending on how you
> deploy them.  i.e. give me the groups of a user that I am allowed
> to READ... does your directory handle that?  mine doesn't and I
> think most do not.

Actually, mine does.  But that isn't exactly what I meant.  We have 
a fastcgi application that provides a subset of our group web service API
and uses the ldap directories for a database.  (It is a read-only copy of 
our old web service.)  This responds only to program clients (simple xhtml)
and only for the members-of-group and groups-for-members searches. 
If we didn't already have the cgi app I would provide the same capability
through the tomcat app.

> Anyways, your concerns are noted, we need to make sure performance
> is acceptable and discuss this further obviously... :)

Fair enough.

Jim