Grouper Users - Open Discussion List

["Cramton, James" <>]

There are 2 different kinds of 'large groups' in this space. One is
groups with many members...Cornell has direct experience trying to add
and update groups with upwards of 200,000 members. From what I recall in
their trials and tribulations, there are some LDAP performance issues
writing that many group members to LDAP. So I don't believe the
bottleneck for that use case is Grouper itself, rather the interface to
LDAP.  Perhaps Cornell can comment further.

At Brown, our largest group contains 30,000 members, so we have not had
trouble of Cornell's sort. But we do have on the order of 250,000 groups
in MACE Grouper, and our average group size, if I remember correctly,
was somewhere between 20 and 30. We have about 1,000 demographic,
departmental, or ad-hoc groups, and the rest (the vast majority) of our
groups are course groups. We organize our course groups into 12 distinct
ldap-visible groups for each of 2500 course sections per term. After a
few terms, it adds up pretty fast--each term has around 30,000 course
groups.

We actively provision the demographic, departmental, and ad-hoc groups
into LDAP in batches, but we stop provisioning course groups for past
terms after the term ends.  This leaves us with around 30,000 course
groups for the current term that we could potentially provision daily.
For all intents and purposes, we run an analog of ldappc, which we start
daily after we receive a nightly feed from the upstream business and
student systems. Once the first run of the day completes, a new LDAP
provisioning process kicks off, one after the other, until we're close
to one cycle before the end of the day. We typically run 10 or 11
provisioning runs in a day, averaging around an hour and a half per
cycle. At the beginning of the term, our provisioning runtimes
definitely increase, to the neighborhood of 3 hours, finishing 5 cycles
per day. 

Reading between the lines, it takes our ldappc-like software 90 minutes
just to cycle through 250,000 groups and determine which should be
provisioned to LDAP, based on config file settings (that identify the
stems visible in LDAP, and the current course group term) and on whether
or not there were changes to the LDAP-provisioned groups. Even if there
are no changes.  Once we start adding the high level of change at the
beginning of a term, with students adding and dropping courses daily,
our runtime doubles. But even with a lot of groups, we manage to
provision several times a day.  

When we free up some cycles, we plan to use Grouper's hooks to devise a
real-time provisioning mechanism, which will make our groups
provisioning service vastly more responsive to the community's needs
than it is today.

And it should be noted, as others have pointed out, that hardware
matters throughout the infrastructure. Most of the systems in our mix
(MACE Grouper, Oracle, and SunOne LDAP registry) are quad-cpu 64-bit x86
SunFire 4100s on a fiber network with disks mounted from the SAN. But we
are not clustered, except the directory and ldapauth ldap servers that
replicate changes MACE Grouper sends to the registry LDAP server.

Geez! Ask a simple question, and look what  you get!

James

> -----Original Message-----
> From: Gabby Capitanchik 
> [mailto:]
> Sent: Tuesday, January 27, 2009 11:21 AM
> To: GW Brown, Information Systems and Computing
> Cc: 
> ;
>  Tim gray
> Subject: Re: [grouper-users] More Grouper Question
> 
> Hi,
> 
>  > Duke, Brown, Cornell and others probably have direct experience,
but
>  > generating lots of new groups and/or memberships will take some
time
>  > 8hrs+ I would guess depending on your systems and the numbers
> involved
> 
> By lots do you mean 100s, 1000s or 10000s?
> 
> Cheers
> 
> Gabby Capitanchik
> 
> GW Brown, Information Systems and Computing wrote:
> > --On 27 January 2009 15:49 +0000 Gabby Capitanchik
> > <>
> >  wrote:
> >
> >> Hi,
> >>
> >> Thanks for the responses so far! I have been reading through the
> >> documentation and have a couple of more questions for you guys:-
> >>
> >> 1) Does Grouper take a copy of Identity information from an IDM/ID
> Store
> >> or does it hold a reference to that data. For example, a uuid?
> > Grouper maintains subject ids in the grouper_members table. It is up
> to
> > the source adapter to map the subject id to a uuid in the IDM/ID
> Store.
> >>
> >> 2) Has anyone got experience of how Grouper handle large volumes of
> group
> >> change. For example, at the start of an academic year (for student
> >> groups) or when an Organisational structure changes (for staff)
> > Duke, Brown, Cornell and others probably have direct experience, but
> > generating lots of new groups and/or memberships will take some time
> > 8hrs+ I would guess depending on your systems and the numbers
> involved
> >>
> >> 3)What is the maximum number of groups that Grouper can handle?
> > There is no limit as such - assuming your RDBMS can hold the data
and
> > indexes. Depending on how groups are structured in a stem hierarchy
> and
> > how privileges are assigned some operations, particularly in the UI,
> may
> > be slow.
> >
> > I believe sites are using 100,000s of groups and millions of
> memberships.
> >>
> >> Cheers
> >>
> >> Gabby
> >>
> >>
> >> --
> >> The University of Edinburgh is a charitable body, registered in
> >> Scotland, with registration number SC005336.
> >>
> >
> >
> >
> > ----------------------
> > GW Brown, Information Systems and Computing
> > 
> >
> 
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.