Grouper Developers Forum

Brown's original work with ldappc was using an ldap-based person 
source and an oracle-based repository for the Grouper tables. We had 
three top level stems that contained large numbers of groups; we 
experimented with replicating various combinations into ldap. The 
COMMUNITY stem has about 700 groups (some of which contain ~ 18K 
members); the EAB stem also has about 700 groups (some of which 
contain ~ 18K members); the COURSES stem has about 10K groups (most 
of which are very small (eg < 10 members); several hundred might have 
50-100 members; and there are probably a few groups with 100-500 
members). Note that we have been asking ldappc to a) maintain the 
hasMember attribute of the group objects, and b) maintain the 
isMemberOf attribute of the user objects.

Using the ldap-based person registry, we had a great deal of 
difficulty getting ldappc to run to completion. It could replicate 
smaller groups without problems. However, as soon as it encountered 
the larger groups, problems started. Most of this work was done 5-6 
weeks ago (before CAMP), and my memory is a bit hazy. I do remember 
typing to copy one of the stems with large groups; ldappc ran for 
four days, at which point I killed it. I also ran into lots of 
problems where the java jndi connector reported all sorts of problems 
(which appeared to mean that it had mismanaged its buffer pool -- but 
that's just a guess). Although it ran to completion,it effectively 
stopped replicating when the buffer pool got blown. At that point, we 
gave up on the ldap-based person source.

We created a simple table in oracle that could serve as a person 
source, and began using that. We then started running ldappc, using 
oracle as a person source (replacing ldap).

1) the initial replication of the EAB groups from Grouper to ldap 
took four hours.

2) re-running ldappc, replicating the EAB groups (but no changes were 
detected, or needed) took 40 minutes.

3) re-running ldappc, and pruning back some of the logging, dropped 
the run time to 30 minutes.

4) we deleted all the groups from ldap, to give a clean starting point

5) re-running ldappc, replicating all of  the EAB and COURSES groups, 
took five hours.

6) pruned some more logging, re-ran step 5 (should make no changes), 
and it took ten minutes. (This number truly shocked me -- I think we 
have to re-validate this number).

We're going to re-run ldappc again, using an ldap-based person 
source, the see whether the problem is a) somewhere in the jndi 
support, or b) we've just learned a whole lot more about ldappc and 
can get it to run in a useful fashion with either person source. 
Based on other comments, tho, we think most of the problems are in 
the jndi support...

One other comment, based on reviewing the above numbers......

The operation which seemed to take an immense amount of time was 
maintaining the membership of the very large groups in the EAB stem. 
We're continuing to prune down the contents of the logs, and hope to 
soon be able to look at time stamps in the logs and provide some 
numbers.

However, my sense is that the algorithm that ldappc uses to maintain 
the membership of these groups -- when a run includes a number of 
membership changes -- may be less than optimal....

Other than that... if the performance numbers we're now seeing 
continue to hold up.... we're now thinking this will be sufficient 
for our planned Fall Semester rollout...

... note that by the time we get to phases two, three, and beyond, 
things will to improve further. But, we're happy for now!