Subject: Grouper Developers Forum
- Subject: some results with ldappc
- Date: Tue, 17 Jul 2007 13:35:39 -0400
Brown's original work with ldappc was using an ldap-based person source and an oracle-based repository for the Grouper tables. We had three top level stems that contained large numbers of groups; we experimented with replicating various combinations into ldap. The COMMUNITY stem has about 700 groups (some of which contain ~ 18K members); the EAB stem also has about 700 groups (some of which contain ~ 18K members); the COURSES stem has about 10K groups (most of which are very small (eg < 10 members); several hundred might have 50-100 members; and there are probably a few groups with 100-500 members). Note that we have been asking ldappc to a) maintain the hasMember attribute of the group objects, and b) maintain the isMemberOf attribute of the user objects.
Using the ldap-based person registry, we had a great deal of difficulty getting ldappc to run to completion. It could replicate smaller groups without problems. However, as soon as it encountered the larger groups, problems started. Most of this work was done 5-6 weeks ago (before CAMP), and my memory is a bit hazy. I do remember typing to copy one of the stems with large groups; ldappc ran for four days, at which point I killed it. I also ran into lots of problems where the java jndi connector reported all sorts of problems (which appeared to mean that it had mismanaged its buffer pool -- but that's just a guess). Although it ran to completion,it effectively stopped replicating when the buffer pool got blown. At that point, we gave up on the ldap-based person source.
We created a simple table in oracle that could serve as a person source, and began using that. We then started running ldappc, using oracle as a person source (replacing ldap).
1) the initial replication of the EAB groups from Grouper to ldap took four hours.
2) re-running ldappc, replicating the EAB groups (but no changes were detected, or needed) took 40 minutes.
3) re-running ldappc, and pruning back some of the logging, dropped the run time to 30 minutes.
4) we deleted all the groups from ldap, to give a clean starting point
5) re-running ldappc, replicating all of the EAB and COURSES groups, took five hours.
6) pruned some more logging, re-ran step 5 (should make no changes), and it took ten minutes. (This number truly shocked me -- I think we have to re-validate this number).
We're going to re-run ldappc again, using an ldap-based person source, the see whether the problem is a) somewhere in the jndi support, or b) we've just learned a whole lot more about ldappc and can get it to run in a useful fashion with either person source. Based on other comments, tho, we think most of the problems are in the jndi support...
One other comment, based on reviewing the above numbers......
The operation which seemed to take an immense amount of time was maintaining the membership of the very large groups in the EAB stem. We're continuing to prune down the contents of the logs, and hope to soon be able to look at time stamps in the logs and provide some numbers.
However, my sense is that the algorithm that ldappc uses to maintain the membership of these groups -- when a run includes a number of membership changes -- may be less than optimal....
Other than that... if the performance numbers we're now seeing continue to hold up.... we're now thinking this will be sufficient for our planned Fall Semester rollout...
... note that by the time we get to phases two, three, and beyond, things will to improve further. But, we're happy for now!
- some results with ldappc, Steven_Carmody, 07/17/2007
- Re: [grouper-dev] some results with ldappc, Tom Barton, 07/17/2007
Archive powered by MHonArc 2.6.16.