Skip to Content.
Sympa Menu

grouper-dev - Re: [grouper-dev] Brown LDAPpc 1.0 Stats

Subject: Grouper Developers Forum

List archive

Re: [grouper-dev] Brown LDAPpc 1.0 Stats


Chronological Thread 
  • From: Tom Barton <>
  • To: "Cramton, James" <>
  • Cc:
  • Subject: Re: [grouper-dev] Brown LDAPpc 1.0 Stats
  • Date: Sat, 21 Jul 2007 11:14:30 -0500



Cramton, James wrote:
1.
The only oddity we see in the quality of the data output in LDAP is that
some groups--and we haven't identified the pattern yet--get an
isMemberOf attribute that includes their parent groups. Also, groups
with child groups list the children as hasMember values. Neither of
these is necessarily bad for us, but they are not necessary, and I
suspect they are due to a config file setting we haven't understood
correctly. Any suggestions?

Does every group that is a subgroup of another group get an isMemberOf attribute, or are some subgroups missing the attribute?

Ldappc provisions all of the direct + indirect members of a group, and in particular it treats groups in membership lists just the same as any other type of subject that may appear in a membership list. So, a group entry will receive a populated isMemberOf attribute if that group is a member of any other groups, and likewise, a group's membership list will include a reference to any subgroup that belongs to it.

In a membership context a group is treated like any other type of subject. As with any other subject source, the ldappc config file can contain a source-subject-identifier element that specifies how to identify the ldap entry corresponding to a subject from that source. The g:gsa source adapter presents grouper groups as subjects, and so a config fragment such as the following enables ldappc to maintain isMemberOf type attributes for groups-as-subjects, and DN-valued membership attributes can also reference groups that are members, just as is done for persons or any other type of subject:

<source-subject-identifier source="g:gsa" subject-attribute="name" >
<ldap-search base="ou=groups,dc=brown,dc=edu"
scope="subtree_scope"
filter="(brownGroupRDN={0})" />
</source-subject-identifier>

2. Performancewise, we're running Grouper and LDAPpc on Red Hat Enterprise
Server 3 on quad processor 64-bit SunFire X4100 intel boxes with 4GB RAM
and a nice quick SAN disk. The app server runs apache 2.2.4, tomcat 5.x
and Sun jdk 5.x. The db server runs Oracle 10g. Our SunOne LDAP server
runs under Solaris on a substantially less beefy box--a SunFire 280R
dual UltraSparcIII with 2GB RAM, without a SAN connection. All servers
are on the same subnet of a Gigabit Ethernetwork. The app server and DB
server tend to clip right along, rarely topping 10% or 20% of total CPU
time. Typically, the ldappc java process runs 50% - 70% of 1 CPU, so
we're not taking full advantage of the multiprocessor systems. If Oracle
bothers to notice the load, it distributes the load nicely across the 4
CPUs, but I have yet to see it break a sweat--typically in the low
single digit percentages of total CPU time. The bottleneck is i/o on the
LDAP server. SunOne tends to chug along at 20% - 30% of CPU, with 40% -
50% of process time waiting on i/o. Reducing access logging on the LDAP
server may help mitigate this issue, as we're producing 250 - 300MB of
LDAP access logs per provisioning run. We're looking into mounting the
LDAP logs on a separate disk or maybe the SAN as well, so we can hold
onto some of our access stats.

Sure seems that faster ldap logging (with some combination of faster logging I/O and less logging) would net a substantial performance gain in provisioning performance with the current version of ldappc. I noticed the same in my own tests of ldappc provisioning. It really inundates the ldap server with tons of queries.

About 2 months back we spoke about redesigning part of ldappc to cache a great deal more info in order to speed up its computation of the incremental changes that need to be provisioned. Since that approach would reduce the number of ldap queries needed, the overall time to provision would also be reduced because less activity is logged by the ldap server.

I think a modification to ldappc along these lines is currently being contemplated by Kathryn, but I'll let her speak for herself.

3.
Our current LDAPpc log output is much cleaner than it once was. We had
to edit the Grouper StemFinder to prevent thousands of internal
isChild:null log entries. But all we did was disable the logging of the
exception. Tom and Blair have identified the issue, though. Even though
our LDAP provisioning is performing correctly as far as we've been able
to determine, we still produce 8MB of error logs out of LDAPpc due to
entries like these:

2007-07-20 04:07:00,816:
[edu.internet2.middleware.ldappc.synchronize.GroupEntrySynchronizer]
SUBJECT[[ NAME = COURSE:AFRI:1050E:2007Fall:S01:TeachingAssistant ][ ID
= 552adac8-92c1-472d-92c1-31a2ca8f2334 ]] Subject not found using [
subject id = COURSE:AFRI:1050E:2007Fall:S01:TeachingAssistant ][ source
= g:gsa ][ filter =
[base=ou=groups,dc=brown,dc=edu][scope=2][filter=(brownGroupRDN={0})] ]

The "Subject not found" errors elude us at the moment. It strikes me as
odd to be looking in the g:gsa source adapter with an LDAP filter.

My response to #1 above explains that the logged ldap filter is what ldappc is using to find the ldap entry corresponding to the referenced group. Basically, it is saying that the ldap search

base=ou=groups,dc=brown,dc=edu
scope=2
filter=(brownGroupRDN=COURSE:AFRI:1050E:2007Fall:S01:TeachingAssistant)

produced no hits, ie, there was no ldap entry corresponding to that group.

Since this was an initial provisioning run, I wonder if this group's ldap entry wasn't yet created at the time ldappc was looking to refer to it as a subject, ie, as a member of another group. If you do a second, incremental, provisioning run, is the same error logged for the same group?

4.
Finally, LDAPpc isn't satisfied that we've completed the run
successfully, we think because it gets a null dto in GrouperSession. I
wonder if the GrouperSession has expired by the time the process
finishes?

2007-07-20 06:29:07,230:
[edu.internet2.middleware.ldappc.LdappcGrouperProvisioner] Grouper
Provision Failed: null dto in class
edu.internet2.middleware.grouper.GrouperSession

Dunno what this means, but I bet blair does. Blair?

Tom

PS. When I write long messages, I sometimes reward those who read to the end with an offer to buy them a beer... hint... :-)



Archive powered by MHonArc 2.6.16.

Top of Page