Grouper Developers Forum

[Tom Barton <>]

Cramton, James wrote:
> 1.
> The only oddity we see in the quality of the data output in LDAP is that
> some groups--and we haven't identified the pattern yet--get an
> isMemberOf attribute that includes their parent groups. Also, groups
> with child groups list the children as hasMember values. Neither of
> these is necessarily bad for us, but they are not necessary, and I
> suspect they are due to a config file setting we haven't understood
> correctly. Any suggestions?

Does every group that is a subgroup of another group get an isMemberOf 
attribute, or are some subgroups missing the attribute?

Ldappc provisions all of the direct + indirect members of a group, and 
in particular it treats groups in membership lists just the same as any 
other type of subject that may appear in a membership list. So, a group 
entry will receive a populated isMemberOf attribute if that group is a 
member of any other groups, and likewise, a group's membership list will 
include a reference to any subgroup that belongs to it.

In a membership context a group is treated like any other type of 
subject. As with any other subject source, the ldappc config file can 
contain a source-subject-identifier element that specifies how to 
identify the ldap entry corresponding to a subject from that source. The 
g:gsa source adapter presents grouper groups as subjects, and so a 
config fragment such as the following enables ldappc to maintain 
isMemberOf type attributes for groups-as-subjects, and DN-valued 
membership attributes can also reference groups that are members, just 
as is done for persons or any other type of subject:

<source-subject-identifier source="g:gsa" subject-attribute="name" >
    <ldap-search base="ou=groups,dc=brown,dc=edu"
    scope="subtree_scope"
    filter="(brownGroupRDN={0})" />
</source-subject-identifier>

> 2. 
> Performancewise, we're running Grouper and LDAPpc on Red Hat Enterprise
> Server 3 on quad processor 64-bit SunFire X4100 intel boxes with 4GB RAM
> and a nice quick SAN disk. The app server runs apache 2.2.4, tomcat 5.x
> and Sun jdk 5.x. The db server runs Oracle 10g. Our SunOne LDAP server
> runs under Solaris on a substantially less beefy box--a SunFire 280R
> dual UltraSparcIII with 2GB RAM, without a SAN connection. All servers
> are on the same subnet of a Gigabit Ethernetwork. The app server and DB
> server tend to clip right along, rarely topping 10% or 20% of total CPU
> time. Typically, the ldappc java process runs 50% - 70% of 1 CPU, so
> we're not taking full advantage of the multiprocessor systems. If Oracle
> bothers to notice the load, it distributes the load nicely across the 4
> CPUs, but I have yet to see it break a sweat--typically in the low
> single digit percentages of total CPU time. The bottleneck is i/o on the
> LDAP server. SunOne tends to chug along at 20% - 30% of CPU, with 40% -
> 50% of process time waiting on i/o. Reducing access logging on the LDAP
> server may help mitigate this issue, as we're producing 250 - 300MB of
> LDAP access logs per provisioning run. We're looking into mounting the
> LDAP logs on a separate disk or maybe the SAN as well, so we can hold
> onto some of our access stats.

Sure seems that faster ldap logging (with some combination of faster 
logging I/O and less logging) would net a substantial performance gain 
in provisioning performance with the current version of ldappc. I 
noticed the same in my own tests of ldappc provisioning. It really 
inundates the ldap server with tons of queries.

About 2 months back we spoke about redesigning part of ldappc to cache a 
great deal more info in order to speed up its computation of the 
incremental changes that need to be provisioned. Since that approach 
would reduce the number of ldap queries needed, the overall time to 
provision would also be reduced because less activity is logged by the 
ldap server.

I think a modification to ldappc along these lines is currently being 
contemplated by Kathryn, but I'll let her speak for herself.

> 3.
> Our current LDAPpc log output is much cleaner than it once was. We had
> to edit the Grouper StemFinder to prevent thousands of internal
> isChild:null log entries. But all we did was disable the logging of the
> exception. Tom and Blair have identified the issue, though. Even though
> our LDAP provisioning is performing correctly as far as we've been able
> to determine, we still produce 8MB of error logs out of LDAPpc due to
> entries like these:
>
> 2007-07-20 04:07:00,816:
> [edu.internet2.middleware.ldappc.synchronize.GroupEntrySynchronizer]
> SUBJECT[[ NAME = COURSE:AFRI:1050E:2007Fall:S01:TeachingAssistant ][ ID
> = 552adac8-92c1-472d-92c1-31a2ca8f2334 ]] Subject not found using [
> subject id = COURSE:AFRI:1050E:2007Fall:S01:TeachingAssistant ][ source
> = g:gsa ][ filter =
> [base=ou=groups,dc=brown,dc=edu][scope=2][filter=(brownGroupRDN={0})] ]
>
> The "Subject not found" errors elude us at the moment. It strikes me as
> odd to be looking in the g:gsa source adapter with an LDAP filter.

My response to #1 above explains that the logged ldap filter is what 
ldappc is using to find the ldap entry corresponding to the referenced 
group. Basically, it is saying that the ldap search

base=ou=groups,dc=brown,dc=edu
scope=2
filter=(brownGroupRDN=COURSE:AFRI:1050E:2007Fall:S01:TeachingAssistant)

produced no hits, ie, there was no ldap entry corresponding to that group.

Since this was an initial provisioning run, I wonder if this group's 
ldap entry wasn't yet created at the time ldappc was looking to refer to 
it as a subject, ie, as a member of another group. If you do a second, 
incremental, provisioning run, is the same error logged for the same group?

> 4.
> Finally, LDAPpc isn't satisfied that we've completed the run
> successfully, we think because it gets a null dto in GrouperSession. I
> wonder if the GrouperSession has expired by the time the process
> finishes?
>
> 2007-07-20 06:29:07,230:
> [edu.internet2.middleware.ldappc.LdappcGrouperProvisioner] Grouper
> Provision Failed: null dto in class
> edu.internet2.middleware.grouper.GrouperSession

Dunno what this means, but I bet blair does. Blair?

Tom

PS. When I write long messages, I sometimes reward those who read to the 
end with an offer to buy them a beer... hint... :-)