Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] ldap source vs jdbc source

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] ldap source vs jdbc source

Chronological Thread 
  • From: Lynn Garrison <>
  • To: Chris Hyzer <>
  • Cc: "" <>
  • Subject: Re: [grouper-users] ldap source vs jdbc source
  • Date: Fri, 11 Mar 2011 14:08:01 -0500


On Mar 11, 2011, at 12:55 PM, Chris Hyzer wrote:

> You could try Jim Fox's vt-ldap source which has better performance. I
> think the link for it is here:
I need to take a look at that. I know that you are planning on
incorporating it into the standard grouper api package. Will that be 1.7 or
> How are you loading your group into Grouper? Grouper loader, WS, GSH, API?
> 15 minutes for 30k members is 30 members/sec which is pretty good I think.

I am using gsh. I wanted to go a quick test. The next step is to
try other ways to load the data.
> I cant really comment on ldappcng performance.
> For loading into grouper though, it has to do a bunch of work, so even if
> it does a subject query for each row, the order of magnitude of the
> operation overall will still be the same if you remove the subject query.
> I would assume it would be similar for ldappcng as well. However, you will
> be better off if you can load by specifying the subjectId and sourceId so
> it goes to one source with one query to resolve. If you only specify a
> netId, and call idOrIdentifier, and not sourceId, that will query each
> source (~4?), with two queries (8 total). The loader is pretty efficient
> about this.
> Generally Grouper is designed assuming there is not a lot of membership
> churn, right? I.e. you will have your 30k students in a group, and each
> day, you might add or remove a couple hundred, and then a few days a year,
> you have +- 5k. Those loads will take a while as you see for the few days
> where the new students are entered.
> Do you think these performance numbers will be a problem for you?

The provisioning to ldap timing will be a problem for us. We use GPFS
for our all share file system and we control group access to share file
systems with groups in ldap. So all of are groups have to be provisioned to
ldap. Currently we support three types of groups - standing (department),
course and user managed groups. The standing groups are created once and
changes to the groups are made once a day. Course groups are created once a
semester and changes are made once a day. The user managed groups are
maintained only in ldap and changes appear as soon as they occur. One of
the reasons that we are looking at grouper is to make the changes to standing
and course groups in real time.
During our testing we ran into several areas of concern. We were
running ldappcng with a bulksync and interval of 180 seconds. On the first
interval, the sync took about 45 minutes because the large group had to be
provisioned. During that interval, I added several more small groups (2
members) to grouper. The groups didn't appear in ldap until the large group
was provisioned. Once the large group was in sync, provisioning of new
groups and members happened very quickly. I added one member to the large
group, and the sync was back up to taking 45 minutes. It appeared that it
modified all members of the group instead of just the one added. At least
that is what I believe that the SPML was telling me. The large group is our
faculty staff group and could potentially change multiple times a day.

1. Minor change to membership appears to re-provision the group
2. Provisioning appears to be single threaded

> Btw, I don't think it will change your performance that much, but I think
> that a JDBC source has opportunities in the future to have some performance
> improvements and feature improvements since it could bulk load subjects and
> page/sort better, but Im not an ldap person, so maybe we could do a similar
> thing there too. At Penn we just the jdbc2 source...
> Thanks,
> Chris
> -----Original Message-----
> From:
> [mailto:]
> On Behalf Of Lynn Garrison
> Sent: Friday, March 11, 2011 11:02 AM
> To:
> Subject: [grouper-users] ldap source vs jdbc source
> Our test environment at Penn State is configure with an ldap source;
> oracle for the gsa. We recently executed a test load with our largest
> group - ~32k faculty/staff members. We loaded all the members to grouper
> and then provisioned them to lpad using ldappcng. We executed the test
> several times.
> The load into grouper executed in 15 to 18 minutes.
> The provision to ldap with ldappcng executed in 45 to 50 minutes.
> We are using the source api and ldappcng version 1.6.3.
> Questions:
> 1. Are these reasonable times?
> 2. Would we see an improvement in the ldappcng execution time if we
> were using a jdbc source?
> We are looking at replacing the current mechanism for managing groups
> - 66000+ groups, ~32k members in the largest group. One of the
> requirements is that all groups be provision to ldap and available for use
> as soon as they are created.

Archive powered by MHonArc 2.6.16.

Top of Page