Grouper Users - Open Discussion List

[Lynn Garrison <>]

Chris,

On Mar 11, 2011, at 12:55 PM, Chris Hyzer wrote:

> You could try Jim Fox's vt-ldap source which has better performance.  I 
> think the link for it is here:
> 
> http://staff.washington.edu/fox/grouper/dist/
> 
        I need to take a look at that.  I know that you are planning on 
incorporating it into the standard grouper api package.  Will that be 1.7 or 
2.0?
> How are you loading your group into Grouper?  Grouper loader, WS, GSH, API?
> 15 minutes for 30k members is 30 members/sec which is pretty good I think.
> 

        I am using gsh.  I wanted to go a quick test.  The next step is to 
try other ways to load the data.
> I cant really comment on ldappcng performance.
> 
> For loading into grouper though, it has to do a bunch of work, so even if 
> it does a subject query for each row, the order of magnitude of the 
> operation overall will still be the same if you remove the subject query.  
> I would assume it would be similar for ldappcng as well.  However, you will 
> be better off if you can load by specifying the subjectId and sourceId so 
> it goes to one source with one query to resolve.  If you only specify a 
> netId, and call idOrIdentifier, and not sourceId, that will query each 
> source (~4?), with two queries (8 total).  The loader is pretty efficient 
> about this.
> 
> Generally Grouper is designed assuming there is not a lot of membership 
> churn, right?  I.e. you will have your 30k students in a group, and each 
> day, you might add or remove a couple hundred, and then a few days a year, 
> you have +- 5k.  Those loads will take a while as you see for the few days 
> where the new students are entered.  
> 
> Do you think these performance numbers will be a problem for you?
> 

        The provisioning to ldap timing will be a problem for us. We use GPFS 
for our all share file system and we control group access to share file 
systems with groups in ldap.  So all of are groups have to be provisioned to 
ldap.  Currently we support three types of groups - standing (department), 
course and user managed groups.    The standing groups are created once and 
changes to the groups are made once a day.  Course groups are created once a 
semester and changes are made once a day.  The user managed groups are 
maintained only in ldap and changes appear as soon as they occur.   One of 
the reasons that we are looking at grouper is to make the changes to standing 
and course groups in real time.    
        During our testing we ran into several areas of concern.   We were 
running ldappcng with a bulksync  and interval of 180 seconds.   On the first 
interval, the sync took about 45 minutes because the large group had to be 
provisioned.  During that interval, I added several more  small groups (2 
members) to grouper.  The groups didn't appear in ldap until the large group 
was provisioned.  Once  the large group was in sync, provisioning of new 
groups and members happened very quickly.    I added one member to the large 
group, and  the sync was back up to taking 45 minutes.  It appeared that it 
modified all members of the group instead of just  the one added.  At least 
that is what I believe that the SPML was telling me.  The large group is our 
faculty staff group and could potentially change multiple times a day.  

Concerns
        1.  Minor change to membership appears to re-provision the group
        2.  Provisioning appears to be single threaded 

> Btw, I don't think it will change your performance that much, but I think 
> that a JDBC source has opportunities in the future to have some performance 
> improvements and feature improvements since it could bulk load subjects and 
> page/sort better, but Im not an ldap person, so maybe we could do a similar 
> thing there too.  At Penn we just the jdbc2 source...
> 
> Thanks,
> Chris
> 
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Lynn Garrison
> Sent: Friday, March 11, 2011 11:02 AM
> To: 
> 
> Subject: [grouper-users] ldap source vs jdbc source
> 
>       Our test environment at Penn State is configure with an ldap source; 
> oracle for the gsa.  We recently executed a test load  with our largest 
> group - ~32k faculty/staff members.  We loaded all the members to grouper 
> and then provisioned them to lpad using ldappcng.  We executed the test 
> several times.
> 
>       The load into grouper executed in 15 to 18 minutes.  
>       The provision to ldap with ldappcng executed in 45 to 50 minutes.  
> 
>       We are using the source api and ldappcng version 1.6.3.
> Questions:
> 
>       1.  Are these reasonable times?   
>       2.  Would we see an improvement in the ldappcng execution time if we 
> were using a jdbc source?  
> 
> 
>       We are looking at replacing the current mechanism for managing groups 
> - 66000+ groups, ~32k members in the largest group.  One of the 
> requirements is that all groups be provision to ldap and available for use 
> as soon as they are created.
>