Grouper Users - Open Discussion List

[James Vuccolo <>]

We are using IBMs directory server v62

James "Jimmy" Vuccolo
Penn State University 
215B Computer Building
University Park, PA 16802

On Mar 11, 2011, at 2:37 PM, "Michael R. Gettes" 
<>
 wrote:

> Lynn, I am terribly sorry for not properly reading your email... I see you 
> did the test properly.  I'm still curious about which LDAP server you are 
> using.  I agree that re-provisioning the entire group is not desirable.
> 
> (thanks Chris for noting my inability to read properly :-) )
> 
> /mrg
> 
> On Mar 11, 2011, at 14:22, Michael R. Gettes wrote:
> 
>> Lynn,
>> 
>> I'm curious... what LDAP server is involved here?  It may not be relevant 
>> to the issues you're having, but I am curious.
>> 
>> Otherwise, I do find the single-threaded nature of the provisioning 
>> interesting.  LDAP servers are inherently multi-threaded and quite capable 
>> so this would be an interesting area to explore in speeding things up.  
>> I'd also add that Chris' note was indicating the problems of initial 
>> provisioning versus ongoing.  Initial always takes longer no matter what 
>> the technology used.  Would be interesting to note your timings on 
>> subsequent updates where only a few changes are involved.
>> 
>> /mrg
>> 
>> On Mar 11, 2011, at 14:08, Lynn Garrison wrote:
>> 
>>> Chris,
>>> 
>>> On Mar 11, 2011, at 12:55 PM, Chris Hyzer wrote:
>>> 
>>>> You could try Jim Fox's vt-ldap source which has better performance.  I 
>>>> think the link for it is here:
>>>> 
>>>> http://staff.washington.edu/fox/grouper/dist/
>>>> 
>>>    I need to take a look at that.  I know that you are planning on 
>>> incorporating it into the standard grouper api package.  Will that be 1.7 
>>> or 2.0?
>>>> How are you loading your group into Grouper?  Grouper loader, WS, GSH, 
>>>> API?
>>>> 15 minutes for 30k members is 30 members/sec which is pretty good I 
>>>> think.
>>>> 
>>> 
>>>    I am using gsh.  I wanted to go a quick test.  The next step is to try 
>>> other ways to load the data.
>>>> I cant really comment on ldappcng performance.
>>>> 
>>>> For loading into grouper though, it has to do a bunch of work, so even 
>>>> if it does a subject query for each row, the order of magnitude of the 
>>>> operation overall will still be the same if you remove the subject 
>>>> query.  I would assume it would be similar for ldappcng as well.  
>>>> However, you will be better off if you can load by specifying the 
>>>> subjectId and sourceId so it goes to one source with one query to 
>>>> resolve.  If you only specify a netId, and call idOrIdentifier, and not 
>>>> sourceId, that will query each source (~4?), with two queries (8 total). 
>>>>  The loader is pretty efficient about this.
>>>> 
>>>> Generally Grouper is designed assuming there is not a lot of membership 
>>>> churn, right?  I.e. you will have your 30k students in a group, and each 
>>>> day, you might add or remove a couple hundred, and then a few days a 
>>>> year, you have +- 5k.  Those loads will take a while as you see for the 
>>>> few days where the new students are entered.  
>>>> 
>>>> Do you think these performance numbers will be a problem for you?
>>>> 
>>> 
>>>    The provisioning to ldap timing will be a problem for us. We use GPFS 
>>> for our all share file system and we control group access to share file 
>>> systems with groups in ldap.  So all of are groups have to be provisioned 
>>> to ldap.  Currently we support three types of groups - standing 
>>> (department), course and user managed groups.    The standing groups are 
>>> created once and changes to the groups are made once a day.  Course 
>>> groups are created once a semester and changes are made once a day.  The 
>>> user managed groups are maintained only in ldap and changes appear as 
>>> soon as they occur.   One of the reasons that we are looking at grouper 
>>> is to make the changes to standing and course groups in real time.    
>>>    During our testing we ran into several areas of concern.   We were 
>>> running ldappcng with a bulksync  and interval of 180 seconds.   On the 
>>> first interval, the sync took about 45 minutes because the large group 
>>> had to be provisioned.  During that interval, I added several more  small 
>>> groups (2 members) to grouper.  The groups didn't appear in ldap until 
>>> the large group was provisioned.  Once  the large group was in sync, 
>>> provisioning of new groups and members happened very quickly.    I added 
>>> one member to the large group, and  the sync was back up to taking 45 
>>> minutes.  It appeared that it modified all members of the group instead 
>>> of just  the one added.  At least that is what I believe that the SPML 
>>> was telling me.  The large group is our faculty staff group and could 
>>> potentially change multiple times a day.  
>>> 
>>> Concerns
>>>    1.  Minor change to membership appears to re-provision the group
>>>    2.  Provisioning appears to be single threaded 
>>> 
>>>> Btw, I don't think it will change your performance that much, but I 
>>>> think that a JDBC source has opportunities in the future to have some 
>>>> performance improvements and feature improvements since it could bulk 
>>>> load subjects and page/sort better, but Im not an ldap person, so maybe 
>>>> we could do a similar thing there too.  At Penn we just the jdbc2 
>>>> source...
>>>> 
>>>> Thanks,
>>>> Chris
>>>> 
>>>> -----Original Message-----
>>>> From: 
>>>> 
>>>>  
>>>> [mailto:]
>>>>  On Behalf Of Lynn Garrison
>>>> Sent: Friday, March 11, 2011 11:02 AM
>>>> To: 
>>>> 
>>>> Subject: [grouper-users] ldap source vs jdbc source
>>>> 
>>>>    Our test environment at Penn State is configure with an ldap source; 
>>>> oracle for the gsa.  We recently executed a test load  with our largest 
>>>> group - ~32k faculty/staff members.  We loaded all the members to 
>>>> grouper and then provisioned them to lpad using ldappcng.  We executed 
>>>> the test several times.
>>>> 
>>>>    The load into grouper executed in 15 to 18 minutes.  
>>>>    The provision to ldap with ldappcng executed in 45 to 50 minutes.  
>>>> 
>>>>    We are using the source api and ldappcng version 1.6.3.
>>>> Questions:
>>>> 
>>>>    1.  Are these reasonable times?   
>>>>    2.  Would we see an improvement in the ldappcng execution time if we 
>>>> were using a jdbc source?  
>>>> 
>>>> 
>>>>    We are looking at replacing the current mechanism for managing groups 
>>>> - 66000+ groups, ~32k members in the largest group.  One of the 
>>>> requirements is that all groups be provision to ldap and available for 
>>>> use as soon as they are created.
>>>> 
>>> 
>>> 
>> 
>> 
>