Grouper Users - Open Discussion List

[Chris Hyzer <>]

I don’t know off the top of my head, but we are interested in working with 
you to figure it out at some point...  hopefully there is caching that would 
help :).  Ideally you could run a profiler like yourkit to see what the 
bottleneck is.  If that is not possible, you could describe in great detail 
the size/distribution of your database, the environment and sizing of 
hardwar, and what operations you are doing, and we could try to reproduce it 
and use yourkit...

Thanks,
Chris

-----Original Message-----
From: Julien Gribonvald 
[mailto:]
 
Sent: Monday, August 29, 2011 12:20 PM
To: Chris Hyzer
Cc: 

Subject: Re: [grouper-users] Grouper 1.6.3 : Problem to desactivate log 
generation in database

Thanks for your response Chris.

Our problem is that grouper is slower version after version for us, and we 
think that some tweaks like the possibility of disabling insert of chanlog in 
some case by a flag could a be good thing. Also my purpose to have two level 
of subject object used by the api could be a good tweak too, mainly for 
ldappc where it's not usefull in some context to get all information of the 
user. Such options would be usefull for us, for example we deconnected on 
ldappc the actions to make the request on ldap (user source for us) as we are 
sure that users exists and so it does only ldap provisioning of groups 
attributes. For example a such modification for us permitted to have in less 
than 15 min the provisionning of all groups attributes (16 000) for all users 
(200 000) with an average of 8 groups per user. If we didn't make this 
modification the provisionning run during several hours before to be ok.

After for our provisionning we developped some code that use the grouper api, 
this worked very well with grouper 1.4 and 1.5, but now it's slow with 
grouper 1.6.3 (but nothing was modified on method that we use), do you have 
any idea on our problem ?

Thanks

Julien


Le 29/08/2011 17:43, Chris Hyzer a écrit :
> Originally those flags did something, but then we went down the path of not 
> being sure what the impact of turning it off would be, so there is no way 
> of turning it off.  i.e. real time ldappcng will use the change log, and 
> all notifications already do, etc...
>
> I put in a Jira to clean up those flags though:
>
> https://bugs.internet2.edu/jira/browse/GRP-638
>
> When resolving this issue, maybe we can make the flag work if we thought it 
> was ok to not use the change log for some things...
>
> If there is more discussion Im happy to discuss :)
>
> Thanks,
> Chris
>
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Julien 
> Gribonvald
> Sent: Monday, August 29, 2011 9:36 AM
> Cc: 
> 
> Subject: [grouper-users] Grouper 1.6.3 : Problem to desactivate log 
> generation in database
>
> Hi,
>
> I'm having problems to avoid that grouper generate logs in database for 
> each operations done (in some case only, because we need logs by the UI 
> only). I want to avoid this generation because we have massive 
> modifications (16000 groups and around 200 000 users to add in some 
> groups), and so it adds useless insert with these modifications. Maybe it's 
> an other problem because with a good configuration of grouper.properties 
> file in grouper 1.5.2 it's doesn't take so much time and now after 
> migration on grouper 1.6.3 it's really slow, the provisioning time is 
> multiplied at least by 2. I have done an analyze and an optimize on the 
> database but nothing changed.
>
> I think the first optimisation would be to avoid log generation in 
> database, I tried this configuration in grouper.properties :
>
> changeLog.enabled = false
> groups.updateLastMembershipTime = false stems.updateLastMembershipTime 
> = false
>
> But this configuration doesn't seem to works because insert are done in 
> grouper_change_log_entry_temp and grouper_audit_entry tables. Do you know 
> why this change nothing ?
> Also we are using ldappcng and after tests changeLog.enabled to false 
> doesn't change anything... But we modified a bit ldappcng to avoid that it 
> gets user information in our ldap (the source of user) which was very 
> expansive in time. As purpose, a good improvement could be done by avoiding 
> request on user source which gets all user parameters because a lot of time 
> we don't need user informations, only source identifier and id of the user 
> are sufficient, for example a simple subject and a complete subject which 
> expand simpleSubject would be great.
>
> Thanks for your help (and sorry for my English).
>
> Julien