Grouper Developers Forum

[Chris Hyzer <>]

Subjects should be cached in grouper.ehcache.xml, all the entries with 
CachingResolver, maybe bumping those up would help.  Also, I know the caching 
in 2.0.0 ldappcng is improved.

Thanks,
Chris


-----Original Message-----
From: Julien Gribonvald 
[mailto:]
 
Sent: Monday, September 05, 2011 8:21 AM
To: Chris Hyzer
Subject: Re: [grouper-users] Grouper 1.6.3 : Problem to desactivate log 
generation in database

Hi Chris,

Thanks for your help, finaly I found the problem whci came from one LDAP wich 
have problem in requesting with the grouper user... But if there were a 
possibility to avoid that grouper do request on ldap for each request it 
would be nice, like have some option to create a subject without requesting 
ldap directly (maybe having an option on the subject who tell that the user 
was check from his source and an other option for runtime environment which 
tell to verify/get or not the user from his source). This could make faster 
significantly grouper in many request, we do that with ldappc, because we 
consider that all user in grouper are present in our ldap, we have some 
script that make such checks. And this modification in ldappc is really 
usefull for us, we don't need to wait too much time for provisioning at 
demand and for the whole provisioning of memberships in ldap.

Else thanks for your help, but now I have an other question, which will be an 
other post ;-)

Thanks

Julien


Le 29/08/2011 18:24, Chris Hyzer a écrit :
> I don’t know off the top of my head, but we are interested in working with 
> you to figure it out at some point...  hopefully there is caching that 
> would help :).  Ideally you could run a profiler like yourkit to see what 
> the bottleneck is.  If that is not possible, you could describe in great 
> detail the size/distribution of your database, the environment and sizing 
> of hardwar, and what operations you are doing, and we could try to 
> reproduce it and use yourkit...
>
> Thanks,
> Chris
>
> -----Original Message-----
> From: Julien Gribonvald 
> [mailto:]
> Sent: Monday, August 29, 2011 12:20 PM
> To: Chris Hyzer
> Cc: 
> 
> Subject: Re: [grouper-users] Grouper 1.6.3 : Problem to desactivate 
> log generation in database
>
> Thanks for your response Chris.
>
> Our problem is that grouper is slower version after version for us, and we 
> think that some tweaks like the possibility of disabling insert of chanlog 
> in some case by a flag could a be good thing. Also my purpose to have two 
> level of subject object used by the api could be a good tweak too, mainly 
> for ldappc where it's not usefull in some context to get all information of 
> the user. Such options would be usefull for us, for example we deconnected 
> on ldappc the actions to make the request on ldap (user source for us) as 
> we are sure that users exists and so it does only ldap provisioning of 
> groups attributes. For example a such modification for us permitted to have 
> in less than 15 min the provisionning of all groups attributes (16 000) for 
> all users (200 000) with an average of 8 groups per user. If we didn't make 
> this modification the provisionning run during several hours before to be 
> ok.
>
> After for our provisionning we developped some code that use the grouper 
> api, this worked very well with grouper 1.4 and 1.5, but now it's slow with 
> grouper 1.6.3 (but nothing was modified on method that we use), do you have 
> any idea on our problem ?
>
> Thanks
>
> Julien
>
>
> Le 29/08/2011 17:43, Chris Hyzer a écrit :
>> Originally those flags did something, but then we went down the path of 
>> not being sure what the impact of turning it off would be, so there is no 
>> way of turning it off.  i.e. real time ldappcng will use the change log, 
>> and all notifications already do, etc...
>>
>> I put in a Jira to clean up those flags though:
>>
>> https://bugs.internet2.edu/jira/browse/GRP-638
>>
>> When resolving this issue, maybe we can make the flag work if we thought 
>> it was ok to not use the change log for some things...
>>
>> If there is more discussion Im happy to discuss :)
>>
>> Thanks,
>> Chris
>>
>> -----Original Message-----
>> From: 
>> 
>> [mailto:]
>>  On Behalf Of Julien 
>> Gribonvald
>> Sent: Monday, August 29, 2011 9:36 AM
>> Cc: 
>> 
>> Subject: [grouper-users] Grouper 1.6.3 : Problem to desactivate log 
>> generation in database
>>
>> Hi,
>>
>> I'm having problems to avoid that grouper generate logs in database for 
>> each operations done (in some case only, because we need logs by the UI 
>> only). I want to avoid this generation because we have massive 
>> modifications (16000 groups and around 200 000 users to add in some 
>> groups), and so it adds useless insert with these modifications. Maybe 
>> it's an other problem because with a good configuration of 
>> grouper.properties file in grouper 1.5.2 it's doesn't take so much time 
>> and now after migration on grouper 1.6.3 it's really slow, the 
>> provisioning time is multiplied at least by 2. I have done an analyze and 
>> an optimize on the database but nothing changed.
>>
>> I think the first optimisation would be to avoid log generation in 
>> database, I tried this configuration in grouper.properties :
>>
>> changeLog.enabled = false
>> groups.updateLastMembershipTime = false 
>> stems.updateLastMembershipTime = false
>>
>> But this configuration doesn't seem to works because insert are done in 
>> grouper_change_log_entry_temp and grouper_audit_entry tables. Do you know 
>> why this change nothing ?
>> Also we are using ldappcng and after tests changeLog.enabled to false 
>> doesn't change anything... But we modified a bit ldappcng to avoid that it 
>> gets user information in our ldap (the source of user) which was very 
>> expansive in time. As purpose, a good improvement could be done by 
>> avoiding request on user source which gets all user parameters because a 
>> lot of time we don't need user informations, only source identifier and id 
>> of the user are sufficient, for example a simple subject and a complete 
>> subject which expand simpleSubject would be great.
>>
>> Thanks for your help (and sorry for my English).
>>
>> Julien