Grouper Users - Open Discussion List

[Julien Gribonvald <>]

Thanks for your response,
I didn't run an analyse, I figured that we made a dump and re-insert it, 
but it was before the migration (as we moved from mysql 5.5 to mariadb 
10.1 on debian 9)  so I will try that and test if it improve group/stem 
creation. Also I have to analyze with visualVM which part of the code 
take too long time during the process. But the thing that is clear is 
that nothing change in my GrouperDAO between version 1.6.3 and 2.3.0 of 
grouper (only 2 method name on the api call), so for me the main problem 
isn't here.

For the multi threads part I fear that will be useless as we will have 
some locks on tables and that we will need to manage some "already 
creation done".

In all case I should analyze the code that take too long time.

Thanks.
- Julien


Le 16/03/2018 à 20:38, Shilen Patel a écrit :
> After you upgraded to 2.3, did you analyze your tables?
>
> https://spaces.internet2.edu/pages/viewpage.action?pageId=14517958#APIBuilding&Configuration-AnalyzingTablestoImproveQueryPerformance
>
> What database do you use?  How long does it take to run the create method in 
> your GrouperDAO once on average?   Also, just wondering, have you tried 
> running multiple threads to allow creations to happen in parallel?
>
> Thanks!
>
> - Shilen
>
> On 3/16/18, 5:50 AM, 
> "
>  on behalf of Julien Gribonvald" 
> <
>  on behalf of 
> >
>  wrote:
>
>      I'm using directly the grouper-api, and it uses api generic methods.
>      The project is here : 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_GIP-2DRECIA_esco-2Dgrouper-2Dmodules_&d=DwIDaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=sWqutME58phurE0oO57Icg&m=ZtuV_vgGx5WjIRe39F0QbEfwJYcxAMFvJZbn2_tzxpg&s=Nb0iZ22adxZreTvg-8Rvy2lXQ07_IuONWAJo-sEDBLo&e=
>      And the main class used is here :
>      
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_GIP-2DRECIA_esco-2Dgrouper-2Dmodules_blob_master_esco-2Dgrouper-2Dsarapis-2DalimManager_src_main_java_org_esco_grouper_dao_GrouperDAO.java&d=DwIDaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=sWqutME58phurE0oO57Icg&m=ZtuV_vgGx5WjIRe39F0QbEfwJYcxAMFvJZbn2_tzxpg&s=SDKio10_-xuExwLipGPXqd9ch-lWEwC5eUPNzr4mTLo&e=
>      
>      This project provide a template system to create folder and groups with
>      permission and to provision membership depending on user's informations
>      provided.
>      
>      Thanks,
>      Julien
>      
>      Le 15/03/2018 à 19:12, Hyzer, Chris a écrit :
>      > Are you using GSH or the loader?
>      >
>      > Can you send a portion of a script that takes a while and we can take 
> a look?
>      >
>      > Manipulating the database directly would probably result in 
> corruption...
>      >
>      > Thanks
>      > Chris
>      >
>      > -----Original Message-----
>      > From: 
>
>  
> [mailto:]
>  On Behalf Of Julien Gribonvald
>      > Sent: Thursday, March 15, 2018 2:07 PM
>      > To: 
>
>      > Subject: [grouper-users] Optimized group provisionning in grouper 
> database
>      >
>      > Hi,
>      >
>      > I would like to know if there are somes optimized process to be able to
>      > manage groups and folders, permissions on theses groups and the
>      > membership provisionning from a batch ?
>      >
>      > We just moved in production from grouper 1.6.3 to grouper 2.3.0 and I
>      > used for a long time the grouper api for this process, but with the
>      > version 2.3 when you have to manager around 40K groups by 
> automatization
>      > it takes really too long time (the main time is lost on groups and
>      > folders creation with privilegies) ! Hopefully this kind of management
>      > (such number of groups to update) appear only few times during the 
> year,
>      > but we would need something really more fast for such process.
>      >
>      > I watched on other grouper tools, but all are using the same "slow"
>      > grouper-api methods for managing groups, after my problems is that i
>      > don't know how I can create groups and folders more directly into the
>      > grouper dabatase, so for me it's really dangerous to do that, or maybe 
> ?
>      >
>      > So I would like to know if there could be more optimization, like
>      > disabling ldap request on subject api (has we are provisionning ldap
>      > user's account at the same time), and other stuff that could permit to
>      > manage a such number of groups in few/some hours instead of several 
> days ?
>      >
>      >
>      > Thanks for your help.
>      >
>      
>      --
>      Julien Gribonvald
>      

--
Julien Gribonvald