Grouper Users - Open Discussion List

[Siju Jacob <>]

Hi Chris,

     Thanks for the quick response. We do have plans to migrate to 2.4 in next 6 months but as of now we want to move 2.3 to production linux servers from solaris asap. We could live with this performance for initial loading for now.

 

Below is the result of the loader I ran, which took 2 days to complete. Although it says inserted 1 membership it actually loaded all 234364 memberships for the first time.

===> loader ran successfully, inserted 1 memberships, deleted 0 memberships, total membership count: 234364, unresolvable subjects: 1

 

Could you please explain briefly my questions on loader.groupThreadPoolSize and hibernate.c3p0.max_size settings in grouper 2.3

whenever you get sometime.

 

1)      If there is 10 loaders with more than 1000 groups and 200k members each, running at the same time going to spawn 20 threads in total or is it going to start 10*20=200 threads in total?

2)      Does each thread  uses one connection from the connection pool?

 

3)  If hibernate.c3p0.max_size = 100 and all 100 connections is going to be used by loaders

 

there won’t be any connection left for grouper UI?

 

4)      Does these settings behave differently in grouper 2.4?

 

Thanks,

Siju

 

From: Hyzer, Chris <>
Sent: Thursday, March 21, 2019 3:29 PM
To: Siju Jacob <>; Mailing List <>
Subject: RE: loader.groupThreadPoolSize and hibernate.c3p0.max_size

 

Im thinking turning these knobs is not your problem…

 

When it takes a few days, what do the grouper_loader_logs report as the number of inserts/updates/deletes?

 

I hate to say it, but im not sure threads will help… any chance you can upgrade to 2.4?  We just did some performance work on loaders…  If not, maybe we could make an option to have the 2.4 loader performance patch in 2.3 (default to false to not introduce risk)…

 

Thanks

Chris

 

From: <> On Behalf Of Siju Jacob
Sent: Thursday, March 21, 2019 1:44 PM
To: Mailing List <>
Subject: [grouper-users] loader.groupThreadPoolSize and hibernate.c3p0.max_size

 

Hi Team,

     We are using grouper 2.3 with latest updates. We are trying to scale our grouper to enterprise level. We are facing some performance issue due to my limited knowledge on grouper.

    While trying to run the loaders with below configuration its taking couple of days for a loader with 3000 groups and 300k members to load.

 

    We are running on JDK 1.7 , Linux system with X86 Large - 4vCPU 16GB Mem and Centos, Oracle 11g database.

 

    I am having little trouble in understanding below configuration (groupThreadPoolSize) and the connection pool size (hibernate.c3p0.max_size = 100).

 

    Based on the below configuration, if there is 10 loaders with more than 1000 groups and 200k members each, running at the same time going to spawn 20 threads in total or is it going to start 10*20=200 threads in total?

 

     I believe each thread is going to use one connection from the connection pool. Please correct me if I am wrong.

 

     If hibernate.c3p0.max_size = 100 and all 100 connections is going to be used by loaders there won’t be any connection left for grouper UI?

 

     Please let me know what will be the ideal setting for the below configurations based on your experience for the enterprise level grouper with above mentioned load.

 

Any advice or guidance will be of great help and would be greatly appreciated..!

 

 

-à grouper-loader.base.properties

# number of threads in the loader threadpool.  Only this number of jobs can run at once

# jobs which are on deck will block, or will fail if the blocking timeout occurs

# a job is running if it is loading (not just scheduled)

loader.thread.pool.size=10

 

# if should use threads in the loader for add/remove member

loader.use.membershipThreads=true

 

# number of threads to use for each group job (not shared among jobs)

loader.membershipThreadPoolSize=10

 

# if should use threads in the loader for each group in a group of groups

loader.use.groupThreads=true

 

# number of threads to use for each list of groups job (not shared among jobs)

loader.groupThreadPoolSize=20

 

 

-àgrouper.hibernate.base.properties

 

hibernate.c3p0.max_size = 100

 

Thanks,

Siju