Grouper Users - Open Discussion List

[Richard Frovarp <>]

Okay. It's an initial load. Several of the legacy groups are going to basically have our entire population in them. So that would cause a very large change to be processed. Thanks.

Richard

- Yes, I know that in Grouper I shouldn't be duplicating our user base across all of the application groups and instead should be using group math. Doing this now to figure out how to make the loader work with data I have available and understand.

---

From: Shilen Patel <>
Sent: Monday, January 7, 2019 7:56:42 PM
To: Hyzer, Chris; Richard Frovarp
Cc:
Subject: Re: [grouper-users] Re: Performance tuning guide?

 

Right if it’s a single group list, then it should be 20 total (not 20 per group in the group list).   However, there’s also loader.membershipThreadPoolSize.

 

# number of threads to use for each group job (not shared among jobs)

loader.membershipThreadPoolSize=10

 

So I think each of those 20 threads might use up to 10 threads (by default) if you have a lot of membership changes.

 

Thanks!

 

- Shilen

 

From: <> on behalf of "Hyzer, Chris" <>
Date: Monday, January 7, 2019 at 7:10 PM
To: Richard Frovarp <>, "" <>
Subject: [grouper-users] Re: Performance tuning guide?

 

I dont think thats how it works but Shilen can confirm

---

From: <> on behalf of Richard Frovarp <>
Sent: Monday, January 7, 2019 5:58:27 PM
To:
Subject: [grouper-users] Re: Performance tuning guide?

 

Possibly increasing max connections in MariaDB, but if it is going for 20k threads, that's a little crazy.

 

Yes, I'm doing a group list. It's the other setting that makes me think it is multiplying out:

 

 

# if should use threads in the loader for each group in a group of groups
loader.use.groupThreads=true

 

So since that is true, I think it is using the groupThreadPoolSize for each department for instance. Which even for departments, 90 departments * 20 groupThreadPoolSize equals 1,800 connections. I'm guessing that I'm going to bottleneck on something else even if I up MariaDB to 2k connections.

 

 

On 1/7/19 4:52 PM, Hyzer, Chris wrote:

You can decrease the thread pool size if you want.  But maybe you should also increase the max_connections var in mariadb...

 

The docs in the config say its one pool for each list of groups job, is that the type of job you are running?

 

# number of threads to use for each list of groups job (not shared among jobs)

# {valueType: "integer", required: true}

loader.groupThreadPoolSize=20

 

 

Thanks

Chris

 

-----Original Message-----
From: On Behalf Of Richard Frovarp
Sent: Monday, January 07, 2019 5:12 PM
To:
Subject: [grouper-users] Performance tuning guide?

 

Is there a performance tuning guide somewhere. I know to kick up the

memory offered to Tomcat. That helped out a good amount for the UI.

 

In particular at the moment I'm having trouble with the loader. I'm

trying to load in our legacy groups for lack of a better term, and it's

firing up way too many threads. We have around 1k legacy groups. I can

see from jconsole that it's easily starting over 200 pool threads. Then

they start to time out, and MariaDB starts to complain about too many

connections. I see in the loader config file reference to

loader.use.groupThreads, and loader.groupThreadPoolSize, but not

entirely sure how they work.

 

loader.use.groupThreads means that it is trying to use threads for each

of my 1k groups? And loader.groupThreadPoolSize means it is trying to

use 20 threads for each group? So is it trying to start 20k threads?

That would seem to indicate that the right option would be to set

loader.use.groupThreads to false. Or is there something else? I've hit

the same problem trying to load departments, and we're a mid-sized school.

 

Is there anything else to performance tune the system?

 

Thanks,

 

Richard