The grouper_loader_log table shows some information about time to get data and time to manipulate groups. You could also use p6spy to get information about
how long queries take. In log4j.properties you can set DEBUG level to edu.internet2.middleware.grouper.app.loader package and see how it goes
Thanks,
Chris
From: Wayne Hall [mailto:]
Sent: Tuesday, November 17, 2015 12:38 PM
To: Chris Hyzer;
Subject: Re: [grouper-core] grouper loader process
Hi,
First, sorry I sent this to the core list, didn't realize I did until Emily's note. When composing the message, I noticed grouper auto populated in my TO field, I selected it assuming I was sending to the users group.
Grouper version is 2.2.2 (java7) and running a query on the sql server database seems to run fine (Elapsed time: 00:00:01.000 for 70642 rows). Running a query from grouper on the mariadb seems to run fine (70673 rows in set (0.10 sec)).
Is there a way to run ./gsh -loader verbose so I can watch that process from grouper, or other methods to check performance, or places I should check?
Thanks,
Wayne
On 11/16/2015 06:22 PM, Chris Hyzer wrote:
What grouper version and patch level are you at? When you run the queries against the database do they take a long time?
From:
[]
On Behalf Of Wayne Hall
Sent: Monday, November 16, 2015 6:25 PM
To:
Subject: [grouper-core] grouper loader process
Hi,
Our grouper loader task takes about 19 minutes to complete. This seems longer than I would expect, so wondering if this should be considered "normal" performance?
We have members ranging from 70 to 33,900 across 72 groups. The grouper database resides on a MariaDB instance querying (72) views on a sql server. I have added to the grouper_loader.properties files the following:
# if should use threads in the loader for add/remove member
loader.use.membershipThreads=true
# number of threads to use for each group job (not shared among jobs)
loader.membershipThreadPoolSize=10
# if should use threads in the loader for each group in a group of groups
loader.use.groupThreads=true
# number of threads to use for each list of groups job (not shared among jobs)
loader.groupThreadPoolSize=20
#################################
## Performance enhancements
#################################
# if you want to bulk retrieve subjects to add/remove
loader.bulkLookupSubjects = true
Does this seem right, and if not, what other performance enhancements should I consider?
Thanks,
Wayne
--
Wayne Hall
Middleware Services (970) 491-4763
Academic Computing and Networking Services
Colorado State University Fort Collins CO 80523
