Grouper Users - Open Discussion List

[Wayne Hall <>]

Chris,

Thank you.

The grouper_loader_logs provided the data needed to 'think outside the box' of our setup.  I ended 
up creating a local file on grouper with these statements for each of our grouper-loader groups 
(edited for our instance):

# Next line begins file
grouperSession = GrouperSession.startRootSession();

# Next two lines added for each grouper-loader group
loaderGroup = GroupFinder.findByName(grouperSession, "stem:group");
loaderRunOneJob(loaderGroup);

I can ssh into grouper, execute the gsh command, point it to my loader file and it works great!  
Now the process takes 3:21 minutes to complete.

Previously we had a remote system logged into grouper running each of those lines, one task at a 
time.  The local file is a much cleaner option.

Thanks,

Wayne

On 11/17/2015 01:39 PM, Chris Hyzer wrote:
> The grouper_loader_log table shows some information about time to get data and time to manipulate 
> groups.  You could also use p6spy to get information about how long queries take.  In 
> log4j.properties you can set DEBUG level to edu.internet2.middleware.grouper.app.loader package 
> and see how it goes
>
> Thanks,
>
> Chris
>
> *From:*Wayne Hall 
> [mailto:]
> *Sent:* Tuesday, November 17, 2015 12:38 PM
> *To:* Chris Hyzer; 
>
> *Subject:* Re: [grouper-core] grouper loader process
>
> Hi,
>
> First, sorry I sent this to the core list, didn't realize I did until Emily's note.  When 
> composing the message, I noticed grouper auto populated in my TO field, I selected it assuming I 
> was sending to the users group.
>
> Grouper version is 2.2.2 (java7) and running a query on the sql server database seems to run fine 
> (Elapsed time: 00:00:01.000 for 70642 rows).  Running a query from grouper on the mariadb seems 
> to run fine (70673 rows in set (0.10 sec)).
>
> Is there a way to run ./gsh -loader verbose so I can watch that process from grouper, or other 
> methods to check performance, or places I should check?
>
> Thanks,
> Wayne
>
> On 11/16/2015 06:22 PM, Chris Hyzer wrote:
>
>     What grouper version and patch level are you at?  When you run the 
> queries against the
>     database do they take a long time?
>
>     
> *From:*
>  
> <mailto:>
>     
> [mailto:]
>  *On Behalf Of *Wayne Hall
>     *Sent:* Monday, November 16, 2015 6:25 PM
>     *To:* 
>
>  
> <mailto:>
>     *Subject:* [grouper-core] grouper loader process
>
>     Hi,
>
>     Our grouper loader task takes about 19 minutes to complete. This seems 
> longer than I would
>     expect, so wondering if this should be considered "normal" performance?
>
>     We have members ranging from 70 to 33,900 across 72 groups. The grouper 
> database resides on a
>     MariaDB instance querying (72) views on a sql server.  I have added to the
>     grouper_loader.properties files the following:
>
>     # if should use threads in the loader for add/remove member
>     loader.use.membershipThreads=true
>
>     # number of threads to use for each group job (not shared among jobs)
>     loader.membershipThreadPoolSize=10
>
>     # if should use threads in the loader for each group in a group of groups
>     loader.use.groupThreads=true
>
>     # number of threads to use for each list of groups job (not shared among 
> jobs)
>     loader.groupThreadPoolSize=20
>
>     #################################
>     ## Performance enhancements
>     #################################
>
>     # if you want to bulk retrieve subjects to add/remove
>     loader.bulkLookupSubjects = true
>
>
>     Does this seem right, and if not, what other performance enhancements 
> should I consider?
>
>     Thanks,
>     Wayne
>
>
>     -- 
>
>     Wayne 
>
>  
> <mailto:>
>
>     Middleware Services                    (970) 491-4763
>
>     Academic Computing and Networking Services
>
>     Colorado State University        Fort Collins CO 80523