Grouper Users - Open Discussion List

[Chris Hyzer <>]

We need to consider that when we look at performance of Grouper (hopefully at some point soon).  The API wasn’t really designed for batching, but its something we can look at.  Right now a membership add does a bunch of inserts into various tables, and commits that as a transaction.  I think the loader tries to do each membership change independently so that it can get as far as it can, though maybe it should have a transaction of batches and see if Hibernate can batch up the SQL with how the data access layer works.

 

Thanks,

Chris

 

From: Doppala, Karthik [mailto:]
Sent: Thursday, February 27, 2014 5:07 PM
To: Chris Hyzer
Cc:
Subject: RE: [grouper-users] Slow SQL Loader Job

 

Hi Chris,

Sorry about reopening this thread but had a related question regarding addMember method and how it’s being currently used. I noticed that the current logic process one records (adding membership) at a time; is there a particular reason for  not using batch insert or deletes?

 

Thanks,

Karthik

 

From: Chris Hyzer []
Sent: Thursday, January 30, 2014 11:52 AM
To: Doppala, Karthik; Bryan E. Wooten; David Langenberg
Cc:
Subject: RE: [grouper-users] Slow SQL Loader Job

 

I think a task after 2.2 is to look at the Grouper API performance for common tasks.  (e.g. addMember in this case)

 

If you are doing 1.8 million in a week, and no memberships exist, then that is 300ms per insert (right?)

 

One thing that will speed things up is to cache netIds in the Grouper DB.  There are obvious pros and cons in doing that, and we can discuss doing that at some point.

 

I could imaging breaking up “list of list” jobs into multiple threads which should make it go quicker… we could try J

 

Thanks,

Chris

 

From: Doppala, Karthik []
Sent: Thursday, January 30, 2014 2:38 PM
To: Bryan E. Wooten; Chris Hyzer; David Langenberg
Cc:
Subject: RE: [grouper-users] Slow SQL Loader Job

 

I wanted to get involved in this conversation as well since I am currently trying to analyze Grouper-Loader code. The last time we ran the loader it almost took a week to sync 1.8 million membership data from SQL database, we would like to bring it down to 5-8 hours which is more acceptable. We are currently using SQL Server 2008 , I would definitely try analyzing and indexing the db tables as Chirs suggested. Is there any additional documentation specific to Grouper-Loader’s architecture? Also, I would appreciate if you can share your experience with Loader’s performance.

 

Thanks,

Karthik

 

From: [] On Behalf Of Bryan E. Wooten
Sent: Thursday, January 30, 2014 10:16 AM
To: Chris Hyzer; David Langenberg
Cc:
Subject: RE: [grouper-users] Slow SQL Loader Job

 

Thanks,

 

I am currently rebuilding my LDAP indexes. It seems my LDAP server ran out of disk space and so the indexes were never built L

 

-Bryan

 

From: Chris Hyzer []
Sent: Thursday, January 30, 2014 11:15 AM
To: David Langenberg; Bryan E. Wooten
Cc:
Subject: RE: [grouper-users] Slow SQL Loader Job

 

Also, the first thing to do with Grouper slowness is to analyze your database tables…

 

This has an example for Oracle, though other databases are similar

 

https://spaces.internet2.edu/display/Grouper/Upgrade+notes+from+Grouper+v1.6+to+v1.7

 

Generate script:

select 'ANALYZE TABLE ' || table_name || ' COMPUTE STATISTICS;' as script from user_Tables where table_name like 'GROUPER%'

ANALYZE TABLE GROUPER_ATTRIBUTES COMPUTE STATISTICS;            
ANALYZE TABLE GROUPER_ATTRIBUTE_ASSIGN COMPUTE STATISTICS;      
ANALYZE TABLE GROUPER_ATTRIBUTE_ASSIGN_VALUE COMPUTE STATISTICS;
…

 

Then run that script

 

Thanks,

Chris

 

From: [] On Behalf Of David Langenberg
Sent: Thursday, January 30, 2014 10:38 AM
To: Bryan E. Wooten
Cc:
Subject: Re: [grouper-users] Slow SQL Loader Job

 

Hi Bryan,

 

The first place I'd start is looking at my subject source configuration and the LDAP server and ensure that: the queries that grouper is performing are searching against indexed attributes, the indexes are fully built-out, and they are being used in the search.  If not, tweak each end (subject config & ldap) until those are snappy.  

 

Dave

 

On Thu, Jan 30, 2014 at 8:24 AM, Bryan E. Wooten <> wrote:

So I have refreshed by LDAP subject source server and my test data base had also been refreshed.

 

At this point I obliterated my stem containing all the course enrollment loader groups.

 

I then restarted Grouper. The Grouper loader job (SQL_GROUP_LIST) has now be running for over 18 hours and has still not completed, it is continuing to add course groups but is not anywhere near completion.

 

How do I go about diagnosing this slowness?

 

Thanks,

 

Bryan

 

--
David Langenberg

Identity & Access Management

The University of Chicago