Subject: Grouper Users - Open Discussion List
- From: Chris Hyzer <>
- To: "Doppala, Karthik" <>
- Cc: "" <>
- Subject: RE: [grouper-users] Slow SQL Loader Job
- Date: Thu, 27 Feb 2014 23:11:35 +0000
- Accept-language: en-US
We need to consider that when we look at performance of Grouper (hopefully at some point soon). The API wasn’t really designed for batching, but its something we can look at. Right now a membership add does a bunch of inserts into various tables, and commits that as a transaction. I think the loader tries to do each membership change independently so that it can get as far as it can, though maybe it should have a transaction of batches and see if Hibernate can batch up the SQL with how the data access layer works.
Sorry about reopening this thread but had a related question regarding addMember method and how it’s being currently used. I noticed that the current logic process one records (adding membership) at a time; is there a particular reason for not using batch insert or deletes?
I think a task after 2.2 is to look at the Grouper API performance for common tasks. (e.g. addMember in this case)
If you are doing 1.8 million in a week, and no memberships exist, then that is 300ms per insert (right?)
One thing that will speed things up is to cache netIds in the Grouper DB. There are obvious pros and cons in doing that, and we can discuss doing that at some point.
I could imaging breaking up “list of list” jobs into multiple threads which should make it go quicker… we could try J
I wanted to get involved in this conversation as well since I am currently trying to analyze Grouper-Loader code. The last time we ran the loader it almost took a week to sync 1.8 million membership data from SQL database, we would like to bring it down to 5-8 hours which is more acceptable. We are currently using SQL Server 2008 , I would definitely try analyzing and indexing the db tables as Chirs suggested. Is there any additional documentation specific to Grouper-Loader’s architecture? Also, I would appreciate if you can share your experience with Loader’s performance.
I am currently rebuilding my LDAP indexes. It seems my LDAP server ran out of disk space and so the indexes were never built L
Also, the first thing to do with Grouper slowness is to analyze your database tables…
This has an example for Oracle, though other databases are similar
select 'ANALYZE TABLE ' || table_name || ' COMPUTE STATISTICS;' as script from user_Tables where table_name like 'GROUPER%'
ANALYZE TABLE GROUPER_ATTRIBUTES COMPUTE STATISTICS;
Then run that script
The first place I'd start is looking at my subject source configuration and the LDAP server and ensure that: the queries that grouper is performing are searching against indexed attributes, the indexes are fully built-out, and they are being used in the search. If not, tweak each end (subject config & ldap) until those are snappy.
On Thu, Jan 30, 2014 at 8:24 AM, Bryan E. Wooten <> wrote:
So I have refreshed by LDAP subject source server and my test data base had also been refreshed.
At this point I obliterated my stem containing all the course enrollment loader groups.
I then restarted Grouper. The Grouper loader job (SQL_GROUP_LIST) has now be running for over 18 hours and has still not completed, it is continuing to add course groups but is not anywhere near completion.
How do I go about diagnosing this slowness?
Identity & Access Management
The University of Chicago
- RE: [grouper-users] Slow SQL Loader Job, Doppala, Karthik, 02/27/2014
- RE: [grouper-users] Slow SQL Loader Job, Chris Hyzer, 02/27/2014
Archive powered by MHonArc 2.6.16.