Skip to Content.
Sympa Menu

grouper-users - RE: [grouper-users] Slow SQL Loader Job

Subject: Grouper Users - Open Discussion List

List archive

RE: [grouper-users] Slow SQL Loader Job

Chronological Thread 
  • From: Chris Hyzer <>
  • To: "Doppala, Karthik" <>, "Bryan E. Wooten" <>, David Langenberg <>
  • Cc: "" <>
  • Subject: RE: [grouper-users] Slow SQL Loader Job
  • Date: Thu, 30 Jan 2014 19:51:41 +0000
  • Accept-language: en-US

I think a task after 2.2 is to look at the Grouper API performance for common tasks.  (e.g. addMember in this case)


If you are doing 1.8 million in a week, and no memberships exist, then that is 300ms per insert (right?)


One thing that will speed things up is to cache netIds in the Grouper DB.  There are obvious pros and cons in doing that, and we can discuss doing that at some point.


I could imaging breaking up “list of list” jobs into multiple threads which should make it go quicker… we could try J





From: Doppala, Karthik [mailto:]
Sent: Thursday, January 30, 2014 2:38 PM
To: Bryan E. Wooten; Chris Hyzer; David Langenberg
Subject: RE: [grouper-users] Slow SQL Loader Job


I wanted to get involved in this conversation as well since I am currently trying to analyze Grouper-Loader code. The last time we ran the loader it almost took a week to sync 1.8 million membership data from SQL database, we would like to bring it down to 5-8 hours which is more acceptable. We are currently using SQL Server 2008 , I would definitely try analyzing and indexing the db tables as Chirs suggested. Is there any additional documentation specific to Grouper-Loader’s architecture? Also, I would appreciate if you can share your experience with Loader’s performance.





From: [] On Behalf Of Bryan E. Wooten
Sent: Thursday, January 30, 2014 10:16 AM
To: Chris Hyzer; David Langenberg
Subject: RE: [grouper-users] Slow SQL Loader Job




I am currently rebuilding my LDAP indexes. It seems my LDAP server ran out of disk space and so the indexes were never built L




From: Chris Hyzer []
Sent: Thursday, January 30, 2014 11:15 AM
To: David Langenberg; Bryan E. Wooten
Subject: RE: [grouper-users] Slow SQL Loader Job


Also, the first thing to do with Grouper slowness is to analyze your database tables…


This has an example for Oracle, though other databases are similar


Generate script:

select 'ANALYZE TABLE ' || table_name || ' COMPUTE STATISTICS;' as script from user_Tables where table_name like 'GROUPER%'



Then run that script





From: [] On Behalf Of David Langenberg
Sent: Thursday, January 30, 2014 10:38 AM
To: Bryan E. Wooten
Subject: Re: [grouper-users] Slow SQL Loader Job


Hi Bryan,


The first place I'd start is looking at my subject source configuration and the LDAP server and ensure that: the queries that grouper is performing are searching against indexed attributes, the indexes are fully built-out, and they are being used in the search.  If not, tweak each end (subject config & ldap) until those are snappy.  




On Thu, Jan 30, 2014 at 8:24 AM, Bryan E. Wooten <> wrote:

So I have refreshed by LDAP subject source server and my test data base had also been refreshed.


At this point I obliterated my stem containing all the course enrollment loader groups.


I then restarted Grouper. The Grouper loader job (SQL_GROUP_LIST) has now be running for over 18 hours and has still not completed, it is continuing to add course groups but is not anywhere near completion.


How do I go about diagnosing this slowness?






David Langenberg

Identity & Access Management

The University of Chicago

Archive powered by MHonArc 2.6.16.

Top of Page