Grouper Users - Open Discussion List

[Chris Hyzer <>]

Shilen always says to analyze your tables…  try this in the grouper DB to all the grouper tables, and I guess in your source.  Is it indexed for ID and IDENTIFIER?  I don’t think it should be slower if a different DB than the Grouper one, unless it is on a different continent or something.

 

The loader will commit as it goes, so it is partially done.  See how many users are in the group:

 

select count(*) from grouper_memberships_lw_v where group_name = 'test:testGroup' and list_name = 'members'

 

Start up that job again, and as it runs, check the progress with the count query.  The first run is always slow, since it is adding each membership, and subsequent runs will be a lot faster since it only has the diffs to do, e.g. a few dozen or hundred memberships or something

 

Thanks,

Chris

 

 

From: [mailto:] On Behalf Of Jon Gorrono
Sent: Wednesday, October 10, 2012 5:47 PM
To:
Subject: [grouper-users] loader performance

 

 

I am still down in the shallow end here :)

 

I am a little surprised at the time it is taking to load a group with the loader

 

I've created a simple view with subject_id and subject_source_id and defined the group in the ui and created a loader job in the group attributes to select users from the view for the group

 

There are about 130k users and in this case they are all being shoved into one group

 

The source for the subjects api lookups is a different view on the same remote machine as the view used by the loader to populate the group and is using the C3p0 jdbc connection provider

 

The network is not likely the bottleneck, and the machines, both 'development-quality', are not top-notch but their mid-range performance is usually adequate for sane debug cycles etc.

 

The loader had been 40 minutes and had not yet finished when I had it was stopped (abruptly, heh) for scheduled patching.

 

I am guessing that the struggle it is having is with the remote source for the subjects...

 

So I guess my question is... is it really practical to have a remote subject source? Given the answer is 'yes' then there would be some better alternate questions, but I am not sure what good ones might be right now.

 

Any comments, questions, suggestions are welcome.

 

--
Jon Gorrono
PGP Key: 0x5434509D - http{pgp.mit.edu:11371/pks/lookup?search=0x5434509D&op=index}
http{middleware.ucdavis.edu}