grouper-users - [grouper-users] threaded loader jobs
Subject: Grouper Users - Open Discussion List
- From: Chris Hyzer <>
- To: "" <>
- Subject: [grouper-users] threaded loader jobs
- Date: Mon, 3 Nov 2014 14:49:43 +0000
- Accept-language: en-US
I added threads to loader jobs in two places: when processing a list of groups, each group will get its own thread, and in a group, each membership add or delete will get its own thread.
This seems to speed things up by a factor of 4.
When loading 200 groups with 10 members each, the old loader took 41.4 seconds.
With threads on memberships (wont see a big gain in this case since it is only 10 members), it took: 34.5 seconds.
With threads on groups, it took 15.5 seconds.
With threads on groups and memberships it took: 11.2 seconds.
This was on my dev PC with mysql. Maybe with a production database the gains will be more dramatic.
These four settings control the threading in the grouper-loader.properties:
# if should use threads in the loader for add/remove member
# number of threads to use for each group job (not shared among jobs)
# if should use threads in the loader for each group in a group of groups
# number of threads to use for each list of groups job (not shared among jobs)
This is sort of experimental, but I think we should default to on for 2.2.1 since it helps a lot and if it is a problem we can turn it off easily.
- [grouper-users] threaded loader jobs, Chris Hyzer, 11/03/2014
Archive powered by MHonArc 2.6.16.