Grouper Users - Open Discussion List

[Chris Hyzer <>]

I added threads to loader jobs in two places:  when processing a list of groups, each group will get its own thread, and in a group, each membership add or delete will get its own thread.

 

https://github.com/Internet2/grouper/commit/1401006b5abcb07e1dcd82456385381a80ec4453

 

https://bugs.internet2.edu/jira/browse/GRP-1067

 

This seems to speed things up by a factor of 4.

 

When loading 200 groups with 10 members each, the old loader took 41.4 seconds.

 

With threads on memberships (wont see a big gain in this case since it is only 10 members), it took: 34.5 seconds.

 

With threads on groups, it took 15.5 seconds.

 

With threads on groups and memberships it took: 11.2 seconds.

 

This was on my dev PC with mysql.  Maybe with a production database the gains will be more dramatic.

 

These four settings control the threading in the grouper-loader.properties:

 

# if should use threads in the loader for add/remove member

loader.use.membershipThreads=true

 

# number of threads to use for each group job (not shared among jobs)

loader.membershipThreadPoolSize=10

 

# if should use threads in the loader for each group in a group of groups

loader.use.groupThreads=true

 

# number of threads to use for each list of groups job (not shared among jobs)

loader.groupThreadPoolSize=20

 

 

This is sort of experimental, but I think we should default to on for 2.2.1 since it helps a lot and if it is a problem we can turn it off easily.

 

Thanks,

Chris