Grouper Developers Forum

[Chris Hyzer <>]

I implemented threads and a threadpool, and it didn’t speed things up, I think because I have a sql search on the same db as grouper.  Well, I think we still need the paging part, maybe the threading should default to off…  I still need to do some checks about the paging…  Shilen, do you mind looking the code in the v2.0 branch (api, subject, ui, ws)?

 

These control the paging in grouper.properties

 

# if finding across multiple threadable sources, use threads to do the work faster

subjects.allPage.useThreadForkJoin = true

 

# if finding across multiple threadable sources, use threads to do the work faster

subjects.idOrIdentifier.useThreadForkJoin = true

 

You can set logging to debug here:

 

log4j.logger.edu.internet2.middleware.grouper.subj.SourcesXmlResolver = DEBUG

 

Then you can run this benchmark (mine has 80k groups, and 500k subjects in pennperson and a large search string):

 

SubjectFinderBenchmark

 

You will see output like this for threading:

 

2011-10-30 02:27:21,480: [pool-1-thread-5] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: g:isa, 'bah', time in millis: 0

2011-10-30 02:27:21,487: [pool-1-thread-2] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: servPrinc, 'bah', time in millis: 9

2011-10-30 02:27:21,488: [pool-1-thread-4] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: grouperExternal, 'bah', time in millis: 9

2011-10-30 02:27:22,127: [pool-1-thread-1] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: g:gsa, 'bah', time in millis: 647

2011-10-30 02:27:22,255: [pool-1-thread-3] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: pennperson, 'bah', time in millis: 775

2011-10-30 02:27:22,255: [main] DEBUG SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using threads: true, time in millis: 789

 

Or like this without threading:

 

2011-10-30 02:39:28,592: [main] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: servPrinc, 'bah', time in millis: 4

2011-10-30 02:39:28,802: [main] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: g:gsa, 'bah', time in millis: 209

2011-10-30 02:39:29,442: [main] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: pennperson, 'bah', time in millis: 640

2011-10-30 02:39:29,448: [main] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: grouperExternal, 'bah', time in millis: 5

2011-10-30 02:39:29,448: [main] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: g:isa, 'bah', time in millis: 0

2011-10-30 02:39:29,448: [main] DEBUG SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using threads: false, time in millis: 860

 

There are also tests in TestSubjectFinder.testSearchPageMax()

 

Well, just thought of something… if there are lots of results, I wonder if threading helps, my benchmark now wont return a lot of results… let me try.

 

Thanks,

Chris

 

From: Chris Hyzer
Sent: Friday, October 28, 2011 8:27 AM
To: Grouper Dev
Subject: grouper 2.0 subject issue

 

When I do a findall (e.g. UI search) for something that returns 1000+ Subjects (which I have as my maxResults) in my sources.xml, it gives an exception.  It is supposed to throw a SubjectTooManyResults exception which is caught in UI and displayed on the screen.  I could fix this immediate problem, but I kind of think that there should be another method in the source called searchPage(searchString) which is just like search, but if the source has a

 

    <init-param>

      <param-name>maxPageSize</param-name>

      <param-value>100</param-value>

    </init-param>

 

Then on a searchPage(), only 100 results are returned, in addition it will say if there are more or not.  The UI could just show the 100, and the user would narrow their search.  This seems better than showing nothing and having them narrow their search.  One reason is if the search string matches a subjectId or subjectIdentifier, it goes to the top.  If the subjectIdentifier is “fred”, and that returns more than 100 results, but the netId “fred” is at the top of the list, the user is happy…  this is for 2.0.2

 

I discussed this with Shilen and his reaction was positive J

 

Thoughts?

 

Thanks,

Chris