Grouper Developers Forum

[Chris Hyzer <>]

Ok, im done working on this (its in the 2.0 branch, slated for whenever 2.0.2 comes out), and I have it in Penn’s test env and it is a lot better.  i.e. in prod (2.0.1),  certain searches can make the UI run out of memory:

 

https://bugs.internet2.edu/jira/browse/GRP-687

 

There is a new source method called searchPage where if the source has a page limit, then it will only return the first X+1 records, and the result is a bean which has the subjects and a boolean which says if there are more records:

    <!-- on a findPage() this is the most results returned -->
    <init-param>
      <param-name>maxPageSize</param-name>
      <param-value>100</param-value>
    </init-param>


If there is no maxPageSize then it will work without paging like a normal search. All UI/WS calls to find subjects will use the paging method. Note, there is no hibernate in the subject api, so the query is edited in java. This works in hsql, oracle, mysql, postgres (note: sql server paging seems too complicated). If your query is too complex and the paging causes errors, then you can disable this behavior with this setting:

    <init-param>
      <param-name>changeSearchQueryForMaxResults</param-name>
      <param-value>false</param-value>
    </init-param>


Also, groups returned as subjects have a query for each group to get attributes, this is now done in 1 or a small number of queries.

Group subjects should probably not have editedBy and createdBy. There is a switch to turn this off. This also causes a query per group returned.

# if the creator and last updater should be group subject attributes (you get
# a performance gain if you set to false, but if true you can see subject id from UI in 2.0
subjects.group.useCreatorAndModifierAsSubjectAttributes = true


That currently defaults to true so there are no changes.

 

Note, there is more work to do at some point:

 

https://bugs.internet2.edu/jira/browse/GRP-686

 

Thanks,

Chris

 

From: Chris Hyzer
Sent: Sunday, October 30, 2011 2:41 AM
To: Chris Hyzer; 'Grouper Dev'
Subject: RE: grouper 2.0 subject issue

 

I implemented threads and a threadpool, and it didn’t speed things up, I think because I have a sql search on the same db as grouper.  Well, I think we still need the paging part, maybe the threading should default to off…  I still need to do some checks about the paging…  Shilen, do you mind looking the code in the v2.0 branch (api, subject, ui, ws)?

 

These control the paging in grouper.properties

 

# if finding across multiple threadable sources, use threads to do the work faster

subjects.allPage.useThreadForkJoin = true

 

# if finding across multiple threadable sources, use threads to do the work faster

subjects.idOrIdentifier.useThreadForkJoin = true

 

You can set logging to debug here:

 

log4j.logger.edu.internet2.middleware.grouper.subj.SourcesXmlResolver = DEBUG

 

Then you can run this benchmark (mine has 80k groups, and 500k subjects in pennperson and a large search string):

 

SubjectFinderBenchmark

 

You will see output like this for threading:

 

2011-10-30 02:27:21,480: [pool-1-thread-5] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: g:isa, 'bah', time in millis: 0

2011-10-30 02:27:21,487: [pool-1-thread-2] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: servPrinc, 'bah', time in millis: 9

2011-10-30 02:27:21,488: [pool-1-thread-4] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: grouperExternal, 'bah', time in millis: 9

2011-10-30 02:27:22,127: [pool-1-thread-1] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: g:gsa, 'bah', time in millis: 647

2011-10-30 02:27:22,255: [pool-1-thread-3] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: pennperson, 'bah', time in millis: 775

2011-10-30 02:27:22,255: [main] DEBUG SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using threads: true, time in millis: 789

 

Or like this without threading:

 

2011-10-30 02:39:28,592: [main] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: servPrinc, 'bah', time in millis: 4

2011-10-30 02:39:28,802: [main] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: g:gsa, 'bah', time in millis: 209

2011-10-30 02:39:29,442: [main] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: pennperson, 'bah', time in millis: 640

2011-10-30 02:39:29,448: [main] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: grouperExternal, 'bah', time in millis: 5

2011-10-30 02:39:29,448: [main] DEBUG SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source: g:isa, 'bah', time in millis: 0

2011-10-30 02:39:29,448: [main] DEBUG SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using threads: false, time in millis: 860

 

There are also tests in TestSubjectFinder.testSearchPageMax()

 

Well, just thought of something… if there are lots of results, I wonder if threading helps, my benchmark now wont return a lot of results… let me try.

 

Thanks,

Chris

 

From: Chris Hyzer
Sent: Friday, October 28, 2011 8:27 AM
To: Grouper Dev
Subject: grouper 2.0 subject issue

 

When I do a findall (e.g. UI search) for something that returns 1000+ Subjects (which I have as my maxResults) in my sources.xml, it gives an exception.  It is supposed to throw a SubjectTooManyResults exception which is caught in UI and displayed on the screen.  I could fix this immediate problem, but I kind of think that there should be another method in the source called searchPage(searchString) which is just like search, but if the source has a

 

    <init-param>

      <param-name>maxPageSize</param-name>

      <param-value>100</param-value>

    </init-param>

 

Then on a searchPage(), only 100 results are returned, in addition it will say if there are more or not.  The UI could just show the 100, and the user would narrow their search.  This seems better than showing nothing and having them narrow their search.  One reason is if the search string matches a subjectId or subjectIdentifier, it goes to the top.  If the subjectIdentifier is “fred”, and that returns more than 100 results, but the netId “fred” is at the top of the list, the user is happy…  this is for 2.0.2

 

I discussed this with Shilen and his reaction was positive J

 

Thoughts?

 

Thanks,

Chris