Grouper Developers Forum

[Chris Hyzer <>]

Argh... we are still getting bad performance on the subject picker we use 
from grouper on our eforms (search for someone in the employee group by name 
in a combobox).

I think the reason is we haven't migrated to use the find subject in group 
method.  However, I noticed that even with that method, it will find the 
relevant subjects, but I don't think it does the group security in one query. 
 So I will add that.  Also, once it finds the subjects, to display them on 
the screen, it needs to resolve them, which is N queries (100?  200?)  We 
have a batch method on the subject API Source interface, but it is not 
implemented.  I will implement it in the JDBC and JDBC2 source adapters so 
they can retrieve 100 subjects in one query.  Do we want to do that in the 
ldap and vt-ldap adapters?  Is getting 100 subjects with 100 jndi queries 
much slower than one filter which gets 100 (or some batched number) at once?

Thanks,
Chris

-----Original Message-----
From: Chris Hyzer 
Sent: Wednesday, November 02, 2011 2:25 PM
To: 'Tom Zeller'
Cc: Grouper Dev
Subject: RE: [grouper-dev] RE: grouper 2.0 subject issue

Yes, if you can test the jndi and ldap source adapter, I put the limit in 
there too for paging, though as people will tell me, there is (typically?) a 
server-side limit anyways so it might not be a big deal...

Basically to test it, set the paging to a small number, and run it...  see 
TestSubjectFinder.testSearchPageMax() in the 2.0 branch, something like this:

    <!-- on a findPage() this is the most results returned --> 
    <init-param>
      <param-name>maxPageSize</param-name>
      <param-value>2</param-value>
    </init-param>

(from GSH?)

        searchPageResult = SubjectFinder.findPage("searchString", "sourceId");
        
  
        searchPageResult.isTooManyResults();  //should be true if too many 
results
        searchPageResult.getResults().size();   //should be 2 if page size if 
2

If you look in the ldap logs, you should only see 2 records sent back for 
this query


Thanks,
Chris

-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Tom Zeller
Sent: Wednesday, November 02, 2011 1:28 PM
To: Chris Hyzer
Cc: Grouper Dev
Subject: Re: [grouper-dev] RE: grouper 2.0 subject issue

Is this relevant to the ldap source adapter ?

As an aside, it would be nice to know which source adapters are used
by deployers, i.e. sql, ldap, or custom.

TomZ

> There is a new source method called searchPage where if the source has a
> page limit, then it will only return the first X+1 records, and the result
> is a bean which has the subjects and a boolean which says if there are more
> records:
>
>     <!-- on a findPage() this is the most results returned -->
>
>     <init-param>
>       <param-name>maxPageSize</param-name>
>       <param-value>100</param-value>
>     </init-param>
>
>
> If there is no maxPageSize then it will work without paging like a normal
> search. All UI/WS calls to find subjects will use the paging method. Note,
> there is no hibernate in the subject api, so the query is edited in java.
> This works in hsql, oracle, mysql, postgres (note: sql server paging seems
> too complicated). If your query is too complex and the paging causes errors,
> then you can disable this behavior with this setting:
>
>     <init-param>
>       <param-name>changeSearchQueryForMaxResults</param-name>
>       <param-value>false</param-value>
>     </init-param>
>
>
> Also, groups returned as subjects have a query for each group to get
> attributes, this is now done in 1 or a small number of queries.
>
> Group subjects should probably not have editedBy and createdBy. There is a
> switch to turn this off. This also causes a query per group returned.
>
> # if the creator and last updater should be group subject attributes (you
> get
> # a performance gain if you set to false, but if true you can see subject id
> from UI in 2.0
> subjects.group.useCreatorAndModifierAsSubjectAttributes = true
>
>
> That currently defaults to true so there are no changes.
>
>
>
> Note, there is more work to do at some point:
>
>
>
> https://bugs.internet2.edu/jira/browse/GRP-686
>
>
>
> Thanks,
>
> Chris
>
>
>
> From: Chris Hyzer
> Sent: Sunday, October 30, 2011 2:41 AM
> To: Chris Hyzer; 'Grouper Dev'
> Subject: RE: grouper 2.0 subject issue
>
>
>
> I implemented threads and a threadpool, and it didn't speed things up, I
> think because I have a sql search on the same db as grouper.  Well, I think
> we still need the paging part, maybe the threading should default to off.  I
> still need to do some checks about the paging.  Shilen, do you mind looking
> the code in the v2.0 branch (api, subject, ui, ws)?
>
>
>
> These control the paging in grouper.properties
>
>
>
> # if finding across multiple threadable sources, use threads to do the work
> faster
>
> subjects.allPage.useThreadForkJoin = true
>
>
>
> # if finding across multiple threadable sources, use threads to do the work
> faster
>
> subjects.idOrIdentifier.useThreadForkJoin = true
>
>
>
> You can set logging to debug here:
>
>
>
> log4j.logger.edu.internet2.middleware.grouper.subj.SourcesXmlResolver =
> DEBUG
>
>
>
> Then you can run this benchmark (mine has 80k groups, and 500k subjects in
> pennperson and a large search string):
>
>
>
> SubjectFinderBenchmark
>
>
>
> You will see output like this for threading:
>
>
>
> 2011-10-30 02:27:21,480: [pool-1-thread-5] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> g:isa, 'bah', time in millis: 0
>
> 2011-10-30 02:27:21,487: [pool-1-thread-2] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> servPrinc, 'bah', time in millis: 9
>
> 2011-10-30 02:27:21,488: [pool-1-thread-4] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> grouperExternal, 'bah', time in millis: 9
>
> 2011-10-30 02:27:22,127: [pool-1-thread-1] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> g:gsa, 'bah', time in millis: 647
>
> 2011-10-30 02:27:22,255: [pool-1-thread-3] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> pennperson, 'bah', time in millis: 775
>
> 2011-10-30 02:27:22,255: [main] DEBUG
> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
> threads: true, time in millis: 789
>
>
>
> Or like this without threading:
>
>
>
> 2011-10-30 02:39:28,592: [main] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> servPrinc, 'bah', time in millis: 4
>
> 2011-10-30 02:39:28,802: [main] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> g:gsa, 'bah', time in millis: 209
>
> 2011-10-30 02:39:29,442: [main] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> pennperson, 'bah', time in millis: 640
>
> 2011-10-30 02:39:29,448: [main] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> grouperExternal, 'bah', time in millis: 5
>
> 2011-10-30 02:39:29,448: [main] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> g:isa, 'bah', time in millis: 0
>
> 2011-10-30 02:39:29,448: [main] DEBUG
> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
> threads: false, time in millis: 860
>
>
>
> There are also tests in TestSubjectFinder.testSearchPageMax()
>
>
>
> Well, just thought of something. if there are lots of results, I wonder if
> threading helps, my benchmark now wont return a lot of results. let me try.
>
>
>
> Thanks,
>
> Chris
>
>
>
> From: Chris Hyzer
> Sent: Friday, October 28, 2011 8:27 AM
> To: Grouper Dev
> Subject: grouper 2.0 subject issue
>
>
>
> When I do a findall (e.g. UI search) for something that returns 1000+
> Subjects (which I have as my maxResults) in my sources.xml, it gives an
> exception.  It is supposed to throw a SubjectTooManyResults exception which
> is caught in UI and displayed on the screen.  I could fix this immediate
> problem, but I kind of think that there should be another method in the
> source called searchPage(searchString) which is just like search, but if the
> source has a
>
>
>
>     <init-param>
>
>       <param-name>maxPageSize</param-name>
>
>       <param-value>100</param-value>
>
>     </init-param>
>
>
>
> Then on a searchPage(), only 100 results are returned, in addition it will
> say if there are more or not.  The UI could just show the 100, and the user
> would narrow their search.  This seems better than showing nothing and
> having them narrow their search.  One reason is if the search string matches
> a subjectId or subjectIdentifier, it goes to the top.  If the
> subjectIdentifier is "fred", and that returns more than 100 results, but the
> netId "fred" is at the top of the list, the user is happy.  this is for
> 2.0.2
>
>
>
> I discussed this with Shilen and his reaction was positive J
>
>
>
> Thoughts?
>
>
>
> Thanks,
>
> Chris