Grouper Developers Forum

[Tom Zeller <>]

Depends on the query complexity and the particular implementation.

Ldap filter or'ing might be ok, but I think would be frowned upon by server 
operators.

On Nov 19, 2011, at 2:09 PM, Chris Hyzer 
<>
 wrote:

> Argh... we are still getting bad performance on the subject picker we use 
> from grouper on our eforms (search for someone in the employee group by 
> name in a combobox).
> 
> I think the reason is we haven't migrated to use the find subject in group 
> method.  However, I noticed that even with that method, it will find the 
> relevant subjects, but I don't think it does the group security in one 
> query.  So I will add that.  Also, once it finds the subjects, to display 
> them on the screen, it needs to resolve them, which is N queries (100?  
> 200?)  We have a batch method on the subject API Source interface, but it 
> is not implemented.  I will implement it in the JDBC and JDBC2 source 
> adapters so they can retrieve 100 subjects in one query.  Do we want to do 
> that in the ldap and vt-ldap adapters?  Is getting 100 subjects with 100 
> jndi queries much slower than one filter which gets 100 (or some batched 
> number) at once?
> 
> Thanks,
> Chris
> 
> -----Original Message-----
> From: Chris Hyzer 
> Sent: Wednesday, November 02, 2011 2:25 PM
> To: 'Tom Zeller'
> Cc: Grouper Dev
> Subject: RE: [grouper-dev] RE: grouper 2.0 subject issue
> 
> Yes, if you can test the jndi and ldap source adapter, I put the limit in 
> there too for paging, though as people will tell me, there is (typically?) 
> a server-side limit anyways so it might not be a big deal...
> 
> Basically to test it, set the paging to a small number, and run it...  see 
> TestSubjectFinder.testSearchPageMax() in the 2.0 branch, something like 
> this:
> 
>    <!-- on a findPage() this is the most results returned --> 
>    <init-param>
>      <param-name>maxPageSize</param-name>
>      <param-value>2</param-value>
>    </init-param>
> 
> (from GSH?)
> 
>        searchPageResult = SubjectFinder.findPage("searchString", 
> "sourceId");
> 
> 
>        searchPageResult.isTooManyResults();  //should be true if too many 
> results
>        searchPageResult.getResults().size(); //should be 2 if page size if 2
> 
> If you look in the ldap logs, you should only see 2 records sent back for 
> this query
> 
> 
> Thanks,
> Chris
> 
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Tom Zeller
> Sent: Wednesday, November 02, 2011 1:28 PM
> To: Chris Hyzer
> Cc: Grouper Dev
> Subject: Re: [grouper-dev] RE: grouper 2.0 subject issue
> 
> Is this relevant to the ldap source adapter ?
> 
> As an aside, it would be nice to know which source adapters are used
> by deployers, i.e. sql, ldap, or custom.
> 
> TomZ
> 
>> There is a new source method called searchPage where if the source has a
>> page limit, then it will only return the first X+1 records, and the result
>> is a bean which has the subjects and a boolean which says if there are more
>> records:
>> 
>>     <!-- on a findPage() this is the most results returned -->
>> 
>>     <init-param>
>>       <param-name>maxPageSize</param-name>
>>       <param-value>100</param-value>
>>     </init-param>
>> 
>> 
>> If there is no maxPageSize then it will work without paging like a normal
>> search. All UI/WS calls to find subjects will use the paging method. Note,
>> there is no hibernate in the subject api, so the query is edited in java.
>> This works in hsql, oracle, mysql, postgres (note: sql server paging seems
>> too complicated). If your query is too complex and the paging causes 
>> errors,
>> then you can disable this behavior with this setting:
>> 
>>     <init-param>
>>       <param-name>changeSearchQueryForMaxResults</param-name>
>>       <param-value>false</param-value>
>>     </init-param>
>> 
>> 
>> Also, groups returned as subjects have a query for each group to get
>> attributes, this is now done in 1 or a small number of queries.
>> 
>> Group subjects should probably not have editedBy and createdBy. There is a
>> switch to turn this off. This also causes a query per group returned.
>> 
>> # if the creator and last updater should be group subject attributes (you
>> get
>> # a performance gain if you set to false, but if true you can see subject 
>> id
>> from UI in 2.0
>> subjects.group.useCreatorAndModifierAsSubjectAttributes = true
>> 
>> 
>> That currently defaults to true so there are no changes.
>> 
>> 
>> 
>> Note, there is more work to do at some point:
>> 
>> 
>> 
>> https://bugs.internet2.edu/jira/browse/GRP-686
>> 
>> 
>> 
>> Thanks,
>> 
>> Chris
>> 
>> 
>> 
>> From: Chris Hyzer
>> Sent: Sunday, October 30, 2011 2:41 AM
>> To: Chris Hyzer; 'Grouper Dev'
>> Subject: RE: grouper 2.0 subject issue
>> 
>> 
>> 
>> I implemented threads and a threadpool, and it didn't speed things up, I
>> think because I have a sql search on the same db as grouper.  Well, I think
>> we still need the paging part, maybe the threading should default to off.  
>> I
>> still need to do some checks about the paging.  Shilen, do you mind looking
>> the code in the v2.0 branch (api, subject, ui, ws)?
>> 
>> 
>> 
>> These control the paging in grouper.properties
>> 
>> 
>> 
>> # if finding across multiple threadable sources, use threads to do the work
>> faster
>> 
>> subjects.allPage.useThreadForkJoin = true
>> 
>> 
>> 
>> # if finding across multiple threadable sources, use threads to do the work
>> faster
>> 
>> subjects.idOrIdentifier.useThreadForkJoin = true
>> 
>> 
>> 
>> You can set logging to debug here:
>> 
>> 
>> 
>> log4j.logger.edu.internet2.middleware.grouper.subj.SourcesXmlResolver =
>> DEBUG
>> 
>> 
>> 
>> Then you can run this benchmark (mine has 80k groups, and 500k subjects in
>> pennperson and a large search string):
>> 
>> 
>> 
>> SubjectFinderBenchmark
>> 
>> 
>> 
>> You will see output like this for threading:
>> 
>> 
>> 
>> 2011-10-30 02:27:21,480: [pool-1-thread-5] DEBUG
>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>> g:isa, 'bah', time in millis: 0
>> 
>> 2011-10-30 02:27:21,487: [pool-1-thread-2] DEBUG
>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>> servPrinc, 'bah', time in millis: 9
>> 
>> 2011-10-30 02:27:21,488: [pool-1-thread-4] DEBUG
>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>> grouperExternal, 'bah', time in millis: 9
>> 
>> 2011-10-30 02:27:22,127: [pool-1-thread-1] DEBUG
>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>> g:gsa, 'bah', time in millis: 647
>> 
>> 2011-10-30 02:27:22,255: [pool-1-thread-3] DEBUG
>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>> pennperson, 'bah', time in millis: 775
>> 
>> 2011-10-30 02:27:22,255: [main] DEBUG
>> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
>> threads: true, time in millis: 789
>> 
>> 
>> 
>> Or like this without threading:
>> 
>> 
>> 
>> 2011-10-30 02:39:28,592: [main] DEBUG
>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>> servPrinc, 'bah', time in millis: 4
>> 
>> 2011-10-30 02:39:28,802: [main] DEBUG
>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>> g:gsa, 'bah', time in millis: 209
>> 
>> 2011-10-30 02:39:29,442: [main] DEBUG
>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>> pennperson, 'bah', time in millis: 640
>> 
>> 2011-10-30 02:39:29,448: [main] DEBUG
>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>> grouperExternal, 'bah', time in millis: 5
>> 
>> 2011-10-30 02:39:29,448: [main] DEBUG
>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>> g:isa, 'bah', time in millis: 0
>> 
>> 2011-10-30 02:39:29,448: [main] DEBUG
>> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
>> threads: false, time in millis: 860
>> 
>> 
>> 
>> There are also tests in TestSubjectFinder.testSearchPageMax()
>> 
>> 
>> 
>> Well, just thought of something. if there are lots of results, I wonder if
>> threading helps, my benchmark now wont return a lot of results. let me try.
>> 
>> 
>> 
>> Thanks,
>> 
>> Chris
>> 
>> 
>> 
>> From: Chris Hyzer
>> Sent: Friday, October 28, 2011 8:27 AM
>> To: Grouper Dev
>> Subject: grouper 2.0 subject issue
>> 
>> 
>> 
>> When I do a findall (e.g. UI search) for something that returns 1000+
>> Subjects (which I have as my maxResults) in my sources.xml, it gives an
>> exception.  It is supposed to throw a SubjectTooManyResults exception which
>> is caught in UI and displayed on the screen.  I could fix this immediate
>> problem, but I kind of think that there should be another method in the
>> source called searchPage(searchString) which is just like search, but if 
>> the
>> source has a
>> 
>> 
>> 
>>     <init-param>
>> 
>>       <param-name>maxPageSize</param-name>
>> 
>>       <param-value>100</param-value>
>> 
>>     </init-param>
>> 
>> 
>> 
>> Then on a searchPage(), only 100 results are returned, in addition it will
>> say if there are more or not.  The UI could just show the 100, and the user
>> would narrow their search.  This seems better than showing nothing and
>> having them narrow their search.  One reason is if the search string 
>> matches
>> a subjectId or subjectIdentifier, it goes to the top.  If the
>> subjectIdentifier is "fred", and that returns more than 100 results, but 
>> the
>> netId "fred" is at the top of the list, the user is happy.  this is for
>> 2.0.2
>> 
>> 
>> 
>> I discussed this with Shilen and his reaction was positive J
>> 
>> 
>> 
>> Thoughts?
>> 
>> 
>> 
>> Thanks,
>> 
>> Chris