Grouper Developers Forum

[Tom Zeller <>]

Actually, we could perform n searches in parallel. That might be more generic.

On Nov 19, 2011, at 3:12 PM, Tom Zeller 
<>
 wrote:

> Depends on the query complexity and the particular implementation.
> 
> Ldap filter or'ing might be ok, but I think would be frowned upon by server 
> operators.
> 
> On Nov 19, 2011, at 2:09 PM, Chris Hyzer 
> <>
>  wrote:
> 
>> Argh... we are still getting bad performance on the subject picker we use 
>> from grouper on our eforms (search for someone in the employee group by 
>> name in a combobox).
>> 
>> I think the reason is we haven't migrated to use the find subject in group 
>> method.  However, I noticed that even with that method, it will find the 
>> relevant subjects, but I don't think it does the group security in one 
>> query.  So I will add that.  Also, once it finds the subjects, to display 
>> them on the screen, it needs to resolve them, which is N queries (100?  
>> 200?)  We have a batch method on the subject API Source interface, but it 
>> is not implemented.  I will implement it in the JDBC and JDBC2 source 
>> adapters so they can retrieve 100 subjects in one query.  Do we want to do 
>> that in the ldap and vt-ldap adapters?  Is getting 100 subjects with 100 
>> jndi queries much slower than one filter which gets 100 (or some batched 
>> number) at once?
>> 
>> Thanks,
>> Chris
>> 
>> -----Original Message-----
>> From: Chris Hyzer 
>> Sent: Wednesday, November 02, 2011 2:25 PM
>> To: 'Tom Zeller'
>> Cc: Grouper Dev
>> Subject: RE: [grouper-dev] RE: grouper 2.0 subject issue
>> 
>> Yes, if you can test the jndi and ldap source adapter, I put the limit in 
>> there too for paging, though as people will tell me, there is (typically?) 
>> a server-side limit anyways so it might not be a big deal...
>> 
>> Basically to test it, set the paging to a small number, and run it...  see 
>> TestSubjectFinder.testSearchPageMax() in the 2.0 branch, something like 
>> this:
>> 
>>   <!-- on a findPage() this is the most results returned --> 
>>   <init-param>
>>     <param-name>maxPageSize</param-name>
>>     <param-value>2</param-value>
>>   </init-param>
>> 
>> (from GSH?)
>> 
>>       searchPageResult = SubjectFinder.findPage("searchString", 
>> "sourceId");
>> 
>> 
>>       searchPageResult.isTooManyResults();  //should be true if too many 
>> results
>>       searchPageResult.getResults().size(); //should be 2 if page size if 2
>> 
>> If you look in the ldap logs, you should only see 2 records sent back for 
>> this query
>> 
>> 
>> Thanks,
>> Chris
>> 
>> -----Original Message-----
>> From: 
>> 
>>  
>> [mailto:]
>>  On Behalf Of Tom Zeller
>> Sent: Wednesday, November 02, 2011 1:28 PM
>> To: Chris Hyzer
>> Cc: Grouper Dev
>> Subject: Re: [grouper-dev] RE: grouper 2.0 subject issue
>> 
>> Is this relevant to the ldap source adapter ?
>> 
>> As an aside, it would be nice to know which source adapters are used
>> by deployers, i.e. sql, ldap, or custom.
>> 
>> TomZ
>> 
>>> There is a new source method called searchPage where if the source has a
>>> page limit, then it will only return the first X+1 records, and the result
>>> is a bean which has the subjects and a boolean which says if there are 
>>> more
>>> records:
>>> 
>>>    <!-- on a findPage() this is the most results returned -->
>>> 
>>>    <init-param>
>>>      <param-name>maxPageSize</param-name>
>>>      <param-value>100</param-value>
>>>    </init-param>
>>> 
>>> 
>>> If there is no maxPageSize then it will work without paging like a normal
>>> search. All UI/WS calls to find subjects will use the paging method. Note,
>>> there is no hibernate in the subject api, so the query is edited in java.
>>> This works in hsql, oracle, mysql, postgres (note: sql server paging seems
>>> too complicated). If your query is too complex and the paging causes 
>>> errors,
>>> then you can disable this behavior with this setting:
>>> 
>>>    <init-param>
>>>      <param-name>changeSearchQueryForMaxResults</param-name>
>>>      <param-value>false</param-value>
>>>    </init-param>
>>> 
>>> 
>>> Also, groups returned as subjects have a query for each group to get
>>> attributes, this is now done in 1 or a small number of queries.
>>> 
>>> Group subjects should probably not have editedBy and createdBy. There is a
>>> switch to turn this off. This also causes a query per group returned.
>>> 
>>> # if the creator and last updater should be group subject attributes (you
>>> get
>>> # a performance gain if you set to false, but if true you can see subject 
>>> id
>>> from UI in 2.0
>>> subjects.group.useCreatorAndModifierAsSubjectAttributes = true
>>> 
>>> 
>>> That currently defaults to true so there are no changes.
>>> 
>>> 
>>> 
>>> Note, there is more work to do at some point:
>>> 
>>> 
>>> 
>>> https://bugs.internet2.edu/jira/browse/GRP-686
>>> 
>>> 
>>> 
>>> Thanks,
>>> 
>>> Chris
>>> 
>>> 
>>> 
>>> From: Chris Hyzer
>>> Sent: Sunday, October 30, 2011 2:41 AM
>>> To: Chris Hyzer; 'Grouper Dev'
>>> Subject: RE: grouper 2.0 subject issue
>>> 
>>> 
>>> 
>>> I implemented threads and a threadpool, and it didn't speed things up, I
>>> think because I have a sql search on the same db as grouper.  Well, I 
>>> think
>>> we still need the paging part, maybe the threading should default to off. 
>>>  I
>>> still need to do some checks about the paging.  Shilen, do you mind 
>>> looking
>>> the code in the v2.0 branch (api, subject, ui, ws)?
>>> 
>>> 
>>> 
>>> These control the paging in grouper.properties
>>> 
>>> 
>>> 
>>> # if finding across multiple threadable sources, use threads to do the 
>>> work
>>> faster
>>> 
>>> subjects.allPage.useThreadForkJoin = true
>>> 
>>> 
>>> 
>>> # if finding across multiple threadable sources, use threads to do the 
>>> work
>>> faster
>>> 
>>> subjects.idOrIdentifier.useThreadForkJoin = true
>>> 
>>> 
>>> 
>>> You can set logging to debug here:
>>> 
>>> 
>>> 
>>> log4j.logger.edu.internet2.middleware.grouper.subj.SourcesXmlResolver =
>>> DEBUG
>>> 
>>> 
>>> 
>>> Then you can run this benchmark (mine has 80k groups, and 500k subjects in
>>> pennperson and a large search string):
>>> 
>>> 
>>> 
>>> SubjectFinderBenchmark
>>> 
>>> 
>>> 
>>> You will see output like this for threading:
>>> 
>>> 
>>> 
>>> 2011-10-30 02:27:21,480: [pool-1-thread-5] DEBUG
>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>> g:isa, 'bah', time in millis: 0
>>> 
>>> 2011-10-30 02:27:21,487: [pool-1-thread-2] DEBUG
>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>> servPrinc, 'bah', time in millis: 9
>>> 
>>> 2011-10-30 02:27:21,488: [pool-1-thread-4] DEBUG
>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>> grouperExternal, 'bah', time in millis: 9
>>> 
>>> 2011-10-30 02:27:22,127: [pool-1-thread-1] DEBUG
>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>> g:gsa, 'bah', time in millis: 647
>>> 
>>> 2011-10-30 02:27:22,255: [pool-1-thread-3] DEBUG
>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>> pennperson, 'bah', time in millis: 775
>>> 
>>> 2011-10-30 02:27:22,255: [main] DEBUG
>>> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
>>> threads: true, time in millis: 789
>>> 
>>> 
>>> 
>>> Or like this without threading:
>>> 
>>> 
>>> 
>>> 2011-10-30 02:39:28,592: [main] DEBUG
>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>> servPrinc, 'bah', time in millis: 4
>>> 
>>> 2011-10-30 02:39:28,802: [main] DEBUG
>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>> g:gsa, 'bah', time in millis: 209
>>> 
>>> 2011-10-30 02:39:29,442: [main] DEBUG
>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>> pennperson, 'bah', time in millis: 640
>>> 
>>> 2011-10-30 02:39:29,448: [main] DEBUG
>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>> grouperExternal, 'bah', time in millis: 5
>>> 
>>> 2011-10-30 02:39:29,448: [main] DEBUG
>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>> g:isa, 'bah', time in millis: 0
>>> 
>>> 2011-10-30 02:39:29,448: [main] DEBUG
>>> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
>>> threads: false, time in millis: 860
>>> 
>>> 
>>> 
>>> There are also tests in TestSubjectFinder.testSearchPageMax()
>>> 
>>> 
>>> 
>>> Well, just thought of something. if there are lots of results, I wonder if
>>> threading helps, my benchmark now wont return a lot of results. let me 
>>> try.
>>> 
>>> 
>>> 
>>> Thanks,
>>> 
>>> Chris
>>> 
>>> 
>>> 
>>> From: Chris Hyzer
>>> Sent: Friday, October 28, 2011 8:27 AM
>>> To: Grouper Dev
>>> Subject: grouper 2.0 subject issue
>>> 
>>> 
>>> 
>>> When I do a findall (e.g. UI search) for something that returns 1000+
>>> Subjects (which I have as my maxResults) in my sources.xml, it gives an
>>> exception.  It is supposed to throw a SubjectTooManyResults exception 
>>> which
>>> is caught in UI and displayed on the screen.  I could fix this immediate
>>> problem, but I kind of think that there should be another method in the
>>> source called searchPage(searchString) which is just like search, but if 
>>> the
>>> source has a
>>> 
>>> 
>>> 
>>>    <init-param>
>>> 
>>>      <param-name>maxPageSize</param-name>
>>> 
>>>      <param-value>100</param-value>
>>> 
>>>    </init-param>
>>> 
>>> 
>>> 
>>> Then on a searchPage(), only 100 results are returned, in addition it will
>>> say if there are more or not.  The UI could just show the 100, and the 
>>> user
>>> would narrow their search.  This seems better than showing nothing and
>>> having them narrow their search.  One reason is if the search string 
>>> matches
>>> a subjectId or subjectIdentifier, it goes to the top.  If the
>>> subjectIdentifier is "fred", and that returns more than 100 results, but 
>>> the
>>> netId "fred" is at the top of the list, the user is happy.  this is for
>>> 2.0.2
>>> 
>>> 
>>> 
>>> I discussed this with Shilen and his reaction was positive J
>>> 
>>> 
>>> 
>>> Thoughts?
>>> 
>>> 
>>> 
>>> Thanks,
>>> 
>>> Chris