Grouper Developers Forum

[Tom Barton <>]

I have to agree with Jim on this point, ie, about the
search-as-user-types approach. It's not under-performant in our current
grouper deploy at uchicago, but it sure could become that way. I run
Touchdown on my droid, a rather full-featured mobile exchange client,
which uses this approach in some under-performant contexts and it's one
of the most annoying aspects of otherwise good software.

Tom

On 11/20/2011 1:24 PM, Jim Fox wrote:
> Alternatively, you could eschew the combo box.  That widget seems like a
> hammer looking for a nail.  Unless you can provide instant updates the
> display always lags the user's typing.  It's very distracting.
>
> Allowing a user to type as much of a name as she sees fit and then doing
> the lookup makes a cleaner user experience and a lot less overhead on
> the servers.
>
> Jim
>
> On Sat, 2011-11-19 at 17:37 -0700, Tom Zeller wrote:
>> Actually, we could perform n searches in parallel. That might be more 
>> generic.
>>
>> On Nov 19, 2011, at 3:12 PM, Tom Zeller 
>> <>
>>  wrote:
>>
>>> Depends on the query complexity and the particular implementation.
>>>
>>> Ldap filter or'ing might be ok, but I think would be frowned upon by 
>>> server operators.
>>>
>>> On Nov 19, 2011, at 2:09 PM, Chris Hyzer 
>>> <>
>>>  wrote:
>>>
>>>> Argh... we are still getting bad performance on the subject picker we 
>>>> use from grouper on our eforms (search for someone in the employee group 
>>>> by name in a combobox).
>>>>
>>>> I think the reason is we haven't migrated to use the find subject in 
>>>> group method.  However, I noticed that even with that method, it will 
>>>> find the relevant subjects, but I don't think it does the group security 
>>>> in one query.  So I will add that.  Also, once it finds the subjects, to 
>>>> display them on the screen, it needs to resolve them, which is N queries 
>>>> (100?  200?)  We have a batch method on the subject API Source 
>>>> interface, but it is not implemented.  I will implement it in the JDBC 
>>>> and JDBC2 source adapters so they can retrieve 100 subjects in one 
>>>> query.  Do we want to do that in the ldap and vt-ldap adapters?  Is 
>>>> getting 100 subjects with 100 jndi queries much slower than one filter 
>>>> which gets 100 (or some batched number) at once?
>>>>
>>>> Thanks,
>>>> Chris
>>>>
>>>> -----Original Message-----
>>>> From: Chris Hyzer
>>>> Sent: Wednesday, November 02, 2011 2:25 PM
>>>> To: 'Tom Zeller'
>>>> Cc: Grouper Dev
>>>> Subject: RE: [grouper-dev] RE: grouper 2.0 subject issue
>>>>
>>>> Yes, if you can test the jndi and ldap source adapter, I put the limit 
>>>> in there too for paging, though as people will tell me, there is 
>>>> (typically?) a server-side limit anyways so it might not be a big deal...
>>>>
>>>> Basically to test it, set the paging to a small number, and run it...  
>>>> see TestSubjectFinder.testSearchPageMax() in the 2.0 branch, something 
>>>> like this:
>>>>
>>>>   <!-- on a findPage() this is the most results returned -->
>>>>   <init-param>
>>>>     <param-name>maxPageSize</param-name>
>>>>     <param-value>2</param-value>
>>>>   </init-param>
>>>>
>>>> (from GSH?)
>>>>
>>>>       searchPageResult = SubjectFinder.findPage("searchString", 
>>>> "sourceId");
>>>>
>>>>
>>>>       searchPageResult.isTooManyResults();  //should be true if too many 
>>>> results
>>>>       searchPageResult.getResults().size(); //should be 2 if page size 
>>>> if 2
>>>>
>>>> If you look in the ldap logs, you should only see 2 records sent back 
>>>> for this query
>>>>
>>>>
>>>> Thanks,
>>>> Chris
>>>>
>>>> -----Original Message-----
>>>> From: 
>>>> 
>>>>  
>>>> [mailto:]
>>>>  On Behalf Of Tom Zeller
>>>> Sent: Wednesday, November 02, 2011 1:28 PM
>>>> To: Chris Hyzer
>>>> Cc: Grouper Dev
>>>> Subject: Re: [grouper-dev] RE: grouper 2.0 subject issue
>>>>
>>>> Is this relevant to the ldap source adapter ?
>>>>
>>>> As an aside, it would be nice to know which source adapters are used
>>>> by deployers, i.e. sql, ldap, or custom.
>>>>
>>>> TomZ
>>>>
>>>>> There is a new source method called searchPage where if the source has a
>>>>> page limit, then it will only return the first X+1 records, and the 
>>>>> result
>>>>> is a bean which has the subjects and a boolean which says if there are 
>>>>> more
>>>>> records:
>>>>>
>>>>>    <!-- on a findPage() this is the most results returned -->
>>>>>
>>>>>    <init-param>
>>>>>      <param-name>maxPageSize</param-name>
>>>>>      <param-value>100</param-value>
>>>>>    </init-param>
>>>>>
>>>>>
>>>>> If there is no maxPageSize then it will work without paging like a 
>>>>> normal
>>>>> search. All UI/WS calls to find subjects will use the paging method. 
>>>>> Note,
>>>>> there is no hibernate in the subject api, so the query is edited in 
>>>>> java.
>>>>> This works in hsql, oracle, mysql, postgres (note: sql server paging 
>>>>> seems
>>>>> too complicated). If your query is too complex and the paging causes 
>>>>> errors,
>>>>> then you can disable this behavior with this setting:
>>>>>
>>>>>    <init-param>
>>>>>      <param-name>changeSearchQueryForMaxResults</param-name>
>>>>>      <param-value>false</param-value>
>>>>>    </init-param>
>>>>>
>>>>>
>>>>> Also, groups returned as subjects have a query for each group to get
>>>>> attributes, this is now done in 1 or a small number of queries.
>>>>>
>>>>> Group subjects should probably not have editedBy and createdBy. There 
>>>>> is a
>>>>> switch to turn this off. This also causes a query per group returned.
>>>>>
>>>>> # if the creator and last updater should be group subject attributes 
>>>>> (you
>>>>> get
>>>>> # a performance gain if you set to false, but if true you can see 
>>>>> subject id
>>>>> from UI in 2.0
>>>>> subjects.group.useCreatorAndModifierAsSubjectAttributes = true
>>>>>
>>>>>
>>>>> That currently defaults to true so there are no changes.
>>>>>
>>>>>
>>>>>
>>>>> Note, there is more work to do at some point:
>>>>>
>>>>>
>>>>>
>>>>> https://bugs.internet2.edu/jira/browse/GRP-686
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>>
>>>>> From: Chris Hyzer
>>>>> Sent: Sunday, October 30, 2011 2:41 AM
>>>>> To: Chris Hyzer; 'Grouper Dev'
>>>>> Subject: RE: grouper 2.0 subject issue
>>>>>
>>>>>
>>>>>
>>>>> I implemented threads and a threadpool, and it didn't speed things up, I
>>>>> think because I have a sql search on the same db as grouper.  Well, I 
>>>>> think
>>>>> we still need the paging part, maybe the threading should default to 
>>>>> off.  I
>>>>> still need to do some checks about the paging.  Shilen, do you mind 
>>>>> looking
>>>>> the code in the v2.0 branch (api, subject, ui, ws)?
>>>>>
>>>>>
>>>>>
>>>>> These control the paging in grouper.properties
>>>>>
>>>>>
>>>>>
>>>>> # if finding across multiple threadable sources, use threads to do the 
>>>>> work
>>>>> faster
>>>>>
>>>>> subjects.allPage.useThreadForkJoin = true
>>>>>
>>>>>
>>>>>
>>>>> # if finding across multiple threadable sources, use threads to do the 
>>>>> work
>>>>> faster
>>>>>
>>>>> subjects.idOrIdentifier.useThreadForkJoin = true
>>>>>
>>>>>
>>>>>
>>>>> You can set logging to debug here:
>>>>>
>>>>>
>>>>>
>>>>> log4j.logger.edu.internet2.middleware.grouper.subj.SourcesXmlResolver =
>>>>> DEBUG
>>>>>
>>>>>
>>>>>
>>>>> Then you can run this benchmark (mine has 80k groups, and 500k subjects 
>>>>> in
>>>>> pennperson and a large search string):
>>>>>
>>>>>
>>>>>
>>>>> SubjectFinderBenchmark
>>>>>
>>>>>
>>>>>
>>>>> You will see output like this for threading:
>>>>>
>>>>>
>>>>>
>>>>> 2011-10-30 02:27:21,480: [pool-1-thread-5] DEBUG
>>>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>>>> g:isa, 'bah', time in millis: 0
>>>>>
>>>>> 2011-10-30 02:27:21,487: [pool-1-thread-2] DEBUG
>>>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>>>> servPrinc, 'bah', time in millis: 9
>>>>>
>>>>> 2011-10-30 02:27:21,488: [pool-1-thread-4] DEBUG
>>>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>>>> grouperExternal, 'bah', time in millis: 9
>>>>>
>>>>> 2011-10-30 02:27:22,127: [pool-1-thread-1] DEBUG
>>>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>>>> g:gsa, 'bah', time in millis: 647
>>>>>
>>>>> 2011-10-30 02:27:22,255: [pool-1-thread-3] DEBUG
>>>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>>>> pennperson, 'bah', time in millis: 775
>>>>>
>>>>> 2011-10-30 02:27:22,255: [main] DEBUG
>>>>> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
>>>>> threads: true, time in millis: 789
>>>>>
>>>>>
>>>>>
>>>>> Or like this without threading:
>>>>>
>>>>>
>>>>>
>>>>> 2011-10-30 02:39:28,592: [main] DEBUG
>>>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>>>> servPrinc, 'bah', time in millis: 4
>>>>>
>>>>> 2011-10-30 02:39:28,802: [main] DEBUG
>>>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>>>> g:gsa, 'bah', time in millis: 209
>>>>>
>>>>> 2011-10-30 02:39:29,442: [main] DEBUG
>>>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>>>> pennperson, 'bah', time in millis: 640
>>>>>
>>>>> 2011-10-30 02:39:29,448: [main] DEBUG
>>>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>>>> grouperExternal, 'bah', time in millis: 5
>>>>>
>>>>> 2011-10-30 02:39:29,448: [main] DEBUG
>>>>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
>>>>> g:isa, 'bah', time in millis: 0
>>>>>
>>>>> 2011-10-30 02:39:29,448: [main] DEBUG
>>>>> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
>>>>> threads: false, time in millis: 860
>>>>>
>>>>>
>>>>>
>>>>> There are also tests in TestSubjectFinder.testSearchPageMax()
>>>>>
>>>>>
>>>>>
>>>>> Well, just thought of something. if there are lots of results, I wonder 
>>>>> if
>>>>> threading helps, my benchmark now wont return a lot of results. let me 
>>>>> try.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>>
>>>>> From: Chris Hyzer
>>>>> Sent: Friday, October 28, 2011 8:27 AM
>>>>> To: Grouper Dev
>>>>> Subject: grouper 2.0 subject issue
>>>>>
>>>>>
>>>>>
>>>>> When I do a findall (e.g. UI search) for something that returns 1000+
>>>>> Subjects (which I have as my maxResults) in my sources.xml, it gives an
>>>>> exception.  It is supposed to throw a SubjectTooManyResults exception 
>>>>> which
>>>>> is caught in UI and displayed on the screen.  I could fix this immediate
>>>>> problem, but I kind of think that there should be another method in the
>>>>> source called searchPage(searchString) which is just like search, but 
>>>>> if the
>>>>> source has a
>>>>>
>>>>>
>>>>>
>>>>>    <init-param>
>>>>>
>>>>>      <param-name>maxPageSize</param-name>
>>>>>
>>>>>      <param-value>100</param-value>
>>>>>
>>>>>    </init-param>
>>>>>
>>>>>
>>>>>
>>>>> Then on a searchPage(), only 100 results are returned, in addition it 
>>>>> will
>>>>> say if there are more or not.  The UI could just show the 100, and the 
>>>>> user
>>>>> would narrow their search.  This seems better than showing nothing and
>>>>> having them narrow their search.  One reason is if the search string 
>>>>> matches
>>>>> a subjectId or subjectIdentifier, it goes to the top.  If the
>>>>> subjectIdentifier is "fred", and that returns more than 100 results, 
>>>>> but the
>>>>> netId "fred" is at the top of the list, the user is happy.  this is for
>>>>> 2.0.2
>>>>>
>>>>>
>>>>>
>>>>> I discussed this with Shilen and his reaction was positive J
>>>>>
>>>>>
>>>>>
>>>>> Thoughts?
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Chris
>