Grouper Developers Forum

[Jim Fox <>]

Alternatively, you could eschew the combo box.  That widget seems like a
hammer looking for a nail.  Unless you can provide instant updates the
display always lags the user's typing.  It's very distracting.

Allowing a user to type as much of a name as she sees fit and then doing
the lookup makes a cleaner user experience and a lot less overhead on
the servers. 

Jim

On Sat, 2011-11-19 at 17:37 -0700, Tom Zeller wrote:
> Actually, we could perform n searches in parallel. That might be more 
> generic.
> 
> On Nov 19, 2011, at 3:12 PM, Tom Zeller 
> <>
>  wrote:
> 
> > Depends on the query complexity and the particular implementation.
> > 
> > Ldap filter or'ing might be ok, but I think would be frowned upon by 
> > server operators.
> > 
> > On Nov 19, 2011, at 2:09 PM, Chris Hyzer 
> > <>
> >  wrote:
> > 
> >> Argh... we are still getting bad performance on the subject picker we 
> >> use from grouper on our eforms (search for someone in the employee group 
> >> by name in a combobox).
> >> 
> >> I think the reason is we haven't migrated to use the find subject in 
> >> group method.  However, I noticed that even with that method, it will 
> >> find the relevant subjects, but I don't think it does the group security 
> >> in one query.  So I will add that.  Also, once it finds the subjects, to 
> >> display them on the screen, it needs to resolve them, which is N queries 
> >> (100?  200?)  We have a batch method on the subject API Source 
> >> interface, but it is not implemented.  I will implement it in the JDBC 
> >> and JDBC2 source adapters so they can retrieve 100 subjects in one 
> >> query.  Do we want to do that in the ldap and vt-ldap adapters?  Is 
> >> getting 100 subjects with 100 jndi queries much slower than one filter 
> >> which gets 100 (or some batched number) at once?
> >> 
> >> Thanks,
> >> Chris
> >> 
> >> -----Original Message-----
> >> From: Chris Hyzer 
> >> Sent: Wednesday, November 02, 2011 2:25 PM
> >> To: 'Tom Zeller'
> >> Cc: Grouper Dev
> >> Subject: RE: [grouper-dev] RE: grouper 2.0 subject issue
> >> 
> >> Yes, if you can test the jndi and ldap source adapter, I put the limit 
> >> in there too for paging, though as people will tell me, there is 
> >> (typically?) a server-side limit anyways so it might not be a big deal...
> >> 
> >> Basically to test it, set the paging to a small number, and run it...  
> >> see TestSubjectFinder.testSearchPageMax() in the 2.0 branch, something 
> >> like this:
> >> 
> >>   <!-- on a findPage() this is the most results returned --> 
> >>   <init-param>
> >>     <param-name>maxPageSize</param-name>
> >>     <param-value>2</param-value>
> >>   </init-param>
> >> 
> >> (from GSH?)
> >> 
> >>       searchPageResult = SubjectFinder.findPage("searchString", 
> >> "sourceId");
> >> 
> >> 
> >>       searchPageResult.isTooManyResults();  //should be true if too many 
> >> results
> >>       searchPageResult.getResults().size(); //should be 2 if page size 
> >> if 2
> >> 
> >> If you look in the ldap logs, you should only see 2 records sent back 
> >> for this query
> >> 
> >> 
> >> Thanks,
> >> Chris
> >> 
> >> -----Original Message-----
> >> From: 
> >> 
> >>  
> >> [mailto:]
> >>  On Behalf Of Tom Zeller
> >> Sent: Wednesday, November 02, 2011 1:28 PM
> >> To: Chris Hyzer
> >> Cc: Grouper Dev
> >> Subject: Re: [grouper-dev] RE: grouper 2.0 subject issue
> >> 
> >> Is this relevant to the ldap source adapter ?
> >> 
> >> As an aside, it would be nice to know which source adapters are used
> >> by deployers, i.e. sql, ldap, or custom.
> >> 
> >> TomZ
> >> 
> >>> There is a new source method called searchPage where if the source has a
> >>> page limit, then it will only return the first X+1 records, and the 
> >>> result
> >>> is a bean which has the subjects and a boolean which says if there are 
> >>> more
> >>> records:
> >>> 
> >>>    <!-- on a findPage() this is the most results returned -->
> >>> 
> >>>    <init-param>
> >>>      <param-name>maxPageSize</param-name>
> >>>      <param-value>100</param-value>
> >>>    </init-param>
> >>> 
> >>> 
> >>> If there is no maxPageSize then it will work without paging like a 
> >>> normal
> >>> search. All UI/WS calls to find subjects will use the paging method. 
> >>> Note,
> >>> there is no hibernate in the subject api, so the query is edited in 
> >>> java.
> >>> This works in hsql, oracle, mysql, postgres (note: sql server paging 
> >>> seems
> >>> too complicated). If your query is too complex and the paging causes 
> >>> errors,
> >>> then you can disable this behavior with this setting:
> >>> 
> >>>    <init-param>
> >>>      <param-name>changeSearchQueryForMaxResults</param-name>
> >>>      <param-value>false</param-value>
> >>>    </init-param>
> >>> 
> >>> 
> >>> Also, groups returned as subjects have a query for each group to get
> >>> attributes, this is now done in 1 or a small number of queries.
> >>> 
> >>> Group subjects should probably not have editedBy and createdBy. There 
> >>> is a
> >>> switch to turn this off. This also causes a query per group returned.
> >>> 
> >>> # if the creator and last updater should be group subject attributes 
> >>> (you
> >>> get
> >>> # a performance gain if you set to false, but if true you can see 
> >>> subject id
> >>> from UI in 2.0
> >>> subjects.group.useCreatorAndModifierAsSubjectAttributes = true
> >>> 
> >>> 
> >>> That currently defaults to true so there are no changes.
> >>> 
> >>> 
> >>> 
> >>> Note, there is more work to do at some point:
> >>> 
> >>> 
> >>> 
> >>> https://bugs.internet2.edu/jira/browse/GRP-686
> >>> 
> >>> 
> >>> 
> >>> Thanks,
> >>> 
> >>> Chris
> >>> 
> >>> 
> >>> 
> >>> From: Chris Hyzer
> >>> Sent: Sunday, October 30, 2011 2:41 AM
> >>> To: Chris Hyzer; 'Grouper Dev'
> >>> Subject: RE: grouper 2.0 subject issue
> >>> 
> >>> 
> >>> 
> >>> I implemented threads and a threadpool, and it didn't speed things up, I
> >>> think because I have a sql search on the same db as grouper.  Well, I 
> >>> think
> >>> we still need the paging part, maybe the threading should default to 
> >>> off.  I
> >>> still need to do some checks about the paging.  Shilen, do you mind 
> >>> looking
> >>> the code in the v2.0 branch (api, subject, ui, ws)?
> >>> 
> >>> 
> >>> 
> >>> These control the paging in grouper.properties
> >>> 
> >>> 
> >>> 
> >>> # if finding across multiple threadable sources, use threads to do the 
> >>> work
> >>> faster
> >>> 
> >>> subjects.allPage.useThreadForkJoin = true
> >>> 
> >>> 
> >>> 
> >>> # if finding across multiple threadable sources, use threads to do the 
> >>> work
> >>> faster
> >>> 
> >>> subjects.idOrIdentifier.useThreadForkJoin = true
> >>> 
> >>> 
> >>> 
> >>> You can set logging to debug here:
> >>> 
> >>> 
> >>> 
> >>> log4j.logger.edu.internet2.middleware.grouper.subj.SourcesXmlResolver =
> >>> DEBUG
> >>> 
> >>> 
> >>> 
> >>> Then you can run this benchmark (mine has 80k groups, and 500k subjects 
> >>> in
> >>> pennperson and a large search string):
> >>> 
> >>> 
> >>> 
> >>> SubjectFinderBenchmark
> >>> 
> >>> 
> >>> 
> >>> You will see output like this for threading:
> >>> 
> >>> 
> >>> 
> >>> 2011-10-30 02:27:21,480: [pool-1-thread-5] DEBUG
> >>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> >>> g:isa, 'bah', time in millis: 0
> >>> 
> >>> 2011-10-30 02:27:21,487: [pool-1-thread-2] DEBUG
> >>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> >>> servPrinc, 'bah', time in millis: 9
> >>> 
> >>> 2011-10-30 02:27:21,488: [pool-1-thread-4] DEBUG
> >>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> >>> grouperExternal, 'bah', time in millis: 9
> >>> 
> >>> 2011-10-30 02:27:22,127: [pool-1-thread-1] DEBUG
> >>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> >>> g:gsa, 'bah', time in millis: 647
> >>> 
> >>> 2011-10-30 02:27:22,255: [pool-1-thread-3] DEBUG
> >>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> >>> pennperson, 'bah', time in millis: 775
> >>> 
> >>> 2011-10-30 02:27:22,255: [main] DEBUG
> >>> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
> >>> threads: true, time in millis: 789
> >>> 
> >>> 
> >>> 
> >>> Or like this without threading:
> >>> 
> >>> 
> >>> 
> >>> 2011-10-30 02:39:28,592: [main] DEBUG
> >>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> >>> servPrinc, 'bah', time in millis: 4
> >>> 
> >>> 2011-10-30 02:39:28,802: [main] DEBUG
> >>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> >>> g:gsa, 'bah', time in millis: 209
> >>> 
> >>> 2011-10-30 02:39:29,442: [main] DEBUG
> >>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> >>> pennperson, 'bah', time in millis: 640
> >>> 
> >>> 2011-10-30 02:39:29,448: [main] DEBUG
> >>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> >>> grouperExternal, 'bah', time in millis: 5
> >>> 
> >>> 2011-10-30 02:39:29,448: [main] DEBUG
> >>> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> >>> g:isa, 'bah', time in millis: 0
> >>> 
> >>> 2011-10-30 02:39:29,448: [main] DEBUG
> >>> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
> >>> threads: false, time in millis: 860
> >>> 
> >>> 
> >>> 
> >>> There are also tests in TestSubjectFinder.testSearchPageMax()
> >>> 
> >>> 
> >>> 
> >>> Well, just thought of something. if there are lots of results, I wonder 
> >>> if
> >>> threading helps, my benchmark now wont return a lot of results. let me 
> >>> try.
> >>> 
> >>> 
> >>> 
> >>> Thanks,
> >>> 
> >>> Chris
> >>> 
> >>> 
> >>> 
> >>> From: Chris Hyzer
> >>> Sent: Friday, October 28, 2011 8:27 AM
> >>> To: Grouper Dev
> >>> Subject: grouper 2.0 subject issue
> >>> 
> >>> 
> >>> 
> >>> When I do a findall (e.g. UI search) for something that returns 1000+
> >>> Subjects (which I have as my maxResults) in my sources.xml, it gives an
> >>> exception.  It is supposed to throw a SubjectTooManyResults exception 
> >>> which
> >>> is caught in UI and displayed on the screen.  I could fix this immediate
> >>> problem, but I kind of think that there should be another method in the
> >>> source called searchPage(searchString) which is just like search, but 
> >>> if the
> >>> source has a
> >>> 
> >>> 
> >>> 
> >>>    <init-param>
> >>> 
> >>>      <param-name>maxPageSize</param-name>
> >>> 
> >>>      <param-value>100</param-value>
> >>> 
> >>>    </init-param>
> >>> 
> >>> 
> >>> 
> >>> Then on a searchPage(), only 100 results are returned, in addition it 
> >>> will
> >>> say if there are more or not.  The UI could just show the 100, and the 
> >>> user
> >>> would narrow their search.  This seems better than showing nothing and
> >>> having them narrow their search.  One reason is if the search string 
> >>> matches
> >>> a subjectId or subjectIdentifier, it goes to the top.  If the
> >>> subjectIdentifier is "fred", and that returns more than 100 results, 
> >>> but the
> >>> netId "fred" is at the top of the list, the user is happy.  this is for
> >>> 2.0.2
> >>> 
> >>> 
> >>> 
> >>> I discussed this with Shilen and his reaction was positive J
> >>> 
> >>> 
> >>> 
> >>> Thoughts?
> >>> 
> >>> 
> >>> 
> >>> Thanks,
> >>> 
> >>> Chris