Grouper Developers Forum

[Tom Zeller <>]

Is this relevant to the ldap source adapter ?

As an aside, it would be nice to know which source adapters are used
by deployers, i.e. sql, ldap, or custom.

TomZ

> There is a new source method called searchPage where if the source has a
> page limit, then it will only return the first X+1 records, and the result
> is a bean which has the subjects and a boolean which says if there are more
> records:
>
>     <!-- on a findPage() this is the most results returned -->
>
>     <init-param>
>       <param-name>maxPageSize</param-name>
>       <param-value>100</param-value>
>     </init-param>
>
>
> If there is no maxPageSize then it will work without paging like a normal
> search. All UI/WS calls to find subjects will use the paging method. Note,
> there is no hibernate in the subject api, so the query is edited in java.
> This works in hsql, oracle, mysql, postgres (note: sql server paging seems
> too complicated). If your query is too complex and the paging causes errors,
> then you can disable this behavior with this setting:
>
>     <init-param>
>       <param-name>changeSearchQueryForMaxResults</param-name>
>       <param-value>false</param-value>
>     </init-param>
>
>
> Also, groups returned as subjects have a query for each group to get
> attributes, this is now done in 1 or a small number of queries.
>
> Group subjects should probably not have editedBy and createdBy. There is a
> switch to turn this off. This also causes a query per group returned.
>
> # if the creator and last updater should be group subject attributes (you
> get
> # a performance gain if you set to false, but if true you can see subject id
> from UI in 2.0
> subjects.group.useCreatorAndModifierAsSubjectAttributes = true
>
>
> That currently defaults to true so there are no changes.
>
>
>
> Note, there is more work to do at some point:
>
>
>
> https://bugs.internet2.edu/jira/browse/GRP-686
>
>
>
> Thanks,
>
> Chris
>
>
>
> From: Chris Hyzer
> Sent: Sunday, October 30, 2011 2:41 AM
> To: Chris Hyzer; 'Grouper Dev'
> Subject: RE: grouper 2.0 subject issue
>
>
>
> I implemented threads and a threadpool, and it didn’t speed things up, I
> think because I have a sql search on the same db as grouper.  Well, I think
> we still need the paging part, maybe the threading should default to off…  I
> still need to do some checks about the paging…  Shilen, do you mind looking
> the code in the v2.0 branch (api, subject, ui, ws)?
>
>
>
> These control the paging in grouper.properties
>
>
>
> # if finding across multiple threadable sources, use threads to do the work
> faster
>
> subjects.allPage.useThreadForkJoin = true
>
>
>
> # if finding across multiple threadable sources, use threads to do the work
> faster
>
> subjects.idOrIdentifier.useThreadForkJoin = true
>
>
>
> You can set logging to debug here:
>
>
>
> log4j.logger.edu.internet2.middleware.grouper.subj.SourcesXmlResolver =
> DEBUG
>
>
>
> Then you can run this benchmark (mine has 80k groups, and 500k subjects in
> pennperson and a large search string):
>
>
>
> SubjectFinderBenchmark
>
>
>
> You will see output like this for threading:
>
>
>
> 2011-10-30 02:27:21,480: [pool-1-thread-5] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> g:isa, 'bah', time in millis: 0
>
> 2011-10-30 02:27:21,487: [pool-1-thread-2] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> servPrinc, 'bah', time in millis: 9
>
> 2011-10-30 02:27:21,488: [pool-1-thread-4] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> grouperExternal, 'bah', time in millis: 9
>
> 2011-10-30 02:27:22,127: [pool-1-thread-1] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> g:gsa, 'bah', time in millis: 647
>
> 2011-10-30 02:27:22,255: [pool-1-thread-3] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> pennperson, 'bah', time in millis: 775
>
> 2011-10-30 02:27:22,255: [main] DEBUG
> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
> threads: true, time in millis: 789
>
>
>
> Or like this without threading:
>
>
>
> 2011-10-30 02:39:28,592: [main] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> servPrinc, 'bah', time in millis: 4
>
> 2011-10-30 02:39:28,802: [main] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> g:gsa, 'bah', time in millis: 209
>
> 2011-10-30 02:39:29,442: [main] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> pennperson, 'bah', time in millis: 640
>
> 2011-10-30 02:39:29,448: [main] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> grouperExternal, 'bah', time in millis: 5
>
> 2011-10-30 02:39:29,448: [main] DEBUG
> SourcesXmlResolver$LogLabelCallable.call(155) -  - findPage on source:
> g:isa, 'bah', time in millis: 0
>
> 2011-10-30 02:39:29,448: [main] DEBUG
> SourcesXmlResolver.executeCallables(270) -  - SourcesXmlResolver: Using
> threads: false, time in millis: 860
>
>
>
> There are also tests in TestSubjectFinder.testSearchPageMax()
>
>
>
> Well, just thought of something… if there are lots of results, I wonder if
> threading helps, my benchmark now wont return a lot of results… let me try.
>
>
>
> Thanks,
>
> Chris
>
>
>
> From: Chris Hyzer
> Sent: Friday, October 28, 2011 8:27 AM
> To: Grouper Dev
> Subject: grouper 2.0 subject issue
>
>
>
> When I do a findall (e.g. UI search) for something that returns 1000+
> Subjects (which I have as my maxResults) in my sources.xml, it gives an
> exception.  It is supposed to throw a SubjectTooManyResults exception which
> is caught in UI and displayed on the screen.  I could fix this immediate
> problem, but I kind of think that there should be another method in the
> source called searchPage(searchString) which is just like search, but if the
> source has a
>
>
>
>     <init-param>
>
>       <param-name>maxPageSize</param-name>
>
>       <param-value>100</param-value>
>
>     </init-param>
>
>
>
> Then on a searchPage(), only 100 results are returned, in addition it will
> say if there are more or not.  The UI could just show the 100, and the user
> would narrow their search.  This seems better than showing nothing and
> having them narrow their search.  One reason is if the search string matches
> a subjectId or subjectIdentifier, it goes to the top.  If the
> subjectIdentifier is “fred”, and that returns more than 100 results, but the
> netId “fred” is at the top of the list, the user is happy…  this is for
> 2.0.2
>
>
>
> I discussed this with Shilen and his reaction was positive J
>
>
>
> Thoughts?
>
>
>
> Thanks,
>
> Chris