Grouper Developers Forum

[Tom Zeller <>]

For an ldap source, limiting an OR search filter concatenation to N, defined in sources.xml, may be useful. N would be the max number of OR'd filters, say 100 by default ?

On Nov 27, 2011, at 11:37 AM, Chris Hyzer <> wrote:

For #1, we never decided if we wanted to do a filter that brings back multiple records, or multiple threads.  I don’t have a jndi or vt-ldap source, so best case is someone can give me access to theirs, or we expose them on the demo server or something.  Otherwise, I can try to code it and someone else can test it.  Or someone else can code it.

 

For #2, sure, we can do that.

 

Thanks,

Chris

 

From: [mailto:] On Behalf Of Tom Barton
Sent: Sunday, November 27, 2011 10:22 AM
To:
Subject: Re: [grouper-dev] subject batching performance improvement

 

Thanks Chris. Two questions for the dev team:

1. I think we realized that batching is also possible in JNDI context, yes? Is someone looking into that? For 2.0.2?

2. It'd be great to have performance numbers with and without batching for some simple set of test cases. Is that possible?

Thanks,
Tom

On 11/26/2011 11:21 PM, Chris Hyzer wrote:

I finished coding the subject resolution batching performance improvement… Penn has a subject picker which finds subjects who are employees, and it would take more than 20 seconds to search which caused a timeout.  Now it takes a few seconds, and the number of queries and amount of data is bounded (not by N J ).

 

https://bugs.internet2.edu/jira/browse/GRP-713

 

Subject batching to not require N queries has been added to the following places:

UI subject picker when subject is in a group
UI get members
UI get members lite
WS get members
WS get privileges
WS get memberships

 

https://bugs.internet2.edu/jira/browse/GRP-712

 

There should be batching (for jdbc and jdbc2 source adapters) so that multiple subjects can be resolved at once. Note: for source adapters which do not implement a batched method to retrieve multiple subjects, they will be retrieved as they were before one at a time.

If you use the jdbc2 source adapter, you dont have to configure anything differently, it will just work.

If you use the jdbc source adapter, compare the new sources.example.xml, you can set these properties

     <!-- if you are going to use the inclause attribute
       on the search to make the queries batchable when searching
       by id or identifier -->
     <init-param>
       <param-name>useInClauseForIdAndIdentifier</param-name>
       <param-value>true</param-value>
     </init-param>
     
     <!-- comma separate the identifiers for this row, this is for the findByIdentifiers if using an in clause -->
     <init-param>
       <param-name>identifierAttributes</param-name>
       <param-value>LOGINID</param-value>
     </init-param>

Then you can change your queries for id or identifier to have an inclause variable, and assign that variable for the inclause part:

     <search>
         <searchType>searchSubject</searchType>
         <param>
             <param-name>sql</param-name>
             <param-value>
select
   s.subjectid as id, s.name as name,
   (select sa2.value from subjectattribute sa2 where name='name' and sa2.SUBJECTID = s.subjectid) as lfname,
   (select sa3.value from subjectattribute sa3 where name='loginid' and sa3.SUBJECTID = s.subjectid) as loginid,
   (select sa4.value from subjectattribute sa4 where name='description' and sa4.SUBJECTID = s.subjectid) as description,
   (select sa5.value from subjectattribute sa5 where name='email' and sa5.SUBJECTID = s.subjectid) as email
from
   subject s
where
   {inclause}
            </param-value>
         </param>
         <param>
             <param-name>inclause</param-name>
             <param-value>
s.subjectid = ?
            </param-value>
         </param>
     </search>
     <search>
         <searchType>searchSubjectByIdentifier</searchType>
         <param>
             <param-name>sql</param-name>
             <param-value>
select
   s.subjectid as id, s.name as name,
   (select sa2.value from subjectattribute sa2 where name='name' and sa2.SUBJECTID = s.subjectid) as lfname,
   (select sa3.value from subjectattribute sa3 where name='loginid' and sa3.SUBJECTID = s.subjectid) as loginid,
   (select sa4.value from subjectattribute sa4 where name='description' and sa4.SUBJECTID = s.subjectid) as description,
   (select sa5.value from subjectattribute sa5 where name='email' and sa5.SUBJECTID = s.subjectid) as email
from
   subject s, subjectattribute a
where
   a.name='loginid' and s.subjectid = a.subjectid and {inclause}
             </param-value>
         </param>
         <param>
             <param-name>inclause</param-name>
             <param-value>
   a.value = ?
            </param-value>
         </param>
     </search>




There are now these SubjectFinder methods:

SubjectFinder.findByIds(Collection<String>)
SubjectFinder.findByIds(Collection<String>, String)
SubjectFinder.findByIdentifiers(Collection<String>)
SubjectFinder.findByIdentifiers(Collection<String>, String)
SubjectFinder.findByIdsOrIdentifiers(Collection<String>)
SubjectFinder.findByIdsOrIdentifiers(Collection<String>, String)

There is this method in member to resolve subjects in a collection of members at once:

Member.resolveSubjects(Collection<Member>, boolean)

This method will resolve subjects in membership array results:

Membership.resolveSubjects(Collection<Object[]>)

This method will resolve subjects in privilege objects:

PrivilegeHelper.resolveSubjects(Collection<GrouperPrivilege>, boolean)

 

 

Thanks,

Chris