Skip to Content.
Sympa Menu

grouper-dev - [grouper-dev] subject batching performance improvement

Subject: Grouper Developers Forum

List archive

[grouper-dev] subject batching performance improvement


Chronological Thread 
  • From: Chris Hyzer <>
  • To: Grouper Dev <>
  • Subject: [grouper-dev] subject batching performance improvement
  • Date: Sun, 27 Nov 2011 05:21:10 +0000
  • Accept-language: en-US

I finished coding the subject resolution batching performance improvement… Penn has a subject picker which finds subjects who are employees, and it would take more than 20 seconds to search which caused a timeout.  Now it takes a few seconds, and the number of queries and amount of data is bounded (not by N J ).

 

https://bugs.internet2.edu/jira/browse/GRP-713

 

Subject batching to not require N queries has been added to the following places:

UI subject picker when subject is in a group
UI get members
UI get members lite
WS get members
WS get privileges
WS get memberships

 

https://bugs.internet2.edu/jira/browse/GRP-712

 

There should be batching (for jdbc and jdbc2 source adapters) so that multiple subjects can be resolved at once. Note: for source adapters which do not implement a batched method to retrieve multiple subjects, they will be retrieved as they were before one at a time.

If you use the jdbc2 source adapter, you dont have to configure anything differently, it will just work.

If you use the jdbc source adapter, compare the new sources.example.xml, you can set these properties

     <!-- if you are going to use the inclause attribute
       on the search to make the queries batchable when searching
       by id or identifier -->
     <init-param>
       <param-name>useInClauseForIdAndIdentifier</param-name>
       <param-value>true</param-value>
     </init-param>
     
     <!-- comma separate the identifiers for this row, this is for the findByIdentifiers if using an in clause -->
     <init-param>
       <param-name>identifierAttributes</param-name>
       <param-value>LOGINID</param-value>
     </init-param>

Then you can change your queries for id or identifier to have an inclause variable, and assign that variable for the inclause part:

     <search>
         <searchType>searchSubject</searchType>
         <param>
             <param-name>sql</param-name>
             <param-value>
select
   s.subjectid as id, s.name as name,
   (select sa2.value from subjectattribute sa2 where name='name' and sa2.SUBJECTID = s.subjectid) as lfname,
   (select sa3.value from subjectattribute sa3 where name='loginid' and sa3.SUBJECTID = s.subjectid) as loginid,
   (select sa4.value from subjectattribute sa4 where name='description' and sa4.SUBJECTID = s.subjectid) as description,
   (select sa5.value from subjectattribute sa5 where name='email' and sa5.SUBJECTID = s.subjectid) as email
from
   subject s
where
   {inclause}
            </param-value>
         </param>
         <param>
             <param-name>inclause</param-name>
             <param-value>
s.subjectid = ?
            </param-value>
         </param>
     </search>
     <search>
         <searchType>searchSubjectByIdentifier</searchType>
         <param>
             <param-name>sql</param-name>
             <param-value>
select
   s.subjectid as id, s.name as name,
   (select sa2.value from subjectattribute sa2 where name='name' and sa2.SUBJECTID = s.subjectid) as lfname,
   (select sa3.value from subjectattribute sa3 where name='loginid' and sa3.SUBJECTID = s.subjectid) as loginid,
   (select sa4.value from subjectattribute sa4 where name='description' and sa4.SUBJECTID = s.subjectid) as description,
   (select sa5.value from subjectattribute sa5 where name='email' and sa5.SUBJECTID = s.subjectid) as email
from
   subject s, subjectattribute a
where
   a.name='loginid' and s.subjectid = a.subjectid and {inclause}
             </param-value>
         </param>
         <param>
             <param-name>inclause</param-name>
             <param-value>
   a.value = ?
            </param-value>
         </param>
     </search>




There are now these SubjectFinder methods:

SubjectFinder.findByIds(Collection<String>)
SubjectFinder.findByIds(Collection<String>, String)
SubjectFinder.findByIdentifiers(Collection<String>)
SubjectFinder.findByIdentifiers(Collection<String>, String)
SubjectFinder.findByIdsOrIdentifiers(Collection<String>)
SubjectFinder.findByIdsOrIdentifiers(Collection<String>, String)

There is this method in member to resolve subjects in a collection of members at once:

Member.resolveSubjects(Collection<Member>, boolean)

This method will resolve subjects in membership array results:

Membership.resolveSubjects(Collection<Object[]>)

This method will resolve subjects in privilege objects:

PrivilegeHelper.resolveSubjects(Collection<GrouperPrivilege>, boolean)

 

 

Thanks,

Chris




Archive powered by MHonArc 2.6.16.

Top of Page