Grouper Developers Forum

[Thomas Barton <>]

Thanks Jim, that's great.

Tom (mobile)

-----Original Message----- 
From: Jim Fox []
Received: Sunday, 27 Nov 2011, 11:35pm
To: Chris Hyzer []
CC: Thomas Barton [];  []
Subject: RE: [grouper-dev] subject batching performance improvement

I can take a look at the vt-ladp source we provided.

Jim

On Sun, 2011-11-27 at 17:37 +0000, Chris Hyzer wrote:
> For #1, we never decided if we wanted to do a filter that brings back
> multiple records, or multiple threads.  I don’t have a jndi or vt-ldap
> source, so best case is someone can give me access to theirs, or we
> expose them on the demo server or something.  Otherwise, I can try to
> code it and someone else can test it.  Or someone else can code it.
>
> 
>
> For #2, sure, we can do that.
>
> 
>
> Thanks,
>
> Chris
>
> 
>
> From:
> [] On Behalf Of Tom Barton
> Sent: Sunday, November 27, 2011 10:22 AM
> To:
> Subject: Re: [grouper-dev] subject batching performance improvement
>
>
> 
>
> Thanks Chris. Two questions for the dev team:
>
> 1. I think we realized that batching is also possible in JNDI context,
> yes? Is someone looking into that? For 2.0.2?
>
> 2. It'd be great to have performance numbers with and without batching
> for some simple set of test cases. Is that possible?
>
> Thanks,
> Tom
>
> On 11/26/2011 11:21 PM, Chris Hyzer wrote:
>
> I finished coding the subject resolution batching performance
> improvement… Penn has a subject picker which finds subjects who are
> employees, and it would take more than 20 seconds to search which
> caused a timeout.  Now it takes a few seconds, and the number of
> queries and amount of data is bounded (not by N J ).
>
> 
>
> https://bugs.internet2.edu/jira/browse/GRP-713
>
> 
>
> Subject batching to not require N queries has been added to the
> following places:
>
> UI subject picker when subject is in a group
> UI get members
> UI get members lite
> WS get members
> WS get privileges
> WS get memberships
>
> 
>
> https://bugs.internet2.edu/jira/browse/GRP-712
>
> 
>
> There should be batching (for jdbc and jdbc2 source adapters) so that
> multiple subjects can be resolved at once. Note: for source adapters
> which do not implement a batched method to retrieve multiple subjects,
> they will be retrieved as they were before one at a time.
>
> If you use the jdbc2 source adapter, you dont have to configure
> anything differently, it will just work.
>
> If you use the jdbc source adapter, compare the new
> sources.example.xml, you can set these properties
>
>      <!-- if you are going to use the inclause attribute
>        on the search to make the queries batchable when searching
>        by id or identifier -->
>      <init-param>
>        <param-name>useInClauseForIdAndIdentifier</param-name>
>        <param-value>true</param-value>
>      </init-param>
>     
>      <!-- comma separate the identifiers for this row, this is for the
> findByIdentifiers if using an in clause -->
>      <init-param>
>        <param-name>identifierAttributes</param-name>
>        <param-value>LOGINID</param-value>
>      </init-param>
>
> Then you can change your queries for id or identifier to have an
> inclause variable, and assign that variable for the inclause part:
>
>      <search>
>          <searchType>searchSubject</searchType>
>          <param>
>              <param-name>sql</param-name>
>              <param-value>
> select
>    s.subjectid as id, s.name as name,
>    (select sa2.value from subjectattribute sa2 where name='name' and
> sa2.SUBJECTID = s.subjectid) as lfname,
>    (select sa3.value from subjectattribute sa3 where name='loginid'
> and sa3.SUBJECTID = s.subjectid) as loginid,
>    (select sa4.value from subjectattribute sa4 where
> name='description' and sa4.SUBJECTID = s.subjectid) as description,
>    (select sa5.value from subjectattribute sa5 where name='email' and
> sa5.SUBJECTID = s.subjectid) as email
> from
>    subject s
> where
>    {inclause}
>             </param-value>
>          </param>
>          <param>
>              <param-name>inclause</param-name>
>              <param-value>
> s.subjectid = ?
>             </param-value>
>          </param>
>      </search>
>      <search>
>          <searchType>searchSubjectByIdentifier</searchType>
>          <param>
>              <param-name>sql</param-name>
>              <param-value>
> select
>    s.subjectid as id, s.name as name,
>    (select sa2.value from subjectattribute sa2 where name='name' and
> sa2.SUBJECTID = s.subjectid) as lfname,
>    (select sa3.value from subjectattribute sa3 where name='loginid'
> and sa3.SUBJECTID = s.subjectid) as loginid,
>    (select sa4.value from subjectattribute sa4 where
> name='description' and sa4.SUBJECTID = s.subjectid) as description,
>    (select sa5.value from subjectattribute sa5 where name='email' and
> sa5.SUBJECTID = s.subjectid) as email
> from
>    subject s, subjectattribute a
> where
>    a.name='loginid' and s.subjectid = a.subjectid and {inclause}
>              </param-value>
>          </param>
>          <param>
>              <param-name>inclause</param-name>
>              <param-value>
>    a.value = ?
>             </param-value>
>          </param>
>      </search>
>
>
>
>
> There are now these SubjectFinder methods:
>
> SubjectFinder.findByIds(Collection<String>)
> SubjectFinder.findByIds(Collection<String>, String)
> SubjectFinder.findByIdentifiers(Collection<String>)
> SubjectFinder.findByIdentifiers(Collection<String>, String)
> SubjectFinder.findByIdsOrIdentifiers(Collection<String>)
> SubjectFinder.findByIdsOrIdentifiers(Collection<String>, String)
>
> There is this method in member to resolve subjects in a collection of
> members at once:
>
> Member.resolveSubjects(Collection<Member>, boolean)
>
> This method will resolve subjects in membership array results:
>
> Membership.resolveSubjects(Collection<Object[]>)
>
> This method will resolve subjects in privilege objects:
>
> PrivilegeHelper.resolveSubjects(Collection<GrouperPrivilege>, boolean)
>
> 
>
> 
>
> Thanks,
>
> Chris
>
>