Grouper Developers Forum

[Chris Hyzer <>]

I took a pass at solving these issues, and now I am seeing improved 
performance.  Note there are upgrade instructions listed below.

https://bugs.internet2.edu/jira/browse/GRP-268

Feel free to try it out (or review the code if you are a developer).

The end result is browsing a repository from the UI where there are lots of 
groups involved now takes a few hundred queries/results where it used to be 
thousands).  All unit tests pass.

1. You can list the members in a large group using paging and sorting (though 
we cant sort on anything meaningful quite yet, until 1.5)

2. Browsing the repository is now very fast since everything is done in one 
query.  Also, this logic was moved to the API from the UI so we can unit test 
it and reuse it.  Here is an example, get all the child groups directly in a 
stem where the current user in GrouperSession has at least one list 
membership, and also can either view/read/admin/optin/optout/update, 
sort the output by displayName, and get the first page of 50 results (takes 
less time to write the code than explain what it does :) ).

Set<Group> groups = stem.getChildMembershipGroups(Scope.SUB, 
AccessPrivilege.VIEW_PRIVILEGES, 
    new QueryOptions().sortAsc("displayName").paging(50, 1, true)); 


3. Composites are now querying for only memberships which will be added to 
the composite.  Before it was getting the left and right lists, and doing 
java calculations.  This is not good for intersecting a 60k member group with 
a small group

4. Privileges are refactored.  There is a BaseAccessAdapter which any 
external privilege implementation should extend (not just implement 
AccessAdapter).  This ensures that as we add optimizations usign the 
assumption that the GrouperAccessAdapter house information in the DB, 
external providers dont have to change their code.  Same thing for 
BaseNamingAdapter.  Right now there are only a couple of methods in there, 
here is an example:

  /**
   * 
   * @see 
edu.internet2.middleware.grouper.privs.AccessAdapter#hqlFilterGroupsWhereClause(edu.internet2.middleware.grouper.GrouperSession,
 edu.internet2.middleware.subject.Subject, 
edu.internet2.middleware.grouper.hibernate.HqlQuery, java.lang.StringBuilder, 
java.lang.String, java.util.Set)
   */
  public boolean hqlFilterGroupsWhereClause(GrouperSession grouperSession,
      Subject subject, HqlQuery hqlQuery, StringBuilder hql, String 
groupColumn, Set<Privilege> privInSet) {
    //by default dont change the HQL
    return false;
  }

  /**
   * 
   * @see 
edu.internet2.middleware.grouper.privs.AccessAdapter#postHqlFilterMemberships(edu.internet2.middleware.grouper.GrouperSession,
 edu.internet2.middleware.subject.Subject, java.util.Set)
   */
  public Set<Membership> postHqlFilterMemberships(
      GrouperSession grouperSession, Subject subject,
      Set<Membership> memberships) {

    return PrivilegeHelper.canViewMemberships(grouperSession, memberships);
  }


Note that in the Grouper adapter, we will change the query to filter for 
results, but other implementations can just filter after the resultset is 
returned...  If you dont want to use the integrated DB implementation, you 
can use GrouperNonDbAccessAdapter instead of GrouperAccessAdapter (configured 
in grouper.properties)

5. Some items which cause unnecessary queries were removed.  e.g. there was a 
group.getParentStem().getName() which was accessing the DB to get the stem, 
then getName.  but this can be calculated from the group's name, so I added a 
method which doesnt hit the DB: group.getParentStemName()

6. Some things were batched up.  e.g. if you are looping through memberships 
and calling getGroup on all of them, you can precalculate with 
Membership.retrieveGroups(Collection<Membership> memberships), then loop 
through and call getGroup() on all of them, and wont go back to the DB (only 
went once (well, N/50 times :) ).

7. Hibernate caching was turned on, and accesses to groups/stems/members by 
ID/Name were set to 10 second timeouts.  See the change log for upgrade 
instructions:

https://wiki.internet2.edu/confluence/display/GrouperWG/Grouper+change+log+v1.4

# 2009/4/25: v1.4.2 GROUPER_1_4_BRANCH:set this to true in 
grouper.hibernate.properties:
hibernate.cache.use_query_cache       = true
Compare and merge the new ehcache.example.xml with ehcache.xml.  There are 
stem/group/member caches defined

8. To get the logic of if a wheel group member is acting as admin or not, 
this was moved to GrouperSession

grouperSession.setConsiderIfWheelMember(activeWheelGroupMember);

If false, then it will be as if the user is not a wheel member...


> -----Original Message-----
> From: Chris Hyzer
> Sent: Tuesday, April 07, 2009 4:47 PM
> To: 
> ''
> Subject: RE: grouper 1.4 performance
> 
> I did these composite refactorings in the API.  Before if I made a
> requireInGroups composite (intersection) with a group of 40k members,
> it took 3 minutes on my PC with mysql.  Now it takes 2 seconds.
> 
> I tagged before I committed: GROUPER_1_4_20090407a
> 
> Shilen, if you want to review the code, that would be good.  (just diff
> branch GROUPER_1_4 with that tag).
> 
> All the composite unit tests work, I will kick them all off now.
> 
> Also, the #2 idea from that email (to join to the security tables for
> group listings) doesn't look like it is necessary... in my test env it
> looks quick now (due to my other change I guess).  Not really sure
> though...  will check.
> 
> Thanks,
> Chris
> 
> > #########################################
> > 3. making a composite (intersection or complement) of a small group
> and
> > a large group.  We have a use case where we like to intersect with
> > allEmployees or facultyStudentStaff, etc, and it takes a LONG time to
> > figure out a few members (since it selects all from each group)...  I
> > think we can do this all in one query.  Also, right now all columns
> in
> > the membership object are returned, then only the ID is used, so we
> > only need to select the ID.  Here is an example of an intersection,
> > complement, and union in one query each (again, proof of concepts in
> > the HibernateSessionTest class):
> >
> >     //intersection in one query
> >     List<String> memberUuids = HibernateSession.byHqlStatic()
> >       .createQuery("select distinct theMember.uuid from Member
> > theMember, " +
> >                     "Membership theMembership, Membership theMembership2,
> > Field theField " +
> >                     "where theMembership.ownerUuid = :group1uuid and
> > theMembership2.ownerUuid = :group2uuid " +
> >                     "and theMember.uuid = theMembership.memberUuid and
> > theMember.uuid = theMembership2.memberUuid " +
> >                     "and theMembership.fieldId = theField.uuid and
> > theMembership2.fieldId = theField.uuid " +
> >                     "and theField.typeString = 'list' and theField.name =
> > 'members'")
> >                     .setString("group1uuid", i2.getUuid())
> >                     .setString("group2uuid", i3.getUuid())
> >           .list(String.class);
> >
> >     assertEquals(1, memberUuids.size());
> >     assertEquals(MemberFinder.findBySubject(grouperSession,
> > SubjectTestHelper.SUBJ1).getUuid(), memberUuids.get(0));
> >
> >     //complement in one query
> >     memberUuids = HibernateSession.byHqlStatic()
> >     .createQuery("select distinct theMember.uuid from Member
> theMember,
> > " +
> >         "Membership theMembership, Field theField " +
> >         "where theMembership.ownerUuid = :group1uuid " +
> >         "and theMember.uuid = theMembership.memberUuid " +
> >         "and theMembership.fieldId = theField.uuid " +
> >         "and theField.typeString = 'list' and theField.name =
> 'members'
> > " +
> >         "and not exists (select theMembership2.memberUuid from
> > Membership theMembership2 " +
> >         "where theMembership2.memberUuid = theMember.uuid and
> > theMembership.fieldId = theField.uuid " +
> >         "and theMembership2.ownerUuid = :group2uuid) ")
> >         .setString("group1uuid", i3.getUuid())
> >         .setString("group2uuid", i2.getUuid())
> >         .list(String.class);
> >
> >     assertEquals(1, memberUuids.size());
> >     assertEquals(MemberFinder.findBySubject(grouperSession,
> > SubjectTestHelper.SUBJ4).getUuid(), memberUuids.get(0));
> >
> >     //union in one query (granted, this is less of an issue since you
> > need all data from both,
> >     //but might as well be consistent)
> >     memberUuids = HibernateSession.byHqlStatic()
> >     .createQuery("select distinct theMember.uuid from Member
> theMember,
> > " +
> >         "Membership theMembership, Membership theMembership2, Field
> > theField " +
> >         "where theMembership.ownerUuid = :group1uuid and
> > theMembership2.ownerUuid = :group2uuid " +
> >         "and (theMember.uuid = theMembership.memberUuid or
> > theMember.uuid = theMembership2.memberUuid) " +
> >         "and theMembership.fieldId = theField.uuid and
> > theMembership2.fieldId = theField.uuid " +
> >         "and theField.typeString = 'list' and theField.name =
> > 'members'")
> >         .setString("group1uuid", i2.getUuid())
> >         .setString("group2uuid", i3.getUuid())
> >         .list(String.class);
> >
> >     assertEquals(6, memberUuids.size());