Grouper Developers Forum

["Jessica Bibbee" <>]

Grouper Working Group Meeting
July 25, 2007

 

*Attendees*
Tom Barton, U. Chicago (chair)
Blair Christensen, U. Chicago

Gary Brown, U. Bristol

Dave Donnelly, Stanford U.

Shilen Patel, Duke U.

Celeste Copeland, Duke U.

Joy Veronneau, Cornell U.
Michael Gettes, Internet2

Jessica Bibbee, Internet2 (scribe)

 

New *Action Items*

[AI] {Celeste and Shilen} will share results of their investigation related to query issues, pointing either to methods or underlying code.

[AI] {Celeste and Shilen} will share profiling information regarding group search.

[AI] {Blair} will look into batching of SQL.

[AI] {Tom} will update his list of Grouper priorities to resemble discussion on the Working Group call, including next steps and who will work on which items.

Carry-over *Action Items*

[AI] {Blair} will address performance issues related to proposed method signatures for supporting browsing from {Gary}.  (11-Jul-07)

 

Future *Agenda Items*

- Roadmap updating. Step 1 - review existing docs:

    . Priorities for Functional Enhancements (Aug 2006): < https://wiki.internet2.edu/confluence/x/NBk >
    . Grouper Product Backlog (July 2007): < https://wiki.internet2.edu/confluence/x/dzs >

*Agenda*

1. Sequence and assignment of next steps to address performance issues

The technical issues underlying some of the functional performance issues raised by Duke and Brown are becoming clearer. Here's a table identifying potential work items and the issues they would address. Let's try to make this table more accurate, determine next steps, and who should do what.

a. Browsing My Memberships or Manage Groups
     . [GRP-7] Add methods for finding child stems and groups where a Subject has the specified access or naming privileges
    . modify UI to take advantage of these
     . [GRP-10] Modify Membership indices based upon recommendations from Duke

b. Listing group members is slow
     . improve JNDISourceAdapter performance
     . move certain privilege checking from UI into API
     . [GRP-6] API Subject caching no longer caching everything that it should

c. Ldappc performance
     . improve JNDISourceAdapter performance

     . [GRP-12] Poor integration with Ldappc: subordinate-stem-query causes a full table scan
     . [GRP-13] Poor integration with Ldappc: attribute-matching-queries causes a full table scan
     . [GRP-14] Poor integration with Ldappc: Null DTO error message causes Ldappc  runs to fail
     . modify Ldappc to take advantage of these
     . modify Ldappc to cache more group data

d. Group search
     . [GRP-9] A number of methods for retrieving groups by attribute cause full table scans.  Document methods where this is the situation and provide alternates where possible.
     . modify UI to default to equality-matched & scoped searching

e. TAAdmins using Manage Groups
     . subclass PrepareMenuAction to omit Manage Groups for TAAdmins

f. Add member to TAAdmins takes hours
     . perform this inherently slow operation in a separate JVM, eg, by running a gsh script

g. XML import/export doesn't scale
     . improve JNDISourceAdapter performance
     . profile XMLReader and XMLImporter to see where time and space is being spent

h. Variant Subject presentation depending on user and Subject's FERPA status

*Discussion*

 The call's agenda is detailed with a list of narrowly focused performance issues; to see the original email, c.f. {Tom's} thread on the grouper-dev mailing list: < https://mail.internet2.edu/wws/arc/grouper-dev/2007-07/msg00067.html >. The Group discussed the best sequence of steps to address, the next steps, and who would work on which items.

 

The Group agreed that a focus on [GRP-12] ought to be sooner than later; it would assist in browsing My Memberships.

 

{Tom} discussed future work on the Subject API and JNDISourceAdapter performance. Regarding the Subject API, {Blair} said that it is not caching all that should be. {Michael} asked if it was covered by GH cache, and {Blair} responded by saying that it was intended to, though currently is not. Subject caching will improve LDAPPC performance.

 

Concerns around Group Search stem from the fact that a search performs full table scans, making it slow. {Tom} discussed the idea of designing UI workflow to favor, i.e., make easier, select searches to support indices. Duke is more concerned with changing the way batch queries could be performed.

 

{Tom} asked {Celeste} if she thought that the slowness of Group Search was related to the grouping of SQL query. She thought it might be more related to how the API is implemented, i.e., queries happening at a low level, as opposed to a batch mechanism. She was under the impression that it would be good to take a closer look at the design of how the API interacts with the database. [AI] {Blair} will look into batching of SQL.

 

[AI] {Celeste and Shilen} will share results of their investigation related to query issues, pointing either to methods or underlying code.

[AI] {Celeste and Shilen} will share profiling information regarding group search.

 

Duke is using a workaround to temporarily help them manage TAAdmins, a small group needing high administrative privileges. Import/export and scaling were two issues that {Tom} mentioned.

 

{Shilen} also expressed a desire to have another way of dealing with a null displayName value. Currently, when the displayName is returned as null, an exception is thrown. More ideally, a message would be displayed to inform of the null displayName. {Michael} suggested that another approach would be to use a plugin, implemented on the directory side, which would return a value of choice is someone is without necessary privileges.

 

[AI] {Tom} will update his list of Grouper priorities to resemble discussion on the Working Group call, including next steps and who will work on which items. Changes to the above agenda stemming from the discussion can be found on the Grouper wiki page: Performance Issues – Next Steps < https://wiki.internet2.edu:443/confluence/x/Zz4 >. {Tom} has asked for feedback.

The next Grouper Working Group conference call will be held on Wednesday, August 8, 2007 at 12pm EDT.

 

-- 
Jessica Bibbee, Technical Analyst
Internet2

mobile: +1-734-255-6644

Innovate, participate, educate.
The  Fall 2007 Internet2 Member Meeting Call for Participation
http://events.internet2.edu/2007/fall-mm/calls.cfm