Grouper Developers Forum

[Mark Weber <>]

FYI, I rebuilt the Subject API with JNDI Connection Pooling turned on 
and that had quite a significant effect on the xml-import.  Loading the 
1000, it went from 6 min down to 1 min 50 sec.

However, I still suspect that there is something j'ed up elsewhere with 
our config because the UI is still extremely slow in places.  (When I 
attempt to display the members of the group with a specific privilege)

I included the diff's in case others want to the Subject API with CP 
turned on.

I haven't done any real testing with this change except to load 1000 
subjects into grouper and verify that it's *not* opening and closing a 
connection for each query.  So... mileage may vary.  Also, I think you 
need jdk1.4+ to do the connection pooling.

--Mark

Mark Weber wrote:
> Here is a break down of the relevant pieces of our environments and 
> other info.
>
> (Note: I haven't run again with Ellen's suggestion of allocating more 
> memory to the jvm, but will do that asap)
>
>
> Development Environment:
> ------------------------
> Back-End:
>   Oracle 10g (4-CPU IBM AIX)
>
> Subject Store:
>   Sun ONE Directory Server v5.2 (UltraSPARC III solaris9)
>
> Grouper(tomcat):
>   jvm 1.5.0.06, tomcat5.5, (2.4Ghz p4, Ubuntu 6.06)
>
> Grouper(WebLogic):
>   BEA WebLogic 9.2 (2-way UltraSPARC III, solaris9)
>
>
> We have roughly 200,000 subjects in the subject store for test.  We will 
> likely have 250,000 subjects on a full production load.  For now we are 
> loading small basic groups that are very shallow.
>
> Loading 2 groups of a 1000 each for emp's and students.
>
> uwmsn:[employees1k]
> uwmsn:[students1k]
>
> In this batch, we are giving every entry, every privilege (admin, read, 
> view, update, optin, optout)  We intend to mix things up and try deeply 
> nested groups, fewer privileges etc.  But first, we want to solve the 
> slowness issues.
>
> sources.xml
> ------------
> <init-param>
>   <param-name>SubjectID_AttributeType</param-name>
>   <param-value>wiscEduPVI</param-value>
> </init-param>
>
> <init-param>
>   <param-name>Name_AttributeType</param-name>
>   <param-value>cn</param-value>
> </init-param>
>
> <init-param>
>   <param-name>Description_AttributeType</param-name>
>   <param-value>cn</param-value>
> </init-param>
>
>
> LDAP Indexes:
> -------------
>
> cn:          Equality, Presence, SubString
> WiscEduPVI:  Equality, Presence
>
>
> LDAP ACI's:
> ------------
> uid=gapi,ou=apps,o=isp:
>
> Rights:
>   read, compare, search, selfwrite, write
> Targets:
>   ou=people,o=wisctest.wisc.edu,o=isp
>
>
>
> Tom Barton wrote:
> > Mark,
> >
> > Although the jndi source adapter does not use connection pooling, your 
> > statement of problem makes me wonder if other issues are also at play 
> > here, ie, that these absurdly long times may be the result of more 
> > than just lack of connection pooling.
> >
> > Some particulars below...
> >
> > Mark Weber wrote:
> > > We've started doing some initial load testing and found that it's 
> > > pretty slow.  Our initial estimates put a full load of our population 
> > > into grouper, via the xml-import facility, would take about 8 days.
> >
> > Could you give some basic facts & stats - what back-end database, how 
> > large its overall allocation, about how many "large" groups & "small" 
> > groups in the load, how deeply nested, how many employ non-default 
> > privileges.
> >
> > > I think I've narrowed it down to the Subject API.  Specifically the 
> > > Subject API configured using JNDI/LDAP adapter.  For Instance, the UI 
> > > takes 74 seconds to load the " All Groups   - Current subjects with 
> > > [xyz] privilege" page.  It displays the first 50 members and creates 
> > > a connection, bind, search, result, unbind, and disconnect for each 
> > > person on the page.  Same with the xml-import.
> >
> > This is far from the fraction of a second I'm accustomed to seeing it 
> > take the jndi source adapter to display 50 subjects. I suspect there's 
> > another factor at play in this example, or perhaps your ldap server 
> > configuration (ACLs, authentication mechanism, other unknown 
> > operational params) might produce substantially different connection 
> > setup and query response times compared to the rather vanilla config 
> > of the openldap server I use for prototyping.
> >
> > > So, is there some configuration option that allows for connection 
> > > pooling within the subject API when it's using LDAP?
> >
> > Not in this version.
> >
> > > This is an assumption, but if we used the jdbc source adapter 
> > > instead, would it use connection pooling and thereby eliminate the 
> > > problem altogether?
> >
> > I think it does indeed use connection pooling.
> >
> > Tom

--
.Mark G. Weber----------------------Division of Information Technology.
|
            University Of Wisconsin - Madison|
|608/263.8945                               Room 3159 Computer Science|
`---------------------------------------------------------------------'
Index: JNDISourceAdapter.java
===================================================================
RCS file: 
/home/cvs/i2mi/subject/src/edu/internet2/middleware/subject/provider/JNDISourceAdapter.java,v
retrieving revision 1.5
diff -u -8 -p -r1.5 JNDISourceAdapter.java
--- JNDISourceAdapter.java      27 Jun 2006 22:33:51 -0000      1.5
+++ JNDISourceAdapter.java      2 Oct 2006 21:28:06 -0000
@@ -207,16 +207,17 @@ public class JNDISourceAdapter
          * Setup environment.
          * @throws SourceUnavailableException
          */
         protected void setupEnvironment(Properties props)
         throws SourceUnavailableException {
             
             environment.put(Context.INITIAL_CONTEXT_FACTORY,  
props.getProperty("INITIAL_CONTEXT_FACTORY") );
             environment.put(Context.PROVIDER_URL, 
props.getProperty("PROVIDER_URL") );
+            environment.put("com.sun.jndi.ldap.connect.pool", 
props.getProperty("POOL_CONNECTIONS") );
             environment.put(Context.SECURITY_AUTHENTICATION, 
props.getProperty("SECURITY_AUTHENTICATION"));
             environment.put(Context.SECURITY_PRINCIPAL, 
props.getProperty("SECURITY_PRINCIPAL") );
             
environment.put(Context.SECURITY_CREDENTIALS,props.getProperty("SECURITY_CREDENTIALS"));
             if (props.getProperty("SECURITY_PROTOCOL")!= null) {
                 environment.put(Context.SECURITY_PROTOCOL, "ssl");
             }
             Context context = null;
             try {
Index: sources.xml
===================================================================
RCS file: /home/cvs/i2mi/subject/conf/sources.xml,v
retrieving revision 1.5
diff -u -8 -p -r1.5 sources.xml
--- sources.xml 8 Jun 2006 00:13:01 -0000       1.5
+++ sources.xml 2 Oct 2006 21:30:23 -0000
@@ -138,16 +138,20 @@ $Id: sources.xml,v 1.5 2006/06/08 00:13:
       <param-name>INITIAL_CONTEXT_FACTORY</param-name>
       <param-value>com.sun.jndi.ldap.LdapCtxFactory</param-value>
     </init-param>
     <init-param>
       <param-name>PROVIDER_URL</param-name>
       <param-value>ldap://localhost:389</param-value>
     </init-param>
     <init-param>
+      <param-name>POOL_CONNECTIONS</param-name>
+      <param-value>true</param-value>
+    </init-param>
+    <init-param>
       <param-name>SECURITY_AUTHENTICATION</param-name>
       <param-value>simple</param-value>
     </init-param>
     <init-param>
       <param-name>SECURITY_PRINCIPAL</param-name>
       <param-value>cn=Manager,dc=example,dc=edu</param-value>
     </init-param>
     <init-param>

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature