Grouper Users - Open Discussion List

[Tom Barton <>]

You've clearly demonstrated that the combination of JNDISourceAdapter +
grouper API overwhelms your ldap directory's ability to handle
connections. A maintenance release of grouper is being prepared that
will include several performance related improvements, including
resumption and improvement of subject caching. That should reduce the
number of times that a connection to ldap is created to look something
up about a subject.

However, it's hard to say exactly how much a given caching strategy will
impact a given application. And an ldap directory should be capable of
handling a very high volume of read activity. So, although the next
maintenance release might offer some relief, it will still need to do
lots of reading from ldap (and writing, in the case of provisioning
group info to it). You might want to look into how your directory server
is configured, in terms of indices, query caches, and connection
handling. And possibly also look into how the TCP stack on the
underlying host is configured to see how long it takes to expire an
entry from the TCP connections table.

Allen Chen wrote:
>         Now I make some progress on the ldap grouper provisioning failure 
> problem. The error lies in grouper SubjectFinder.
>         When I use the following code to try to get all the subject id from 
> all the groups, the cycle failed in mid way with an exception.
> ----------------------------------------------------------------------------------
>                    s = GrouperSession.start(root);
>                    // get the groups in the specified stem
>    
> 
>                   Stem rootStem = StemFinder.findRootStem(s);
>                    GrouperQuery gq = GrouperQuery.createQuery(s, new 
> GroupNameFilter(StemName, rootStem));
>                    System.out.println("groups " + gq.getGroups().size());
>                    Set s1 = gq.getGroups();
>                    for (Iterator it = s1.iterator(); it.hasNext();) {
>                                int count = 0;
>                                Group g = (Group) it.next();
> 
>                                System.out.println(g.getName() + " "+ 
> g.getMembers().size());
> 
>                                Set set = g.getMembers();
>                                list.clear();
>                                 for (Iterator it2 = set.iterator(); 
> it2.hasNext();) {
>                                         Member member = (Member) it2.next();
>                                         Subject sub = member.getSubject();
>                                         System.out.println(++count+ " 
> "+sub.getId()+" "+sub.getName());
>                                         list.add(member.getSubjectId());
>                 }
>        }
> 
>           s.stop();
> --------------------------------------------------------------------------
> In the midway of the cycle , an exception is thrown:
>               .
>               .(subject's infomation omited)
>               .
>        edu.internet2.middleware.subject.SubjectNotFoundException: No 
> results: searchSubject filter:(& (uid=%TERM%) (objectclass=person)) 
> searchValue: yijianhui
>        at 
> edu.internet2.middleware.subject.provider.JNDISourceAdapter.getLdapUnique(Unknown
>  Source)
>         at 
> edu.internet2.middleware.subject.provider.JNDISourceAdapter.getSubject(Unknown
>  Source)
>         at 
> edu.internet2.middleware.grouper.SubjectFinder.findById(SubjectFinder.java:208)
>         at 
> edu.internet2.middleware.grouper.Member.getSubject(Member.java:461)
>         at 
> cn.edu.scut.grouper.ladpImport.TestGrouperUtil.getGroups(TestGrouperUtil.java:132)
>         at 
> cn.edu.scut.grouper.ladpImport.TestGrouperUtil.main(TestGrouperUtil.java:163)
> 
> In fact, people with uid "yijianhui" exists in ldap server.          
> In the source JNDISourceAdapter.java of the package 
> edu.internet2.middleware.subject, the getLdapResults method connect server 
> every time  SubjectFinder.findById is called.
> try  {
>             context = new InitialDirContext(environment);
>             SearchControls constraints = new SearchControls( );
>             constraints.setSearchScope( scopeNum );
>             constraints.setReturningAttributes(attributeNames);
>             results = context.search( base, filter, constraints );
>         } catch ( AuthenticationException ex ) {
>             log.error("Ldap Authentication Exception: " + ex.getMessage(), 
> ex);
>         } catch (NamingException ex) {
>             log.error("Ldap NamingException: " + ex.getMessage(), ex);
>             
>         } finally {
>             if (context != null) {
>                 try {
>                     context.close();
>                 } catch (NamingException ne) {
>                     // squelch, since it is already closed
>                 }
>             }
>         }
> 
> I guess whether the fact that the server cannot stand the program's 
> connection's frequently connecting and closeing made the above cycle 
> failed, for some groups have a maximum number of 5000 members.
> Did the program's connection's frequently connecting and closeing make the 
> ldap server refuse to be connected again? Perhaps this is why the ldappc 
> failed to add member attributes to group entry. 
> 
> And I checked the grouper_error.log, found the following error:
> 2007-08-10 09:23:18,671: Ldap NamingException: 192.168.78.16:399
> javax.naming.CommunicationException: 192.168.78.16:399 [Root exception is 
> java.net.ConnectException: Connection timed out: connect]
>  at com.sun.jndi.ldap.Connection.<init>(Connection.java:207)
>  at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:118)
>  at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1580)
>  at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2616)
>  at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:287)
>  at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:175)
>  at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:193)
>  at 
> com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:136)
>  at 
> com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:66)
>  at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
>  at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288)
>  at javax.naming.InitialContext.init(InitialContext.java:223)
>  at javax.naming.InitialContext.<init>(InitialContext.java:197)
>  at 
> javax.naming.directory.InitialDirContext.<init>(InitialDirContext.java:82)
>  at 
> edu.internet2.middleware.subject.provider.JNDISourceAdapter.getLdapResults(Unknown
>  Source)
>  at 
> edu.internet2.middleware.subject.provider.JNDISourceAdapter.getLdapUnique(Unknown
>  Source)
>  at 
> edu.internet2.middleware.subject.provider.JNDISourceAdapter.getSubject(Unknown
>  Source)
>  at 
> edu.internet2.middleware.grouper.SubjectFinder.findById(SubjectFinder.java:208)
>  at edu.internet2.middleware.grouper.Member.getSubject(Member.java:461)
>  at 
> cn.edu.scut.grouper.ladpImport.TestGrouperUtil.getGroups(TestGrouperUtil.java:132)
>  at 
> cn.edu.scut.grouper.ladpImport.TestGrouperUtil.main(TestGrouperUtil.java:163)
> Caused by: java.net.ConnectException: Connection timed out: connect
>  at java.net.PlainSocketImpl.socketConnect(Native Method)
>  at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>  at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>  at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>  at java.net.Socket.connect(Socket.java:519)
>  at java.net.Socket.connect(Socket.java:469)
>  at java.net.Socket.<init>(Socket.java:366)
>  at java.net.Socket.<init>(Socket.java:179)
>  at com.sun.jndi.ldap.Connection.createSocket(Connection.java:349)
>  at com.sun.jndi.ldap.Connection.<init>(Connection.java:184)
>  ... 20 more
> 
> 
> 
> 
> 
> 
> 
> 
> Allen Chen
> 2007-08-10
> 
> 
> 
> 发件人： Tom Barton
> 发送时间： 2007-08-08 10:42:46
> 收件人： Allen Chen
> 抄送： 
> 
> 主题： Re: [grouper-users] Ldap import data from grouper fail
> 
> 
> 
> Allen Chen wrote:
>>   >Is there a limited pool of port numbers higher than 1024 available on 
>>   >the machine running ldappc? What OS is the ldappc host running?
>> The ldappc host is running on windows. The 
>> port numbers higher than 1024 is available on the pc I worked on.
>>  
>> In fact, what makes me confused is the following error:
>>  
>> Ldap NamingException: 192.168.23.11:389
>>   > > javax.naming.CommunicationException: 192.168.23.11:389[Root 
>> exception is 
>>   > > java.net.BindException: Address already in use: connect]
>>  
>> where 192.168.23.11:389 is the address of ldap server.Is there 
>> something wrong with the server?
> 
> Possibly. I don't know whether this error reflects inability to form a
> connection because the server (your LDAP server) refuses to accept
> another connection, or because the client has no more open ports
> available. A quick peruse on the net for the deeper meaning of this java
> error message does not indicate which of the two possibilities is the
> cause of this message.
> 
> There are several ways to debug this. I'd probably use a protocol
> analyzer like ethereal running on the client to look at the TCP
> connection handshake - that should clearly identify which end, server or
> client, is the source of the error condition. If you have access to the
> LDAP server's LDAP logs, and if those logs record sufficient detail, you
> might be able to tell that way.
> 
> Tom