Grouper Users - Open Discussion List

["Wessel, Keith" <>]

Following up on this. Thanks for the advice so far, Julio.

 

The load balancer may be the problem, but we haven’t found an easy way to change that yet, and we’re really hoping we can fix it with LDAP connection pooling instead.

 

I’ve added the following two settings to ldap-sources.properties:

 

edu.vt.middleware.ldap.pool.pruneTimerPeriod = 60000

edu.vt.middleware.ldap.pool.expirationTime=600000

 

What I was expecting to see as a result of this, with a minPoolSize of 2, is new LDAP connections being created every 10 minutes or so. I see no such thing in our logs, though. I see two connections being created on Tomcat start-up, then I see no new connections at all until the error occurs.

 

A sequence looks like this in the LDAP logs:

At 12:04, I start Tomcat, and two connections are created. It uses one when I log in on the web interface a minute later.

I then let my browser sit idle.

At 12:30, my LDAP server logs report an unbind for both connections.

When I click a link in the browser at 1:02, I get an error that my subject couldn’t be retrieved, and I get the error I included in my original post to this thread.

 

The GSLB inactivity timeout is 30 minutes, not 26. So, while that could be the issue, I’m not sure.

 

But the fact is my pool doesn’t seem to be working as I would think it would. Is there something I need to do to enable pooling that I haven’t done? Far as I can tell, with the two initial connections, pooling’s working on start-up. It’s just not purging and expiring connections.

 

Any thoughts appreciated.

 

Keith

 

From: Julio Polo [mailto:]
Sent: Wednesday, December 07, 2016 7:13 PM
To: Wessel, Keith <>
Cc:
Subject: Re: [grouper-users] Subject source LDAP timeouts

 

I would disable idle timeouts in the load balancer and the LDAP server.   It's OK to have timeouts for LDAP queries in both ends, client and server.   If there is no way to turn off the load balancer's idle timeout, can you configure it to 24 hours and ensure that you query a Grouper LDAP subject at least once a day?  Adjust the intervals accordingly to the maximum idle timeout value for your appliance.

I'm not familiar with Grouper's LDAP connection pool validation, but anything that can help remove a zombie connection before it's used can only help.   The validation would have to be a high-level validation, ie make a subject query that returns something without timing out.

-julio

 

On Wed, Dec 7, 2016 at 5:13 AM, Wessel, Keith <> wrote:

Thanks, Julio. We do, in fact, have a load balancer in the mix, too. If I’m going to tweak idle timeouts on the LDAP server or load balancer, what should my max idle timeout be? I assume shorter than the pool expiration time – is that 10 minutes by default?

 

Similarly, is pool validation something that would help solve this problem? I don’t believe we’ve configured it, and I would assume it’s not configured by default since the validation check is specific to our environment.

 

Thanks,

Keith

 

 

From: Julio Polo [mailto:]
Sent: Tuesday, December 06, 2016 6:50 PM
To: Wessel, Keith <>
Cc:
Subject: Re: [grouper-users] Subject source LDAP timeouts

 

If the logs for your LDAP server don't reveal anything related to this timeout, check if there's a firewall or load balancer between Grouper and the network interface associated with the LDAP fqdn you configured Grouper with.

When it comes to persistent connections, we've experienced weird things when such network appliances and idle timeouts are involved.   The connection will appear to be fine from the client side, but there is nothing passing through these appliances onto the server.   Grouper's LDAP connection pooling wouldn't see anything wrong with the connection, but the client request isn't making it through the firewall or load balancer.   You can try tweaking the idle settings on these appliances (we've only had partial success) or you could specify an address that bypasses those appliances (but you'd lose the load balancing and every other benefit that comes with those appliances).

 

-julio

Julio Polo
Enterprise Middleware, Identity and Access Management
Information Technology Services
University of Hawaii

On Tue, Dec 6, 2016 at 9:53 AM, Wessel, Keith <> wrote:

Hi,

We've been seeing LDAP connection read timeout exceptions in our Grouper dev and text environments. (We don't have production running quite yet.) They're somewhat random in terms of when they happen, but sometimes we see them if someone leaves a Grouper UI session idle in their browser for more than a half hour (give or take). Other times, they happen when an environment hasn't been accessed in a while.

I'm blaming connecton pooling, but I'm unclear why a stale connection from the pool would cause this to happen. After all, if our LDAP server closed an idle connection, wouldn't' the pool just create a new one? And is the pool really handing out stale connections?

The LDAP server, for what it's worth, as an IBM Security Systems Directory Server.

Any advice would be appreciated. The Java stack trace is below.

Thanks,
Keith

2016-12-06 13:00:51,534: [TP-Processor3] ERROR GrouperUiFilter.doFilter(1030) -  - UI error
edu.internet2.middleware.subject.SourceUnavailableException: Ldap NamingException: LDAP response read timed out, timeout used:2000ms.,
Cant find subject from login id: ecc
        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResultsHelper(LdapSourceAdapter.java:774)
        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResults(LdapSourceAdapter.java:661)
        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapUnique(LdapSourceAdapter.java:806)
        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getSubject(LdapSourceAdapter.java:374)
        at edu.internet2.middleware.subject.provider.BaseSourceAdapter.getSubjectByIdOrIdentifier(BaseSourceAdapter.java:252)
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver$4.callLogic(SourcesXmlResolver.java:514)
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver$4.callLogic(SourcesXmlResolver.java:511)
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver$LogLabelCallable.call(SourcesXmlResolver.java:169)
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver.executeCallables(SourcesXmlResolver.java:230)
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver.findByIdOrIdentifier(SourcesXmlResolver.java:520)
        at edu.internet2.middleware.grouper.subj.CachingResolver.findByIdOrIdentifier(CachingResolver.java:377)
        at edu.internet2.middleware.grouper.subj.ValidatingResolver.findByIdOrIdentifier(ValidatingResolver.java:203)
        at edu.internet2.middleware.grouper.SubjectFinder.findByIdOrIdentifier(SubjectFinder.java:315)
        at edu.internet2.middleware.grouper.ui.GrouperUiFilter.retrieveSubjectLoggedInHelper(GrouperUiFilter.java:347)
        at edu.internet2.middleware.grouper.ui.GrouperUiFilter.retrieveSubjectLoggedIn(GrouperUiFilter.java:297)
        at edu.internet2.middleware.grouper.ui.GrouperUiFilter.doFilter(GrouperUiFilter.java:1013)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:745)
Caused by: javax.naming.NamingException: LDAP response read timed out, timeout used:2000ms.; remaining name 'ou=people,dc=uiuc,dc=edu'
        at com.sun.jndi.ldap.Connection.readReply(Connection.java:483)
        at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:639)
        at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:562)
        at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1789)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:412)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:394)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:376)
        at edu.vt.middleware.ldap.AbstractLdap.search(AbstractLdap.java:215)
        at edu.vt.middleware.ldap.Ldap.search(Ldap.java:431)
        at edu.vt.middleware.ldap.Ldap.search(Ldap.java:347)
        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResultsHelper(LdapSourceAdapter.java:771)
        ... 30 more