Grouper Users - Open Discussion List

[Julio Polo <>]

I would disable idle timeouts in the load balancer and the LDAP server.   It's OK to have timeouts for LDAP queries in both ends, client and server.   If there is no way to turn off the load balancer's idle timeout, can you configure it to 24 hours and ensure that you query a Grouper LDAP subject at least once a day?  Adjust the intervals accordingly to the maximum idle timeout value for your appliance.

I'm not familiar with Grouper's LDAP connection pool validation, but anything that can help remove a zombie connection before it's used can only help.   The validation would have to be a high-level validation, ie make a subject query that returns something without timing out.

-julio

On Wed, Dec 7, 2016 at 5:13 AM, Wessel, Keith <> wrote:

Thanks, Julio. We do, in fact, have a load balancer in the mix, too. If I’m going to tweak idle timeouts on the LDAP server or load balancer, what should my max idle timeout be? I assume shorter than the pool expiration time – is that 10 minutes by default?

 

Similarly, is pool validation something that would help solve this problem? I don’t believe we’ve configured it, and I would assume it’s not configured by default since the validation check is specific to our environment.

 

Thanks,

Keith

 

 

From: Julio Polo [mailto:]
Sent: Tuesday, December 06, 2016 6:50 PM
To: Wessel, Keith <>
Cc:
Subject: Re: [grouper-users] Subject source LDAP timeouts

 

If the logs for your LDAP server don't reveal anything related to this timeout, check if there's a firewall or load balancer between Grouper and the network interface associated with the LDAP fqdn you configured Grouper with.

When it comes to persistent connections, we've experienced weird things when such network appliances and idle timeouts are involved.   The connection will appear to be fine from the client side, but there is nothing passing through these appliances onto the server.   Grouper's LDAP connection pooling wouldn't see anything wrong with the connection, but the client request isn't making it through the firewall or load balancer.   You can try tweaking the idle settings on these appliances (we've only had partial success) or you could specify an address that bypasses those appliances (but you'd lose the load balancing and every other benefit that comes with those appliances).

 

-julio

Julio Polo
Enterprise Middleware, Identity and Access Management
Information Technology Services
University of Hawaii

On Tue, Dec 6, 2016 at 9:53 AM, Wessel, Keith <> wrote:

Hi,

We've been seeing LDAP connection read timeout exceptions in our Grouper dev and text environments. (We don't have production running quite yet.) They're somewhat random in terms of when they happen, but sometimes we see them if someone leaves a Grouper UI session idle in their browser for more than a half hour (give or take). Other times, they happen when an environment hasn't been accessed in a while.

I'm blaming connecton pooling, but I'm unclear why a stale connection from the pool would cause this to happen. After all, if our LDAP server closed an idle connection, wouldn't' the pool just create a new one? And is the pool really handing out stale connections?

The LDAP server, for what it's worth, as an IBM Security Systems Directory Server.

Any advice would be appreciated. The Java stack trace is below.

Thanks,
Keith

2016-12-06 13:00:51,534: [TP-Processor3] ERROR GrouperUiFilter.doFilter(1030) -  - UI error
edu.internet2.middleware.subject.SourceUnavailableException: Ldap NamingException: LDAP response read timed out, timeout used:2000ms.,
Cant find subject from login id: ecc
        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResultsHelper(LdapSourceAdapter.java:774)
        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResults(LdapSourceAdapter.java:661)
        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapUnique(LdapSourceAdapter.java:806)
        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getSubject(LdapSourceAdapter.java:374)
        at edu.internet2.middleware.subject.provider.BaseSourceAdapter.getSubjectByIdOrIdentifier(BaseSourceAdapter.java:252)
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver$4.callLogic(SourcesXmlResolver.java:514)
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver$4.callLogic(SourcesXmlResolver.java:511)
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver$LogLabelCallable.call(SourcesXmlResolver.java:169)
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver.executeCallables(SourcesXmlResolver.java:230)
        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver.findByIdOrIdentifier(SourcesXmlResolver.java:520)
        at edu.internet2.middleware.grouper.subj.CachingResolver.findByIdOrIdentifier(CachingResolver.java:377)
        at edu.internet2.middleware.grouper.subj.ValidatingResolver.findByIdOrIdentifier(ValidatingResolver.java:203)
        at edu.internet2.middleware.grouper.SubjectFinder.findByIdOrIdentifier(SubjectFinder.java:315)
        at edu.internet2.middleware.grouper.ui.GrouperUiFilter.retrieveSubjectLoggedInHelper(GrouperUiFilter.java:347)
        at edu.internet2.middleware.grouper.ui.GrouperUiFilter.retrieveSubjectLoggedIn(GrouperUiFilter.java:297)
        at edu.internet2.middleware.grouper.ui.GrouperUiFilter.doFilter(GrouperUiFilter.java:1013)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:745)
Caused by: javax.naming.NamingException: LDAP response read timed out, timeout used:2000ms.; remaining name 'ou=people,dc=uiuc,dc=edu'
        at com.sun.jndi.ldap.Connection.readReply(Connection.java:483)
        at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:639)
        at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:562)
        at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1789)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:412)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:394)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:376)
        at edu.vt.middleware.ldap.AbstractLdap.search(AbstractLdap.java:215)
        at edu.vt.middleware.ldap.Ldap.search(Ldap.java:431)
        at edu.vt.middleware.ldap.Ldap.search(Ldap.java:347)
        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResultsHelper(LdapSourceAdapter.java:771)
        ... 30 more