Grouper Users - Open Discussion List

["Wessel, Keith" <>]

I actually haven't set up validation queries yet. I was starting simple. Any 
idea if expirationTime is independent of validation? I assumed it was.

I agree; it sounds like pooling isn't functioning, or at least not 
functioning right.

Keith


-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Tom Poage
Sent: Thursday, December 15, 2016 4:34 PM
To: 

Subject: Re: [grouper-users] Subject source LDAP timeouts

The behavior suggests to me that connection pool validation is not being 
performed.

E.g. our CAS and Shibboleth services perform connection validation every 5 
minutes, with a prune check every 10 minutes. Our LDAP infrastructure closes 
idle connections at one hour, so LDAP connections from these services are (in 
theory) never forcibly closed, and the base number of connections stay open 
in perpetuity (Shibboleth and CAS use ldaptive, vt-ldap being its ancestor). 
If a connection (validation test) fails, new pool connections are attempted. 
Anyhow, CAS and Shib list archives may help.

Tom.

> On Dec 15, 2016, at 2:09 PM, Wessel, Keith 
> <>
>  wrote:
> 
> Yes.
>  
> Other thoughts?
>  
> Keith
>  
>  
> From: Hyzer, Chris 
> [mailto:]
>  
> Sent: Thursday, December 15, 2016 3:08 PM
> To: Wessel, Keith 
> <>;
>  
> 
> Subject: RE: [grouper-users] Subject source LDAP timeouts
>  
> Do you have this in the sources.xml for that source?
>  
> <init-param>
>       <param-name>ldapProperties_file</param-name>
>       <param-value>ldap-sources.properties</param-value>
>     </init-param>
>     
> Thanks
> Chris
>  
>  
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Wessel, Keith
> Sent: Thursday, December 15, 2016 3:03 PM
> To: 
> 
> Subject: RE: [grouper-users] Subject source LDAP timeouts
>  
> Following up on this. Thanks for the advice so far, Julio.
>  
> The load balancer may be the problem, but we haven’t found an easy way to 
> change that yet, and we’re really hoping we can fix it with LDAP connection 
> pooling instead.
>  
> I’ve added the following two settings to ldap-sources.properties:
>  
> edu.vt.middleware.ldap.pool.pruneTimerPeriod = 60000
> edu.vt.middleware.ldap.pool.expirationTime=600000
>  
> What I was expecting to see as a result of this, with a minPoolSize of 2, 
> is new LDAP connections being created every 10 minutes or so. I see no such 
> thing in our logs, though. I see two connections being created on Tomcat 
> start-up, then I see no new connections at all until the error occurs.
>  
> A sequence looks like this in the LDAP logs:
> At 12:04, I start Tomcat, and two connections are created. It uses one when 
> I log in on the web interface a minute later.
> I then let my browser sit idle.
> At 12:30, my LDAP server logs report an unbind for both connections.
> When I click a link in the browser at 1:02, I get an error that my subject 
> couldn’t be retrieved, and I get the error I included in my original post 
> to this thread.
>  
> The GSLB inactivity timeout is 30 minutes, not 26. So, while that could be 
> the issue, I’m not sure.
>  
> But the fact is my pool doesn’t seem to be working as I would think it 
> would. Is there something I need to do to enable pooling that I haven’t 
> done? Far as I can tell, with the two initial connections, pooling’s 
> working on start-up. It’s just not purging and expiring connections.
>  
> Any thoughts appreciated.
>  
> Keith
>  
> From: Julio Polo 
> [mailto:]
>  
> Sent: Wednesday, December 07, 2016 7:13 PM
> To: Wessel, Keith 
> <>
> Cc: 
> 
> Subject: Re: [grouper-users] Subject source LDAP timeouts
>  
> I would disable idle timeouts in the load balancer and the LDAP server.   
> It's OK to have timeouts for LDAP queries in both ends, client and server.  
>  If there is no way to turn off the load balancer's idle timeout, can you 
> configure it to 24 hours and ensure that you query a Grouper LDAP subject 
> at least once a day?  Adjust the intervals accordingly to the maximum idle 
> timeout value for your appliance.
> 
> I'm not familiar with Grouper's LDAP connection pool validation, but 
> anything that can help remove a zombie connection before it's used can only 
> help.   The validation would have to be a high-level validation, ie make a 
> subject query that returns something without timing out.
> 
> -julio
>  
> On Wed, Dec 7, 2016 at 5:13 AM, Wessel, Keith 
> <>
>  wrote:
> Thanks, Julio. We do, in fact, have a load balancer in the mix, too. If I’m 
> going to tweak idle timeouts on the LDAP server or load balancer, what 
> should my max idle timeout be? I assume shorter than the pool expiration 
> time – is that 10 minutes by default?
>  
> Similarly, is pool validation something that would help solve this problem? 
> I don’t believe we’ve configured it, and I would assume it’s not configured 
> by default since the validation check is specific to our environment.
>  
> Thanks,
> Keith
>  
>  
> From: Julio Polo 
> [mailto:]
>  
> Sent: Tuesday, December 06, 2016 6:50 PM
> To: Wessel, Keith 
> <>
> Cc: 
> 
> Subject: Re: [grouper-users] Subject source LDAP timeouts
>  
> If the logs for your LDAP server don't reveal anything related to this 
> timeout, check if there's a firewall or load balancer between Grouper and 
> the network interface associated with the LDAP fqdn you configured Grouper 
> with. 
> 
> When it comes to persistent connections, we've experienced weird things 
> when such network appliances and idle timeouts are involved.   The 
> connection will appear to be fine from the client side, but there is 
> nothing passing through these appliances onto the server.   Grouper's LDAP 
> connection pooling wouldn't see anything wrong with the connection, but the 
> client request isn't making it through the firewall or load balancer.   You 
> can try tweaking the idle settings on these appliances (we've only had 
> partial success) or you could specify an address that bypasses those 
> appliances (but you'd lose the load balancing and every other benefit that 
> comes with those appliances).
>  
> -julio
> 
> Julio Polo
> Enterprise Middleware, Identity and Access Management
> Information Technology Services
> University of Hawaii
> 
> On Tue, Dec 6, 2016 at 9:53 AM, Wessel, Keith 
> <>
>  wrote:
> Hi,
> 
> We've been seeing LDAP connection read timeout exceptions in our Grouper 
> dev and text environments. (We don't have production running quite yet.) 
> They're somewhat random in terms of when they happen, but sometimes we see 
> them if someone leaves a Grouper UI session idle in their browser for more 
> than a half hour (give or take). Other times, they happen when an 
> environment hasn't been accessed in a while.
> 
> I'm blaming connecton pooling, but I'm unclear why a stale connection from 
> the pool would cause this to happen. After all, if our LDAP server closed 
> an idle connection, wouldn't' the pool just create a new one? And is the 
> pool really handing out stale connections?
> 
> The LDAP server, for what it's worth, as an IBM Security Systems Directory 
> Server.
> 
> Any advice would be appreciated. The Java stack trace is below.
> 
> Thanks,
> Keith
> 
> 2016-12-06 13:00:51,534: [TP-Processor3] ERROR 
> GrouperUiFilter.doFilter(1030) -  - UI error
> edu.internet2.middleware.subject.SourceUnavailableException: Ldap 
> NamingException: LDAP response read timed out, timeout used:2000ms.,
> Cant find subject from login id: ecc
>         at 
> edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResultsHelper(LdapSourceAdapter.java:774)
>         at 
> edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResults(LdapSourceAdapter.java:661)
>         at 
> edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapUnique(LdapSourceAdapter.java:806)
>         at 
> edu.internet2.middleware.subject.provider.LdapSourceAdapter.getSubject(LdapSourceAdapter.java:374)
>         at 
> edu.internet2.middleware.subject.provider.BaseSourceAdapter.getSubjectByIdOrIdentifier(BaseSourceAdapter.java:252)
>         at 
> edu.internet2.middleware.grouper.subj.SourcesXmlResolver$4.callLogic(SourcesXmlResolver.java:514)
>         at 
> edu.internet2.middleware.grouper.subj.SourcesXmlResolver$4.callLogic(SourcesXmlResolver.java:511)
>         at 
> edu.internet2.middleware.grouper.subj.SourcesXmlResolver$LogLabelCallable.call(SourcesXmlResolver.java:169)
>         at 
> edu.internet2.middleware.grouper.subj.SourcesXmlResolver.executeCallables(SourcesXmlResolver.java:230)
>         at 
> edu.internet2.middleware.grouper.subj.SourcesXmlResolver.findByIdOrIdentifier(SourcesXmlResolver.java:520)
>         at 
> edu.internet2.middleware.grouper.subj.CachingResolver.findByIdOrIdentifier(CachingResolver.java:377)
>         at 
> edu.internet2.middleware.grouper.subj.ValidatingResolver.findByIdOrIdentifier(ValidatingResolver.java:203)
>         at 
> edu.internet2.middleware.grouper.SubjectFinder.findByIdOrIdentifier(SubjectFinder.java:315)
>         at 
> edu.internet2.middleware.grouper.ui.GrouperUiFilter.retrieveSubjectLoggedInHelper(GrouperUiFilter.java:347)
>         at 
> edu.internet2.middleware.grouper.ui.GrouperUiFilter.retrieveSubjectLoggedIn(GrouperUiFilter.java:297)
>         at 
> edu.internet2.middleware.grouper.ui.GrouperUiFilter.doFilter(GrouperUiFilter.java:1013)
>         at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>         at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>         at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>         at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>         at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>         at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>         at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>         at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>         at 
> org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
>         at 
> org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
>         at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776)
>         at 
> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705)
>         at 
> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
>         at 
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: javax.naming.NamingException: LDAP response read timed out, 
> timeout used:2000ms.; remaining name 'ou=people,dc=uiuc,dc=edu'
>         at com.sun.jndi.ldap.Connection.readReply(Connection.java:483)
>         at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:639)
>         at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:562)
>         at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
>         at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)
>         at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
>         at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1789)
>         at 
> com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:412)
>         at 
> com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:394)
>         at 
> com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:376)
>         at edu.vt.middleware.ldap.AbstractLdap.search(AbstractLdap.java:215)
>         at edu.vt.middleware.ldap.Ldap.search(Ldap.java:431)
>         at edu.vt.middleware.ldap.Ldap.search(Ldap.java:347)
>         at 
> edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResultsHelper(LdapSourceAdapter.java:771)
>         ... 30 more