Grouper Users - Open Discussion List

["Redman, Chad Eric" <>]

If you are setting validatePeriodically in ldap.properties, you may also want 
to set edu.vt.middleware.ldap.pool.validateTimerPeriod, as it will default to 
30 minutes if not set. We have 5 minute stale timeouts, and this has helped 
us keep the connections open. However, it was generally just avoiding excess 
warnings in the logs; the LDAP connections should attempt to create a new 
connection if the existing one fails.

Don't forget -- the validation is partially in ldap.properties and partially 
in sources.xml. The validation wouldn't know what to do without telling it 
how to validate. So sources.xml would look like:

    <init-param>
      <param-name>ldapProperties_file</param-name>
      <param-value>ldap.properties</param-value>
    </init-param>
    <init-param>
      <param-name>VTLDAP_VALIDATOR</param-name>
      <param-value>CompareLdapValidator</param-value>
    </init-param>
    <init-param>
      <param-name>VTLDAP_VALIDATOR_COMPARE_DN</param-name>
      <param-value>ou=people,dc=example,dc=edu</param-value>
    </init-param>
    <init-param>
      <param-name>VTLDAP_VALIDATOR_COMPARE_SEARCH_FILTER_STRING</param-name>
      <param-value>(ou=people)</param-value>
    </init-param>


For more troubleshooting, you can set the log4j.properties log level for 
vt-ldap to TRACE or DEBUG:

    log4j.logger.edu.vt.middleware.ldap = TRACE

That would log the prune, expire, and validation checks, so you can see if 
and when they are triggered.

-Chad



-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Tom Poage
Sent: Thursday, December 15, 2016 6:10 PM
To: 

Subject: Re: [grouper-users] Subject source LDAP timeouts

This might be some help:

https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.internet2.edu%2Fsympa%2Farc%2Fgrouper-users%2F2015-09%2Fmsg00054.html&data=01%7C01%7Cchad_redman%40unc.edu%7C05c9dc1aa73d46e5106508d4253fabc8%7C58b3d54f16c942d3af081fcabd095666%7C1&sdata=G8h06RBBo793Rdf6AHpDr%2FF5K%2FsBWzsDaw0JbdYwTcE%3D&reserved=0

I'd say set validatePeriodically=true and, if your traffic is expected to be 
high, leave validateOnCheck{In,Out} as false (default, I think).

All that's missing are the validation parameters themselves, e.g. base DN, 
query filter, bind DN/creds if used, prune timer period. These are 
implemented in ldaptive; I don't exactly recall if/how they're implemented in 
vt-ldap, but do recall similar features in the vt-ldap JAAS implementation. 
May also depend which version of vt-ldap is used in Grouper.

Tom.

> On Dec 15, 2016, at 2:52 PM, Wessel, Keith 
> <>
>  wrote:
> 
> I actually haven't set up validation queries yet. I was starting simple. 
> Any idea if expirationTime is independent of validation? I assumed it was.
> 
> I agree; it sounds like pooling isn't functioning, or at least not 
> functioning right.
> 
> Keith
> 
> 
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Tom Poage
> Sent: Thursday, December 15, 2016 4:34 PM
> To: 
> 
> Subject: Re: [grouper-users] Subject source LDAP timeouts
> 
> The behavior suggests to me that connection pool validation is not being 
> performed.
> 
> E.g. our CAS and Shibboleth services perform connection validation every 5 
> minutes, with a prune check every 10 minutes. Our LDAP infrastructure 
> closes idle connections at one hour, so LDAP connections from these 
> services are (in theory) never forcibly closed, and the base number of 
> connections stay open in perpetuity (Shibboleth and CAS use ldaptive, 
> vt-ldap being its ancestor). If a connection (validation test) fails, new 
> pool connections are attempted. Anyhow, CAS and Shib list archives may help.
> 
> Tom.
> 
>> On Dec 15, 2016, at 2:09 PM, Wessel, Keith 
>> <>
>>  wrote:
>> 
>> Yes.
>> 
>> Other thoughts?
>> 
>> Keith
>> 
>> 
>> From: Hyzer, Chris 
>> [mailto:]
>>  
>> Sent: Thursday, December 15, 2016 3:08 PM
>> To: Wessel, Keith 
>> <>;
>>  
>> 
>> Subject: RE: [grouper-users] Subject source LDAP timeouts
>> 
>> Do you have this in the sources.xml for that source?
>> 
>> <init-param>
>>      <param-name>ldapProperties_file</param-name>
>>      <param-value>ldap-sources.properties</param-value>
>>    </init-param>
>> 
>> Thanks
>> Chris
>> 
>> 
>> From: 
>> 
>>  
>> [mailto:]
>>  On Behalf Of Wessel, Keith
>> Sent: Thursday, December 15, 2016 3:03 PM
>> To: 
>> 
>> Subject: RE: [grouper-users] Subject source LDAP timeouts
>> 
>> Following up on this. Thanks for the advice so far, Julio.
>> 
>> The load balancer may be the problem, but we haven’t found an easy way to 
>> change that yet, and we’re really hoping we can fix it with LDAP 
>> connection pooling instead.
>> 
>> I’ve added the following two settings to ldap-sources.properties:
>> 
>> edu.vt.middleware.ldap.pool.pruneTimerPeriod = 60000
>> edu.vt.middleware.ldap.pool.expirationTime=600000
>> 
>> What I was expecting to see as a result of this, with a minPoolSize of 2, 
>> is new LDAP connections being created every 10 minutes or so. I see no 
>> such thing in our logs, though. I see two connections being created on 
>> Tomcat start-up, then I see no new connections at all until the error 
>> occurs.
>> 
>> A sequence looks like this in the LDAP logs:
>> At 12:04, I start Tomcat, and two connections are created. It uses one 
>> when I log in on the web interface a minute later.
>> I then let my browser sit idle.
>> At 12:30, my LDAP server logs report an unbind for both connections.
>> When I click a link in the browser at 1:02, I get an error that my subject 
>> couldn’t be retrieved, and I get the error I included in my original post 
>> to this thread.
>> 
>> The GSLB inactivity timeout is 30 minutes, not 26. So, while that could be 
>> the issue, I’m not sure.
>> 
>> But the fact is my pool doesn’t seem to be working as I would think it 
>> would. Is there something I need to do to enable pooling that I haven’t 
>> done? Far as I can tell, with the two initial connections, pooling’s 
>> working on start-up. It’s just not purging and expiring connections.
>> 
>> Any thoughts appreciated.
>> 
>> Keith
>> 
>> From: Julio Polo 
>> [mailto:]
>>  
>> Sent: Wednesday, December 07, 2016 7:13 PM
>> To: Wessel, Keith 
>> <>
>> Cc: 
>> 
>> Subject: Re: [grouper-users] Subject source LDAP timeouts
>> 
>> I would disable idle timeouts in the load balancer and the LDAP server.   
>> It's OK to have timeouts for LDAP queries in both ends, client and server. 
>>   If there is no way to turn off the load balancer's idle timeout, can you 
>> configure it to 24 hours and ensure that you query a Grouper LDAP subject 
>> at least once a day?  Adjust the intervals accordingly to the maximum idle 
>> timeout value for your appliance.
>> 
>> I'm not familiar with Grouper's LDAP connection pool validation, but 
>> anything that can help remove a zombie connection before it's used can 
>> only help.   The validation would have to be a high-level validation, ie 
>> make a subject query that returns something without timing out.
>> 
>> -julio
>> 
>> On Wed, Dec 7, 2016 at 5:13 AM, Wessel, Keith 
>> <>
>>  wrote:
>> Thanks, Julio. We do, in fact, have a load balancer in the mix, too. If 
>> I’m going to tweak idle timeouts on the LDAP server or load balancer, what 
>> should my max idle timeout be? I assume shorter than the pool expiration 
>> time – is that 10 minutes by default?
>> 
>> Similarly, is pool validation something that would help solve this 
>> problem? I don’t believe we’ve configured it, and I would assume it’s not 
>> configured by default since the validation check is specific to our 
>> environment.
>> 
>> Thanks,
>> Keith
>> 
>> 
>> From: Julio Polo 
>> [mailto:]
>>  
>> Sent: Tuesday, December 06, 2016 6:50 PM
>> To: Wessel, Keith 
>> <>
>> Cc: 
>> 
>> Subject: Re: [grouper-users] Subject source LDAP timeouts
>> 
>> If the logs for your LDAP server don't reveal anything related to this 
>> timeout, check if there's a firewall or load balancer between Grouper and 
>> the network interface associated with the LDAP fqdn you configured Grouper 
>> with. 
>> 
>> When it comes to persistent connections, we've experienced weird things 
>> when such network appliances and idle timeouts are involved.   The 
>> connection will appear to be fine from the client side, but there is 
>> nothing passing through these appliances onto the server.   Grouper's LDAP 
>> connection pooling wouldn't see anything wrong with the connection, but 
>> the client request isn't making it through the firewall or load balancer.  
>>  You can try tweaking the idle settings on these appliances (we've only 
>> had partial success) or you could specify an address that bypasses those 
>> appliances (but you'd lose the load balancing and every other benefit that 
>> comes with those appliances).
>> 
>> -julio
>> 
>> Julio Polo
>> Enterprise Middleware, Identity and Access Management
>> Information Technology Services
>> University of Hawaii
>> 
>> On Tue, Dec 6, 2016 at 9:53 AM, Wessel, Keith 
>> <>
>>  wrote:
>> Hi,
>> 
>> We've been seeing LDAP connection read timeout exceptions in our Grouper 
>> dev and text environments. (We don't have production running quite yet.) 
>> They're somewhat random in terms of when they happen, but sometimes we see 
>> them if someone leaves a Grouper UI session idle in their browser for more 
>> than a half hour (give or take). Other times, they happen when an 
>> environment hasn't been accessed in a while.
>> 
>> I'm blaming connecton pooling, but I'm unclear why a stale connection from 
>> the pool would cause this to happen. After all, if our LDAP server closed 
>> an idle connection, wouldn't' the pool just create a new one? And is the 
>> pool really handing out stale connections?
>> 
>> The LDAP server, for what it's worth, as an IBM Security Systems Directory 
>> Server.
>> 
>> Any advice would be appreciated. The Java stack trace is below.
>> 
>> Thanks,
>> Keith
>> 
>> 2016-12-06 13:00:51,534: [TP-Processor3] ERROR 
>> GrouperUiFilter.doFilter(1030) -  - UI error
>> edu.internet2.middleware.subject.SourceUnavailableException: Ldap 
>> NamingException: LDAP response read timed out, timeout used:2000ms.,
>> Cant find subject from login id: ecc
>>        at 
>> edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResultsHelper(LdapSourceAdapter.java:774)
>>        at 
>> edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResults(LdapSourceAdapter.java:661)
>>        at 
>> edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapUnique(LdapSourceAdapter.java:806)
>>        at 
>> edu.internet2.middleware.subject.provider.LdapSourceAdapter.getSubject(LdapSourceAdapter.java:374)
>>        at 
>> edu.internet2.middleware.subject.provider.BaseSourceAdapter.getSubjectByIdOrIdentifier(BaseSourceAdapter.java:252)
>>        at 
>> edu.internet2.middleware.grouper.subj.SourcesXmlResolver$4.callLogic(SourcesXmlResolver.java:514)
>>        at 
>> edu.internet2.middleware.grouper.subj.SourcesXmlResolver$4.callLogic(SourcesXmlResolver.java:511)
>>        at 
>> edu.internet2.middleware.grouper.subj.SourcesXmlResolver$LogLabelCallable.call(SourcesXmlResolver.java:169)
>>        at 
>> edu.internet2.middleware.grouper.subj.SourcesXmlResolver.executeCallables(SourcesXmlResolver.java:230)
>>        at 
>> edu.internet2.middleware.grouper.subj.SourcesXmlResolver.findByIdOrIdentifier(SourcesXmlResolver.java:520)
>>        at 
>> edu.internet2.middleware.grouper.subj.CachingResolver.findByIdOrIdentifier(CachingResolver.java:377)
>>        at 
>> edu.internet2.middleware.grouper.subj.ValidatingResolver.findByIdOrIdentifier(ValidatingResolver.java:203)
>>        at 
>> edu.internet2.middleware.grouper.SubjectFinder.findByIdOrIdentifier(SubjectFinder.java:315)
>>        at 
>> edu.internet2.middleware.grouper.ui.GrouperUiFilter.retrieveSubjectLoggedInHelper(GrouperUiFilter.java:347)
>>        at 
>> edu.internet2.middleware.grouper.ui.GrouperUiFilter.retrieveSubjectLoggedIn(GrouperUiFilter.java:297)
>>        at 
>> edu.internet2.middleware.grouper.ui.GrouperUiFilter.doFilter(GrouperUiFilter.java:1013)
>>        at 
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>        at 
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>        at 
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>        at 
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>        at 
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>        at 
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>        at 
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>        at 
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>>        at 
>> org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
>>        at 
>> org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
>>        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776)
>>        at 
>> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705)
>>        at 
>> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
>>        at 
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
>>        at java.lang.Thread.run(Thread.java:745)
>> Caused by: javax.naming.NamingException: LDAP response read timed out, 
>> timeout used:2000ms.; remaining name 'ou=people,dc=uiuc,dc=edu'
>>        at com.sun.jndi.ldap.Connection.readReply(Connection.java:483)
>>        at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:639)
>>        at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:562)
>>        at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
>>        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)
>>        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
>>        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1789)
>>        at 
>> com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:412)
>>        at 
>> com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:394)
>>        at 
>> com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:376)
>>        at edu.vt.middleware.ldap.AbstractLdap.search(AbstractLdap.java:215)
>>        at edu.vt.middleware.ldap.Ldap.search(Ldap.java:431)
>>        at edu.vt.middleware.ldap.Ldap.search(Ldap.java:347)
>>        at 
>> edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapResultsHelper(LdapSourceAdapter.java:771)
>>        ... 30 more
>