Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] LDAP timeouts after Java upgrade

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] LDAP timeouts after Java upgrade


Chronological Thread 
  • From: "Oulman,James F" <>
  • To: Baron Fujimoto <>, "" <>
  • Subject: Re: [grouper-users] LDAP timeouts after Java upgrade
  • Date: Tue, 19 May 2020 03:52:30 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ufl.edu; dmarc=pass action=none header.from=ufl.edu; dkim=pass header.d=ufl.edu; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xZ6ELki0IS1IyPDnawl1+SairjIfbPLQInP5Eq5/dzw=; b=DGmzJ4WQ2P1HyYdOD2UOW/G81PL8w4uYD6ZO9kDzB81G2DkkRu4n9msgkg7PQqjI0RfUGb5FJSqTzdutF8LJFaryDVDDjYXl58qgCsAz2ieeL2ZXj1jB1bB0/ytIQVTlZ+8/WHp6LnxAUuYZkg7rwZTluNi+EpXbaHlburb2s/Cq0bdQnEu8xwP+ehl+9vyQ05Ru7r7gIM9rAvpxPaODA3skyQspFKunC6RHFpnEn6qIgzkvGS3AxyvnoIwZf1ckgW/Ldp/hvg4NebmTBf2oDRBfSZAiw2z71ViIWnpDvllxWevXxcGJWuowxyfA8nklE6MMMqDuk3o7Gu589uagtQ==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=j4yY+RXABsVSy0YoBEaQUXycNO7jan3XTOr+oeXgXIoEJ7a9SXDBZJAWDh9DmHHKW6J8+hLe4G02RiStspWkicrQYkarBDtXIdnDDO4qnpJMC0SLkTrk7c9SdSxKuYVCpSC6BB+3BjSRiztEzrm4v0vH7n3LedXtGTl2LC8T5NcaRSe5UsEJqsUNiF3qEGv1SUmPd5RD32ksgeiE6nxGD5fnvl2bQs8ePNiNlOO+t8ECObXm39Myo3xD6Cgtt2n4IQXY/Fr5HluEzHB4BAVbJbgnaYQfxKApENmRloCLDbno7ivrwVmxBrSDf3L9ukmFjb24deAb10/bKy14VB5npA==

My understanding is the default idle timeout on the F5 tcp profile is 300
seconds[1]. On our Shibboleth IdPs, we run the ldap validation at 60s and it
resolved our dropped connections errors in the logs.

1. https://support.f5.com/csp/article/K13004262

´╗┐On 5/18/20, 9:55 PM, " on behalf of Baron
Fujimoto" < on behalf of >
wrote:

[External Email]

On Tue, May 05, 2020 at 05:22:17PM -1000, Baron Fujimoto wrote:
>On Tue, Apr 28, 2020 at 10:14:55AM +0100, Robert Bradley wrote:
>>
>>On 28/04/2020 02:27, Baron Fujimoto wrote:
>>>We're running Grouper 2.2.2 with LDAP (389DS) as a subject source.
>>>We were previously using Java 1.0.8_212 successfully. However, I
>>>recently upgraded the instance to use the current version of Java
>>>(251), and after doing so noticed that while it initially appears
>>>to work as expected, the LDAP connections eventually begin to time
>>>out with the following error:
>>>
>>>javax.naming.NamingException: LDAP response read timed out,
>>>timeout used:-1ms
>>>
>>>The timeouts start to occur after ~20 minutes. Netstat shows no
>>>open connections to our LDAP at that point.
>>>
>>>The grouper host is actually a node in a cluster behind a load
>>>balancer, but our lb admins can't find any relevant ~20 minute
>>>timeout value there.
>>>
>>>I've empirically determined that this appears to happen with a
>>>version of Java 8 higher than 221 (i.e. 231, 241, 251). I dodn't
>>>see anything in the JDK release notes for 231 that appear to be
>>>relevant.

>>><https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oracle.com_technetwork_java_javase_8u-2Drelnotes-2D2225394.ht&d=DwIBAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=_L7sACgIQaR0AZonCJxTrg&m=dVAVILzctndF7wXmU6MlIciU0r1tt3WEKz2dnrBpYAU&s=aIg6Cs7FoO3SdtUsO0xWjRA9jO2ieEZPJu8K3vLCGLg&e=
>>ml>
>>>
>>>
>>One thought is that it could be a similar JNDI bug to that described
>>in
https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.shibboleth.net_confluence_display_IDP30_LDAPonJava-253E8&d=DwIBAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=_L7sACgIQaR0AZonCJxTrg&m=dVAVILzctndF7wXmU6MlIciU0r1tt3WEKz2dnrBpYAU&s=rkJHTfS6eSLDQRmJPnCyBIa9W1uLd4EBaU0D8NlwfV8&e=
>>and
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.shibboleth.net_jira_browse_IDP-2D1441&d=DwIBAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=_L7sACgIQaR0AZonCJxTrg&m=dVAVILzctndF7wXmU6MlIciU0r1tt3WEKz2dnrBpYAU&s=h77iKk9Gc3ZMgIqkz8A5_jepIL3J42xnOwH7J6cM63o&e=
. The problem
>>with that is that the fix probably involves upgrading Grouper to get
>>the ldaptive library instead of vt-ldap, and then configuring it to
>>use the UnboundID library instead of JNDI. I doubt that's a practical
>>option in the short term unless vt-ldap has a similar setting you can tr
>>y.
>
>FWIW, I'm also seeing similar behavior when I attempted the same upgrade
in one of our CAS deployments, though the timeout errors happen rather
quickly there, so it doesn't appear to be Grouper specific. AFAIK, the
version of CAS involved is using an ldaptive library, so that suggests that
the problem may not lie with the vt-ldap library. Unfortunately my searches
don't turn up any evidence that this has been a problem for others, so I'm
kind of at a loss now. :/

We're still wrestling with this, but have uncovered a few more details in
case it provides any new insight into the problem.

1) Our LDAP is actually a cluster behind an F5 load balancer. If we point
CAS or Grouper at non-load balanced LDAP host, we do not see the timeout
problem. It appears that both JDK 8u231+ *and* LDAP behind the load balancer
are necessary conditions to trigger the timeour error.

So clearly it's some interaction between the two, possibly some half
closed or zombie connection from the load balancer that the upgraded Java is
not dealing with properly.

2) We've empirically determined that if we shorten the default value for
the LDAP pool validation from 600s to, say, 60s for CAS then this also
mitigates the timeout problem. The shortened pool validation period seems to
be sufficient to function as some sort of keepalive.

I tried something similar for Grouper, setting the validateTimerPeriod to
a very short value with the following in our ldap.properties:

edu.vt.middleware.ldap.pool.validatePeriodically = true
edu.vt.middleware.ldap.pool.validateTimerPeriod = 60

It seems to have the same mitigating effect for the Grouper UI, but not
for the Grouper WS as far as I can tell.

--
UH Information Technology Services : Identity & Access Mgmt, Middleware
minutas cantorum, minutas balorum, minutas carboratum desendus pantorum




Archive powered by MHonArc 2.6.19.

Top of Page