Grouper Users - Open Discussion List

["Hyzer, Chris" <>]

Can you install the latest patches and try again?

-----Original Message-----
From: Prasad Karkhanis 
Sent: Tuesday, March 03, 2020 9:21 AM
To: ; Hyzer, Chris <>
Subject: Re: RE: [grouper-users] Trouble Running Loader Diagnostics v2.3.0

Hi Chris,

Thanks for your response. Sorry for not being clearer earlier - I replied to
Bryan Wooten's post on the mailing group, which contained the specific error
message I was getting, and I forgot to put the error again in my post. The
original post by Bryan is located here:

https://lists.internet2.edu/sympa/arc/grouper-users/2018-05/msg00058.html

And the error message he posted about (which I'm getting as well) is:

Error: Cant execute reflection method: loaderDiagnosticsRun, on:
edu.internet2.middleware.grouper.grouperUi.serviceLogic.UiV2GrouperLoader,
with args: edu.internet2.middleware.grouper.j2ee.GrouperRequestWrapper,
org.owasp.csrfguard.http.InterceptRedirectResponse, Problem calling method
loaderDiagnosticsRun on
edu.internet2.middleware.grouper.grouperUi.serviceLogic.UiV2GrouperLoader

The version we're using is 2.4, and on doing further investigation, I found
the following blurb in the grouper-ui log:

tomcat;console;;;Exception in thread
"C3P0PooledConnectionPoolManager[identityToken->166bos0a810ev41lrida2q|
147dd0a2]-HelperThread-#2" Exception in thread "http-nio-8080-ClientPoller-1"
Exception in thread "C3P0PooledConnectionPoolManager[identityToken-
>166bos0a810ev41lrida2q|4cbcb90c]-HelperThread-#2" 2020-03-02 19:38:42,276:
[Thread-12] ERROR SubjectSourceCache.subjectSourceCacheThread(289) -  - Error
in outer loop of subject source cache
tomcat;console;;;java.lang.OutOfMemoryError: GC overhead limit exceeded
tomcat;console;;;       at java.util.Arrays.copyOf(Arrays.java:3332)
tomcat;console;;;       at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
tomcat;console;;;       at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
tomcat;console;;;       at
java.lang.StringBuffer.append(StringBuffer.java:367)
tomcat;console;;;       at java.io.StringWriter.write(StringWriter.java:94)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.copy(ConfigPropertiesCascadeCommonUtils.java:5089)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.copy(ConfigPropertiesCascadeCommonUtils.java:5074)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.toString(ConfigPropertiesCascadeCommonUtils.java:5057)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase
$ConfigFile.retrieveContents(ConfigPropertiesCascadeBase.java:749)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.filesNeedReloadingBasedOnContents(ConfigPropertiesCascadeBase.java:1127)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.retrieveFromConfigFileOrCache(ConfigPropertiesCascadeBase.java:1069)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.retrieveConfig(ConfigPropertiesCascadeBase.java:104)
tomcat;console;;;       at
edu.internet2.middleware.subject.config.SubjectConfig.retrieveConfig(SubjectConfig.java:53)
tomcat;console;;;       at
edu.internet2.middleware.grouper.subj.cache.SubjectSourceCache.cacheEnabled(SubjectSourceCache.java:1569)
tomcat;console;;;       at
edu.internet2.middleware.grouper.subj.cache.SubjectSourceCache.subjectSourceCacheThread(SubjectSourceCache.java:247)
tomcat;console;;;       at
edu.internet2.middleware.grouper.subj.cache.SubjectSourceCache
$1.run(SubjectSourceCache.java:219)
tomcat;console;;;       at java.lang.Thread.run(Thread.java:748)
tomcat;console;;;2020-03-02 19:38:42,276: [Thread-12] ERROR
SubjectSourceCache.subjectSourceCacheThread(289) -  - Error in outer loop of
subject source cache
tomcat;console;;;java.lang.OutOfMemoryError: GC overhead limit exceeded
tomcat;console;;;       at java.util.Arrays.copyOf(Arrays.java:3332)
tomcat;console;;;       at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
tomcat;console;;;       at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
tomcat;console;;;       at
java.lang.StringBuffer.append(StringBuffer.java:367)
tomcat;console;;;       at java.io.StringWriter.write(StringWriter.java:94)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.copy(ConfigPropertiesCascadeCommonUtils.java:5089)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.copy(ConfigPropertiesCascadeCommonUtils.java:5074)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.toString(ConfigPropertiesCascadeCommonUtils.java:5057)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase
$ConfigFile.retrieveContents(ConfigPropertiesCascadeBase.java:749)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.filesNeedReloadingBasedOnContents(ConfigPropertiesCascadeBase.java:1127)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.retrieveFromConfigFileOrCache(ConfigPropertiesCascadeBase.java:1069)
tomcat;console;;;       at
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.retrieveConfig(ConfigPropertiesCascadeBase.java:104)
tomcat;console;;;       at
edu.internet2.middleware.subject.config.SubjectConfig.retrieveConfig(SubjectConfig.java:53)
tomcat;console;;;       at
edu.internet2.middleware.grouper.subj.cache.SubjectSourceCache.cacheEnabled(SubjectSourceCache.java:1569)
tomcat;console;;;       at
edu.internet2.middleware.grouper.subj.cache.SubjectSourceCache.subjectSourceCacheThread(SubjectSourceCache.java:247)
tomcat;console;;;       at
edu.internet2.middleware.grouper.subj.cache.SubjectSourceCache
$1.run(SubjectSourceCache.java:219)
tomcat;console;;;       at java.lang.Thread.run(Thread.java:748)
tomcat;console;;;Exception in thread
"C3P0PooledConnectionPoolManager[identityToken->166bos0a810ev41lrida2q|
147dd0a2]-HelperThread-#1" java.lang.OutOfMemoryError: GC overhead limit
exceeded
tomcat;console;;;       at
oracle.jdbc.driver.OracleSql.<init>(OracleSql.java:77)
tomcat;console;;;       at
oracle.jdbc.driver.OracleStatement.<init>(OracleStatement.java:931)
tomcat;console;;;       at
oracle.jdbc.driver.T4CStatement.<init>(T4CStatement.java:1793)
tomcat;console;;;       at
oracle.jdbc.driver.T4CDriverExtension.allocateStatement(T4CDriverExtension.java:45)
tomcat;console;;;       at
oracle.jdbc.driver.PhysicalConnection.createStatement(PhysicalConnection.java:4734)
tomcat;console;;;       at
oracle.jdbc.driver.PhysicalConnection.createStatement(PhysicalConnection.java:4656)
tomcat;console;;;       at
com.mchange.v2.c3p0.impl.NewProxyConnection.createStatement(NewProxyConnection.java:176)
tomcat;console;;;       at
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:285)
tomcat;console;;;       at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool
$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:510)
tomcat;console;;;       at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool
$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:464)
tomcat;console;;;       at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool
$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:436)
tomcat;console;;;       at com.mchange.v2.resourcepool.BasicResourcePool
$AsyncTestIdleResourceTask.run(BasicResourcePool.java:2211)
tomcat;console;;;       at com.mchange.v2.async.ThreadPoolAsynchronousRunner
$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
tomcat;console;;;java.lang.OutOfMemoryError: GC overhead limit exceeded
tomcat;console;;;java.lang.OutOfMemoryError: GC overhead limit exceeded
tomcat;console;;;java.lang.OutOfMemoryError: GC overhead limit exceeded
tomcat;catalina.out;${env:ENV};${env:USERTOKEN};2020-03-02 19:39:13,382
[ContainerBackgroundProcessor[StandardEngine[Catalina]]] ERROR
org.apache.catalina.core.ContainerBase- Unexpected death of background thread
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
java.lang.OutOfMemoryError: GC overhead limit exceeded
tomcat;console;;;Exception in thread
"ContainerBackgroundProcessor[StandardEngine[Catalina]]"
java.lang.OutOfMemoryError: GC overhead limit exceeded
tomcat;console;;;2020-03-02 19:39:13,467: [ajp-nio-8009-exec-3] ERROR
GrouperUiRestServlet.doGet(359) -  - Problem calling reflection from URL:
edu.internet2.middleware.grouper.grouperUi.serviceLogic.UiV2GrouperLoader.loaderDiagnosticsRun
tomcat;console;;;
tomcat;console;;;java.lang.RuntimeException: Cant execute reflection method:
loaderDiagnosticsRun, on:
edu.internet2.middleware.grouper.grouperUi.serviceLogic.UiV2GrouperLoader,
with args: edu.internet2.middleware.grouper.j2ee.GrouperRequestWrapper,
org.owasp.csrfguard.http.InterceptRedirectResponse,
tomcat;console;;;Problem calling method loaderDiagnosticsRun on
edu.internet2.middleware.grouper.grouperUi.serviceLogic.UiV2GrouperLoader
tomcat;console;;;       at
edu.internet2.middleware.grouper.util.GrouperUtil.invokeMethod(GrouperUtil.java:4212)
tomcat;console;;;       at
edu.internet2.middleware.grouper.util.GrouperUtil.callMethod(GrouperUtil.java:4147)
tomcat;console;;;       at
edu.internet2.middleware.grouper.j2ee.GrouperUiRestServlet.doGet(GrouperUiRestServlet.java:326)
tomcat;console;;;       at
edu.internet2.middleware.grouper.j2ee.GrouperUiRestServlet.doPost(GrouperUiRestServlet.java:197)
tomcat;console;;;       at
javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
tomcat;console;;;       at
javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
tomcat;console;;;       at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
tomcat;console;;;       at
org.owasp.csrfguard.CsrfGuardFilter.doFilter(CsrfGuardFilter.java:88)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
tomcat;console;;;       at
edu.internet2.middleware.grouper.ui.GrouperUiFilter.doFilter(GrouperUiFilter.java:1139)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
tomcat;console;;;       at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
tomcat;console;;;       at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
tomcat;console;;;       at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:610)
tomcat;console;;;       at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
tomcat;console;;;       at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
tomcat;console;;;       at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
tomcat;console;;;       at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
tomcat;console;;;       at
org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:476)
tomcat;console;;;       at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
tomcat;console;;;       at org.apache.coyote.AbstractProtocol
$ConnectionHandler.process(AbstractProtocol.java:808)
tomcat;console;;;       at org.apache.tomcat.util.net.NioEndpoint
$SocketProcessor.doRun(NioEndpoint.java:1498)
tomcat;console;;;       at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
tomcat;console;;;       at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
tomcat;console;;;       at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:624)
tomcat;console;;;       at org.apache.tomcat.util.threads.TaskThread
$WrappingRunnable.run(TaskThread.java:61)
tomcat;console;;;       at java.lang.Thread.run(Thread.java:748)
tomcat;console;;;Caused by: java.lang.reflect.InvocationTargetException
tomcat;console;;;       at
sun.reflect.GeneratedMethodAccessor1091.invoke(Unknown Source)
tomcat;console;;;       at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
tomcat;console;;;       at java.lang.reflect.Method.invoke(Method.java:498)
tomcat;console;;;       at
edu.internet2.middleware.grouper.util.GrouperUtil.invokeMethod(GrouperUtil.java:4196)
tomcat;console;;;       ... 32 more
tomcat;console;;;Caused by: java.lang.OutOfMemoryError: GC overhead limit
exceeded
tomcat;console;;;
tomcat;console;;;2020-03-02 19:39:13,467: [ajp-nio-8009-exec-3] ERROR
GrouperUiRestServlet.doGet(359) -  - Problem calling reflection from URL:
edu.internet2.middleware.grouper.grouperUi.serviceLogic.UiV2GrouperLoader.loaderDiagnosticsRun
tomcat;console;;;
tomcat;console;;;java.lang.RuntimeException: Cant execute reflection method:
loaderDiagnosticsRun, on:
edu.internet2.middleware.grouper.grouperUi.serviceLogic.UiV2GrouperLoader,
with args: edu.internet2.middleware.grouper.j2ee.GrouperRequestWrapper,
org.owasp.csrfguard.http.InterceptRedirectResponse,
tomcat;console;;;Problem calling method loaderDiagnosticsRun on
edu.internet2.middleware.grouper.grouperUi.serviceLogic.UiV2GrouperLoader
tomcat;console;;;       at
edu.internet2.middleware.grouper.util.GrouperUtil.invokeMethod(GrouperUtil.java:4212)
tomcat;console;;;       at
edu.internet2.middleware.grouper.util.GrouperUtil.callMethod(GrouperUtil.java:4147)
tomcat;console;;;       at
edu.internet2.middleware.grouper.j2ee.GrouperUiRestServlet.doGet(GrouperUiRestServlet.java:326)
tomcat;console;;;       at
edu.internet2.middleware.grouper.j2ee.GrouperUiRestServlet.doPost(GrouperUiRestServlet.java:197)
tomcat;console;;;       at
javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
tomcat;console;;;       at
javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
tomcat;console;;;       at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
tomcat;console;;;       at
org.owasp.csrfguard.CsrfGuardFilter.doFilter(CsrfGuardFilter.java:88)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
tomcat;console;;;       at
edu.internet2.middleware.grouper.ui.GrouperUiFilter.doFilter(GrouperUiFilter.java:1139)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
tomcat;console;;;       at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
tomcat;console;;;       at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
tomcat;console;;;       at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
tomcat;console;;;       at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:610)
tomcat;console;;;       at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
tomcat;console;;;       at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
tomcat;console;;;       at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
tomcat;console;;;       at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
tomcat;console;;;       at
org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:476)
tomcat;console;;;       at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
tomcat;console;;;       at org.apache.coyote.AbstractProtocol
$ConnectionHandler.process(AbstractProtocol.java:808)
tomcat;console;;;       at org.apache.tomcat.util.net.NioEndpoint
$SocketProcessor.doRun(NioEndpoint.java:1498)
tomcat;console;;;       at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
tomcat;console;;;       at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
tomcat;console;;;       at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:624)
tomcat;console;;;       at org.apache.tomcat.util.threads.TaskThread
$WrappingRunnable.run(TaskThread.java:61)
tomcat;console;;;       at java.lang.Thread.run(Thread.java:748)
tomcat;console;;;Caused by: java.lang.reflect.InvocationTargetException
tomcat;console;;;       at
sun.reflect.GeneratedMethodAccessor1091.invoke(Unknown Source)
tomcat;console;;;       at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
tomcat;console;;;       at java.lang.reflect.Method.invoke(Method.java:498)
tomcat;console;;;       at
edu.internet2.middleware.grouper.util.GrouperUtil.invokeMethod(GrouperUtil.java:4196)
tomcat;console;;;       ... 32 more
tomcat;console;;;Caused by: java.lang.OutOfMemoryError: GC overhead limit
exceeded
tomcat;console;;;

This leads to the error message that Bryan referred to in his initial post
being thrown in the UI. I'm guessing it is because the loader I was trying to
set up chewed up all the memory because of a very broad SQL query, and I have
since then narrowed down the scope by adding a few "where" clauses. I haven't
been able to test it out further since our Dev DBs are undergoing routine
maintenance, and I'll check out the modified loader SQL in a couple of days
and post an update here.