grouper-users - [grouper-users] RE: Status Monitoring - Two Errors

Subject: Grouper Users - Open Discussion List

List archive

[grouper-users] RE: Status Monitoring - Two Errors

From: Ryan Rumbaugh <>
To: "" <>
Subject: [grouper-users] RE: Status Monitoring - Two Errors
Date: Mon, 27 Aug 2018 14:11:52 +0000
Accept-language: en-US
Ironport-phdr: 9a23:JSVAMByOE4VeqLTXCy+O+j09IxM/srCxBDY+r6Qd2uIQIJqq85mqBkHD//Il1AaPAd2Eraocw8Pt8InYEVQa5piAtH1QOLdtbDQizfssogo7HcSeAlf6JvO5JwYzHcBFSUM3tyrjaRsdF8nxfUDdrWOv5jAOBBr/KRB1JuPoEYLOksi7ze+/94HRbglSmDaxfa55IQmrownWqsQYm5ZpJLwryhvOrHtIeuBWyn1tKFmOgRvy5dq+8YB6/ShItP0v68BPUaPhf6QlVrNYFygpM3o05MLwqxbOSxaE62YGXWUXlhpIBBXF7A3/U5zsvCb2qvZx1S+HNsDwULs6Wymt771zRRHolikJKiI5/m/UhMx+jq1boQ6uqBNkzoHOfI2ZKOBzcr/Bcd8EQ2dKQ8ZfVzZGAoO5d4YBE+oAMvxer4Lgu1UOsB6+ChK2C+PyxT9IhmL90Kog0+Q7DAHG2RcvEMgTsHTbrNX4L6ESXv6uw6nS1zrDdPBW1i3n5IfWbx8hvOiBULRtesTfzkkvEhnKjlSWqYH9IzOV1v4NvHaB4OV9T+6vkXMnqwdprjiv2scslpHJhpkLxVDe6yV5xJg6JcGiRE55e9GkDoFfuzmHN4tqQ8MiR3tkuCAkxb0av5OwYSsEyIw/yhLCZPGKcJKE7g/iWeufOzt0mX1odbylixqv/kWtyPXwWtSq3FtOtCZIltfBumoD2hHT8MSKRPpw8l+v2TmR1A3f9uRJLEU0mKffJZ4u3rs9lpQJvkvZAiD7nUT7g7WVe0Uq/+Wn9uDqbqngq5SBLYF7kBv+Pb4rmsGnAeQ3LAwOX2+D9OS527zv4Vb2TKxWgvEqjKXUv5HXKMsHqqKnGQNVyp4j6xa4Dze9zNQXg2QIIEhZdxKAiojlI1DOIPbmAvejm1mgjThmyvPcMrH8A5jAIGLPnKrucLtz8UJQ1QU+wNJH65JREL4BIfbzWkHrtNzfCx80KxS0w/z5CNVmzo4TW2CCD62bPa/PrFOF5/8jL/eIaYINojb9MeUl5/v1gnAjhF8dcqmp0IARaH+mBftmOV+WYWHvgtgbC2sFoBcxTPHyhF2YTTFTf2qyX7475jwjB4KmF4DDRp21gLOfxie3B4BWZntYBVCWCnflbIGEW/YXaCKOOc9tjCYIVbmnS489yx6urgn6xKR7LubK4CEXq4/s1MUmr9HUwFs97ztpF8mHlnyWQntvtmIOWzIs2q1j+wpwxkrJmfx3mftFDdFJou5SXx0hHZ/a0+FgDd3uAETMcsrfG3i8RdDzOyA4SJoL3toEaltyG532lAre2iCjHrAVv7qGDoY96aHVmXX9OpAumD79yKA9ggx+EYN0Pmq8i/sn+g==
Spamdiagnosticmetadata: NSPM
Spamdiagnosticoutput: 1:99

An update to this issue that may be helpful to others…

Before I left the office on Friday I ran the gsh command “loaderRunOneJob(“CHANGE_LOG_changeLogTempToChangeLog”)” process and now the number of rows in the change_entry_temp table is zero! I tried running that before, but really didn’t see much of anything happening. Maybe I was just too impatient.

Now when accessing grouper/status?diagnosticType=all the only error is related to “OTHER_JOB_deprovisioningDaemon”. If anyone had any tips on how to get that kick started it would be greatly appreciated.

Ryan Rumbaugh

From: <> On Behalf Of Ryan Rumbaugh
Sent: Friday, August 24, 2018 9:15 AM
To:
Subject: [grouper-users] Status Monitoring - Two Errors

Good morning,

We would like to begin monitoring the status of grouper by using the diagnostic pages at grouper/status?diagnosticType=all, but before doing so I would like to take care of the two issues shown below.

Can anyone provide tips/suggestions on how to fix the two failures for CHANGE_LOG_changeLogTempToChangeLog and OTHER_JOB_deprovisioningDaemon?

We had a Java heap issue late last week which I believe caused the “grouper_change_log_entry_temp” table to keep growing. It’s at 69,886 rows currently while earlier this week it was at 50k. Thanks for any insight.

2 errors in the diagnostic tasks:

DiagnosticLoaderJobTest, Loader job CHANGE_LOG_changeLogTempToChangeLog

DiagnosticLoaderJobTest, Loader job OTHER_JOB_deprovisioningDaemon

Error stack for: loader_CHANGE_LOG_changeLogTempToChangeLog

java.lang.RuntimeException: Cant find a success in job CHANGE_LOG_changeLogTempToChangeLog since: 2018/08/16 14:19:22.000, expecting one in the last 30 minutes

at edu.internet2.middleware.grouper.j2ee.status.DiagnosticLoaderJobTest.doTask(DiagnosticLoaderJobTest.java:175)

at edu.internet2.middleware.grouper.j2ee.status.DiagnosticTask.executeTask(DiagnosticTask.java:78)

at edu.internet2.middleware.grouper.j2ee.status.GrouperStatusServlet.doGet(GrouperStatusServlet.java:180)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.owasp.csrfguard.CsrfGuardFilter.doFilter(CsrfGuardFilter.java:110)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:624)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)

at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:478)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:798)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1441)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:748)

Error stack for: loader_OTHER_JOB_deprovisioningDaemon

java.lang.RuntimeException: Cant find a success in job OTHER_JOB_deprovisioningDaemon, expecting one in the last 3120 minutes

at edu.internet2.middleware.grouper.j2ee.status.DiagnosticLoaderJobTest.doTask(DiagnosticLoaderJobTest.java:173)