grouper-users - [grouper-users] RE: Status Monitoring - Two Errors

Subject: Grouper Users - Open Discussion List

List archive

[grouper-users] RE: Status Monitoring - Two Errors

From: Ryan Rumbaugh <>
To: "" <>
Subject: [grouper-users] RE: Status Monitoring - Two Errors
Date: Mon, 27 Aug 2018 22:13:25 +0000
Accept-language: en-US
Ironport-phdr: 9a23:xN10aRLh29xchIr2wtmcpTZWNBhigK39O0sv0rFitYgXKvT5rarrMEGX3/hxlliBBdydt6obzbKO+4nbGkU4qa6bt34DdJEeHzQksu4x2zIaPcieFEfgJ+TrZSFpVO5LVVti4m3peRMNQJW2aFLduGC94iAPERvjKwV1Ov71GonPhMiryuy+4ZLebxlKiTanfb9+MAi9oBnMuMURnYZsMLs6xAHTontPdeRWxGdoKkyWkh3h+Mq+/4Nt/jpJtf45+MFOTav1f6IjTbxFFzsmKHw65NfqtRbYUwSC4GYXX3gMnRpJBwjF6wz6Xov0vyDnuOdxxDWWMMvrRr0yRD+s7bpkSAXwhSkHKTA37X3XhMJzgqJVoh2hpgBwzIHPbY6PKPZyYrnQcc8GSWZfWMtaSixPApm7b4sKF+cPIeZZoJP7p1ATsRW+GAysC/j1xT9ViX/23bAx3uM7EQHcwQwgGskBv27JrNX6NKcfSua1w7fTwjXZdfxWwjT955LSfh8/vP6MQKt9fMzMwkchEAPFi0+fqY3jPz6N2eQNqW+b7/d4Ve2xkW4rsRt+oiC3xss0i4nJgJ4VxU7e9SV/3ok1OcO0RFRlbtG5DZtcrz2aOJFsQs84TGFpuT42yqEGuZ6hYicF0okoywTFa/yadYiE+hLjVPqNITdgn3JqZqi/hwi28Ue+1u3wTMu030xUriVfitXMrmoN1xvU6siJUvt9+Uah2TCT1wzJ9u5EJkU0mbLGK549w74wkoAfsULdES/qnkj9kayYdl089+S15OnqYa/qqoKdOoJ6kA3yL6sjl86lDeglNgUDXXCX9fmi2LDn50H0RLtHguc4n6TZqpzXK8UWqra3AwBL0Ysv9xOyAjKm3dkck3kIMFdIeBydgIfzJ17DJfH1Au28jlSilTpk2vTLMqHnD57QNHbMiq3hcqx460NEyAo809Rf55VMB7EEL/P/RlX+uMXEAhMlLQC5zOfqBdph2o8AQ26PGreZMKPVsV+T+uIgPfSDaJUJtzb6Lvgp///ujXknll8BZaSlwJQaZXOiEvh7IkiUb2DgjsoOHGoIpAYyUejnhV+aXT5WfXmyXqY85j8hCIKhCIfOXpitgKaf3CegBpBbZGJLB1KOHHv1aomJWuoAZDyWL8J5iDwET6WhS4o62h60qQ/6xLpnI/HS+iIGrp3jzsJ65/bQlR4o7zB7EdmS03yVQ2FugmwIXyM23Lx4oUFlxVeDy694g+FAFdNN/fNFSxo6NYXCwOxgEND/QQbBftaSSFa6WdWqHys9TtM3w98SfUl9AdOigQ7f3ya0GbMaiaGEBIFnup7bijL+PcFg03vckbQ6gkM9aspJKWC8gKNjrU7eC5OD2xGWjaG3bakGmTPW+X2Y5WuIoExCVgNsC+PIUW1JNWXMqtGs3FLLSfeUFLEtOxZOyYbWMLFXb9fgkFpGbPHuMs7XeGG43Wq8GEDblfu3cIP2djBFj23mA08enlVL8A==
Spamdiagnosticmetadata: NSPM
Spamdiagnosticoutput: 1:99

Thanks for your suggestion to check the configuration settings. We do have those settings however.

After a bit more reading I realized I could simply call the loaderRunOneJob method on the deprovisioning process, however, when doing so I now get this exception:

groovy:000> loaderRunOneJob("OTHER_JOB_deprovisioningDaemon");

ERROR java.lang.RuntimeException:

java.lang.RuntimeException: Dont pass more than 100 ids: 3968

at edu.internet2.middleware.grouper.app.loader.GrouperLoader.runOnceByJobName (GrouperLoader.java:1625)

at edu.internet2.middleware.grouper.app.gsh.loaderRunOneJob.invoke (loaderRunOneJob.java:95)

at edu.internet2.middleware.grouper.app.gsh.loaderRunOneJob$invoke.call (Unknown Source)

at groovysh_evaluate.loaderRunOneJob (groovysh_evaluate:4)

I tracked down this issue in JIRA (https://bugs.internet2.edu/jira/browse/GRP-1832?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel

) and the solution appears to have been “sending a batch of 100 stem ids”. Any suggestions on exactly what that means to a Grouper newbie?

Ryan Rumbaugh

From: Black, Carey M. <>
Sent: Monday, August 27, 2018 10:21 AM
To: Ryan Rumbaugh <>;
Subject: RE: Status Monitoring - Two Errors

Ryan,

RE: “I had been restarting the API daemon” … ( due to docker use )

I have often wondered how the “shutdown process” works for the daemon. Is it “graceful” ( and lets all running jobs complete before shutdown) or does it just “pull the plug”?

I think it just pulls the plug.

Which “leaves” running jobs as “in progress”(in the DB status table) and they refuse to immediately start when the loader restarts. Well, until the “in progress” record(s) get old enough that they are assumed to be dead. Then the jobs will no longer refuse to start.

I say that to say this:

If the loader is restarted repeatedly, quickly, and/or often, you may be interrupting the running jobs and leaving them as “in progress” (in the DB) and producing more delay on the jobs re-starting again. But it all depends on how fast/often those things are spinning up and down.

However, maybe If you always spinning up instances (and let the old ones run for a bit) you may be able to “wait till a good time” to turn them off.

Maybe if you cycle out the old instances gracefully by timing it with these settings?

“

##################################

## enabled / disabled cron

##################################

#quartz cron-like schedule for enabled/disabled daemon. Note, this has nothing to do with the changelog

#leave blank to disable this, the default is 12:01am, 11:01am, 3:01pm every day: 0 1 0,11,15 * * ?

changeLog.enabledDisabled.quartz.cron = 0 1 0,11,15 * * ?

“

RE: how to schedule the “deprovisioningDaemon”

Verify that your grouper-loader.base.properties has this block: ( or you can add it to your grouper-loader.properties )

NOTE: it was added to the default base as of GRP-1623. ( which maps to grouper_v2_3_0_api_patch_107 ( and for the UI grouper_v2_3_0_ui_patch_44 ) ) You likely are past those patches… but just saying. J

“

#####################################

## Deprovisioning Job

#####################################

otherJob.deprovisioningDaemon.class = edu.internet2.middleware.grouper.app.deprovisioning.GrouperDeprovisioningJob

otherJob.deprovisioningDaemon.quartzCron = 0 0 2 * * ?

“

HTH.

Carey Matthew

From: <> On Behalf Of Ryan Rumbaugh
Sent: Monday, August 27, 2018 10:12 AM
To:
Subject: [grouper-users] RE: Status Monitoring - Two Errors

An update to this issue that may be helpful to others…

Before I left the office on Friday I ran the gsh command “loaderRunOneJob(“CHANGE_LOG_changeLogTempToChangeLog”)” process and now the number of rows in the change_entry_temp table is zero! I tried running that before, but really didn’t see much of anything happening. Maybe I was just too impatient.

Now when accessing grouper/status?diagnosticType=all the only error is related to “OTHER_JOB_deprovisioningDaemon”. If anyone had any tips on how to get that kick started it would be greatly appreciated.

Ryan Rumbaugh

From: <> On Behalf Of Ryan Rumbaugh
Sent: Friday, August 24, 2018 9:15 AM
To:
Subject: [grouper-users] Status Monitoring - Two Errors

Good morning,

We would like to begin monitoring the status of grouper by using the diagnostic pages at grouper/status?diagnosticType=all, but before doing so I would like to take care of the two issues shown below.

Can anyone provide tips/suggestions on how to fix the two failures for CHANGE_LOG_changeLogTempToChangeLog and OTHER_JOB_deprovisioningDaemon?

We had a Java heap issue late last week which I believe caused the “grouper_change_log_entry_temp” table to keep growing. It’s at 69,886 rows currently while earlier this week it was at 50k. Thanks for any insight.

2 errors in the diagnostic tasks:

DiagnosticLoaderJobTest, Loader job CHANGE_LOG_changeLogTempToChangeLog

DiagnosticLoaderJobTest, Loader job OTHER_JOB_deprovisioningDaemon

Error stack for: loader_CHANGE_LOG_changeLogTempToChangeLog

java.lang.RuntimeException: Cant find a success in job CHANGE_LOG_changeLogTempToChangeLog since: 2018/08/16 14:19:22.000, expecting one in the last 30 minutes

at edu.internet2.middleware.grouper.j2ee.status.DiagnosticLoaderJobTest.doTask(DiagnosticLoaderJobTest.java:175)

at edu.internet2.middleware.grouper.j2ee.status.DiagnosticTask.executeTask(DiagnosticTask.java:78)

at edu.internet2.middleware.grouper.j2ee.status.GrouperStatusServlet.doGet(GrouperStatusServlet.java:180)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.owasp.csrfguard.CsrfGuardFilter.doFilter(CsrfGuardFilter.java:110)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:624)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)

at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:478)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:798)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1441)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:748)

Error stack for: loader_OTHER_JOB_deprovisioningDaemon

java.lang.RuntimeException: Cant find a success in job OTHER_JOB_deprovisioningDaemon, expecting one in the last 3120 minutes

at edu.internet2.middleware.grouper.j2ee.status.DiagnosticLoaderJobTest.doTask(DiagnosticLoaderJobTest.java:173)