Grouper Users - Open Discussion List

[Scott Koranda <>]

Hi,

Before receiving your note below, I found this wiki page

https://spaces.at.internet2.edu/display/Grouper/Grouper+-+Loader

I then stopped the loader and using GSH ran

GrouperSession.startRootSession();
GrouperLoader.schedulerFactory().getScheduler().unscheduleJob(org.quartz.TriggerKey.triggerKey("triggerChangeLog_grouperChangeLogTempToChangeLog"));

I restarted the loader and saw that the ChangeLogTempToChangeLog jobs
were running as expected.

I then stopped the loader again and looked at the grouper_QZ_TRIGGERS
table. I did not find anything else in the BLOCKED state, but I found
others in the ACQUIRED state, even though they had not run for quite
some time.

So I then ran using GSH

GrouperLoader.schedulerFactory().getScheduler().unscheduleJob(org.quartz.TriggerKey.triggerKey("triggerChangeLog_CHANGE_LOG_consumer_pspng_attributes"));

GrouperLoader.schedulerFactory().getScheduler().unscheduleJob(org.quartz.TriggerKey.triggerKey("triggerChangeLog_CHANGE_LOG_consumer_pspng_groupOfNames"));

GrouperLoader.schedulerFactory().getScheduler().unscheduleJob(org.quartz.TriggerKey.triggerKey("triggerChangeLog_CHANGE_LOG_consumer_grouperRules"));

GrouperLoader.schedulerFactory().getScheduler().unscheduleJob(org.quartz.TriggerKey.triggerKey("triggerChangeLog_CHANGE_LOG_consumer_syncGroups"));

That left only triggers in the WAITING state.

I then started the loader again, and now I see in grouper_loader_log the
sequence of jobs I expect for the change log and the PSP-NG.

More below inline...

> There’s a known issue with quartz where jobs may get stuck in a BLOCKED 
> state.  There’s a separate job whose purpose is to fix those (every 30 
> minutes).  Maybe that got stuck too?  Do you have anything else that’s 
> blocked?

I belive it is this:

https://github.com/quartz-scheduler/quartz/pull/146

It looks like that is scheduled to be included in a release relatively
soon. 

Will it be possible to release a patch for 2.3.x that upgrades the quartz jar?

> To manually run the job that unblocks it:
> 
> groovy:000> loaderRunOneJob("OTHER_JOB_schedulerCheckDaemon")

I get this:

groovy:000> GrouperSession.startRootSession();
===> 3d29e472266e4c53a008e10d2de31b63,'GrouperSystem','application'
groovy:000> loaderRunOneJob("OTHER_JOB_schedulerCheckDaemon")
ERROR java.lang.RuntimeException:
java.lang.RuntimeException: Problem loading class: null
        at 
edu.internet2.middleware.grouper.app.loader.GrouperLoader.runOnceByJobName 
(GrouperLoader.java:1625)
        at edu.internet2.middleware.grouper.app.gsh.loaderRunOneJob.invoke 
(loaderRunOneJob.java:95)
        at 
edu.internet2.middleware.grouper.app.gsh.loaderRunOneJob$invoke.call (Unknown 
Source)
        at groovysh_evaluate.loaderRunOneJob (groovysh_evaluate:4)

I don't see any quartz trigger with that name:

MySQL [grouper]> select job_name from grouper_QZ_TRIGGERS;
+----------------------------------------+
| job_name                               |
+----------------------------------------+
| CHANGE_LOG_changeLogTempToChangeLog    |
| CHANGE_LOG_consumer_grouperRules       |
| CHANGE_LOG_consumer_pspng_attributes   |
| CHANGE_LOG_consumer_pspng_groupOfNames |
| CHANGE_LOG_consumer_syncGroups         |
| MAINTENANCE_cleanLogs                  |
| MAINTENANCE__builtinMessagingDaemon    |
| MAINTENANCE__enabledDisabled           |
| MAINTENANCE__rules                     |
| OTHER_JOB_attestationDaemon            |
| OTHER_JOB_deprovisioningDaemon         |
| OTHER_JOB_pspng_attributes_full        |
| OTHER_JOB_pspng_groupOfNames_full      |
+----------------------------------------+
13 rows in set (0.00 sec)

Any ideas?

Thanks,

Scott

> It just basically changes the state from BLOCKED to WAITING for anything 
> that is not currently firing (i.e. actually blocked).  Does that fix the 
> problem?
> 
> Thanks!
> 
> - Shilen
> 
> On 4/29/19, 8:53 PM, "Scott Koranda" <> wrote:
> 
>     Hi,
>     
>     > Are any of your other jobs (that run via the daemon) running 
> correctly? 
>     
>     The only other daemon jobs would be the PSP-NG jobs that follow the
>     changelog. No, those jobs are not running correctly. They appear to have
>     stopped 5 days ago based on the last entries in grouper_loader_log.
>     
>     > What do the quartz tables show?
>     >
>     > select * from grouper_qz_triggers where 
> trigger_name='triggerChangeLog_grouperChangeLogTempToChangeLog'
>     > 
>     > select * from grouper_qz_fired_triggers where 
> trigger_name='triggerChangeLog_grouperChangeLogTempToChangeLog'
>     
>     MySQL [grouper]> select * from grouper_QZ_TRIGGERS where 
> trigger_name='triggerChangeLog_grouperChangeLogTempToChangeLog'\G
>     *************************** 1. row ***************************
>         sched_name: DefaultQuartzScheduler
>       trigger_name: triggerChangeLog_grouperChangeLogTempToChangeLog
>      trigger_group: DEFAULT
>           job_name: CHANGE_LOG_changeLogTempToChangeLog
>          job_group: DEFAULT
>        description: NULL
>     next_fire_time: 1556569425000
>     prev_fire_time: 1556569365000
>           priority: 5
>      trigger_state: BLOCKED
>       trigger_type: CRON
>         start_time: 1551790380000
>           end_time: 0
>      calendar_name: NULL
>      misfire_instr: 0
>           job_data: 
>     1 row in set (0.00 sec)
>     
>     MySQL [grouper]> select * from grouper_QZ_FIRED_TRIGGERS where 
> trigger_name='triggerChangeLog_grouperChangeLogTempToChangeLog'\G
>     Empty set (0.00 sec)
>     
>     Thanks,
>     
>     Scott
>     
>