Grouper Users - Open Discussion List

[Yoann Delattre <>]

Hello Bert,

thanks for your answer !

I will try the "coordinationTimeout_sec" solution until i'm able to upgrade (soon, i hope !)

Yoann

Le 28/02/2019 à 18:43, Bee-Lindgren, Bert a écrit :

The updateStem operations are going to do full-syncs, and these are going to happen synchronously with 2.3, and (new) 2.4Patch3 moves these full-syncs away from the changelog stream (so changelog processing continues unabated).

Until GRP-1902 is addressed or until you're able to upgrade, there are a couple choices:

  - Reduce the effects of the full-syncs by reducing/removing the locking that happens between Full & Incremental syncs: coordinationTimeout_secs=0. This will increase some log noise and incrementals and fulls conflict and have to retry their operations, but will keep them from stalling each other.

  - Do as you suggest and skip over the updateStem changes if they really don't change anything you care about.

Hoping this helps,

  Bert

---

From: on behalf of Yoann Delattre
Sent: Thursday, February 28, 2019 3:07 AM
To:
Subject: Re: [grouper-users] [PSPNG] Cannot start FullSync of .... Incremental provisioning underway since...

 

I checked the changelog and it seems that there is a lot of "updateStem" changelog type.
According to this issue, each one will trigger a full-sync.

So i'm thinking to stop the grouperDaemon, force a full-sync for all provisioner and skip all this "updateStem" changelog type by modifying column "last_sequence_processed" in database

What do you think ?

Thanks,
Yoann.

Le 28/02/2019 à 08:39, Yoann Delattre a écrit :

Hello,

since this morning :

2019-02-28 08:24:42,092: [FullSyncer(pspng_brancheGrouper)-Thread] WARN  ProvisionerCoordinator$ProvisioningStatus.lockForFullSyncWhenNoIncrementalIsUnderway(73) -  - pspng_brancheGrouper: Cannot start FullSync of services:15040:direction. Incremental provisioning underway since Thu Feb 28 07:15:42 CET 2019. We'll give up and move ahead anyway in 259 seconds.

same thing that yesterday but with new lock on the same groups...it's endless :-/

meanwhile, the sequences to be processed not decreased at all for this provisioner :

What can i do ?

Thanks for your help,

Yoann

Le 27/02/2019 à 15:17, Yoann Delattre a écrit :

Hello,

in the log, i get messages like this :

WARN  ProvisionerCoordinator$ProvisioningStatus.lockForFullSyncWhenNoIncrementalIsUnderway(73) -  - pspng_brancheGrouper: Cannot start FullSync of services:12944:tous. Incremental provisioning underway since Wed Feb 27 11:33:36 CET 2019. We'll give up and move ahead anyway in 109 seconds.

that's correct, some groups are locked (around 500):

2019-02-27 11:33:36,685: [DefaultQuartzScheduler_Worker-1] INFO  ProvisionerCoordinator$ProvisioningStatus.lockForIncrementalProvisioningWhenNoFullSyncIsUnderway(139) -  - pspng_brancheGrouper: Locking group before incremental sync: services:12943:tous
2019-02-27 11:33:36,685: [DefaultQuartzScheduler_Worker-1] INFO  ProvisionerCoordinator$ProvisioningStatus.lockForIncrementalProvisioningWhenNoFullSyncIsUnderway(139) -  - pspng_brancheGrouper: Locking group before incremental sync: services:12944:direction
2019-02-27 11:33:36,685: [DefaultQuartzScheduler_Worker-1] INFO  ProvisionerCoordinator$ProvisioningStatus.lockForIncrementalProvisioningWhenNoFullSyncIsUnderway(139) -  - pspng_brancheGrouper: Locking group before incremental sync: services:12944:secretaires
2019-02-27 11:33:36,685: [DefaultQuartzScheduler_Worker-1] INFO  ProvisionerCoordinator$ProvisioningStatus.lockForIncrementalProvisioningWhenNoFullSyncIsUnderway(139) -  - pspng_brancheGrouper: Locking group before incremental sync: services:12944:tous
2019-02-27 11:33:36,685: [DefaultQuartzScheduler_Worker-1] INFO  ProvisionerCoordinator$ProvisioningStatus.lockForIncrementalProvisioningWhenNoFullSyncIsUnderway(139) -  - pspng_brancheGrouper: Locking group before incremental sync: services:12945:direction
2019-02-27 11:33:36,685: [DefaultQuartzScheduler_Worker-1] INFO  ProvisionerCoordinator$ProvisioningStatus.lockForIncrementalProvisioningWhenNoFullSyncIsUnderway(139) -  - pspng_brancheGrouper: Locking group before incremental sync: services:12945:secretaires
2019-02-27 11:33:36,685: [DefaultQuartzScheduler_Worker-1] INFO  ProvisionerCoordinator$ProvisioningStatus.lockForIncrementalProvisioningWhenNoFullSyncIsUnderway(139) -  - pspng_brancheGrouper: Locking group before incremental sync: services:12945:tous
2019-02-27 11:33:36,685: [DefaultQuartzScheduler_Worker-1] INFO  ProvisionerCoordinator$ProvisioningStatus.lockForIncrementalProvisioningWhenNoFullSyncIsUnderway(139) -  - pspng_brancheGrouper: Locking group before incremental sync: services:12946:direction
...

And then shortly after, PSPNG started a full-sync :

2019-02-27 11:34:42,611: [DefaultQuartzScheduler_Worker-1] INFO  Provisioner.workItemShouldBeHandledByFullSyncOfEverything(1767) -  - pspng_brancheGrouper: Change might change group selection: ProvisioningWorkItem[done=false,clog=clog #90287666 / ChangeLog type: stem: updateStem]
2019-02-27 11:34:42,611: [DefaultQuartzScheduler_Worker-1] INFO  Provisioner.provisionItem(1093) -  - pspng_brancheGrouper: Performing sync of all groups to process work item: ProvisioningWorkItem[done=false,clog=clog #90287666 / ChangeLog type: stem: updateStem]
2019-02-27 11:34:42,612: [DefaultQuartzScheduler_Worker-1] INFO  FullSyncProvisioner.queueAllGroupsForFullSync(551) -  - FullSyncer(pspng_brancheGrouper): Queuing all groups for full sync. (Work item invokes full sync: ProvisioningWorkItem[done=false,clog=clog #90287666 / ChangeLog type: stem: updateStem])

Also the log say "We'll give up and move ahead anyway in 300 seconds." but when he reach 300 seconds, he start over. And only after 600 seconds (300 x 2), PSPNG start the full-sync.

2019-02-27 13:24:11,518: [FullSyncer(pspng_brancheGrouper)-Thread] WARN  ProvisionerCoordinator$ProvisioningStatus.lockForFullSyncWhenNoIncrementalIsUnderway(73) -  - pspng_brancheGrouper: Cannot start FullSync of services:12944:tous. Incremental provisioning underway since Wed Feb 27 11:33:36 CET 2019. We'll give up and move ahead anyway in 9 seconds.
2019-02-27 13:24:21,521: [FullSyncer(pspng_brancheGrouper)-Thread] WARN  ProvisionerCoordinator$ProvisioningStatus.lockForFullSyncWhenNoIncrementalIsUnderway(73) -  - pspng_brancheGrouper: Cannot start FullSync of services:12944:tous. Incremental provisioning underway since Wed Feb 27 11:33:36 CET 2019. We'll give up and move ahead anyway in 300 seconds.

Is that a normal behavior ?

Thanks for your help,

Yoann.

PSP : I use PSPNG 2.3 Patch 21

--
Yoann Delattre ✆ 03 20 95 69 10 ✉ Équipe SIAD (Systèmes d'Information et Aide à la Décision)
DSI de l'académie de Lille (Direction des Systèmes d'Information)
110 avenue Gaston Berger - 59000 Lille