Grouper Users - Open Discussion List

[Yoann Delattre <>]

Hello,

download link just expire. To make things easier, i created a jira : https://bugs.internet2.edu/jira/browse/GRP-1902

Thanks,

Yoann

Le 11/09/2018 à 10:54, Yoann Delattre a écrit :

First : thanks everyone for your help :)
I'll try to answer as clearly as possible, i'm not really good in english !

Here the structure of the parent folder i renamed :
app:test-pspng-du-10092018 ---> folder renamed
-- app:test-pspng-du-10092018:test-renommage
-- app:test-pspng-du-10092018:habil:test
-- app:test-pspng-du-10092018:habil:test10092018
-- app:test-pspng-du-10092018:habil:test-titi
-- app:test-pspng-du-10092018:habil:test-titi2

So when i renamed the folder, just 2 provisioners should have been affected : pspng_attrFrEduLilHabilitation and pspng_brancheGrouper

Therefore, editing a group's name would do an independent full-sync of that group in each of the provisioning targets the group was selected for.

That's the problem, this is not the behavior I had : a full sync was started for all the provisioners and not only for the groups concerned but for all groups...(so it takes a lot of time...)

If it can help, an extract from my log when the full sync was triggered : https://filesender.renater.fr/?s=download&token=08075bfc-45fb-8fff-095d-4f667adafddf&lang=en

Thanks,
Yoann

Le 10/09/2018 à 21:25, Bee-Lindgren, Bert a écrit :

Presently, PSPNG #21 does a synchronous[1], high-priority full-sync of a group that has had changed attributes/name/etc. Therefore, editing a group's name would do an independent full-sync of that group in each of the provisioning targets the group was selected for.

[1]-Synchronous means that incremental changes wait for that full-sync to occur so that the relevant Changelog event can be Done'd/Error'ed accurately.

There are two ways that I think PSPNG could be optimized:

a) Start the full-sync in the background and 'done' the Changelog row, or [Easy]

b) Create a specialized group-update sync that doesn't need to read all the memberships from the Grouper DB nor the Target system  [Harder, see below]

As background, there are (at least) 5 theoretical types of provisioning invocations that could be written:

1- Incremental changes thrown quickly at the provisioning target, now with decent error recovery if they change happens to already be there

2- Full-Sync of a group's membership attribute(s)

3- Provisioning of a group's non-membership attributes

4- Full-sync following an incremental error or on request (messaging)

5- Fixing a Subject's membership in a Group

The desire for provisioners to be fast but also easy to write has led PSPNG to implement two flows: (1) [The fast flow] and (2-5) [The complete-correction flow]. Creating a third provisioning flow for GroupUpdates (3) will add a noticeable amount of complexity to each Provisioner's implementation. Also, there's a possibility that a renamed group might be hard to find by subsequent incremental changes to that group.

So...

Are you seeing the changelog queues fall behind with slow GroupUpdates of big groups?

If so, would you like to the Changelog events be forwarded to a Best-Effort, Asynchronous, Group-Updating full sync?

Or, is the next step to instrument the synchronous full-syncs so that the logs help understand the effects?

Thanks much,

  Bert

 

---

From: on behalf of Jeffrey Williams
Sent: Monday, September 10, 2018 10:12 AM
To: Yoann Delattre
Cc: ; Grouper-Users
Subject: Re: [grouper-users] [PSPNG] Renaming a folder trigger a full sync

 

It is!  I was curious to see if all of the provisioners normally sync that renamed folder, or did the rename cause an unrelated provisioner, where the folder would be out-of-scope for it, to sync its own groups as well.

The former would make sense since the provisioners run independently of each other and if an event causes a full-sync for one provisioner, it would happen for the others as well.  The latter would be interesting.

Bear in mind: PSPNG runs the full-sync engine continuously in the background to handle events that the incremental cannot.  

For updating stems(i.e. folders), does PSPNG call the full-sync engine to do a targeted sync of the renamed folder?  

On Mon, Sep 10, 2018 at 9:36 AM Yoann Delattre <> wrote:

pspng_attrFrEduLilHabilitation use groupSelectionExpression parameter :

changeLog.consumer.pspng_attrFrEduLilHabilitation.groupSelectionExpression = ${name.startsWith("app:") && name.contains(":habil:")}

pspng_brancheGrouper : provision_to attribute is set on practically all root folder.

pspng_brancheListes : provision_to attribute is only set on a specific folder.

i'm not sure if this is what you're asking for ?

Anyway thanks for your help !
Yoann

Le 10/09/2018 à 15:22, Jeffrey Williams a écrit :

I'm also curious as to how you're setting your provisioning attributes: immediate folder, parent or *parent folder?  Individual groups?

Thanks,

Jeff

On Mon, Sep 10, 2018 at 9:20 AM Yoann Delattre <> wrote:

with the last patch : 21.

Thanks,
Yoann

Le 10/09/2018 à 15:15, Julio Macavilca a écrit :

Just curious, in which patch are you noticing this?

Thanks,

Julio

On Mon, Sep 10, 2018 at 9:02 AM, Yoann Delattre <> wrote:

Hello,

I'm currently testing PSPNG.
During my tests, rename a folder (with some groups in it) triggered a full sync for all the provisioners :

2018-09-10 13:57:00,594: [DefaultQuartzScheduler_Worker-1] INFO FullSyncProvisioner.queueAllGroupsForFullSync(551) -  - FullSyncer(pspng_attrFrEduLilHabilitation): Queuing all groups for full sync. (Work item invokes full sync: ProvisioningWorkItem[done=false,clog=clog #3931352 / ChangeLog type: stem: updateStem])
2018-09-10 13:57:00,613: [DefaultQuartzScheduler_Worker-2] INFO FullSyncProvisioner.queueAllGroupsForFullSync(551) -  - FullSyncer(pspng_brancheListes): Queuing all groups for full sync. (Work item invokes full sync: ProvisioningWorkItem[done=false,clog=clog #3931352 / ChangeLog type: stem: updateStem])
2018-09-10 13:57:02,047: [DefaultQuartzScheduler_Worker-3] INFO FullSyncProvisioner.queueAllGroupsForFullSync(551) -  - FullSyncer(pspng_brancheGrouper): Queuing all groups for full sync. (Work item invokes full sync: ProvisioningWorkItem[done=false,clog=clog #3931352 / ChangeLog type: stem: updateStem])

Is this the expected behavior ?

i'm a little concern because if a full sync of all provisioners is started during work time, provisioning will take more time than usual.

Thanks for your help,
Yoann

--

Jeffrey Williams 

Identity Engineer
Identity & Access Services
https://its.uncg.edu

--

Jeffrey Williams 

Identity Engineer
Identity & Access Services
https://its.uncg.edu