Grouper Users - Open Discussion List

["Bee-Lindgren, Bert" <>]

(This response is at least slightly related to the "Renaming a folder trigger.")

I generally have concerns with incremental-event processing and re-queuing/reordering because a membership-add failure might get re-queued and override a successful,subsequent membership-delete. (for example)

I think re-queuing should only be done to a different queue with different semantics: The re-queued message no longer means "Add Subject to a Group" but instead means "make a group right" or, perhaps, "make the user right in a group."

This is why PSPNG has a low-priority full-sync queue where these retries can go when incremental provisioning doesn't work.

Perhaps the Changelog feed should track how many failures a Changelog event experiences and move it to a full-sync (or user-sync-in-a-group) queue?

Also, (related to the "Renaming a folder trigger" thread), keep the system simple... there are lots of specialized situations that can occur. However, writing specialized handling of each one means that consumers get more specialized, probably faster, and more complicated. IMNSHO, if errors are infrequent, then it might be okay to use the (slow) Full-Sync hammer to fix all of them. At least, the dual-hammer (fast/incremental & slow/full) approach should be tried, instrumented and evaluated.

Getting back to the point of view of a changelog stream, I think this means to supply three streams:

a) The incremental events

b) The Refresh-Subject-In-Group events (for error'ed membership-changes)

c) The Refresh-Group events (for error'ed non-membership-changes)

A given consumer should be able to then combine (b&c) if they don't think (b) is worth maintaining.

---

From: <> on behalf of Gettes, Michael <>
Sent: Monday, September 10, 2018 2:01 PM
To: Black, Carey M.
Cc:
Subject: Re: [grouper-users] ChangeLogConsumer ... resubmit event to back of the queue?

 

I think a lot of what you want to do could be achieved with messaging… all dependent on the message broker you choose, of course.

/mrg

> On Sep 10, 2018, at 1:59 PM, Black, Carey M. <> wrote:
>
> All,
>
> I expect the answer is "yes", but I have yet to identify the "sample code that I am looking for". :)
>
> In some cases, I can see a desire to defer the processing of a specific event ( in the sequence of events) for a ChangeLogConsumer.
>        Is there a designed way to do that in the Grouper API?
>        Maybe a way to track the "number of times" the event was "deferrd"? ( To avoid "stuck events" that "retry forever". )
>
> In my case I can see some asyncronous dependeances that the ChangeLogConsumer would need to "just wait" till the other thing is "done" before it could continue with the event.
> If I could "come back to the event" then I would not have to "block and wait" on the event and hold up other events that could be processed that are in the queue behind the "async depenedent" event.
>
>
> Part of me would even like to "force these events into a different stream" so that they can be picked up by a different listner. ( Then that listner would need to "send the event back" to the first listner too. )
> I may just be asking for to much from the built in ChangeLog/Messaging stuff. ( But I hope not. )
>
>
> Anyone have a clue brick for me?
>
> --
> Carey Matthew
>
>