Grouper Users - Open Discussion List

["Black, Carey M." <>]

Bert,

 

I understand that all things are temporal.

 

I have often questioned if a “add_Membership” event really shouldn’t always be treated as a "make the user right in a group".

                With the grouper change log consumer being “inside grouper” it is easier to think about doing a quick “is this still the case” check before processing the event. (And doing something to an outside system.)

                If the check does not look like the event is still valid… should it be done? In the general “Message Bug” approach the answer is Yes… but….

 

                I have even considered the idea that it may be wise to “look ahead” for related events (by object, maybe by property too?) and maybe “process them as a unit” as well.

                                if the queue has all of these events at the time of processing: User Add, remove user, add user ,  change user, remove user then should the process not add the user at all? Maybe even kill the other events in the queue?)

                                But that is a long step sideways (AKA: optimization/special case) from the general “Message Bus” approach/design.

 

However, I don’t yet know how to “create a new event in the queue”. ( Any advice / code examples? )

                Short of actually doing a “remove user from group” and “add user to group” in grouper.  (It would work, but no. Let’s not do that. J )

 

I guess I am tripping over the similarities between the “Change log consumers” and  the “Messaging System” in Grouper. ( Maybe I am using the wrong interface/thing? )

 

 

“Messaging System”

                https://spaces.at.internet2.edu/pages/viewpage.action?pageId=14517859#GrouperShell(gsh)-GrouperBuiltinMessaging

                                This has some gsh examples to “Send, receive, acknowledge messages in any message system”.

 

                I guess I could use a change log consumer to send a “grouper message” to a queue. (to “re-Queue” the event. )

                Then maybe use something based on edu.internet2.middleware.grouper.messaging.MessagingListenerToChangeLogConsumer  ?

              /**

              * convert a messaging listener to a change log consumer

              */

                Which appears to let you give it a changeLogConsumerClass class to pass the message back to after “ChangeLogEntriesForMessage = ChangeLogEntry.fromJsonToCollection(grouperMessage.getMessageBody());”

        Hum… seems like a long way around the barn ….. J

 

 

 

                Or am I completely over thinking this…..

                                edu.internet2.middleware.grouper.changeLog.ChangeLogEntry

                                /**

                                * <pre>

                                * represents a user change log record.  This is a change to a record in the DB (insert/update/delete).

                                *

                                 * note: if this object is headed for the temp table, then the getters in the composite key will not be null, will be empty.

                                * this is a hibernate constraint

                                *

                                 * </pre>

                                */

                Sounds like that might be a way to create a new temp change log entry…. ( just need to “convert” the existing event into a new object and save it?)

                                Maybe something like the following ???? :

                                                cle = new ChangeLogEntry();

                                                cle.fromJsonHelper(event_to_requeue.toJson(false(?));  // should that be false or true? Hum….

                     cle.setTempObject(true)

                                                // should I clear any other values? ( IE: “composite key will not be null, will be empty.”     Maybe:  .setId(“”) , changeLogEntryId?? , sequenceNumber ?? … others? )

                                                cle.save()

 

 

                Thoughts?

 

 

Michael,

 

                Yea.. if I had a “real” Message service… then I could do it from an “outside of grouper” message queue. But I don’t have one of those here. L

                Trying to find out if the internal grouper one(s) have a “way” to add events. ( instead of needing to add/remove the thing in grouper to get it to recreate the events. J )

 

--

Carey Matthew

 

From: Bee-Lindgren, Bert <>
Sent: Monday, September 10, 2018 4:56 PM
To: Gettes, Michael <>; Black, Carey M. <>
Cc:
Subject: Re: [grouper-users] ChangeLogConsumer ... resubmit event to back of the queue?

 

(This response is at least slightly related to the "Renaming a folder trigger.")

 

 

I generally have concerns with incremental-event processing and re-queuing/reordering because a membership-add failure might get re-queued and override a successful,subsequent membership-delete. (for example)

 

I think re-queuing should only be done to a different queue with different semantics: The re-queued message no longer means "Add Subject to a Group" but instead means "make a group right" or, perhaps, "make the user right in a group."

 

This is why PSPNG has a low-priority full-sync queue where these retries can go when incremental provisioning doesn't work.

 

Perhaps the Changelog feed should track how many failures a Changelog event experiences and move it to a full-sync (or user-sync-in-a-group) queue?

 

Also, (related to the "Renaming a folder trigger" thread), keep the system simple... there are lots of specialized situations that can occur. However, writing specialized handling of each one means that consumers get more specialized, probably faster, and more complicated. IMNSHO, if errors are infrequent, then it might be okay to use the (slow) Full-Sync hammer to fix all of them. At least, the dual-hammer (fast/incremental & slow/full) approach should be tried, instrumented and evaluated.

 

Getting back to the point of view of a changelog stream, I think this means to supply three streams:

a) The incremental events

b) The Refresh-Subject-In-Group events (for error'ed membership-changes)

c) The Refresh-Group events (for error'ed non-membership-changes)

 

A given consumer should be able to then combine (b&c) if they don't think (b) is worth maintaining.

 

 

---

From: <> on behalf of Gettes, Michael <>
Sent: Monday, September 10, 2018 2:01 PM
To: Black, Carey M.
Cc:
Subject: Re: [grouper-users] ChangeLogConsumer ... resubmit event to back of the queue?

 

I think a lot of what you want to do could be achieved with messaging… all dependent on the message broker you choose, of course.

/mrg

> On Sep 10, 2018, at 1:59 PM, Black, Carey M. <> wrote:
>
> All,
>
> I expect the answer is "yes", but I have yet to identify the "sample code that I am looking for". :)
>
> In some cases, I can see a desire to defer the processing of a specific event ( in the sequence of events) for a ChangeLogConsumer.
>        Is there a designed way to do that in the Grouper API?
>        Maybe a way to track the "number of times" the event was "deferrd"? ( To avoid "stuck events" that "retry forever". )
>
> In my case I can see some asyncronous dependeances that the ChangeLogConsumer would need to "just wait" till the other thing is "done" before it could continue with the event.
> If I could "come back to the event" then I would not have to "block and wait" on the event and hold up other events that could be processed that are in the queue behind the "async depenedent" event.
>
>
> Part of me would even like to "force these events into a different stream" so that they can be picked up by a different listner. ( Then that listner would need to "send the event back" to the first listner too. )
> I may just be asking for to much from the built in ChangeLog/Messaging stuff. ( But I hope not. )
>
>
> Anyone have a clue brick for me?
>
> --
> Carey Matthew
>
>