I appreciate people want to do this with routing keys. But there are also cases where it needs to be in grouper (or people just prefer it that way), so we should have an answer eventually for both.
Thanks
Chris
From: <>
On Behalf Of Gettes, Michael
Sent: Wednesday, January 30, 2019 11:31 PM
To: Black, Carey M. <>
Cc: ; Vivek Sachdeva <>
Subject: Re: [grouper-users] RabbitMq Routing
Sure - I’ll give it a try.
The change Vivek is making is derived from the Unicon code commissioned by University of Hawaii. The routingKey is set to be the full group stem and : is replaced by . causing A:B:C groupName to be A.B.C - you note the 255 length limit
- I am not concerned about that. So, no attribute to set in grouper. I appreciate how it might be desirable to configure all this within grouper, but it is the groupName that is set and in the RabbitMQ administration UI you would go into something like amq.topic
(or some other topic definition) and specify A.B.# goes to Queue Q3. So all messages for the group pattern A:B:* would be sent to Q3. The AMQP pattern matching on the routingKey is specified as A.B.# , not as intuitive. It’s really just that simple.
If we really wanted to put this into attributes on the grouper side, you could implement this with messages. Attribute messages of a particular naming could be consumed and those are interpreted in the appropriate RabbitMQ API calls to
automate the topic routing instructions. It’s possible. You could do the same for Queue definitions as well. Use grouper group names to manage queues and topic routing over in Rabbit. Given the graphing capabilities in grouper - coming soon - this means
you could graphically see how messages are routed based on grouper config and know it was implemented by messages sent. Hmmm. Well, at least that’s how I would think about it.
On Jan 30, 2019, at 10:26 AM, Black, Carey M. <> wrote:
Can you expand on this point a bit more for me/us? In generally it seems fine and good, but I think there are details there that I am not seeing how to operationalize yet.
RE: “A single message from Grouper can become dozens based on this configuration *and* you get to control queues/consumers only getting the messages they are supposed to receive.”
Can you provide any more specifics about how the Message Broker makes that decision?
How to maintain that configuration ( I assume ) in the Message Broker?
Or is it “manually done” somehow?
Is there an attribute based model that might allow that to be driven from Grouper data/config?
Can/Does attributes attached to the object ( or an ancestor of the object) get sent to the broker by default?
Can that attribute data be included in the “routingKey”? ( maybe not…
AMQP 0.9.1 has a routingKey max length of 255 chars. ( Which will be too small for some GroupNames.))
Maybe an multi-valued attribute could be used to create a message for N “routingKey values” from the grouper side?
(ALA: AttributeDef = etc:attribute:MessagingMQ:SendToRoutingKeysDef, AttributeName=routingKeys, multi valued, single assigned, to Stems,groups,attirbutes
Each value found produces a “clone message” with that routingKey value from the attribute value. )
Lacking value(s) found, the “normal/default” routingKey value would be produced/used.
From: <> On
Behalf Of Gettes, Michael
Sent: Wednesday, January 30, 2019 7:48 AM
To: Vivek Sachdeva <>
Cc: Crawford, Jeffrey <>; Ruch,Jeff <>;
Chris Hyzer <>; Julio Polo <>;
Subject: Re: [grouper-users] RabbitMq Routing
Here’s how I would recommend use of messaging, especially with RabbitMQ but ActiveMQ can do it too and I believe many other message brokers have similar capabilities…
Configure a grouper changelog consumer for message publishing for each message “category” you’d like to send. A “category” would be based on what content you want in the
message. To some message consumers you’d like to send them only eduPersonPrincipalName but to others you’d like to send ePPN and email. Each of those would cause you to configure a separate changelog consumer sending messages to the broker. A category can
also differentiate sending only membership related events vs. privilege changes or other messages related to the group. For each category - I recommend sending to a topic, not a queue, and include the groupName in the routingKey for RabbitMQ (or equiv for
your broker) and then you can easily split-out to many different queues on the broker based on the routingKey name. A single message from Grouper can become dozens based on this configuration *and* you get to control queues/consumers only getting the messages
they are supposed to receive.
I further recommend you use RabbitMQ for use by Grouper to get the above capability and, currently, it comes with the TIER packaging. You can always link a RabbitMQ broker
to another broker by MQ style messaging or easily write a consumer which resends the message onto your other message broker. This would give you the best of capabilities out of grouper and integrate your existing non-AMQP capable messaging system.
With this overall mindset of how to configure grouper and the broker you obtain complete control of who gets what messages and doing so rather efficiently. I can generate
10,000 messages from grouper to multiple queues with varied requirements on what they receive in a matter of 45 seconds - all on my laptop. That’s slick. Of course, this is with the latest version of grouper and patches supporting better subject caching
and so on. Really looking forward to Vivek’s latest changes so I can change my config to something even simpler and more powerful.
On Jan 30, 2019, at 12:11 AM, Vivek Sachdeva <> wrote:
Are you planning to use queues or topics? If only queues, then everything is already implemented. If topics, then it's almost done. I just need to merge a branch into master
and publish a patch.
On Tue, Jan 29, 2019 at 6:29 PM Crawford, Jeffrey <> wrote:
Did anything come of this? We have an upcoming project where if we could set up MQ subscription based on folder paths. We could lock down group message changes to specific
queue access.
We don’t want apps to get group membership changes for groups they don’t have business seeing. However we would like to use a queue as the subscription endpoint for grouper
data.
Does that make sense and is that what is being discussed here?
From: <>
on behalf of "Ruch,Jeff" <>
Date: Sunday, December 30, 2018 at 4:27 PM
To: "Gettes, Michael" <>, Chris Hyzer <>
Cc: Julio Polo <>, Vivek Sachdeva <>,
"" <>
Subject: Re: [grouper-users] RabbitMq Routing
Great News!! Let me know when you are ready for me to test.
From: Gettes,
Michael <>
Sent: Saturday, December 29, 2018 9:03:22 PM
To: Chris Hyzer
Cc: Julio Polo; Ruch,Jeff; Vivek Sachdeva;
Subject: Re: [grouper-users] RabbitMq Routing
Wonderful news! Vivek - THANK YOU!
A Happy and Healthy New Year to the entire Grouper Dev team and everyone else!
Thank you for all you do.
On Dec 29, 2018, at 10:13 PM, Hyzer, Chris <> wrote:
Vivek has done most of this and as we reviewed it, we noticed that the Unicon part had spring _expression_ substitution, but Grouper generally uses EL which can do similar
things. I assume that’s not controversial to not add another way to configure expressions… anyways, almost done with this…
From: Julio
Polo <>
Sent: Wednesday, December 19, 2018 2:19 PM
To: Gettes, Michael <>
Cc: Ruch,Jeff <>; Vivek Sachdeva <>;
Hyzer, Chris <>;
Subject: Re: [grouper-users] RabbitMq Routing
Hi Chris,
In case it helps (ie I implore you to implement this :-), here are the specs we gave to Unicon when we commissioned the work to publish Grouper messages to a topic exchange:
The JSON messages will be published to a RabbitMQ topic exchange. The message routing key will be based on the group ID path (a colon-delimited path) and should be configurable in a properties files. This property will consist of one or more regular expressions,
which when matched will lead to a substitution command that is applied to the matched group ID path. Colons should always be converted into a period, even if that is not specified in the substitution command. Here is an example using vi syntax:
s/^hawaii.edu:\(auto:sis:registration:.*\)(enrolled|waitlisted|withdrawn)$|group.modify./\1/
so if group in message is:
hawaii.edu:auto:sis:registration:MAN:MATH:100:123456:201410:enrolled
the message routing key for the message would be:
group.modify.auto.sis.registration.MAN.MATH.100.123456.201410
If no regular expressions are defined, simply substitute colons with periods.
If a JSON message has no group ID path associated with it, the properties file should allow us to use regular _expression_ as done above to map strings from the JSON message into a message routing key. For example:
s/"setAttribute"/attribute.set/
On Wed, Dec 19, 2018 at 7:27 AM Gettes, Michael <> wrote:
Please provide support (or continue the support) for sending directly to a topic exchange. The grouper code does it now and so does the Hawaii code. As you note, setting
group name into the routing key will go a long way.
Chris, do you want me to dig up the email from a few months ago where this was all outlined or do you still have it?
On Dec 19, 2018, at 12:16 PM, Ruch,Jeff <> wrote:
I believe if you can set the path of the group as the routing key, that is all that is needed.
You should then be able to use rabbitmq shovel to move the messages from the grouper configured queue to a topic exchange. The topic exchange will allow the routing of
messages based on the routing key. It supports wildcard filtering as well.
From: Vivek
Sachdeva <>
Sent: Tuesday, December 18, 2018 9:35:53 PM
To: Hyzer, Chris
Cc: Gettes, Michael; Ruch,Jeff;
Subject: Re: [grouper-users] RabbitMq Routing
Yes, routing key is already supported but adding group name is not there yet.
On Tue, Dec 18, 2018 at 7:30 PM Hyzer, Chris <> wrote:
Vivek, isn’t this already in Grouper? Or is the gap having the group name dynamically put in the message? Let us know what is missing and we might be able to fix asap.
Thanks
Chris
-----Original Message-----
From: <>
On Behalf Of Gettes, Michael
Sent: Monday, December 17, 2018 12:51 PM
To: Jeff <>
Cc:
Subject: Re: [grouper-users] RabbitMq Routing
The built-in to Grouper code doesn’t have this ability but the Hawaii code (they commissioned Unicon) does have the ability to pass the group name in the routing code. I have requested the Unicon code be combined with the built-in code. Not sure where all
this stands as this was part of the request prior to TechEx regarding “what do people want”.
The Hawaii code I am referring to is at https://github.com/Unicon/grouper-amqp-esb-publisher
/mrg
> On Dec 17, 2018, at 12:38 PM, Jeff <> wrote:
>
>
> Hi,
>
> I would like Grouper to publish to RabbitMQ queues based on Grouper folder
> path. I believe this is supported in RabbitMQ using a topic exchange and
> routing keys. Currently, Grouper publishes directly to a RabbitMQ queue.
>
> Is there a way to do this either through configuration, or through another
> approach?
>
> Thanks,
>
> Jeff Ruch
> Colorado State University
|
