Grouper Users - Open Discussion List

["Black, Carey M." <>]

Michael,

 

Can you expand on this point a bit more for me/us? In generally it seems fine and good, but I think there are details there that I am not seeing how to operationalize yet.

                RE: “A single message from Grouper can become dozens based on this configuration *and* you get to control queues/consumers only getting the messages they are supposed to receive.”

 

                Can you provide any more specifics about how the Message Broker makes that decision?

                                How to maintain that configuration ( I assume ) in the Message Broker?

                                Or is it “manually done” somehow?

 

Maybe….

                Is there an attribute based model that might allow that to be driven from Grouper data/config?

                                Can/Does attributes attached to the object ( or an ancestor of the object) get sent to the broker by default?

                                Can that attribute data be included in the “routingKey”? ( maybe not… AMQP 0.9.1 has a routingKey max length of 255 chars. ( Which will be too small for some GroupNames.))

                                Maybe an multi-valued attribute could be used to create a message for N “routingKey values” from the grouper side?

                                                (ALA: AttributeDef = etc:attribute:MessagingMQ:SendToRoutingKeysDef, AttributeName=routingKeys, multi valued, single assigned, to Stems,groups,attirbutes

                                                Each value found produces a “clone message” with that routingKey value from the attribute value. )

                                                Lacking value(s) found, the “normal/default” routingKey value would be produced/used.

 

--

Carey Matthew

 

From: <> On Behalf Of Gettes, Michael
Sent: Wednesday, January 30, 2019 7:48 AM
To: Vivek Sachdeva <>
Cc: Crawford, Jeffrey <>; Ruch,Jeff <>; Chris Hyzer <>; Julio Polo <>;
Subject: Re: [grouper-users] RabbitMq Routing

 

Here’s how I would recommend use of messaging, especially with RabbitMQ but ActiveMQ can do it too and I believe many other message brokers have similar capabilities…

 

Configure a grouper changelog consumer for message publishing for each message “category” you’d like to send.  A “category” would be based on what content you want in the message.  To some message consumers you’d like to send them only eduPersonPrincipalName but to others you’d like to send ePPN and email.  Each of those would cause you to configure a separate changelog consumer sending messages to the broker.  A category can also differentiate sending only membership related events vs. privilege changes or other messages related to the group.  For each category - I recommend sending to a topic, not a queue, and include the groupName in the routingKey for RabbitMQ (or equiv for your broker) and then you can easily split-out to many different queues on the broker based on the routingKey name.  A single message from Grouper can become dozens based on this configuration *and* you get to control queues/consumers only getting the messages they are supposed to receive.

 

I further recommend you use RabbitMQ for use by Grouper to get the above capability and, currently, it comes with the TIER packaging.  You can always link a RabbitMQ broker to another broker by MQ style messaging or easily write a consumer which resends the message onto your other message broker.  This would give you the best of capabilities out of grouper and integrate your existing non-AMQP capable messaging system.

 

With this overall mindset of how to configure grouper and the broker you obtain complete control of who gets what messages and doing so rather efficiently.  I can generate 10,000 messages from grouper to multiple queues with varied requirements on what they receive in a matter of 45 seconds - all on my laptop.  That’s slick.  Of course, this is with the latest version of grouper and patches supporting better subject caching and so on.  Really looking forward to Vivek’s latest changes so I can change my config to something even simpler and more powerful.

 

/mrg

On Jan 30, 2019, at 12:11 AM, Vivek Sachdeva <> wrote:

 

Hi Jeffrey,

 

Are you planning to use queues or topics? If only queues, then everything is already implemented. If topics, then it's almost done. I just need to merge a branch into master and publish a patch.

 

Thanks,

Vivek

 

 

On Tue, Jan 29, 2019 at 6:29 PM Crawford, Jeffrey <> wrote:

Hi,

 

Did anything come of this? We have an upcoming project where if we could set up MQ subscription based on folder paths. We could lock down group message changes to specific queue access.

 

We don’t want apps to get group membership changes for groups they don’t have business seeing. However we would like to use a queue as the subscription endpoint for grouper data.

 

Does that make sense and is that what is being discussed here?

 

Thanks

Jeffrey C.

 

 

From: <> on behalf of "Ruch,Jeff" <>
Date: Sunday, December 30, 2018 at 4:27 PM
To: "Gettes, Michael" <>, Chris Hyzer <>
Cc: Julio Polo <>, Vivek Sachdeva <>, "" <>
Subject: Re: [grouper-users] RabbitMq Routing

 

Great News!!  Let me know when you are ready for me to test.

 

Jeff

---

From: Gettes, Michael <>
Sent: Saturday, December 29, 2018 9:03:22 PM
To: Chris Hyzer
Cc: Julio Polo; Ruch,Jeff; Vivek Sachdeva;
Subject: Re: [grouper-users] RabbitMq Routing

 

Wonderful news!  Vivek - THANK YOU!

 

A Happy and Healthy New Year to the entire Grouper Dev team and everyone else!

 

Thank you for all you do.

 

/mrg

 

On Dec 29, 2018, at 10:13 PM, Hyzer, Chris <> wrote:

 

Vivek has done most of this and as we reviewed it, we noticed that the Unicon part had spring _expression_ substitution, but Grouper generally uses EL which can do similar things.  I assume that’s not controversial to not add another way to configure expressions…  anyways, almost done with this…

 

Thanks

Chris

 

From: Julio Polo <> 
Sent: Wednesday, December 19, 2018 2:19 PM
To: Gettes, Michael <>
Cc: Ruch,Jeff <>; Vivek Sachdeva <>; Hyzer, Chris <>;
Subject: Re: [grouper-users] RabbitMq Routing

 

Hi Chris,

In case it helps (ie I implore you to implement this :-), here are the specs we gave to Unicon when we commissioned the work to publish Grouper messages to a topic exchange:

The JSON messages will be published to a RabbitMQ topic exchange. The message routing key will be based on the group ID path (a colon-­delimited path) and should be configurable in a properties files. This property will consist of one or more regular expressions, which when matched will lead to a substitution command that is applied to the matched group ID path. Colons should always be converted into a period, even if that is not specified in the substitution command. Here is an example using vi syntax:

s/^hawaii.edu:\(auto:sis:registration:.*\)(enrolled|waitlisted|withdrawn)$|group.modify./\1/

 so if group in message is:

  hawaii.edu:auto:sis:registration:MAN:MATH:100:123456:201410:enrolled

 the message routing key for the message would be:

  group.modify.auto.sis.registration.MAN.MATH.100.123456.201410


If no regular expressions are defined, simply substitute colons with periods.

If a JSON message has no group ID path associated with it, the properties file should allow us to use regular _expression_ as done above to map strings from the JSON message into a message routing key. For example:

s/"setAttribute"/attribute.set/

-julio

 

On Wed, Dec 19, 2018 at 7:27 AM Gettes, Michael <> wrote:

Please provide support (or continue the support) for sending directly to a topic exchange.  The grouper code does it now and so does the Hawaii code.  As you note, setting group name into the routing key will go a long way.

 

Chris, do you want me to dig up the email from a few months ago where this was all outlined or do you still have it?

 

/mrg

 

On Dec 19, 2018, at 12:16 PM, Ruch,Jeff <> wrote:

 

I believe if you can set the path of the group as the routing key, that is all that is needed.

 

You should then be able to use rabbitmq shovel to move the messages from the grouper configured queue to a topic exchange.  The topic exchange will allow the routing of messages based on the routing key.  It supports wildcard filtering as well.

 

Jeff

 

 

 

---

From: Vivek Sachdeva <>
Sent: Tuesday, December 18, 2018 9:35:53 PM
To: Hyzer, Chris
Cc: Gettes, Michael; Ruch,Jeff; 
Subject: Re: [grouper-users] RabbitMq Routing

 

Yes, routing key is already supported but adding group name is not there yet.

 

Thanks,

Vivek

 

On Tue, Dec 18, 2018 at 7:30 PM Hyzer, Chris <> wrote:

Vivek, isn’t this already in Grouper?  Or is the gap having the group name dynamically put in the message?  Let us know what is missing and we might be able to fix asap.

Thanks
Chris



-----Original Message-----
From:  <> On Behalf Of Gettes, Michael
Sent: Monday, December 17, 2018 12:51 PM
To: Jeff <>
Cc: 
Subject: Re: [grouper-users] RabbitMq Routing

The built-in to Grouper code doesn’t have this ability but the Hawaii code (they commissioned Unicon) does have the ability to pass the group name in the routing code.  I have requested the Unicon code be combined with the built-in code.  Not sure where all this stands as this was part of the request prior to TechEx regarding “what do people want”.

The Hawaii code I am referring to is at https://github.com/Unicon/grouper-amqp-esb-publisher

/mrg

> On Dec 17, 2018, at 12:38 PM, Jeff <> wrote:
> 
> 
> Hi,
> 
> I would like Grouper to publish to RabbitMQ queues based on Grouper folder
> path.  I believe this is supported in RabbitMQ using a topic exchange and
> routing keys.  Currently, Grouper publishes directly to a RabbitMQ queue. 
> 
> Is there a way to do this either through configuration, or through another
> approach?
> 
> Thanks,
> 
> Jeff Ruch
> Colorado State University