Grouper Developers Forum

["McDermott, Michael" <>]

We're not using grouper to manage permissions in ActiveMQ, though now that we've seen it we may borrow it.  We're using ActiveMQ here at Brown to provision downstream groups based services.

Our basic setup is that we use the Grouper changelog to put messages onto a JMS queue (the basic set are create group, delete group, add member, and delete member).  That message will list different target systems based on attributes tied to the group.  For example Google GAE Groups, LDAP Groups, AD, or Instructure's hosted Canvas course memberships.   

All messages are sent to the same queue.  On the other end there is a service that reads the queue and farms copies out to dedicated service queues.  We use a very basic Ruby script to do this, but something like Apache Camel might be a great fit as well.

There is one service per dedicated queue that processes the messages and provisions the service.  It is responsible for aggregating information about the provisioning event and executing it.  

Its been pretty successful provisioning google groups and Canvas, we processed over 120K provisioning events in the last year and the server resources are minimal.

The advantages of ActiveMQ from our standpoint were it supports JMS, STOMP, and (will support AMQP 1).  It is fairly robust, and configuration is straightforward.  I've heard reports on the ActiveMQ mailing list of issues with clustered ActiveMQ instances, but we're nowhere near a load level where that should be an issue.

Steve Carmody secured clearance from our CIO to open source what we've done, but our staff has been mired in other projects.

On Tue, Oct 23, 2012 at 2:48 PM, Chris Hyzer <> wrote:

Some questions regarding SQS...

1. Is there a way to do a topic as opposed to a queue?  i.e. I want to clear an authz cache which is on 10 app servers.  I want to send the message to one topic, and whoever is subscribed will get the message.

2. I don't really like the polling.  I want to send a message and the receiver will get it in real time.  With SQS I have to poll every minute, and even then, it might not be ready and it will be there (hopefully) the next time after that...

3. The polling doesn't seem like it scales... if every need to get a message is a poll (every minute?  every 15 seconds?) then that are a lot of SSL web service requests as opposed to a socket which is just waiting for traffic.  If you have a lot of apps on a machine or a cluster it seems like this polling can add up.

4. Whoops, I think the last time I estimated the cost of SQS I was calculating polling every second, which is expensive.  But I think for one endpoint to poll SQS every minute, it costs 50 cents per year :)  ok, that is pretty cheap...  :)

Thanks,
Chris

-----Original Message-----
From: [mailto:] On Behalf Of Jim Fox
Sent: Tuesday, October 23, 2012 2:33 PM
To: Keith Hazelton
Cc: ;
Subject: Re: [grouper-dev] grouper ActiveMQ integration


>

> ActiveMQ integration is something I've been hoping to look into, especially after hearing JimmyV talk about how they're using it to provision
> identity data out of the Central Person Registry at Penn State.  
> Looks like a prime candidate for a CIFER-recommended tool to use for provisioning and integration tasks when an event-driven message based
> solution is appropriate.
>

At UW we've used ActiveMQ for group change notices for several years.
It has generally worked well enough, but for many reasons all of our
new group messaging activities are using Amazon's SNS/SQS messaging.

ActiveMQ is expensive.  You have to get and maintain hardware,
install and document the software, monitor it, be on-call during the
night, administer it.  It might be open source, but it isn't free.

If you are lucky enough to have an existing ActiveMQ service
operated by another department, and they are willing to let use use
it mostly for free (our situation), hope they will continue
to give you wonderful service after they have migrated their own
stuff to RabbitMQ, or OpenMQ, or some other MQ and you're the
only one left on the ActiveMQ system.

Connections can hang.  We use the CMS client, for c++ with an
interface library for plain c.  (You know there's trouble with an
interface when the first thing you absolutely need is to find an
interface package for your language.)  This interface uses persistent
connections and certain conditions on the server will cause these
connections to hang.  This is a difficult situation to monitor.
Generally an external process has to send test messages and report
when none has been processed for some timeout period.


Amazon's messaging service, by comparison, is nearly free, is
exceptionally easy to use (there are client libraries for many
languages, including both our present favorites: java and python,
but I had no trouble writing a c client from scratch), easy to
configure (a web client make most administration self-service),
and is reliable.  It is also RESTful.

We have, as do most universities nowadays, initiatives to use cloud
services in lieu of a 'running everything ourselves' mentality.
Our people are not so fearful of losing control of even core
services.  It is the future.

There are a couple of advantages ActiveMQ has over Amazon SNS.

ActiveMQ is an order of magnitude faster than SNS.  Is that important?
For our network team, monitoring thousands of routers and whatnot,
and sending millions of messages per day, it is important.  For our
group service, not so much.  Messaging for course enrollment,
operating since summer, now totals about 300,000 adds or drops.
All the messages get to our consumers in a few seconds.  And a few
seconds is a short time for this application.

ActiveMQ always delivers messages in order (I think).  With SNS/SQS
they can get out of order.  Something to deal with.

Jim

--
Michael J. McDermott
Lead Developer, Identity and Access Management
Brown University