Skip to Content.
Sympa Menu

grouper-dev - RE: [grouper-dev] grouper ActiveMQ integration

Subject: Grouper Developers Forum

List archive

RE: [grouper-dev] grouper ActiveMQ integration

Chronological Thread 
  • From: Jim Fox <>
  • To: Chris Hyzer <>
  • Cc: Keith Hazelton <>, "" <>, "" <>
  • Subject: RE: [grouper-dev] grouper ActiveMQ integration
  • Date: Tue, 23 Oct 2012 12:05:20 -0700 (PDT)

[ be aware that I am not authoritative on AWS. see ]

Some questions regarding SQS...

1. Is there a way to do a topic as opposed to a queue? i.e. I want to clear
an authz cache which is on 10 app servers. I want to send the message to one
topic, and whoever is subscribed will get the message.

It would be SNS that corresponds to a topic. It can have any number of
senders and listeners, and a listener can be a http or https port. SNS will
send the messages to the http listener (no need for polling). If the
messages are short enough they can be sent to a phone with SMS.

2. I don't really like the polling. I want to send a message and the
receiver will get it in real time. With SQS I have to poll every minute, and
even then, it might not be ready and it will be there (hopefully) the next
time after that...

As long as you can receive the messages as fast as they are sent you can run
an http listener and do no polling.

3. The polling doesn't seem like it scales... if every need to get a message
is a poll (every minute? every 15 seconds?) then that are a lot of SSL web
service requests as opposed to a socket which is just waiting for traffic.
If you have a lot of apps on a machine or a cluster it seems like this
polling can add up.

We first used the direct connection from SNS to an http app, but it turned
out, for this application, to have the SNS topic send to an SQS queue from
which we pull the messages. We use a sliding polling frequency. As long as
there are messages on the queue we retrieve the next immediately. If the
queue is empty we wait a minute and then poll. If that goes on very long we
back off to five minutes per poll.

4. Whoops, I think the last time I estimated the cost of SQS I was
calculating polling every second, which is expensive. But I think for one
endpoint to poll SQS every minute, it costs 50 cents per year :) ok, that is
pretty cheap... :)

I think we budget about 80 cents per year for the course enrollment messages.



-----Original Message-----

On Behalf Of Jim Fox
Sent: Tuesday, October 23, 2012 2:33 PM
To: Keith Hazelton

Subject: Re: [grouper-dev] grouper ActiveMQ integration

ActiveMQ integration is something I've been hoping to look into, especially
after hearing JimmyV talk about how they're using it to provision
identity data out of the Central Person Registry at Penn State.  
Looks like a prime candidate for a CIFER-recommended tool to use for
provisioning and integration tasks when an event-driven message based
solution is appropriate.

At UW we've used ActiveMQ for group change notices for several years.
It has generally worked well enough, but for many reasons all of our
new group messaging activities are using Amazon's SNS/SQS messaging.

ActiveMQ is expensive. You have to get and maintain hardware,
install and document the software, monitor it, be on-call during the
night, administer it. It might be open source, but it isn't free.

If you are lucky enough to have an existing ActiveMQ service
operated by another department, and they are willing to let use use
it mostly for free (our situation), hope they will continue
to give you wonderful service after they have migrated their own
stuff to RabbitMQ, or OpenMQ, or some other MQ and you're the
only one left on the ActiveMQ system.

Connections can hang. We use the CMS client, for c++ with an
interface library for plain c. (You know there's trouble with an
interface when the first thing you absolutely need is to find an
interface package for your language.) This interface uses persistent
connections and certain conditions on the server will cause these
connections to hang. This is a difficult situation to monitor.
Generally an external process has to send test messages and report
when none has been processed for some timeout period.

Amazon's messaging service, by comparison, is nearly free, is
exceptionally easy to use (there are client libraries for many
languages, including both our present favorites: java and python,
but I had no trouble writing a c client from scratch), easy to
configure (a web client make most administration self-service),
and is reliable. It is also RESTful.

We have, as do most universities nowadays, initiatives to use cloud
services in lieu of a 'running everything ourselves' mentality.
Our people are not so fearful of losing control of even core
services. It is the future.

There are a couple of advantages ActiveMQ has over Amazon SNS.

ActiveMQ is an order of magnitude faster than SNS. Is that important?
For our network team, monitoring thousands of routers and whatnot,
and sending millions of messages per day, it is important. For our
group service, not so much. Messaging for course enrollment,
operating since summer, now totals about 300,000 adds or drops.
All the messages get to our consumers in a few seconds. And a few
seconds is a short time for this application.

ActiveMQ always delivers messages in order (I think). With SNS/SQS
they can get out of order. Something to deal with.


Archive powered by MHonArc 2.6.16.

Top of Page