Grouper Developers Forum

[Chris Hyzer <>]

This helps alleviate concerns about polling:

http://aws.amazon.com/about-aws/whats-new/2012/11/08/amazon-sqs-announces-long-polling-and-enhanced-client-support/

SQS now supports long-polling, which reduces extraneous polling associated 
with empty receives. With long polling, receiving a message from an empty 
queue will no longer return immediately. Instead, SQS waits up to 20 seconds 
for a message to arrive in the queue, at which point it returns it 
immediately. You can enable long polling for the entire queue, or 
alternatively for individual polling requests.

Thanks,
Chris

-----Original Message-----
From: Jim Fox 
[mailto:]
 
Sent: Tuesday, October 23, 2012 3:05 PM
To: Chris Hyzer
Cc: Keith Hazelton; 
;
 

Subject: RE: [grouper-dev] grouper ActiveMQ integration


[ be aware that I am not authoritative on AWS.  see 
http://aws.amazon.com/documentation/ ]

> Some questions regarding SQS...
>
> 1. Is there a way to do a topic as opposed to a queue?  i.e. I want to 
> clear an authz cache which is on 10 app servers.  I want to send the 
> message to one topic, and whoever is subscribed will get the message.

It would be SNS that corresponds to a topic.  It can have any number of 
senders and listeners, and a listener can be a http or https port.  SNS will 
send the messages to the http listener (no need for polling).  If the 
messages are short enough they can be sent to a phone with SMS.

>
> 2. I don't really like the polling.  I want to send a message and the 
> receiver will get it in real time.  With SQS I have to poll every minute, 
> and even then, it might not be ready and it will be there (hopefully) the 
> next time after that...

As long as you can receive the messages as fast as they are sent you can run 
an http listener and do no polling.

>
> 3. The polling doesn't seem like it scales... if every need to get a 
> message is a poll (every minute?  every 15 seconds?) then that are a lot of 
> SSL web service requests as opposed to a socket which is just waiting for 
> traffic.  If you have a lot of apps on a machine or a cluster it seems like 
> this polling can add up.

We first used the direct connection from SNS to an http app, but it turned 
out, for this application, to have the SNS topic send to an SQS queue from 
which we pull the messages.  We use a sliding polling frequency.  As long as 
there are messages on the queue we retrieve the next immediately.  If the 
queue is empty we wait a minute and then poll.  If that goes on very long we 
back off to five minutes per poll.

>
> 4. Whoops, I think the last time I estimated the cost of SQS I was 
> calculating polling every second, which is expensive.  But I think for one 
> endpoint to poll SQS every minute, it costs 50 cents per year :)  ok, that 
> is pretty cheap...  :)

I think we budget about 80 cents per year for the course enrollment messages.

Jim

>
> Thanks,
> Chris
>
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Jim Fox
> Sent: Tuesday, October 23, 2012 2:33 PM
> To: Keith Hazelton
> Cc: 
> ;
>  
> 
> Subject: Re: [grouper-dev] grouper ActiveMQ integration
>
>
>>
>> ActiveMQ integration is something I've been hoping to look into, 
>> especially after hearing JimmyV talk about how they're using it to 
>> provision
>> identity data out of the Central Person Registry at Penn State.  
>> Looks like a prime candidate for a CIFER-recommended tool to use for 
>> provisioning and integration tasks when an event-driven message based
>> solution is appropriate.
>>
>
> At UW we've used ActiveMQ for group change notices for several years.
> It has generally worked well enough, but for many reasons all of our
> new group messaging activities are using Amazon's SNS/SQS messaging.
>
> ActiveMQ is expensive.  You have to get and maintain hardware,
> install and document the software, monitor it, be on-call during the
> night, administer it.  It might be open source, but it isn't free.
>
> If you are lucky enough to have an existing ActiveMQ service
> operated by another department, and they are willing to let use use
> it mostly for free (our situation), hope they will continue
> to give you wonderful service after they have migrated their own
> stuff to RabbitMQ, or OpenMQ, or some other MQ and you're the
> only one left on the ActiveMQ system.
>
> Connections can hang.  We use the CMS client, for c++ with an
> interface library for plain c.  (You know there's trouble with an
> interface when the first thing you absolutely need is to find an
> interface package for your language.)  This interface uses persistent
> connections and certain conditions on the server will cause these
> connections to hang.  This is a difficult situation to monitor.
> Generally an external process has to send test messages and report
> when none has been processed for some timeout period.
>
>
> Amazon's messaging service, by comparison, is nearly free, is
> exceptionally easy to use (there are client libraries for many
> languages, including both our present favorites: java and python,
> but I had no trouble writing a c client from scratch), easy to
> configure (a web client make most administration self-service),
> and is reliable.  It is also RESTful.
>
> We have, as do most universities nowadays, initiatives to use cloud
> services in lieu of a 'running everything ourselves' mentality.
> Our people are not so fearful of losing control of even core
> services.  It is the future.
>
> There are a couple of advantages ActiveMQ has over Amazon SNS.
>
> ActiveMQ is an order of magnitude faster than SNS.  Is that important?
> For our network team, monitoring thousands of routers and whatnot,
> and sending millions of messages per day, it is important.  For our
> group service, not so much.  Messaging for course enrollment,
> operating since summer, now totals about 300,000 adds or drops.
> All the messages get to our consumers in a few seconds.  And a few
> seconds is a short time for this application.
>
> ActiveMQ always delivers messages in order (I think).  With SNS/SQS
> they can get out of order.  Something to deal with.
>
> Jim
>
>