Grouper Developers Forum

[Jim Fox <>]

This is to add to the provisioning via messages discussion.

We've used messaging to provision group caches for several years.
These are some of our experiences.


ActiveMQ vs. AWS

We used Apache's AMQ for a couple of years prior to our switch to
Amazon's AWS.  AMQ not in any way a turnkey product.  It has to be
installed, have lots of disk allocated, and it has to be monitored.
Someone has to manage keys, accounts and permissions of the producers
and consumers of messages.

We had a few issues with our MQ service.  Occasionally one of the
persistent connections from AMQ to a consumer would hang.  We found
no way easy way to detect the condition except to send heartbeats
and force a restart on the client.  If a consumer stops reading
messages AMQ's disk fills and the whole thing stops.  Addition of
new consumers required a restart of the service, which is turn
tended to require a restart of the producers and consumers.

We've used AWS for a couple of years.  It has worked without fail.
We send to a notification service (SNS) and let AWS distribute the
messages to all the customers.  These can be direct to an http
or https endpoint or can be through the customers queue service
(SQS). AWS provides monitors, alerts, and statistics.  In addition,
it is very easy to add both producers and consumers to the service.
Thus far the million or so messages we've sent through AWS has cost
us about six dollars.

There are some concerns over the lack of order of messages out of AWS.
See below: 'Content of the messages'.

The idea of requiring all the machinery of AMQ solely to provision a
local LDAP seems very irresponsible to me.  I also worry that if the
project distributes AMQ with Grouper it will end up supporting AMQ.


What is the LDAP for?

If the LDAP is simply another downstream cache of group data it could
be provisioned by a message service.  We provision our AD this way.
I repeat that I would not recommend this for the simple case of a
single LDAP.  If, however, it is a cache used by the group service
itself, as we use our openldap, then the message system would entail
too much latency.  In this case the LDAP ought to be provisioned
directly out of the change log.



Content of the messages.

For us this was trivially easy.  Our group service provides a REST
API with essentially two resources:  group information except
membership, and the individual members.  Our messages are the same
representations of these resources as one would get from the API.

This leads to an easy way to deal with AWS's lack of guaranteed
order.  We keep a window of resource_id/event_time pairs.  For each
message, if we have already processed a later message for this
resource we toss the new one.


Signatures, encryption, and headers.

To guarantee message integrity we feel it necessary to sign them.
Since some of the content is confidential we also have to encrypt
some of them.  Signing is easy enough, as a public key can be
distributed on a separate web site.  Encryption is more difficult.
You have to distribute the keys out of band, and only to trusted
'world-readers'.  In addition the messages have to have meaning to
ordinary clients, who do not have the decryption key.  We put some
information, group name, type of resource, etc. in an unencrypted
message header.  This way a client can at least see when a group's
membership has been updated and query the web service for the
changes.

Jim