Subject: Grouper Users - Open Discussion List
[grouper-users] Grouper ActiveMQ Provisioner (GAP)
- From: "Michael R. Gettes" <>
- To: "" <>
- Subject: [grouper-users] Grouper ActiveMQ Provisioner (GAP)
- Date: Mon, 14 Jul 2014 20:21:52 +0000
- Accept-language: en-US
Kind Grouper Community,
Carnegie Mellon University’s Identity Services Team has developed some code
that may be of interest. We weren’t satisfied with the flexibility and
performance of PSP. In our environment, we saw an average of 3 grouper
change operations per second - sometimes as high as 9 or 10, using PSP. We
have a number of large groups and lots of group dependencies based on group
math and group nesting for many purposes. We use groups to provision
services like GoogleApps and are in process of deploying OIM (still !!!)
where we use groups to describe who gets various services. In other words, I
think we are effectively trying to use Grouper itself. However, our 389 and
AD LDAP environments weren’t being updated fast enough.
ActiveMQ can be found at activemq.apache.org. With AMQ we can easily monitor
queue activity, usage over time, graphing usage and monitor for backlogs so
we can actually see how our environment is doing. This is not something
easily achieved within Grouper itself or with PSP. There is no real
dependency on AMQ with what we have developed other than the API calls used
to plug into AMQ. We make use of JMS and the Stomp messaging protocol. It
shouldn’t be hard to substitute another messaging service if you want to do
so. Our AMQ environment is 2 servers in redundant mode running 5.9.0 and
this wasn’t hard to get going. We, identity services, run the AMQ
environment. In our testing, we have processed many many millions of
messages. AMQ is LDAP enabled for authentication and access control. Our
AMQ is only for Identity related activities - we do not run a generalized
message service for applications. We do NOT run this as an ESB even though
AMQ is used to create ESBs. Our mindset is to “keep it simple”.
Much of the code I am about to describe was written by Rahul Doshi of CMU.
The design was a team effort. This was Rahul’s first significant development
in Perl. He’s rather used to developing in Java. We got some input from
Brown University and their grouper changelog to AMQ code, but we ended up
redoing everything from scratch.
We process the grouper changelog and express the changes in simple messages
to ActiveMQ. We experienced the ability to write about 10,000 messages per
minute to AMQ (that’s 167/sec we believe this number has gone up quite a bit
since our original tests). We have a dispatcher which, via a config file,
will resend the grouper change messages to other queues within AMQ. This
allows us to filter changes so a queue will only see what it needs to see.
For example, we don’t want to publish to AD our course groups, so we don’t
send those changes by specifying an appropriate pattern. It’s not so much
how fast the messages get passed around - at least we know with AMQ it isn’t
a concern. Of course, how fast we process messages by the consumers
(described below) is the real issue.
For our 389 LDAP server (which could be any LDAP server really), we have 2
consumers. One to process standard LDAP group objects and the other consumer
processes isMemberOf attributes in the user object - similar to the AD
memberOf. A single change for add userid X to group Y would get translated
into 2 messages sent to the 389.groups and 389.ismemberof queues in AMQ. The
consumers, written in perl, read a message off the queue, processes the
change, and moves on to the next. The consumers support a full-sync mode to
allow for a push of all groups to be re-sync’d as well as the normal mode of
handling each individual change as it comes.
Some numbers: under load, we have seen 30 messages/sec for normal mode (not
full sync) messages. Each result in multiple LDAP operations. Performance
really depends on the LDAP server and its configuration. We have seen LDAP
ops as high as 174/sec and these are irrespective of whether MOD ops are
done. We can create or delete a group with about 30,000 members in about 5-7
min (not including LDAP replication). A full group sync of about 5000 groups
where little changes are made takes about 25 min. For isMemberOf processing,
a full sync takes about 33 minutes. We believe we can still speed up the
full-sync aspects of this - so this effort is still a work in progress. Our
hope is full-sync operations would be done no more frequently than weekly.
We found PSP getting out of sync and the need for full-sync operations. We
are working towards full-sync operations being performed on an exceptional
For AD, we support bushy style LDAP structure and handle DNs within MS specs
(each DN component can’t be more than 64 chars) and the samAccountName is
reversed to make it usable by normal AD group UI mechanisms.
We are in process of working through licensing issues with our tech transfer
office (hence my questions about licensing). PSU is interested in using this
code and we will work with them to make sure it is sufficiently flexible in
configuration and works outside of CMU. We expect to have the code available
on github at some point in the coming couple of months.
Given our work, we still believe Grouper should package ActiveMQ with it so
custom changelog processors no longer need to be written - and it would just
be AMQ consumers for people to do whatever they want. Consumers are so easy
to write. We have external to Identity Services various applications reading
identity data from AMQ, in message formats we specify and control who can
The intent with this email was to give a brief overview of what we have been
doing. The numbers are complicated but look pretty good, we haven’t provided
gory details because of the complexity of things. GAP has been in
production, fully replacing PSP, since June 1, 2014.
Lastly, THANK YOU SO MUCH to the Grouper team for Grouper 2.2 and the new UI.
CMU is very excited to begin work (probably tomorrow) on getting this new
version of Grouper up and running and to have it deployed to production as
soon as is practical.
Happy to answer questions about GAP, if you have any.
- [grouper-users] Grouper ActiveMQ Provisioner (GAP), Michael R. Gettes, 07/14/2014
Archive powered by MHonArc 2.6.16.