Grouper Users - Open Discussion List

["Michael R. Gettes" <>]

Kind Grouper Community,

Carnegie Mellon University’s Identity Services Team has developed some code 
that may be of interest.  We weren’t satisfied with the flexibility and 
performance of PSP.  In our environment, we saw an average of 3 grouper 
change operations per second - sometimes as high as 9 or 10, using PSP.  We 
have a number of large groups and lots of group dependencies based on group 
math and group nesting for many purposes.  We use groups to provision 
services like GoogleApps and are in process of deploying OIM (still !!!) 
where we use groups to describe who gets various services.  In other words, I 
think we are effectively trying to use Grouper itself.  However, our 389 and 
AD LDAP environments weren’t being updated fast enough.  

ActiveMQ can be found at activemq.apache.org.  With AMQ we can easily monitor 
queue activity, usage over time, graphing usage and monitor for backlogs so 
we can actually see how our environment is doing.  This is not something 
easily achieved within Grouper itself or with PSP.  There is no real 
dependency on AMQ with what we have developed other than the API calls used 
to plug into AMQ.  We make use of JMS and the Stomp messaging protocol.  It 
shouldn’t be hard to substitute another messaging service if you want to do 
so.  Our AMQ environment is 2 servers in redundant mode running 5.9.0 and 
this wasn’t hard to get going.  We, identity services, run the AMQ 
environment.  In our testing, we have processed many many millions of 
messages.  AMQ is LDAP enabled for authentication and access control.  Our 
AMQ is only for Identity related activities - we do not run a generalized 
message service for applications.  We do NOT run this as an ESB even though 
AMQ is used to create ESBs.  Our mindset is to “keep it simple”.

Much of the code I am about to describe was written by Rahul Doshi of CMU.  
The design was a team effort.  This was Rahul’s first significant development 
in Perl.  He’s rather used to developing in Java.  We got some input from 
Brown University and their grouper changelog to AMQ code, but we ended up 
redoing everything from scratch.

We process the grouper changelog and express the changes in simple messages 
to ActiveMQ.  We experienced the ability to write about 10,000 messages per 
minute to AMQ (that’s 167/sec we believe this number has gone up quite a bit 
since our original tests).  We have a dispatcher which, via a config file, 
will resend the grouper change messages to other queues within AMQ.  This 
allows us to filter changes so a queue will only see what it needs to see.  
For example, we don’t want to publish to AD our course groups, so we don’t 
send those changes by specifying an appropriate pattern.  It’s not so much 
how fast the messages get passed around - at least we know with AMQ it isn’t 
a concern.  Of course, how fast we process messages by the consumers 
(described below) is the real issue.

For our 389 LDAP server (which could be any LDAP server really), we have 2 
consumers.  One to process standard LDAP group objects and the other consumer 
processes isMemberOf attributes in the user object - similar to the AD 
memberOf.  A single change for add userid X to group Y would get translated 
into 2 messages sent to the 389.groups and 389.ismemberof queues in AMQ.  The 
consumers, written in perl, read a message off the queue, processes the 
change, and moves on to the next.  The consumers support a full-sync mode to 
allow for a push of all groups to be re-sync’d as well as the normal mode of 
handling each individual change as it comes.

Some numbers:  under load, we have seen 30 messages/sec for normal mode (not 
full sync) messages.  Each result in multiple LDAP operations.  Performance 
really depends on the LDAP server and its configuration.   We have seen LDAP 
ops as high as 174/sec and these are irrespective of whether MOD ops are 
done.  We can create or delete a group with about 30,000 members in about 5-7 
min (not including LDAP replication).  A full group sync of about 5000 groups 
where little changes are made takes about 25 min.  For isMemberOf processing, 
a full sync takes about 33 minutes.  We believe we can still speed up the 
full-sync aspects of this - so this effort is still a work in progress.  Our 
hope is full-sync operations would be done no more frequently than weekly.  
We found PSP getting out of sync and the need for full-sync operations.  We 
are working towards full-sync operations being performed on an exceptional 
basis.

For AD, we support bushy style LDAP structure and handle DNs within MS specs 
(each DN component can’t be more than 64 chars) and the samAccountName is 
reversed to make it usable by normal AD group UI mechanisms.

We are in process of working through licensing issues with our tech transfer 
office (hence my questions about licensing).  PSU is interested in using this 
code and we will work with them to make sure it is sufficiently flexible in 
configuration and works outside of CMU.  We expect to have the code available 
on github at some point in the coming couple of months.

Given our work, we still believe Grouper should package ActiveMQ with it so 
custom changelog processors no longer need to be written - and it would just 
be AMQ consumers for people to do whatever they want.  Consumers are so easy 
to write.  We have external to Identity Services various applications reading 
identity data from AMQ, in message formats we specify and control who can 
read/write queues.

The intent with this email was to give a brief overview of what we have been 
doing.  The numbers are complicated but look pretty good, we haven’t provided 
gory details because of the complexity of things.  GAP has been in 
production, fully replacing PSP, since June 1, 2014.

Lastly, THANK YOU SO MUCH to the Grouper team for Grouper 2.2 and the new UI. 
 CMU is very excited to begin work (probably tomorrow) on getting this new 
version of Grouper up and running and to have it deployed to production as 
soon as is practical.

Happy to answer questions about GAP, if you have any.

/mrg