Grouper Developers Forum

["Cramton, James" <>]

At Brown this week, we've successfully provisioned our 11,000 groups
into MACE Grouper 1.1 and 1.2 in our QA environment. I've been promising
some numbers on performance for some time, and we have some interesting
preliminary figures to share. We have not drilled down yet to determine
exactly where major bottlenecks are, but at this early stage, we're
pleased with the performance for what we've seen. Additionally, our
figures give some high level performance metrics differences between
Grouper 1.1 and 1.2 rc1.
 
1. Environment
 
Our environment in QA is identical to our production environment, with 1
server dedicated to the Oracle DB, and 1 server dedicated to the
application & provisioning server. The only difference is that the
production DB server has an HBA card for SAN connectivity, which will
help with DB I/O. Development is significantly less robust--in
development, Oracle and the app server run on the same 3 year old single
CPU server with 1GB of RAM. Production and QA specs look like this:

Hardware: 
Sun SunFire X4100 M2 x64 Servers
- Four 2.4Ghz/1Mb processors
- 4Gb RAM (DDR2-667)
- Two 73Gb 10K RPM SAS drives (hardware mirrored)
- 1Gb/full duplex network


Software:
DB server: 
- RedHat Linux Enterprise Server 3
- Oracle 10g pretty much straight out of the box
 
App server: 
- RedHat Linux Enterprise Server 3
- Apache 2.2.4
- apr and apr-util with ldaps support
- mod_ssl for https support in apache
- mod_authnz_ldap for apache authentication via Brown's ldap(s) server
- Tomcat 5.5.20
- Sun JDK 1.5.0_10
- mod_jk 1.2.21
- ant 1.7.0
- MACE Grouper 1.1 or 1.2, at various times. Running 1.2 rc1 now.
 


2. The task
Brown's existing group management tool (aka Brown Grouper) currently
manages 11,000 groups, with 8,500 groups as members of other groups, and
a total of 370,000 person memberships in all groups. Brown has 18,000
users in our SunOne LDAP registry. The current task is to script the
one-time provisioning effort of moving all existing Brown Grouper
memberships to the MACE Grouper database. On an ongoing basis, there
will be a differential provisioning run every night to update
provisioned group changes, but this will be a small fraction of the
total, except during registration periods 2 or 3 times per year. Our
script currently only handles stem creation and group memberships, but
we are incorporating ACLs next, and this is expected to increase run
time to some degree. We also have some fine tuning of the provisioning
script to better handle edge cases of complicated group math we
currently use in Brown Grouper. We will be working to improve
performance of the script and the overall system at the same time.
 


3. Results
Grouper 1.1:
We first ran the provisioning script in QA on Grouper 1.1, and the
results were better than our days-long tests in development, but not yet
acceptable. We successfully provisioned the 11,000 groups in MACE
Grouper in about 6 hours. 
 
Grouper 1.2:
Last night, we ran the provisioning script in QA on a clean installation
of Grouper 1.2 rc1, with a dramatic improvement just in the new release.
We successfully provisioned the 11,000 groups in MACE Grouper 1.2 in
about 3.5 hours.
 
From briefly inspecting the systems during the provisioning run, there
was a mild to moderate iowait bottleneck on the Oracle server, although
we have more work to do to identify the direct cause of the bottleneck.
We've done essentially no tuning of the Oracle system (yet), and no
analysis of what queries are taking the longest (yet). We expect some
potentially significant performance increase to come from database
hardware and software configuration changes, along with some query
optimization.
 
The web interface performance [speed] has been entirely acceptable on
any of the systems we've used, even with a large data set. Faster
systems did provide quicker web interface response, and of course pages
displaying large membership lists were slower than others, but the
slowest pages were acceptable. We have not load tested with multiple
simultaneous users in the web interface.
 
One interesting lesson we've learned in our first cursory analysis of
our data in MACE Grouper is that the web interface provides much greater
visibility into the nature of our data. Having the ability to list an
individual subject's memberships was not supported by Brown Grouper, and
it is seen as a big win for the group that will manage the data. On the
down side, MACE Grouper has revealed a number of undesirable issues in
our data. There will be a significant data management project of
unknown scope to come from the increased visibility MACE Grouper
provides.
 

 
4. Next steps
These are the figures we have at this point. If you would find other
information useful, let me know, and I can supplement. Our next tasks
proceed on 2 fronts...technical and administrative. Technically, we are
adding ACLs to our provisioning process to maintain existing rules in
Brown Grouper that define who can view groups or list memberships, etc.
Once that is done, we will get our daily differential provisioning
script in place and start monitoring speed and accuracy over several
weeks. We also have odds and ends in the MACE Grouper software to
address, such as building a custom UI for at least 2 audiences.
Administratively, we are interviewing our current group management staff
with our full data set in MACE Grouper 1.2. We'll meet to go through
common management tasks to identify any functional changes we may need
to make in order to accomplish current needs. During this process, we
will also be identifying any data issues we may want to clean up, and
we'll be identifying the skill set we think will be needed to support
this application going forward.  Out of those administrative
conversations we'll establish policy to govern the use of MACE Grouper
in production.
 
I hope this helps!
 
 
James Cramton
Lead Programmer/Analyst
Brown University

401 863-7324