Grouper Users - Open Discussion List

[Jeffrey Eaton <>]

For what it’s worth, we were never able to get PSP in Grouper 2.2 to work 
efficiently enough to actually be usable in production for our needs (though 
we were pushing changes out to just 389, not AD).  That was what led us to 
develop our own system for doing provisioning, consisting of a relatively 
simple changelog consumer that pushes out messages of changes to ActiveMQ, a 
tool which consumes those messages and dispatches them to one or more target 
ActiveMQ queues based on the group name and/or transaction type, and then a 
set of Perl code that consumes messages from ActiveMQ and applies the changes 
out to target systems.  We now provision out to 389 (both as Group members 
and account isMemberOf attributes), AD, and some other custom systems.  It’s 
scaled pretty well for us, and can generally keep up with messages as fast as 
Grouper can generate changes.

The code is out at https://github.com/cmu-ids/Grouper-ActiveMQ-Provisioner/

Note I have not used this with 2.3 at all. 

-jeaton



> On Sep 14, 2016, at 10:13 AM, Dave Churchley 
> <>
>  wrote:
> 
> Good afternoon
> 
> I'd just like to follow up on this. We've spotted that the issue Phil 
> described below is a known issue with real-time PSP - 
> https://spaces.internet2.edu/pages/viewpage.action?pageId=25204227#GrouperProvisioningServiceProvider(PSP)-KnownIssueswith"RealTime"Provisioning
>  - which partly explains why it's taking so long.
> 
> Is anyone else using PSP to populate AD at a relatively large scale? If so, 
> how is the performance? We still haven't caught up with a large number of 
> changes made on 1st September, so we have not had "real-time" provisioning 
> for a fortnight now. 
> 
> As Phil said, we're currently running Grouper 2.2.2. We're planning to 
> start looking into upgrading and using PSPNG. Can anyone please reassure me 
> that we will not suffer similar issues if we do that?
> 
> Thanks
> Dave 
> 
>> -----Original Message-----
>> From: 
>> 
>>  [
>> ]
>>  On Behalf Of Philip Harle
>> Sent: 08 September 2016 16:39
>> To: 
>> 
>> Subject: [grouper-users] RE: PSP speed issues
>> 
>> Hi,
>> 
>> Just as a follow up to this, as mentioned previously we only publish a 
>> subset of
>> our Grouper groups to Active Directory. This is configured in 
>> ldap.properties to
>> only publish groups in the 'Applications' stem:
>> 
>> # The base Grouper stem to be provisioned.
>> edu.internet2.middleware.psp.baseStem=Applications
>> 
>> However, in grouper_error.log I can see that the PSP is working its way 
>> through
>> all groups regardless of which stem they're located in. Groups in the
>> 'Applications' stem do take a second or so longer than the others to 
>> process.
>> Is it correct that the PSP operates in this manner, or can it be 
>> configured to
>> ignore those groups that ultimately it won't be publishing to the Active
>> Directory?
>> Being able to filter out unnecessary PSP enumeration is likely to give us a
>> significant improvement in processing time.
>> 
>> Thanks,
>> Phil
>> 
>>> -----Original Message-----
>>> From: Philip Harle
>>> Sent: 06 September 2016 14:02
>>> To: 
>>> 
>>> Subject: PSP speed issues
>>> 
>>> Hi,
>>> 
>>> Last year we migrated (via a fresh install) our instance of Grouper to 
>>> version
>>> 2.2.2. Since then we have experienced a significant degradation around the
>>> speed of provisioning with the PSP. We've been using the PSP to
>>> incrementally provision groups since the upgrade, and accepted the reduced
>>> speed.
>>> 
>>> However, as we've recently loaded all student data for our new academic
>>> year into Grouper, the PSP has been working through over a quarter of a
>>> million changes. This is taking between 1 and 4 seconds for each change, 
>>> and
>>> we're estimating that this could take anywhere up to 2 weeks to work
>>> through all the changes. The server itself isn't stressed and is showing 
>>> low
>>> CPU and RAM usage. I've attached a small snippet of the grouper_error.log
>>> showing the groups currently being provisioned. Some of these are being
>>> published to our Active Directory, others are Grouper-only.
>>> 
>>> We're aware of performance issues with the PSP and the advice to upgrade
>>> to PSPNG in 2.3, unfortunately this isn't feasible for us at the moment.
>>> Hence, are there any configuration settings we should investigate/change 
>>> in
>>> an effort to give some improved performance with the current PSP in the
>>> meantime?
>>> 
>>> Thanks,
>>> Phil
>>> 
>>> ---
>>> Phil Harle
>>> IT Service
>>> Newcastle University
>