Grouper Users - Open Discussion List

[Chris Hyzer <>]

You could keep another pointer (or pointers) and if there is a problem, you could try to move the changelog consumer pointer back to where you need it… (need to make sure that isn’t cached somewhere in the controller…)

 

However, another solution is:

 

Get a batch from the change log (which is currently a batch of 100), and send that along as one operation to activemq, then mark it complete.  So it would not be 1 huge batch, but rather, a bunch of medium sized batches…

 

That 100 number is hardcoded right now, but we could easily make that configurable, or configurable per consumer if that helps.  Wouldn’t make it too large though… you can experiment with it to see what works.

 

Thanks,

Chris

 

 

From: [mailto:] On Behalf Of Rahul Doshi
Sent: Monday, April 29, 2013 11:13 AM
To:
Subject: [grouper-users] PSP performance

 

Hello,

 

While  jira https://bugs.internet2.edu/jira/browse/GRP-882 to improve the PSP real-time provisioning  is worked on we are looking to implement a stop gap solution to improve PSP real-time provisioning performance.   This is how we are planning to do that.  We will implement a custom change log consumer that will combine multiple updates to same object and publish it to a queue like ActiveMQ.  We will implement a client that will grab the message from the queue and perform the update as single ldap operation. 

 

One limitation we already know of this approach is we would have to continuously advance change log consumer pointer even though we are still aggregating changes in a single message to push it to ActiveMQ.  If change log consumer dies for whatever reason before we could publish the message to the queue we will lose changes and only way to recover from that would be to do bulkSync.  Are there any other issues that we could potentially  face?  Also if anyone has alternative approach to improve PSP real-time performance we would very much like to hear.

 

Thanks,

Rahul