Grouper Users - Open Discussion List

[John Schrader <>]

+1 on multi-threaded change_log consumers

Thanks!

-John

On Fri, Jul 13, 2018 at 1:30 AM Hyzer, Chris <> wrote:

> Now all that said, the only item in all this that worries me is the tempchangelog to changelog processing.  

> It’s a necessary evil cuz this is where changes are labelled/categorized.  It’s also the single threading aspect

> of processing changes that worries me the most.  Make sure these tables are on fast storage, properly indexed

> and so on - usual stuff.  Beyond that - like I said - good stuff.

 

It also attaches a sequential number to all the records.

 

I feel like some change log consumers could also read from the change_log_temp.  This would give multi-threaded faster provisioning during high load periods under certain circumstances.  Provisioners might get a duplicate message but would be able to figure out it’s a no-op…  Or maybe they could only read from change_log_temp.

 

We could explore this to see if it is feasible…

 

 

 

 

 

From: [mailto:] On Behalf Of Gettes, Michael
Sent: Thursday, July 12, 2018 4:22 PM
To: Crawford, Jeffrey <>
Cc:
Subject: Re: [grouper-users] Hopefully quick question regarding rabbitmq and grouper

 

Yes, it is the same - but, as an example, i have had messaging running about every 5 seconds.  So it can be rather quick response.  The whole loader process is actually quite elegant.  The quartz mechanism scales to many hosts - all those running the loader.  TIER packaging has moved to calling it the daemon - better terminology (I personally don’t like the word but loader isn’t good either).  Don’t get hung up on the loader and changeelog - just use it.  It’s cool stuff.

 

Now all that said, the only item in all this that worries me is the tempchangelog to changelog processing.  It’s a necessary evil cuz this is where changes are labelled/categorized.  It’s also the single threading aspect of processing changes that worries me the most.  Make sure these tables are on fast storage, properly indexed and so on - usual stuff.  Beyond that - like I said - good stuff.

 

I hope this helps.

 

/mrg

On Jul 12, 2018, at 3:43 PM, Crawford, Jeffrey <> wrote:

 

Greetings team,

 

How does the RabbitMQ messages get processed. Is it similar to the psp where changes go into the changelog and a process then periodically polls it? Or do messages get sent as soon as they are sent to the changelog? My guess it’s still a polling system since changes can come in from the WS or UI but the loader has traditionally been the “batch” processing component which does the back end stuff.

 

Thanks

Jeffrey C.

 

--

John Schrader

Identity and Access Management

Office of Information Technologies

University of Notre Dame

EVERYTHING SHOULD BE MADE AS SIMPLE AS POSSIBLE, BUT NOT ANY SIMPLER

—ALBERT EINSTEIN