Grouper Users - Open Discussion List

[Shilen Patel <>]

The temp change log to change log is also where membership changes are flattened.  Only immediate membership changes are written to the temp change log.  But this process will write the effective changes into the actual change log. In addition to duplicates, this also makes it so that consumers don’t have to worry about checking if a user is a member of a group in multiple ways when there’s a membership delete.  From what I’ve seen in the past, this calculation is often the most expensive part about it.  But perhaps improvements can be made there.  You can also disable the flatten calculations if they aren’t desired.

 

- Shilen

 

From: <> on behalf of "Hyzer, Chris" <>
Date: Friday, July 13, 2018 at 1:30 AM
To: "Gettes, Michael" <>, "Crawford, Jeffrey" <>
Cc: "" <>
Subject: RE: [grouper-users] Hopefully quick question regarding rabbitmq and grouper

 

> Now all that said, the only item in all this that worries me is the tempchangelog to changelog processing.  

> It’s a necessary evil cuz this is where changes are labelled/categorized.  It’s also the single threading aspect

> of processing changes that worries me the most.  Make sure these tables are on fast storage, properly indexed

> and so on - usual stuff.  Beyond that - like I said - good stuff.

 

It also attaches a sequential number to all the records.

 

I feel like some change log consumers could also read from the change_log_temp.  This would give multi-threaded faster provisioning during high load periods under certain circumstances.  Provisioners might get a duplicate message but would be able to figure out it’s a no-op…  Or maybe they could only read from change_log_temp.

 

We could explore this to see if it is feasible…

 

 

 

 

 

From: [mailto:] On Behalf Of Gettes, Michael
Sent: Thursday, July 12, 2018 4:22 PM
To: Crawford, Jeffrey <>
Cc:
Subject: Re: [grouper-users] Hopefully quick question regarding rabbitmq and grouper

 

Yes, it is the same - but, as an example, i have had messaging running about every 5 seconds.  So it can be rather quick response.  The whole loader process is actually quite elegant.  The quartz mechanism scales to many hosts - all those running the loader.  TIER packaging has moved to calling it the daemon - better terminology (I personally don’t like the word but loader isn’t good either).  Don’t get hung up on the loader and changeelog - just use it.  It’s cool stuff.

 

Now all that said, the only item in all this that worries me is the tempchangelog to changelog processing.  It’s a necessary evil cuz this is where changes are labelled/categorized.  It’s also the single threading aspect of processing changes that worries me the most.  Make sure these tables are on fast storage, properly indexed and so on - usual stuff.  Beyond that - like I said - good stuff.

 

I hope this helps.

 

/mrg

On Jul 12, 2018, at 3:43 PM, Crawford, Jeffrey <> wrote:

 

Greetings team,

 

How does the RabbitMQ messages get processed. Is it similar to the psp where changes go into the changelog and a process then periodically polls it? Or do messages get sent as soon as they are sent to the changelog? My guess it’s still a polling system since changes can come in from the WS or UI but the loader has traditionally been the “batch” processing component which does the back end stuff.

 

Thanks

Jeffrey C.