Grouper Users - Open Discussion List

["Black, Carey M." <>]

RE: tempchangelog to changelog processing

 

Michael, I kind of agree.

                However, it really should only be a bottle neck if you have LOTS of changes happening at the exact same time, for a sustained time.

                                Or the daemon is “off line for a while”…. ( Don’t do that. J )

                So don’t do large batch processes and the “Event/message model” works much better. AKA: Even flow or tsunami?  J

 

 

I think the other thing to keep in mind is that the loader ( er… “daemon”) can be very RAM hungry.

                In my experience, especially for large LDAP jobs. (100K+ members in a group)

                And since the loader jobs are currently only schedulable to ‘start at “x”’ it can be hard to schedule them to not overload the daemon if some are long running and others “start up” while the big jobs are still running”.

 

 

As a “wish list idea”….

                If a change log consumer gets “to far behind” send notifications to the admins for the change log consumer. ( and the Grouper Admins too. )

                Which could also include the tempchangelog size too. J

 

 

 

As far as the name goes it is both a loader and a daemon. I am just not clear about how to make it “just one of them”. So I think it has a bit of a split personality.

                Pull data in,  (loader jobs: SQL/LDAP/custom)

                Push data out, ( change log consumers, provisioner feed by a change log consumer (IE: box) ,  )

                and do house cleaning too. ( rules engine, …. )

 

                Can we call it the “Grouper Swiss army knife”? ( Or is the fish afraid of knives?  J  )

 

--

Carey Matthew

 

From: <> On Behalf Of Gettes, Michael
Sent: Thursday, July 12, 2018 4:22 PM
To: Crawford, Jeffrey <>
Cc:
Subject: Re: [grouper-users] Hopefully quick question regarding rabbitmq and grouper

 

Yes, it is the same - but, as an example, i have had messaging running about every 5 seconds.  So it can be rather quick response.  The whole loader process is actually quite elegant.  The quartz mechanism scales to many hosts - all those running the loader.  TIER packaging has moved to calling it the daemon - better terminology (I personally don’t like the word but loader isn’t good either).  Don’t get hung up on the loader and changeelog - just use it.  It’s cool stuff.

 

Now all that said, the only item in all this that worries me is the tempchangelog to changelog processing.  It’s a necessary evil cuz this is where changes are labelled/categorized.  It’s also the single threading aspect of processing changes that worries me the most.  Make sure these tables are on fast storage, properly indexed and so on - usual stuff.  Beyond that - like I said - good stuff.

 

I hope this helps.

 

/mrg

On Jul 12, 2018, at 3:43 PM, Crawford, Jeffrey <> wrote:

 

Greetings team,

 

How does the RabbitMQ messages get processed. Is it similar to the psp where changes go into the changelog and a process then periodically polls it? Or do messages get sent as soon as they are sent to the changelog? My guess it’s still a polling system since changes can come in from the WS or UI but the loader has traditionally been the “batch” processing component which does the back end stuff.

 

Thanks

Jeffrey C.