Grouper Users - Open Discussion List

["Gettes, Michael" <>]

Regarding changes, yup, you are right.  BUT, we don’t control when changes get thrown at us.  For example, when admissions chooses to accept 35,000 people, they pull the trigger whenever their heart desires.  That’s just one example.  So, lots of changes at once is a fact of life.  Also, as you design your access and provisioning policies (Carl - do I get brownie points for not calling them groups?) there will likely be a cascade effect given composite groups and groups within groups and so on.  So a single add can result in 10s and more of other groups being modified.  I would much prefer to no longer have to worry about when and how many changes happen - it’s the nature of the beast - and concentrate on efficient ways of handling the changes.  And as for memory issues… just add more.  :-)

Your wish-list idea is really just monitoring (https://spaces.internet2.edu/display/Grouper/Grouper+diagnostics).  The grouper status page can help with that and make Nagios or something look for SUCCESS.  If not SUCCESS - then something in grouper land needs attention - notify admins to go look.

GSAK - Grouper Swiss Army Knife.  cough. cough.  :-)

Mr. Hyzer… regarding the temp-changelog processors idea you floated.  Sorry if I sound like a broken record… if you could fix the problem of getting messages processed (maybe using the temp-changelog processor idea) as fast possible to the tune of messages out of grouper at a rate of 100/sec or better (messaging systems can definitely process at this and higher rates) then the concern about changelog consumers handling the temp-changelog issues really goes away and also allows the community to get more involved in writing code to provision applications with interesting strategies like bulk processing of message queues (JMS has some interesting abilities here) and high volume processing with multiple queue consumers and so on.  I’d even consider a redo of PSPNG as a message consumer getting it out of grouper altogether.  Yes, I am still floating that idea.  I appreciate why the grouper changelog is what it is but I see so much more flexibility by getting messages out of grouper as quickly as possible and passing to queues and letting other stuff outside of grouper solve provisioning problems.  Just sayin… again.  :-)

And with TIER packaging including rabbitmq along with Grouper that addresses what I was wanting a few years ago of including messaging with Grouper and not having to have grouper be the “provisioner to all apps” concern.  This allows for a re-think on the deployment strategy.  Personally, I still believe all changes should go out of grouper as messages and then the changelog issues for grouper become rather simple and then grouper can concentrate on being, well, grouper.  Yes, I know, we are effectively getting this now.  There’s limited dev resources (you, Shilen, Bert and all the other wonderful talent - I just worry about there being too much to do with not enough talent and this is why I think about these sorts of issues).  No need to reply unless you feel it’s warranted.

/mrg

On Jul 13, 2018, at 9:48 AM, Black, Carey M. <> wrote:

RE: tempchangelog to changelog processing

 

Michael, I kind of agree.

                However, it really should only be a bottle neck if you have LOTS of changes happening at the exact same time, for a sustained time.

                                Or the daemon is “off line for a while”…. ( Don’t do that. J )

                So don’t do large batch processes and the “Event/message model” works much better. AKA: Even flow or tsunami?  J

 

 

I think the other thing to keep in mind is that the loader ( er… “daemon”) can be very RAM hungry.

                In my experience, especially for large LDAP jobs. (100K+ members in a group)

                And since the loader jobs are currently only schedulable to ‘start at “x”’ it can be hard to schedule them to not overload the daemon if some are long running and others “start up” while the big jobs are still running”.

 

 

As a “wish list idea”….

                If a change log consumer gets “to far behind” send notifications to the admins for the change log consumer. ( and the Grouper Admins too. )

                Which could also include the tempchangelog size too. J

 

 

 

As far as the name goes it is both a loader and a daemon. I am just not clear about how to make it “just one of them”. So I think it has a bit of a split personality.

                Pull data in,  (loader jobs: SQL/LDAP/custom)

                Push data out, ( change log consumers, provisioner feed by a change log consumer (IE: box) ,  )

                and do house cleaning too. ( rules engine, …. )

 

                Can we call it the “Grouper Swiss army knife”? ( Or is the fish afraid of knives?  J  )

 

--

Carey Matthew 

 

From:  <> On Behalf Of Gettes, Michael
Sent: Thursday, July 12, 2018 4:22 PM
To: Crawford, Jeffrey <>
Cc: 
Subject: Re: [grouper-users] Hopefully quick question regarding rabbitmq and grouper

 

Yes, it is the same - but, as an example, i have had messaging running about every 5 seconds.  So it can be rather quick response.  The whole loader process is actually quite elegant.  The quartz mechanism scales to many hosts - all those running the loader.  TIER packaging has moved to calling it the daemon - better terminology (I personally don’t like the word but loader isn’t good either).  Don’t get hung up on the loader and changeelog - just use it.  It’s cool stuff.

 

Now all that said, the only item in all this that worries me is the tempchangelog to changelog processing.  It’s a necessary evil cuz this is where changes are labelled/categorized.  It’s also the single threading aspect of processing changes that worries me the most.  Make sure these tables are on fast storage, properly indexed and so on - usual stuff.  Beyond that - like I said - good stuff.

 

I hope this helps.

 

/mrg

On Jul 12, 2018, at 3:43 PM, Crawford, Jeffrey <> wrote:

 

Greetings team,

 

How does the RabbitMQ messages get processed. Is it similar to the psp where changes go into the changelog and a process then periodically polls it? Or do messages get sent as soon as they are sent to the changelog? My guess it’s still a polling system since changes can come in from the WS or UI but the loader has traditionally been the “batch” processing component which does the back end stuff.

 

Thanks

Jeffrey C.