Skip to Content.
Sympa Menu

grouper-users - RE: [grouper-users] loader FIFO

Subject: Grouper Users - Open Discussion List

List archive

RE: [grouper-users] loader FIFO


Chronological Thread 
  • From: "Black, Carey M." <>
  • To: "" <>
  • Cc: "Hyzer, Chris" <>, "" <>
  • Subject: RE: [grouper-users] loader FIFO
  • Date: Tue, 11 Dec 2018 15:06:08 +0000
  • Accept-language: en-US
  • Authentication-results: spf=pass (sender IP is 128.146.138.8) smtp.mailfrom=osu.edu; internet2.edu; dkim=pass (signature was verified) header.d=osu.edu;internet2.edu; dmarc=pass action=none header.from=osu.edu;
  • Authentication-results-original: spf=none (sender IP is ) ;
  • Ironport-phdr: 9a23:QzfosBxokCKyc9fXCy+O+j09IxM/srCxBDY+r6Qd2u4VIJqq85mqBkHD//Il1AaPAd2Lraocw8Pt8InYEVQa5piAtH1QOLdtbDQizfssogo7HcSeAlf6JvO5JwYzHcBFSUM3tyrjaRsdF8nxfUDdrWOv5jAOBBr/KRB1JuPoEYLOksi7ze+/94HQbglSmDaxfa55IQmrownWqsQYm5ZpJLwryhvOrHtIeuBWyn1tKFmOgRvy5dq+8YB6/ShItP0v68BPUaPhf6QlVrNYFygpM3o05MLwqxbOSxaE62YGXWUXlhpIBBXF7A3/U5zsvCb2qvZx1S+HNsDwULs6Wymt771zRRHolikJKiI5/m/UhMx+jq1boQ6uqBNkzoHOfI2ZKOBzcr/Bcd4YQ2dKQ8ZfVzZGAoO5d4YDAfcMPeJCoIbhpFsCtQa1CBe2C+P11DBJhmL906og3+gvDArL2xcvH9YUv3vIr9X5LroeXf67waTTyjXDdO5W1S3j54fVbxAsuPeBVq9zf8rJ0UQjCRnKgkmNpYHgIj+Zy/kBvm2V7+dvSe6jl2sqqw9vrTWg2sghj4zEi4AJxl3E7Sl13oc4KcOkREN+fdKpFYZcuDuHO4dqR84vQWNluCgmxbADupO2cycHxZQiyhPRd/OIbo2F7Q/9W+mLJTp1gXBod6+/ihuy6USgxPPzW8qo3FtPqydJjMTAuW0P2hHW5MWLVPtw8V2k1DuP1Q3e6v1ILEU7mKXBL5Mu2Lg9nYcJv0vZBC/5gkD2gbeWdko6/uio7PzqbK36q5GbK4N5hRjyPLk2lMCiGOg4NRMBUHaB9eS7yb3j4Vb2QLJXjv0wj6bVqojaJd4cpq6lHQBazpoj6xe4DzegytgYmmQHLE5BeBKAiIjpOEvCL+z/Dfe6m1iskTFryO7aPrD5HJnCNGTPnKrkcLpg9kJQ1Q8+zd5Q6p5IFr0MJfzzVVH+udPECxI0Ng65z/rnBdll04MRQ2OPAquXMKPItl+I4/oiLPKIZIAPpDn9KOIp6+PwgXI3glIderKp3YYNZH+mA/tmPl2VYWDwjdcZDWcKog0+QfTliF2YVj5Tem6yULwm5j0iFYKmFpnMRpq2gLGa2Ce7H4ZWZnxdClySC3vodoOEW+sSZyKIJM9ujCAEWaa7R4A/yB6uqVyy970yZOPS9jcVnY/oz9Mz6uHO31lm/Dp4Fcm1zmqWRCd5kn5eFBEs26Uq62Z5w1yAleBTiuZVBJR2obkBBg0+PJXfibUgUPj1QR+Hc9uUHgX1Cu66CC08G4pii+QFZFxwTo341ELKwjaqDrkJlreCGJ0z9OfG0mPsI9pmkieehrI5gQw+S9BUfS29i6hz+hKbJraBkl7Rzvz6M/1AgmiUrzfFnSvr3glDVRJoF6DMXHQRfEzT+NPi+wXPQ6L9QbUhLgZbz8OecO1HZsC6xVlFRfK2IN3FeCrxgGa/Ax+U26mBJJXjYC0D0T/cBkkJn0F2nz6GOAEyCz3nrzfZFyEoGF7yMELq7eRkrn6nFAk5wxzZJ0Fk3qC+rwYcnufUQvgP37UC7SEmrTg8HFu009/MTdSaoA80e75BJ94x/QRK
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:99

Andre,

 

Maybe you are using the wrong tools in Grouper to achieve your goals?

 

My personal philosophy ( IMHO and subject to debate by the list J ) :

 

                Change Log Consumers (CLC) should be for outbound flow of changes from Grouper to other systems.

                                Note: You can also loop the CLC’s through a messaging systems for this too.

 

                Hooks are more on the “interactive UI” ( but not strictly interactive the work can be done via async to the UI threads as well) activities for the system.

 

 

Which leads me to these question:

 

                What are your CLC’s doing that you would expect to see in the UI?

                Should those things only made in Grouper after some external system state is changed?

                                Maybe that order should be changed to allow the user to “do all of their work” and “let the system process the CLC” later?

 

 

Additionally another “random” (yet maybe related) question:

                What RDMBS are you using?

                Are your DBA’s seeing any long running queries? ( If so, what are they?)

 

 

And note: You should always have at least one loader/daemon running.

                If it is not running all of the time, then there will be “a lot of work” to catch up on when it starts and that then will be an avalanche of CLC activities too. ( And some things that are done with Grouper rules will also not be enforced regularly either. )

 

--

Carey Matthew

 

From: <> On Behalf Of Hyzer, Chris
Sent: Monday, December 10, 2018 5:00 PM
To: ;
Subject: RE: [grouper-users] loader FIFO

 

The change log starts as a single queue as id’s get assigned and point in time and flattened memberships are calculated.  The UI will still work though.

 

If you have a recent grouper then you can/should run multiple loaders.  We have 2.3 at penn and run 5 loader processes.  Might be overkill, but just as a datapoint

 

Thanks

Chris

 

From: <> On Behalf Of Andre Daniels
Sent: Monday, December 10, 2018 3:48 PM
To:
Subject: [grouper-users] loader FIFO

 

Are all grouper changesets put into a single queue? It appears that new changes are starved while while large jobs are processing. Is there a way to keep the UI responsive while a sync or some other large job is being processed? Similarly, is it possible to run 2 loaders? I was cautioned not to, but I would like to have a loader that starts up periodically and runs a sync.

 

Thanks,

Andre
--

Andre Daniels 

Sr. Developer/Security Analyst

University of California Santa Cruz

(831) 459-1980




Archive powered by MHonArc 2.6.19.

Top of Page