Grouper Users - Open Discussion List

["Bee-Lindgren, Bert" <>]

Within PSPNG, there are four provisioning "queues:"

1) Incremental

  - driven by the Changelog

2) Full-Sync-Retry queue

  - Where failed FullSyncs go when they fail

3) Full-Sync queue

  - Scheduled full sync

4) Full-Sync-ASAP queue

  - From messaging group name to pspng_full_sync_<provisioner name>

  - From certain incremental changes [GRP-1971 describes making these less urgent])

The problem is there is no way to stop a FullSync provisioner once it is working on a group. It's  technically possible to turn a gsh session into a one-shot provisioner for a group, but I just created GRP-1970 to document doing this with a more supportable interface.

So...

For now, the best you can do

a) Run multiple Daemons, and

b) Send a grouper-messaging message to the FullSync provisioner to put your group at the front of the line:

Something like:

provisioner_name="ad";

group_name="app:xyz:pdq";

gs = GrouperSession.startRootSession();
edu.internet2.middleware.grouperClient.messaging.GrouperMessagingEngine.send(new edu.internet2.middleware.grouperClient.messaging.GrouperMessageSendParam().assignGrouperMessageSystemName(GrouperBuiltinMessagingSystem.BUILTIN_NAME).assignQueueType(edu.internet2.middleware.grouperClient.messaging.GrouperMessageQueueType.queue).assignQueueOrTopicName("pspng_full_sync_" + provisioner_name).addMessageBody(group_name));

---

From: <> on behalf of Andre Daniels <>
Sent: Sunday, December 16, 2018 4:48 PM
To:
Cc: Andre Daniels; Chris Hyzer;
Subject: Re: [grouper-users] loader FIFO

 

Carey,

Thanks for the reply. It's a busy year-end so I haven't had time to look into your/suggestions/questions. 

Hooks are more on the “interactive UI” ( but not strictly interactive the work can be done via async to the UI threads as well) activities for the system.

 I looked into hooks and I am not yet sure how this applies. I am looking for a way to have UI driven changes provision to ldap immediately even while a large group is being provisioned. This may not fit in with the design of grouper or good practice for data management. Are you suggesting that UI changes use a hook to run custom code?

Should those things only made in Grouper after some external system state is changed?

                                Maybe that order should be changed to allow the user to “do all of their work” and “let the system process the CLC” later?

 

Sorry, again, I am not sure what you are describing here. My (maybe too) simple view of Grouper is that all provisioning runs through the two change log entry tables and that the provisioner picks data off GROUPER_CHANGE_LOG_ENTRY to act upon. I see there are point in time tables but I don't yet know how they fit into the picture. Again, the basic problem I am trying to solve is that user A creates a large group, shortly after user B creates a large group, how can I get user B's data to provision to ldap quickly? Some large groups take a long time to provision and it seems that user B's change is place at the end of the queue. 

Are there some additional docs and diagrams that I need to read maybe?

Andre

On Tue, Dec 11, 2018 at 7:06 AM Black, Carey M. <> wrote:

Andre,

 

Maybe you are using the wrong tools in Grouper to achieve your goals?

 

My personal philosophy ( IMHO and subject to debate by the list J ) :

 

                Change Log Consumers (CLC) should be for outbound flow of changes from Grouper to other systems.

                                Note: You can also loop the CLC’s through a messaging systems for this too.

 

                Hooks are more on the “interactive UI” ( but not strictly interactive the work can be done via async to the UI threads as well) activities for the system.

 

 

Which leads me to these question:

 

                What are your CLC’s doing that you would expect to see in the UI?

                Should those things only made in Grouper after some external system state is changed?

                                Maybe that order should be changed to allow the user to “do all of their work” and “let the system process the CLC” later?

 

 

Additionally another “random” (yet maybe related) question:

                What RDMBS are you using?

                Are your DBA’s seeing any long running queries? ( If so, what are they?)

 

 

And note: You should always have at least one loader/daemon running.

                If it is not running all of the time, then there will be “a lot of work” to catch up on when it starts and that then will be an avalanche of CLC activities too. ( And some things that are done with Grouper rules will also not be enforced regularly either. )

 

--

Carey Matthew

 

From: <> On Behalf Of Hyzer, Chris
Sent: Monday, December 10, 2018 5:00 PM
To: ;
Subject: RE: [grouper-users] loader FIFO

 

The change log starts as a single queue as id’s get assigned and point in time and flattened memberships are calculated.  The UI will still work though.

 

If you have a recent grouper then you can/should run multiple loaders.  We have 2.3 at penn and run 5 loader processes.  Might be overkill, but just as a datapoint

 

Thanks

Chris

 

From: <> On Behalf Of Andre Daniels
Sent: Monday, December 10, 2018 3:48 PM
To:
Subject: [grouper-users] loader FIFO

 

Are all grouper changesets put into a single queue? It appears that new changes are starved while while large jobs are processing. Is there a way to keep the UI responsive while a sync or some other large job is being processed? Similarly, is it possible to run 2 loaders? I was cautioned not to, but I would like to have a loader that starts up periodically and runs a sync.

 

Thanks,

Andre
--

Andre Daniels 

Sr. Developer/Security Analyst

University of California Santa Cruz

(831) 459-1980

--

Andre Daniels 

Sr. Developer/Security Analyst

University of California Santa Cruz

(831) 459-1980