Subject: Grouper Developers Forum
[grouper-dev] RE: Bad Membership Finder Utility .... Was: ( [grouper-users] Composite group problems)
- From: "Hyzer, Chris" <>
- To: "Black, Carey M." <>, "" <>
- Subject: [grouper-dev] RE: Bad Membership Finder Utility .... Was: ( [grouper-users] Composite group problems)
- Date: Mon, 23 Oct 2017 05:15:39 +0000
- Accept-language: en-US
- Authentication-results: spf=none (sender IP is ) ;
- Spamdiagnosticmetadata: NSPM
- Spamdiagnosticoutput: 1:99
The problem is the queries to find bad memberships can be very costly and disruptive. My DBAs have yelled at me about it. Maybe a targeted change log consumer to check only affected groups could be a good idea. We can think about it after 2.4 is released… Otherwise getting to 2.3 and letting the daemon run nightly should be sufficient as a stopgap…
Shilen ( et. al.) ,
I suspect that this condition is barely understood about the inner workings of the system by most deployers and/or users.
Giving the AM ( Access Management ) team control over the promptness of the “fully effective By” window would be a very powerful tool/control that Grouper could assert. ( Or at least knowledge of when the changes are fully verified/effective in Grouper.)
I remember reading about the Bad Membership Finder Utility and thinking:
“What??!?!? Why?? Can that not be done on any membership change? So that the “invalid state” for the composites is avoided all together?”
Then I wandered off to other things, and have only bumped into the though a few times sense. (and every time decided “I don’t have time to think about it right now.” And wandered off again.)
Well here are some more thoughts on the implementation details:
I think I see some of the complexities here, but I doubt that I see all of them.
The graph of dependencies for all of the composite groups seems knowable. (Possibly changing as a result of the change too. )
When any one of those dependencies are changed (for a given subject/group) it seems possible to walk/correct the affected parts of the graphs. ( in line, or at least triggering an event to config/fix as needed.)
And since circular membership loops are possible it gets a bit more well…. Interesting…. to know where to start and stop some of these things too. :^/ Which is why the process is run repeatedly till it does not make changes. (Inferring that the dependency graph may not be used to optimize the path of changes, or the solution may not be done in a way that “pushes multiple peer changes” in a “top down” kind of flow.)
You alluded to a possible race condition in the system too. ( in the thread below. )
So apparently the backend does not do “locking” in a way to deal with overlapping dependency tree changes in a full ACID compliance way. ( no doubt complicated by the “circular membership loops” )
Likely trying to avoid DB thread deadlocks… ok… but…. Maybe, in some cases, that blocking would actually be desired by the users/system?
So, in this non-blocking model, with an attempt to not “thrash the system” and make every save take minutes, I could see the idea of stacking up a queue (change log consumer?) to process these checks/corrections in a timely ( a few minutes, < 2 ? , < 5 ? ) manor.
But maybe that “acceptable delay” should be a property of the composite and/or the groups that are involved?
I could imagine some cases were you always want:
) change in “this special group” to be 100% completed “NOW”. ( Global Exclude groups, or just “important” Exclude groups, etc… And maybe even in some “include groups” too.)
) While other groups may be ok to be “loosely consistent”. ( “Provision that group in the next hour and no harm done. “ )
Especially when in the context of Access Control Policies and/or Provisioning/DE provisioning. So maybe membership changes also needs to force (queue/call) Prov/DeProv processes too?
I doubt all of that would be easy to do. ( In fact, I think most of it would be fairly hard to implement without potentially causing other issues.)
But does the core idea of making this “loosely consistent model tunable by the users” make sense to anyone else?
That could have happened if you have composites that are factors of other composites. There may be other reasons too though. FWIW, the daemon in 2.3 is designed to take that into account and runs multiple times until all the issues are corrected.
Sent from my Verizon, Samsung Galaxy smartphone
-------- Original message --------
From: Dave Churchley <>
Date: 10/20/17 10:12 AM (GMT-05:00)
To: Shilen Patel <>
Subject: RE: [grouper-users] Composite group problems
Yes, that makes sense. I’ve now scheduled the Bad Membership Finder Utility to run each morning, just after the Grouper loader groups are populated.
One more question, if you don’t mind, do you know why it took four goes to fix all the bad memberships this morning? Each time I ran it, it fixed the majority that it had found, but not all of them.
- [grouper-dev] Bad Membership Finder Utility .... Was: ( [grouper-users] Composite group problems), Black, Carey M., 10/20/2017
- Re: [grouper-dev] Bad Membership Finder Utility .... Was: ( [grouper-users] Composite group problems), Shilen Patel, 10/20/2017
- [grouper-dev] RE: Bad Membership Finder Utility .... Was: ( [grouper-users] Composite group problems), Hyzer, Chris, 10/23/2017
- [grouper-dev] RE: Bad Membership Finder Utility .... Was: ( [grouper-users] Composite group problems), Black, Carey M., 10/23/2017
Archive powered by MHonArc 2.6.19.