Grouper Developers Forum

[Shilen Patel <>]

Yeah we talked about doing locking before but I think avoided it due to the costs.  We also talked about having checks either on post commit or a change log consumer.  At the time we discussed it last, we decided a daemon would be an easy/good enough fix for now.  But I think a better fix is worth considering.  I'm also going to look at the code and especially the loader (where these issues are presumably most happening) to see if there's a better approach. 

- Shilen 

Sent from my Verizon, Samsung Galaxy smartphone

-------- Original message --------

From: "Black, Carey M." <>

Date: 10/20/17 2:58 PM (GMT-05:00)

To:

Subject: [grouper-dev] Bad Membership Finder Utility  .... Was: ( [grouper-users] Composite group problems)

Shilen ( et. al.) ,

 

I suspect that this condition is barely understood about the inner workings of the system by most deployers and/or users.

Giving the AM ( Access Management ) team control over the promptness of the “fully effective By” window would be a very powerful tool/control that Grouper could assert.  ( Or at least knowledge of when the changes are fully verified/effective in Grouper.)

 

 

I remember reading about the Bad Membership Finder Utility and thinking:

                “What??!?!? Why?? Can that not be done on any membership change? So that the “invalid state” for the composites is avoided all together?”

Then I wandered off to other things, and have only bumped into the though a few times sense. (and every time decided “I don’t have time to think about it right now.” And wandered off again.)

 

Well here are some more thoughts on the implementation details:

 

 

I think I see some of the complexities here, but I doubt that I see all of them.

                The graph of dependencies for all of the composite groups seems knowable. (Possibly changing as a result of the change too. )

                When any one of those dependencies are changed (for a given subject/group) it seems possible to walk/correct the affected parts of the graphs. ( in line, or at least triggering an event to config/fix as needed.)

                And since circular membership loops are possible it gets a bit more well…. Interesting…. to know where to start and stop some of these things too.  :^/ Which is why the process is run repeatedly till it does not make changes. (Inferring that the dependency graph may not be used to optimize the path of changes, or the solution may not be done in a way that “pushes multiple peer changes” in a “top down” kind of flow.)

 

                You alluded to a possible race condition in the system too. ( in the thread below. )

                                So apparently the backend does not do “locking” in a way to deal with overlapping dependency tree changes in a full ACID compliance way. ( no doubt complicated by the “circular membership loops” )

                                Likely trying to avoid DB thread deadlocks… ok… but…. Maybe, in some cases, that blocking would actually be desired by the users/system?

                So, in this non-blocking model, with an attempt to not “thrash the system” and make every save take minutes, I could see the idea of stacking up a queue (change log consumer?) to process these checks/corrections in a timely ( a few minutes, < 2 ? , < 5 ? ) manor.

                But maybe that “acceptable delay” should be a property of the composite and/or the groups that are involved?

                                I could imagine some cases were you always want:

                                                ) change in “this special group” to be 100% completed “NOW”. ( Global Exclude groups, or just “important” Exclude groups, etc… And maybe even in some “include groups” too.)

                                                ) While other groups may be ok to be “loosely consistent”. ( “Provision that group in the next hour and no harm done. “ )

                                Especially when in the context of Access Control Policies and/or Provisioning/DE provisioning. So maybe membership changes also needs to force (queue/call) Prov/DeProv processes too?

 

 

I doubt all of that would be easy to do. ( In fact, I think most of it would be fairly hard to implement without potentially causing other issues.)

 

But does the core idea of making this “loosely consistent model tunable by the users” make sense to anyone else?

 

--

Carey Matthew

 

From: [mailto:] On Behalf Of Shilen Patel
Sent: Friday, October 20, 2017 11:40 AM
To: Dave Churchley <>
Cc:
Subject: Re: [grouper-users] Composite group problems

 

That could have happened if you have composites that are factors of other composites.  There may be other reasons too though.  FWIW, the daemon in 2.3 is designed to take that into account and runs multiple times until all the issues are corrected.

 

- Shilen 

 

 

 

Sent from my Verizon, Samsung Galaxy smartphone

 

 

-------- Original message --------

From: Dave Churchley <>

Date: 10/20/17 10:12 AM (GMT-05:00)

To: Shilen Patel <>

Cc:

Subject: RE: [grouper-users] Composite group problems

 

Yes, that makes sense. I’ve now scheduled the Bad Membership Finder Utility to run each morning, just after the Grouper loader groups are populated.

 

One more question, if you don’t mind, do you know why it took four goes to fix all the bad memberships this morning? Each time I ran it, it fixed the majority that it had found, but not all of them.

 

Thanks
Dave

 

From: [] On Behalf Of Shilen Patel
Sent: 20 October 2017 11:46
To: Dave Churchley <>
Cc:
Subject: Re: [grouper-users] Composite group problems

 

OK cool.  The scenario you described is a similar situation where you probably have two threads running at the same time that deleted and added the user from/to the groups under Group B.  The thread that deleted the user from the group under B would have thought the user wasn't an effective member of B anymore and would have deleted the user from A.  Meanwhile the thread that added the user to the group under B would have seen that the user is already in A (since the other transaction may not have committed yet) and therefore wouldn't have done anything else.  Does that at least explain the issue?  We created the bad membership daemon in 2.3 because of this issue but we may need to handle it better.

 

Thanks!

 

- Shilen 

 

 

 

Sent from my Verizon, Samsung Galaxy smartphone

 

 

-------- Original message --------

From: Dave Churchley <>

Date: 10/20/17 1:28 AM (GMT-08:00)

To: Shilen Patel <>

Cc:

Subject: RE: [grouper-users] Composite group problems

 

Thanks again Shilen

 

The Bad Membership Finder Utility cleaned up 118 bad memberships this morning. (I had to run it four times though until it found no membership errors.) I’m sure this utility will be useful to us!

 

I don’t know if you’re able to explain why the composite group memberships go wrong as in the example below?

 

Thanks
Dave

 

From: [] On Behalf Of Dave Churchley
Sent: 19 October 2017 20:56
To: Shilen Patel <>
Cc:
Subject: Re: [grouper-users] Composite group problems

 

Thanks Shilen, I'll take a look at that in the morning.

 

For at least one of the problem groups, however, this is definitely not the issue. Group A is made up of Group B complement Group C. Person X was in B and not C and so he rightly appeared in A. Two days ago he dropped out of A but he remained in B and not in C. Those groups' memberships hadn't changed. We did notice that Group B was built from several Grouper loader groups and Person X had moved from one of them into another, but this meant that he was still a member of B.

 

I hope I've managed to explain that!

 

Thanks

Dave

 

---

From: Shilen Patel <>
Sent: Thursday, October 19, 2017 6:50:14 PM
To: Dave Churchley
Cc:
Subject: Re: [grouper-users] Composite group problems

 

If both factors in the composite are being updated at about the same time for the same subject, then a membership may be missed.  For example, if Group A is an intersection of Group B and Group C and you add the same person to B and C at almost the same time, then due to the way the database transactions work, the composite membership may not be added to Group A.  Does that seem like the problem you’re having?

If so, this can be corrected using bad membership finder.  You can automatically run it (e.g. every hour) using a built-in daemon though that’s only available in 2.3.  Prior to 2.3, it can be run manually.

https://spaces.internet2.edu/display/Grouper/Bad+Membership+Finder+Utility

- Shilen

On 10/19/17, 11:56 AM, "" <> wrote:

    Good afternoon
   
    We've been using composite groups for a few years now, mainly using the complement operation with exclusion groups. Since the summer, we've also had a few intersection groups.
   
    We've not noticed any problems with group memberships until a couple of weeks ago. (That's not to say there definitely weren't any issues, but there certainly was not a noticeable impact.) At first it was just with the intersection groups but today we've seen it with a complement group, too.
   
    It looks like the composite isn't calculating its membership correctly. Editing the composite and resaving it forces it to recalculate the membership. We've been using this as a workaround but it's obviously not ideal. Some of these composite groups are critical to access control for various systems across the University, including physical access into buildings.
   
    Is this a known issue? Is there anything we can do about it? We're running Grouper v2.2.2. Is there a patch or a better workaround?
   
    Any help would be greatly appreciated as this is a big issue for us at the moment.
   
    Thanks in advance.
   
    Dave Churchley
    Newcastle University