Grouper Developers Forum

["Gettes, Michael" <>]

I have made comment similar to this concern to Hyzer and Bert as well related 
to PSPNG.  PSPNG goes into a “degraded” mode when it tries to handle a group 
that has been deleted - it essentially assumes it must handle it.  I deleted 
a group with 100K members and it took hours but the group was never 
provisioned by PSPNG.  I have suggested that certain operations (like a group 
delete) needs to be able to look into the PIT and get the attributes at time 
of deletion regarding that group.  Maybe we need to do the same thing for 
changelog consumers to look into the PIT for deleted users and the attributes 
they had at the time.  I think that would avoid the “multi-phase” processing 
you are suggesting.  Now, I am sure, with all this input and ideas, our dear 
talented grouper devs will come up with the most appropriate way of handling 
these situations going forward.

As for discussing things on wiki’s and such.  I really don’t care - I am just 
following how Chris wants it done.  There are lots of ways of doing things 
and I am pretty much only interested in viable outcomes - how we get there - 
as long as we get there and it doesn’t take forever - is what I care about.

I hope this helps.  Again, I understand your concerns — and I agree with them 
— it’s a matter of how to address them and not just for subjects.  It’s a 
more general problem.

/mrg

> On Feb 6, 2019, at 4:04 PM, Black, Carey M. <> wrote:
> 
> Michael,
> 
> My concern is if the subject it removed before the membership are 
> _processed_ by the "deprovisioner".
>       Things in Grouper are not instantaneous. ( More like "short time 
> bounded batches". And there are different queues and processes that are not 
> guaranteed to be serialized nor synchronized. )
> 
> If the USDU removes the memberships AND deletes the subject (in a single 
> pass), the likely hood of the deprovisioners would have ANY data about the 
> subject is just about zero. ( All the data would need to be "on the 
> Membership" being removed, and that still depends on Attribute point in 
> time data sources too. )
> 
> However, If the deprovisioning process needs Subject data ( that might be 
> cached in Grouper attributes on the subject ) to deprovision  then it is 
> likely not able to do what it needs to do.
> 
> However, if the process is that the USDU removes the memberships and waits 
> to delete the subject until all of those memberships have been processed by 
> all provisioners ( AKA: change log consumers) before the subject is deleted 
> in grouper, then the local Grouper data integrity can sustain the Grouper 
> based processes.
> 
> So to me, the USDU should look like a multi-phase process (with a time 
> delay/verification check before the next phase for that subject) to not 
> break the deprovisioning processes.
> 
> 
> FWIW: 
>       I don't think Wiki's are a good space to make a feature request. ( 
> That what things like Jira are for. :) ) 
>       And I am trying to talk through the details before a formal 
> request/altered request is made. Mostly because I am very open to the 
> reality that I may not be seeing something that others might. :)
>       ( I like discussing details to build full plans. :) )
> 
> -- 
> Carey Matthew 
> 
> 
> -----Original Message-----
> From: Gettes, Michael <> 
> Sent: Wednesday, February 6, 2019 3:15 PM
> To: Black, Carey M. <>
> Cc: 
> Subject: Re: USDU delete subjects after unresolvable for X days
> 
> USDU is supposed to remove the subject from all groups it is a member of as 
> part of it’s processing.  So I think 2 is a non-issue.  Different schedules 
> for different subject sources is certainly an interesting point and one I 
> hadn’t thought of.  As I hope Chris would agree - you should definitely add 
> this to the wiki page so it gets taken into account.
> 
> /mrg
> 
>> On Feb 6, 2019, at 12:59 PM, Black, Carey M. <> wrote:
>> 
>> Michael,
>> 
>> RE: 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__spaces.at.internet2.edu_display_Grouper_USDU-2Bdelete-2Bsubjects-2Bafter-2Bunresolvable-2Bfor-2BX-2Bdays&d=DwIFAg&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=EUBHI54mtQDlbcqo5rUTdQ&m=jd3_GrfSb_TB0hg-n5713kPCCOdcxGAYn3lNN_8eYMI&s=vpZ-XmLY7X99xlpW8gs6wkOnd1umdN9oxb7XjTfChl0&e=
>> 
>> 
>> While I understand the idea of this request, there might be some 
>> unintended consequences if other things are not considered.
>> 
>> 1) some subject sources might need to be treated differently than others.
>>      Maybe different timelines (remove after N days), 
>>      Maybe different logic based on that source. ( some sources may never 
>> need cleaned., or only yearly? )
>>      Some might expect to see lots of subject changes, others might expect 
>> very view. ( so a single "failsafe" might be high for some and low for 
>> others.)
>> 
>> 
>> 2) if an USDU subject is still "provisioned" to some downstream system 
>> then there may be other problems caused by removing the Subject from the 
>> downstream system after the USDU scrubs the subject from grouper.
>>      Maybe there needs to be a step that identifies the unresolvable 
>> subject that triggers downstream provision/de-provision processes before 
>> the subject is removed.
>> 
>> 
>> What are your thoughts along those lines?
>> 
>> -- 
>> Carey Matthew 
>> 
>