Grouper Developers Forum

["Michael R. Gettes" <>]

So, I am sure you guys have all thought about this a bit and I am  
curious
about why not the following approach...

instead of creating a software structure (a layering of plugins and such
to handle notifications and modification of data and yada yada) why not
come up with some protocol at the data layer?  at some scratch on the
wall in the db when an item is added to the db and let some other
process notice the change.  if you need "pre-commit" functionality
then add it to some temp db and go from there.  Then you don't have
to worry about putting in this layer in all software components - it's
just a matter of adding some state to an entry and the state can change
to indicate a lifecycle to the entry.

/mrg

On May 21, 2008, at 12:42, Chris Hyzer wrote:

> Well, if we want reliable notifications, then we need transactional  
> hooks.
> If we want augmenters, that is a pre-hook or post-hook.  If we need  
> to know the hibernate id of whatever was inserted, that is a post  
> hook (post the hibernate flush) (e.g. auto add a member to a group  
> just created).  In both cases we definitely want the ability to be  
> in the same transaction.  I think all that is pretty simple since  
> transactions in threadlocal exist in grouper so the hook can use it  
> or not (for the same database).  Reliable notifications is trickier,  
> and something like the daemon design in my previous email would  
> solve it).  I agree with you that if the auditing is in an external  
> database then we don’t need to worry about transactions (as much).
>
> I think a complication is as we discussed in the previous phone call  
> where we have multiple chained hooks, and which order they come in,  
> etc.  If we can cut that down (even to one non-API hook for pre and  
> one for post) it will make it easier to work with.
>
> About the hooks generating more hook events, I think we should  
> ignore this for now, I think it will work itself out.  For instance  
> for a group with a type of “requireFaculty”, maybe the creaetGroup  
> hook will build out a composite group that and’s with a faculty  
> group (if the composite doesn’t already exist).  The composite group  
> will also trigger the createGroup hook, but the hook will see that  
> the composite group doesn’t have the type “requireFaculty” so it  
> will ignore it…  Once we come up with an endless loop requirement, I  
> bet we will be able to do something clever.  Furthermore, I could  
> see the hook code solving it themselves with a threadlocal (set a  
> flag that says we are in the specific hook, clear it in a finally  
> block, and check for the flag at the beginning of the hook).  I  
> don’t think we should allow a hook to turn off all events, there  
> might have been some good ones in there…  J
>
> Kind regards,
> Chris
>
>
>
> From: Bert Bee-Lindgren 
> [mailto:]
> Sent: Wednesday, May 21, 2008 11:53 AM
> To: Mike Olive
> Cc: Chris Hyzer; 'Grouper Dev'; 
>
> Subject: Re: [grouper-dev] notifications vs hooks
>
> Thanks Mike & Chris,
>
> I just get the feeling that pre-hooks and same-transaction semantics  
> (especially across data sources) are solving problems I don't see: I  
> see huge near-term values of pre-hooks as filters. I don't see how  
> these need coordinated commit/rollback with the GrouperDB transaction.
>
> From reviewing Grouper/Signet roadmap issues, I see the following  
> most affected by Hooks & Plugins.
>   -Notification of changes - email/grouper-to-signet/triggered  
> provisioning
>   -History & audit
>   -Rule-based action (not sure if this uses plugins, but some rules  
> could be considered augmentation)
>
> Maybe my ACID religion will take a hit from this, but I'm not sure I  
> see the harm in Auditing and (membership and attribute) Augmentation  
> happening based on reliable notifications after the group's  
> transaction commits. Further, when the group is changed via Post- 
> Hook augmentation, it's possible that other pre-hooks should be  
> triggered (Orig action, pre-hook, hib flush, post-hook that  
> augments, pre-hook because request changed, hib flush, post-hook,  
> commit.... ugh... double-ugh if it loops further or if a pre-hook  
> later disagrees)
>
> Generally, I see the following levels of increasing functionality  
> and complexity. And I see current plans for #4 while #1 or 2 seem  
> (to me) to be so much easier and offer so much value.I see 80% of  
> vision I've heard being done with #1 and 90+% with #2.
>           1) Pre-hooks as filters, reliable post-action  
> notifications [I'd call this the minimum]
>           2) #1 with pre-hooks also as augmenters
>           3) #2 with all pre-hooks seeing the final (augmented) group
>           4) #3 with post-hooks and same-transaction semantics
>
> Am I missing Use Cases, or not considering ACID-lite problems, or am  
> I seeing difficulty where there is none?
> Put differently, if this is all truly necessary, great. Or if this  
> is easier than it seems, even better.
>
> Thanks, yet again, for you patience,
>   Bert
>
> On May 21, 2008, at 11:03 AM, Mike Olive wrote:
>
>
> Bert,
>
> Pre-hooks is certainly the more complicated aspect of this design  
> with the implementation
> most likely requiring the use of a transaction manager so that the  
> pre-hook plug-ins may
> participate in the final commit or rollback of the actual transaction.
>
> On post-hooks persistence, the current design is agnostic of any  
> messaging solution or
> transportation mechanism of the change notification. The design is  
> extensible such that
> the developer could incorporate a guaranteed messaging solution  
> (client) such as JMS.
> --
> Michael Olive
>
>
>
> From: Bert Bee-Lindgren 
> [mailto:]
> Sent: Wednesday, May 14, 2008 12:05 PM
> To: Chris Hyzer
> Cc: Grouper Dev; 
>
> Subject: Re: [grouper-dev] notifications vs hooks
> Combining our approach to similar problems with UPenn's plans... I  
> think we should consider very different mechanisms for pre-hooks and  
> post-hooks.
>
> Pre-hooks:
> Normal, synchronous method calls
> Pre-hook plugin developers should expect the event to possibly not  
> occur even if they approve it. They should not notify, log, etc  
> anything that might indicate to a user/auditor/sysadmin that an  
> event happened... because they won't know about downstream rejections.
>
>
> On the post-hooks, I think we should consider a persistent post-hook:
>   a) A plugin would have two inbound methods
> a1) "This event happened, do you care?"  [Boolean return, no  
> external processing allowed, must be "fast"]
> a2) "Process this event, let us know when you've succeeded in  
> handling it" [Boolean return, TRUE means this plugin succeeded]
>   b) Create plugin-specific/event-specific database rows in an event  
> table based on the TRUE returns of a1's
>   c) Immediately after all the plugins have had a chance to answer  
> a1, hand the event to all the interested plugins a2's.
>   d) Delete the plugin-specific/event-specific row when that  
> plugin's a2 returns true
>   e) Retry the failed a2's for a plugin before any new a2
>   f) Possibly retry the failed a2's every couple minutes or with  
> some backoff approach (or, disappointingly, wait for the next event)
>
> Yes, this is basically a message queue, but simple to implement (we  
> use a python-based version of this for several event queues in our  
> system). I've looked for a JMS library as simple to use as this two- 
> method approach and haven't found one.
>
>
> On May 14, 2008, at 2:27 PM, Chris Hyzer wrote:
>
> Gary put comments on my hooks page about transactions and  
> notifications.
> https://wiki.internet2.edu/confluence/display/GrouperWG/Hooks+simple
> It makes this whole thing very complicated… if the actions happens  
> in a long running transaction, and you want to be notified at the  
> end, there are a few issues:
> 1.       It is a different architecture than we had been discussing  
> since we need to know about the action at the time of the  
> (successful) commit.  Perhaps using Hibernate’s events could do the  
> trick, but you don’t have any object model anymore, you just have a  
> list of column data for one table
> 2.       Like Gary points out, if the thing you are updating  
> external to grouper fails, how do you log that and catch up later  
> (e.g. if you are calling a web service, and there is a network issue)
> 3.       There are lots of different producers of events (UI, WS,  
> extensions e.g. gsh, and direct db edits [granted they shouldn’t]).   
> Must make sure the notification hooks are registered everywhere, and  
> test them to make sure they are firing everywhere (seems tedious /  
> risky)
> Lets take the use case of writing your own ldappc via notifications  
> (something we will start out with at penn).  We want to know about  
> new members, memberships, and groups.  We will just make 3 tables  
> with the id’s and timestamps of when these change:
> select * from ldap_change_memberships lcm where rownum < 4 order by  
> lcm.LAST_UPDATED desc
> MEMBERSHIP_UUID      LAST_UPDATED
> cd5f23d2-a8c8-44c0-a8b1-a3c3210da3c5                5/8/2008  
> 1:21:37.569272 PM
> 74710fee-40b2-48e4-a8dd-b750876bc4ea             5/8/2008  
> 1:21:37.462896 PM
> f2cf6b80-377c-41c3-981d-0eb9274dc74a                5/8/2008  
> 1:21:36.076199 PM
> On the grouper tables I have some simple triggers that check for  
> diffs and insert to the change tables (and delete old records since  
> all the daemon cares about is the most recently changed record).
> Then we also have friendly views for the daemon to use to query the  
> data:
> select gmv.GROUP_NAME, gmv.SUBJECT_ID, gmv.SUBJECT_SOURCE,  
> gmv.MSHIP_TYPE from grouper_memberships_v gmv where rownum <4
> GROUP_NAME                                  
> SUBJECT_ID                        SUBJECT_SOURCE           MSHIP_TYPE
> Centers:ISC:PennAccess -ext     GrouperSystem                 
> g:isa                                       immediate
> Centers:ISC:PennAccess -ext     GrouperSystem                 
> g:isa                                       immediate
> Centers:ISC:PennAccess -ext     GrouperSystem                 
> g:isa                                       immediate
> Then a daemon will run every 5 minutes to push the new data to ldap,  
> and delete the record from the change table when done.
> So this is all transactional, nothing can slip by (since trigger),  
> and nothing can happen wrong if ldap update fails.
> Triggers are DB specific, and it requires different tables for each  
> notification type.
> For notifications, grouper could provide:
> 1.       If you have certain table structures for last_updated dates  
> (perhaps with name prefixes to support multiple)
> 2.       We could populate with Java hooks (perhaps not reliable per  
> discussion above), or you could do triggers (we could provide  
> examples for certain DB’s) which would be more reliable and probably  
> more performant
> 3.       We could provide a daemon (e.g. runs every 5 minutes) with  
> callbacks that would process the change tables (based on name  
> prefix), and gives you a callback to take the data and put it  
> somewhere
> 4.       We could provide a scheduled way to get all data not just  
> the diffs (e.g. daily or weekly or monthly to ensure things are in  
> sync)
> Just brainstorming here…
> Thanks,
> Chris