Grouper Developers Forum

[Tom Barton <>]

Catching up, reflecting, and prepping for our call tomorrow leads me to 
the following thoughts:

BertBL and MRG remind us to remain pragmatic and stay focused on solving 
actual rather than hypothetical problems. Amen.

I know of actual use cases for veto (Bert, is that what you mean by 
"filter"?) and for notification (including notification to an audit 
log). I don't think I know any, as yet, for plug-ins to enhance events. 
Nor do I see why any such uses would have to be tightly-coupled to the 
originating event, in the same transaction. Should we leave custom 
enhancement capability out of the hooks architecture, for the time 
being? Would that also make moot special considerations regarding 
asynchronism & loops?

Regarding reliability of notification, I don't want us to build a 
reliable asynchronous messaging system. If reliable async messaging is 
needed in Grouper, I want to rely on 3rd parties for it.

However, I *do* want reliable audit history. And I don't want that to 
depend on a separate process calling back to the DB for the details on 
what changed, in case it may have changed again in the meanwhile. I'd 
like instead to have a logical description of each completed transaction 
preserved somewhere in the grouper DB when the transaction completes. 
This history-of-changes table could have a configurable depth (time or 
size) and would necessarily serialize all transactions, thus also 
providing the basis for reliable notification by 3rd party 
implementations. The hard part of this is deciding what the "logical 
descriptions" are - it's not which columns in which tables have what new 
values. Think of LDIF as a motivating example. The grouper event log, at 
INFO level, is probably capturing the right events, but not with the 
right schema.

Regarding how grouper & signet hooks implementations should or shouldn't 
be alike, I think that they can share a common "contract" with plug-in 
developers, but implementation within each respective API will be 
particular to each.

Tom

Chris Hyzer wrote:
> Let me try to summarize how the discussion on hooks / notifications
> went today.
>
> First of all there will be some differences between Grouper and
> Signet with hooks.  E.g. the pre and post in Grouper are in the same
> transaction, and not so in Signet (unless and until long running
> transactions are implemented in signet).
>
> In general we would like to start out with hooks and notifications by
> designing and implementing a straightforward system, and then augment
> it later as needs arise.  Here are some examples:
>
> 1. Toss out the reliable notification mechanism for now.
> Implementers of hooks would need to write their own notification
> mechanism (could be reliable in Grouper, could not be reliable in
> Signet due to lack of long running transactions).  We can put that in
> the API later since having it would save a lot of work on the
> implementer.
>
> 2. Have only one hook implementation per event.  The side effect here
> is that instead of registering multiple listeners like this:
>
> GrouperHooks.registerGroupAddPreListener(AddAuditing.class);
> GrouperHooks.registerGroupAddPreListever(CheckGroupName.class);
>
> It would be in the listener itself
>
> public void preGroupAdd(HookPreGroupAddContext context) {
>   AddAuditing.preGroupAdd(context);
>   CheckGroupName.preGroupAdd(context);
> }
>
> Since this doesn't save much work on the part of the implementer,
> they can worry about the order, what happens when one fails, etc
>
> 3. Since only one hook implementation per event type, we could give
> an API to make the asynchronous call very easy...
>
> Instead of
>
> public class AddAuditing implement AsynchronousHook ...
>
> You would just call with an API method of some sort that passes data 
> correctly etc, e.g.
>
> public void preGroupAdd(HookPreGroupAddContext context) {
>   HooksUtils.asynchronous(context, new AsynchronousHandler() {
>     public void hookLogic(HooksContext asyncContext) {
>       AddAuditing.preGroupAdd((HookPreGroupAddContext)asyncContext);
>     }});
> }
>
> We are talking about 3 lines of code the implementer needs to add to
> make all or part of the hook asynchronous.
>
> If everyone is all good with all of this, we need to decide what is
> in Grouper/Signet and what is in i2mi-common... I think most of the work
> involved is identifying exactly where the hook points are, and what data
> goes into the context, and coding the contexts. This is all specific to
> Grouper or Signet, so I don't think we need anything in i2mi-common at
> this point... and that would make development easier and faster. But we
> can discuss... :) Maybe another point that Grouper needs is vetoing back
> via exception since we don't want to affect our deeply nested API much
> by passing objects around.
>
> Kind regards,
> Chris
>
> Ps. I said I would give an example of a long running transaction in Grouper, 
> see this page:
>
> https://wiki.internet2.edu/confluence/display/GrouperWG/Hibernate+and+data+layer+updates#Hibernateanddatalayerupdates-Addsupportfortransactions
>
> The example is:
>
>     //note the default is READ_WRITE_OR_USE_EXISTING
>     GrouperTransaction.callbackGrouperTransaction(new 
> GrouperTransactionHandler() {
>
>       public Object callback(GrouperTransaction grouperTransaction)
>           throws GrouperDAOException {
>
>         try {
>           Group.saveGroup(rootSession, null,null,groupName,displayExtension,  
> groupDescription,
>               SaveMode.INSERT, false);
>
>           Group.saveGroup(rootSession, null,null, 
> groupName2,displayExtension, groupDescription,
>               SaveMode.INSERT, false);
>
>           //I believe this has an implicit commit, but the user could also do 
> it explicitly or
>           //with the jdbc Connection
>           return null;
>         } catch (Exception e) {
>           //note, if specific exceptions need to be handled, do that here
>           throw new RuntimeException(e);
>         }
>       }
>
>     });
>
> > -----Original Message-----
> > From: Chris Hyzer
> > Sent: Thursday, May 22, 2008 5:18 PM
> > To: 'Michael R. Gettes'; Mike Olive
> > Cc: 'Grouper Dev'; 
> >
> > Subject: RE: [signet-dev] RE: [grouper-dev] notifications vs hooks
> >
> > > Based on this response I'm not sure you understand my original
> > > argument.
> > Im not sure I completely understood it either, but if it is about db
> > triggers, I think it is a moot point if we understand it.  Now instead
> > of being in the weeds discussing how we will implement, we are
> > discussing how we wont implement.  :)  I don't think we can use
> > triggers, though I see how they do have some advantages...
> >
> > What do people think about my current idea:
> >
> > 1. There are no more "multiple event" registration, since per our last
> > call it gets complicated what order they go in, how they affect each
> > other, etc
> > 2. The API can put code in hooks/notifications, even though it is not
> > registered and implemented like the custom hooks (so we don't need
> > event queue)
> > 3. There is one place to register hook code in pre/post of an
> > operation, and it can be transactional or not or mixture, your choice
> > (well, this is free for grouper, signet might need to implement long
> > running transaction to benefit from this... maybe to start signet is
> > not transactional, I don't know what is easy there)
> > 4. Code a class for hook code (extend a base class), and implement the
> > methods you need.  Register in config file.  If you need multiple, just
> > put multiple things in your method.  You can control order,
> > dependencies, etc.  I picture a handful of hook classes organized by
> > api objects (groups, folders, memberships, etc)
> > 5. Veto by throwing specific vetoexceptions which are unchecked (in
> > hook, not in notification).
> > 6. Hooks are synchronous, notifications are asynchronous.  We will
> > document and provide helper methods for doing asynchronous hooks easily
> > and safely (I think most asynchronous activity will be notifications
> > anyway)
> > 7. In the post-hook, you can call a method to register interest in a
> > notification.  You can do this based on the data that is changing hands
> > (e.g. if you want notification on membership change, but not security
> > membership change).  This will put data in a notification queue table
> > transactionally (e.g. type of data, id of data, type of callback
> > (custom), timestamp, type of change (insert/update/delete), status,
> > errorMessage (if notification fails) etc.
> > 8. We have a daemon which runs every so configurably often to get
> > records off the notification queue, call your callback, if success,
> > remove from queue, if not, use a configurable or customizable retry
> > algorithm.  It will hand you the ID and type of what has changed and
> > you can do what you want it (lookup the data, lookup a custom db view,
> > call a stored proc that you write, whatever).  The notification daemon
> > would be one-to-one with the DB.  Whereas the hooks would be installed
> > in each grouper/signet API (e.g. GSH, UI, WS, loader, etc).
> >
> > I think this design covers many use cases, but is simple and easy to
> > implement.  The things that are cut out (sync/async, multiple logic per
> > hook, etc) can still be accomplished with this design, and based on
> > need we can focus on specific areas in future releases if we like...
> >
> > Thoughts?
> > Thanks,
> > Chris
begin:vcard
fn:Tom Barton
n:Barton;Tom
org:University of Chicago;Networking Services & Information Technology
adr;dom:1155 E. 60th St.;;Rm 309, 1155 Bldg;Chicago;IL;60637
email;internet:
title:Sr. Director - Integration
tel;work:+1 773 834 1700
version:2.1
end:vcard