Grouper Developers Forum

[Chris Hyzer <>]

So is my description below, plus a built in api "hook" for audit history 
satisfactory?

> log). I don't think I know any, as yet, for plug-ins to enhance events.
> Nor do I see why any such uses would have to be tightly-coupled to the
> originating event, in the same transaction. Should we leave custom
> enhancement capability out of the hooks architecture, for the time
> being?

One example is I would like to be able to set a group type, and have that 
enhance the data for me.  E.g. an activeEmployee group type which would 
create the necessary composite groups for me if they don't exist.  If you are 
talking about changing the same object as it is saved, then this is not a 
valid example, since this would be querying/adding ancillary objects.  Either 
way, as for whether or not the event enhancements are available or occur in 
the same transaction or not, for Grouper I think it is simpler to just have 
that available...

Thanks,
Chris

> -----Original Message-----
> From: Tom Barton 
> [mailto:]
> Sent: Tuesday, May 27, 2008 5:31 PM
> To: 'Grouper Dev'; 
> 
> Subject: Re: [grouper-dev] hooks and notifications
>
> Catching up, reflecting, and prepping for our call tomorrow leads me to
> the following thoughts:
>
> BertBL and MRG remind us to remain pragmatic and stay focused on
> solving actual rather than hypothetical problems. Amen.
>
> I know of actual use cases for veto (Bert, is that what you mean by
> "filter"?) and for notification (including notification to an audit
> log). I don't think I know any, as yet, for plug-ins to enhance events.
> Nor do I see why any such uses would have to be tightly-coupled to the
> originating event, in the same transaction. Should we leave custom
> enhancement capability out of the hooks architecture, for the time
> being? Would that also make moot special considerations regarding
> asynchronism & loops?
>
> Regarding reliability of notification, I don't want us to build a
> reliable asynchronous messaging system. If reliable async messaging is
> needed in Grouper, I want to rely on 3rd parties for it.
>
> However, I *do* want reliable audit history. And I don't want that to
> depend on a separate process calling back to the DB for the details on
> what changed, in case it may have changed again in the meanwhile. I'd
> like instead to have a logical description of each completed
> transaction preserved somewhere in the grouper DB when the transaction
> completes.
> This history-of-changes table could have a configurable depth (time or
> size) and would necessarily serialize all transactions, thus also
> providing the basis for reliable notification by 3rd party
> implementations. The hard part of this is deciding what the "logical
> descriptions" are - it's not which columns in which tables have what
> new values. Think of LDIF as a motivating example. The grouper event
> log, at INFO level, is probably capturing the right events, but not
> with the right schema.
>
> Regarding how grouper & signet hooks implementations should or
> shouldn't be alike, I think that they can share a common "contract"
> with plug-in developers, but implementation within each respective API
> will be particular to each.
>
> Tom
>
> Chris Hyzer wrote:
> > Let me try to summarize how the discussion on hooks / notifications
> > went today.
> >
> > First of all there will be some differences between Grouper and
> Signet
> > with hooks.  E.g. the pre and post in Grouper are in the same
> > transaction, and not so in Signet (unless and until long running
> > transactions are implemented in signet).
> >
> > In general we would like to start out with hooks and notifications by
> > designing and implementing a straightforward system, and then augment
> > it later as needs arise.  Here are some examples:
> >
> > 1. Toss out the reliable notification mechanism for now.
> > Implementers of hooks would need to write their own notification
> > mechanism (could be reliable in Grouper, could not be reliable in
> > Signet due to lack of long running transactions).  We can put that in
> > the API later since having it would save a lot of work on the
> > implementer.
> >
> > 2. Have only one hook implementation per event.  The side effect here
> > is that instead of registering multiple listeners like this:
> >
> > GrouperHooks.registerGroupAddPreListener(AddAuditing.class);
> > GrouperHooks.registerGroupAddPreListever(CheckGroupName.class);
> >
> > It would be in the listener itself
> >
> > public void preGroupAdd(HookPreGroupAddContext context) {
> >   AddAuditing.preGroupAdd(context);
> >   CheckGroupName.preGroupAdd(context);
> > }
> >
> > Since this doesn't save much work on the part of the implementer,
> they
> > can worry about the order, what happens when one fails, etc
> >
> > 3. Since only one hook implementation per event type, we could give
> an
> > API to make the asynchronous call very easy...
> >
> > Instead of
> >
> > public class AddAuditing implement AsynchronousHook ...
> >
> > You would just call with an API method of some sort that passes data
> correctly etc, e.g.
> >
> > public void preGroupAdd(HookPreGroupAddContext context) {
> >   HooksUtils.asynchronous(context, new AsynchronousHandler() {
> >     public void hookLogic(HooksContext asyncContext) {
> >       AddAuditing.preGroupAdd((HookPreGroupAddContext)asyncContext);
> >     }});
> > }
> >
> > We are talking about 3 lines of code the implementer needs to add to
> > make all or part of the hook asynchronous.
> >
> > If everyone is all good with all of this, we need to decide what is
> in
> > Grouper/Signet and what is in i2mi-common... I think most of the work
> > involved is identifying exactly where the hook points are, and what
> > data goes into the context, and coding the contexts. This is all
> > specific to Grouper or Signet, so I don't think we need anything in
> > i2mi-common at this point... and that would make development easier
> > and faster. But we can discuss... :) Maybe another point that Grouper
> > needs is vetoing back via exception since we don't want to affect our
> > deeply nested API much by passing objects around.
> >
> > Kind regards,
> > Chris
> >
> > Ps. I said I would give an example of a long running transaction in
> Grouper, see this page:
> >
> >
> https://wiki.internet2.edu/confluence/display/GrouperWG/Hibernate+and+
> > data+layer+updates#Hibernateanddatalayerupdates-
> Addsupportfortransacti
> > ons
> >
> > The example is:
> >
> >     //note the default is READ_WRITE_OR_USE_EXISTING
> >     GrouperTransaction.callbackGrouperTransaction(new
> > GrouperTransactionHandler() {
> >
> >       public Object callback(GrouperTransaction grouperTransaction)
> >           throws GrouperDAOException {
> >
> >         try {
> >           Group.saveGroup(rootSession,
> null,null,groupName,displayExtension,  groupDescription,
> >               SaveMode.INSERT, false);
> >
> >           Group.saveGroup(rootSession, null,null,
> groupName2,displayExtension, groupDescription,
> >               SaveMode.INSERT, false);
> >
> >           //I believe this has an implicit commit, but the user could
> also do it explicitly or
> >           //with the jdbc Connection
> >           return null;
> >         } catch (Exception e) {
> >           //note, if specific exceptions need to be handled, do that
> here
> >           throw new RuntimeException(e);
> >         }
> >       }
> >
> >     });
> >
> >> -----Original Message-----
> >> From: Chris Hyzer
> >> Sent: Thursday, May 22, 2008 5:18 PM
> >> To: 'Michael R. Gettes'; Mike Olive
> >> Cc: 'Grouper Dev'; 
> >> 
> >> Subject: RE: [signet-dev] RE: [grouper-dev] notifications vs hooks
> >>
> >>> Based on this response I'm not sure you understand my original
> >>> argument.
> >> Im not sure I completely understood it either, but if it is about db
> >> triggers, I think it is a moot point if we understand it.  Now
> >> instead of being in the weeds discussing how we will implement, we
> >> are discussing how we wont implement.  :)  I don't think we can use
> >> triggers, though I see how they do have some advantages...
> >>
> >> What do people think about my current idea:
> >>
> >> 1. There are no more "multiple event" registration, since per our
> >> last call it gets complicated what order they go in, how they affect
> >> each other, etc 2. The API can put code in hooks/notifications, even
> >> though it is not registered and implemented like the custom hooks
> (so
> >> we don't need event queue) 3. There is one place to register hook
> >> code in pre/post of an operation, and it can be transactional or not
> >> or mixture, your choice (well, this is free for grouper, signet
> might
> >> need to implement long running transaction to benefit from this...
> >> maybe to start signet is not transactional, I don't know what is
> easy
> >> there) 4. Code a class for hook code (extend a base class), and
> >> implement the methods you need.  Register in config file.  If you
> >> need multiple, just put multiple things in your method.  You can
> >> control order, dependencies, etc.  I picture a handful of hook
> >> classes organized by api objects (groups, folders, memberships, etc)
> >> 5. Veto by throwing specific vetoexceptions which are unchecked (in
> >> hook, not in notification).
> >> 6. Hooks are synchronous, notifications are asynchronous.  We will
> >> document and provide helper methods for doing asynchronous hooks
> >> easily and safely (I think most asynchronous activity will be
> >> notifications
> >> anyway)
> >> 7. In the post-hook, you can call a method to register interest in a
> >> notification.  You can do this based on the data that is changing
> >> hands (e.g. if you want notification on membership change, but not
> >> security membership change).  This will put data in a notification
> >> queue table transactionally (e.g. type of data, id of data, type of
> >> callback (custom), timestamp, type of change (insert/update/delete),
> >> status, errorMessage (if notification fails) etc.
> >> 8. We have a daemon which runs every so configurably often to get
> >> records off the notification queue, call your callback, if success,
> >> remove from queue, if not, use a configurable or customizable retry
> >> algorithm.  It will hand you the ID and type of what has changed and
> >> you can do what you want it (lookup the data, lookup a custom db
> >> view, call a stored proc that you write, whatever).  The
> notification
> >> daemon would be one-to-one with the DB.  Whereas the hooks would be
> >> installed in each grouper/signet API (e.g. GSH, UI, WS, loader,
> etc).
> >>
> >> I think this design covers many use cases, but is simple and easy to
> >> implement.  The things that are cut out (sync/async, multiple logic
> >> per hook, etc) can still be accomplished with this design, and based
> >> on need we can focus on specific areas in future releases if we
> like...
> >>
> >> Thoughts?
> >> Thanks,
> >> Chris