Grouper Developers Forum

[Chris Hyzer <>]

> Based on this response I'm not sure you understand my original
> argument.

Im not sure I completely understood it either, but if it is about db 
triggers, I think it is a moot point if we understand it.  Now instead of 
being in the weeds discussing how we will implement, we are discussing how we 
wont implement.  :)  I don't think we can use triggers, though I see how they 
do have some advantages...

What do people think about my current idea:

1. There are no more "multiple event" registration, since per our last call 
it gets complicated what order they go in, how they affect each other, etc
2. The API can put code in hooks/notifications, even though it is not 
registered and implemented like the custom hooks (so we don't need event 
queue)
3. There is one place to register hook code in pre/post of an operation, and 
it can be transactional or not or mixture, your choice (well, this is free 
for grouper, signet might need to implement long running transaction to 
benefit from this... maybe to start signet is not transactional, I don't know 
what is easy there)
4. Code a class for hook code (extend a base class), and implement the 
methods you need.  Register in config file.  If you need multiple, just put 
multiple things in your method.  You can control order, dependencies, etc.  I 
picture a handful of hook classes organized by api objects (groups, folders, 
memberships, etc)
5. Veto by throwing specific vetoexceptions which are unchecked (in hook, not 
in notification).
6. Hooks are synchronous, notifications are asynchronous.  We will document 
and provide helper methods for doing asynchronous hooks easily and safely (I 
think most asynchronous activity will be notifications anyway)
7. In the post-hook, you can call a method to register interest in a 
notification.  You can do this based on the data that is changing hands (e.g. 
if you want notification on membership change, but not security membership 
change).  This will put data in a notification queue table transactionally 
(e.g. type of data, id of data, type of callback (custom), timestamp, type of 
change (insert/update/delete), status, errorMessage (if notification fails) 
etc.
8. We have a daemon which runs every so configurably often to get records off 
the notification queue, call your callback, if success, remove from queue, if 
not, use a configurable or customizable retry algorithm.  It will hand you 
the ID and type of what has changed and you can do what you want it (lookup 
the data, lookup a custom db view, call a stored proc that you write, 
whatever).  The notification daemon would be one-to-one with the DB.  Whereas 
the hooks would be installed in each grouper/signet API (e.g. GSH, UI, WS, 
loader, etc).

I think this design covers many use cases, but is simple and easy to 
implement.  The things that are cut out (sync/async, multiple logic per hook, 
etc) can still be accomplished with this design, and based on need we can 
focus on specific areas in future releases if we like...

Thoughts?
Thanks,
Chris

> -----Original Message-----
> From: Michael R. Gettes 
> [mailto:]
> Sent: Thursday, May 22, 2008 4:34 PM
> To: Mike Olive
> Cc: Chris Hyzer; 'Grouper Dev'; 
> 
> Subject: Re: [signet-dev] RE: [grouper-dev] notifications vs hooks
>
> Based on this response I'm not sure you understand my original
> argument.
>
> /mrg
>
> On May 22, 2008, at 16:25, Mike Olive wrote:
>
> > In addition to being database agnostic the solution should be
> > such that plug-ins (integrations) are unaffected by new releases
> > of Grouper/Signet that may include changes in the data model.
> > Also, it would be better to provide a solution that does not mandate
> > that a plug-in developer require innate knowledge of Grouper/Signet
> > storage.
> >
> > --
> > Michael Olive
> >
> >
> > < -----Original Message-----
> > < From: Chris Hyzer 
> > [mailto:]
> > < Sent: Thursday, May 22, 2008 7:33 AM
> > < To: Michael R. Gettes; Grouper Dev; 
> > 
> > < Subject: [signet-dev] RE: [grouper-dev] notifications vs hooks
> > <
> > < Thanks for the reality check :).  I hope we are almost there.
> > <  Ideally by Friday or Wednesday we can nail down the design
> > < and then we can get it done quickly (it might seem like the
> > < contrary, but its not that complicated)... also, this
> > < discussion has definitely changed the design for the better
> > < (so far), so I think it has been productive...
> > <
> > < Btw, even if everyone were oracle, Im afraid DB triggers are
> > < not the way to go, since the java API of grouper or signet
> > < would not be there (there are Java stored procs in oracle,
> > < but if you have used them, you know that they are painful).
> > < Also we need to be consistent with how other customizations
> > < are coded, in java and configured in config files (e.g.
> > < authentication in the UI or WS)...
> > <
> > < Regards,
> > < Chris
> > <
> > < > -----Original Message-----
> > < > From: Michael R. Gettes 
> > [mailto:]
> > < > Sent: Thursday, May 22, 2008 8:32 AM
> > < > To: Grouper Dev; 
> > 
> > < > Subject: Re: [grouper-dev] notifications vs hooks
> > < >
> > < > It's always nice to try and be DB agnostic... but why not
> > < say "if you
> > < > want realtime
> > < > UI response" or DB triggers specifically, then use a DB
> technology
> > < > that supports
> > < > it like Oracle.  And then engineer things to support triggers.
> > What
> > < > really worries
> > < > me is our gazing at navels whilst we try and figure out the
> > perfect
> > < > solution
> > < > instead of getting something done to find out how useful
> > < stuff can be
> > < > in the
> > < > real world.  We need to find that balance between perfection and
> > < > getting stuff done.
> > < >
> > < > And thanks for the clarification chris.
> > < >
> > < > /mrg
> > < >
> > < > On May 22, 2008, at 1:01, Chris Hyzer wrote:
> > < >
> > < > > I believe what you are describing is similar to what I was
> > < > > describing (below) for notifications based on Bert's email.
> For
> > < > > hooks I believe we need other features (like veto, or
> > < transactional
> > < > > augmenters), so Im not sure it can happen quite like
> > < this...  Im not
> > < > > sure I understand your "temp db" suggestion, but when
> > < thinking about
> > < > > a user using the UI, they need instant hook
> > < action/results for when
> > < > > their screen displays a response, and it shouldn't impact
> > < > > performance much (e.g. having to wait for another process
> > < to decide
> > < > > something).  Also, I agree that DB triggers could be a
> > < way to solve
> > < > > some of these issues, but since we are db agnostic, and we like
> > < > > Java, it doesn't seem viable.
> > < > >
> > < > > Thanks,
> > < > > Chris
> > < > >
> > < > >
> > < > > From: Chris Hyzer
> > < > > Sent: Wednesday, May 14, 2008 4:46 PM
> > < > > To: 'Bert Bee-Lindgren'
> > < > > Cc: Grouper Dev; 
> > 
> > < > > Subject: RE: [grouper-dev] notifications vs hooks
> > < > >
> > < > > Bert, thanks for your email, that also seems like a more
> > < > > transactional solution, though Im still not exactly sure
> > < how we know
> > < > > when something is committed in grouper in java (now that we
> have
> > < > > these fancy new long running transactions. dote!  ? ).  Also,
> if
> > < > > the process terminates while it is in the middle of
> > < notifying, does
> > < > > it restart when it starts back up.  Is there one central daemon
> > < > > process that does that, or will it happen in UI / WS / GSH /
> > etc.
> > < > > if there is a notifier in UI / WS / GSH then how will it
> > < know which
> > < > > records in the table are for them to process and not someone
> > else?
> > < > > But that aside, I think we should separate hooks and
> > < notifications.
> > < > > I don't think all post-hooks are notification based.  Two
> > < examples:
> > < > >
> > < > > 1.      Auditing.  I want to insert an audit record about
> > < something
> > < > > done, and it should be done after the method call which
> > < is after all
> > < > > pre-hooks complete.  This should still be in the same DB
> > < transaction
> > < > > (hopefully)
> > < > > 2.      Augmenting with other objects.  I want to wait for a
> new
> > < > > group to be inserted (and the hibernate id created, which
> > < is done on
> > < > > hibernate flush), then attach a type to it, and maybe some
> > < > > memberships or something, still in the same DB transaction
> > < > >
> > < > > So I think we have pre-hooks, post-hooks (which could be used
> > for
> > < > > non-reliable notifications), and notifications.
> > < > >
> > < > > If your description below (items c-f) are processed on a
> > separate
> > < > > daemon, then I would be up for that (1 to 1 mapping between
> > daemon
> > < > > and database).  The insert into the table could happen
> > < with a post-
> > < > > hook, and be in the same DB transaction.  The separate
> > < daemon would
> > < > > poll the changed table for items and the destination
> > < (which would be
> > < > > a configured listener), hand-off and delete (repeat if errors).
> > < > > Sounds great!  ?
> > < > >
> > < > > Chris
> > < > >
> > < > >> -----Original Message-----
> > < > >> From: Michael R. Gettes 
> > [mailto:]
> > < > >> Sent: Wednesday, May 21, 2008 4:38 PM
> > < > >> To: Chris Hyzer
> > < > >> Cc: Bert Bee-Lindgren; Mike Olive; 'Grouper Dev'; signet-
> > < > >> 
> > 
> > < > >> Subject: Re: [grouper-dev] notifications vs hooks
> > < > >>
> > < > >> So, I am sure you guys have all thought about this a bit and
> > I am
> > < > >> curious
> > < > >> about why not the following approach...
> > < > >>
> > < > >> instead of creating a software structure (a layering of
> > < plugins and
> > < > >> such
> > < > >> to handle notifications and modification of data and
> > < yada yada) why
> > < > >> not
> > < > >> come up with some protocol at the data layer?  at some
> > scratch on
> > < > the
> > < > >> wall in the db when an item is added to the db and let some
> > other
> > < > >> process notice the change.  if you need "pre-commit"
> > < functionality
> > < > >> then add it to some temp db and go from there.  Then you
> > < don't have
> > < > >> to worry about putting in this layer in all software
> > components -
> > < > >> it's
> > < > >> just a matter of adding some state to an entry and the state
> > can
> > < > >> change
> > < > >> to indicate a lifecycle to the entry.
> > < > >>
> > < > >> /mrg
> > < > >>
> > < > >> On May 21, 2008, at 12:42, Chris Hyzer wrote:
> > < > >>
> > < > >>> Well, if we want reliable notifications, then we need
> > < transactional
> > < > >>> hooks.
> > < > >>> If we want augmenters, that is a pre-hook or post-hook.
> > <  If we need
> > < > >>> to know the hibernate id of whatever was inserted, that
> > < is a post
> > < > >>> hook (post the hibernate flush) (e.g. auto add a member
> > < to a group
> > < > >>> just created).  In both cases we definitely want the
> > < ability to be
> > < > >>> in the same transaction.  I think all that is pretty
> > < simple since
> > < > >>> transactions in threadlocal exist in grouper so the
> > < hook can use it
> > < > >>> or not (for the same database).  Reliable notifications is
> > < > trickier,
> > < > >>> and something like the daemon design in my previous email
> > would
> > < > >>> solve it).  I agree with you that if the auditing is in
> > < an external
> > < > >>> database then we don't need to worry about transactions
> > < (as much).
> > < > >>>
> > < > >>> I think a complication is as we discussed in the previous
> > phone
> > < > call
> > < > >>> where we have multiple chained hooks, and which order
> > < they come in,
> > < > >>> etc.  If we can cut that down (even to one non-API hook
> > < for pre and
> > < > >>> one for post) it will make it easier to work with.
> > < > >>>
> > < > >>> About the hooks generating more hook events, I think we
> should
> > < > >>> ignore this for now, I think it will work itself out.
> > < For instance
> > < > >>> for a group with a type of "requireFaculty", maybe the
> > < > >>> creaetGroup
> > < > >>> hook will build out a composite group that and's with a
> > faculty
> > < > >>> group (if the composite doesn't already exist).  The
> composite
> > < > >>> group
> > < > >>> will also trigger the createGroup hook, but the hook
> > < will see that
> > < > >>> the composite group doesn't have the type "requireFaculty" so
> > < > >>> it
> > < > >>> will ignore it.  Once we come up with an endless loop
> > < > >>> requirement, I
> > < > >>> bet we will be able to do something clever.
> > < Furthermore, I could
> > < > >>> see the hook code solving it themselves with a
> > < threadlocal (set a
> > < > >>> flag that says we are in the specific hook, clear it in
> > < a finally
> > < > >>> block, and check for the flag at the beginning of the
> > hook).  I
> > < > >>> don't think we should allow a hook to turn off all events,
> > there
> > < > >>> might have been some good ones in there.  J
> > < > >>>
> > < > >>> Kind regards,
> > < > >>> Chris
> > < > >>>
> > < > >>>
> > < > >>>
> > < > >>> From: Bert Bee-Lindgren
> > < 
> > [mailto:]
> > < > >>> Sent: Wednesday, May 21, 2008 11:53 AM
> > < > >>> To: Mike Olive
> > < > >>> Cc: Chris Hyzer; 'Grouper Dev'; 
> > 
> > < > >>> Subject: Re: [grouper-dev] notifications vs hooks
> > < > >>>
> > < > >>> Thanks Mike & Chris,
> > < > >>>
> > < > >>> I just get the feeling that pre-hooks and same-transaction
> > < > semantics
> > < > >>> (especially across data sources) are solving problems I
> > < don't see:
> > < > I
> > < > >>> see huge near-term values of pre-hooks as filters. I
> > < don't see how
> > < > >>> these need coordinated commit/rollback with the GrouperDB
> > < > >> transaction.
> > < > >>>
> > < > >>> From reviewing Grouper/Signet roadmap issues, I see the
> > < following
> > < > >>> most affected by Hooks & Plugins.
> > < > >>>  -Notification of changes - email/grouper-to-signet/triggered
> > < > >>> provisioning
> > < > >>>  -History & audit
> > < > >>>  -Rule-based action (not sure if this uses plugins, but
> > < some rules
> > < > >>> could be considered augmentation)
> > < > >>>
> > < > >>> Maybe my ACID religion will take a hit from this, but
> > < I'm not sure
> > < > I
> > < > >>> see the harm in Auditing and (membership and attribute)
> > < > Augmentation
> > < > >>> happening based on reliable notifications after the group's
> > < > >>> transaction commits. Further, when the group is changed
> > < via Post-
> > < > >>> Hook augmentation, it's possible that other pre-hooks should
> > be
> > < > >>> triggered (Orig action, pre-hook, hib flush, post-hook that
> > < > >>> augments, pre-hook because request changed, hib flush,
> > < post-hook,
> > < > >>> commit.... ugh... double-ugh if it loops further or if
> > < a pre-hook
> > < > >>> later disagrees)
> > < > >>>
> > < > >>> Generally, I see the following levels of increasing
> > < functionality
> > < > >>> and complexity. And I see current plans for #4 while #1
> > < or 2 seem
> > < > >>> (to me) to be so much easier and offer so much value.I
> > < see 80% of
> > < > >>> vision I've heard being done with #1 and 90+% with #2.
> > < > >>>          1) Pre-hooks as filters, reliable post-action
> > < > >>> notifications [I'd call this the minimum]
> > < > >>>          2) #1 with pre-hooks also as augmenters
> > < > >>>          3) #2 with all pre-hooks seeing the final
> (augmented)
> > < > group
> > < > >>>          4) #3 with post-hooks and same-transaction semantics
> > < > >>>
> > < > >>> Am I missing Use Cases, or not considering ACID-lite
> > < problems, or
> > < > am
> > < > >>> I seeing difficulty where there is none?
> > < > >>> Put differently, if this is all truly necessary, great.
> > < Or if this
> > < > >>> is easier than it seems, even better.
> > < > >>>
> > < > >>> Thanks, yet again, for you patience,
> > < > >>>  Bert
> > < > >>>
> > < > >>> On May 21, 2008, at 11:03 AM, Mike Olive wrote:
> > < > >>>
> > < > >>>
> > < > >>> Bert,
> > < > >>>
> > < > >>> Pre-hooks is certainly the more complicated aspect of
> > < this design
> > < > >>> with the implementation
> > < > >>> most likely requiring the use of a transaction manager
> > < so that the
> > < > >>> pre-hook plug-ins may
> > < > >>> participate in the final commit or rollback of the actual
> > < > >> transaction.
> > < > >>>
> > < > >>> On post-hooks persistence, the current design is agnostic of
> > any
> > < > >>> messaging solution or
> > < > >>> transportation mechanism of the change notification.
> > < The design is
> > < > >>> extensible such that
> > < > >>> the developer could incorporate a guaranteed messaging
> > solution
> > < > >>> (client) such as JMS.
> > < > >>> --
> > < > >>> Michael Olive
> > < > >>>
> > < > >>>
> > < > >>>
> > < > >>> From: Bert Bee-Lindgren
> > < 
> > [mailto:]
> > < > >>> Sent: Wednesday, May 14, 2008 12:05 PM
> > < > >>> To: Chris Hyzer
> > < > >>> Cc: Grouper Dev; 
> > 
> > < > >>> Subject: Re: [grouper-dev] notifications vs hooks
> > < > >>> Combining our approach to similar problems with UPenn's
> > < plans... I
> > < > >>> think we should consider very different mechanisms for pre-
> > hooks
> > < > and
> > < > >>> post-hooks.
> > < > >>>
> > < > >>> Pre-hooks:
> > < > >>> Normal, synchronous method calls
> > < > >>> Pre-hook plugin developers should expect the event to
> > < possibly not
> > < > >>> occur even if they approve it. They should not notify, log,
> > etc
> > < > >>> anything that might indicate to a user/auditor/sysadmin that
> > an
> > < > >>> event happened... because they won't know about downstream
> > < > >> rejections.
> > < > >>>
> > < > >>>
> > < > >>> On the post-hooks, I think we should consider a persistent
> > post-
> > < > >>> hook:
> > < > >>>  a) A plugin would have two inbound methods
> > < > >>> a1) "This event happened, do you care?"  [Boolean return, no
> > < > >>> external processing allowed, must be "fast"]
> > < > >>> a2) "Process this event, let us know when you've succeeded in
> > < > >>> handling it" [Boolean return, TRUE means this plugin
> > succeeded]
> > < > >>>  b) Create plugin-specific/event-specific database rows
> > < in an event
> > < > >>> table based on the TRUE returns of a1's
> > < > >>>  c) Immediately after all the plugins have had a chance
> > < to answer
> > < > >>> a1, hand the event to all the interested plugins a2's.
> > < > >>>  d) Delete the plugin-specific/event-specific row when that
> > < > >>> plugin's a2 returns true
> > < > >>>  e) Retry the failed a2's for a plugin before any new a2
> > < > >>>  f) Possibly retry the failed a2's every couple minutes or
> > with
> > < > >>> some backoff approach (or, disappointingly, wait for the next
> > < > event)
> > < > >>>
> > < > >>> Yes, this is basically a message queue, but simple to
> > < implement (we
> > < > >>> use a python-based version of this for several event
> > < queues in our
> > < > >>> system). I've looked for a JMS library as simple to use as
> > this
> > < > two-
> > < > >>> method approach and haven't found one.
> > < > >>>
> > < > >>>
> > < > >>> On May 14, 2008, at 2:27 PM, Chris Hyzer wrote:
> > < > >>>
> > < > >>> Gary put comments on my hooks page about transactions and
> > < > >>> notifications.
> > < > >>>
> > < >
> https://wiki.internet2.edu/confluence/display/GrouperWG/Hooks+simple
> > < > >>> It makes this whole thing very complicated. if the
> > < actions happens
> > < > >>> in a long running transaction, and you want to be
> > < notified at the
> > < > >>> end, there are a few issues:
> > < > >>> 1.       It is a different architecture than we had
> > < been discussing
> > < > >>> since we need to know about the action at the time of the
> > < > >>> (successful) commit.  Perhaps using Hibernate's events could
> > do
> > < > >>> the
> > < > >>> trick, but you don't have any object model anymore, you
> > < just have
> > < > >>> a
> > < > >>> list of column data for one table
> > < > >>> 2.       Like Gary points out, if the thing you are updating
> > < > >>> external to grouper fails, how do you log that and
> > < catch up later
> > < > >>> (e.g. if you are calling a web service, and there is a
> network
> > < > >>> issue)
> > < > >>> 3.       There are lots of different producers of
> > < events (UI, WS,
> > < > >>> extensions e.g. gsh, and direct db edits [granted they
> > < > >>> shouldn't]).
> > < > >>> Must make sure the notification hooks are registered
> > everywhere,
> > < > and
> > < > >>> test them to make sure they are firing everywhere
> > < (seems tedious /
> > < > >>> risky)
> > < > >>> Lets take the use case of writing your own ldappc via
> > < notifications
> > < > >>> (something we will start out with at penn).  We want to
> > < know about
> > < > >>> new members, memberships, and groups.  We will just
> > < make 3 tables
> > < > >>> with the id's and timestamps of when these change:
> > < > >>> select * from ldap_change_memberships lcm where rownum
> > < < 4 order by
> > < > >>> lcm.LAST_UPDATED desc
> > < > >>> MEMBERSHIP_UUID      LAST_UPDATED
> > < > >>> cd5f23d2-a8c8-44c0-a8b1-a3c3210da3c5                5/8/2008
> > < > >>> 1:21:37.569272 PM
> > < > >>> 74710fee-40b2-48e4-a8dd-b750876bc4ea             5/8/2008
> > < > >>> 1:21:37.462896 PM
> > < > >>> f2cf6b80-377c-41c3-981d-0eb9274dc74a                5/8/2008
> > < > >>> 1:21:36.076199 PM
> > < > >>> On the grouper tables I have some simple triggers that check
> > for
> > < > >>> diffs and insert to the change tables (and delete old
> > < records since
> > < > >>> all the daemon cares about is the most recently changed
> > record).
> > < > >>> Then we also have friendly views for the daemon to use
> > < to query the
> > < > >>> data:
> > < > >>> select gmv.GROUP_NAME, gmv.SUBJECT_ID, gmv.SUBJECT_SOURCE,
> > < > >>> gmv.MSHIP_TYPE from grouper_memberships_v gmv where rownum <4
> > < > >>> GROUP_NAME
> > < > >>> SUBJECT_ID                        SUBJECT_SOURCE
> > < > >>> MSHIP_TYPE
> > < > >>> Centers:ISC:PennAccess -ext     GrouperSystem
> > < > >>> g:isa                                       immediate
> > < > >>> Centers:ISC:PennAccess -ext     GrouperSystem
> > < > >>> g:isa                                       immediate
> > < > >>> Centers:ISC:PennAccess -ext     GrouperSystem
> > < > >>> g:isa                                       immediate
> > < > >>> Then a daemon will run every 5 minutes to push the new data
> to
> > < > ldap,
> > < > >>> and delete the record from the change table when done.
> > < > >>> So this is all transactional, nothing can slip by
> > < (since trigger),
> > < > >>> and nothing can happen wrong if ldap update fails.
> > < > >>> Triggers are DB specific, and it requires different
> > < tables for each
> > < > >>> notification type.
> > < > >>> For notifications, grouper could provide:
> > < > >>> 1.       If you have certain table structures for
> last_updated
> > < > dates
> > < > >>> (perhaps with name prefixes to support multiple)
> > < > >>> 2.       We could populate with Java hooks (perhaps not
> > reliable
> > < > per
> > < > >>> discussion above), or you could do triggers (we could provide
> > < > >>> examples for certain DB's) which would be more reliable and
> > < > >>> probably
> > < > >>> more performant
> > < > >>> 3.       We could provide a daemon (e.g. runs every 5
> > < minutes) with
> > < > >>> callbacks that would process the change tables (based on name
> > < > >>> prefix), and gives you a callback to take the data and put it
> > < > >>> somewhere
> > < > >>> 4.       We could provide a scheduled way to get all
> > < data not just
> > < > >>> the diffs (e.g. daily or weekly or monthly to ensure
> > < things are in
> > < > >>> sync)
> > < > >>> Just brainstorming here.
> > < > >>> Thanks,
> > < > >>> Chris
> > < > >>>
> > < > >>>
> > < > >
> > <
> > <
> >