Grouper Developers Forum

["Michael R. Gettes" <>]

Based on this response I'm not sure you understand my original argument.

/mrg

On May 22, 2008, at 16:25, Mike Olive wrote:

> In addition to being database agnostic the solution should be
> such that plug-ins (integrations) are unaffected by new releases
> of Grouper/Signet that may include changes in the data model.
> Also, it would be better to provide a solution that does not mandate
> that a plug-in developer require innate knowledge of Grouper/Signet
> storage.
>
> --
> Michael Olive
>
>
> < -----Original Message-----
> < From: Chris Hyzer 
> [mailto:]
> < Sent: Thursday, May 22, 2008 7:33 AM
> < To: Michael R. Gettes; Grouper Dev; 
>
> < Subject: [signet-dev] RE: [grouper-dev] notifications vs hooks
> <
> < Thanks for the reality check :).  I hope we are almost there.
> <  Ideally by Friday or Wednesday we can nail down the design
> < and then we can get it done quickly (it might seem like the
> < contrary, but its not that complicated)... also, this
> < discussion has definitely changed the design for the better
> < (so far), so I think it has been productive...
> <
> < Btw, even if everyone were oracle, Im afraid DB triggers are
> < not the way to go, since the java API of grouper or signet
> < would not be there (there are Java stored procs in oracle,
> < but if you have used them, you know that they are painful).
> < Also we need to be consistent with how other customizations
> < are coded, in java and configured in config files (e.g.
> < authentication in the UI or WS)...
> <
> < Regards,
> < Chris
> <
> < > -----Original Message-----
> < > From: Michael R. Gettes 
> [mailto:]
> < > Sent: Thursday, May 22, 2008 8:32 AM
> < > To: Grouper Dev; 
>
> < > Subject: Re: [grouper-dev] notifications vs hooks
> < >
> < > It's always nice to try and be DB agnostic... but why not
> < say "if you
> < > want realtime
> < > UI response" or DB triggers specifically, then use a DB technology
> < > that supports
> < > it like Oracle.  And then engineer things to support triggers.   
> What
> < > really worries
> < > me is our gazing at navels whilst we try and figure out the  
> perfect
> < > solution
> < > instead of getting something done to find out how useful
> < stuff can be
> < > in the
> < > real world.  We need to find that balance between perfection and
> < > getting stuff done.
> < >
> < > And thanks for the clarification chris.
> < >
> < > /mrg
> < >
> < > On May 22, 2008, at 1:01, Chris Hyzer wrote:
> < >
> < > > I believe what you are describing is similar to what I was
> < > > describing (below) for notifications based on Bert's email.  For
> < > > hooks I believe we need other features (like veto, or
> < transactional
> < > > augmenters), so Im not sure it can happen quite like
> < this...  Im not
> < > > sure I understand your "temp db" suggestion, but when
> < thinking about
> < > > a user using the UI, they need instant hook
> < action/results for when
> < > > their screen displays a response, and it shouldn't impact
> < > > performance much (e.g. having to wait for another process
> < to decide
> < > > something).  Also, I agree that DB triggers could be a
> < way to solve
> < > > some of these issues, but since we are db agnostic, and we like
> < > > Java, it doesn't seem viable.
> < > >
> < > > Thanks,
> < > > Chris
> < > >
> < > >
> < > > From: Chris Hyzer
> < > > Sent: Wednesday, May 14, 2008 4:46 PM
> < > > To: 'Bert Bee-Lindgren'
> < > > Cc: Grouper Dev; 
>
> < > > Subject: RE: [grouper-dev] notifications vs hooks
> < > >
> < > > Bert, thanks for your email, that also seems like a more
> < > > transactional solution, though Im still not exactly sure
> < how we know
> < > > when something is committed in grouper in java (now that we have
> < > > these fancy new long running transactions. dote!  ? ).  Also, if
> < > > the process terminates while it is in the middle of
> < notifying, does
> < > > it restart when it starts back up.  Is there one central daemon
> < > > process that does that, or will it happen in UI / WS / GSH /  
> etc.
> < > > if there is a notifier in UI / WS / GSH then how will it
> < know which
> < > > records in the table are for them to process and not someone  
> else?
> < > > But that aside, I think we should separate hooks and
> < notifications.
> < > > I don't think all post-hooks are notification based.  Two
> < examples:
> < > >
> < > > 1.      Auditing.  I want to insert an audit record about
> < something
> < > > done, and it should be done after the method call which
> < is after all
> < > > pre-hooks complete.  This should still be in the same DB
> < transaction
> < > > (hopefully)
> < > > 2.      Augmenting with other objects.  I want to wait for a new
> < > > group to be inserted (and the hibernate id created, which
> < is done on
> < > > hibernate flush), then attach a type to it, and maybe some
> < > > memberships or something, still in the same DB transaction
> < > >
> < > > So I think we have pre-hooks, post-hooks (which could be used  
> for
> < > > non-reliable notifications), and notifications.
> < > >
> < > > If your description below (items c-f) are processed on a  
> separate
> < > > daemon, then I would be up for that (1 to 1 mapping between  
> daemon
> < > > and database).  The insert into the table could happen
> < with a post-
> < > > hook, and be in the same DB transaction.  The separate
> < daemon would
> < > > poll the changed table for items and the destination
> < (which would be
> < > > a configured listener), hand-off and delete (repeat if errors).
> < > > Sounds great!  ?
> < > >
> < > > Chris
> < > >
> < > >> -----Original Message-----
> < > >> From: Michael R. Gettes 
> [mailto:]
> < > >> Sent: Wednesday, May 21, 2008 4:38 PM
> < > >> To: Chris Hyzer
> < > >> Cc: Bert Bee-Lindgren; Mike Olive; 'Grouper Dev'; signet-
> < > >> 
>
> < > >> Subject: Re: [grouper-dev] notifications vs hooks
> < > >>
> < > >> So, I am sure you guys have all thought about this a bit and  
> I am
> < > >> curious
> < > >> about why not the following approach...
> < > >>
> < > >> instead of creating a software structure (a layering of
> < plugins and
> < > >> such
> < > >> to handle notifications and modification of data and
> < yada yada) why
> < > >> not
> < > >> come up with some protocol at the data layer?  at some  
> scratch on
> < > the
> < > >> wall in the db when an item is added to the db and let some  
> other
> < > >> process notice the change.  if you need "pre-commit"
> < functionality
> < > >> then add it to some temp db and go from there.  Then you
> < don't have
> < > >> to worry about putting in this layer in all software  
> components -
> < > >> it's
> < > >> just a matter of adding some state to an entry and the state  
> can
> < > >> change
> < > >> to indicate a lifecycle to the entry.
> < > >>
> < > >> /mrg
> < > >>
> < > >> On May 21, 2008, at 12:42, Chris Hyzer wrote:
> < > >>
> < > >>> Well, if we want reliable notifications, then we need
> < transactional
> < > >>> hooks.
> < > >>> If we want augmenters, that is a pre-hook or post-hook.
> <  If we need
> < > >>> to know the hibernate id of whatever was inserted, that
> < is a post
> < > >>> hook (post the hibernate flush) (e.g. auto add a member
> < to a group
> < > >>> just created).  In both cases we definitely want the
> < ability to be
> < > >>> in the same transaction.  I think all that is pretty
> < simple since
> < > >>> transactions in threadlocal exist in grouper so the
> < hook can use it
> < > >>> or not (for the same database).  Reliable notifications is
> < > trickier,
> < > >>> and something like the daemon design in my previous email  
> would
> < > >>> solve it).  I agree with you that if the auditing is in
> < an external
> < > >>> database then we don't need to worry about transactions
> < (as much).
> < > >>>
> < > >>> I think a complication is as we discussed in the previous  
> phone
> < > call
> < > >>> where we have multiple chained hooks, and which order
> < they come in,
> < > >>> etc.  If we can cut that down (even to one non-API hook
> < for pre and
> < > >>> one for post) it will make it easier to work with.
> < > >>>
> < > >>> About the hooks generating more hook events, I think we should
> < > >>> ignore this for now, I think it will work itself out.
> < For instance
> < > >>> for a group with a type of "requireFaculty", maybe the
> < > >>> creaetGroup
> < > >>> hook will build out a composite group that and's with a  
> faculty
> < > >>> group (if the composite doesn't already exist).  The composite
> < > >>> group
> < > >>> will also trigger the createGroup hook, but the hook
> < will see that
> < > >>> the composite group doesn't have the type "requireFaculty" so
> < > >>> it
> < > >>> will ignore it.  Once we come up with an endless loop
> < > >>> requirement, I
> < > >>> bet we will be able to do something clever.
> < Furthermore, I could
> < > >>> see the hook code solving it themselves with a
> < threadlocal (set a
> < > >>> flag that says we are in the specific hook, clear it in
> < a finally
> < > >>> block, and check for the flag at the beginning of the  
> hook).  I
> < > >>> don't think we should allow a hook to turn off all events,  
> there
> < > >>> might have been some good ones in there.  J
> < > >>>
> < > >>> Kind regards,
> < > >>> Chris
> < > >>>
> < > >>>
> < > >>>
> < > >>> From: Bert Bee-Lindgren
> < 
> [mailto:]
> < > >>> Sent: Wednesday, May 21, 2008 11:53 AM
> < > >>> To: Mike Olive
> < > >>> Cc: Chris Hyzer; 'Grouper Dev'; 
>
> < > >>> Subject: Re: [grouper-dev] notifications vs hooks
> < > >>>
> < > >>> Thanks Mike & Chris,
> < > >>>
> < > >>> I just get the feeling that pre-hooks and same-transaction
> < > semantics
> < > >>> (especially across data sources) are solving problems I
> < don't see:
> < > I
> < > >>> see huge near-term values of pre-hooks as filters. I
> < don't see how
> < > >>> these need coordinated commit/rollback with the GrouperDB
> < > >> transaction.
> < > >>>
> < > >>> From reviewing Grouper/Signet roadmap issues, I see the
> < following
> < > >>> most affected by Hooks & Plugins.
> < > >>>  -Notification of changes - email/grouper-to-signet/triggered
> < > >>> provisioning
> < > >>>  -History & audit
> < > >>>  -Rule-based action (not sure if this uses plugins, but
> < some rules
> < > >>> could be considered augmentation)
> < > >>>
> < > >>> Maybe my ACID religion will take a hit from this, but
> < I'm not sure
> < > I
> < > >>> see the harm in Auditing and (membership and attribute)
> < > Augmentation
> < > >>> happening based on reliable notifications after the group's
> < > >>> transaction commits. Further, when the group is changed
> < via Post-
> < > >>> Hook augmentation, it's possible that other pre-hooks should  
> be
> < > >>> triggered (Orig action, pre-hook, hib flush, post-hook that
> < > >>> augments, pre-hook because request changed, hib flush,
> < post-hook,
> < > >>> commit.... ugh... double-ugh if it loops further or if
> < a pre-hook
> < > >>> later disagrees)
> < > >>>
> < > >>> Generally, I see the following levels of increasing
> < functionality
> < > >>> and complexity. And I see current plans for #4 while #1
> < or 2 seem
> < > >>> (to me) to be so much easier and offer so much value.I
> < see 80% of
> < > >>> vision I've heard being done with #1 and 90+% with #2.
> < > >>>          1) Pre-hooks as filters, reliable post-action
> < > >>> notifications [I'd call this the minimum]
> < > >>>          2) #1 with pre-hooks also as augmenters
> < > >>>          3) #2 with all pre-hooks seeing the final (augmented)
> < > group
> < > >>>          4) #3 with post-hooks and same-transaction semantics
> < > >>>
> < > >>> Am I missing Use Cases, or not considering ACID-lite
> < problems, or
> < > am
> < > >>> I seeing difficulty where there is none?
> < > >>> Put differently, if this is all truly necessary, great.
> < Or if this
> < > >>> is easier than it seems, even better.
> < > >>>
> < > >>> Thanks, yet again, for you patience,
> < > >>>  Bert
> < > >>>
> < > >>> On May 21, 2008, at 11:03 AM, Mike Olive wrote:
> < > >>>
> < > >>>
> < > >>> Bert,
> < > >>>
> < > >>> Pre-hooks is certainly the more complicated aspect of
> < this design
> < > >>> with the implementation
> < > >>> most likely requiring the use of a transaction manager
> < so that the
> < > >>> pre-hook plug-ins may
> < > >>> participate in the final commit or rollback of the actual
> < > >> transaction.
> < > >>>
> < > >>> On post-hooks persistence, the current design is agnostic of  
> any
> < > >>> messaging solution or
> < > >>> transportation mechanism of the change notification.
> < The design is
> < > >>> extensible such that
> < > >>> the developer could incorporate a guaranteed messaging  
> solution
> < > >>> (client) such as JMS.
> < > >>> --
> < > >>> Michael Olive
> < > >>>
> < > >>>
> < > >>>
> < > >>> From: Bert Bee-Lindgren
> < 
> [mailto:]
> < > >>> Sent: Wednesday, May 14, 2008 12:05 PM
> < > >>> To: Chris Hyzer
> < > >>> Cc: Grouper Dev; 
>
> < > >>> Subject: Re: [grouper-dev] notifications vs hooks
> < > >>> Combining our approach to similar problems with UPenn's
> < plans... I
> < > >>> think we should consider very different mechanisms for pre- 
> hooks
> < > and
> < > >>> post-hooks.
> < > >>>
> < > >>> Pre-hooks:
> < > >>> Normal, synchronous method calls
> < > >>> Pre-hook plugin developers should expect the event to
> < possibly not
> < > >>> occur even if they approve it. They should not notify, log,  
> etc
> < > >>> anything that might indicate to a user/auditor/sysadmin that  
> an
> < > >>> event happened... because they won't know about downstream
> < > >> rejections.
> < > >>>
> < > >>>
> < > >>> On the post-hooks, I think we should consider a persistent  
> post-
> < > >>> hook:
> < > >>>  a) A plugin would have two inbound methods
> < > >>> a1) "This event happened, do you care?"  [Boolean return, no
> < > >>> external processing allowed, must be "fast"]
> < > >>> a2) "Process this event, let us know when you've succeeded in
> < > >>> handling it" [Boolean return, TRUE means this plugin  
> succeeded]
> < > >>>  b) Create plugin-specific/event-specific database rows
> < in an event
> < > >>> table based on the TRUE returns of a1's
> < > >>>  c) Immediately after all the plugins have had a chance
> < to answer
> < > >>> a1, hand the event to all the interested plugins a2's.
> < > >>>  d) Delete the plugin-specific/event-specific row when that
> < > >>> plugin's a2 returns true
> < > >>>  e) Retry the failed a2's for a plugin before any new a2
> < > >>>  f) Possibly retry the failed a2's every couple minutes or  
> with
> < > >>> some backoff approach (or, disappointingly, wait for the next
> < > event)
> < > >>>
> < > >>> Yes, this is basically a message queue, but simple to
> < implement (we
> < > >>> use a python-based version of this for several event
> < queues in our
> < > >>> system). I've looked for a JMS library as simple to use as  
> this
> < > two-
> < > >>> method approach and haven't found one.
> < > >>>
> < > >>>
> < > >>> On May 14, 2008, at 2:27 PM, Chris Hyzer wrote:
> < > >>>
> < > >>> Gary put comments on my hooks page about transactions and
> < > >>> notifications.
> < > >>>
> < > https://wiki.internet2.edu/confluence/display/GrouperWG/Hooks+simple
> < > >>> It makes this whole thing very complicated. if the
> < actions happens
> < > >>> in a long running transaction, and you want to be
> < notified at the
> < > >>> end, there are a few issues:
> < > >>> 1.       It is a different architecture than we had
> < been discussing
> < > >>> since we need to know about the action at the time of the
> < > >>> (successful) commit.  Perhaps using Hibernate's events could  
> do
> < > >>> the
> < > >>> trick, but you don't have any object model anymore, you
> < just have
> < > >>> a
> < > >>> list of column data for one table
> < > >>> 2.       Like Gary points out, if the thing you are updating
> < > >>> external to grouper fails, how do you log that and
> < catch up later
> < > >>> (e.g. if you are calling a web service, and there is a network
> < > >>> issue)
> < > >>> 3.       There are lots of different producers of
> < events (UI, WS,
> < > >>> extensions e.g. gsh, and direct db edits [granted they
> < > >>> shouldn't]).
> < > >>> Must make sure the notification hooks are registered  
> everywhere,
> < > and
> < > >>> test them to make sure they are firing everywhere
> < (seems tedious /
> < > >>> risky)
> < > >>> Lets take the use case of writing your own ldappc via
> < notifications
> < > >>> (something we will start out with at penn).  We want to
> < know about
> < > >>> new members, memberships, and groups.  We will just
> < make 3 tables
> < > >>> with the id's and timestamps of when these change:
> < > >>> select * from ldap_change_memberships lcm where rownum
> < < 4 order by
> < > >>> lcm.LAST_UPDATED desc
> < > >>> MEMBERSHIP_UUID      LAST_UPDATED
> < > >>> cd5f23d2-a8c8-44c0-a8b1-a3c3210da3c5                5/8/2008
> < > >>> 1:21:37.569272 PM
> < > >>> 74710fee-40b2-48e4-a8dd-b750876bc4ea             5/8/2008
> < > >>> 1:21:37.462896 PM
> < > >>> f2cf6b80-377c-41c3-981d-0eb9274dc74a                5/8/2008
> < > >>> 1:21:36.076199 PM
> < > >>> On the grouper tables I have some simple triggers that check  
> for
> < > >>> diffs and insert to the change tables (and delete old
> < records since
> < > >>> all the daemon cares about is the most recently changed  
> record).
> < > >>> Then we also have friendly views for the daemon to use
> < to query the
> < > >>> data:
> < > >>> select gmv.GROUP_NAME, gmv.SUBJECT_ID, gmv.SUBJECT_SOURCE,
> < > >>> gmv.MSHIP_TYPE from grouper_memberships_v gmv where rownum <4
> < > >>> GROUP_NAME
> < > >>> SUBJECT_ID                        SUBJECT_SOURCE
> < > >>> MSHIP_TYPE
> < > >>> Centers:ISC:PennAccess -ext     GrouperSystem
> < > >>> g:isa                                       immediate
> < > >>> Centers:ISC:PennAccess -ext     GrouperSystem
> < > >>> g:isa                                       immediate
> < > >>> Centers:ISC:PennAccess -ext     GrouperSystem
> < > >>> g:isa                                       immediate
> < > >>> Then a daemon will run every 5 minutes to push the new data to
> < > ldap,
> < > >>> and delete the record from the change table when done.
> < > >>> So this is all transactional, nothing can slip by
> < (since trigger),
> < > >>> and nothing can happen wrong if ldap update fails.
> < > >>> Triggers are DB specific, and it requires different
> < tables for each
> < > >>> notification type.
> < > >>> For notifications, grouper could provide:
> < > >>> 1.       If you have certain table structures for last_updated
> < > dates
> < > >>> (perhaps with name prefixes to support multiple)
> < > >>> 2.       We could populate with Java hooks (perhaps not  
> reliable
> < > per
> < > >>> discussion above), or you could do triggers (we could provide
> < > >>> examples for certain DB's) which would be more reliable and
> < > >>> probably
> < > >>> more performant
> < > >>> 3.       We could provide a daemon (e.g. runs every 5
> < minutes) with
> < > >>> callbacks that would process the change tables (based on name
> < > >>> prefix), and gives you a callback to take the data and put it
> < > >>> somewhere
> < > >>> 4.       We could provide a scheduled way to get all
> < data not just
> < > >>> the diffs (e.g. daily or weekly or monthly to ensure
> < things are in
> < > >>> sync)
> < > >>> Just brainstorming here.
> < > >>> Thanks,
> < > >>> Chris
> < > >>>
> < > >>>
> < > >
> <
> <