Grouper Developers Forum

["Mike Olive" <>]

I don't believe that we are actually gazing at navels, but are more
attempting to flesh out the requirements applied against use-cases
for this feature.

< -----Original Message-----
< From: Michael R. Gettes 
[mailto:]
 
< Sent: Thursday, May 22, 2008 1:34 PM
< To: Mike Olive
< Cc: 'Chris Hyzer'; 'Grouper Dev'; 

< Subject: Re: [signet-dev] RE: [grouper-dev] notifications vs hooks
< 
< Based on this response I'm not sure you understand my 
< original argument.
< 
< /mrg
< 
< On May 22, 2008, at 16:25, Mike Olive wrote:
< 
< > In addition to being database agnostic the solution should be
< > such that plug-ins (integrations) are unaffected by new releases
< > of Grouper/Signet that may include changes in the data model.
< > Also, it would be better to provide a solution that does not mandate
< > that a plug-in developer require innate knowledge of Grouper/Signet
< > storage.
< >
< > --
< > Michael Olive
< >
< >
< > < -----Original Message-----
< > < From: Chris Hyzer 
[mailto:]
< > < Sent: Thursday, May 22, 2008 7:33 AM
< > < To: Michael R. Gettes; Grouper Dev; 

< > < Subject: [signet-dev] RE: [grouper-dev] notifications vs hooks
< > <
< > < Thanks for the reality check :).  I hope we are almost there.
< > <  Ideally by Friday or Wednesday we can nail down the design
< > < and then we can get it done quickly (it might seem like the
< > < contrary, but its not that complicated)... also, this
< > < discussion has definitely changed the design for the better
< > < (so far), so I think it has been productive...
< > <
< > < Btw, even if everyone were oracle, Im afraid DB triggers are
< > < not the way to go, since the java API of grouper or signet
< > < would not be there (there are Java stored procs in oracle,
< > < but if you have used them, you know that they are painful).
< > < Also we need to be consistent with how other customizations
< > < are coded, in java and configured in config files (e.g.
< > < authentication in the UI or WS)...
< > <
< > < Regards,
< > < Chris
< > <
< > < > -----Original Message-----
< > < > From: Michael R. Gettes 
[mailto:]
< > < > Sent: Thursday, May 22, 2008 8:32 AM
< > < > To: Grouper Dev; 

< > < > Subject: Re: [grouper-dev] notifications vs hooks
< > < >
< > < > It's always nice to try and be DB agnostic... but why not
< > < say "if you
< > < > want realtime
< > < > UI response" or DB triggers specifically, then use a DB 
< technology
< > < > that supports
< > < > it like Oracle.  And then engineer things to support 
< triggers.   
< > What
< > < > really worries
< > < > me is our gazing at navels whilst we try and figure out the  
< > perfect
< > < > solution
< > < > instead of getting something done to find out how useful
< > < stuff can be
< > < > in the
< > < > real world.  We need to find that balance between perfection and
< > < > getting stuff done.
< > < >
< > < > And thanks for the clarification chris.
< > < >
< > < > /mrg
< > < >
< > < > On May 22, 2008, at 1:01, Chris Hyzer wrote:
< > < >
< > < > > I believe what you are describing is similar to what I was
< > < > > describing (below) for notifications based on Bert's 
< email.  For
< > < > > hooks I believe we need other features (like veto, or
< > < transactional
< > < > > augmenters), so Im not sure it can happen quite like
< > < this...  Im not
< > < > > sure I understand your "temp db" suggestion, but when
< > < thinking about
< > < > > a user using the UI, they need instant hook
< > < action/results for when
< > < > > their screen displays a response, and it shouldn't impact
< > < > > performance much (e.g. having to wait for another process
< > < to decide
< > < > > something).  Also, I agree that DB triggers could be a
< > < way to solve
< > < > > some of these issues, but since we are db agnostic, 
< and we like
< > < > > Java, it doesn't seem viable.
< > < > >
< > < > > Thanks,
< > < > > Chris
< > < > >
< > < > >
< > < > > From: Chris Hyzer
< > < > > Sent: Wednesday, May 14, 2008 4:46 PM
< > < > > To: 'Bert Bee-Lindgren'
< > < > > Cc: Grouper Dev; 

< > < > > Subject: RE: [grouper-dev] notifications vs hooks
< > < > >
< > < > > Bert, thanks for your email, that also seems like a more
< > < > > transactional solution, though Im still not exactly sure
< > < how we know
< > < > > when something is committed in grouper in java (now 
< that we have
< > < > > these fancy new long running transactions. dote!  ? 
< ).  Also, if
< > < > > the process terminates while it is in the middle of
< > < notifying, does
< > < > > it restart when it starts back up.  Is there one 
< central daemon
< > < > > process that does that, or will it happen in UI / WS / GSH /  
< > etc.
< > < > > if there is a notifier in UI / WS / GSH then how will it
< > < know which
< > < > > records in the table are for them to process and not someone  
< > else?
< > < > > But that aside, I think we should separate hooks and
< > < notifications.
< > < > > I don't think all post-hooks are notification based.  Two
< > < examples:
< > < > >
< > < > > 1.      Auditing.  I want to insert an audit record about
< > < something
< > < > > done, and it should be done after the method call which
< > < is after all
< > < > > pre-hooks complete.  This should still be in the same DB
< > < transaction
< > < > > (hopefully)
< > < > > 2.      Augmenting with other objects.  I want to 
< wait for a new
< > < > > group to be inserted (and the hibernate id created, which
< > < is done on
< > < > > hibernate flush), then attach a type to it, and maybe some
< > < > > memberships or something, still in the same DB transaction
< > < > >
< > < > > So I think we have pre-hooks, post-hooks (which could 
< be used  
< > for
< > < > > non-reliable notifications), and notifications.
< > < > >
< > < > > If your description below (items c-f) are processed on a  
< > separate
< > < > > daemon, then I would be up for that (1 to 1 mapping between  
< > daemon
< > < > > and database).  The insert into the table could happen
< > < with a post-
< > < > > hook, and be in the same DB transaction.  The separate
< > < daemon would
< > < > > poll the changed table for items and the destination
< > < (which would be
< > < > > a configured listener), hand-off and delete (repeat 
< if errors).
< > < > > Sounds great!  ?
< > < > >
< > < > > Chris
< > < > >
< > < > >> -----Original Message-----
< > < > >> From: Michael R. Gettes 
[mailto:]
< > < > >> Sent: Wednesday, May 21, 2008 4:38 PM
< > < > >> To: Chris Hyzer
< > < > >> Cc: Bert Bee-Lindgren; Mike Olive; 'Grouper Dev'; signet-
< > < > >> 

< > < > >> Subject: Re: [grouper-dev] notifications vs hooks
< > < > >>
< > < > >> So, I am sure you guys have all thought about this a 
< bit and  
< > I am
< > < > >> curious
< > < > >> about why not the following approach...
< > < > >>
< > < > >> instead of creating a software structure (a layering of
< > < plugins and
< > < > >> such
< > < > >> to handle notifications and modification of data and
< > < yada yada) why
< > < > >> not
< > < > >> come up with some protocol at the data layer?  at some  
< > scratch on
< > < > the
< > < > >> wall in the db when an item is added to the db and let some  
< > other
< > < > >> process notice the change.  if you need "pre-commit"
< > < functionality
< > < > >> then add it to some temp db and go from there.  Then you
< > < don't have
< > < > >> to worry about putting in this layer in all software  
< > components -
< > < > >> it's
< > < > >> just a matter of adding some state to an entry and 
< the state  
< > can
< > < > >> change
< > < > >> to indicate a lifecycle to the entry.
< > < > >>
< > < > >> /mrg
< > < > >>
< > < > >> On May 21, 2008, at 12:42, Chris Hyzer wrote:
< > < > >>
< > < > >>> Well, if we want reliable notifications, then we need
< > < transactional
< > < > >>> hooks.
< > < > >>> If we want augmenters, that is a pre-hook or post-hook.
< > <  If we need
< > < > >>> to know the hibernate id of whatever was inserted, that
< > < is a post
< > < > >>> hook (post the hibernate flush) (e.g. auto add a member
< > < to a group
< > < > >>> just created).  In both cases we definitely want the
< > < ability to be
< > < > >>> in the same transaction.  I think all that is pretty
< > < simple since
< > < > >>> transactions in threadlocal exist in grouper so the
< > < hook can use it
< > < > >>> or not (for the same database).  Reliable notifications is
< > < > trickier,
< > < > >>> and something like the daemon design in my previous email  
< > would
< > < > >>> solve it).  I agree with you that if the auditing is in
< > < an external
< > < > >>> database then we don't need to worry about transactions
< > < (as much).
< > < > >>>
< > < > >>> I think a complication is as we discussed in the previous  
< > phone
< > < > call
< > < > >>> where we have multiple chained hooks, and which order
< > < they come in,
< > < > >>> etc.  If we can cut that down (even to one non-API hook
< > < for pre and
< > < > >>> one for post) it will make it easier to work with.
< > < > >>>
< > < > >>> About the hooks generating more hook events, I 
< think we should
< > < > >>> ignore this for now, I think it will work itself out.
< > < For instance
< > < > >>> for a group with a type of "requireFaculty", maybe the
< > < > >>> creaetGroup
< > < > >>> hook will build out a composite group that and's with a  
< > faculty
< > < > >>> group (if the composite doesn't already exist).  
< The composite
< > < > >>> group
< > < > >>> will also trigger the createGroup hook, but the hook
< > < will see that
< > < > >>> the composite group doesn't have the type 
< "requireFaculty" so
< > < > >>> it
< > < > >>> will ignore it.  Once we come up with an endless loop
< > < > >>> requirement, I
< > < > >>> bet we will be able to do something clever.
< > < Furthermore, I could
< > < > >>> see the hook code solving it themselves with a
< > < threadlocal (set a
< > < > >>> flag that says we are in the specific hook, clear it in
< > < a finally
< > < > >>> block, and check for the flag at the beginning of the  
< > hook).  I
< > < > >>> don't think we should allow a hook to turn off all events,  
< > there
< > < > >>> might have been some good ones in there.  J
< > < > >>>
< > < > >>> Kind regards,
< > < > >>> Chris
< > < > >>>
< > < > >>>
< > < > >>>
< > < > >>> From: Bert Bee-Lindgren
< > < 
[mailto:]
< > < > >>> Sent: Wednesday, May 21, 2008 11:53 AM
< > < > >>> To: Mike Olive
< > < > >>> Cc: Chris Hyzer; 'Grouper Dev'; 

< > < > >>> Subject: Re: [grouper-dev] notifications vs hooks
< > < > >>>
< > < > >>> Thanks Mike & Chris,
< > < > >>>
< > < > >>> I just get the feeling that pre-hooks and same-transaction
< > < > semantics
< > < > >>> (especially across data sources) are solving problems I
< > < don't see:
< > < > I
< > < > >>> see huge near-term values of pre-hooks as filters. I
< > < don't see how
< > < > >>> these need coordinated commit/rollback with the GrouperDB
< > < > >> transaction.
< > < > >>>
< > < > >>> From reviewing Grouper/Signet roadmap issues, I see the
< > < following
< > < > >>> most affected by Hooks & Plugins.
< > < > >>>  -Notification of changes - 
< email/grouper-to-signet/triggered
< > < > >>> provisioning
< > < > >>>  -History & audit
< > < > >>>  -Rule-based action (not sure if this uses plugins, but
< > < some rules
< > < > >>> could be considered augmentation)
< > < > >>>
< > < > >>> Maybe my ACID religion will take a hit from this, but
< > < I'm not sure
< > < > I
< > < > >>> see the harm in Auditing and (membership and attribute)
< > < > Augmentation
< > < > >>> happening based on reliable notifications after the group's
< > < > >>> transaction commits. Further, when the group is changed
< > < via Post-
< > < > >>> Hook augmentation, it's possible that other 
< pre-hooks should  
< > be
< > < > >>> triggered (Orig action, pre-hook, hib flush, post-hook that
< > < > >>> augments, pre-hook because request changed, hib flush,
< > < post-hook,
< > < > >>> commit.... ugh... double-ugh if it loops further or if
< > < a pre-hook
< > < > >>> later disagrees)
< > < > >>>
< > < > >>> Generally, I see the following levels of increasing
< > < functionality
< > < > >>> and complexity. And I see current plans for #4 while #1
< > < or 2 seem
< > < > >>> (to me) to be so much easier and offer so much value.I
< > < see 80% of
< > < > >>> vision I've heard being done with #1 and 90+% with #2.
< > < > >>>          1) Pre-hooks as filters, reliable post-action
< > < > >>> notifications [I'd call this the minimum]
< > < > >>>          2) #1 with pre-hooks also as augmenters
< > < > >>>          3) #2 with all pre-hooks seeing the final 
< (augmented)
< > < > group
< > < > >>>          4) #3 with post-hooks and same-transaction 
< semantics
< > < > >>>
< > < > >>> Am I missing Use Cases, or not considering ACID-lite
< > < problems, or
< > < > am
< > < > >>> I seeing difficulty where there is none?
< > < > >>> Put differently, if this is all truly necessary, great.
< > < Or if this
< > < > >>> is easier than it seems, even better.
< > < > >>>
< > < > >>> Thanks, yet again, for you patience,
< > < > >>>  Bert
< > < > >>>
< > < > >>> On May 21, 2008, at 11:03 AM, Mike Olive wrote:
< > < > >>>
< > < > >>>
< > < > >>> Bert,
< > < > >>>
< > < > >>> Pre-hooks is certainly the more complicated aspect of
< > < this design
< > < > >>> with the implementation
< > < > >>> most likely requiring the use of a transaction manager
< > < so that the
< > < > >>> pre-hook plug-ins may
< > < > >>> participate in the final commit or rollback of the actual
< > < > >> transaction.
< > < > >>>
< > < > >>> On post-hooks persistence, the current design is 
< agnostic of  
< > any
< > < > >>> messaging solution or
< > < > >>> transportation mechanism of the change notification.
< > < The design is
< > < > >>> extensible such that
< > < > >>> the developer could incorporate a guaranteed messaging  
< > solution
< > < > >>> (client) such as JMS.
< > < > >>> --
< > < > >>> Michael Olive
< > < > >>>
< > < > >>>
< > < > >>>
< > < > >>> From: Bert Bee-Lindgren
< > < 
[mailto:]
< > < > >>> Sent: Wednesday, May 14, 2008 12:05 PM
< > < > >>> To: Chris Hyzer
< > < > >>> Cc: Grouper Dev; 

< > < > >>> Subject: Re: [grouper-dev] notifications vs hooks
< > < > >>> Combining our approach to similar problems with UPenn's
< > < plans... I
< > < > >>> think we should consider very different mechanisms for pre- 
< > hooks
< > < > and
< > < > >>> post-hooks.
< > < > >>>
< > < > >>> Pre-hooks:
< > < > >>> Normal, synchronous method calls
< > < > >>> Pre-hook plugin developers should expect the event to
< > < possibly not
< > < > >>> occur even if they approve it. They should not 
< notify, log,  
< > etc
< > < > >>> anything that might indicate to a 
< user/auditor/sysadmin that  
< > an
< > < > >>> event happened... because they won't know about downstream
< > < > >> rejections.
< > < > >>>
< > < > >>>
< > < > >>> On the post-hooks, I think we should consider a persistent  
< > post-
< > < > >>> hook:
< > < > >>>  a) A plugin would have two inbound methods
< > < > >>> a1) "This event happened, do you care?"  [Boolean return, no
< > < > >>> external processing allowed, must be "fast"]
< > < > >>> a2) "Process this event, let us know when you've 
< succeeded in
< > < > >>> handling it" [Boolean return, TRUE means this plugin  
< > succeeded]
< > < > >>>  b) Create plugin-specific/event-specific database rows
< > < in an event
< > < > >>> table based on the TRUE returns of a1's
< > < > >>>  c) Immediately after all the plugins have had a chance
< > < to answer
< > < > >>> a1, hand the event to all the interested plugins a2's.
< > < > >>>  d) Delete the plugin-specific/event-specific row when that
< > < > >>> plugin's a2 returns true
< > < > >>>  e) Retry the failed a2's for a plugin before any new a2
< > < > >>>  f) Possibly retry the failed a2's every couple minutes or  
< > with
< > < > >>> some backoff approach (or, disappointingly, wait 
< for the next
< > < > event)
< > < > >>>
< > < > >>> Yes, this is basically a message queue, but simple to
< > < implement (we
< > < > >>> use a python-based version of this for several event
< > < queues in our
< > < > >>> system). I've looked for a JMS library as simple to use as  
< > this
< > < > two-
< > < > >>> method approach and haven't found one.
< > < > >>>
< > < > >>>
< > < > >>> On May 14, 2008, at 2:27 PM, Chris Hyzer wrote:
< > < > >>>
< > < > >>> Gary put comments on my hooks page about transactions and
< > < > >>> notifications.
< > < > >>>
< > < > 
< https://wiki.internet2.edu/confluence/display/GrouperWG/Hooks+simple
< > < > >>> It makes this whole thing very complicated. if the
< > < actions happens
< > < > >>> in a long running transaction, and you want to be
< > < notified at the
< > < > >>> end, there are a few issues:
< > < > >>> 1.       It is a different architecture than we had
< > < been discussing
< > < > >>> since we need to know about the action at the time of the
< > < > >>> (successful) commit.  Perhaps using Hibernate's 
< events could  
< > do
< > < > >>> the
< > < > >>> trick, but you don't have any object model anymore, you
< > < just have
< > < > >>> a
< > < > >>> list of column data for one table
< > < > >>> 2.       Like Gary points out, if the thing you are updating
< > < > >>> external to grouper fails, how do you log that and
< > < catch up later
< > < > >>> (e.g. if you are calling a web service, and there 
< is a network
< > < > >>> issue)
< > < > >>> 3.       There are lots of different producers of
< > < events (UI, WS,
< > < > >>> extensions e.g. gsh, and direct db edits [granted they
< > < > >>> shouldn't]).
< > < > >>> Must make sure the notification hooks are registered  
< > everywhere,
< > < > and
< > < > >>> test them to make sure they are firing everywhere
< > < (seems tedious /
< > < > >>> risky)
< > < > >>> Lets take the use case of writing your own ldappc via
< > < notifications
< > < > >>> (something we will start out with at penn).  We want to
< > < know about
< > < > >>> new members, memberships, and groups.  We will just
< > < make 3 tables
< > < > >>> with the id's and timestamps of when these change:
< > < > >>> select * from ldap_change_memberships lcm where rownum
< > < < 4 order by
< > < > >>> lcm.LAST_UPDATED desc
< > < > >>> MEMBERSHIP_UUID      LAST_UPDATED
< > < > >>> cd5f23d2-a8c8-44c0-a8b1-a3c3210da3c5                5/8/2008
< > < > >>> 1:21:37.569272 PM
< > < > >>> 74710fee-40b2-48e4-a8dd-b750876bc4ea             5/8/2008
< > < > >>> 1:21:37.462896 PM
< > < > >>> f2cf6b80-377c-41c3-981d-0eb9274dc74a                5/8/2008
< > < > >>> 1:21:36.076199 PM
< > < > >>> On the grouper tables I have some simple triggers 
< that check  
< > for
< > < > >>> diffs and insert to the change tables (and delete old
< > < records since
< > < > >>> all the daemon cares about is the most recently changed  
< > record).
< > < > >>> Then we also have friendly views for the daemon to use
< > < to query the
< > < > >>> data:
< > < > >>> select gmv.GROUP_NAME, gmv.SUBJECT_ID, gmv.SUBJECT_SOURCE,
< > < > >>> gmv.MSHIP_TYPE from grouper_memberships_v gmv where 
< rownum <4
< > < > >>> GROUP_NAME
< > < > >>> SUBJECT_ID                        SUBJECT_SOURCE
< > < > >>> MSHIP_TYPE
< > < > >>> Centers:ISC:PennAccess -ext     GrouperSystem
< > < > >>> g:isa                                       immediate
< > < > >>> Centers:ISC:PennAccess -ext     GrouperSystem
< > < > >>> g:isa                                       immediate
< > < > >>> Centers:ISC:PennAccess -ext     GrouperSystem
< > < > >>> g:isa                                       immediate
< > < > >>> Then a daemon will run every 5 minutes to push the 
< new data to
< > < > ldap,
< > < > >>> and delete the record from the change table when done.
< > < > >>> So this is all transactional, nothing can slip by
< > < (since trigger),
< > < > >>> and nothing can happen wrong if ldap update fails.
< > < > >>> Triggers are DB specific, and it requires different
< > < tables for each
< > < > >>> notification type.
< > < > >>> For notifications, grouper could provide:
< > < > >>> 1.       If you have certain table structures for 
< last_updated
< > < > dates
< > < > >>> (perhaps with name prefixes to support multiple)
< > < > >>> 2.       We could populate with Java hooks (perhaps not  
< > reliable
< > < > per
< > < > >>> discussion above), or you could do triggers (we 
< could provide
< > < > >>> examples for certain DB's) which would be more reliable and
< > < > >>> probably
< > < > >>> more performant
< > < > >>> 3.       We could provide a daemon (e.g. runs every 5
< > < minutes) with
< > < > >>> callbacks that would process the change tables 
< (based on name
< > < > >>> prefix), and gives you a callback to take the data 
< and put it
< > < > >>> somewhere
< > < > >>> 4.       We could provide a scheduled way to get all
< > < data not just
< > < > >>> the diffs (e.g. daily or weekly or monthly to ensure
< > < things are in
< > < > >>> sync)
< > < > >>> Just brainstorming here.
< > < > >>> Thanks,
< > < > >>> Chris
< > < > >>>
< > < > >>>
< > < > >
< > <
< > <
< >
< 
<