Grouper Developers Forum

["Mike Olive" <>]

In addition to being database agnostic the solution should be
such that plug-ins (integrations) are unaffected by new releases
of Grouper/Signet that may include changes in the data model.
Also, it would be better to provide a solution that does not mandate
that a plug-in developer require innate knowledge of Grouper/Signet
storage.

--
Michael Olive


< -----Original Message-----
< From: Chris Hyzer 
[mailto:]
 
< Sent: Thursday, May 22, 2008 7:33 AM
< To: Michael R. Gettes; Grouper Dev; 

< Subject: [signet-dev] RE: [grouper-dev] notifications vs hooks
< 
< Thanks for the reality check :).  I hope we are almost there. 
<  Ideally by Friday or Wednesday we can nail down the design 
< and then we can get it done quickly (it might seem like the 
< contrary, but its not that complicated)... also, this 
< discussion has definitely changed the design for the better 
< (so far), so I think it has been productive...
< 
< Btw, even if everyone were oracle, Im afraid DB triggers are 
< not the way to go, since the java API of grouper or signet 
< would not be there (there are Java stored procs in oracle, 
< but if you have used them, you know that they are painful).  
< Also we need to be consistent with how other customizations 
< are coded, in java and configured in config files (e.g. 
< authentication in the UI or WS)...
< 
< Regards,
< Chris
< 
< > -----Original Message-----
< > From: Michael R. Gettes 
[mailto:]
< > Sent: Thursday, May 22, 2008 8:32 AM
< > To: Grouper Dev; 

< > Subject: Re: [grouper-dev] notifications vs hooks
< >
< > It's always nice to try and be DB agnostic... but why not 
< say "if you
< > want realtime
< > UI response" or DB triggers specifically, then use a DB technology
< > that supports
< > it like Oracle.  And then engineer things to support triggers.  What
< > really worries
< > me is our gazing at navels whilst we try and figure out the perfect
< > solution
< > instead of getting something done to find out how useful 
< stuff can be
< > in the
< > real world.  We need to find that balance between perfection and
< > getting stuff done.
< >
< > And thanks for the clarification chris.
< >
< > /mrg
< >
< > On May 22, 2008, at 1:01, Chris Hyzer wrote:
< >
< > > I believe what you are describing is similar to what I was
< > > describing (below) for notifications based on Bert's email.  For
< > > hooks I believe we need other features (like veto, or 
< transactional
< > > augmenters), so Im not sure it can happen quite like 
< this...  Im not
< > > sure I understand your "temp db" suggestion, but when 
< thinking about
< > > a user using the UI, they need instant hook 
< action/results for when
< > > their screen displays a response, and it shouldn't impact
< > > performance much (e.g. having to wait for another process 
< to decide
< > > something).  Also, I agree that DB triggers could be a 
< way to solve
< > > some of these issues, but since we are db agnostic, and we like
< > > Java, it doesn't seem viable.
< > >
< > > Thanks,
< > > Chris
< > >
< > >
< > > From: Chris Hyzer
< > > Sent: Wednesday, May 14, 2008 4:46 PM
< > > To: 'Bert Bee-Lindgren'
< > > Cc: Grouper Dev; 

< > > Subject: RE: [grouper-dev] notifications vs hooks
< > >
< > > Bert, thanks for your email, that also seems like a more
< > > transactional solution, though Im still not exactly sure 
< how we know
< > > when something is committed in grouper in java (now that we have
< > > these fancy new long running transactions. dote!  ? ).  Also, if
< > > the process terminates while it is in the middle of 
< notifying, does
< > > it restart when it starts back up.  Is there one central daemon
< > > process that does that, or will it happen in UI / WS / GSH / etc.
< > > if there is a notifier in UI / WS / GSH then how will it 
< know which
< > > records in the table are for them to process and not someone else?
< > > But that aside, I think we should separate hooks and 
< notifications.
< > > I don't think all post-hooks are notification based.  Two 
< examples:
< > >
< > > 1.      Auditing.  I want to insert an audit record about 
< something
< > > done, and it should be done after the method call which 
< is after all
< > > pre-hooks complete.  This should still be in the same DB 
< transaction
< > > (hopefully)
< > > 2.      Augmenting with other objects.  I want to wait for a new
< > > group to be inserted (and the hibernate id created, which 
< is done on
< > > hibernate flush), then attach a type to it, and maybe some
< > > memberships or something, still in the same DB transaction
< > >
< > > So I think we have pre-hooks, post-hooks (which could be used for
< > > non-reliable notifications), and notifications.
< > >
< > > If your description below (items c-f) are processed on a separate
< > > daemon, then I would be up for that (1 to 1 mapping between daemon
< > > and database).  The insert into the table could happen 
< with a post-
< > > hook, and be in the same DB transaction.  The separate 
< daemon would
< > > poll the changed table for items and the destination 
< (which would be
< > > a configured listener), hand-off and delete (repeat if errors).
< > > Sounds great!  ?
< > >
< > > Chris
< > >
< > >> -----Original Message-----
< > >> From: Michael R. Gettes 
[mailto:]
< > >> Sent: Wednesday, May 21, 2008 4:38 PM
< > >> To: Chris Hyzer
< > >> Cc: Bert Bee-Lindgren; Mike Olive; 'Grouper Dev'; signet-
< > >> 

< > >> Subject: Re: [grouper-dev] notifications vs hooks
< > >>
< > >> So, I am sure you guys have all thought about this a bit and I am
< > >> curious
< > >> about why not the following approach...
< > >>
< > >> instead of creating a software structure (a layering of 
< plugins and
< > >> such
< > >> to handle notifications and modification of data and 
< yada yada) why
< > >> not
< > >> come up with some protocol at the data layer?  at some scratch on
< > the
< > >> wall in the db when an item is added to the db and let some other
< > >> process notice the change.  if you need "pre-commit" 
< functionality
< > >> then add it to some temp db and go from there.  Then you 
< don't have
< > >> to worry about putting in this layer in all software components -
< > >> it's
< > >> just a matter of adding some state to an entry and the state can
< > >> change
< > >> to indicate a lifecycle to the entry.
< > >>
< > >> /mrg
< > >>
< > >> On May 21, 2008, at 12:42, Chris Hyzer wrote:
< > >>
< > >>> Well, if we want reliable notifications, then we need 
< transactional
< > >>> hooks.
< > >>> If we want augmenters, that is a pre-hook or post-hook. 
<  If we need
< > >>> to know the hibernate id of whatever was inserted, that 
< is a post
< > >>> hook (post the hibernate flush) (e.g. auto add a member 
< to a group
< > >>> just created).  In both cases we definitely want the 
< ability to be
< > >>> in the same transaction.  I think all that is pretty 
< simple since
< > >>> transactions in threadlocal exist in grouper so the 
< hook can use it
< > >>> or not (for the same database).  Reliable notifications is
< > trickier,
< > >>> and something like the daemon design in my previous email would
< > >>> solve it).  I agree with you that if the auditing is in 
< an external
< > >>> database then we don't need to worry about transactions 
< (as much).
< > >>>
< > >>> I think a complication is as we discussed in the previous phone
< > call
< > >>> where we have multiple chained hooks, and which order 
< they come in,
< > >>> etc.  If we can cut that down (even to one non-API hook 
< for pre and
< > >>> one for post) it will make it easier to work with.
< > >>>
< > >>> About the hooks generating more hook events, I think we should
< > >>> ignore this for now, I think it will work itself out.  
< For instance
< > >>> for a group with a type of "requireFaculty", maybe the
< > >>> creaetGroup
< > >>> hook will build out a composite group that and's with a faculty
< > >>> group (if the composite doesn't already exist).  The composite
< > >>> group
< > >>> will also trigger the createGroup hook, but the hook 
< will see that
< > >>> the composite group doesn't have the type "requireFaculty" so
< > >>> it
< > >>> will ignore it.  Once we come up with an endless loop
< > >>> requirement, I
< > >>> bet we will be able to do something clever.  
< Furthermore, I could
< > >>> see the hook code solving it themselves with a 
< threadlocal (set a
< > >>> flag that says we are in the specific hook, clear it in 
< a finally
< > >>> block, and check for the flag at the beginning of the hook).  I
< > >>> don't think we should allow a hook to turn off all events, there
< > >>> might have been some good ones in there.  J
< > >>>
< > >>> Kind regards,
< > >>> Chris
< > >>>
< > >>>
< > >>>
< > >>> From: Bert Bee-Lindgren 
< 
[mailto:]
< > >>> Sent: Wednesday, May 21, 2008 11:53 AM
< > >>> To: Mike Olive
< > >>> Cc: Chris Hyzer; 'Grouper Dev'; 

< > >>> Subject: Re: [grouper-dev] notifications vs hooks
< > >>>
< > >>> Thanks Mike & Chris,
< > >>>
< > >>> I just get the feeling that pre-hooks and same-transaction
< > semantics
< > >>> (especially across data sources) are solving problems I 
< don't see:
< > I
< > >>> see huge near-term values of pre-hooks as filters. I 
< don't see how
< > >>> these need coordinated commit/rollback with the GrouperDB
< > >> transaction.
< > >>>
< > >>> From reviewing Grouper/Signet roadmap issues, I see the 
< following
< > >>> most affected by Hooks & Plugins.
< > >>>  -Notification of changes - email/grouper-to-signet/triggered
< > >>> provisioning
< > >>>  -History & audit
< > >>>  -Rule-based action (not sure if this uses plugins, but 
< some rules
< > >>> could be considered augmentation)
< > >>>
< > >>> Maybe my ACID religion will take a hit from this, but 
< I'm not sure
< > I
< > >>> see the harm in Auditing and (membership and attribute)
< > Augmentation
< > >>> happening based on reliable notifications after the group's
< > >>> transaction commits. Further, when the group is changed 
< via Post-
< > >>> Hook augmentation, it's possible that other pre-hooks should be
< > >>> triggered (Orig action, pre-hook, hib flush, post-hook that
< > >>> augments, pre-hook because request changed, hib flush, 
< post-hook,
< > >>> commit.... ugh... double-ugh if it loops further or if 
< a pre-hook
< > >>> later disagrees)
< > >>>
< > >>> Generally, I see the following levels of increasing 
< functionality
< > >>> and complexity. And I see current plans for #4 while #1 
< or 2 seem
< > >>> (to me) to be so much easier and offer so much value.I 
< see 80% of
< > >>> vision I've heard being done with #1 and 90+% with #2.
< > >>>          1) Pre-hooks as filters, reliable post-action
< > >>> notifications [I'd call this the minimum]
< > >>>          2) #1 with pre-hooks also as augmenters
< > >>>          3) #2 with all pre-hooks seeing the final (augmented)
< > group
< > >>>          4) #3 with post-hooks and same-transaction semantics
< > >>>
< > >>> Am I missing Use Cases, or not considering ACID-lite 
< problems, or
< > am
< > >>> I seeing difficulty where there is none?
< > >>> Put differently, if this is all truly necessary, great. 
< Or if this
< > >>> is easier than it seems, even better.
< > >>>
< > >>> Thanks, yet again, for you patience,
< > >>>  Bert
< > >>>
< > >>> On May 21, 2008, at 11:03 AM, Mike Olive wrote:
< > >>>
< > >>>
< > >>> Bert,
< > >>>
< > >>> Pre-hooks is certainly the more complicated aspect of 
< this design
< > >>> with the implementation
< > >>> most likely requiring the use of a transaction manager 
< so that the
< > >>> pre-hook plug-ins may
< > >>> participate in the final commit or rollback of the actual
< > >> transaction.
< > >>>
< > >>> On post-hooks persistence, the current design is agnostic of any
< > >>> messaging solution or
< > >>> transportation mechanism of the change notification. 
< The design is
< > >>> extensible such that
< > >>> the developer could incorporate a guaranteed messaging solution
< > >>> (client) such as JMS.
< > >>> --
< > >>> Michael Olive
< > >>>
< > >>>
< > >>>
< > >>> From: Bert Bee-Lindgren 
< 
[mailto:]
< > >>> Sent: Wednesday, May 14, 2008 12:05 PM
< > >>> To: Chris Hyzer
< > >>> Cc: Grouper Dev; 

< > >>> Subject: Re: [grouper-dev] notifications vs hooks
< > >>> Combining our approach to similar problems with UPenn's 
< plans... I
< > >>> think we should consider very different mechanisms for pre-hooks
< > and
< > >>> post-hooks.
< > >>>
< > >>> Pre-hooks:
< > >>> Normal, synchronous method calls
< > >>> Pre-hook plugin developers should expect the event to 
< possibly not
< > >>> occur even if they approve it. They should not notify, log, etc
< > >>> anything that might indicate to a user/auditor/sysadmin that an
< > >>> event happened... because they won't know about downstream
< > >> rejections.
< > >>>
< > >>>
< > >>> On the post-hooks, I think we should consider a persistent post-
< > >>> hook:
< > >>>  a) A plugin would have two inbound methods
< > >>> a1) "This event happened, do you care?"  [Boolean return, no
< > >>> external processing allowed, must be "fast"]
< > >>> a2) "Process this event, let us know when you've succeeded in
< > >>> handling it" [Boolean return, TRUE means this plugin succeeded]
< > >>>  b) Create plugin-specific/event-specific database rows 
< in an event
< > >>> table based on the TRUE returns of a1's
< > >>>  c) Immediately after all the plugins have had a chance 
< to answer
< > >>> a1, hand the event to all the interested plugins a2's.
< > >>>  d) Delete the plugin-specific/event-specific row when that
< > >>> plugin's a2 returns true
< > >>>  e) Retry the failed a2's for a plugin before any new a2
< > >>>  f) Possibly retry the failed a2's every couple minutes or with
< > >>> some backoff approach (or, disappointingly, wait for the next
< > event)
< > >>>
< > >>> Yes, this is basically a message queue, but simple to 
< implement (we
< > >>> use a python-based version of this for several event 
< queues in our
< > >>> system). I've looked for a JMS library as simple to use as this
< > two-
< > >>> method approach and haven't found one.
< > >>>
< > >>>
< > >>> On May 14, 2008, at 2:27 PM, Chris Hyzer wrote:
< > >>>
< > >>> Gary put comments on my hooks page about transactions and
< > >>> notifications.
< > >>>
< > https://wiki.internet2.edu/confluence/display/GrouperWG/Hooks+simple
< > >>> It makes this whole thing very complicated. if the 
< actions happens
< > >>> in a long running transaction, and you want to be 
< notified at the
< > >>> end, there are a few issues:
< > >>> 1.       It is a different architecture than we had 
< been discussing
< > >>> since we need to know about the action at the time of the
< > >>> (successful) commit.  Perhaps using Hibernate's events could do
< > >>> the
< > >>> trick, but you don't have any object model anymore, you 
< just have
< > >>> a
< > >>> list of column data for one table
< > >>> 2.       Like Gary points out, if the thing you are updating
< > >>> external to grouper fails, how do you log that and 
< catch up later
< > >>> (e.g. if you are calling a web service, and there is a network
< > >>> issue)
< > >>> 3.       There are lots of different producers of 
< events (UI, WS,
< > >>> extensions e.g. gsh, and direct db edits [granted they
< > >>> shouldn't]).
< > >>> Must make sure the notification hooks are registered everywhere,
< > and
< > >>> test them to make sure they are firing everywhere 
< (seems tedious /
< > >>> risky)
< > >>> Lets take the use case of writing your own ldappc via 
< notifications
< > >>> (something we will start out with at penn).  We want to 
< know about
< > >>> new members, memberships, and groups.  We will just 
< make 3 tables
< > >>> with the id's and timestamps of when these change:
< > >>> select * from ldap_change_memberships lcm where rownum 
< < 4 order by
< > >>> lcm.LAST_UPDATED desc
< > >>> MEMBERSHIP_UUID      LAST_UPDATED
< > >>> cd5f23d2-a8c8-44c0-a8b1-a3c3210da3c5                5/8/2008
< > >>> 1:21:37.569272 PM
< > >>> 74710fee-40b2-48e4-a8dd-b750876bc4ea             5/8/2008
< > >>> 1:21:37.462896 PM
< > >>> f2cf6b80-377c-41c3-981d-0eb9274dc74a                5/8/2008
< > >>> 1:21:36.076199 PM
< > >>> On the grouper tables I have some simple triggers that check for
< > >>> diffs and insert to the change tables (and delete old 
< records since
< > >>> all the daemon cares about is the most recently changed record).
< > >>> Then we also have friendly views for the daemon to use 
< to query the
< > >>> data:
< > >>> select gmv.GROUP_NAME, gmv.SUBJECT_ID, gmv.SUBJECT_SOURCE,
< > >>> gmv.MSHIP_TYPE from grouper_memberships_v gmv where rownum <4
< > >>> GROUP_NAME
< > >>> SUBJECT_ID                        SUBJECT_SOURCE
< > >>> MSHIP_TYPE
< > >>> Centers:ISC:PennAccess -ext     GrouperSystem
< > >>> g:isa                                       immediate
< > >>> Centers:ISC:PennAccess -ext     GrouperSystem
< > >>> g:isa                                       immediate
< > >>> Centers:ISC:PennAccess -ext     GrouperSystem
< > >>> g:isa                                       immediate
< > >>> Then a daemon will run every 5 minutes to push the new data to
< > ldap,
< > >>> and delete the record from the change table when done.
< > >>> So this is all transactional, nothing can slip by 
< (since trigger),
< > >>> and nothing can happen wrong if ldap update fails.
< > >>> Triggers are DB specific, and it requires different 
< tables for each
< > >>> notification type.
< > >>> For notifications, grouper could provide:
< > >>> 1.       If you have certain table structures for last_updated
< > dates
< > >>> (perhaps with name prefixes to support multiple)
< > >>> 2.       We could populate with Java hooks (perhaps not reliable
< > per
< > >>> discussion above), or you could do triggers (we could provide
< > >>> examples for certain DB's) which would be more reliable and
< > >>> probably
< > >>> more performant
< > >>> 3.       We could provide a daemon (e.g. runs every 5 
< minutes) with
< > >>> callbacks that would process the change tables (based on name
< > >>> prefix), and gives you a callback to take the data and put it
< > >>> somewhere
< > >>> 4.       We could provide a scheduled way to get all 
< data not just
< > >>> the diffs (e.g. daily or weekly or monthly to ensure 
< things are in
< > >>> sync)
< > >>> Just brainstorming here.
< > >>> Thanks,
< > >>> Chris
< > >>>
< > >>>
< > >
< 
<