Skip to Content.
Sympa Menu

grouper-dev - RE: [signet-dev] RE: [grouper-dev] notifications vs hooks

Subject: Grouper Developers Forum

List archive

RE: [signet-dev] RE: [grouper-dev] notifications vs hooks


Chronological Thread 
  • From: "Mike Olive" <>
  • To: "'Chris Hyzer'" <>, "'Michael R. Gettes'" <>, "'Grouper Dev'" <>, <>
  • Subject: RE: [signet-dev] RE: [grouper-dev] notifications vs hooks
  • Date: Thu, 22 May 2008 13:25:49 -0700

In addition to being database agnostic the solution should be
such that plug-ins (integrations) are unaffected by new releases
of Grouper/Signet that may include changes in the data model.
Also, it would be better to provide a solution that does not mandate
that a plug-in developer require innate knowledge of Grouper/Signet
storage.

--
Michael Olive


< -----Original Message-----
< From: Chris Hyzer
[mailto:]

< Sent: Thursday, May 22, 2008 7:33 AM
< To: Michael R. Gettes; Grouper Dev;

< Subject: [signet-dev] RE: [grouper-dev] notifications vs hooks
<
< Thanks for the reality check :). I hope we are almost there.
< Ideally by Friday or Wednesday we can nail down the design
< and then we can get it done quickly (it might seem like the
< contrary, but its not that complicated)... also, this
< discussion has definitely changed the design for the better
< (so far), so I think it has been productive...
<
< Btw, even if everyone were oracle, Im afraid DB triggers are
< not the way to go, since the java API of grouper or signet
< would not be there (there are Java stored procs in oracle,
< but if you have used them, you know that they are painful).
< Also we need to be consistent with how other customizations
< are coded, in java and configured in config files (e.g.
< authentication in the UI or WS)...
<
< Regards,
< Chris
<
< > -----Original Message-----
< > From: Michael R. Gettes
[mailto:]
< > Sent: Thursday, May 22, 2008 8:32 AM
< > To: Grouper Dev;

< > Subject: Re: [grouper-dev] notifications vs hooks
< >
< > It's always nice to try and be DB agnostic... but why not
< say "if you
< > want realtime
< > UI response" or DB triggers specifically, then use a DB technology
< > that supports
< > it like Oracle. And then engineer things to support triggers. What
< > really worries
< > me is our gazing at navels whilst we try and figure out the perfect
< > solution
< > instead of getting something done to find out how useful
< stuff can be
< > in the
< > real world. We need to find that balance between perfection and
< > getting stuff done.
< >
< > And thanks for the clarification chris.
< >
< > /mrg
< >
< > On May 22, 2008, at 1:01, Chris Hyzer wrote:
< >
< > > I believe what you are describing is similar to what I was
< > > describing (below) for notifications based on Bert's email. For
< > > hooks I believe we need other features (like veto, or
< transactional
< > > augmenters), so Im not sure it can happen quite like
< this... Im not
< > > sure I understand your "temp db" suggestion, but when
< thinking about
< > > a user using the UI, they need instant hook
< action/results for when
< > > their screen displays a response, and it shouldn't impact
< > > performance much (e.g. having to wait for another process
< to decide
< > > something). Also, I agree that DB triggers could be a
< way to solve
< > > some of these issues, but since we are db agnostic, and we like
< > > Java, it doesn't seem viable.
< > >
< > > Thanks,
< > > Chris
< > >
< > >
< > > From: Chris Hyzer
< > > Sent: Wednesday, May 14, 2008 4:46 PM
< > > To: 'Bert Bee-Lindgren'
< > > Cc: Grouper Dev;

< > > Subject: RE: [grouper-dev] notifications vs hooks
< > >
< > > Bert, thanks for your email, that also seems like a more
< > > transactional solution, though Im still not exactly sure
< how we know
< > > when something is committed in grouper in java (now that we have
< > > these fancy new long running transactions. dote! ? ). Also, if
< > > the process terminates while it is in the middle of
< notifying, does
< > > it restart when it starts back up. Is there one central daemon
< > > process that does that, or will it happen in UI / WS / GSH / etc.
< > > if there is a notifier in UI / WS / GSH then how will it
< know which
< > > records in the table are for them to process and not someone else?
< > > But that aside, I think we should separate hooks and
< notifications.
< > > I don't think all post-hooks are notification based. Two
< examples:
< > >
< > > 1. Auditing. I want to insert an audit record about
< something
< > > done, and it should be done after the method call which
< is after all
< > > pre-hooks complete. This should still be in the same DB
< transaction
< > > (hopefully)
< > > 2. Augmenting with other objects. I want to wait for a new
< > > group to be inserted (and the hibernate id created, which
< is done on
< > > hibernate flush), then attach a type to it, and maybe some
< > > memberships or something, still in the same DB transaction
< > >
< > > So I think we have pre-hooks, post-hooks (which could be used for
< > > non-reliable notifications), and notifications.
< > >
< > > If your description below (items c-f) are processed on a separate
< > > daemon, then I would be up for that (1 to 1 mapping between daemon
< > > and database). The insert into the table could happen
< with a post-
< > > hook, and be in the same DB transaction. The separate
< daemon would
< > > poll the changed table for items and the destination
< (which would be
< > > a configured listener), hand-off and delete (repeat if errors).
< > > Sounds great! ?
< > >
< > > Chris
< > >
< > >> -----Original Message-----
< > >> From: Michael R. Gettes
[mailto:]
< > >> Sent: Wednesday, May 21, 2008 4:38 PM
< > >> To: Chris Hyzer
< > >> Cc: Bert Bee-Lindgren; Mike Olive; 'Grouper Dev'; signet-
< > >>

< > >> Subject: Re: [grouper-dev] notifications vs hooks
< > >>
< > >> So, I am sure you guys have all thought about this a bit and I am
< > >> curious
< > >> about why not the following approach...
< > >>
< > >> instead of creating a software structure (a layering of
< plugins and
< > >> such
< > >> to handle notifications and modification of data and
< yada yada) why
< > >> not
< > >> come up with some protocol at the data layer? at some scratch on
< > the
< > >> wall in the db when an item is added to the db and let some other
< > >> process notice the change. if you need "pre-commit"
< functionality
< > >> then add it to some temp db and go from there. Then you
< don't have
< > >> to worry about putting in this layer in all software components -
< > >> it's
< > >> just a matter of adding some state to an entry and the state can
< > >> change
< > >> to indicate a lifecycle to the entry.
< > >>
< > >> /mrg
< > >>
< > >> On May 21, 2008, at 12:42, Chris Hyzer wrote:
< > >>
< > >>> Well, if we want reliable notifications, then we need
< transactional
< > >>> hooks.
< > >>> If we want augmenters, that is a pre-hook or post-hook.
< If we need
< > >>> to know the hibernate id of whatever was inserted, that
< is a post
< > >>> hook (post the hibernate flush) (e.g. auto add a member
< to a group
< > >>> just created). In both cases we definitely want the
< ability to be
< > >>> in the same transaction. I think all that is pretty
< simple since
< > >>> transactions in threadlocal exist in grouper so the
< hook can use it
< > >>> or not (for the same database). Reliable notifications is
< > trickier,
< > >>> and something like the daemon design in my previous email would
< > >>> solve it). I agree with you that if the auditing is in
< an external
< > >>> database then we don't need to worry about transactions
< (as much).
< > >>>
< > >>> I think a complication is as we discussed in the previous phone
< > call
< > >>> where we have multiple chained hooks, and which order
< they come in,
< > >>> etc. If we can cut that down (even to one non-API hook
< for pre and
< > >>> one for post) it will make it easier to work with.
< > >>>
< > >>> About the hooks generating more hook events, I think we should
< > >>> ignore this for now, I think it will work itself out.
< For instance
< > >>> for a group with a type of "requireFaculty", maybe the
< > >>> creaetGroup
< > >>> hook will build out a composite group that and's with a faculty
< > >>> group (if the composite doesn't already exist). The composite
< > >>> group
< > >>> will also trigger the createGroup hook, but the hook
< will see that
< > >>> the composite group doesn't have the type "requireFaculty" so
< > >>> it
< > >>> will ignore it. Once we come up with an endless loop
< > >>> requirement, I
< > >>> bet we will be able to do something clever.
< Furthermore, I could
< > >>> see the hook code solving it themselves with a
< threadlocal (set a
< > >>> flag that says we are in the specific hook, clear it in
< a finally
< > >>> block, and check for the flag at the beginning of the hook). I
< > >>> don't think we should allow a hook to turn off all events, there
< > >>> might have been some good ones in there. J
< > >>>
< > >>> Kind regards,
< > >>> Chris
< > >>>
< > >>>
< > >>>
< > >>> From: Bert Bee-Lindgren
<
[mailto:]
< > >>> Sent: Wednesday, May 21, 2008 11:53 AM
< > >>> To: Mike Olive
< > >>> Cc: Chris Hyzer; 'Grouper Dev';

< > >>> Subject: Re: [grouper-dev] notifications vs hooks
< > >>>
< > >>> Thanks Mike & Chris,
< > >>>
< > >>> I just get the feeling that pre-hooks and same-transaction
< > semantics
< > >>> (especially across data sources) are solving problems I
< don't see:
< > I
< > >>> see huge near-term values of pre-hooks as filters. I
< don't see how
< > >>> these need coordinated commit/rollback with the GrouperDB
< > >> transaction.
< > >>>
< > >>> From reviewing Grouper/Signet roadmap issues, I see the
< following
< > >>> most affected by Hooks & Plugins.
< > >>> -Notification of changes - email/grouper-to-signet/triggered
< > >>> provisioning
< > >>> -History & audit
< > >>> -Rule-based action (not sure if this uses plugins, but
< some rules
< > >>> could be considered augmentation)
< > >>>
< > >>> Maybe my ACID religion will take a hit from this, but
< I'm not sure
< > I
< > >>> see the harm in Auditing and (membership and attribute)
< > Augmentation
< > >>> happening based on reliable notifications after the group's
< > >>> transaction commits. Further, when the group is changed
< via Post-
< > >>> Hook augmentation, it's possible that other pre-hooks should be
< > >>> triggered (Orig action, pre-hook, hib flush, post-hook that
< > >>> augments, pre-hook because request changed, hib flush,
< post-hook,
< > >>> commit.... ugh... double-ugh if it loops further or if
< a pre-hook
< > >>> later disagrees)
< > >>>
< > >>> Generally, I see the following levels of increasing
< functionality
< > >>> and complexity. And I see current plans for #4 while #1
< or 2 seem
< > >>> (to me) to be so much easier and offer so much value.I
< see 80% of
< > >>> vision I've heard being done with #1 and 90+% with #2.
< > >>> 1) Pre-hooks as filters, reliable post-action
< > >>> notifications [I'd call this the minimum]
< > >>> 2) #1 with pre-hooks also as augmenters
< > >>> 3) #2 with all pre-hooks seeing the final (augmented)
< > group
< > >>> 4) #3 with post-hooks and same-transaction semantics
< > >>>
< > >>> Am I missing Use Cases, or not considering ACID-lite
< problems, or
< > am
< > >>> I seeing difficulty where there is none?
< > >>> Put differently, if this is all truly necessary, great.
< Or if this
< > >>> is easier than it seems, even better.
< > >>>
< > >>> Thanks, yet again, for you patience,
< > >>> Bert
< > >>>
< > >>> On May 21, 2008, at 11:03 AM, Mike Olive wrote:
< > >>>
< > >>>
< > >>> Bert,
< > >>>
< > >>> Pre-hooks is certainly the more complicated aspect of
< this design
< > >>> with the implementation
< > >>> most likely requiring the use of a transaction manager
< so that the
< > >>> pre-hook plug-ins may
< > >>> participate in the final commit or rollback of the actual
< > >> transaction.
< > >>>
< > >>> On post-hooks persistence, the current design is agnostic of any
< > >>> messaging solution or
< > >>> transportation mechanism of the change notification.
< The design is
< > >>> extensible such that
< > >>> the developer could incorporate a guaranteed messaging solution
< > >>> (client) such as JMS.
< > >>> --
< > >>> Michael Olive
< > >>>
< > >>>
< > >>>
< > >>> From: Bert Bee-Lindgren
<
[mailto:]
< > >>> Sent: Wednesday, May 14, 2008 12:05 PM
< > >>> To: Chris Hyzer
< > >>> Cc: Grouper Dev;

< > >>> Subject: Re: [grouper-dev] notifications vs hooks
< > >>> Combining our approach to similar problems with UPenn's
< plans... I
< > >>> think we should consider very different mechanisms for pre-hooks
< > and
< > >>> post-hooks.
< > >>>
< > >>> Pre-hooks:
< > >>> Normal, synchronous method calls
< > >>> Pre-hook plugin developers should expect the event to
< possibly not
< > >>> occur even if they approve it. They should not notify, log, etc
< > >>> anything that might indicate to a user/auditor/sysadmin that an
< > >>> event happened... because they won't know about downstream
< > >> rejections.
< > >>>
< > >>>
< > >>> On the post-hooks, I think we should consider a persistent post-
< > >>> hook:
< > >>> a) A plugin would have two inbound methods
< > >>> a1) "This event happened, do you care?" [Boolean return, no
< > >>> external processing allowed, must be "fast"]
< > >>> a2) "Process this event, let us know when you've succeeded in
< > >>> handling it" [Boolean return, TRUE means this plugin succeeded]
< > >>> b) Create plugin-specific/event-specific database rows
< in an event
< > >>> table based on the TRUE returns of a1's
< > >>> c) Immediately after all the plugins have had a chance
< to answer
< > >>> a1, hand the event to all the interested plugins a2's.
< > >>> d) Delete the plugin-specific/event-specific row when that
< > >>> plugin's a2 returns true
< > >>> e) Retry the failed a2's for a plugin before any new a2
< > >>> f) Possibly retry the failed a2's every couple minutes or with
< > >>> some backoff approach (or, disappointingly, wait for the next
< > event)
< > >>>
< > >>> Yes, this is basically a message queue, but simple to
< implement (we
< > >>> use a python-based version of this for several event
< queues in our
< > >>> system). I've looked for a JMS library as simple to use as this
< > two-
< > >>> method approach and haven't found one.
< > >>>
< > >>>
< > >>> On May 14, 2008, at 2:27 PM, Chris Hyzer wrote:
< > >>>
< > >>> Gary put comments on my hooks page about transactions and
< > >>> notifications.
< > >>>
< > https://wiki.internet2.edu/confluence/display/GrouperWG/Hooks+simple
< > >>> It makes this whole thing very complicated. if the
< actions happens
< > >>> in a long running transaction, and you want to be
< notified at the
< > >>> end, there are a few issues:
< > >>> 1. It is a different architecture than we had
< been discussing
< > >>> since we need to know about the action at the time of the
< > >>> (successful) commit. Perhaps using Hibernate's events could do
< > >>> the
< > >>> trick, but you don't have any object model anymore, you
< just have
< > >>> a
< > >>> list of column data for one table
< > >>> 2. Like Gary points out, if the thing you are updating
< > >>> external to grouper fails, how do you log that and
< catch up later
< > >>> (e.g. if you are calling a web service, and there is a network
< > >>> issue)
< > >>> 3. There are lots of different producers of
< events (UI, WS,
< > >>> extensions e.g. gsh, and direct db edits [granted they
< > >>> shouldn't]).
< > >>> Must make sure the notification hooks are registered everywhere,
< > and
< > >>> test them to make sure they are firing everywhere
< (seems tedious /
< > >>> risky)
< > >>> Lets take the use case of writing your own ldappc via
< notifications
< > >>> (something we will start out with at penn). We want to
< know about
< > >>> new members, memberships, and groups. We will just
< make 3 tables
< > >>> with the id's and timestamps of when these change:
< > >>> select * from ldap_change_memberships lcm where rownum
< < 4 order by
< > >>> lcm.LAST_UPDATED desc
< > >>> MEMBERSHIP_UUID LAST_UPDATED
< > >>> cd5f23d2-a8c8-44c0-a8b1-a3c3210da3c5 5/8/2008
< > >>> 1:21:37.569272 PM
< > >>> 74710fee-40b2-48e4-a8dd-b750876bc4ea 5/8/2008
< > >>> 1:21:37.462896 PM
< > >>> f2cf6b80-377c-41c3-981d-0eb9274dc74a 5/8/2008
< > >>> 1:21:36.076199 PM
< > >>> On the grouper tables I have some simple triggers that check for
< > >>> diffs and insert to the change tables (and delete old
< records since
< > >>> all the daemon cares about is the most recently changed record).
< > >>> Then we also have friendly views for the daemon to use
< to query the
< > >>> data:
< > >>> select gmv.GROUP_NAME, gmv.SUBJECT_ID, gmv.SUBJECT_SOURCE,
< > >>> gmv.MSHIP_TYPE from grouper_memberships_v gmv where rownum <4
< > >>> GROUP_NAME
< > >>> SUBJECT_ID SUBJECT_SOURCE
< > >>> MSHIP_TYPE
< > >>> Centers:ISC:PennAccess -ext GrouperSystem
< > >>> g:isa immediate
< > >>> Centers:ISC:PennAccess -ext GrouperSystem
< > >>> g:isa immediate
< > >>> Centers:ISC:PennAccess -ext GrouperSystem
< > >>> g:isa immediate
< > >>> Then a daemon will run every 5 minutes to push the new data to
< > ldap,
< > >>> and delete the record from the change table when done.
< > >>> So this is all transactional, nothing can slip by
< (since trigger),
< > >>> and nothing can happen wrong if ldap update fails.
< > >>> Triggers are DB specific, and it requires different
< tables for each
< > >>> notification type.
< > >>> For notifications, grouper could provide:
< > >>> 1. If you have certain table structures for last_updated
< > dates
< > >>> (perhaps with name prefixes to support multiple)
< > >>> 2. We could populate with Java hooks (perhaps not reliable
< > per
< > >>> discussion above), or you could do triggers (we could provide
< > >>> examples for certain DB's) which would be more reliable and
< > >>> probably
< > >>> more performant
< > >>> 3. We could provide a daemon (e.g. runs every 5
< minutes) with
< > >>> callbacks that would process the change tables (based on name
< > >>> prefix), and gives you a callback to take the data and put it
< > >>> somewhere
< > >>> 4. We could provide a scheduled way to get all
< data not just
< > >>> the diffs (e.g. daily or weekly or monthly to ensure
< things are in
< > >>> sync)
< > >>> Just brainstorming here.
< > >>> Thanks,
< > >>> Chris
< > >>>
< > >>>
< > >
<
<




Archive powered by MHonArc 2.6.16.

Top of Page