Grouper Developers Forum

[Chris Hyzer <>]

Bert, thanks for your email, that also seems like a more transactional solution, though Im still not exactly sure how we know when something is committed in grouper in java (now that we have these fancy new long running transactions… dote!  J ).  Also, if the process terminates while it is in the middle of notifying, does it restart when it starts back up.  Is there one central daemon process that does that, or will it happen in UI / WS / GSH / etc…   if there is a notifier in UI / WS / GSH then how will it know which records in the table are for them to process and not someone else?

But that aside, I think we should separate hooks and notifications.  I don’t think all post-hooks are notification based.  Two examples:

 

1.       Auditing.  I want to insert an audit record about something done, and it should be done after the method call which is after all pre-hooks complete.  This should still be in the same DB transaction (hopefully)

2.       Augmenting with other objects.  I want to wait for a new group to be inserted (and the hibernate id created, which is done on hibernate flush), then attach a type to it, and maybe some memberships or something, still in the same DB transaction

 

So I think we have pre-hooks, post-hooks (which could be used for non-reliable notifications), and notifications…

 

If your description below (items c-f) are processed on a separate daemon, then I would be up for that (1 to 1 mapping between daemon and database).  The insert into the table could happen with a post-hook, and be in the same DB transaction…  The separate daemon would poll the changed table for items and the destination (which would be a configured listener), hand-off and delete (repeat if errors).  Sounds great!  J

 

Chris

 

 

 

 

From: Bert Bee-Lindgren [mailto:]
Sent: Wednesday, May 14, 2008 3:05 PM
To: Chris Hyzer
Cc: Grouper Dev;
Subject: Re: [grouper-dev] notifications vs hooks

 

Combining our approach to similar problems with UPenn's plans... I think we should consider very different mechanisms for pre-hooks and post-hooks.

 

Pre-hooks:

          Normal, synchronous method calls

          Pre-hook plugin developers should expect the event to possibly not occur even if they approve it. They should not notify, log, etc anything that might indicate to a user/auditor/sysadmin that an event happened... because they won't know about downstream rejections.

 

 

On the post-hooks, I think we should consider a persistent post-hook:

  a) A plugin would have two inbound methods

          a1) "This event happened, do you care?"  [Boolean return, no external processing allowed, must be "fast"]

          a2) "Process this event, let us know when you've succeeded in handling it" [Boolean return, TRUE means this plugin succeeded]

  b) Create plugin-specific/event-specific database rows in an event table based on the TRUE returns of a1's

  c) Immediately after all the plugins have had a chance to answer a1, hand the event to all the interested plugins a2's.

  d) Delete the plugin-specific/event-specific row when that plugin's a2 returns true

  e) Retry the failed a2's for a plugin before any new a2

  f) Possibly retry the failed a2's every couple minutes or with some backoff approach (or, disappointingly, wait for the next event)

 

Yes, this is basically a message queue, but simple to implement (we use a python-based version of this for several event queues in our system). I've looked for a JMS library as simple to use as this two-method approach and haven't found one.

 

 

On May 14, 2008, at 2:27 PM, Chris Hyzer wrote:

Gary put comments on my hooks page about transactions and notifications.

 

https://wiki.internet2.edu/confluence/display/GrouperWG/Hooks+simple

 

It makes this whole thing very complicated… if the actions happens in a long running transaction, and you want to be notified at the end, there are a few issues:

 

1.       It is a different architecture than we had been discussing since we need to know about the action at the time of the (successful) commit.  Perhaps using Hibernate’s events could do the trick, but you don’t have any object model anymore, you just have a list of column data for one table

2.       Like Gary points out, if the thing you are updating external to grouper fails, how do you log that and catch up later (e.g. if you are calling a web service, and there is a network issue)

3.       There are lots of different producers of events (UI, WS, extensions e.g. gsh, and direct db edits [granted they shouldn’t]).  Must make sure the notification hooks are registered everywhere, and test them to make sure they are firing everywhere (seems tedious / risky)

 

Lets take the use case of writing your own ldappc via notifications (something we will start out with at penn).  We want to know about new members, memberships, and groups.  We will just make 3 tables with the id’s and timestamps of when these change:

 

select * from ldap_change_memberships lcm where rownum < 4 order by lcm.LAST_UPDATED desc

MEMBERSHIP_UUID      LAST_UPDATED

cd5f23d2-a8c8-44c0-a8b1-a3c3210da3c5                5/8/2008 1:21:37.569272 PM

74710fee-40b2-48e4-a8dd-b750876bc4ea             5/8/2008 1:21:37.462896 PM

f2cf6b80-377c-41c3-981d-0eb9274dc74a                5/8/2008 1:21:36.076199 PM

 

On the grouper tables I have some simple triggers that check for diffs and insert to the change tables (and delete old records since all the daemon cares about is the most recently changed record).

 

Then we also have friendly views for the daemon to use to query the data:

 

select gmv.GROUP_NAME, gmv.SUBJECT_ID, gmv.SUBJECT_SOURCE, gmv.MSHIP_TYPE from grouper_memberships_v gmv where rownum <4

GROUP_NAME                                 SUBJECT_ID                        SUBJECT_SOURCE           MSHIP_TYPE

 

Centers:ISC:PennAccess -ext     GrouperSystem                g:isa                                       immediate

Centers:ISC:PennAccess -ext     GrouperSystem                g:isa                                       immediate

Centers:ISC:PennAccess -ext     GrouperSystem                g:isa                                       immediate

 

Then a daemon will run every 5 minutes to push the new data to ldap, and delete the record from the change table when done.

 

So this is all transactional, nothing can slip by (since trigger), and nothing can happen wrong if ldap update fails.

 

Triggers are DB specific, and it requires different tables for each notification type.

 

For notifications, grouper could provide:

 

1.       If you have certain table structures for last_updated dates (perhaps with name prefixes to support multiple)

2.       We could populate with Java hooks (perhaps not reliable per discussion above), or you could do triggers (we could provide examples for certain DB’s) which would be more reliable and probably more performant

3.       We could provide a daemon (e.g. runs every 5 minutes) with callbacks that would process the change tables (based on name prefix), and gives you a callback to take the data and put it somewhere

4.       We could provide a scheduled way to get all data not just the diffs (e.g. daily or weekly or monthly to ensure things are in sync)

 

Just brainstorming here…

 

Thanks,

Chris