Grouper Developers Forum

["GW Brown, Information Systems and Computing" <>]

--On 27 May 2008 16:30 -0500 Tom Barton 
<>
 wrote:

> Catching up, reflecting, and prepping for our call tomorrow leads me to
> the following thoughts:
>
> BertBL and MRG remind us to remain pragmatic and stay focused on solving
> actual rather than hypothetical problems. Amen.
>
> I know of actual use cases for veto (Bert, is that what you mean by
> "filter"?) and for notification (including notification to an audit log).
> I don't think I know any, as yet, for plug-ins to enhance events. Nor do
> I see why any such uses would have to be tightly-coupled to the
> originating event, in the same transaction.
I would like the ability, when a group is created, to set some group 
attribute values (and possibly add some group types) and have those be part 
of the transaction. New values would depend on existing values and location 
in the hierarchy - but you don’t need to know the details...

The current way of using lots of individual API calls to build a group 
isn’t very satisfactory in this respect. Ideally these types of changes 
could be effected using new single API methods i.e.

Stem.addChildGroup(String extn, String displayExtn, Set types, Map 
attributes)

Group.setAttributes(Map attributes)
Group.setAttributes(Set newTypes, Map attributes)
Group.setAttributes(Set newTypes, Map attributes, Set removeTypes);

Similarly, for adding / removing members or privilegees:

Group.addMembers(Set subjects)
Group.addMembers(Set subjects,Set subjectsToRemove)

Rather than generating lots of ‘small’ changes, a single ‘large’ change can 
be vetoed, amended* or notified). I can’t sensibly choose to veto 
type/attribute changes to a group without knowing what all the changes are 
- and currently when you make one change the API has no knowledge of any 
API calls you intend to make in the next lines of code.

*By making the Sets and Maps available to a pre-hook it could modify them 
before the API effects the changes. I wouldn’t worry about earlier hooks 
having the chance to examine the changes.

> Should we leave custom
> enhancement capability out of the hooks architecture, for the time being?
> Would that also make moot special considerations regarding asynchronism &
> loops?
>
> Regarding reliability of notification, I don't want us to build a
> reliable asynchronous messaging system. If reliable async messaging is
> needed in Grouper, I want to rely on 3rd parties for it.
>
> However, I *do* want reliable audit history. And I don't want that to
> depend on a separate process calling back to the DB for the details on
> what changed, in case it may have changed again in the meanwhile. I'd
> like instead to have a logical description of each completed transaction
> preserved somewhere in the grouper DB when the transaction completes.
> This history-of-changes table could have a configurable depth (time or
> size) and would necessarily serialize all transactions, thus also
> providing the basis for reliable notification by 3rd party
> implementations. The hard part of this is deciding what the "logical
> descriptions" are - it's not which columns in which tables have what new
> values. Think of LDIF as a motivating example. The grouper event log, at
> INFO level, is probably capturing the right events, but not with the
> right schema.
So are we looking at tables with:

date, transaction_id, member_id

transaction_id, object_id, object_type, change*

*where change is a clob, or similar, which has a representation of changes 
to the object.

New API methods would allow a client to query for all changes since a 
certain date (filterable by object type, and optional object_id) and would 
return consolidated changes for each object changed.

The devil would be in the detail of how you represent the changes - they 
need to be easily machine readable. Also, do we ‘log’ effective changes?
> Regarding how grouper & signet hooks implementations should or shouldn't
> be alike, I think that they can share a common "contract" with plug-in
> developers, but implementation within each respective API will be
> particular to each.
Ideally, configuration would be similar..
> Tom
>
> Chris Hyzer wrote:
> > Let me try to summarize how the discussion on hooks / notifications
> > went today.
> >
> > First of all there will be some differences between Grouper and
> > Signet with hooks.  E.g. the pre and post in Grouper are in the same
> > transaction, and not so in Signet (unless and until long running
> > transactions are implemented in signet).
> >
> > In general we would like to start out with hooks and notifications by
> > designing and implementing a straightforward system, and then augment
> > it later as needs arise.  Here are some examples:
> >
> > 1. Toss out the reliable notification mechanism for now.
> > Implementers of hooks would need to write their own notification
> > mechanism (could be reliable in Grouper, could not be reliable in
> > Signet due to lack of long running transactions).  We can put that in
> > the API later since having it would save a lot of work on the
> > implementer.
> >
> > 2. Have only one hook implementation per event.  The side effect here
> > is that instead of registering multiple listeners like this:
> >
> > GrouperHooks.registerGroupAddPreListener(AddAuditing.class);
> > GrouperHooks.registerGroupAddPreListever(CheckGroupName.class);
> >
> > It would be in the listener itself
> >
> > public void preGroupAdd(HookPreGroupAddContext context) {
> >   AddAuditing.preGroupAdd(context);
> >   CheckGroupName.preGroupAdd(context);
> > }
> >
> > Since this doesn't save much work on the part of the implementer,
> > they can worry about the order, what happens when one fails, etc
> >
> > 3. Since only one hook implementation per event type, we could give
> > an API to make the asynchronous call very easy...
> >
> > Instead of
> >
> > public class AddAuditing implement AsynchronousHook ...
> >
> > You would just call with an API method of some sort that passes data
> > correctly etc, e.g.
> >
> > public void preGroupAdd(HookPreGroupAddContext context) {
> >   HooksUtils.asynchronous(context, new AsynchronousHandler() {
> >     public void hookLogic(HooksContext asyncContext) {
> >       AddAuditing.preGroupAdd((HookPreGroupAddContext)asyncContext);
> >     }});
> > }
> >
> > We are talking about 3 lines of code the implementer needs to add to
> > make all or part of the hook asynchronous.
> >
> > If everyone is all good with all of this, we need to decide what is
> > in Grouper/Signet and what is in i2mi-common... I think most of the work
> > involved is identifying exactly where the hook points are, and what data
> > goes into the context, and coding the contexts. This is all specific to
> > Grouper or Signet, so I don't think we need anything in i2mi-common at
> > this point... and that would make development easier and faster. But we
> > can discuss... :) Maybe another point that Grouper needs is vetoing back
> > via exception since we don't want to affect our deeply nested API much
> > by passing objects around.
> >
> > Kind regards,
> > Chris
> >
> > Ps. I said I would give an example of a long running transaction in
> > Grouper, see this page:
> >
> > https://wiki.internet2.edu/confluence/display/GrouperWG/Hibernate+and+da
> > ta+layer+updates#Hibernateanddatalayerupdates-Addsupportfortransactions
> >
> > The example is:
> >
> >     //note the default is READ_WRITE_OR_USE_EXISTING
> >     GrouperTransaction.callbackGrouperTransaction(new
> >     GrouperTransactionHandler() {
> >
> >       public Object callback(GrouperTransaction grouperTransaction)
> >           throws GrouperDAOException {
> >
> >         try {
> >           Group.saveGroup(rootSession,
> >           null,null,groupName,displayExtension,  groupDescription,
> >               SaveMode.INSERT, false);
> >
> >           Group.saveGroup(rootSession, null,null,
> >           groupName2,displayExtension, groupDescription,
> >               SaveMode.INSERT, false);
> >
> >           //I believe this has an implicit commit, but the user could
> >           also do it explicitly or //with the jdbc Connection
> >           return null;
> >         } catch (Exception e) {
> >           //note, if specific exceptions need to be handled, do that here
> >           throw new RuntimeException(e);
> >         }
> >       }
> >
> >     });
> >
> > > -----Original Message-----
> > > From: Chris Hyzer
> > > Sent: Thursday, May 22, 2008 5:18 PM
> > > To: 'Michael R. Gettes'; Mike Olive
> > > Cc: 'Grouper Dev'; 
> > >
> > > Subject: RE: [signet-dev] RE: [grouper-dev] notifications vs hooks
> > >
> > > > Based on this response I'm not sure you understand my original
> > > > argument.
> > > Im not sure I completely understood it either, but if it is about db
> > > triggers, I think it is a moot point if we understand it.  Now instead
> > > of being in the weeds discussing how we will implement, we are
> > > discussing how we wont implement.  :)  I don't think we can use
> > > triggers, though I see how they do have some advantages...
> > >
> > > What do people think about my current idea:
> > >
> > > 1. There are no more "multiple event" registration, since per our last
> > > call it gets complicated what order they go in, how they affect each
> > > other, etc
> > > 2. The API can put code in hooks/notifications, even though it is not
> > > registered and implemented like the custom hooks (so we don't need
> > > event queue)
> > > 3. There is one place to register hook code in pre/post of an
> > > operation, and it can be transactional or not or mixture, your choice
> > > (well, this is free for grouper, signet might need to implement long
> > > running transaction to benefit from this... maybe to start signet is
> > > not transactional, I don't know what is easy there)
> > > 4. Code a class for hook code (extend a base class), and implement the
> > > methods you need.  Register in config file.  If you need multiple, just
> > > put multiple things in your method.  You can control order,
> > > dependencies, etc.  I picture a handful of hook classes organized by
> > > api objects (groups, folders, memberships, etc)
> > > 5. Veto by throwing specific vetoexceptions which are unchecked (in
> > > hook, not in notification).
> > > 6. Hooks are synchronous, notifications are asynchronous.  We will
> > > document and provide helper methods for doing asynchronous hooks easily
> > > and safely (I think most asynchronous activity will be notifications
> > > anyway)
> > > 7. In the post-hook, you can call a method to register interest in a
> > > notification.  You can do this based on the data that is changing hands
> > > (e.g. if you want notification on membership change, but not security
> > > membership change).  This will put data in a notification queue table
> > > transactionally (e.g. type of data, id of data, type of callback
> > > (custom), timestamp, type of change (insert/update/delete), status,
> > > errorMessage (if notification fails) etc.
> > > 8. We have a daemon which runs every so configurably often to get
> > > records off the notification queue, call your callback, if success,
> > > remove from queue, if not, use a configurable or customizable retry
> > > algorithm.  It will hand you the ID and type of what has changed and
> > > you can do what you want it (lookup the data, lookup a custom db view,
> > > call a stored proc that you write, whatever).  The notification daemon
> > > would be one-to-one with the DB.  Whereas the hooks would be installed
> > > in each grouper/signet API (e.g. GSH, UI, WS, loader, etc).
> > >
> > > I think this design covers many use cases, but is simple and easy to
> > > implement.  The things that are cut out (sync/async, multiple logic per
> > > hook, etc) can still be accomplished with this design, and based on
> > > need we can focus on specific areas in future releases if we like...
> > >
> > > Thoughts?
> > > Thanks,
> > > Chris



----------------------
GW Brown, Information Systems and Computing