Grouper Developers Forum

[Chris Hyzer <>]

Do you picture push diffs as popular in a federated (external) sense?  I 
don't think the target system will want the source system to know the current 
state of the data, there might be sensitive parts, right?  Maybe the source 
sends the change to a daemon which resides on the target somewhere, that can 
do a diff based on that somehow...  know what I mean?

Thanks,
Chris

-----Original Message-----
From: 

 
[mailto:]
 On Behalf Of Tom Zeller
Sent: Monday, August 30, 2010 6:01 PM
To: 

Cc: Jim Fox
Subject: Re: [grouper-dev] federated/provisioned groups mockup

>>> Will the Target Federated Grouper's Grouper Connector make SPML requests
>>> (a pull model), or will the source Grouper (gsh) make the SPML requests
>>> for the Target Federated Grouper?
>>>
>>> TomB
>>
>> I think there are three options:
>>
>> 1. Diffs, sent via push, based on the change log
>> 2. Periodic refreshes of groups, pushed by the source
>> 3. Periodic refreshes of groups, pulled by the target
>>
>> I think we need diffs for real time, and we also need full refreshes.  If
>> we already have the authn etc for #1, then we might as well do #2 instead 
>> of
>> #3, though it shouldn't really matter we can support all if people want
>> it...  Note, I think we had discussed it before, but there is currently no
>> way to pull diffs from WS...
>>
>> Chris
>
> We have something of a similar issue here at UW, where we need to
> keep up-to-date several remote group systems, google groups, Windows
> AD, and etc.  We chose to send updates (diffs) via JMS messaging
> -- implemented with ActiveMQ.  This has several advantages over,
> say, sending to a RESTful WS, which would be our other choice.
> Most notably the remote site doesn't have to be up all the time,
> so our change log processor can keep running whithout worrying what
> to do with dropped notices.
>
> Haven't decided yet how to do the reconciliation.  It's not as
> simple as a periodic refresh.  These are large and dynamic things.
> There will necessarily be a lot of activity between the start and
> end of a refresh operation.  One possibility we're considering
> is a background, continuous reconciliation pushed by the source.
> It would be appropriately throttled so as not to overload anyone.
>
> Jim

Diffs and syncs/refreshes either push or pull seem reasonable. A
persistent queue built-in (but optional) to ldappc might be nice to
handle "unreliable messaging".

An SPML interface which Grouper reads and writes to will allow
deployers to choose their methodology (push/pull, etc). Ldappc
shouldn't be required, but provides attribute massaging and filtering,
ala shib.

So, if Grouper writes changes/diffs in SPML, UW can pick those up and
process them however they see fit.

Large group sync/refresh/reconciliation is interesting from a
performance perspective, mainly because most person based attributes
have far less than thousands or millions of values (in my experience).
Perhaps QoS or priority would help with target specific throttling.
I've thought that diff operations should have a notion of TTL before a
full sync is required. The complexity of these operations was noted on
the PSTC call today, and detailed use cases would be helpful.

TomZ