Grouper Developers Forum

[Tom Zeller <>]

I agree that sources and targets will want to filter exposed data.
However, I'm thinking that push diffs are a good idea and will be
useful, perhaps I don't understand the details.

For example, a push diff such as "add member M to group G" received by
a target is more efficient than a push refresh/sync such as "group G's
membership changed" and/or "member M's membership changed". The target
doesn't have to diff, but still may want to, that's up to it. But you
know that already, I'm sure.

In terms of SPML, a provider (PSP) receives SPML requests and returns
SPML responses to a requester (RA). The target (PST) is never exposed
directly to a requester.

So, a source Grouper instance may act as a requestor and send changes
(SPML requests) to a PSP in front of the target Grouper. Ldappcng acts
as a PSP, but only currently supports an LDAP target - I need to write
the connector so that the PSP supports a Grouper target. I also need
to write Grouper's RA, which outputs SPML requests based on the
changelog. With these two pieces written, we can support
bi-directional communication, allowing deployers to prefer direction.

Probably a next step is to mockup the SPML request/responses for
federated provisioning based on the changelog.

And again, I don't want to *require* ldappcng in the middle of
federated Grouper provisioning, since sites may want to process the
SPML however they see fit.

Tom

On Mon, Aug 30, 2010 at 10:30 PM, Chris Hyzer 
<>
 wrote:
> Do you picture push diffs as popular in a federated (external) sense?  I 
> don't think the target system will want the source system to know the 
> current state of the data, there might be sensitive parts, right?  Maybe 
> the source sends the change to a daemon which resides on the target 
> somewhere, that can do a diff based on that somehow...  know what I mean?
>
> Thanks,
> Chris
>
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Tom Zeller
> Sent: Monday, August 30, 2010 6:01 PM
> To: 
> 
> Cc: Jim Fox
> Subject: Re: [grouper-dev] federated/provisioned groups mockup
>
>>>> Will the Target Federated Grouper's Grouper Connector make SPML requests
>>>> (a pull model), or will the source Grouper (gsh) make the SPML requests
>>>> for the Target Federated Grouper?
>>>>
>>>> TomB
>>>
>>> I think there are three options:
>>>
>>> 1. Diffs, sent via push, based on the change log
>>> 2. Periodic refreshes of groups, pushed by the source
>>> 3. Periodic refreshes of groups, pulled by the target
>>>
>>> I think we need diffs for real time, and we also need full refreshes.  If
>>> we already have the authn etc for #1, then we might as well do #2 instead 
>>> of
>>> #3, though it shouldn't really matter we can support all if people want
>>> it...  Note, I think we had discussed it before, but there is currently no
>>> way to pull diffs from WS...
>>>
>>> Chris
>>
>> We have something of a similar issue here at UW, where we need to
>> keep up-to-date several remote group systems, google groups, Windows
>> AD, and etc.  We chose to send updates (diffs) via JMS messaging
>> -- implemented with ActiveMQ.  This has several advantages over,
>> say, sending to a RESTful WS, which would be our other choice.
>> Most notably the remote site doesn't have to be up all the time,
>> so our change log processor can keep running whithout worrying what
>> to do with dropped notices.
>>
>> Haven't decided yet how to do the reconciliation.  It's not as
>> simple as a periodic refresh.  These are large and dynamic things.
>> There will necessarily be a lot of activity between the start and
>> end of a refresh operation.  One possibility we're considering
>> is a background, continuous reconciliation pushed by the source.
>> It would be appropriately throttled so as not to overload anyone.
>>
>> Jim
>
> Diffs and syncs/refreshes either push or pull seem reasonable. A
> persistent queue built-in (but optional) to ldappc might be nice to
> handle "unreliable messaging".
>
> An SPML interface which Grouper reads and writes to will allow
> deployers to choose their methodology (push/pull, etc). Ldappc
> shouldn't be required, but provides attribute massaging and filtering,
> ala shib.
>
> So, if Grouper writes changes/diffs in SPML, UW can pick those up and
> process them however they see fit.
>
> Large group sync/refresh/reconciliation is interesting from a
> performance perspective, mainly because most person based attributes
> have far less than thousands or millions of values (in my experience).
> Perhaps QoS or priority would help with target specific throttling.
> I've thought that diff operations should have a notion of TTL before a
> full sync is required. The complexity of these operations was noted on
> the PSTC call today, and detailed use cases would be helpful.
>
> TomZ
>