Grouper Developers Forum

[Tom Zeller <>]

FYI, for housekeeping, I will track progress on implementing
incremental provisioning on the jira issue itself.

[1] https://bugs.internet2.edu/jira/browse/GRP-592

On Tue, May 10, 2011 at 12:13 AM, Tom Zeller 
<>
 wrote:
> So far, I have a wobbly proof-of-concept for real-time provisioning.
> Here are some details and tasks :
>
> By "real-time provisioning" I mean that changes to groups and
> memberships are reflected to provisioned targets after the
> corresponding changes are made in Grouper with little or no humanly
> measurable delay. Of course, I need to assume that provisioned targets
> have very little latency. In other words, "real-time" = fast.
>
> By "incremental provisioning", I mean the provisioning of a single
> membership. Currently, ldappc[ng] provisions groups in their entirety,
> with performance ramifications for groups with a large number of
> members, where the definition of "large" depends upon a deployer's
> infrastructure. In my testing, I consider large to be a group with
> 100,000 members. I do not equate incremental provisioning with the
> provisioning of a single value of a possibly multi-valued attribute -
> this just is not in the SPMLv2 specification.
>
> Incremental provisioning is a step towards real-time provisioning,
> since provisioning "add member M to group G" should be faster than
> provisioning all of group G's members. However, when creating a group
> for the first time with a large number of members, incremental
> provisioning is likely slower than provisioning a group in its
> entirety.
>
> Real-time provisioning will be based on the Grouper changelog. I have
> begun work on a ChangeLogConsumer for ldappcng and a
> ChangeLogDataConnector for the Shibboleth AttributeResolver.
>
> Real-time provisioning will also be based on representing a membership
> as an additional object in the ldappcng configuration. Currently,
> there are typically 3 objects in the ldappcng configuration: stem,
> group, and member. A fourth object will be a membership object.
>
> The current style of ldappc[ng] provisioning has been called "full
> sync" or "bulkSync". When ldappc[ng] starts up, it gathers all
> identifiers from sources, as well as all identifiers from provisioned
> objects, performs a diff, and then provisions the resultant changes.
> On subsequent ldappc[ng] runs, only changed objects are considered for
> provisioning. New in 2.0 will be an option to perform full/bulkSyncs
> at a configurable interval.
>
> In order to perform full/bulkSyncs, ldappcng needs to gather the
> identifiers of all objects in sources (e.g. Grouper), which I
> implemented by extending data connectors to provide
> getAllIdentifiers() and getAllIdentifiers(Date updatedSince). This is
> more of a kludge than a hack. I will replace this not-well thought out
> extension with new data connectors which will return all identifiers
> for their corresponding objects, for example,
> GetAllGroupsInStemDataConnector. Since we may still desire to
> provision all objects which have been changed since a particular time,
> I am looking at extending AttributeQuery to include the last
> modification time, which Grouper data connectors will need to know how
> to handle.
>
> And, the LdapTargetProvider will need to support the SPMLv2
> hasReference QueryType to support the incremental provisioning of
> memberships.
>
> So, here is an abbreviated task list :
>
> - extend AttributeQuery to provide a last modify time for subsequent
> ldappcng runs
> - remove the SourceDataConnector interface
> - provide "find all" data connectors for groups, stems, and members.
> - implement a ChangeLogDataConnector and ChangeLogConsumer
> - support hasReference QueryType in the LdapTargetProvider
>
> I think it is unlikely, barring an act of <insert a belief system
> here>, that this will be written and tested before the 2.0 code freeze
> on Monday May 16. Perhaps I will find time during the upcoming
> jasig/acamp meeting to do some coding as I did during the last I2MM.
> At the least, there are fairly significant performance improvements to
> ldappcng based on caching that should be helpful for folks for 2.0.0,
> if real-time provisioning needs to wait for 2.0.1.
>
> TomZ
>