Skip to Content.
Sympa Menu

grouper-dev - [grouper-dev] Re: [ldappcng] incremental/real-time provisioning : implementation tasks

Subject: Grouper Developers Forum

List archive

[grouper-dev] Re: [ldappcng] incremental/real-time provisioning : implementation tasks

Chronological Thread 
  • From: Tom Zeller <>
  • To: Grouper Dev <>
  • Subject: [grouper-dev] Re: [ldappcng] incremental/real-time provisioning : implementation tasks
  • Date: Tue, 10 May 2011 15:51:21 -0500
  • Domainkey-signature: a=rsa-sha1; c=nofws;; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:content-type; b=mq9Q51DUYP85xoNboLA9MhefNjn0B/iNuJ19hO9kO4vhxPpr/uDyYwsCWOw5mXpaUG wpQEVFBPHv7y7KVM+NXTMo2GNUBPec+HIf3bt3FL8PRMLp9n6+13cqeRby+sUIJTspYw CP9+Ac5RjXhMEeMvZrzi/yuQE6kNscHdg2jGc=

FYI, for housekeeping, I will track progress on implementing
incremental provisioning on the jira issue itself.


On Tue, May 10, 2011 at 12:13 AM, Tom Zeller
> So far, I have a wobbly proof-of-concept for real-time provisioning.
> Here are some details and tasks :
> By "real-time provisioning" I mean that changes to groups and
> memberships are reflected to provisioned targets after the
> corresponding changes are made in Grouper with little or no humanly
> measurable delay. Of course, I need to assume that provisioned targets
> have very little latency. In other words, "real-time" = fast.
> By "incremental provisioning", I mean the provisioning of a single
> membership. Currently, ldappc[ng] provisions groups in their entirety,
> with performance ramifications for groups with a large number of
> members, where the definition of "large" depends upon a deployer's
> infrastructure. In my testing, I consider large to be a group with
> 100,000 members. I do not equate incremental provisioning with the
> provisioning of a single value of a possibly multi-valued attribute -
> this just is not in the SPMLv2 specification.
> Incremental provisioning is a step towards real-time provisioning,
> since provisioning "add member M to group G" should be faster than
> provisioning all of group G's members. However, when creating a group
> for the first time with a large number of members, incremental
> provisioning is likely slower than provisioning a group in its
> entirety.
> Real-time provisioning will be based on the Grouper changelog. I have
> begun work on a ChangeLogConsumer for ldappcng and a
> ChangeLogDataConnector for the Shibboleth AttributeResolver.
> Real-time provisioning will also be based on representing a membership
> as an additional object in the ldappcng configuration. Currently,
> there are typically 3 objects in the ldappcng configuration: stem,
> group, and member. A fourth object will be a membership object.
> The current style of ldappc[ng] provisioning has been called "full
> sync" or "bulkSync". When ldappc[ng] starts up, it gathers all
> identifiers from sources, as well as all identifiers from provisioned
> objects, performs a diff, and then provisions the resultant changes.
> On subsequent ldappc[ng] runs, only changed objects are considered for
> provisioning. New in 2.0 will be an option to perform full/bulkSyncs
> at a configurable interval.
> In order to perform full/bulkSyncs, ldappcng needs to gather the
> identifiers of all objects in sources (e.g. Grouper), which I
> implemented by extending data connectors to provide
> getAllIdentifiers() and getAllIdentifiers(Date updatedSince). This is
> more of a kludge than a hack. I will replace this not-well thought out
> extension with new data connectors which will return all identifiers
> for their corresponding objects, for example,
> GetAllGroupsInStemDataConnector. Since we may still desire to
> provision all objects which have been changed since a particular time,
> I am looking at extending AttributeQuery to include the last
> modification time, which Grouper data connectors will need to know how
> to handle.
> And, the LdapTargetProvider will need to support the SPMLv2
> hasReference QueryType to support the incremental provisioning of
> memberships.
> So, here is an abbreviated task list :
> - extend AttributeQuery to provide a last modify time for subsequent
> ldappcng runs
> - remove the SourceDataConnector interface
> - provide "find all" data connectors for groups, stems, and members.
> - implement a ChangeLogDataConnector and ChangeLogConsumer
> - support hasReference QueryType in the LdapTargetProvider
> I think it is unlikely, barring an act of <insert a belief system
> here>, that this will be written and tested before the 2.0 code freeze
> on Monday May 16. Perhaps I will find time during the upcoming
> jasig/acamp meeting to do some coding as I did during the last I2MM.
> At the least, there are fairly significant performance improvements to
> ldappcng based on caching that should be helpful for folks for 2.0.0,
> if real-time provisioning needs to wait for 2.0.1.
> TomZ

Archive powered by MHonArc 2.6.16.

Top of Page