Grouper Developers Forum

[Tom Zeller <>]

So far, I have a wobbly proof-of-concept for real-time provisioning.
Here are some details and tasks :

By "real-time provisioning" I mean that changes to groups and
memberships are reflected to provisioned targets after the
corresponding changes are made in Grouper with little or no humanly
measurable delay. Of course, I need to assume that provisioned targets
have very little latency. In other words, "real-time" = fast.

By "incremental provisioning", I mean the provisioning of a single
membership. Currently, ldappc[ng] provisions groups in their entirety,
with performance ramifications for groups with a large number of
members, where the definition of "large" depends upon a deployer's
infrastructure. In my testing, I consider large to be a group with
100,000 members. I do not equate incremental provisioning with the
provisioning of a single value of a possibly multi-valued attribute -
this just is not in the SPMLv2 specification.

Incremental provisioning is a step towards real-time provisioning,
since provisioning "add member M to group G" should be faster than
provisioning all of group G's members. However, when creating a group
for the first time with a large number of members, incremental
provisioning is likely slower than provisioning a group in its
entirety.

Real-time provisioning will be based on the Grouper changelog. I have
begun work on a ChangeLogConsumer for ldappcng and a
ChangeLogDataConnector for the Shibboleth AttributeResolver.

Real-time provisioning will also be based on representing a membership
as an additional object in the ldappcng configuration. Currently,
there are typically 3 objects in the ldappcng configuration: stem,
group, and member. A fourth object will be a membership object.

The current style of ldappc[ng] provisioning has been called "full
sync" or "bulkSync". When ldappc[ng] starts up, it gathers all
identifiers from sources, as well as all identifiers from provisioned
objects, performs a diff, and then provisions the resultant changes.
On subsequent ldappc[ng] runs, only changed objects are considered for
provisioning. New in 2.0 will be an option to perform full/bulkSyncs
at a configurable interval.

In order to perform full/bulkSyncs, ldappcng needs to gather the
identifiers of all objects in sources (e.g. Grouper), which I
implemented by extending data connectors to provide
getAllIdentifiers() and getAllIdentifiers(Date updatedSince). This is
more of a kludge than a hack. I will replace this not-well thought out
extension with new data connectors which will return all identifiers
for their corresponding objects, for example,
GetAllGroupsInStemDataConnector. Since we may still desire to
provision all objects which have been changed since a particular time,
I am looking at extending AttributeQuery to include the last
modification time, which Grouper data connectors will need to know how
to handle.

And, the LdapTargetProvider will need to support the SPMLv2
hasReference QueryType to support the incremental provisioning of
memberships.

So, here is an abbreviated task list :

- extend AttributeQuery to provide a last modify time for subsequent
ldappcng runs
- remove the SourceDataConnector interface
- provide "find all" data connectors for groups, stems, and members.
- implement a ChangeLogDataConnector and ChangeLogConsumer
- support hasReference QueryType in the LdapTargetProvider

I think it is unlikely, barring an act of <insert a belief system
here>, that this will be written and tested before the 2.0 code freeze
on Monday May 16. Perhaps I will find time during the upcoming
jasig/acamp meeting to do some coding as I did during the last I2MM.
At the least, there are fairly significant performance improvements to
ldappcng based on caching that should be helpful for folks for 2.0.0,
if real-time provisioning needs to wait for 2.0.1.

TomZ