Grouper Users - Open Discussion List

[Tom Zeller <>]

I committed an example configuration project

 psp-example-grouper-uchicago

to

 
http://anonsvn.internet2.edu/viewvc/viewvc.py/i2mi/java-provisioning-provider/trunk/

Example configuration files are in

 src/test/resource

Example xml responses and ldif files are in

 src/test/resources/data

The junit test class is GrouperUChicagoTest.java in

 src/test/java/edu/...

I stored a ';' separated list of container OUs in a custom attribute
"Containers", and assigned this to 'groupA'

  // The 'Containers' attribute contains container DNs separated by a ';'.
  groupA.setAttribute("Containers", "ou=groups," + getLdapBaseDn() +
";ou=applications," + getLdapBaseDn());

so groupA is provisioned as

 dn: cn=Group A,ou=groups,dc=example,edu
 dn: cn=Group A,ou=applications,dc=example,dc=edu

A second group, groupB, will not have the "Containers" attribute, so
it will not be provisioned. But, how do you provision the dynamic
group member attribute ? Something like

 uid=test.subject.0,ou=people,dc=example,dc=edu
 isMemberOf: edu:groupA
 isMemberOf: edu:groupB
 memberOf: cn=Group A,ou=groups,dc=example,edu
 memberOf: cn=Group A,ou=applications,dc=example,dc=edu
 memberOf: Group B

?

On Thu, Apr 12, 2012 at 12:34 PM, Tom Zeller 
<>
 wrote:
> Colin,
>
> Sure, I'll help.
>
> I started a psp-example-grouper-uchicago project. Once a basic
> configuration is working, I can commit it to svn so you can give it a
> whirl.
>
> What are your target ldap directories ? (I think you mentioned Active 
> Directory)
>
> Let's get the non-realtime configuration working, and then figure out
> the change log.
>
> TomZ
>
> On Thu, Apr 12, 2012 at 11:49 AM, Colin Hudler 
> <>
>  wrote:
>> Just to reiterate, my goal is to have a single-instance provisioner that 
>> can
>> write groups to different and multiple LDAP container OUs. Very often there
>> is no static group (ou), yet the group name is written to a common member
>> attribute (dynamic groups).
>>
>> **We do have the ability to change how this happens at our site**
>>
>> If doing away with our custom Types and Attributes makes that possible,
>> UChicago is very much in favor of changing. Thus far, provisioning 
>> selection
>> has been in the hands of our users; if it becomes a server-side psp
>> configuration, we can probably accept the cost of it becoming an admin 
>> task.
>>
>> My most serious obstacle is groking PSP well enough to make it write groups
>> into multiple containers, or none at all, without individually configuring
>> each group (there's hundreds). Can you help with this?
>>
>> My ideal is to decorate the groups and/or stems with their intended
>> provisioning targets and parameters, but I understand if that is very hard.
>> Our group provisioning changes at least weekly, so whatever it is would be
>> as lite as possible. Way back when, I accomplished this with an automated
>> ldappc config generator which created beastly configurations for hundreds 
>> of
>> groups based on the registry Types & Attributes and wrangled several ldappc
>> instances together. It was slow, hard to debug, and almost impossible to 
>> use
>> for a realtime system.
>>
>>
>> On 04/12/2012 11:25 AM, Tom Zeller wrote:
>>>
>>> I don't remember, are old-style or new-style attributes cached by
>>> grouper ? with PIT auditing too ?
>>>
>>> If a group with 1,000 members is added to another group, there will be
>>> 1,000 change log add_membership entries. If a provisioner needs to
>>> query grouper for old-style or new-style attributes to determine
>>> provisioned target identifiers, that's 1,000 "find group" queries
>>> (cached) and then 1,000 "get old-style attribute" queries and/or 1,000
>>> "get new-style attribute" queries, correct ?
>>>
>>> If we decide that we need to use PIT, are the PIT "find group", "get
>>> old-style attribute", and "get new-style attribute" queries cached too
>>> ?
>>>
>>> Deleting a group whose provisioned target identifier is not solely
>>> based on the group name will require PIT queries as well.
>>>
>>> Maybe we should say we don't support real-time provisioning if
>>> provisioned target identifiers are not based only on the object
>>> (group, stem, member) name, excluding displayName.
>>>
>>> If attribute queries are cached by grouper, then maybe deployers will
>>> want to have a separate ehcache configuration for provisioning with
>>> longer timeouts than the UI.
>>>
>>> TomZ
>>>
>>> On Wed, Apr 11, 2012 at 3:29 PM, Chris 
>>> Hyzer<>
>>>  wrote:
>>>
>>>>>
>>>>> The latency I am concerned about is on the order of hours rather than
>>>>> minutes, for example, patch tuesday or holiday break maintenance when
>>>>> domain controllers might be down for many hours. So, I am not sure the
>>>>> 5 minute rule helps.
>>>>>
>>>>
>>>> Im saying if you have 2 queries in 5 minutes, then you could get from
>>>> cache.  If you query 2 hours later, the provisioning attributes would 
>>>> still
>>>> be there, I don't think it is a problem.
>>>>
>>>> Lets discuss this at the member meeting I guess...  maybe a solution is a
>>>> JSON 4k field we could pack data into, or with the new UI try to get 
>>>> people
>>>> off of the old-style attributes and then things would be in PIT...  Im 
>>>> still
>>>> not sure why this is a large risk, in general the metadata about
>>>> provisioning wouldn't change too often, it seems like an edge case which 
>>>> we
>>>> might be overengineering...
>>>>
>>>> Thanks,
>>>> Chris
>>>>
>>>> -----Original Message-----
>>>> From: 
>>>> 
>>>>  
>>>> [mailto:]
>>>>  On Behalf Of Tom
>>>> Zeller
>>>> Sent: Wednesday, April 11, 2012 4:15 PM
>>>> To: 
>>>> 
>>>> Cc: Colin Hudler
>>>> Subject: Re: [grouper-users] Replicating peculiar functionality with psp
>>>>
>>>> The latency I am concerned about is on the order of hours rather than
>>>> minutes, for example, patch tuesday or holiday break maintenance when
>>>> domain controllers might be down for many hours. So, I am not sure the
>>>> 5 minute rule helps.
>>>>
>>>> I understand the scaling issues, and I am also unsure what to cache
>>>> for each kind of change log event. For example, for a membership
>>>> change log event, do I query for the group, member, subject, and
>>>> membership objects and cache all of them ?
>>>>
>>>> The idea I am floating is for grouper to provide an attribute, e.g.
>>>> changeLogAttribute1, which if present on any object whose identifier
>>>> exists in a change log event, the values of the attribute would be
>>>> added to the change log event. So, UChicago would store the custom
>>>> ldap identifier and the target identifier in changeLogAttribute1. They
>>>> would need to use some sort of string separator, or xml, or whatever,
>>>> to concatenate two old-style attributes into a single new-style
>>>> attribute. If we have enough change log event columns available, we
>>>> could provide changeLogAttribute1, changeLogAttribute2, etc.
>>>>
>>>> When I first encountered userAttribute1 or customAttribute1 in active
>>>> directory or windows live I thought they were clumsy, but at least
>>>> folks might not be surprised.
>>>>
>>>> And, yeah, I think this is hacky but trying to cache a slice of the
>>>> grouper database for every change log event seems prone to performance
>>>> issues.
>>>>
>>>> Thoughts ?
>>>>
>>>> TomZ
>>>>
>>>> On Fri, Apr 6, 2012 at 6:07 PM, Chris 
>>>> Hyzer<>
>>>>  wrote:
>>>>
>>>>>
>>>>> I think we should document that changes in provisioning attributes take
>>>>> 5 minutes to propagate to the provisioner... you can do a second query 
>>>>> and
>>>>> cache for 5 minutes...
>>>>> Sound good?  :)
>>>>>
>>>>> I don't think we should add attributes to the change log since we would
>>>>> need to add for each type of query it will get complicated to configure 
>>>>> and
>>>>> what if someone needs more attributes for various provisioners than are
>>>>> available.  It doesn't really scale...
>>>>>
>>>>> Thanks,
>>>>> Chris
>>>>>
>>>>> -----Original Message-----
>>>>> From: 
>>>>> 
>>>>>  
>>>>> [mailto:]
>>>>>  On Behalf Of Tom
>>>>> Zeller
>>>>> Sent: Friday, April 06, 2012 12:05 AM
>>>>> To: Colin Hudler
>>>>> Cc: 
>>>>> 
>>>>> Subject: Re: [grouper-users] Replicating peculiar functionality with psp
>>>>>
>>>>> Here is one issue worth talking about.
>>>>>
>>>>> The current code for change log based provisioning does not handle the
>>>>> custom attributes which determine whether or not a group should be
>>>>> provisioned as well as the desired ldap dn.
>>>>>
>>>>> For the first attempt at change log based provisioning, I decided to
>>>>> try to provision based on change log entry data only without
>>>>> additional queries. An "add membership" change log entry does not
>>>>> contain the custom group attributes, so another query would be needed
>>>>> to return those attributes. Not a big deal, except ... I have been
>>>>> worried about phasing, meaning the potential time difference between
>>>>> when a change log entry is processed and when the action producing the
>>>>> change log entry occurred. I think I am alone in my concern.
>>>>>
>>>>> A potential solution is to use PIT (point-in-time) auditing to return
>>>>> the group as it was when the change log event occurred. However, if I
>>>>> remember correctly, I don't think that custom attributes are included
>>>>> in PIT auditing, but new attribute framework attributes are.
>>>>>
>>>>> In general, I prefer provisioning a change log entry without
>>>>> performing additional queries, meaning, a change log entry should
>>>>> contain all of the data necessary for provisioning.
>>>>>
>>>>> Maybe it could be possible to include custom attributes in change log
>>>>> entries ? A change log entry has capacity for 12 strings (columns),
>>>>> maybe deployers could specify additional group (or member or stem)
>>>>> attributes to be returned in addition to the group (or member or stem)
>>>>> name or id ?
>>>>>
>>>>> So, rather than the provisioner querying for additional data at the
>>>>> time of provisioning, perhaps grouper could include necessary
>>>>> additional data in the change log entry at the time the change log
>>>>> event occurred, which would alleviate my concerns regarding time
>>>>> delays.
>>>>>
>>>>> Whether or not folks use the psp for provisioning based on the change
>>>>> log, I think they will face similar issues, and customizing change log
>>>>> entries may help deployers regardless of their provisioning gear.
>>>>>
>>>>> This response is probably too long already, but it might be a good
>>>>> idea to standardize and perhaps provide a default configuration for
>>>>> attributes used to determine whether or not objects should be
>>>>> provisioned, as well as target identifiers.
>>>>>
>>>>> TomZ
>>>>>
>>>>> On Thu, Apr 5, 2012 at 6:39 PM, Tom 
>>>>> Zeller<>
>>>>>   wrote:
>>>>>
>>>>>>
>>>>>> Apologies for the delay in my response.
>>>>>>
>>>>>> In general, I think you will be ok, meaning : you should be able to do
>>>>>> what you want.
>>>>>>
>>>>>> It will take me a minute to parse your configuration, in the meantime,
>>>>>> could you help me understand  :
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Our static groups may not have a canonical groupDn, and often not any
>>>>>>> static
>>>>>>> group at all.
>>>>>>>
>>>>>>
>>>>>> I think you are saying that the ldap dn of a static group is user
>>>>>> specified, and stored as a group attribute in grouper. This seems
>>>>>> fine.
>>>>>>
>>>>>> And, regarding :
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> allows the user to also specify the DOMAIN where groups are stored (we
>>>>>>> do
>>>>>>> not always have them in the same domain as users).
>>>>>>>
>>>>>>
>>>>>> What is the impact of the user specified domain ? Does each domain
>>>>>> require an ldap connection ?
>>>>>>
>>>>>> Thanks for trying out the psp.
>>>>>>
>>>>>> TomZ
>>>>>>
>>>>>> On Wed, Apr 4, 2012 at 12:05 PM, Colin 
>>>>>> Hudler<>
>>>>>>  wrote:
>>>>>>
>>>>>>>
>>>>>>> We have custom provisioner to LDAP. Users are allowed to add a Group
>>>>>>> Type
>>>>>>> and attribute value that the provisioner looks for to enable the
>>>>>>> synchronization to LDAP. One of the attributes allows the user to
>>>>>>> write the
>>>>>>> group into one or many pre-defined LDAP containers (e.g,
>>>>>>> ou=groups,ou=groups,ou=application,...). The attribute value is a
>>>>>>> string
>>>>>>> containing DNs separated by semicolon.
>>>>>>>
>>>>>>> My approach so far is the goal of psp writing exactly the same
>>>>>>> static/dynamic groups as our custom sync. Success is measured by
>>>>>>> bulkCalc
>>>>>>> returning whatever is the equivalent of no-op. Alternatively, I can
>>>>>>> restructure how we use Types and Attributes. The immovable is our
>>>>>>> ability to
>>>>>>> direct static groups using Type/Attribute for one or more LDAP
>>>>>>> containers.
>>>>>>>
>>>>>>> I have been successful in plucking the groups in GroupDataConnector,
>>>>>>> AllGroupNamesConnector, et al, by using GroupExactAttribute filters.
>>>>>>> However, I wonder about the feasibility of the dynamic user-specified
>>>>>>> static
>>>>>>> group DNs. Besides that I cannot currently conceive of the correct
>>>>>>> psp-resolver configuration, I think this might be opposed to the
>>>>>>> design
>>>>>>> (please let me know). Will it cause problems with the groupDn
>>>>>>> attribute? Our
>>>>>>> static groups may not have a canonical groupDn, and often not any
>>>>>>> static
>>>>>>> group at all. Attached is my attempt at a psp-resolver and psp.xml
>>>>>>> configuration just for POC. I think the log message during bulkCalc
>>>>>>> "PSO
>>>>>>> Identifier Definition 'groupDn' - Source attribute 'groupDn' does not
>>>>>>> exist"
>>>>>>> points at least at part of my problem.
>>>>>>>
>>>>>>> As if that isn't enough, our custom AD synchronizer works similarly,
>>>>>>> except
>>>>>>> allows the user to also specify the DOMAIN where groups are stored (we
>>>>>>> do
>>>>>>> not always have them in the same domain as users). Perhaps I'll tackle
>>>>>>> this
>>>>>>> another time, but we also have user objects from a forest trust which
>>>>>>> are
>>>>>>> resolved by the provisioner (the group-member in AD must be a
>>>>>>> specially
>>>>>>> formatted foreign SID obtained from the other forest, not a vanilla
>>>>>>> local
>>>>>>> user object). WHAT A MESS!
>>>>>>>
>>>
>>>
>>
>>