Grouper Users - Open Discussion List

[Tom Zeller <>]

Colin,

Sure, I'll help.

I started a psp-example-grouper-uchicago project. Once a basic
configuration is working, I can commit it to svn so you can give it a
whirl.

What are your target ldap directories ? (I think you mentioned Active 
Directory)

Let's get the non-realtime configuration working, and then figure out
the change log.

TomZ

On Thu, Apr 12, 2012 at 11:49 AM, Colin Hudler 
<>
 wrote:
> Just to reiterate, my goal is to have a single-instance provisioner that can
> write groups to different and multiple LDAP container OUs. Very often there
> is no static group (ou), yet the group name is written to a common member
> attribute (dynamic groups).
>
> **We do have the ability to change how this happens at our site**
>
> If doing away with our custom Types and Attributes makes that possible,
> UChicago is very much in favor of changing. Thus far, provisioning selection
> has been in the hands of our users; if it becomes a server-side psp
> configuration, we can probably accept the cost of it becoming an admin task.
>
> My most serious obstacle is groking PSP well enough to make it write groups
> into multiple containers, or none at all, without individually configuring
> each group (there's hundreds). Can you help with this?
>
> My ideal is to decorate the groups and/or stems with their intended
> provisioning targets and parameters, but I understand if that is very hard.
> Our group provisioning changes at least weekly, so whatever it is would be
> as lite as possible. Way back when, I accomplished this with an automated
> ldappc config generator which created beastly configurations for hundreds of
> groups based on the registry Types & Attributes and wrangled several ldappc
> instances together. It was slow, hard to debug, and almost impossible to use
> for a realtime system.
>
>
> On 04/12/2012 11:25 AM, Tom Zeller wrote:
>>
>> I don't remember, are old-style or new-style attributes cached by
>> grouper ? with PIT auditing too ?
>>
>> If a group with 1,000 members is added to another group, there will be
>> 1,000 change log add_membership entries. If a provisioner needs to
>> query grouper for old-style or new-style attributes to determine
>> provisioned target identifiers, that's 1,000 "find group" queries
>> (cached) and then 1,000 "get old-style attribute" queries and/or 1,000
>> "get new-style attribute" queries, correct ?
>>
>> If we decide that we need to use PIT, are the PIT "find group", "get
>> old-style attribute", and "get new-style attribute" queries cached too
>> ?
>>
>> Deleting a group whose provisioned target identifier is not solely
>> based on the group name will require PIT queries as well.
>>
>> Maybe we should say we don't support real-time provisioning if
>> provisioned target identifiers are not based only on the object
>> (group, stem, member) name, excluding displayName.
>>
>> If attribute queries are cached by grouper, then maybe deployers will
>> want to have a separate ehcache configuration for provisioning with
>> longer timeouts than the UI.
>>
>> TomZ
>>
>> On Wed, Apr 11, 2012 at 3:29 PM, Chris 
>> Hyzer<>
>>  wrote:
>>
>>>>
>>>> The latency I am concerned about is on the order of hours rather than
>>>> minutes, for example, patch tuesday or holiday break maintenance when
>>>> domain controllers might be down for many hours. So, I am not sure the
>>>> 5 minute rule helps.
>>>>
>>>
>>> Im saying if you have 2 queries in 5 minutes, then you could get from
>>> cache.  If you query 2 hours later, the provisioning attributes would 
>>> still
>>> be there, I don't think it is a problem.
>>>
>>> Lets discuss this at the member meeting I guess...  maybe a solution is a
>>> JSON 4k field we could pack data into, or with the new UI try to get 
>>> people
>>> off of the old-style attributes and then things would be in PIT...  Im 
>>> still
>>> not sure why this is a large risk, in general the metadata about
>>> provisioning wouldn't change too often, it seems like an edge case which 
>>> we
>>> might be overengineering...
>>>
>>> Thanks,
>>> Chris
>>>
>>> -----Original Message-----
>>> From: 
>>> 
>>>  
>>> [mailto:]
>>>  On Behalf Of Tom
>>> Zeller
>>> Sent: Wednesday, April 11, 2012 4:15 PM
>>> To: 
>>> 
>>> Cc: Colin Hudler
>>> Subject: Re: [grouper-users] Replicating peculiar functionality with psp
>>>
>>> The latency I am concerned about is on the order of hours rather than
>>> minutes, for example, patch tuesday or holiday break maintenance when
>>> domain controllers might be down for many hours. So, I am not sure the
>>> 5 minute rule helps.
>>>
>>> I understand the scaling issues, and I am also unsure what to cache
>>> for each kind of change log event. For example, for a membership
>>> change log event, do I query for the group, member, subject, and
>>> membership objects and cache all of them ?
>>>
>>> The idea I am floating is for grouper to provide an attribute, e.g.
>>> changeLogAttribute1, which if present on any object whose identifier
>>> exists in a change log event, the values of the attribute would be
>>> added to the change log event. So, UChicago would store the custom
>>> ldap identifier and the target identifier in changeLogAttribute1. They
>>> would need to use some sort of string separator, or xml, or whatever,
>>> to concatenate two old-style attributes into a single new-style
>>> attribute. If we have enough change log event columns available, we
>>> could provide changeLogAttribute1, changeLogAttribute2, etc.
>>>
>>> When I first encountered userAttribute1 or customAttribute1 in active
>>> directory or windows live I thought they were clumsy, but at least
>>> folks might not be surprised.
>>>
>>> And, yeah, I think this is hacky but trying to cache a slice of the
>>> grouper database for every change log event seems prone to performance
>>> issues.
>>>
>>> Thoughts ?
>>>
>>> TomZ
>>>
>>> On Fri, Apr 6, 2012 at 6:07 PM, Chris 
>>> Hyzer<>
>>>  wrote:
>>>
>>>>
>>>> I think we should document that changes in provisioning attributes take
>>>> 5 minutes to propagate to the provisioner... you can do a second query 
>>>> and
>>>> cache for 5 minutes...
>>>> Sound good?  :)
>>>>
>>>> I don't think we should add attributes to the change log since we would
>>>> need to add for each type of query it will get complicated to configure 
>>>> and
>>>> what if someone needs more attributes for various provisioners than are
>>>> available.  It doesn't really scale...
>>>>
>>>> Thanks,
>>>> Chris
>>>>
>>>> -----Original Message-----
>>>> From: 
>>>> 
>>>>  
>>>> [mailto:]
>>>>  On Behalf Of Tom
>>>> Zeller
>>>> Sent: Friday, April 06, 2012 12:05 AM
>>>> To: Colin Hudler
>>>> Cc: 
>>>> 
>>>> Subject: Re: [grouper-users] Replicating peculiar functionality with psp
>>>>
>>>> Here is one issue worth talking about.
>>>>
>>>> The current code for change log based provisioning does not handle the
>>>> custom attributes which determine whether or not a group should be
>>>> provisioned as well as the desired ldap dn.
>>>>
>>>> For the first attempt at change log based provisioning, I decided to
>>>> try to provision based on change log entry data only without
>>>> additional queries. An "add membership" change log entry does not
>>>> contain the custom group attributes, so another query would be needed
>>>> to return those attributes. Not a big deal, except ... I have been
>>>> worried about phasing, meaning the potential time difference between
>>>> when a change log entry is processed and when the action producing the
>>>> change log entry occurred. I think I am alone in my concern.
>>>>
>>>> A potential solution is to use PIT (point-in-time) auditing to return
>>>> the group as it was when the change log event occurred. However, if I
>>>> remember correctly, I don't think that custom attributes are included
>>>> in PIT auditing, but new attribute framework attributes are.
>>>>
>>>> In general, I prefer provisioning a change log entry without
>>>> performing additional queries, meaning, a change log entry should
>>>> contain all of the data necessary for provisioning.
>>>>
>>>> Maybe it could be possible to include custom attributes in change log
>>>> entries ? A change log entry has capacity for 12 strings (columns),
>>>> maybe deployers could specify additional group (or member or stem)
>>>> attributes to be returned in addition to the group (or member or stem)
>>>> name or id ?
>>>>
>>>> So, rather than the provisioner querying for additional data at the
>>>> time of provisioning, perhaps grouper could include necessary
>>>> additional data in the change log entry at the time the change log
>>>> event occurred, which would alleviate my concerns regarding time
>>>> delays.
>>>>
>>>> Whether or not folks use the psp for provisioning based on the change
>>>> log, I think they will face similar issues, and customizing change log
>>>> entries may help deployers regardless of their provisioning gear.
>>>>
>>>> This response is probably too long already, but it might be a good
>>>> idea to standardize and perhaps provide a default configuration for
>>>> attributes used to determine whether or not objects should be
>>>> provisioned, as well as target identifiers.
>>>>
>>>> TomZ
>>>>
>>>> On Thu, Apr 5, 2012 at 6:39 PM, Tom 
>>>> Zeller<>
>>>>   wrote:
>>>>
>>>>>
>>>>> Apologies for the delay in my response.
>>>>>
>>>>> In general, I think you will be ok, meaning : you should be able to do
>>>>> what you want.
>>>>>
>>>>> It will take me a minute to parse your configuration, in the meantime,
>>>>> could you help me understand  :
>>>>>
>>>>>
>>>>>>
>>>>>> Our static groups may not have a canonical groupDn, and often not any
>>>>>> static
>>>>>> group at all.
>>>>>>
>>>>>
>>>>> I think you are saying that the ldap dn of a static group is user
>>>>> specified, and stored as a group attribute in grouper. This seems
>>>>> fine.
>>>>>
>>>>> And, regarding :
>>>>>
>>>>>
>>>>>>
>>>>>> allows the user to also specify the DOMAIN where groups are stored (we
>>>>>> do
>>>>>> not always have them in the same domain as users).
>>>>>>
>>>>>
>>>>> What is the impact of the user specified domain ? Does each domain
>>>>> require an ldap connection ?
>>>>>
>>>>> Thanks for trying out the psp.
>>>>>
>>>>> TomZ
>>>>>
>>>>> On Wed, Apr 4, 2012 at 12:05 PM, Colin 
>>>>> Hudler<>
>>>>>  wrote:
>>>>>
>>>>>>
>>>>>> We have custom provisioner to LDAP. Users are allowed to add a Group
>>>>>> Type
>>>>>> and attribute value that the provisioner looks for to enable the
>>>>>> synchronization to LDAP. One of the attributes allows the user to
>>>>>> write the
>>>>>> group into one or many pre-defined LDAP containers (e.g,
>>>>>> ou=groups,ou=groups,ou=application,...). The attribute value is a
>>>>>> string
>>>>>> containing DNs separated by semicolon.
>>>>>>
>>>>>> My approach so far is the goal of psp writing exactly the same
>>>>>> static/dynamic groups as our custom sync. Success is measured by
>>>>>> bulkCalc
>>>>>> returning whatever is the equivalent of no-op. Alternatively, I can
>>>>>> restructure how we use Types and Attributes. The immovable is our
>>>>>> ability to
>>>>>> direct static groups using Type/Attribute for one or more LDAP
>>>>>> containers.
>>>>>>
>>>>>> I have been successful in plucking the groups in GroupDataConnector,
>>>>>> AllGroupNamesConnector, et al, by using GroupExactAttribute filters.
>>>>>> However, I wonder about the feasibility of the dynamic user-specified
>>>>>> static
>>>>>> group DNs. Besides that I cannot currently conceive of the correct
>>>>>> psp-resolver configuration, I think this might be opposed to the
>>>>>> design
>>>>>> (please let me know). Will it cause problems with the groupDn
>>>>>> attribute? Our
>>>>>> static groups may not have a canonical groupDn, and often not any
>>>>>> static
>>>>>> group at all. Attached is my attempt at a psp-resolver and psp.xml
>>>>>> configuration just for POC. I think the log message during bulkCalc
>>>>>> "PSO
>>>>>> Identifier Definition 'groupDn' - Source attribute 'groupDn' does not
>>>>>> exist"
>>>>>> points at least at part of my problem.
>>>>>>
>>>>>> As if that isn't enough, our custom AD synchronizer works similarly,
>>>>>> except
>>>>>> allows the user to also specify the DOMAIN where groups are stored (we
>>>>>> do
>>>>>> not always have them in the same domain as users). Perhaps I'll tackle
>>>>>> this
>>>>>> another time, but we also have user objects from a forest trust which
>>>>>> are
>>>>>> resolved by the provisioner (the group-member in AD must be a
>>>>>> specially
>>>>>> formatted foreign SID obtained from the other forest, not a vanilla
>>>>>> local
>>>>>> user object). WHAT A MESS!
>>>>>>
>>
>>
>
>