Grouper Users - Open Discussion List

[Tom Zeller <>]

I don't remember, are old-style or new-style attributes cached by
grouper ? with PIT auditing too ?

If a group with 1,000 members is added to another group, there will be
1,000 change log add_membership entries. If a provisioner needs to
query grouper for old-style or new-style attributes to determine
provisioned target identifiers, that's 1,000 "find group" queries
(cached) and then 1,000 "get old-style attribute" queries and/or 1,000
"get new-style attribute" queries, correct ?

If we decide that we need to use PIT, are the PIT "find group", "get
old-style attribute", and "get new-style attribute" queries cached too
?

Deleting a group whose provisioned target identifier is not solely
based on the group name will require PIT queries as well.

Maybe we should say we don't support real-time provisioning if
provisioned target identifiers are not based only on the object
(group, stem, member) name, excluding displayName.

If attribute queries are cached by grouper, then maybe deployers will
want to have a separate ehcache configuration for provisioning with
longer timeouts than the UI.

TomZ

On Wed, Apr 11, 2012 at 3:29 PM, Chris Hyzer 
<>
 wrote:
>> The latency I am concerned about is on the order of hours rather than
>> minutes, for example, patch tuesday or holiday break maintenance when
>> domain controllers might be down for many hours. So, I am not sure the
>> 5 minute rule helps.
>
> Im saying if you have 2 queries in 5 minutes, then you could get from 
> cache.  If you query 2 hours later, the provisioning attributes would still 
> be there, I don't think it is a problem.
>
> Lets discuss this at the member meeting I guess...  maybe a solution is a 
> JSON 4k field we could pack data into, or with the new UI try to get people 
> off of the old-style attributes and then things would be in PIT...  Im 
> still not sure why this is a large risk, in general the metadata about 
> provisioning wouldn't change too often, it seems like an edge case which we 
> might be overengineering...
>
> Thanks,
> Chris
>
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Tom Zeller
> Sent: Wednesday, April 11, 2012 4:15 PM
> To: 
> 
> Cc: Colin Hudler
> Subject: Re: [grouper-users] Replicating peculiar functionality with psp
>
> The latency I am concerned about is on the order of hours rather than
> minutes, for example, patch tuesday or holiday break maintenance when
> domain controllers might be down for many hours. So, I am not sure the
> 5 minute rule helps.
>
> I understand the scaling issues, and I am also unsure what to cache
> for each kind of change log event. For example, for a membership
> change log event, do I query for the group, member, subject, and
> membership objects and cache all of them ?
>
> The idea I am floating is for grouper to provide an attribute, e.g.
> changeLogAttribute1, which if present on any object whose identifier
> exists in a change log event, the values of the attribute would be
> added to the change log event. So, UChicago would store the custom
> ldap identifier and the target identifier in changeLogAttribute1. They
> would need to use some sort of string separator, or xml, or whatever,
> to concatenate two old-style attributes into a single new-style
> attribute. If we have enough change log event columns available, we
> could provide changeLogAttribute1, changeLogAttribute2, etc.
>
> When I first encountered userAttribute1 or customAttribute1 in active
> directory or windows live I thought they were clumsy, but at least
> folks might not be surprised.
>
> And, yeah, I think this is hacky but trying to cache a slice of the
> grouper database for every change log event seems prone to performance
> issues.
>
> Thoughts ?
>
> TomZ
>
> On Fri, Apr 6, 2012 at 6:07 PM, Chris Hyzer 
> <>
>  wrote:
>> I think we should document that changes in provisioning attributes take 5 
>> minutes to propagate to the provisioner... you can do a second query and 
>> cache for 5 minutes...
>> Sound good?  :)
>>
>> I don't think we should add attributes to the change log since we would 
>> need to add for each type of query it will get complicated to configure 
>> and what if someone needs more attributes for various provisioners than 
>> are available.  It doesn't really scale...
>>
>> Thanks,
>> Chris
>>
>> -----Original Message-----
>> From: 
>> 
>>  
>> [mailto:]
>>  On Behalf Of Tom Zeller
>> Sent: Friday, April 06, 2012 12:05 AM
>> To: Colin Hudler
>> Cc: 
>> 
>> Subject: Re: [grouper-users] Replicating peculiar functionality with psp
>>
>> Here is one issue worth talking about.
>>
>> The current code for change log based provisioning does not handle the
>> custom attributes which determine whether or not a group should be
>> provisioned as well as the desired ldap dn.
>>
>> For the first attempt at change log based provisioning, I decided to
>> try to provision based on change log entry data only without
>> additional queries. An "add membership" change log entry does not
>> contain the custom group attributes, so another query would be needed
>> to return those attributes. Not a big deal, except ... I have been
>> worried about phasing, meaning the potential time difference between
>> when a change log entry is processed and when the action producing the
>> change log entry occurred. I think I am alone in my concern.
>>
>> A potential solution is to use PIT (point-in-time) auditing to return
>> the group as it was when the change log event occurred. However, if I
>> remember correctly, I don't think that custom attributes are included
>> in PIT auditing, but new attribute framework attributes are.
>>
>> In general, I prefer provisioning a change log entry without
>> performing additional queries, meaning, a change log entry should
>> contain all of the data necessary for provisioning.
>>
>> Maybe it could be possible to include custom attributes in change log
>> entries ? A change log entry has capacity for 12 strings (columns),
>> maybe deployers could specify additional group (or member or stem)
>> attributes to be returned in addition to the group (or member or stem)
>> name or id ?
>>
>> So, rather than the provisioner querying for additional data at the
>> time of provisioning, perhaps grouper could include necessary
>> additional data in the change log entry at the time the change log
>> event occurred, which would alleviate my concerns regarding time
>> delays.
>>
>> Whether or not folks use the psp for provisioning based on the change
>> log, I think they will face similar issues, and customizing change log
>> entries may help deployers regardless of their provisioning gear.
>>
>> This response is probably too long already, but it might be a good
>> idea to standardize and perhaps provide a default configuration for
>> attributes used to determine whether or not objects should be
>> provisioned, as well as target identifiers.
>>
>> TomZ
>>
>> On Thu, Apr 5, 2012 at 6:39 PM, Tom Zeller 
>> <>
>>  wrote:
>>> Apologies for the delay in my response.
>>>
>>> In general, I think you will be ok, meaning : you should be able to do
>>> what you want.
>>>
>>> It will take me a minute to parse your configuration, in the meantime,
>>> could you help me understand  :
>>>
>>>> Our static groups may not have a canonical groupDn, and often not any 
>>>> static
>>>> group at all.
>>>
>>> I think you are saying that the ldap dn of a static group is user
>>> specified, and stored as a group attribute in grouper. This seems
>>> fine.
>>>
>>> And, regarding :
>>>
>>>> allows the user to also specify the DOMAIN where groups are stored (we do
>>>> not always have them in the same domain as users).
>>>
>>> What is the impact of the user specified domain ? Does each domain
>>> require an ldap connection ?
>>>
>>> Thanks for trying out the psp.
>>>
>>> TomZ
>>>
>>> On Wed, Apr 4, 2012 at 12:05 PM, Colin Hudler 
>>> <>
>>>  wrote:
>>>> We have custom provisioner to LDAP. Users are allowed to add a Group Type
>>>> and attribute value that the provisioner looks for to enable the
>>>> synchronization to LDAP. One of the attributes allows the user to write 
>>>> the
>>>> group into one or many pre-defined LDAP containers (e.g,
>>>> ou=groups,ou=groups,ou=application,...). The attribute value is a string
>>>> containing DNs separated by semicolon.
>>>>
>>>> My approach so far is the goal of psp writing exactly the same
>>>> static/dynamic groups as our custom sync. Success is measured by bulkCalc
>>>> returning whatever is the equivalent of no-op. Alternatively, I can
>>>> restructure how we use Types and Attributes. The immovable is our 
>>>> ability to
>>>> direct static groups using Type/Attribute for one or more LDAP 
>>>> containers.
>>>>
>>>> I have been successful in plucking the groups in GroupDataConnector,
>>>> AllGroupNamesConnector, et al, by using GroupExactAttribute filters.
>>>> However, I wonder about the feasibility of the dynamic user-specified 
>>>> static
>>>> group DNs. Besides that I cannot currently conceive of the correct
>>>> psp-resolver configuration, I think this might be opposed to the design
>>>> (please let me know). Will it cause problems with the groupDn attribute? 
>>>> Our
>>>> static groups may not have a canonical groupDn, and often not any static
>>>> group at all. Attached is my attempt at a psp-resolver and psp.xml
>>>> configuration just for POC. I think the log message during bulkCalc "PSO
>>>> Identifier Definition 'groupDn' - Source attribute 'groupDn' does not 
>>>> exist"
>>>> points at least at part of my problem.
>>>>
>>>> As if that isn't enough, our custom AD synchronizer works similarly, 
>>>> except
>>>> allows the user to also specify the DOMAIN where groups are stored (we do
>>>> not always have them in the same domain as users). Perhaps I'll tackle 
>>>> this
>>>> another time, but we also have user objects from a forest trust which are
>>>> resolved by the provisioner (the group-member in AD must be a specially
>>>> formatted foreign SID obtained from the other forest, not a vanilla local
>>>> user object). WHAT A MESS!