Grouper Users - Open Discussion List

[Colin Hudler <>]

Just to reiterate, my goal is to have a single-instance provisioner that 
can write groups to different and multiple LDAP container OUs. Very 
often there is no static group (ou), yet the group name is written to a 
common member attribute (dynamic groups).

**We do have the ability to change how this happens at our site**

If doing away with our custom Types and Attributes makes that possible, 
UChicago is very much in favor of changing. Thus far, provisioning 
selection has been in the hands of our users; if it becomes a 
server-side psp configuration, we can probably accept the cost of it 
becoming an admin task.

My most serious obstacle is groking PSP well enough to make it write 
groups into multiple containers, or none at all, without individually 
configuring each group (there's hundreds). Can you help with this?

My ideal is to decorate the groups and/or stems with their intended 
provisioning targets and parameters, but I understand if that is very 
hard. Our group provisioning changes at least weekly, so whatever it is 
would be as lite as possible. Way back when, I accomplished this with an 
automated ldappc config generator which created beastly configurations 
for hundreds of groups based on the registry Types & Attributes and 
wrangled several ldappc instances together. It was slow, hard to debug, 
and almost impossible to use for a realtime system.

On 04/12/2012 11:25 AM, Tom Zeller wrote:
> I don't remember, are old-style or new-style attributes cached by
> grouper ? with PIT auditing too ?
>
> If a group with 1,000 members is added to another group, there will be
> 1,000 change log add_membership entries. If a provisioner needs to
> query grouper for old-style or new-style attributes to determine
> provisioned target identifiers, that's 1,000 "find group" queries
> (cached) and then 1,000 "get old-style attribute" queries and/or 1,000
> "get new-style attribute" queries, correct ?
>
> If we decide that we need to use PIT, are the PIT "find group", "get
> old-style attribute", and "get new-style attribute" queries cached too
> ?
>
> Deleting a group whose provisioned target identifier is not solely
> based on the group name will require PIT queries as well.
>
> Maybe we should say we don't support real-time provisioning if
> provisioned target identifiers are not based only on the object
> (group, stem, member) name, excluding displayName.
>
> If attribute queries are cached by grouper, then maybe deployers will
> want to have a separate ehcache configuration for provisioning with
> longer timeouts than the UI.
>
> TomZ
>
> On Wed, Apr 11, 2012 at 3:29 PM, Chris 
> Hyzer<>
>   wrote:
>    
> > > The latency I am concerned about is on the order of hours rather than
> > > minutes, for example, patch tuesday or holiday break maintenance when
> > > domain controllers might be down for many hours. So, I am not sure the
> > > 5 minute rule helps.
> > >        
> > Im saying if you have 2 queries in 5 minutes, then you could get from cache.  
> > If you query 2 hours later, the provisioning attributes would still be there, 
> > I don't think it is a problem.
> >
> > Lets discuss this at the member meeting I guess...  maybe a solution is a 
> > JSON 4k field we could pack data into, or with the new UI try to get people 
> > off of the old-style attributes and then things would be in PIT...  Im still 
> > not sure why this is a large risk, in general the metadata about provisioning 
> > wouldn't change too often, it seems like an edge case which we might be 
> > overengineering...
> >
> > Thanks,
> > Chris
> >
> > -----Original Message-----
> > From: 
> >
> >  
> > [mailto:]
> >  On Behalf Of Tom Zeller
> > Sent: Wednesday, April 11, 2012 4:15 PM
> > To: 
> >
> > Cc: Colin Hudler
> > Subject: Re: [grouper-users] Replicating peculiar functionality with psp
> >
> > The latency I am concerned about is on the order of hours rather than
> > minutes, for example, patch tuesday or holiday break maintenance when
> > domain controllers might be down for many hours. So, I am not sure the
> > 5 minute rule helps.
> >
> > I understand the scaling issues, and I am also unsure what to cache
> > for each kind of change log event. For example, for a membership
> > change log event, do I query for the group, member, subject, and
> > membership objects and cache all of them ?
> >
> > The idea I am floating is for grouper to provide an attribute, e.g.
> > changeLogAttribute1, which if present on any object whose identifier
> > exists in a change log event, the values of the attribute would be
> > added to the change log event. So, UChicago would store the custom
> > ldap identifier and the target identifier in changeLogAttribute1. They
> > would need to use some sort of string separator, or xml, or whatever,
> > to concatenate two old-style attributes into a single new-style
> > attribute. If we have enough change log event columns available, we
> > could provide changeLogAttribute1, changeLogAttribute2, etc.
> >
> > When I first encountered userAttribute1 or customAttribute1 in active
> > directory or windows live I thought they were clumsy, but at least
> > folks might not be surprised.
> >
> > And, yeah, I think this is hacky but trying to cache a slice of the
> > grouper database for every change log event seems prone to performance
> > issues.
> >
> > Thoughts ?
> >
> > TomZ
> >
> > On Fri, Apr 6, 2012 at 6:07 PM, Chris 
> > Hyzer<>
> >   wrote:
> >      
> > > I think we should document that changes in provisioning attributes take 5 
> > > minutes to propagate to the provisioner... you can do a second query and 
> > > cache for 5 minutes...
> > > Sound good?  :)
> > >
> > > I don't think we should add attributes to the change log since we would need 
> > > to add for each type of query it will get complicated to configure and what 
> > > if someone needs more attributes for various provisioners than are available. 
> > >  It doesn't really scale...
> > >
> > > Thanks,
> > > Chris
> > >
> > > -----Original Message-----
> > > From: 
> > >
> > >  
> > > [mailto:]
> > >  On Behalf Of Tom Zeller
> > > Sent: Friday, April 06, 2012 12:05 AM
> > > To: Colin Hudler
> > > Cc: 
> > >
> > > Subject: Re: [grouper-users] Replicating peculiar functionality with psp
> > >
> > > Here is one issue worth talking about.
> > >
> > > The current code for change log based provisioning does not handle the
> > > custom attributes which determine whether or not a group should be
> > > provisioned as well as the desired ldap dn.
> > >
> > > For the first attempt at change log based provisioning, I decided to
> > > try to provision based on change log entry data only without
> > > additional queries. An "add membership" change log entry does not
> > > contain the custom group attributes, so another query would be needed
> > > to return those attributes. Not a big deal, except ... I have been
> > > worried about phasing, meaning the potential time difference between
> > > when a change log entry is processed and when the action producing the
> > > change log entry occurred. I think I am alone in my concern.
> > >
> > > A potential solution is to use PIT (point-in-time) auditing to return
> > > the group as it was when the change log event occurred. However, if I
> > > remember correctly, I don't think that custom attributes are included
> > > in PIT auditing, but new attribute framework attributes are.
> > >
> > > In general, I prefer provisioning a change log entry without
> > > performing additional queries, meaning, a change log entry should
> > > contain all of the data necessary for provisioning.
> > >
> > > Maybe it could be possible to include custom attributes in change log
> > > entries ? A change log entry has capacity for 12 strings (columns),
> > > maybe deployers could specify additional group (or member or stem)
> > > attributes to be returned in addition to the group (or member or stem)
> > > name or id ?
> > >
> > > So, rather than the provisioner querying for additional data at the
> > > time of provisioning, perhaps grouper could include necessary
> > > additional data in the change log entry at the time the change log
> > > event occurred, which would alleviate my concerns regarding time
> > > delays.
> > >
> > > Whether or not folks use the psp for provisioning based on the change
> > > log, I think they will face similar issues, and customizing change log
> > > entries may help deployers regardless of their provisioning gear.
> > >
> > > This response is probably too long already, but it might be a good
> > > idea to standardize and perhaps provide a default configuration for
> > > attributes used to determine whether or not objects should be
> > > provisioned, as well as target identifiers.
> > >
> > > TomZ
> > >
> > > On Thu, Apr 5, 2012 at 6:39 PM, Tom 
> > > Zeller<>
> > >   wrote:
> > >        
> > > > Apologies for the delay in my response.
> > > >
> > > > In general, I think you will be ok, meaning : you should be able to do
> > > > what you want.
> > > >
> > > > It will take me a minute to parse your configuration, in the meantime,
> > > > could you help me understand  :
> > > >
> > > >          
> > > > > Our static groups may not have a canonical groupDn, and often not any static
> > > > > group at all.
> > > > >            
> > > > I think you are saying that the ldap dn of a static group is user
> > > > specified, and stored as a group attribute in grouper. This seems
> > > > fine.
> > > >
> > > > And, regarding :
> > > >
> > > >          
> > > > > allows the user to also specify the DOMAIN where groups are stored (we do
> > > > > not always have them in the same domain as users).
> > > > >            
> > > > What is the impact of the user specified domain ? Does each domain
> > > > require an ldap connection ?
> > > >
> > > > Thanks for trying out the psp.
> > > >
> > > > TomZ
> > > >
> > > > On Wed, Apr 4, 2012 at 12:05 PM, Colin 
> > > > Hudler<>
> > > >   wrote:
> > > >          
> > > > > We have custom provisioner to LDAP. Users are allowed to add a Group Type
> > > > > and attribute value that the provisioner looks for to enable the
> > > > > synchronization to LDAP. One of the attributes allows the user to write the
> > > > > group into one or many pre-defined LDAP containers (e.g,
> > > > > ou=groups,ou=groups,ou=application,...). The attribute value is a string
> > > > > containing DNs separated by semicolon.
> > > > >
> > > > > My approach so far is the goal of psp writing exactly the same
> > > > > static/dynamic groups as our custom sync. Success is measured by bulkCalc
> > > > > returning whatever is the equivalent of no-op. Alternatively, I can
> > > > > restructure how we use Types and Attributes. The immovable is our ability to
> > > > > direct static groups using Type/Attribute for one or more LDAP containers.
> > > > >
> > > > > I have been successful in plucking the groups in GroupDataConnector,
> > > > > AllGroupNamesConnector, et al, by using GroupExactAttribute filters.
> > > > > However, I wonder about the feasibility of the dynamic user-specified static
> > > > > group DNs. Besides that I cannot currently conceive of the correct
> > > > > psp-resolver configuration, I think this might be opposed to the design
> > > > > (please let me know). Will it cause problems with the groupDn attribute? Our
> > > > > static groups may not have a canonical groupDn, and often not any static
> > > > > group at all. Attached is my attempt at a psp-resolver and psp.xml
> > > > > configuration just for POC. I think the log message during bulkCalc "PSO
> > > > > Identifier Definition 'groupDn' - Source attribute 'groupDn' does not exist"
> > > > > points at least at part of my problem.
> > > > >
> > > > > As if that isn't enough, our custom AD synchronizer works similarly, except
> > > > > allows the user to also specify the DOMAIN where groups are stored (we do
> > > > > not always have them in the same domain as users). Perhaps I'll tackle this
> > > > > another time, but we also have user objects from a forest trust which are
> > > > > resolved by the provisioner (the group-member in AD must be a specially
> > > > > formatted foreign SID obtained from the other forest, not a vanilla local
> > > > > user object). WHAT A MESS!
> > > > >            
>