Grouper Users - Open Discussion List

[Chris Hyzer <>]

You can easily do this in the custom class you wrote J

 

new GroupSave(GrouperSession.staticGrouperSession()).assignName(groupNameFromAd).assignCreateParentStemsIfNotExist(true).save();

 

But yes, we could improve the loader at some point so the loader can do this…

 

I added a jira, but not sure when this would get done since it is not a permanent problem J  Note: in the SQL loader there are two queries, one for groups, one for memberships, this addresses the problem, and handles orphans, maybe we need a similar thing in the ldap loader…

 

https://bugs.internet2.edu/jira/browse/GRP-790

 

Thanks,

Chris

 

From: Gagné Sébastien [mailto:]
Sent: Monday, May 14, 2012 10:50 AM
To: Chris Hyzer
Cc:
Subject: RE: Grouper Loader LDAP with AD

 

I agree terminology is confusing at first, but you get used to it.

 

Back on subject : yes it works now ! I didn’t catch the difference between SubjectID and SubjectIdentifier for groups. So I did a search by subjectIdentifier and converted to DN to either sAMAccount name (CN / login ID) and Group Name (stemA:stemB:GroupID) with the custom class and it works, I didn’t need to call the backdoor class to get the UUID. It also works with the UUID backdoor, but prefer the “standard” way.

 

Further testing showed one weakness in the loader : if one of the member of a parent-group in AD is a child-group that doesn’t exist yet in Grouper, but does in AD, the loader will throw an exception because it cannot find the child-group because it wasn’t yet created. This can be patched by running the Loader twice (the second time the child-group will be there since it was created in the first run), but I think it’s a poor solution.

 

Could it be improved in Grouper ? You could automatically create child-groups when they aren’t existing, but this might cause problems if some of the child-group are “out of scope” of the loader or Grouper altogether (e.g. if Grouper is managing only a subset of AD groups). The other solution might be to do the process in two steps : 1- create all the groups, 2- add the memberships in the groups. This might not be practical from a performance standpoint, but it would ensure that all SNF exceptions are justified and must be looked into.

 

Thanks for your help

 

De : Chris Hyzer []
Envoyé : 11 mai 2012 14:08
À : Gagné Sébastien
Cc :
Objet : RE: Grouper Loader LDAP with AD

 

(moving back to list)

 

The terminology is confusing.

 

If you can generate the group id path (UI term) in the EL, that is actually the API “name”, or the g:gsa “identifier”

 

So… if you are doing a loader job by identifier, and you return the group id path, then it should work (like you said, don’t specify the source)… if you need to convert to UUID in the EL, then try this:

 

    Group group = GrouperDAOFactory.getFactory().getGroup().findByName(idPath, false, null) ;

 

You don’t need a grouper session for that method (it’s a backdoor).  Then if you get null, the group isn’t there.  If it is there, then the uuid is group.getId()…

 

Does it work?  J

 

Thanks,

Chris

 

Ps. the GrouperSourceAdapter search by identifier is here:

 

  public Map<String, Subject> getSubjectsByIdentifiers(final Collection<String> identifiers) {

    Set<Group> groups = null;

   

    groups = (Set<Group>)GrouperSession.callbackGrouperSession(

        GrouperSourceAdapter.internal_getSessionOrRootForSubjectFinder(), new GrouperSessionHandler() {

 

      public Object callback(GrouperSession grouperSession)

          throws GrouperSessionException {

        return GrouperDAOFactory.getFactory().getGroup().findByNamesSecure(identifiers, null, GrouperSourceAdapter.this.typeOfGroups());

      }

    });

 

 

 

From: Gagné Sébastien
Sent: Friday, May 11, 2012 1:58 PM
To: Chris Hyzer
Subject: RE: Grouper Loader LDAP with AD

 

Hello Chris,

I did the EL class and it works well by itself, BUT it’s still not finding the Group in g:gsa. ID Path is correct and I tried  to specify no sources  or g:gsa (to force the check) with no luck.

 

I dug in the source and I think that the find subject for g:gsa only accepts  UUIDS :

 

In : edu.internet2.middleware.grouper.GrouperSourceAdapter.java

 

    groups = (Set<Group>)GrouperSession.callbackGrouperSession(

        GrouperSourceAdapter.internal_getSessionOrRootForSubjectFinder(), new GrouperSessionHandler() {

 

      public Object callback(GrouperSession grouperSession)

          throws GrouperSessionException {

        return GrouperDAOFactory.getFactory().getGroup().findByUuidsSecure(ids, null, GrouperSourceAdapter.this.typeOfGroups());

      }

    });

 

Is there an easy way to get Group UUIDS based on Group ID or Group ID Path ? Going with Group ID would be awesome since I would have to convert the DN to an ID path.

 

Maybe this would require more global change in Grouper (I understand that IDs might not be unique, but ID Paths are) ?

 

Thanks for your help

 

 

 

De : Chris Hyzer []
Envoyé : 10 mai 2012 15:39
À : Gagné Sébastien
Objet : RE: Grouper Loader LDAP with AD

 

> Thanks again, but how frequently should the loader run ? I was thinking about once a day so 5 hours might be too long

 

Whatever you want to do…

 

>

> How will the loader react if there are new members in Grouper’s group that aren’t in AD when the loader runs  (lets say there was a problem with the PSP) ?

> -           Will the loader delete the new members to reflect what is in AD

 

Yes.  The loader assumes that group is the system of record.  It should only be edited by the loader.  If you want ad hoc additions, then add that system of record group to another “overall” group, and add more members to that group.  Or you could do a restrict list too with composites.  Know what I mean?

 

> -           Will the loader add the missing members from the AD group in the grouper group without removing them

> -           Will the loader add the new members in AD (doubtful)

>

> I was thinking of scheduling the PSP full sync before the loader, but it might be somewhat redundant.

 

Sure, not sure which parts of AD are in the loader and which are in the PSP…  up to you

 

chris

 

From: Gagné Sébastien
Sent: Thursday, May 10, 2012 3:35 PM
To: Chris Hyzer
Subject: RE: Grouper Loader LDAP with AD

 

Thanks again, but how frequently should the loader run ? I was thinking about once a day so 5 hours might be too long

 

How will the loader react if there are new members in Grouper’s group that aren’t in AD when the loader runs  (lets say there was a problem with the PSP) ?

-          Will the loader delete the new members to reflect what is in AD

-          Will the loader add the missing members from the AD group in the grouper group without removing them

-          Will the loader add the new members in AD (doubtful)

 

I was thinking of scheduling the PSP full sync before the loader, but it might be somewhat redundant.

 

De : Chris Hyzer []
Envoyé : 10 mai 2012 15:23
À : Gagné Sébastien
Objet : RE: Grouper Loader LDAP with AD

 

Btw, you should definitely cache this.  Maybe a permanent or an expirable cache?  J

 

  /** cache if cn is a person (false is group), for 5 hours */

  private static ExpirableCache<String, Boolean> cacheIsPerson = new ExpirableCache<String, Boolean>(60*5);

 

  /**

   * convert a user dn to a user, and a group dn to a group

   * @param dn

   * @return the dn

   */

  public static String convertAdDnToSpecificValue(String dn) {

   

    String cn = LoaderLdapElUtils.convertDnToSpecificValue(dn);

   

    Boolean isPerson = cacheIsPerson.get(cn);

   

    if (isPerson == null) {

      //do ldap call, figure it out

      ...

     

      //put back in cache

      cacheIsPerson.put(cn, isPerson);

    }

   

    if (isPerson) {

      return cn;

    }

   

    //is group

    //e.g. umontreal.grouper.ad.group.prefix = umontreal:adgroups:

    String grouperGroupPrefix = GrouperLoaderConfig.getPropertyString("umontreal.grouper.ad.group.prefix", true);

   

    return grouperGroupPrefix + cn;

  }   

 

 

From: Gagné Sébastien
Sent: Thursday, May 10, 2012 2:57 PM
To: Chris Hyzer
Subject: RE: Grouper Loader LDAP with AD

 

Ok, thanks I’ll try that also.

 

About LdapSession :

Do I need to get an object or can directly use “LdapSession.list” (static method) ? Will it manage everything ? Or should I use a callback or helper somewhere ?

 

The “list” function seems to be returning a list of attributes so I’ll try to get the objectClass from there.

 

What I find strange here is that this object seems to be made for the Loader, but is outside the Loader directory, it might confuse some people

 

Anyway I got the source code so I think I’ll be able to get something decent. I’ll check back with you later

 

Thanks a lot.

 

De : Chris Hyzer []
Envoyé : 10 mai 2012 14:18
À : Gagné Sébastien;
Objet : RE: Grouper Loader LDAP with AD

 

You can do an EL that does a directory call…  LdapSession might help you do a lookup by object class, if you want to bind as the user, you need to not use the same pool of directory connections I think…  if you need help let  me know

 

Chris

 

From: Gagné Sébastien
Sent: Thursday, May 10, 2012 2:10 PM
To: Chris Hyzer;
Subject: RE: Grouper Loader LDAP with AD

 

Wow thanks a lot, I figure that with that option I have to leave the Source ID empty for the loader to try the AD source then the g :gsa source (which strangely doesn’t have any search or filter configuration but I guess it’s managed in grouper).

 

Unfortunately, groups and users are both under People (I know… you don’t have to tell me it’s wrong, we’re trying to fix this) so the Suffix condition might not work. I’ll have to check with our experts here, there might be some string we could check for if where only managing a subset of groups, but I might not be foolproof (or any-user proof). I will give it a try to make it work if we want (or can) to go that way, it will eventually be useful.

 

What would be best is if we could check against LDAP with loginID and if it fails check in Grouper Groups with a GroupID, but it might not be possible to configure that part of the process.

 

Another option would be to get the objectClass of the member DN and then transform the ID accordingly, but that would require a new connexion and search in LDAP. Is there some utilities in Grouper to help with that ?

 

Could I just create my own EL class that does the insert instead of the loader’s standard way ? Would it be practical ?

 

Thanks

 

 

De : Chris Hyzer []
Envoyé : 10 mai 2012 11:48
À : Gagné Sébastien;
Objet : RE: Grouper Loader LDAP with AD

 

I think it can do it, but we have to adapt the convertDbToSpecificValue part…

 

Ie. A person DN is CN=gagns,OU=People,DC=dev,DC=umontreal,DC=ca

A group DN is (Im making this up): CN=someGroup,OU=Groups,DC=dev,DC=umontreal,DC=ca

 

Then we need a simple Java method that we will add to the LDAP subject _expression_ which:

 

Looks at the DB, and if it is in People, then return the CN value, if it is Groups (or wherever), then we need the group name in Grouper: umontreal:adgroups:someGroup or whatever the translation is…

 

1.       Edit the grouper-loader.properties:

 

(edit this existing entry)

loader.ldap.el.classes = ca.umontreal.grouper.UmontrealLoaderElUtils

 

(add these [and configure to your env] )

umontreal.ad.group.suffix =,OU=Groups,DC=dev,DC=umontreal,DC=ca

umontreal.grouper.ad.group.prefix = umontreal:adgroups:

 

2.       Add the attached jar to your lib dir

 

3.       Change your attribute:

 

LDAP subject _expression_              ${umontrealLoaderElUtils.convertAdDnToSpecificValue(subjectId)}

 

I didn’t test it… see if it helps… J

 

Chris

 

Btw, the source of that jar is one file.. look at the logic, will it work or do you need something different?

 

/**

* @author mchyzer

* $Id$

*/

package ca.umontreal.grouper;

 

import org.apache.commons.lang.StringUtils;

 

import edu.internet2.middleware.grouper.app.loader.GrouperLoaderConfig;

import edu.internet2.middleware.grouper.app.loader.ldap.LoaderLdapElUtils;

import edu.internet2.middleware.grouper.util.GrouperUtil;

 

 

/**

*

*/

public class UmontrealLoaderElUtils {

 

 

  /**

   * convert a user dn to a user, and a group dn to a group

   * @param dn

   * @return the dn

   */

  public static String convertAdDnToSpecificValue(String dn) {

   

    //e.g. ,OU=Groups,DC=dev,DC=umontreal,DC=ca

    String adGroupSuffix = GrouperLoaderConfig.getPropertyString("umontreal.ad.group.suffix", true);

   

    //not sure why this would happen

    if (dn == null) {

      return dn;

    }

   

    if (dn.toLowerCase().endsWith(adGroupSuffix.toLowerCase())) {

      String cn = dn.substring(0, dn.length()-adGroupSuffix.length());

     

      if (StringUtils.countMatches(cn, "=") != 1) {

        throw new RuntimeException("Why is there not 1 equals in this CN??? '" + cn + "'");

      }

     

      //this should be CN=groupName, convert to groupName

      cn = GrouperUtil.prefixOrSuffix(cn, "=", false);

     

      //e.g. umontreal.grouper.ad.group.prefix = umontreal:adgroups:

      String grouperGroupPrefix = GrouperLoaderConfig.getPropertyString("umontreal.grouper.ad.group.prefix", true);

     

      return grouperGroupPrefix + cn;

 

    }

    //not a group

    return LoaderLdapElUtils.convertDnToSpecificValue(dn);

  }

 

}

 

 

From: On Behalf Of Gagné Sébastien
Sent: Thursday, May 10, 2012 11:13 AM
To:
Subject: RE: [grouper-users] RE: Grouper Loader LDAP with AD

 

Hello,

I have a new question : Does the Loader LDAP support group membership in a group (i.e GroupA is a member of GroupB in LDAP)

 

It seems the standard Grouper Loader can do this : “Penn is using it in production to load membership for groups, and for groups of groups”

 

When I tried it, the loader couldn’t find GroupA in the subject source (since I filter using objectClass=person). I removed this part of the filter, it added the group, but not as a Grouper group but as an AD subject. So I put the filter back like it was with the objectClass=person and removed the source ID thinking the loader could search the Group Source Adapter (g:gsa), but it didn’t work.

 

Here are my current configuration, I tried to keep it as plain as possible :

LDAP type	LDAP_GROUP_LIST
LDAP filter	(objectClass=group)
LDAP subject attribute name	member
LDAP source ID
LDAP subject ID type	subjectId
LDAP server ID	personLdap
LDAP quartz cron	30 * * * * ?
LDAP subject _expression_	${loaderLdapElUtils.convertDnToSpecificValue(subjectId)}

 

In the Group’s member attribute contains both Group DNs and Member DNs with no

 

Thank you again.

 

De : [] De la part de Chris Hyzer
Envoyé : 9 mai 2012 11:51
À : Gagné Sébastien;
Objet : [grouper-users] RE: Grouper Loader LDAP with AD

 

This is what TomB asked at the member meeting and I said, “Uh, I think it does that”  J

 

I think you need:

 

grouperLoaderLdapSubjectExpression

 

To the value:

 

${loaderLdapElUtils.convertDnToSpecificValue(subjectId)}

 

That should work for subjectId, or subjectIdentifier, or subjectIdOrIdentifier.  See if that works.

 

If not can you let me know all the loader job attributes and values you have set?

 

Thanks,

Chris

 

 

From: On Behalf Of Gagné Sébastien
Sent: Wednesday, May 09, 2012 11:30 AM
To:
Subject: [grouper-users] Grouper Loader LDAP with AD

 

Hello,

I’m trying to configure the Grouper Loader LDAP to work with our AD, but I’m stuck with a problem. Basically what we would like is to retrieve all groups in AD and load them in Grouper.

 

I’ve set the loader type to LDAP_GROUP_LIST with a filter (objectClass=group) (dev AD has 5-6 groups). This properly returns all groups and they are added in Grouper.

 

The problem is with the members : it sees them in the AD group, but cannot find them in the source, which is the same AD.

 

I’ve set the subject attribute name to member and my source ID is ldap (I’m also using the PSP)

The group’s member attribute is a multivalued set of “Distinguished Name” (e.g. CN=gagns,OU=People,DC=dev,DC=umontreal,DC=ca). This is a problem since my subject ID is the sAMAccountName (i.e. gagns). What I did, is use the search type subjectIdentifier and modified the filter in sources.xml to use the full DN :

 

<searchType>searchSubjectByIdentifier</searchType>

    <param>

        <param-name>filter</param-name>

        <param-value>

            (&amp;(distinguishedName=%TERM%)(objectclass=person))

        </param-value>

    </param>

 

Unfortunately it doesn’t work and I don’t know why. When manually running the loader job (i.e. loaderRunOneJob(group) in gsh.sh), in the logs I see that it find the members of the groups, but that the every search for one of the subject fails :

 

2012-05-09 09:06:24,001: [main] ERROR GrouperLoaderResultset$Row.getSubject(1112) -  - Problem with subjectIdentifier: CN=gagns,OU=People,DC=dev,DC=umontreal,DC=ca, subjectSourceId: ldap, in jobName: LDAP_GROUP_LIST__etc:confLDAPLoader__e7b97262558b477fab09c0b48f98ed1c

 

edu.internet2.middleware.subject.SubjectNotFoundException: No results: searchSubjectByIdentifier filter:(&(distinguishedName=%TERM%)(objectclass=person)) searchValue: CN=gagns,OU=People,DC=dev,DC=umontreal,DC=ca

 

        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getLdapUnique(LdapSourceAdapter.java:655)

        at edu.internet2.middleware.subject.provider.LdapSourceAdapter.getSubjectByIdentifier(LdapSourceAdapter.java:336)

        at edu.internet2.middleware.grouper.subj.SourcesXmlResolver.findByIdentifier(SourcesXmlResolver.java:403)

        [contd...]

 

 

I tried an ldapsearch and manually changing the %TERM% to the DN and it works fine :

 

ldapsearch -h husky.devsim.umontreal.ca -D "<grouper service account DN>" -w <pass> -b "OU=People,DC=dev,DC=umontreal,DC=ca"  "(&(distinguishedName=CN=gagns,OU=People,DC=dev,DC=umontreal,DC=ca)(objectclass=person))"

 

CN=gagns,OU=People,DC=devsim,DC=umontreal,DC=ca

objectClass=top

objectClass=person

objectClass=organizationalPerson

objectClass=user

cn=gagns

[continued...]

 

 

Anyone has an idea what might be happening here ? Anyone did a successful AD configuration ?

 

Thank you

 

 

What I find weird is if I use subjectId (not subjectIdentifier), I get slightly different results in the exceptions. JobName is different and membership count also.

 

2012-05-09 10:22:49,165: [main] ERROR GrouperLoaderResultset$Row.getSubject(1112) -  - Problem with subjectId: CN=gagns,OU=People,DC=devsim,DC=umontreal,DC=ca, subjectSourceId: ldap, in jobName: groups:UdeM:fromAD1

 

edu.internet2.middleware.subject.SubjectNotFoundException: No results: searchSubject filter:(&(sAMAccountName=%TERM%)(objectclass=person)) searchValue: CN=gagns,OU=People,DC=devsim,DC=umontreal,DC=ca

 

with subjectIdentifier :

loader ran successfully, inserted 0 memberships, deleted 0 memberships, total membership count: 0

with subjectId :

loader ran successfully, inserted 0 memberships, deleted 0 memberships, total membership count: 40

 

 

 

 

Sébastien Gagné,     | Analyste en informatique

514-343-6111 x33844  | Université de Montréal,

                     | Pavillon Roger-Gaudry, local X-100-11