Grouper Users - Open Discussion List

[Jeffrey Williams <>]

Hi Oliver,

I looked at this with Bert and we'll be putting in a JIRA for this( GRP-2481 - Adding Groups does not add new group to selectedGroups ).  Your good notes lead us to what we think is a case where a group is created, but not added to the selectedGroups set until its rebuilt(by the changeLog events surrounding attribute maniupation, stem or group updates) or the service is restarted(which also triggers a rebuild.

We'll see if we can replicate it and if the behavior is what we think, a patch should be able to be made for it.

Again, thanks for you footwork on this.  We'll keep you posted.

-Jeff

On Tue, Dec 3, 2019 at 8:35 AM Oliver Trieu <> wrote:

Dear Grouper Users,

To solve my problem I download the grouper source-code and attached a remote debugger to my grouper instance to see where the problem is.

Here are my findings:

In the class "edu.internet2.middleware.grouper.pspng.Provisioner" the variable "selectedGroups" contains all groups selected for provisioning and on each action,

before they are sent to the actual LDAP-Provisioner a check is being made if your group is in the "selectedGroups".

If it isnt in the selectedGroups you get the famous message "Ignoring work item because (deleted) group was not provisioned before it was deleted"

Here is the code snippet (function filterWorkItems):

        if ( group.hasGroupBeenDeleted() && !selectedGroups.get().contains(group) ) {
          workItem.markAsSkippedAndWarn("Ignoring work item because (deleted) group was not provisioned before it was deleted");
          continue;
        }

As you can see a deleted group will be skipped if it is not contained in the selectedGroups set.

The function "processAnyChangesInGroupSelection" is used to update this group-cache in the "selectedGroups".

However this method is not called when a new group is created, even if it should be in the "selectedGroups" (because it is actually provisioned as you can see in my log snippets from my previous message).

I dont see any check for this in the code.

The function "provisionItem" will check for this:

      if ( workItem.matchesChangelogType(ChangelogHandlingConfig.changelogTypesThatAreHandledIncrementally) ) {
        processIncrementalSyncEvent(workItem);
        return;
      }

      if ( workItemMightChangeGroupSelection(workItem) ) {
        processAnyChangesInGroupSelection(workItem);
      }

Incremental handling is used for the following types:

            ChangeLogTypeBuiltin.GROUP_ADD,
            ChangeLogTypeBuiltin.GROUP_DELETE,
            ChangeLogTypeBuiltin.MEMBERSHIP_ADD,
            ChangeLogTypeBuiltin.MEMBERSHIP_DELETE)

So Group-Add and Group-Delete will only trigger an incremental sync not updating the seletedGroups cache (which thus does not yet contain our newly created or deleted group...).

I also found out why it sometimes works!

If you alter any attribute the cache is rebuilt. Also a restart will rebuild the cache and then the delete will work too.

The following change log types will trigger a rebuild of the selectedGroups Set:

            ChangeLogTypeBuiltin.ATTRIBUTE_ASSIGN_ADD,
            ChangeLogTypeBuiltin.ATTRIBUTE_ASSIGN_DELETE,
            ChangeLogTypeBuiltin.ATTRIBUTE_ASSIGN_VALUE_ADD,
            ChangeLogTypeBuiltin.ATTRIBUTE_ASSIGN_VALUE_DELETE,
            ChangeLogTypeBuiltin.GROUP_UPDATE,
            ChangeLogTypeBuiltin.STEM_UPDATE

How is that supposed to work or am i missing an important point here?

Kind Regards

Oliver

On 10/24/19 3:08 AM, Jeffrey Williams wrote:

Hi Oliver,

That is all good information you've provided, including that it is not consistently having the issue.  I'm still looking at your logs and wanted to ask about the URL that Grouper uses to point to AD.  If it's not pointed to a single DC or a pool that balanced active/standby, can you point it to a single DC and see if the error continues?

I don't believe that this will resolve the issue, I wanted to make a side note:  Searching by CN and objectClass is fine if you're provisioning flat and CN: ${group.name}. However, I would not use CN=${group.extension} to search a bushy OU structure.  Bushy OU's can hold multiple groups with the same name extension as their CN. In those cases, this will cause your queries to return multiple objects, which neither PSPNG nor you will like :).  For bushy provisioning in AD, I've found using gidNumber=${group.idIndex} works well because it's not influenced by the end user and won't cause AD to create a new AD object when all that was required was to update the name of the old one.  You can also use DN, but be warned that AD will create a new object each time the group extension or its location(i.e. its DN) changes.

On Wed, Oct 23, 2019 at 5:40 AM Oliver Trieu <> wrote:

Hi Jeffrey,

thank you for your time, of course i can.

I created a new Group using the grouper-gui (GROUP2).

I waited until it was provisioned to AD and then deleted it in grouper.

The attached log file shows the creation and deletion of the group (not_working.txt).

I should add that oddly enough on my first try it actually worked!

It seems to be kind of random as i did the exact same thing (created group, waited for provisioning, then deleted).

I also attached this logfile (worked_strange.txt).

I did not change any settings between these tries...

Kind Regards

Oliver

On 10/22/19 11:41 PM, Jeffrey Williams wrote:

Hi Oliver,

Can you turn up the logging on your provisioner to DEBUG and delete a group?

On Tue, Oct 22, 2019, 1:21 AM Oliver Trieu <> wrote:

Dear Chris,

thank you for your fast response!

Here is the information you asked for:

grouperPatchStatus read from: /opt/grouper.apiBinary-2.4.0/grouperPatchStatus.properties

api patches installed: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74

pspng patches installed: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11

Kind Regards

Oliver

Am 21.10.2019 um 17:28 schrieb Hyzer, Chris:

Can you please confirm which patches you have installed?

-----Original Message-----
From:   On Behalf Of Oliver Trieu
Sent: Thursday, October 17, 2019 4:13 AM
To: 
Subject: [grouper-users] AD Provisioning not working on delete

Hi fellow Grouper Users,


i have the grouper 2.4.0 up and running provisioning into our Microsoft 
AD server.

The provisioning of new Groups works just fine, but when i delete the 
new group it does NOT get deleted in the AD.

Somehow grouper thinks it did not provision the group and thus does not 
delete it.

What could be the cause of this issue?

Here is the log-snippet to show the problem:

2019-10-16 11:31:00 INFO  Starting provisioning batch of 1 items
2019-10-16 11:31:00 INFO  Information cached before 
2019-10-16T11:30:04.881+02:00 will be ignored
2019-10-16 11:31:00 INFO  LDAP search returned 0 entries
2019-10-16 11:31:00 INFO  Creating LDAP group for GrouperGroup: 
...:NEW_GROUP/#12110(Existing)
2019-10-16 11:31:00 INFO  pspng_activedirectory: Creating LDAP object: 
cn=NEW_GROUP,ou=...
2019-10-16 11:31:00 INFO  pspng_activedirectory: Checking for (and 
creating) missing OUs in DN: cn=NEW_GROUP,ou=... (wholeDnIsOu=false)
2019-10-16 11:31:00 INFO  active_directory: Creating LDAP object Entry 
Summary: dn=cn=NEW_GROUP,ou=...|1 cn values|2 objectclass values|
2019-10-16 11:31:00 INFO  LDAP search returned 1 entries
2019-10-16 11:31:00 INFO  Work item handled: 
ProvisioningWorkItem[done=true,successful=true,msg=Group 
...l:NEW_GROUP/#12110(Existing) already exists,clog=clog #2071223 / 
ChangeLog type: group: addGroup,group=..:NEW_GROUP]

2019-10-16 11:31:00 INFO  Provisioning batch summary: 3 successes/0 
failures. Duration=00.788s

Now it is provisioned but when i delete the same object:

2019-10-16 11:40:00 INFO  pspng_activedirectory: +processChangeLogEntries(1)
2019-10-16 11:40:00 WARN  Unable to refresh object from database, 
probably because it has been deleted: 
Group[name=...:NEW_GROUP,uuid=3221ae657f2b4666b008e40086d8350d]
2019-10-16 11:40:00 WARN  Work item handled: 
ProvisioningWorkItem[done=true,successful=true,msg=Ignoring work item 
because (deleted) group was not provisioned before it was 
deleted,clog=clog #2071225 / ChangeLog type: group: 
deleteGroup,group=...:NEW_GROUP]
2019-10-16 11:40:00 INFO  pspng_activedirectory: 0 work items need to be 
processed further
2019-10-16 11:40:00 INFO  Starting provisioning batch of 0 items
2019-10-16 11:40:00 INFO  Information cached before null will be ignored
2019-10-16 11:40:00 INFO  Provisioning batch summary: 1 successes/0 
failures. Duration=00.005s

Grouper decides to not delete it!


I have a similar issue with Folders where grouper will only provision 
the folder if i create a group in that folder and deleting the folder 
will not provision the delete to the AD.

A full sync however will bring the groups in sync but not the folders ...


Kind Regards


Oliver


PS: here is my grouper-loader config for the AD Provisioner:

changeLog.consumer.pspng_activedirectory.class = 
edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim
changeLog.consumer.pspng_activedirectory.type = 
edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner
changeLog.consumer.pspng_activedirectory.quartzCron = 0 * * * * ?
changeLog.consumer.pspng_activedirectory.ldapPoolName = active_directory
changeLog.consumer.pspng_activedirectory.isActiveDirectory = true
changeLog.consumer.pspng_activedirectory.grouperIsAuthoritative = true
changeLog.consumer.pspng_activedirectory.memberAttributeName = member
changeLog.consumer.pspng_activedirectory.memberAttributeValueFormat = 
${ldapUser.getDn()}
changeLog.consumer.pspng_activedirectory.groupSearchBaseDn = ou=...
changeLog.consumer.pspng_activedirectory.allGroupsSearchFilter = 
objectclass=group
changeLog.consumer.pspng_activedirectory.singleGroupSearchFilter = 
(&(objectclass=group)(cn=${grouperUtil.extensionFromName(group.name)}))
changeLog.consumer.pspng_activedirectory.groupCreationLdifTemplate = dn: 
${utils.bushyDn(group.name, "cn", "ou")}||cn: 
${grouperUtil.extensionFromName(group.name)}||objectclass: 
group||objectclass: top
changeLog.consumer.pspng_activedirectory.userSearchBaseDn = ou=...
changeLog.consumer.pspng_activedirectory.userSearchFilter = 
samAccountName=${subject.id}

-- 
Oliver Trieu
Managed Services 
Server and Data Management

Universität Wien
Zentraler Informatikdienst
Universitätsstrasse 7, 1010 Wien

T +43-1-4277-14161
M: +43-664-60277-14161

zid.univie.ac.at

--

Jeffrey Williams 

Identity Engineer
Identity & Access Services
https://its.uncg.edu

-- 
Oliver Trieu
Managed Services 
Server and Data Management

Universität Wien
Zentraler Informatikdienst
Universitätsstrasse 7, 1010 Wien

T +43-1-4277-14161
M: +43-664-60277-14161

zid.univie.ac.at

--

Jeffrey Williams 

Identity Engineer
Identity & Access Services
https://its.uncg.edu