Grouper Users - Open Discussion List

[Jeffrey Williams <>]

Hi Dominique,

What do you have ldap.groupLdap1.pagedResultsSize set to? 

-Jeff

On Fri, Sep 18, 2020 at 4:21 AM Dominique Petitpierre <> wrote:

Hello,

any advice on how to debug this problem?:

When synchronizing a target group that has more than 1500 members, even if there is nothing to do the PSPNG provisioner tries to replace all the members and fails with a LDAP error ENTRY_ALREADY_EXISTS.
This is with a Microsoft Active Directory target directory. It does not happen with 1500 members or less.
The 1501th member can be added without problem, but a subsequent full sync triggers the problem
This is clearly linked to this Active Directory feature:
3.1.1.3.1.3.3 Range Retrieval of Attribute Values
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/e27b48db-6f82-44cd-9038-2e54f790cc1f

dn: CN=Default Query Policy,CN=Query-Policies,CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,DC=isis-klif,DC=unige,DC=ch
lDAPAdminLimits: MaxValRange=1500

Adding the searchResultHandler property to the groupLdap1 LDAP connector configuration does not help:

ldap.groupLdap1.searchResultHandlers=org.ldaptive.handler.DnAttributeEntryHandler,edu.internet2.middleware.grouper.ldap.ldaptive.GrouperRangeEntryHandler

cf. vt-ldap to ldaptive migration for LDAP access
https://spaces.at.internet2.edu/display/Grouper/vt-ldap+to+ldaptive+migration+for+LDAP+access#vt-ldaptoldaptivemigrationforLDAPaccess-Configurationoptions

And of course the LdapProvisioner isActiveDirectory property is set:

changeLog.consumer.activedirectory_student.isActiveDirectory = true

A search in grouper-users shows old problems, in particular:

Re: [grouper-users] Active Directory >1500 group members
https://lists.internet2.edu/sympa/arc/grouper-users/2016-11/msg00005.html

Re: [grouper-users] AD Range Attribute support with the PSP
https://lists.internet2.edu/sympa/arc/grouper-users/2017-01/msg00012.html

There is a Jira issue GRP-2343 that shows the same symptom, although it is considered minor and is not fixed:

LdapSystem.performLdapSearchRequest doesn't return any members when group has more than 1500 memberships
https://todos.internet2.edu/browse/GRP-2343

- What could be the problem? Surely PSNG works in other sites with large Active Directory groups!

There must be something obvious I am missing, but I am at loss of ideas.

Thanks in advance for your lights!

Regards.

Annexe: Relevant extract of the logs:

2020-09-18 07:30:06,640: [main] WARN  GrouperStartup.printConfigOnce(232) -  - Grouper starting up: version: 2.4.0, build date: null, env: test_2.4.0_grouper-loader
grouperPatchStatus read from: /usr/local/grouper-loader/grouperPatchStatus.properties
api patches installed:        0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96
pspng patches installed:      0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
...
2020-09-18 08:23:11,479: [main] DEBUG LdapProvisionerConfiguration.readConfiguration(126) -  - Ldap Provisioner activedirectory_student - Setting isActiveDirectory to true
...
2020-09-18 08:23:21,963: [FullSyncer(activedirectory_student)-Thread] INFO  ProgressMonitor.completelyDone(81) -  - Fetching subjects Completed (Success): 1501 items in 08.833s (10195.9 items/minute)
...
2020-09-18 08:23:21,977: [FullSyncer(activedirectory_student)-Thread] INFO  Provisioner.doFullSync(1591) -  - activedirectory_student-full/application:bpm-poc:bpm-oracle:workflow1:testgroup/#16234(Existing): 1501 correct member subjects. Sample: [''/'person'/'people-test.unige.ch', ''/'person'/'people-test.unige.ch', ''/'person'/'people-test.unige.ch', ''/'person'/'people-test.unige.ch', '674039@u
nige.ch'/'person'/'people-test.unige.ch', ''/'person'/'people-test.unige.ch', ''/'person'/'people-test.unige.ch', ''/'person'/'people-test.unige.ch', ''/'person'/'people-test.unige.ch', ''/'person'/'people-test.unige.ch']...
...
2020-09-18 08:23:22,076: [FullSyncer(activedirectory_student)-Thread] INFO  LdapSystem.performLdapRead(688) -  - Active Directory: Searching with Ldap RangeEntryHandler
...
2020-09-18 08:23:51,589: [FullSyncer(activedirectory_student)-Thread] INFO  LdapGroupProvisioner.doFullSync(194) -  - activedirectory_student-full: Full-sync comparison for application:bpm-poc:bpm-oracle:workflow1:testgroup/#16234(Existing): Target-subject count: Correct/Actual: 1501/0
...
2020-09-18 08:23:51,765: [FullSyncer(activedirectory_student)-Thread] WARN  LdapSystem.performLdapModify(450) -  - groupLdap1: Problem while modifying ldap system based on grouper expectations. Starting to perform adaptive modifications based on data already on server: cn=testgroup,ou=workflow1,ou=bpm-oracle,ou=bpm-poc,ou=application,ou=student,ou=grouper,ou=groups,ou=_unige,dc=isis-klif,dc=unige,dc=ch: ENTRY_ALREADY_EXISTS

-- 
Mr Dominique Petitpierre, user=Dominique.Petitpierre domain=unige.ch
IT Division, University of Geneva, Switzerland

--

Jeffrey Williams 

Identity & Access Engineer
Identity & Access Services
https://its.uncg.edu