Grouper Users - Open Discussion List

[Dominique Petitpierre <>]

Hello,

lots of testing show that the proposed change to LdapSystem.java fixes the issue:
full sync to an Active Directory target directory now also resolves discrepancies in large groups (more that 1500 (or MaxValRange) members).

- performLdapSearchRequest now returns all members of a large group in the member attribute array (instead of returning an empty array).

- full sync now can count them properly (instead of finding 0 members in large groups) and thus does not try anymore to re-add the first 1500 members (which caused the ENTRY_ALREADY_EXISTS warning).

- if members are added to or removed from a large target group, full sync now removes or, respectively, restores the members (instead of changing nothing).

- incremental syncing is not affected.

- bonus: because full sync does not try to re-add 1500 members (with retries) for all large groups, the total synchronizing time is reduced (in some proportion to the memberships affected vs total memberships).

   In my case a provisioner's full sync, that has nothing to reconciliate, that took 1h26 before the correction now finishes in 43 minutes (50% gain), and another one went from 3h57 to 1h21 (62% gain)!

As mentioned in a previous message (https://lists.internet2.edu/sympa/arc/grouper-users/2020-09/msg00042.html) that part of LdapSystem.java has not changed since Grouper 2.4.0 patch 8,
so this fix should also be beneficial to 2.5.x installations that have a PSPNG provisioner syncing to large Active Directory groups.

- If you try the change, I'd be interested in your findings!

Regards.

On 24.09.20 05:41, Dominique Petitpierre wrote:

Hello,

On 24.09.20 04:57, Dominique Petitpierre wrote:

      Currently
        I am testing a fix: it looks like it corrects the problem and
        makes the full sync job finish quicker.

No spurious LDAP MOD so far (no more ENTRY_ALREADY_EXISTS warnings), and it resolved the leftover test discrepancy.
Still waiting for the second long full sync to finish.

Here is the proposed fix (original from https://github.com/Internet2/grouper/archive/grouper_2_4_0-a95-u57-w11-p12.zip):

diff -c ./src/main/java/edu/internet2/middleware/grouper/pspng/LdapSystem.java{.01,}
*** ./src/main/java/edu/internet2/middleware/grouper/pspng/LdapSystem.java.01    2020-03-24 19:45:37.000000000 +0100
--- ./src/main/java/edu/internet2/middleware/grouper/pspng/LdapSystem.java    2020-09-24 00:53:41.808924113 +0200
***************
*** 738,750 ****
        // Turn on attribute-value paging if this is an active directory target
        if ( isActiveDirectory() ) {
          LOG.debug("Using attribute-value paging");
!         request.setSearchEntryHandlers(new RangeEntryHandler());
        }
! 
!       request.setSearchEntryHandlers(
!               new LdapSearchProgressHandler(approximateNumResultsExpected, LOG, "Performing ldap search"),
!               callback);
! 
        // Perform search. This is slightly different if paging is enabled or not.
        if ( isSearchResultPagingEnabled() ) {
          PagedResultsClient client = new PagedResultsClient(conn, getSearchResultPagingSize());
--- 738,755 ----
        // Turn on attribute-value paging if this is an active directory target
        if ( isActiveDirectory() ) {
          LOG.debug("Using attribute-value paging");
!         request.setSearchEntryHandlers(
!                 new RangeEntryHandler(),
!                 new LdapSearchProgressHandler(approximateNumResultsExpected, LOG, "Performing ldap search"),
!                 callback);
        }
!       else {
!         LOG.debug("Not using attribute-value paging");
!         request.setSearchEntryHandlers(
!                 new LdapSearchProgressHandler(approximateNumResultsExpected, LOG, "Performing ldap search"),
!                 callback);
!       }
!       
        // Perform search. This is slightly different if paging is enabled or not.
        if ( isSearchResultPagingEnabled() ) {
          PagedResultsClient client = new PagedResultsClient(conn, getSearchResultPagingSize());

Comments from the developers are welcome!
In particular if it is OK to put RangeEntryHandler first in the list of SearchEntryHandlers.

Regards.

-- 
Mr Dominique Petitpierre, user=Dominique.Petitpierre domain=unige.ch
IT Division, University of Geneva, Switzerland

-- 
Mr Dominique Petitpierre, user=Dominique.Petitpierre domain=unige.ch
IT Division, University of Geneva, Switzerland