Grouper Users - Open Discussion List

["Crawford, Jeffrey" <>]

Hi Yoann,

 

Try adding the following two to your provisioner configs. If you run out of cache space it may be performing excessive searches.

 

…grouperSubjectCacheSize = 1000000

…targetSystemUserCacheSize = 1000000

 

 

Adjust the actual number to around how many you expect to be loading, and make sure you are running with enough memory.

 

Obviously the regular check of LDAP indexes being applied correctly still apply 😊

 

Jeffrey C.

 

 

From: <> on behalf of Yoann Delattre <>
Reply-To: Yoann Delattre <>
Date: Monday, March 2, 2020 at 5:10 AM
To: Grouper Users <>
Subject: [grouper-users] Slow incremental provisioning with PSPNG

 

Hello everyone,

I just upgraded to 2.4 and i use PSPNG with latest patches (12).
Since the upgrade, processing change log entries can take a lot of times (up to 6hours for 110k entries).

I launched PSPNG with debug log and there is a lot of lines like this :

Mar  2 10:20:44 grouper2.in.ac-lille.fr grouper-api-pspng[8117]: 2020-03-02 10:20:44,888: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.evaluateJexlExpression(777) -  - Evaluated GroupSelection Jexl _expression_: 'true'
Mar  2 10:20:44 grouper2.in.ac-lille.fr grouper-api-pspng[8117]: 2020-03-02 10:20:44,889: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.evaluateJexlExpression(797) -  - Evaluated entire GroupSelection Jexl _expression_: 'true'     Mar  2 10:20:44 grouper2 grouper-api-pspng[8117]: 2020-03-02 10:20:44,889: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.shouldGroupBeProvisioned(1823) -  - pspng_brancheGrouper-full: Group etab-pub:1d:ens:circ:0620235U:direction/#27033(Existing) matches group-selection filter.

It's take around 40 min to evaluate all the groups.

See log attached.

Below the config for all provisioners :

 

## Alimentation des groupes dans la branche ou=Grouper,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheGrouper.class =

edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim

changeLog.consumer.pspng_brancheGrouper.type =

edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner

changeLog.consumer.pspng_brancheGrouper.quartzCron =

0 * * * * ? changeLog.consumer.pspng_brancheGrouper.ldapPoolName =

ldapLille changeLog.consumer.pspng_brancheGrouper.memberAttributeName =

uniqueMember

changeLog.consumer.pspng_brancheGrouper.memberAttributeValueFormat =

${ldapUser.getDn()}

changeLog.consumer.pspng_brancheGrouper.groupSearchBaseDn =

ou=Grouper,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheGrouper.allGroupsSearchFilter =

objectclass=groupOfUniqueNames

changeLog.consumer.pspng_brancheGrouper.singleGroupSearchFilter =

(&(objectclass=groupOfUniqueNames)(cn=${group.name}))

changeLog.consumer.pspng_brancheGrouper.groupSearchAttributes =

cn,objectclass

changeLog.consumer.pspng_brancheGrouper.groupCreationLdifTemplate =

dn: cn=${group.name}||cn: ${group.name}||description: ${group.description}||ou: ${group.displayName}||objectclass: groupOfUniqueNames||objectclass: educationnationale

changeLog.consumer.pspng_brancheGrouper.groupSelectionExpression =

${!name.endsWith("_systemOfRecord") && !name.endsWith("_systemOfRecordAndIncludes") && !name.endsWith("_includes") && !name.endsWith("_excludes")}

changeLog.consumer.pspng_brancheGrouper.userSearchBaseDn =

ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheGrouper.userSearchFilter =

uid=${subject.id}

changeLog.consumer.pspng_brancheGrouper.grouperIsAuthoritative =

true ## Alimentation des groupes dans la branche ou=listes,ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheListes.class =

edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim

changeLog.consumer.pspng_brancheListes.type =

edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner

changeLog.consumer.pspng_brancheListes.quartzCron =

0 * * * * ? changeLog.consumer.pspng_brancheListes.ldapPoolName =

ldapLille changeLog.consumer.pspng_brancheListes.memberAttributeName =

uniqueMember

changeLog.consumer.pspng_brancheListes.memberAttributeValueFormat =

${ldapUser.getDn()}

changeLog.consumer.pspng_brancheListes.groupSearchBaseDn =

ou=listes,ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheListes.allGroupsSearchFilter =

(&(objectclass=groupOfUniqueNames)(typensi=grouper))

changeLog.consumer.pspng_brancheListes.singleGroupSearchFilter =

(&(objectclass=groupOfUniqueNames)(typensi=grouper)(cn=${group.extension}))

changeLog.consumer.pspng_brancheListes.groupSearchAttributes =

cn,objectclass,typensi

changeLog.consumer.pspng_brancheListes.groupCreationLdifTemplate =

dn: cn=${group.extension}||cn: ${group.extension}||objectclass: groupOfUniqueNames||objectclass: educationnationale||typensi: grouper

changeLog.consumer.pspng_brancheListes.userSearchBaseDn =

ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheListes.userSearchFilter =

uid=${subject.id}

changeLog.consumer.pspng_brancheListes.grouperIsAuthoritative =

true ## Alimentation de l'attribut FrEduLilHabilitation dans la branche ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_attrFrEduLilHabilitation.class =

edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim

changeLog.consumer.pspng_attrFrEduLilHabilitation.type =

edu.internet2.middleware.grouper.pspng.LdapAttributeProvisioner

changeLog.consumer.pspng_attrFrEduLilHabilitation.quartzCron =

0 * * * * ? changeLog.consumer.pspng_attrFrEduLilHabilitation.ldapPoolName

= ldapLille

changeLog.consumer.pspng_attrFrEduLilHabilitation.provisionedAttributeName =

FrEduLilHabilitation

changeLog.consumer.pspng_attrFrEduLilHabilitation.provisionedAttributeValueFormat =

Grouper|${group.name}

changeLog.consumer.pspng_attrFrEduLilHabilitation.userSearchBaseDn =

ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_attrFrEduLilHabilitation.userSearchFilter =

uid=${subject.id}

changeLog.consumer.pspng_attrFrEduLilHabilitation.groupSelectionExpression =

${name.startsWith("app:") && name.contains(":habil:")}

changeLog.consumer.pspng_attrFrEduLilHabilitation.grouperIsAuthoritative =

true changeLog.consumer.pspng_attrFrEduLilHabilitation.allProvisionedValuesPrefix

= Grouper\\|

Is there a way to improve performance ?

Maybe i need to stop using the JEXL _expression_ and used only provisoning attributes ?

Any suggestion ?

Thanks a lot !

Regards,

Yoann

--

Yoann Delattre	✆	03 20 95 69 10	✉
Équipe SIAD (Systèmes d'Information et Aide à la Décision) DSI de l'académie de Lille (Direction des Systèmes d'Information) 110 avenue Gaston Berger - 59000 Lille