Grouper Users - Open Discussion List

[Yoann Delattre <>]

Hi Jeffrey,

thanks for the suggestion.
according to the logs, cache size seems correct (but not used ??) :

Mar  2 10:28:58 grouper2 grouper-api-pspng[8117]: 2020-03-02 10:28:58,261: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.warnAboutCacheSizeConcerns(684) -  - pspng_brancheGrouper: Cache of grouper groups is sufficiently sized (0% full) (property grouperGroupCacheSize=10000)
Mar  2 10:28:58 grouper2 grouper-api-pspng[8117]: 2020-03-02 10:28:58,261: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.warnAboutCacheSizeConcerns(684) -  - pspng_brancheGrouper: Cache of provisioned groups is sufficiently sized (0% full) (property targetSystemGroupCacheSize=10000)
Mar  2 10:28:58 grouper2 grouper-api-pspng[8117]: 2020-03-02 10:28:58,261: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.warnAboutCacheSizeConcerns(684) -  - pspng_brancheGrouper: Cache of grouper subjects is sufficiently sized (0% full) (property grouperSubjectCacheSize=10000)
Mar  2 10:28:58 grouper2 grouper-api-pspng[8117]: 2020-03-02 10:28:58,261: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.warnAboutCacheSizeConcerns(684) -  - pspng_brancheGrouper: Cache of provisioned subjects is sufficiently sized (0% full) (property targetSystemUserCacheSize=10000)

For me, it's not ldap related, this process check if a group need to be provisioning by a specific provisioner or not. In our case, it's take a lot of time to evaluate every group with the JEXL _expression_.

If someone can confirm but, apparently, a full groups selection evaluation are performed only when the grouperDaemon restart. it means that, for us, grouperDaemon need more than one hour to be fully functional.

It's not a problem (even less with multiple grouperDaemon) but it could be if this evaluation is launch regularly.

Thanks !

Yoann

Le 02/03/2020 à 17:53, Crawford, Jeffrey a écrit :

Hi Yoann,

 

Try adding the following two to your provisioner configs. If you run out of cache space it may be performing excessive searches.

 

…grouperSubjectCacheSize = 1000000

…targetSystemUserCacheSize = 1000000

 

 

Adjust the actual number to around how many you expect to be loading, and make sure you are running with enough memory.

 

Obviously the regular check of LDAP indexes being applied correctly still apply 😊

 

Jeffrey C.

 

 

From: on behalf of Yoann Delattre
Reply-To: Yoann Delattre
Date: Monday, March 2, 2020 at 5:10 AM
To: Grouper Users
Subject: [grouper-users] Slow incremental provisioning with PSPNG

 

Hello everyone,

I just upgraded to 2.4 and i use PSPNG with latest patches (12).
Since the upgrade, processing change log entries can take a lot of times (up to 6hours for 110k entries).

I launched PSPNG with debug log and there is a lot of lines like this :

Mar  2 10:20:44 grouper2.in.ac-lille.fr grouper-api-pspng[8117]: 2020-03-02 10:20:44,888: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.evaluateJexlExpression(777) -  - Evaluated GroupSelection Jexl _expression_: 'true'
Mar  2 10:20:44 grouper2.in.ac-lille.fr grouper-api-pspng[8117]: 2020-03-02 10:20:44,889: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.evaluateJexlExpression(797) -  - Evaluated entire GroupSelection Jexl _expression_: 'true'     Mar  2 10:20:44 grouper2 grouper-api-pspng[8117]: 2020-03-02 10:20:44,889: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.shouldGroupBeProvisioned(1823) -  - pspng_brancheGrouper-full: Group etab-pub:1d:ens:circ:0620235U:direction/#27033(Existing) matches group-selection filter.

It's take around 40 min to evaluate all the groups.

See log attached.

Below the config for all provisioners :

 

## Alimentation des groupes dans la branche ou=Grouper,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheGrouper.class =

edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim

changeLog.consumer.pspng_brancheGrouper.type =

edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner

changeLog.consumer.pspng_brancheGrouper.quartzCron =

0 * * * * ? changeLog.consumer.pspng_brancheGrouper.ldapPoolName =

ldapLille changeLog.consumer.pspng_brancheGrouper.memberAttributeName =

uniqueMember

changeLog.consumer.pspng_brancheGrouper.memberAttributeValueFormat =

${ldapUser.getDn()}

changeLog.consumer.pspng_brancheGrouper.groupSearchBaseDn =

ou=Grouper,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheGrouper.allGroupsSearchFilter =

objectclass=groupOfUniqueNames

changeLog.consumer.pspng_brancheGrouper.singleGroupSearchFilter =

(&(objectclass=groupOfUniqueNames)(cn=${group.name}))

changeLog.consumer.pspng_brancheGrouper.groupSearchAttributes =

cn,objectclass

changeLog.consumer.pspng_brancheGrouper.groupCreationLdifTemplate =

dn: cn=${group.name}||cn: ${group.name}||description: ${group.description}||ou: ${group.displayName}||objectclass: groupOfUniqueNames||objectclass: educationnationale

changeLog.consumer.pspng_brancheGrouper.groupSelectionExpression =

${!name.endsWith("_systemOfRecord") && !name.endsWith("_systemOfRecordAndIncludes") && !name.endsWith("_includes") && !name.endsWith("_excludes")}

changeLog.consumer.pspng_brancheGrouper.userSearchBaseDn =

ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheGrouper.userSearchFilter =

uid=${subject.id}

changeLog.consumer.pspng_brancheGrouper.grouperIsAuthoritative =

true ## Alimentation des groupes dans la branche ou=listes,ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheListes.class =

edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim

changeLog.consumer.pspng_brancheListes.type =

edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner

changeLog.consumer.pspng_brancheListes.quartzCron =

0 * * * * ? changeLog.consumer.pspng_brancheListes.ldapPoolName =

ldapLille changeLog.consumer.pspng_brancheListes.memberAttributeName =

uniqueMember

changeLog.consumer.pspng_brancheListes.memberAttributeValueFormat =

${ldapUser.getDn()}

changeLog.consumer.pspng_brancheListes.groupSearchBaseDn =

ou=listes,ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheListes.allGroupsSearchFilter =

(&(objectclass=groupOfUniqueNames)(typensi=grouper))

changeLog.consumer.pspng_brancheListes.singleGroupSearchFilter =

(&(objectclass=groupOfUniqueNames)(typensi=grouper)(cn=${group.extension}))

changeLog.consumer.pspng_brancheListes.groupSearchAttributes =

cn,objectclass,typensi

changeLog.consumer.pspng_brancheListes.groupCreationLdifTemplate =

dn: cn=${group.extension}||cn: ${group.extension}||objectclass: groupOfUniqueNames||objectclass: educationnationale||typensi: grouper

changeLog.consumer.pspng_brancheListes.userSearchBaseDn =

ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheListes.userSearchFilter =

uid=${subject.id}

changeLog.consumer.pspng_brancheListes.grouperIsAuthoritative =

true ## Alimentation de l'attribut FrEduLilHabilitation dans la branche ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_attrFrEduLilHabilitation.class =

edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim

changeLog.consumer.pspng_attrFrEduLilHabilitation.type =

edu.internet2.middleware.grouper.pspng.LdapAttributeProvisioner

changeLog.consumer.pspng_attrFrEduLilHabilitation.quartzCron =

0 * * * * ? changeLog.consumer.pspng_attrFrEduLilHabilitation.ldapPoolName

= ldapLille

changeLog.consumer.pspng_attrFrEduLilHabilitation.provisionedAttributeName =

FrEduLilHabilitation

changeLog.consumer.pspng_attrFrEduLilHabilitation.provisionedAttributeValueFormat =

Grouper|${group.name}

changeLog.consumer.pspng_attrFrEduLilHabilitation.userSearchBaseDn =

ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_attrFrEduLilHabilitation.userSearchFilter =

uid=${subject.id}

changeLog.consumer.pspng_attrFrEduLilHabilitation.groupSelectionExpression =

${name.startsWith("app:") && name.contains(":habil:")}

changeLog.consumer.pspng_attrFrEduLilHabilitation.grouperIsAuthoritative =

true changeLog.consumer.pspng_attrFrEduLilHabilitation.allProvisionedValuesPrefix

= Grouper\\|

Is there a way to improve performance ?

Maybe i need to stop using the JEXL _expression_ and used only provisoning attributes ?

Any suggestion ?

Thanks a lot !

Regards,

Yoann

--

Yoann Delattre

✆

03 20 95 69 10

✉

Équipe SIAD (Systèmes d'Information et Aide à la Décision)
DSI de l'académie de Lille (Direction des Systèmes d'Information)
110 avenue Gaston Berger - 59000 Lille