Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] Slow incremental provisioning with PSPNG

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] Slow incremental provisioning with PSPNG


Chronological Thread 
  • From: Yoann Delattre <>
  • To: "Crawford, Jeffrey" <>, "" <>
  • Subject: Re: [grouper-users] Slow incremental provisioning with PSPNG
  • Date: Mon, 9 Mar 2020 10:24:26 +0100
  • Arc-authentication-results: i=1; smtp.ac-lille.fr; auth=pass
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=ac-lille.fr; s=dkim201910; t=1583745868; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references; bh=c+DkvyfdqXpU+fXSm1Bfu80UYJU48m1IZLf/2yOGoiw=; b=qU9Opzn+dnUbTozDnB5DDdccJqXlGCp2tu29Dme45Wya8rSuDWIfW+w3u6rHvP3NR1w0+4 n7zxYUcRgdQxg+IYxgDuWMoo0hMvL0uVerCmXcSAQHAIM83RCTjPkOKzfVJusRHzRWzQux jkETif7/L+8zFdm6oKQ+uUnsvs5gl+fPpNZK31w10F47UB/ZFF+lh7yhipUviJr2qdh4wd MxzEdebM1ZVhImbFWapANB8AwjxFRDUg3p2KQM72tw9GJCfGZ2m3/ETUBjam4b50GggVrv 71z7XlNy1U4+xWM5m0qh7sAIl13wTwei1Xpi5vJDUL0HWEwrz8oz4CCFAsclyw==
  • Arc-seal: i=1; s=dkim201910; d=ac-lille.fr; t=1583745868; a=rsa-sha256; cv=none; b=sbsDpTUcahk1MloiGeks1RF217PBsBAeRZeie+PGZBUD1jGq2vph4LZBYsXzjNJ4+Gpbbw wpYAIFl+m2B1SNU5L9Y2awKH++MIgu6IbDKOyilNH2+MrrWwisWQKkrWwxdv+Xe2aRwMq+ h3CpVdbJhqBJu6LVjjBxjhRWbaHdN3OPHPkY6CRuWFs6NB2dJcnioigdtA3dX49qtiEf7c UTHwtfVmjeTU1QCpb40cuHVA2+HUVA1+kLxN3jTX8HTA7c9b3qyG6qFbkt6YFWb0Nw9Ksz Tjxc9sTDGC5R8cj73SOturOtit32Osti0rDTaWBESgDnOFirZD0/TghgztUqCg==

Hello,

i didn't dig enough in jira, i just found two issues related to my problem : GRP-2348 and GRP-2349.
Hope it will be solved soon !

Yoann.


Le 04/03/2020 à 13:57, Yoann Delattre a écrit :

Hi Jeffrey,

thanks for the suggestion.
according to the logs, cache size seems correct (but not used ??) :

Mar  2 10:28:58 grouper2 grouper-api-pspng[8117]: 2020-03-02 10:28:58,261: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.warnAboutCacheSizeConcerns(684) -  - pspng_brancheGrouper: Cache of grouper groups is sufficiently sized (0% full) (property grouperGroupCacheSize=10000)
Mar  2 10:28:58 grouper2 grouper-api-pspng[8117]: 2020-03-02 10:28:58,261: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.warnAboutCacheSizeConcerns(684) -  - pspng_brancheGrouper: Cache of provisioned groups is sufficiently sized (0% full) (property targetSystemGroupCacheSize=10000)
Mar  2 10:28:58 grouper2 grouper-api-pspng[8117]: 2020-03-02 10:28:58,261: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.warnAboutCacheSizeConcerns(684) -  - pspng_brancheGrouper: Cache of grouper subjects is sufficiently sized (0% full) (property grouperSubjectCacheSize=10000)
Mar  2 10:28:58 grouper2 grouper-api-pspng[8117]: 2020-03-02 10:28:58,261: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.warnAboutCacheSizeConcerns(684) -  - pspng_brancheGrouper: Cache of provisioned subjects is sufficiently sized (0% full) (property targetSystemUserCacheSize=10000)

For me, it's not ldap related, this process check if a group need to be provisioning by a specific provisioner or not. In our case, it's take a lot of time to evaluate every group with the JEXL _expression_.

If someone can confirm but, apparently, a full groups selection evaluation are performed only when the grouperDaemon restart. it means that, for us, grouperDaemon need more than one hour to be fully functional.
It's not a problem (even less with multiple grouperDaemon) but it could be if this evaluation is launch regularly.

Thanks !
Yoann

Le 02/03/2020 à 17:53, Crawford, Jeffrey a écrit :

Hi Yoann,

 

Try adding the following two to your provisioner configs. If you run out of cache space it may be performing excessive searches.

 

…grouperSubjectCacheSize = 1000000

…targetSystemUserCacheSize = 1000000

 

 

Adjust the actual number to around how many you expect to be loading, and make sure you are running with enough memory.

 

Obviously the regular check of LDAP indexes being applied correctly still apply 😊

 

Jeffrey C.

 

 

From: <> on behalf of Yoann Delattre <>
Reply-To: Yoann Delattre <>
Date: Monday, March 2, 2020 at 5:10 AM
To: Grouper Users <>
Subject: [grouper-users] Slow incremental provisioning with PSPNG

 

Hello everyone,

I just upgraded to 2.4 and i use PSPNG with latest patches (12).
Since the upgrade, processing change log entries can take a lot of times (up to 6hours for 110k entries).

I launched PSPNG with debug log and there is a lot of lines like this :

Mar  2 10:20:44 grouper2.in.ac-lille.fr grouper-api-pspng[8117]: 2020-03-02 10:20:44,888: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.evaluateJexlExpression(777) -  - Evaluated GroupSelection Jexl _expression_: 'true'
Mar  2 10:20:44 grouper2.in.ac-lille.fr grouper-api-pspng[8117]: 2020-03-02 10:20:44,889: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.evaluateJexlExpression(797) -  - Evaluated entire GroupSelection Jexl _expression_: 'true'     Mar  2 10:20:44 grouper2 grouper-api-pspng[8117]: 2020-03-02 10:20:44,889: [DefaultQuartzScheduler_Worker-10] DEBUG Provisioner.shouldGroupBeProvisioned(1823) -  - pspng_brancheGrouper-full: Group etab-pub:1d:ens:circ:0620235U:direction/#27033(Existing) matches group-selection filter.

It's take around 40 min to evaluate all the groups.

See log attached.

Below the config for all provisioners :

 

## Alimentation des groupes dans la branche ou=Grouper,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheGrouper.class =

edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim

changeLog.consumer.pspng_brancheGrouper.type =

edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner

changeLog.consumer.pspng_brancheGrouper.quartzCron =

0 * * * * ? changeLog.consumer.pspng_brancheGrouper.ldapPoolName =

ldapLille changeLog.consumer.pspng_brancheGrouper.memberAttributeName =

uniqueMember

changeLog.consumer.pspng_brancheGrouper.memberAttributeValueFormat =

${ldapUser.getDn()}

changeLog.consumer.pspng_brancheGrouper.groupSearchBaseDn =

ou=Grouper,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheGrouper.allGroupsSearchFilter =

objectclass=groupOfUniqueNames

changeLog.consumer.pspng_brancheGrouper.singleGroupSearchFilter =

(&(objectclass=groupOfUniqueNames)(cn=${group.name}))

changeLog.consumer.pspng_brancheGrouper.groupSearchAttributes =

cn,objectclass

changeLog.consumer.pspng_brancheGrouper.groupCreationLdifTemplate =

dn: cn=${group.name}||cn: ${group.name}||description: ${group.description}||ou: ${group.displayName}||objectclass: groupOfUniqueNames||objectclass: educationnationale

changeLog.consumer.pspng_brancheGrouper.groupSelectionExpression =

${!name.endsWith("_systemOfRecord") && !name.endsWith("_systemOfRecordAndIncludes") && !name.endsWith("_includes") && !name.endsWith("_excludes")}

changeLog.consumer.pspng_brancheGrouper.userSearchBaseDn =

ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheGrouper.userSearchFilter =

uid=${subject.id}

changeLog.consumer.pspng_brancheGrouper.grouperIsAuthoritative =

true ## Alimentation des groupes dans la branche ou=listes,ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheListes.class =

edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim

changeLog.consumer.pspng_brancheListes.type =

edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner

changeLog.consumer.pspng_brancheListes.quartzCron =

0 * * * * ? changeLog.consumer.pspng_brancheListes.ldapPoolName =

ldapLille changeLog.consumer.pspng_brancheListes.memberAttributeName =

uniqueMember

changeLog.consumer.pspng_brancheListes.memberAttributeValueFormat =

${ldapUser.getDn()}

changeLog.consumer.pspng_brancheListes.groupSearchBaseDn =

ou=listes,ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheListes.allGroupsSearchFilter =

(&(objectclass=groupOfUniqueNames)(typensi=grouper))

changeLog.consumer.pspng_brancheListes.singleGroupSearchFilter =

(&(objectclass=groupOfUniqueNames)(typensi=grouper)(cn=${group.extension}))

changeLog.consumer.pspng_brancheListes.groupSearchAttributes =

cn,objectclass,typensi

changeLog.consumer.pspng_brancheListes.groupCreationLdifTemplate =

dn: cn=${group.extension}||cn: ${group.extension}||objectclass: groupOfUniqueNames||objectclass: educationnationale||typensi: grouper

changeLog.consumer.pspng_brancheListes.userSearchBaseDn =

ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_brancheListes.userSearchFilter =

uid=${subject.id}

changeLog.consumer.pspng_brancheListes.grouperIsAuthoritative =

true ## Alimentation de l'attribut FrEduLilHabilitation dans la branche ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_attrFrEduLilHabilitation.class =

edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim

changeLog.consumer.pspng_attrFrEduLilHabilitation.type =

edu.internet2.middleware.grouper.pspng.LdapAttributeProvisioner

changeLog.consumer.pspng_attrFrEduLilHabilitation.quartzCron =

0 * * * * ? changeLog.consumer.pspng_attrFrEduLilHabilitation.ldapPoolName

= ldapLille

changeLog.consumer.pspng_attrFrEduLilHabilitation.provisionedAttributeName =

FrEduLilHabilitation

changeLog.consumer.pspng_attrFrEduLilHabilitation.provisionedAttributeValueFormat =

Grouper|${group.name}

changeLog.consumer.pspng_attrFrEduLilHabilitation.userSearchBaseDn =

ou=ac-lille,ou=education,o=gouv,c=fr

changeLog.consumer.pspng_attrFrEduLilHabilitation.userSearchFilter =

uid=${subject.id}

changeLog.consumer.pspng_attrFrEduLilHabilitation.groupSelectionExpression =

${name.startsWith("app:") && name.contains(":habil:")}

changeLog.consumer.pspng_attrFrEduLilHabilitation.grouperIsAuthoritative =

true changeLog.consumer.pspng_attrFrEduLilHabilitation.allProvisionedValuesPrefix

= Grouper\\|

Is there a way to improve performance ?

Maybe i need to stop using the JEXL _expression_ and used only provisoning attributes ?

Any suggestion ?

Thanks a lot !

Regards,

Yoann

--

Yoann Delattre

03 20 95 69 10

Équipe SIAD (Systèmes d'Information et Aide à la Décision)
DSI de l'académie de Lille (Direction des Systèmes d'Information)
110 avenue Gaston Berger - 59000 Lille

 




Archive powered by MHonArc 2.6.19.

Top of Page