Grouper Developers Forum

["Gettes, Michael" <>]

After a bit more head-banging (and Carey pointing out my not seeing what Bert 
saw with top)

Here is the config I finally settled on and it works without the errors I was 
reporting (except for the caching message which Bert is kind enough to 
address in an upcoming patch).

The problem I was trying to solve was to provide groups via PSPNG into AD 
preserving DIT structure for compatibility so I can use Grouper to take over 
already provided groups through custom software.  I settled on using 
displayName as a means of linking Grouper and AD instead of the traditional 
CN which was clearly getting me in trouble.  On the user objects we have 
usernames that end in ‘.’ which is not allowed for samAccountName.

I am terribly sorry to have wasted Bert’s time with my mistakes and I am 
grateful to Bert and Carey for taking time to help me.

/mrg

changeLog.consumer.psp_UFADdev.provisionerName = psp_UFADdev
changeLog.consumer.psp_UFADdev.class = 
edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim
changeLog.consumer.psp_UFADdev.type = 
edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner
changeLog.consumer.psp_UFADdev.quartzCron = 1/20 * * * * ?
changeLog.consumer.psp_UFADdev.ldapPoolName = UFADdev
changeLog.consumer.psp_UFADdev.retryOnError = false
changeLog.consumer.psp_UFADdev.isActiveDirectory = true
changeLog.consumer.psp_UFADdev.grouperIsAuthoritative = true
changeLog.consumer.psp_UFADdev.memberAttributeName = member
changeLog.consumer.psp_UFADdev.memberAttributeValueFormat = 
${ldapUser.getDn()}
changeLog.consumer.psp_UFADdev.groupSearchBaseDn = 
OU=Grouper,OU=Groups,OU=UF,DC=dev-ad,DC=ufl,DC=edu
changeLog.consumer.psp_UFADdev.groupCreationBaseDn = 
OU=Grouper,OU=Groups,OU=UF,DC=dev-ad,DC=ufl,DC=edu
changeLog.consumer.psp_UFADdev.allGroupsSearchFilter = 
(&(objectclass=group)(objectclass=posixGroup))
changeLog.consumer.psp_UFADdev.singleGroupSearchFilter = 
(&(objectclass=group)(displayName=${group.name}))
changeLog.consumer.psp_UFADdev.groupSearchAttributes = 
cn,gidNumber,objectclass,samAccountName,name,displayName
changeLog.consumer.psp_UFADdev.groupCreationLdifTemplate = dn: 
${utils.bushyDn("${group.name.replaceAll('(App:.+:UF:Groups:)(.+):(.*)','$2:'+'$3'.replace(':','_'))}","cn","ou")}||objectclass:
 top||objectclass: group||objectclass: posixGroup||gidNumber: 
${group.idIndex}||displayName: ${group.name}
changeLog.consumer.psp_UFADdev.userSearchBaseDn = 
OU=People,OU=UF,DC=dev-ad,DC=ufl,DC=edu
changeLog.consumer.psp_UFADdev.userSearchFilter = 
userPrincipalName=${subject.id}
changeLog.consumer.psp_UFADdev.userSearchAttributes = 
dn,cn,uid,mail,samAccountName,uidNumber,objectclass,memberOf,userPrincipalName
changeLog.consumer.psp_UFADdev.groupSearch_batchSize = 100
changeLog.consumer.psp_UFADdev.ldapSearchResultPagingSize = 1000
changelog.consumer.psp_UFADdev.grouperSubjectCacheSize = 400000
changelog.consumer.psp_UFADdev.grouperGroupCacheSize = 50000
changelog.consumer.psp_UFADdev.ldapUserCacheSize = 400000
changelog.consumer.psp_UFADdev.ldapUserCacheTime_secs = 3600
changelog.consumer.psp_UFADdev.maxValuesToChangePerOperation = 900
changeLog.consumer.psp_UFADdev.needsTargetSystemUsers = true
changeLog.consumer.psp_UFADdev.supportsEmptyGroups = true
changeLog.consumer.psp_UFADdev.createMissingUsers = true
changeLog.consumer.psp_UFADdev.userCreationBaseDn = 
OU=People,OU=UF,DC=dev-ad,DC=ufl,DC=edu
changeLog.consumer.psp_UFADdev.userCreationLdifTemplate = dn: 
CN=${subject.getId().toLowerCase().replaceAll('@ufl.edu','').replaceAll('\\.$','')}||sAMAccountName:
 
${subject.getId().toLowerCase().replaceAll('@ufl.edu','').replaceAll('\\.$','')}||objectclass:
 top||objectclass: person||objectclass: user||sn: 
${subject.getId().toLowerCase().replaceAll('@ufl.edu','')}||userPrincipalName:
 ${subject.getId()}||displayName: ${subject.name}

> On Apr 29, 2019, at 4:43 PM, Gettes, Michael <> wrote:
> 
> see inline...
> 
>> On Apr 29, 2019, at 12:58 PM, Bee-Lindgren, Bert 
>> <> wrote:
>> 
>> Hello,
>> 
>> 1) Please add the top objectclass to your group template, though this 
>> problem looks different to me
> 
> objectclass: top is already part of the template.
> 
>> 2) Can you privately send me what you replaced with ' > … (long list of 
>> DNs) … ‘?
> 
> being sent separately.
> 
>> 3) (particularly with PSPNG p5), Can you run it with INFO or DEBUG log 
>> level and send the result to me?
> 
> being sent separately.
> 
>> 4) I would suggest adding gidNumber to your groupCreationTemplate and then 
>> (after gidNumbers are fully provisioned by a full-sync sweep) using 
>> gidNumber-based searching in your singleGroupSearchFilter
> 
> Done.  Much smarter way to go for Active Directory - i think this should be 
> more strongly recommended in the documentation.
> 
>> 
>> 
>> Thanks,
>>  Bert
>> 
>> 
>> 
>> 
>> From:  
>> <> on behalf of Gettes, Michael 
>> <>
>> Sent: Friday, April 26, 2019 11:04 PM
>> To: 
>> Subject: [grouper-dev] PSPNG error repeating (latest)
>> 
>> Going against AD - this error keeps repeating.  It’s a group with 17K 
>> members.  my pspng config is below.  This is on tier/grouper:latest 
>> published today (2.4.0-a42-u23-w5-p4-20190426-rc1).
>> 
>> 2019-04-27T02:19:09+00:00 DAEMON:dev 
>> grouper-api;grouper.log;grouper_dev;daemon;2019-04-26 22:19:09,381: 
>> [FullSyncer(psp_UFADdev)-Thread] WARN  LdapSystem.performLdapModify(405) - 
>>  - UFADdev: Problem while modifying ldap system based on grouper 
>> expectations. Starting to perform adaptive modifications based on data 
>> already on server: [org.ldaptive.ModifyRequest@1988375000::modifyDn
>> … (long list of DNs) … 
>> controls=null, referralHandler=null, intermediateResponseHandlers=null]: 
>> ENTRY_ALREADY_EXISTS
>> 2019-04-27T02:15:38+00:00 DAEMON:dev 
>> grouper-api;grouper.log;grouper_dev;daemon;2019-04-26 22:15:38,044: 
>> [FullSyncer(psp_UFADdev)-Thread] ERROR 
>> FullSyncProvisioner.fullSyncGroup(739) -  - FullSyncer(psp_UFADdev): 
>> Problem doing full sync. Requeuing group 
>> App:UFAD:UF:Groups:Services:Zoom:Service uflphi
>> 2019-04-27T02:15:38+00:00 DAEMON:dev java.lang.IllegalStateException: The 
>> active provisioner should be defined when creating LdapObjects
>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>> edu.internet2.middleware.grouper.pspng.LdapObject.<init>(LdapObject.java:95)
>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>> edu.internet2.middleware.grouper.pspng.LdapSystem.performLdapRead(LdapSystem.java:626)
>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>> edu.internet2.middleware.grouper.pspng.LdapSystem.performLdapRead(LdapSystem.java:596)
>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>> edu.internet2.middleware.grouper.pspng.LdapSystem.performLdapModify(LdapSystem.java:448)
>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>> edu.internet2.middleware.grouper.pspng.LdapSystem.performLdapModify(LdapSystem.java:377)
>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>> edu.internet2.middleware.grouper.pspng.LdapProvisioner.makeIndividualLdapChanges(LdapProvisioner.java:713)
>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>> edu.internet2.middleware.grouper.pspng.LdapProvisioner.finishProvisioningBatch(LdapProvisioner.java:455)
>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner.fullSyncGroup(FullSyncProvisioner.java:721)
>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner.processQueueItem(FullSyncProvisioner.java:433)
>> 9-04-27T02:19:09+00:00 DAEMON:dev        at 
>> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner.thread_manageFullSyncProcessing(FullSyncProvisioner.java:254)
>> 2019-04-27T02:19:09+00:00 DAEMON:dev     at 
>> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner$1.run(FullSyncProvisioner.java:146)
>> 2019-04-27T02:19:09+00:00 DAEMON:dev     at 
>> java.lang.Thread.run(Thread.java:748)
>> 2019-04-27T02:53:48+00:00 DAEMON:dev 
>> grouper-api;grouper.log;grouper_dev;daemon;2019-04-26 22:53:48,840: 
>> [FullSyncer(psp_UFADdev)-Thread] WARN  
>> ProvisionerCoordinator$ProvisioningStatus.lockForFullSyncWhenNoIncrementalIsUnderway(73)
>>  -  - psp_UFADdev: Cannot start FullSync of 
>> App:UFAD:UF:Groups:Services:Zoom:Service uflphi. Incremental provisioning 
>> underway since Fri Apr 26 22:53:48 EDT 2019. We'll give up and move ahead 
>> anyway in 300 seconds.
>> 
>> ———
>> 
>> AND this is a poorly formatted message that comes next:  not sure what i 
>> should do to address it if anything.
>> 
>> 2019-04-27T02:19:09+00:00 DAEMON:dev 
>> grouper-api;grouper.log;grouper_dev;daemon;2019-04-26 22:19:09,461: 
>> [FullSyncer(psp_UFADdev)-Thread] WARN  
>> Provisioner.warnAboutCacheSizeConcerns(620) -  - Cache is very full 
>> (%.0f%%). Performance is much higher if 100.0 is large enough to hold the 
>> number provisioned groups or subjects
>> 
>> ———
>> 
>> pspng config:
>> 
>> ldap.UFADdev.pass = XXXXXXXXXXXX
>> ldap.UFADdev.user = XXXXXXXXXXXXXXXXXXXXXXXXX
>> ldap.UFADdev.url = ldap://ufadXXXXXXXXXXXXXXXXX.ufl.edu
>> ldap.UFADdev.pagedResultsSize = 1000
>> ldap.UFADdev.timeout = 100000
>> ldap.UFADdev.tls = false
>> ldap.UFADdev.searchResultHandlers=org.ldaptive.handler.DnAttributeEntryHandler,edu.internet2.middleware.grouper.ldap.ldaptive.GrouperRangeEntryHandler
>> # comma-delimited list of result codes (org.ldaptive.ResultCode) to 
>> ignore, e.g. TIME_LIMIT_EXCEEDED, SIZE_LIMIT_EXCEEDED, PARTIAL_RESULTS
>> ldap.UFADdev.searchIgnoreResultCodes=SIZE_LIMIT_EXCEEDED,ATTRIBUTE_OR_VALUE_EXISTS
>> 
>> changeLog.consumer.psp_UFADdev.provisionerName = psp_UFADdev
>> changeLog.consumer.psp_UFADdev.class = 
>> edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim
>> changeLog.consumer.psp_UFADdev.type = 
>> edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner
>> changeLog.consumer.psp_UFADdev.quartzCron = 0/15 * * * * ?
>> changeLog.consumer.psp_UFADdev.ldapPoolName = UFADdev
>> changeLog.consumer.psp_UFADdev.retryOnError = false
>> changeLog.consumer.psp_UFADdev.isActiveDirectory = true
>> changeLog.consumer.psp_UFADdev.grouperIsAuthoritative = true
>> changeLog.consumer.psp_UFADdev.memberAttributeName = member
>> changeLog.consumer.psp_UFADdev.memberAttributeValueFormat = 
>> ${ldapUser.getDn()}
>> changeLog.consumer.psp_UFADdev.groupSearchBaseDn = 
>> OU=Grouper,OU=Groups,OU=UF,DC=dev-ad,DC=ufl,DC=edu
>> changeLog.consumer.psp_UFADdev.groupCreationBaseDn = 
>> OU=Grouper,OU=Groups,OU=UF,DC=dev-ad,DC=ufl,DC=edu
>> changeLog.consumer.psp_UFADdev.allGroupsSearchFilter = objectclass=group
>> changeLog.consumer.psp_UFADdev.singleGroupSearchFilter = 
>> (&(objectclass=group)(cn=${group.name.replaceAll("(App:.+:UF:Groups:)(.+):(.*)","$3").replace(":","_")}))
>> changeLog.consumer.psp_UFADdev.groupSearchAttributes = 
>> cn,gidNumber,objectclass,samAccountName,name
>> changeLog.consumer.psp_UFADdev.groupCreationLdifTemplate = dn: 
>> ${utils.bushyDn("${group.name.replaceAll('(App:.+:UF:Groups:)(.+):(.*)','$2:'+'$3'.replace(':','_'))}","cn","ou")}||objectclass:
>>  group
>> changeLog.consumer.psp_UFADdev.userSearchBaseDn = 
>> OU=People,OU=UF,DC=dev-ad,DC=ufl,DC=edu
>> changeLog.consumer.psp_UFADdev.userSearchFilter = 
>> userPrincipalName=${subject.id}
>> changeLog.consumer.psp_UFADdev.userSearchAttributes = 
>> dn,cn,uid,mail,samAccountName,uidNumber,objectclass,memberOf,userPrincipalName
>> changeLog.consumer.psp_UFADdev.groupSearch_batchSize = 100
>> changeLog.consumer.psp_UFADdev.ldapSearchResultPagingSize = 1000
>> changelog.consumer.psp_UFADdev.grouperSubjectCacheSize = 400000
>> changelog.consumer.psp_UFADdev.grouperGroupCacheSize = 50000
>> changelog.consumer.psp_UFADdev.ldapUserCacheSize = 200000
>> changelog.consumer.psp_UFADdev.ldapUserCacheTime_secs = 3600
>> changelog.consumer.psp_UFADdev.maxValuesToChangePerOperation = 500
>> changeLog.consumer.psp_UFADdev.needsTargetSystemUsers = true
>> changeLog.consumer.psp_UFADdev.supportsEmptyGroups = true
>> changeLog.consumer.psp_UFADdev.createMissingUsers = true
>> changeLog.consumer.psp_UFADdev.userCreationBaseDn = 
>> OU=People,OU=UF,DC=dev-ad,DC=ufl,DC=edu
>> changeLog.consumer.psp_UFADdev.userCreationLdifTemplate = dn: 
>> CN=${subject.getId().toLowerCase().replaceAll('@ufl.edu','').replaceAll('\\.$','')}||sAMAccountName:
>>  
>> ${subject.getId().toLowerCase().replaceAll('@ufl.edu','').replaceAll('\\.$','')}||objectclass:
>>  top||objectclass: person||objectclass: user||sn: 
>> ${subject.getId().toLowerCase().replaceAll('@ufl.edu','')}||userPrincipalName:
>>  ${subject.getId()}||displayName: ${subject.name}||cn: 
>> ${subject.getId().toLowerCase().replaceAll('@ufl.edu','')}
>