Grouper Developers Forum

["Gettes, Michael" <>]

and I am sad to report that the fullsyncer seems to be complaining again.  
Same error - ENTRY_ALREADY_EXISTS.  I thought I had this fixed.  Guess not.  
I will try to isolate this problem and turn on DEBUG log.  It’s now the 17K 
member group with the problem.  Any pearls of wisdom of what to look for are 
appreciated.  (bang head here).

/mrg

> On Apr 30, 2019, at 2:09 PM, Gettes, Michael <> wrote:
> 
> After a bit more head-banging (and Carey pointing out my not seeing what 
> Bert saw with top)
> 
> Here is the config I finally settled on and it works without the errors I 
> was reporting (except for the caching message which Bert is kind enough to 
> address in an upcoming patch).
> 
> The problem I was trying to solve was to provide groups via PSPNG into AD 
> preserving DIT structure for compatibility so I can use Grouper to take 
> over already provided groups through custom software.  I settled on using 
> displayName as a means of linking Grouper and AD instead of the traditional 
> CN which was clearly getting me in trouble.  On the user objects we have 
> usernames that end in ‘.’ which is not allowed for samAccountName.
> 
> I am terribly sorry to have wasted Bert’s time with my mistakes and I am 
> grateful to Bert and Carey for taking time to help me.
> 
> /mrg
> 
> changeLog.consumer.psp_UFADdev.provisionerName = psp_UFADdev
> changeLog.consumer.psp_UFADdev.class = 
> edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim
> changeLog.consumer.psp_UFADdev.type = 
> edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner
> changeLog.consumer.psp_UFADdev.quartzCron = 1/20 * * * * ?
> changeLog.consumer.psp_UFADdev.ldapPoolName = UFADdev
> changeLog.consumer.psp_UFADdev.retryOnError = false
> changeLog.consumer.psp_UFADdev.isActiveDirectory = true
> changeLog.consumer.psp_UFADdev.grouperIsAuthoritative = true
> changeLog.consumer.psp_UFADdev.memberAttributeName = member
> changeLog.consumer.psp_UFADdev.memberAttributeValueFormat = 
> ${ldapUser.getDn()}
> changeLog.consumer.psp_UFADdev.groupSearchBaseDn = 
> OU=Grouper,OU=Groups,OU=UF,DC=dev-ad,DC=ufl,DC=edu
> changeLog.consumer.psp_UFADdev.groupCreationBaseDn = 
> OU=Grouper,OU=Groups,OU=UF,DC=dev-ad,DC=ufl,DC=edu
> changeLog.consumer.psp_UFADdev.allGroupsSearchFilter = 
> (&(objectclass=group)(objectclass=posixGroup))
> changeLog.consumer.psp_UFADdev.singleGroupSearchFilter = 
> (&(objectclass=group)(displayName=${group.name}))
> changeLog.consumer.psp_UFADdev.groupSearchAttributes = 
> cn,gidNumber,objectclass,samAccountName,name,displayName
> changeLog.consumer.psp_UFADdev.groupCreationLdifTemplate = dn: 
> ${utils.bushyDn("${group.name.replaceAll('(App:.+:UF:Groups:)(.+):(.*)','$2:'+'$3'.replace(':','_'))}","cn","ou")}||objectclass:
>  top||objectclass: group||objectclass: posixGroup||gidNumber: 
> ${group.idIndex}||displayName: ${group.name}
> changeLog.consumer.psp_UFADdev.userSearchBaseDn = 
> OU=People,OU=UF,DC=dev-ad,DC=ufl,DC=edu
> changeLog.consumer.psp_UFADdev.userSearchFilter = 
> userPrincipalName=${subject.id}
> changeLog.consumer.psp_UFADdev.userSearchAttributes = 
> dn,cn,uid,mail,samAccountName,uidNumber,objectclass,memberOf,userPrincipalName
> changeLog.consumer.psp_UFADdev.groupSearch_batchSize = 100
> changeLog.consumer.psp_UFADdev.ldapSearchResultPagingSize = 1000
> changelog.consumer.psp_UFADdev.grouperSubjectCacheSize = 400000
> changelog.consumer.psp_UFADdev.grouperGroupCacheSize = 50000
> changelog.consumer.psp_UFADdev.ldapUserCacheSize = 400000
> changelog.consumer.psp_UFADdev.ldapUserCacheTime_secs = 3600
> changelog.consumer.psp_UFADdev.maxValuesToChangePerOperation = 900
> changeLog.consumer.psp_UFADdev.needsTargetSystemUsers = true
> changeLog.consumer.psp_UFADdev.supportsEmptyGroups = true
> changeLog.consumer.psp_UFADdev.createMissingUsers = true
> changeLog.consumer.psp_UFADdev.userCreationBaseDn = 
> OU=People,OU=UF,DC=dev-ad,DC=ufl,DC=edu
> changeLog.consumer.psp_UFADdev.userCreationLdifTemplate = dn: 
> CN=${subject.getId().toLowerCase().replaceAll('@ufl.edu','').replaceAll('\\.$','')}||sAMAccountName:
>  
> ${subject.getId().toLowerCase().replaceAll('@ufl.edu','').replaceAll('\\.$','')}||objectclass:
>  top||objectclass: person||objectclass: user||sn: 
> ${subject.getId().toLowerCase().replaceAll('@ufl.edu','')}||userPrincipalName:
>  ${subject.getId()}||displayName: ${subject.name}
> 
>> On Apr 29, 2019, at 4:43 PM, Gettes, Michael <> wrote:
>> 
>> see inline...
>> 
>>> On Apr 29, 2019, at 12:58 PM, Bee-Lindgren, Bert 
>>> <> wrote:
>>> 
>>> Hello,
>>> 
>>> 1) Please add the top objectclass to your group template, though this 
>>> problem looks different to me
>> 
>> objectclass: top is already part of the template.
>> 
>>> 2) Can you privately send me what you replaced with ' > … (long list of 
>>> DNs) … ‘?
>> 
>> being sent separately.
>> 
>>> 3) (particularly with PSPNG p5), Can you run it with INFO or DEBUG log 
>>> level and send the result to me?
>> 
>> being sent separately.
>> 
>>> 4) I would suggest adding gidNumber to your groupCreationTemplate and 
>>> then (after gidNumbers are fully provisioned by a full-sync sweep) using 
>>> gidNumber-based searching in your singleGroupSearchFilter
>> 
>> Done.  Much smarter way to go for Active Directory - i think this should 
>> be more strongly recommended in the documentation.
>> 
>>> 
>>> 
>>> Thanks,
>>> Bert
>>> 
>>> 
>>> 
>>> 
>>> From:  
>>> <> on behalf of Gettes, Michael 
>>> <>
>>> Sent: Friday, April 26, 2019 11:04 PM
>>> To: 
>>> Subject: [grouper-dev] PSPNG error repeating (latest)
>>> 
>>> Going against AD - this error keeps repeating.  It’s a group with 17K 
>>> members.  my pspng config is below.  This is on tier/grouper:latest 
>>> published today (2.4.0-a42-u23-w5-p4-20190426-rc1).
>>> 
>>> 2019-04-27T02:19:09+00:00 DAEMON:dev 
>>> grouper-api;grouper.log;grouper_dev;daemon;2019-04-26 22:19:09,381: 
>>> [FullSyncer(psp_UFADdev)-Thread] WARN  LdapSystem.performLdapModify(405) 
>>> -  - UFADdev: Problem while modifying ldap system based on grouper 
>>> expectations. Starting to perform adaptive modifications based on data 
>>> already on server: [org.ldaptive.ModifyRequest@1988375000::modifyDn
>>> … (long list of DNs) … 
>>> controls=null, referralHandler=null, intermediateResponseHandlers=null]: 
>>> ENTRY_ALREADY_EXISTS
>>> 2019-04-27T02:15:38+00:00 DAEMON:dev 
>>> grouper-api;grouper.log;grouper_dev;daemon;2019-04-26 22:15:38,044: 
>>> [FullSyncer(psp_UFADdev)-Thread] ERROR 
>>> FullSyncProvisioner.fullSyncGroup(739) -  - FullSyncer(psp_UFADdev): 
>>> Problem doing full sync. Requeuing group 
>>> App:UFAD:UF:Groups:Services:Zoom:Service uflphi
>>> 2019-04-27T02:15:38+00:00 DAEMON:dev java.lang.IllegalStateException: The 
>>> active provisioner should be defined when creating LdapObjects
>>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>>> edu.internet2.middleware.grouper.pspng.LdapObject.<init>(LdapObject.java:95)
>>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>>> edu.internet2.middleware.grouper.pspng.LdapSystem.performLdapRead(LdapSystem.java:626)
>>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>>> edu.internet2.middleware.grouper.pspng.LdapSystem.performLdapRead(LdapSystem.java:596)
>>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>>> edu.internet2.middleware.grouper.pspng.LdapSystem.performLdapModify(LdapSystem.java:448)
>>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>>> edu.internet2.middleware.grouper.pspng.LdapSystem.performLdapModify(LdapSystem.java:377)
>>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>>> edu.internet2.middleware.grouper.pspng.LdapProvisioner.makeIndividualLdapChanges(LdapProvisioner.java:713)
>>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>>> edu.internet2.middleware.grouper.pspng.LdapProvisioner.finishProvisioningBatch(LdapProvisioner.java:455)
>>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>>> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner.fullSyncGroup(FullSyncProvisioner.java:721)
>>> 2019-04-27T02:15:38+00:00 DAEMON:dev     at 
>>> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner.processQueueItem(FullSyncProvisioner.java:433)
>>> 9-04-27T02:19:09+00:00 DAEMON:dev        at 
>>> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner.thread_manageFullSyncProcessing(FullSyncProvisioner.java:254)
>>> 2019-04-27T02:19:09+00:00 DAEMON:dev     at 
>>> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner$1.run(FullSyncProvisioner.java:146)
>>> 2019-04-27T02:19:09+00:00 DAEMON:dev     at 
>>> java.lang.Thread.run(Thread.java:748)
>>> 2019-04-27T02:53:48+00:00 DAEMON:dev 
>>> grouper-api;grouper.log;grouper_dev;daemon;2019-04-26 22:53:48,840: 
>>> [FullSyncer(psp_UFADdev)-Thread] WARN  
>>> ProvisionerCoordinator$ProvisioningStatus.lockForFullSyncWhenNoIncrementalIsUnderway(73)
>>>  -  - psp_UFADdev: Cannot start FullSync of 
>>> App:UFAD:UF:Groups:Services:Zoom:Service uflphi. Incremental provisioning 
>>> underway since Fri Apr 26 22:53:48 EDT 2019. We'll give up and move ahead 
>>> anyway in 300 seconds.
>>> 
>>> ———
>>> 
>>> AND this is a poorly formatted message that comes next:  not sure what i 
>>> should do to address it if anything.
>>> 
>>> 2019-04-27T02:19:09+00:00 DAEMON:dev 
>>> grouper-api;grouper.log;grouper_dev;daemon;2019-04-26 22:19:09,461: 
>>> [FullSyncer(psp_UFADdev)-Thread] WARN  
>>> Provisioner.warnAboutCacheSizeConcerns(620) -  - Cache is very full 
>>> (%.0f%%). Performance is much higher if 100.0 is large enough to hold the 
>>> number provisioned groups or subjects
>>> 
>>> ———
>>> 
>>> pspng config:
>>> 
>>> ldap.UFADdev.pass = XXXXXXXXXXXX
>>> ldap.UFADdev.user = XXXXXXXXXXXXXXXXXXXXXXXXX
>>> ldap.UFADdev.url = ldap://ufadXXXXXXXXXXXXXXXXX.ufl.edu
>>> ldap.UFADdev.pagedResultsSize = 1000
>>> ldap.UFADdev.timeout = 100000
>>> ldap.UFADdev.tls = false
>>> ldap.UFADdev.searchResultHandlers=org.ldaptive.handler.DnAttributeEntryHandler,edu.internet2.middleware.grouper.ldap.ldaptive.GrouperRangeEntryHandler
>>> # comma-delimited list of result codes (org.ldaptive.ResultCode) to 
>>> ignore, e.g. TIME_LIMIT_EXCEEDED, SIZE_LIMIT_EXCEEDED, PARTIAL_RESULTS
>>> ldap.UFADdev.searchIgnoreResultCodes=SIZE_LIMIT_EXCEEDED,ATTRIBUTE_OR_VALUE_EXISTS
>>> 
>>> changeLog.consumer.psp_UFADdev.provisionerName = psp_UFADdev
>>> changeLog.consumer.psp_UFADdev.class = 
>>> edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim
>>> changeLog.consumer.psp_UFADdev.type = 
>>> edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner
>>> changeLog.consumer.psp_UFADdev.quartzCron = 0/15 * * * * ?
>>> changeLog.consumer.psp_UFADdev.ldapPoolName = UFADdev
>>> changeLog.consumer.psp_UFADdev.retryOnError = false
>>> changeLog.consumer.psp_UFADdev.isActiveDirectory = true
>>> changeLog.consumer.psp_UFADdev.grouperIsAuthoritative = true
>>> changeLog.consumer.psp_UFADdev.memberAttributeName = member
>>> changeLog.consumer.psp_UFADdev.memberAttributeValueFormat = 
>>> ${ldapUser.getDn()}
>>> changeLog.consumer.psp_UFADdev.groupSearchBaseDn = 
>>> OU=Grouper,OU=Groups,OU=UF,DC=dev-ad,DC=ufl,DC=edu
>>> changeLog.consumer.psp_UFADdev.groupCreationBaseDn = 
>>> OU=Grouper,OU=Groups,OU=UF,DC=dev-ad,DC=ufl,DC=edu
>>> changeLog.consumer.psp_UFADdev.allGroupsSearchFilter = objectclass=group
>>> changeLog.consumer.psp_UFADdev.singleGroupSearchFilter = 
>>> (&(objectclass=group)(cn=${group.name.replaceAll("(App:.+:UF:Groups:)(.+):(.*)","$3").replace(":","_")}))
>>> changeLog.consumer.psp_UFADdev.groupSearchAttributes = 
>>> cn,gidNumber,objectclass,samAccountName,name
>>> changeLog.consumer.psp_UFADdev.groupCreationLdifTemplate = dn: 
>>> ${utils.bushyDn("${group.name.replaceAll('(App:.+:UF:Groups:)(.+):(.*)','$2:'+'$3'.replace(':','_'))}","cn","ou")}||objectclass:
>>>  group
>>> changeLog.consumer.psp_UFADdev.userSearchBaseDn = 
>>> OU=People,OU=UF,DC=dev-ad,DC=ufl,DC=edu
>>> changeLog.consumer.psp_UFADdev.userSearchFilter = 
>>> userPrincipalName=${subject.id}
>>> changeLog.consumer.psp_UFADdev.userSearchAttributes = 
>>> dn,cn,uid,mail,samAccountName,uidNumber,objectclass,memberOf,userPrincipalName
>>> changeLog.consumer.psp_UFADdev.groupSearch_batchSize = 100
>>> changeLog.consumer.psp_UFADdev.ldapSearchResultPagingSize = 1000
>>> changelog.consumer.psp_UFADdev.grouperSubjectCacheSize = 400000
>>> changelog.consumer.psp_UFADdev.grouperGroupCacheSize = 50000
>>> changelog.consumer.psp_UFADdev.ldapUserCacheSize = 200000
>>> changelog.consumer.psp_UFADdev.ldapUserCacheTime_secs = 3600
>>> changelog.consumer.psp_UFADdev.maxValuesToChangePerOperation = 500
>>> changeLog.consumer.psp_UFADdev.needsTargetSystemUsers = true
>>> changeLog.consumer.psp_UFADdev.supportsEmptyGroups = true
>>> changeLog.consumer.psp_UFADdev.createMissingUsers = true
>>> changeLog.consumer.psp_UFADdev.userCreationBaseDn = 
>>> OU=People,OU=UF,DC=dev-ad,DC=ufl,DC=edu
>>> changeLog.consumer.psp_UFADdev.userCreationLdifTemplate = dn: 
>>> CN=${subject.getId().toLowerCase().replaceAll('@ufl.edu','').replaceAll('\\.$','')}||sAMAccountName:
>>>  
>>> ${subject.getId().toLowerCase().replaceAll('@ufl.edu','').replaceAll('\\.$','')}||objectclass:
>>>  top||objectclass: person||objectclass: user||sn: 
>>> ${subject.getId().toLowerCase().replaceAll('@ufl.edu','')}||userPrincipalName:
>>>  ${subject.getId()}||displayName: ${subject.name}||cn: 
>>> ${subject.getId().toLowerCase().replaceAll('@ufl.edu','')}
>> 
>