Grouper Users - Open Discussion List

["Peter St. Onge" <>]

As a follow up to my earlier email, looking at the LDAP logs it seems
like I'm running into a bit of a corner (or more likely missing
something obvious).

As background (and mindful of the earlier "bushy" vs "flat" discussion):

We use a bushy group layout, mostly to allow subtree queries which feed
into different vendor applications (eg LifeRay so that it pulls
identities from the campus IdMS, a local AlFresco is similarly
provisioned AFAIK).

This also allows our production Grouper environment to support four
separate environments for divisional/departmental purposes -- dev, qa,
uat, and production. As a consequence, we have some redunancy in group
names (eg. dev:bio:web:deptsite, qa:bio:web:deptsite,
uat:bio:web:deptsite, prod:qa:bio:web:deptsite) -- All of these require
the use of production credentials to access, thus they're in the
production Grouper.

Plans to migrate Grouper to our enterprise AD environment are in the
works, but this instance is needed currently to work against our
non-production IdMS where identities live in OpenLDAP.

Looking through the LDAP logs ...

slapd[29943]: conn=8544 op=20 SRCH
base="ou=groupertest,dc=maple,dc=utoronto,dc=ca" scope=2 deref=0
filter="(|(&(objectClass=groupOfNames)(?=undefined)))"

... seems my singleGroupSearchFilter wasn't working. I simplified my
query to see what I was messing up.

changeLog.consumer.pspng_idp-easi.singleGroupSearchFilter =
(&(objectclass=groupOfNames)(${group.name}))
# Problem fetching groups with filter
'(|(&(objectclass=groupOfNames)(utormaple:test1:pete-test)))' on base
'ou=groupertest,dc=maple,dc=utoronto,dc=ca'

If we were a "flat" set up, this would make sense. We aren't, so the
group query returns no results. (The leading OR operator "(|" is
curious, not quite sure where that comes from but the query runs...)


changeLog.consumer.pspng_idp-easi.singleGroupSearchFilter =
(&(objectclass=groupOfNames)(${utils.bushyDn(group.name, "cn", "ou")}))
# slapd[29943]: conn=8535 op=20 SRCH
base="ou=groupertest,dc=maple,dc=utoronto,dc=ca" scope=2 deref=0
filter="(|(&(objectClass=groupOfNames)(cn=pete-test,ou=test1,ou=utormaple)))"

This is somewhat better (but I again note the "(|" OR operator. hmm)

From what little I understand of LDAP, I'm under the impression that we
can't really do searches based on the entity's dn. Since I cannot
guarantee that our groups are unique (I can actually guarantee the
opposite, that they are not distinct), I can't search on the cn of the
group either.

Again, I suspect I'm missing something obvious here, and any pointers
appreciated.

Thanks and best, -- pete



On 2017-01-18 11:17 AM, Peter St. Onge wrote:
> Hi Bert,
> 
> As of this morning, I should be up to date on patches - in
> grouper.apiBinary-2.3.0/grouperPatchStatus.properties I have:
> 
> grouper_v2_3_0_pspng_patch_7.date = 2017/01/18 08:50:30
> grouper_v2_3_0_pspng_patch_7.state = applied
> grouper_v2_3_0_api_patch_44.date = 2017/01/18 08:30:08
> grouper_v2_3_0_api_patch_44.state = applied
> 
> (I similarly updated the UI to current)
> 
> I stopped the UI before making these changes, and the grouper loader
> wasn't working at the time.
> 
> Still seeing this in the logs when I start the Grouper loader tho ....
> 
> 2017-01-18 10:54:23,357: [pspng_idp-easi-FullSync] DEBUG
> HibernateSession.callbackHibernateSession(716) -  -
> grouperTransactionType: READONLY_OR_USE_EXISTING, auditControl:
> WILL_NOT_AUDIT, hibernateSession: HibernateSession (24c95b1f): new,
> readonly, READONLY_NEW, notActiveTransaction, session (1f7d6317),
> hibernateSessionsInThreadLocal: 24c95b1f, willCreateAudit: false,
> createdContext: false, closedSession: true
> 2017-01-18 10:54:23,358: [pspng_idp-easi-FullSync] DEBUG
> GrouperUtil.substituteExpressionLanguage(9332) -  - Subsituting EL:
> '${utils.containedWithin(provisionerName,
> stemAttributes['etc:pspng:provision_to'],
> groupAttributes['etc:pspng:provision_to']) &&
> !utils.containedWithin(provisionerName,
> stemAttributes['etc:pspng:do_not_provision_to'],
> groupAttributes['etc:pspng:do_not_provision_to'])}', and with env vars:
> idIndex, userSearchBaseDn, groupAttributes, groupCreationBaseDn,
> stemAttributes, utils, groupSearchBaseDn, name, provisionerName, group,
> provisionerType, grouperUtil with result: 'true'
> 2017-01-18 10:54:23,358: [pspng_idp-easi-FullSync] DEBUG
> Provisioner.evaluateJexlExpression(518) -  - Evaluated Jexl expression:
> true FROM ${utils.containedWithin(provisionerName,
> stemAttributes['etc:pspng:provision_to'],
> groupAttributes['etc:pspng:provision_to']) &&
> !utils.containedWithin(provisionerName,
> stemAttributes['etc:pspng:do_not_provision_to'],
> groupAttributes['etc:pspng:do_not_provision_to'])} WITH variables
> {idIndex=10051, userSearchBaseDn=dc=maple,dc=utoronto,dc=ca,
> groupAttributes={},
> groupCreationBaseDn=ou=groupertest,dc=maple,dc=utoronto,dc=ca,
> stemAttributes={etc:pspng:provision_to=[pspng_idp-easi]},
> utils=edu.internet2.middleware.grouper.pspng.PspJexlUtils@555a820a,
> groupSearchBaseDn=ou=groupertest,dc=maple,dc=utoronto,dc=ca,
> name=utormaple:test1:pete-test, provisionerName=pspng_idp-easi,
> group=Group[name=utormaple:test1:pete-test,uuid=0a1c1f738bea49078ba50ecf717f223b],
> provisionerType=LdapGroupProvisioner}
> 2017-01-18 10:54:23,358: [pspng_idp-easi-FullSync] DEBUG
> Provisioner.shouldGroupBeProvisioned(1230) -  - pspng_idp-easi: Group
> utormaple:test1:pete-test matches group-selection filter.
> 2017-01-18 10:54:23,358: [pspng_idp-easi-FullSync] INFO
> LdapGroupProvisioner.createGroup(251) -  - Creating LDAP group for
> GrouperGroup: utormaple:test1:pete-test
> 2017-01-18 10:54:23,358: [pspng_idp-easi-FullSync] ERROR
> FullSyncProvisioner.processGroup(327) -  - pspng_idp-easi: Problem doing
> full sync. Requeuing utormaple:test1:pete-test
> java.lang.NullPointerException
>         at
> edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner.createGroup(LdapGroupProvisioner.java:253)
>         at
> edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner.createGroup(LdapGroupProvisioner.java:54)
>         at
> edu.internet2.middleware.grouper.pspng.Provisioner.prepareGroupCache(Provisioner.java:715)
>         at
> edu.internet2.middleware.grouper.pspng.Provisioner.startProvisioningBatch(Provisioner.java:468)
>         at
> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner.processGroup(FullSyncProvisioner.java:311)
>         at
> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner.thread_manageFullSyncProcessing(FullSyncProvisioner.java:174)
>         at
> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner$1.run(FullSyncProvisioner.java:132)
>         at java.lang.Thread.run(Thread.java:745)
> 
> I'm happy to experiment as you see fit.
> 
> Thanks and best, -- pete
> 
> On 2017-01-17 01:31 AM, Bee-Lindgren, Bert wrote:
>> Peter,
>>
>>
>> I've compared your logs to the current version of PSPNG, and they don't
>> line up. This is probably because there are patches that haven't yet
>> been applied. If this is the case, then there's a good chance the
>> problem has been addressed.
>>
>>
>> Please check on the patch status of the api/daemon installation
>> (grouperPatchStatus.properties).
>>
>>
>> Sincerely,
>>
>>   Bert Bee-Lindgren
>>
>>
>>
>> ------------------------------------------------------------------------
>> *From:* 
>> 
>> <>
>>  on behalf of Bee-Lindgren, Bert
>> <>
>> *Sent:* Monday, January 16, 2017 6:29 PM
>> *To:* Peter St. Onge
>> *Cc:* 
>> 
>> *Subject:* Re: [grouper-users] Adventures in PSPNG
>>  
>> I'll look at the cause in a minute, but it will help to confirm that
>> 'current 2.3 Grouper' includes all the patches?
>>
>> Thanks
>>
>>> On Jan 16, 2017, at 2:24 PM, Peter St. Onge 
>>> <>
>>>  wrote:
>>>
>>> I'm working on spinning up a current 2.3 Grouper machine for our dev
>>> group but am running into some issues writing into our OpenLDAP backend.
>>>
>>> This error message keeps popping up ...
>>>
>>>
>>> 2017-01-16 13:55:50,424: [pspng_idp-easi-FullSync] DEBUG
>>> GrouperUtil.substituteExpressionLanguage(9332) -  - Subsituting EL:
>>> '${utils.containedWithin(provisionerName,
>>> stemAttributes['etc:pspng:provision_to'],
>>> groupAttributes['etc:pspng:provision_to']) &&
>>> !utils.containedWithin(provisionerName,
>>> stemAttributes['etc:pspng:do_not_provision_to'],
>>> groupAttributes['etc:pspng:do_not_provision_to'])}', and with env vars:
>>> idIndex, userSearchBaseDn, groupAttributes, groupCreationBaseDn,
>>> stemAttributes, utils, groupSearchBaseDn, name, provisionerName, group,
>>> provisionerType, grouperUtil with result: 'true'
>>> 2017-01-16 13:55:50,424: [pspng_idp-easi-FullSync] DEBUG
>>> Provisioner.evaluateJexlExpression(510) -  - Evaluated Jexl expression:
>>> true FROM ${utils.containedWithin(provisionerName,
>>> stemAttributes['etc:pspng:provision_to'],
>>> groupAttributes['etc:pspng:provision_to']) &&
>>> !utils.containedWithin(provisionerName,
>>> stemAttributes['etc:pspng:do_not_provision_to'],
>>> groupAttributes['etc:pspng:do_not_provision_to'])} WITH variables
>>> {idIndex=10051, userSearchBaseDn=dc=maple,dc=utoronto,dc=ca,
>>> groupAttributes={},
>>> groupCreationBaseDn=ou=groupertest,dc=maple,dc=utoronto,dc=ca,
>>> stemAttributes={etc:pspng:provision_to=[pspng_idp-easi]},
>>> utils=edu.internet2.middleware.grouper.pspng.PspJexlUtils@56935138,
>>> groupSearchBaseDn=ou=groupertest,dc=maple,dc=utoronto,dc=ca,
>>> name=utormaple:test1:pete-test, provisionerName=pspng_idp-easi,
>>> group=Group[name=utormaple:test1:pete-test,uuid=0a1c1f738bea49078ba50ecf717f223b],
>>> provisionerType=LdapGroupProvisioner}
>>> 2017-01-16 13:55:50,424: [pspng_idp-easi-FullSync] DEBUG
>>> Provisioner.shouldGroupBeProvisioned(1208) -  - pspng_idp-easi: Group
>>> utormaple:test1:pete-test matches group-selection filter.
>>> 2017-01-16 13:55:50,424: [pspng_idp-easi-FullSync] INFO
>>> LdapGroupProvisioner.createGroup(231) -  - Creating LDAP group for
>>> GrouperGroup: utormaple:test1:pete-test
>>> 2017-01-16 13:55:50,424: [pspng_idp-easi-FullSync] ERROR
>>> FullSyncProvisioner.processGroup(328) -  - pspng_idp-easi: Problem doing
>>> full sync. Requeuing utormaple:test1:pete-test
>>> java.lang.NullPointerException
>>>        at
>>> edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner.createGroup(LdapGroupProvisioner.java:233)
>>>        at
>>> edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner.createGroup(LdapGroupProvisioner.java:54)
>>>        at
>>> edu.internet2.middleware.grouper.pspng.Provisioner.prepareGroupCache(Provisioner.java:705)
>>>        at
>>> edu.internet2.middleware.grouper.pspng.Provisioner.startProvisioningBatch(Provisioner.java:461)
>>>        at
>>> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner.processGroup(FullSyncProvisioner.java:312)
>>>        at
>>> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner.thread_manageFullSyncProcessing(FullSyncProvisioner.java:175)
>>>        at
>>> edu.internet2.middleware.grouper.pspng.FullSyncProvisioner$1.run(FullSyncProvisioner.java:133)
>>>        at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> I've tried
>>> * provisioning empty groups,
>>> * provisioning new groups with a default (known-good) user that would
>>>  get replaced at first update
>>>
>>> Any pointers most welcome!
>>>
>>>
>>> In passing, I'm getting a metric truckload of these ...
>>>
>>> 2017-01-16 13:55:51,425: [pspng_idp-easi-FullSync] DEBUG
>>> GrouperClientLog.debug(82) -  - Cant find text for variable: 'newline'
>>>
>>> I did a recursive search for any file containing "[^\$]newline" but
>>> didn't find this defined anywhere. I wonder if this is a java-specific
>>> thing, and whether there any way I can fix this.
>>>
>>>
>>>
>>> Thanks and best, -- pete

-- 
Peter St. Onge                           

Information Security Architect                     (416)978-5030
Business Continuity and Communications
Information + Technology Services          University of Toronto