Grouper Users - Open Discussion List

["Crawford, Jeffrey" <>]

Hi Shilin,

 

I think we are talking about slightly different things because this thread started off with one problem that I’ve partially solved, I have the paging working via the grouper-loader.properties file with “ldap.personLdap.pagedResultsSize = 1000”, what isn’t working is limiting the end number of someone searching for records in the UI. I’m trying to tell the UI to only return 1000 or less subjects when someone uses the search box. I thought that would be “subjectApi.source.ldap.param.maxResults.value = 1000”.

 

I’ve kinda gotten things working by putting the following in the loader grouper-loader.properties file:

------------------------------------------------------------------------

ldap.personLdap.url = "ldaps://host.ucla.edu:636

ldap.personLdap.user = uid=grprsvc, dc=ucla, dc=edu

ldap.personLdap.pass = /opt/grpr/grouper.apiBinary/conf/encryptedLDAP.pass

ldap.personLdap.tls = false

ldap.personLdap.pagedResultsSize = 1000

 

And the following for the ui and ws version of the file:

------------------------------------------------------------------------

ldap.personLdap.url = "ldaps://host.ucla.edu:636

ldap.personLdap.user = uid=grprsvc, dc=ucla, dc=edu

ldap.personLdap.pass = /opt/grpr/grouper.apiBinary/conf/encryptedLDAP.pass

ldap.personLdap.tls = false

ldap.personLdap.pagedResultsSize = 1000

ldap.personLdap.countLimit = 1000

ldap.personLdap.timeLimit = 3000

ldap.personLdap.searchIgnoreResultCodes = SIZE_LIMIT_EXCEEDED, TIME_LIMIT_EXCEEDED

 

The above is not my favorite as it means I have different configs in the ui and ws versions of the file, which I’m trying to avoid. The loader can’t have the countLimit applied because we have one LDAP loader job we need to run, and if the UI and WS version of configs are in the grouper loader instance, then we only get a max of 1000 members.

 

I thought if we could put the configs in subject.properties then it would only impact searches, but at least in my case the additional parameters seem to be ignored.

 

With the above configs in the different subproducts “loader”, “ui”, “ws” the LDAP searches are behaving as I would expect.

 

At the moment I have a couple of different configs in the grouper-loader.properties file labeled with “CHANGE ME” so when I make a change in the grouper loader file, I remember which elements to uncomment for the UI and WS versions.

 

Anyway, I’m not sure what does what in what files, if you think it’s something to be fixed, go for it and let me know, if not then I guess the above works for me.

 

Thanks

Jeffrey C.

 

From: Shilen Patel <>
Date: Wednesday, September 4, 2019 at 8:31 AM
To: "Crawford, Jeffrey" <>
Cc: Daniel Burtenshaw <>, Grouper Users <>, Jeffrey Williams <>, Greg Haverkamp <>
Subject: Re: [grouper-users] [Ext] Re: pspng ldapSearchResultPagingSize doesn't seem to do anything

 

My example was using an LDAP – maxPageSize is supported there.   Though I noticed a possible problem but perhaps not the problem you’re seeing.

 

When using LDAP with a maxPageSize (in subject.properties) of 10 and I search for “joh”, three queries are sent to my LDAP server. 

 

The first is a findById, which would be based on the query you have configured for subjectApi.source.example.search.searchSubject.param.filter.value.  But the UI appends an asterisk to the query (joh*) and doesn’t use a size limit here.  So the LDAP server may return everyone with a NetID starting with “joh”, which could be more than the page size.  However, since a findById isn’t supposed to return multiple results, an exception is thrown and the results are basically ignored. 

 

The second is a findPage, which does set the size limit correctly and only returns maxPage + 1 results.  And this is what appears in the search results in the UI.  And for me, this part is working correctly.  But it sounds like you’re saying that you’re getting all 14k results here instead of 100?  Do you have access to your ldap logs in case that explains what is happening there?  And looking at a sanitized copy of subject.properties might help too.  I’m assuming you’ve also verified that your subject.properties is the same between gsh, ui, etc.

 

Thanks!

 

- Shilen

 

From: "Crawford, Jeffrey" <>
Date: Wednesday, September 4, 2019 at 10:41 AM
To: Shilen Patel <>
Cc: Daniel Burtenshaw <>, Grouper-Users <>, Jeffrey Williams <>, Greg Haverkamp <>
Subject: Re: [grouper-users] [Ext] Re: pspng ldapSearchResultPagingSize doesn't seem to do anything

 

I’m pretty much getting the entire list back, something like 14k when I search for “lee”. However this is on the LDAP config, I’m noticing the two elements only show up on the JDBC config and not on the LDAP config example, is it just not supported there?

 

Jeffrey C.

 

From: Shilen Patel <>
Date: Wednesday, September 4, 2019 at 6:14 AM
To: "Crawford, Jeffrey" <>
Cc: Daniel Burtenshaw <>, Grouper Users <>, Jeffrey Williams <>, Greg Haverkamp <>
Subject: Re: [grouper-users] [Ext] Re: pspng ldapSearchResultPagingSize doesn't seem to do anything

 

In my subject.properties, I set the following:

 

subjectApi.source.example.param.maxPageSize.value = 10

 

(my source id was “example”)

 

Then when I go to the UI and click on Add Members and type in a value (e.g. “joh”), it would typically return hundreds of results but the search box only shows 11 (1 more than the page size).

 

The ldap logs show this as well.

 

[04/Sep/2019:09:05:14 -0400] conn=32305 op=10 RESULT err=4 tag=101 nentries=11 etime=0

 

I’m curious what you’re seeing?

 

Thanks!

 

- Shilen

 

 

From: <> on behalf of "Crawford, Jeffrey" <>
Reply-To: "Crawford, Jeffrey" <>
Date: Tuesday, September 3, 2019 at 6:15 PM
To: Greg Haverkamp <>
Cc: Daniel Burtenshaw <>, Grouper-Users <>, Jeffrey Williams <>
Subject: Re: [grouper-users] [Ext] Re: pspng ldapSearchResultPagingSize doesn't seem to do anything

 

I’ve gotten further, but there seems to be multiple places to configure these things. I got the main paging setup in grouper-loader.properties by using:

ldap.personLdap.pagedResultsSize = 1000

 

However I want to limit the search to < 1000 entries in the search box, I’ve tried the following in the subject.properties file but it’s not working:

subjectApi.source.ldap.param.maxResults.value = 1000

subjectApi.source.ldap.param.maxPageSize.value = 100

 

Should the above work?

Jeffrey C.

 

From: <> on behalf of "Crawford, Jeffrey" <>
Reply-To: "Crawford, Jeffrey" <>
Date: Friday, August 30, 2019 at 4:57 PM
To: Greg Haverkamp <>
Cc: Daniel Burtenshaw <>, Grouper Users <>, Jeffrey Williams <>
Subject: Re: [grouper-users] [Ext] Re: pspng ldapSearchResultPagingSize doesn't seem to do anything

 

So this is what I’ve done now, but still paging only 100. The LDAP loader isn’t paging at all. I also tried “batchSize” since it was one of the examples but that didn’t work either.

 

ldap.personLdap.url = ldaps://ldap.iam.ucla.edu:636

ldap.personLdap.user = uid=svcacct, ou=edimi consumers, dc=edtest, dc=ucla, dc=edu

ldap.personLdap.pass = /path/to/encryptedLDAP.pass

ldap.personLdap.tls = false

ldap.personLdap.searchResultPagingEnabled = true

ldap.personLdap.searchResultPagingSize = 1000

 

 

 

From: Greg Haverkamp <>
Date: Thursday, August 29, 2019 at 1:18 PM
To: "Crawford, Jeffrey" <>
Cc: Daniel Burtenshaw <>, Grouper Users <>, Jeffrey Williams <>
Subject: Re: [Ext] Re: [grouper-users] pspng ldapSearchResultPagingSize doesn't seem to do anything

 

I assume when you put it in your personLdap you changed it from ldapSearchResultPagingSize to just searchResultPagingSize?

 

The former, as I noted, appears nowhere in the code. (You can go to GitHub and search to confirm.) The latter does, with the code I quoted sending what should be read from your ldap config, or sending the default if it’s not there.

 

I think the documentation is simply wrong.  If you find that you can make it work, you might correct it.  :)

 

Greg

 

On Thu, Aug 29, 2019 at 1:13 PM Crawford, Jeffrey <> wrote:

Humm, I tried that too, seems to have been ignored by pspng, also on this page it’s listed as an option:

https://spaces.at.internet2.edu/display/Grouper/Grouper+Provisioning:+PSPNG

 

but it seems to have a default so I wonder if the code is “resetting” it.

 

From: Daniel Burtenshaw <>
Date: Thursday, August 29, 2019 at 1:06 PM
To: Greg Haverkamp <>, "Crawford, Jeffrey" <>
Cc: Grouper Users <>, Jeffrey Williams <>
Subject: RE: [Ext] Re: [grouper-users] pspng ldapSearchResultPagingSize doesn't seem to do anything

 

I think that Greg is correct, I’ve used it with success in the ldap connection settings rather than the changelog.consumer settings, so something like:

 

                ldap.personLdap.pagedResultsSize = 1000

 

should work.

 

From: <> On Behalf Of Greg Haverkamp
Sent: Thursday, August 29, 2019 1:48 PM
To: Crawford, Jeffrey <>
Cc: Grouper-Users <>; Jeffrey Williams <>
Subject: [Ext] Re: [grouper-users] pspng ldapSearchResultPagingSize doesn't seem to do anything

 

WARNING: Stop. Think. Read. This is an external email.

I think you want it in your personLdap configuration.  Unless I botched the search, “ldapSearchResultPagingSize” doesn’t exist anywhere in the Grouper repo (in master).

 

However, it does appear to pull the searchResultPagingSize in LdapSystem.java:

  public int getSearchResultPagingSize() {

    Object searchResultPagingSize = getLdaptiveProperties().get("org.ldaptive.searchResultPagingSize");

    return GrouperUtil.intValue(searchResultPagingSize, searchResultPagingSize_default_value);

  }

 

And getLdaptiveProperties should pull that in from your ldap.personLdap.* properties.

 

Greg

 

 

On Thu, Aug 29, 2019 at 11:42 AM Crawford, Jeffrey <> wrote:

The following is what we have, but it’s pretty simple:

 

ldap.personLdap.url = "ldaps://ldap.iam.ucla.edu:636

ldap.personLdap.user = uid=ldappc, dc=some, dc=path

ldap.personLdap.pass = /opt/grpr/grouper.apiBinary/conf/encryptedLDAP.pass

ldap.personLdap.tls = false

ldap.personLdap.countLimit = 1000

ldap.personLdap.timeLimit = 5000

 

From: Jeffrey Williams <>
Date: Thursday, August 29, 2019 at 9:51 AM
To: "Crawford, Jeffrey" <>
Cc: Grouper Users <>
Subject: Re: [grouper-users] pspng ldapSearchResultPagingSize doesn't seem to do anything

 

Can we see your personLdap config?

On Thu, Aug 29, 2019, 11:59 AM Crawford, Jeffrey <> wrote:

I have the following entry in our pspng config, I noticed that on FullSyncer it pages at 100 elements, our system performs better at 1000 so I tried setting the page size, but the LDAP logs still show that it’s paging at 100.

 

Am I using the parameter correctly?

 

changeLog.consumer.pspng_entitlement.class = edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim

changeLog.consumer.pspng_entitlement.type = edu.internet2.middleware.grouper.pspng.LdapAttributeProvisioner

changeLog.consumer.pspng_entitlement.quartzCron = 0 * * * * ?

changeLog.consumer.pspng_entitlement.retryOnError = true

changeLog.consumer.pspng_entitlement.ldapPoolName = personLdap

changeLog.consumer.pspng_entitlement.provisionedAttributeName = eduPersonEntitlement

changeLog.consumer.pspng_entitlement.provisionedAttributeValueFormat = urn:mace:ucla.edu:entitlement${group.name.replaceAll("^ucla(?::org)?(.*:)app:(.*)$", "$1$2").replaceFirst(":entitlement:",":")}

changeLog.consumer.pspng_entitlement.userSearchBaseDn = ou=people,dc=ucla,dc=edu

changeLog.consumer.pspng_entitlement.userSearchFilter = uclaPPID=${subject.id}

changeLog.consumer.pspng_entitlement.ldapSearchResultPagingSize = 1000

changeLog.consumer.pspng_entitlement.userSearchAttributes = dn,cn,uid,mail,samAccountName,uidNumber,objectClass,uclaPPID

 

Thanks

Jeffrey Crawford