Grouper Users - Open Discussion List

[Chris Hyzer <>]

I can do an AD query with more than 1000 records and reproduce your issue…

 

From my reading of vt-ldap docs and code, I think you need pagedResultsSize of 800 and referral set to “follow”.

 

Unfortunately the grouper ldap loader doesn’t support the referral param, or external param files (e.g. ldap.properties).

 

I added some code (need to test more), that should work in 2.1.4.  Can you build that grouper.jar and try it?  J

 

https://bugs.internet2.edu/jira/browse/GRP-897

 

Thanks,

Chris

 

From: [mailto:] On Behalf Of Bryan E. Wooten
Sent: Wednesday, April 17, 2013 5:48 PM
To: Gagné Sébastien;
Subject: RE: [grouper-users] Weird behavior with LDAP_GROUPS_FROM_ATTRIBUTES

 

Thanks for the suggestion. I set it to 800 but got the same results. The log file entries make the mystery even more weird.

 

 

2013-04-17 15:27:04,103: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderResultset$2.callback(888) -  - Found 1 results, (18038 sub-results) for serverId: personLdap, searchDn: ou=people,dc=testad,dc=utah,dc=edu, filter: '(memberof=cn=currentstudents,ou=academic,ou=security groups,dc=testad,dc=utah,dc=edu)', returning subject attribute: null, some results: {ActiveDirectory:groups:currentStudent=[u0254263, u0524534, u0524534, u0524534, u0524534, u052453...

 

Note u0524534 (out net id) appearing multiple times. Then

 

2013-04-17 15:27:04,131: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderType$8.runJob(994) -  - ActiveDirectory:groupsFromAttributesLdapGroup: start syncing membership

2013-04-17 15:27:04,133: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderType.syncGroupList(1114) -  - ActiveDirectory:groupsFromAttributesLdapGroup: found 37 members overall

 

It says found 37 members, but above there are 18038 sub-results? There are in fact 46k members.

 

 

to group: LdapSourceAdapter/u0524534, 10 of 34 subjects

2013-04-17 15:27:04,167: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderType.syncOneGroupMembership(2149) -  - ActiveDirectory:groups:currentStudent will add subject to group: LdapSourceAdapter/u0524534, 11 of 34 subjects

2013-04-17 15:27:04,167: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderType.syncOneGroupMembership(2149) -  - ActiveDirectory:groups:currentStudent will add subject to group: LdapSourceAdapter/u0524534, 12 of 34 subjects

2013-04-17 15:27:04,167: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderType.syncOneGroupMembership(2149) -  - ActiveDirectory:groups:currentStudent will add subject to group: LdapSourceAdapter/u0524534, 13 of 34 subjects

2013-04-17 15:27:04,167: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderType.syncOneGroupMembership(2149) -  - ActiveDirectory:groups:currentStudent will add subject to group: LdapSourceAdapter/u0524534, 14 of 34 subjects

2013-04-17 15:27:04,167: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderType.syncOneGroupMembership(2149) -  - ActiveDirectory:groups:currentStudent will add subject to group: LdapSourceAdapter/u0714509, 15 of 34 subjects

2013-04-17 15:27:04,167: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderType.syncOneGroupMembership(2149) -  - ActiveDirectory:groups:currentStudent will add subject to group: LdapSourceAdapter/u0714509, 16 of 34 subjects

2013-04-17 15:27:04,167: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderType.syncOneGroupMembership(2149) -  - ActiveDirectory:groups:currentStudent will add subject to group: LdapSourceAdapter/u0714509, 17 of 34 subjects

2013-04-17 15:27:04,168: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderType.syncOneGroupMembership(2149) -  - ActiveDirectory:groups:currentStudent will add subject to group: LdapSourceAdapter/u0714509, 18 of 34 subjects

2013-04-17 15:27:04,168: [DefaultQuartzScheduler_Worker-2] DEBUG GrouperLoaderType.syncOneGroupMembership(2149) -  - ActiveDirectory:groups:currentStudent will add subject to group: LdapSourceAdapter/u0714509, 19 of 34 subjects

 

Note the repeated net ids again.

 

The end result is 3 members in my Grouper group.

 

I guess I’ll go talk to the AD guys and see if they have any ideas.

 

-Bryan

From: Gagné Sébastien []
Sent: Wednesday, April 17, 2013 2:04 PM
To: Bryan E. Wooten
Subject: RE: [grouper-users] Weird behavior with LDAP_GROUPS_FROM_ATTRIBUTES

 

I’ve defined it to 800 to be « safer » (I’m not sure why).

 

My ldap.properties :

# paged results

edu.vt.middleware.ldap.pagedResultsSize=800

 

I would lower this one to a value <=1000. I think that when the query is over 1000  (2000 in your case) it will simply fail to do the paging

ldap.personlLdap.pagedResultsSize=2000

 

As for the commented line, it’s still commented in my configuration.

 

De : Bryan E. Wooten []
Envoyé : 17 avril 2013 16:00
À : Gagné Sébastien
Objet : RE: [grouper-users] Weird behavior with LDAP_GROUPS_FROM_ATTRIBUTES

 

I have this in ldap.properties:

edu.vt.middleware.ldap.pagedResultsSize=1000

 

I also tried it set to 2000.

 

In grouper.loader.properties I have this:

 

ldap.personLdap.batchSize = 900000

ldap.personlLdap.pagedResultsSize=2000

ldap.personLdap.countLimit = 900000

ldap.personLdap.timeLimit = 300000

 

I found this comment in the ldap.properties:

 

# handle Active Directory groups with a large (>1500) number of members

# see https://bugs.internet2.edu/jira/browse/GRP-335

# see http://code.google.com/p/vt-middleware/wiki/vtldapAD#Range_Attributes

#edu.vt.middleware.ldap.searchResultHandlers=edu.vt.middleware.ldap.handler.FqdnSearchResultHandler,edu.internet2.middleware.ldappc.util.RangeSearchResultHandler

 

When I uncomment that line I get class def not found and the loader won’t start.

 

-Bryan

 

 

From: Gagné Sébastien []
Sent: Wednesday, April 17, 2013 1:53 PM
To: Bryan E. Wooten;
Subject: RE: [grouper-users] Weird behavior with LDAP_GROUPS_FROM_ATTRIBUTES

 

Did you define your paging configuration in ldap.properties ? AD has a maximum of 1000 results returned, that could be the problem for sn=jones

 

As for the loader I’m not sure

 

De : [] De la part de Bryan E. Wooten
Envoyé : 17 avril 2013 15:28
À :
Objet : [grouper-users] Weird behavior with LDAP_GROUPS_FROM_ATTRIBUTES

 

I am trying to create a group where the search filter is (memberof=cn=currentstudents). When I use that filter against AD I get 46k results. However when the loader runs I only get 3 results. There was a time earlier today when I only got one result.

 

I changed the configuration so that the filter is (sn=jones). I get back 1000 results. The AD has 1007 entries where sn=jones.

 

Does anyone have any I ideas what is going on? Below is my Attribute assignments for the group.

 

Thanks,

 

Bryan

 

Attribute assignments

Owner group     Attribute name                Enabled?            Assignment values         Attribute definition        Assignment UUID

                groupsFromAttributesLdapGroup            Grouper loader LDAP     enabled                               grouperLoaderLdapDef                10635...

Metadata on assignment                             Grouper loader LDAP group attribute name        enabled                  memberof                grouperLoaderLdapValueDef     0b9ea...

Metadata on assignment                             Grouper loader LDAP quartz cron             enabled                  0 * * * * ?                grouperLoaderLdapValueDef     33c16...

Metadata on assignment                             Grouper loader LDAP type           enabled                  LDAP_GROUPS_FROM_ATTRIBUTES      grouperLoaderLdapValueDef     35d89...

Metadata on assignment                             Grouper loader LDAP subject _expression_              enabled                  ${subjectAttributes['cn']}             grouperLoaderLdapValueDef     4d525...

Metadata on assignment                             Grouper loader LDAP server ID  enabled                  personLdap                grouperLoaderLdapValueDef     69096...

Metadata on assignment                             Grouper loader LDAP group name _expression_    enabled                  groups:currentStudent grouperLoaderLdapValueDef     9243e...

Metadata on assignment                             Grouper loader LDAP subject ID type     enabled                  subjectIdentifier                grouperLoaderLdapValueDef     92bb6...

Metadata on assignment                             Grouper loader LDAP extra attributes    enabled                  cn                grouperLoaderLdapValueDef     9a5a6...

Metadata on assignment                             Grouper loader LDAP filter          enabled                  (memberof=cn=currentstudents,ou=academic,ou=security groups,dc=testad,dc=utah,dc=edu)                grouperLoaderLdapValueDef     a5522...

Metadata on assignment                             Grouper loader LDAP search base DN     enabled                  ou=people,dc=testad,dc=utah,dc=edu grouperLoaderLdapValueDef     ba12d...

Metadata on assignment                             Grouper loader LDAP source ID enabled                  ldap                grouperLoaderLdapValueDef     c9756...