Grouper Users - Open Discussion List

[Mark Cairney <>]

Sorry- just to confirm the issue was with the ordering of the syncrepl 
requests due to the clock skew- the ordering from ldappc is correct (and I 
confirmed this by analysis of the LDIF log).

We're using 1.5.3 as we did hit this bug previously in version 1.5.0  when we 
moved from using a flat structure to a bushy one.

Cheers,

Mark

On 20 Jun 2011, at 13:41, Tom Zeller wrote:

> So, "it" below is slapd ? or ldappc ?
> 
> The bug is ldappc OU ordering but the failure was syncrepl+ntp ? correct ?
> 
> I think there was an ordering bug in 1.5.1 and 1.5.2 fixed in 1.5.3
> but I will need to confirm.
> 
> On Monday, June 20, 2011, Mark Cairney 
> <>
>  wrote:
>> Hi Tom,
>> 
>> Essentially what happened was that after running a Grouper provisioning to 
>> our multi-master OpenLDAP setup slapd.d kept on running and running until 
>> it consumed all RAM and swap.
>> 
>> Looking at the log it looked like it was constantly trying to create a 
>> parent OU and failing because it already existed.
>> 
>> Looking closer it appeared that the child OU had been created before the 
>> parent OU and as a placeholder in occordance with the LDAP RFCs a 
>> placeholder OU with the "glue" objectclass is created by the OpenLDAP 
>> server in this instance to hold the child objects in.
>> 
>> To cut a long story short it ends up with a chicken and egg situation 
>> where it  is constantly trying to create the parent OU via syncrepl but 
>> ignores the sync because it thinks it already exists, stores the request 
>> in memory and tries again.
>> 
>> Eventually the system memory ends up saturated with the same repeated 
>> syncrepl request and falls over :-)
>> 
>> In terms of reproduceability providing you have the same initial 
>> conditions of 3 or more OpenLDAP servers, with one of them having a system 
>> clock which is close enough to the others that it accepts the syncrepl 
>> cookie but far enough out that that it receives them in the wrong order 
>> then it is reproducable if you throw a fairly large, nested set of ldap 
>> changes to it. This could be an ldapadd request as well as via Grouper ( I 
>> was able to reproduce it by doing a Grouper "dry Run" and then ldapadding 
>> the resultant LDIF).
>> 
>> In my case I had 2 of my 3 servers pointed at our local NTP server but the 
>> third was still going to the default RHEL NTP servers.
>> 
>> 
>> Ultimately it turned out to be an OpenLDAP issue rather than a Grouper 
>> issue per se, but it was an interesting (if infuriating) problem. I 
>> suppose it just goes to illustrate the golden rule of running OpenLDAP in 
>> multi-master mode- keep your clocks in sync or bad things(tm) happen!
>> 
>> 
>> 
>> On 18 Jun 2011, at 17:30, Tom Zeller wrote:
>> 
>>> Whoa, ntp. Could you share more details ?
>>> 
>>> I read your original message when you sent it ages ago, apologies for the 
>>> delay.
>>> 
>>> Is the bug reproduceable ? (I would guess probably not)
>>> 
>>> On Wednesday, June 15, 2011, Mark Cairney 
>>> <>
>>>  wrote:
>>>> Hi,
>>>> 
>>>> For completeness this was an issue at the OpenLDAP side of things rather 
>>>> than with Grouper- one of the nodes had a clock that was slightly out of 
>>>> sync with the others as it was configured with the default list of NTP 
>>>> servers rather than pointing at a local NTP server. Once I corrected 
>>>> this Grouper appears to be provisioning correctly (touch wood) and the 
>>>> changes appear to be being replicated.
>>>> 
>>>> Apologies!
>>>> 
>>>> Cheers,
>>>> 
>>>> Mark
>>>> 
>>>> On 10 Jun 2011, at 15:45, Mark Cairney wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> Im wondering if anyone has ever came across this issue? Essentially it 
>>>>> looks like Im hitting a race condition in that children ou's are being 
>>>>> created before their parent ou's thus resulting in the creation of 
>>>>> "ghost" OU's with the "glue" objectClass. This behaviour seems to have 
>>>>> coincided with me setting initial cache sizes and changing the subject 
>>>>> attribute to id now that the corresponding LDAP attribute is now 
>>>>> present.
>>>>> 
>>>>> 
>>>>> 
>>>>> My ldappc.xml is currently:
>>>>> 
>>>>> 
>>>>> <ldappc>
>>>>> <grouper>
>>>>>  <group-queries>
>>>>>    <subordinate-stem-queries>
>>>>>      <stem-list>
>>>>>        <stem>affiliations</stem>
>>>>>        <stem>pos</stem>
>>>>>        <stem>courses</stem>
>>>>>        <stem>org</stem>
>>>>>        <stem>adhoc</stem>
>>>>>      </stem-list>
>>>>>    </subordinate-stem-queries>
>>>>>  </group-queries>
>>>>> 
>>>>>  <groups
>>>>>    structure="bushy"
>>>>>    root-dn="ou=grouper,${edu.vt.middleware.ldap.base}"
>>>>>    ldap-object-class="groupOfNames"
>>>>>    ldap-rdn-attribute="cn"
>>>>>    grouper-attribute="name">
>>>>> 
>>>>>    <group-members-dn-list list-object-class="groupOfNames" 
>>>>> list-attribute="memb
>>>>> er" list-empty-value="" />
>>>>> 
>>>>>    <!-- <group-attribute-mapping ldap-object-class="groupOfNames">
>>>>>      <group-attribute-map group-attribute="description" 
>>>>> ldap-attribute="descrip
>>>>> tion" />
>>>>>    </group-attribute-mapping> -->
>>>>>    <group-attribute-mapping ldap-object-class="posixGroup">
>>>>>      <group-attribute-map group-attribute="gid" 
>>>>> ldap-attribute="gidNumber" />
>>>>>    </group-attribute-mapping>
>>>>> </groups>
>>>>> 
>>>>> </grouper>
>>>>> 
>>>>> <source-subject-identifiers>
>>>>>  <source-subject-identifier source="sourceId" subject-attribute="id">
>>>>>    <ldap-search
>>>>>      base="ou=people,ou=central,${edu.vt.middleware.ldap.base}"
>>>>>      scope="onelevel_scope"
>>>>>      filter="(eduniIdmsId={0})"
>>>>>      on-not-found="warn" />
>>>>>       </source-subject-identifier>
>>>>> </source-subject-identifiers>
>>>>> 
>>>>> </ldappc>
>>>>> 
>>>>> /*********************************
>>>>> Mark Cairney
>>>>> ITI UNIX Section
>>>>> Information Services
>>>>> University of Edinburgh
>>>>> 
>>>>> Tel: 0131 650 6565
>>>>> Email: /*********************************
>> Mark Cairney
>> ITI UNIX Section
>> Information Services
>> University of Edinburgh
>> 
>> Tel: 0131 650 6565
>> Email: 
>> 
>> 
>> *********************************/
>> 
>> 
>> --
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>> 
>> 
> 

/********************************* 
Mark Cairney
ITI UNIX Section
Information Services
University of Edinburgh

Tel: 0131 650 6565
Email: 


*********************************/


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.