Grouper Users - Open Discussion List

[Mark Cairney <>]

Hi Tom,

Essentially what happened was that after running a Grouper provisioning to 
our multi-master OpenLDAP setup slapd.d kept on running and running until it 
consumed all RAM and swap.

Looking at the log it looked like it was constantly trying to create a parent 
OU and failing because it already existed.

Looking closer it appeared that the child OU had been created before the 
parent OU and as a placeholder in occordance with the LDAP RFCs a placeholder 
OU with the "glue" objectclass is created by the OpenLDAP server in this 
instance to hold the child objects in.

To cut a long story short it ends up with a chicken and egg situation where 
it  is constantly trying to create the parent OU via syncrepl but ignores the 
sync because it thinks it already exists, stores the request in memory and 
tries again.

Eventually the system memory ends up saturated with the same repeated 
syncrepl request and falls over :-)

In terms of reproduceability providing you have the same initial conditions 
of 3 or more OpenLDAP servers, with one of them having a system clock which 
is close enough to the others that it accepts the syncrepl cookie but far 
enough out that that it receives them in the wrong order then it is 
reproducable if you throw a fairly large, nested set of ldap changes to it. 
This could be an ldapadd request as well as via Grouper ( I was able to 
reproduce it by doing a Grouper "dry Run" and then ldapadding the resultant 
LDIF). 

In my case I had 2 of my 3 servers pointed at our local NTP server but the 
third was still going to the default RHEL NTP servers.


Ultimately it turned out to be an OpenLDAP issue rather than a Grouper issue 
per se, but it was an interesting (if infuriating) problem. I suppose it just 
goes to illustrate the golden rule of running OpenLDAP in multi-master mode- 
keep your clocks in sync or bad things(tm) happen!



On 18 Jun 2011, at 17:30, Tom Zeller wrote:

> Whoa, ntp. Could you share more details ?
> 
> I read your original message when you sent it ages ago, apologies for the 
> delay.
> 
> Is the bug reproduceable ? (I would guess probably not)
> 
> On Wednesday, June 15, 2011, Mark Cairney 
> <>
>  wrote:
>> Hi,
>> 
>> For completeness this was an issue at the OpenLDAP side of things rather 
>> than with Grouper- one of the nodes had a clock that was slightly out of 
>> sync with the others as it was configured with the default list of NTP 
>> servers rather than pointing at a local NTP server. Once I corrected this 
>> Grouper appears to be provisioning correctly (touch wood) and the changes 
>> appear to be being replicated.
>> 
>> Apologies!
>> 
>> Cheers,
>> 
>> Mark
>> 
>> On 10 Jun 2011, at 15:45, Mark Cairney wrote:
>> 
>>> Hi,
>>> 
>>> Im wondering if anyone has ever came across this issue? Essentially it 
>>> looks like Im hitting a race condition in that children ou's are being 
>>> created before their parent ou's thus resulting in the creation of 
>>> "ghost" OU's with the "glue" objectClass. This behaviour seems to have 
>>> coincided with me setting initial cache sizes and changing the subject 
>>> attribute to id now that the corresponding LDAP attribute is now present.
>>> 
>>> 
>>> 
>>> My ldappc.xml is currently:
>>> 
>>> 
>>> <ldappc>
>>> <grouper>
>>>  <group-queries>
>>>    <subordinate-stem-queries>
>>>      <stem-list>
>>>        <stem>affiliations</stem>
>>>        <stem>pos</stem>
>>>        <stem>courses</stem>
>>>        <stem>org</stem>
>>>        <stem>adhoc</stem>
>>>      </stem-list>
>>>    </subordinate-stem-queries>
>>>  </group-queries>
>>> 
>>>  <groups
>>>    structure="bushy"
>>>    root-dn="ou=grouper,${edu.vt.middleware.ldap.base}"
>>>    ldap-object-class="groupOfNames"
>>>    ldap-rdn-attribute="cn"
>>>    grouper-attribute="name">
>>> 
>>>    <group-members-dn-list list-object-class="groupOfNames" 
>>> list-attribute="memb
>>> er" list-empty-value="" />
>>> 
>>>    <!-- <group-attribute-mapping ldap-object-class="groupOfNames">
>>>      <group-attribute-map group-attribute="description" 
>>> ldap-attribute="descrip
>>> tion" />
>>>    </group-attribute-mapping> -->
>>>    <group-attribute-mapping ldap-object-class="posixGroup">
>>>      <group-attribute-map group-attribute="gid" 
>>> ldap-attribute="gidNumber" />
>>>    </group-attribute-mapping>
>>> </groups>
>>> 
>>> </grouper>
>>> 
>>> <source-subject-identifiers>
>>>  <source-subject-identifier source="sourceId" subject-attribute="id">
>>>    <ldap-search
>>>      base="ou=people,ou=central,${edu.vt.middleware.ldap.base}"
>>>      scope="onelevel_scope"
>>>      filter="(eduniIdmsId={0})"
>>>      on-not-found="warn" />
>>>       </source-subject-identifier>
>>> </source-subject-identifiers>
>>> 
>>> </ldappc>
>>> 
>>> /*********************************
>>> Mark Cairney
>>> ITI UNIX Section
>>> Information Services
>>> University of Edinburgh
>>> 
>>> Tel: 0131 650 6565
>>> Email: 
>>> 
>>> 
>>> *********************************/
>>> 
>>> 
>>> --
>>> The University of Edinburgh is a charitable body, registered in
>>> Scotland, with registration number SC005336.
>>> 
>>> 
>> 
>> /*********************************
>> Mark Cairney
>> ITI UNIX Section
>> Information Services
>> University of Edinburgh
>> 
>> Tel: 0131 650 6565
>> Email: 
>> 
>> 
>> *********************************/
>> 
>> 
>> --
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>> 
>> 
> 

/********************************* 
Mark Cairney
ITI UNIX Section
Information Services
University of Edinburgh

Tel: 0131 650 6565
Email: 


*********************************/


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.