Grouper Developers Forum

[Gagné Sébastien <>]

Hi,

I’ve been testing the PSP, aside from performance issue, large quantity of object creation results in OutOfMemory issues.

 

My setup :

I create 10 500 groups base on academic’s data.

I try to bulkSync these groups to AD (since the changelog provisioning isn’t working well for something that large)

I set MEM_MAX to 2000m

 

The process will fail after about 270 minutes and the creation of about 75% of groups, in “top” the process was using 1.8g and 100% of a CPU (probably the GC running like crazy, usually the bulk sync uses about 2-5%) :

Caused by: java.lang.OutOfMemoryError: Java heap space

        at java.util.Arrays.copyOf(Arrays.java:2882)

        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)

        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)

        at java.lang.StringBuffer.append(StringBuffer.java:224)

        at org.openspml.v2.util.xml.XmlBuffer.append(XmlBuffer.java:126)

        at org.openspml.v2.util.xml.XmlBuffer.addAnyElement(XmlBuffer.java:519)

        at org.openspml.v2.util.xml.ReflectiveXMLMarshaller.addOpenContentElements(ReflectiveXMLMarshaller.java:330)

        at org.openspml.v2.util.xml.ReflectiveXMLMarshaller.processElementFields(ReflectiveXMLMarshaller.java:210)

        at org.openspml.v2.util.xml.ReflectiveXMLMarshaller.default_marshall(ReflectiveXMLMarshaller.java:342)

        at org.openspml.v2.util.xml.ReflectiveXMLMarshaller.redispatch(ReflectiveXMLMarshaller.java:133)

        at org.openspml.v2.util.xml.ReflectiveXMLMarshaller.marshall(ReflectiveXMLMarshaller.java:109)

        at org.openspml.v2.util.xml.ReflectiveXMLMarshaller.marshall(ReflectiveXMLMarshaller.java:74)

        at org.openspml.v2.util.xml.ReflectiveXMLMarshaller.marshall(ReflectiveXMLMarshaller.java:372)

        at org.openspml.v2.msg.spml.Response.toXML(Response.java:144)

        at edu.internet2.middleware.psp.spml.provider.BaseSpmlProvider.toXML(BaseSpmlProvider.java:796)

        at edu.internet2.middleware.psp.Psp.execute(Psp.java:1436)

        at edu.internet2.middleware.psp.Psp.execute(Psp.java:879)

        at edu.internet2.middleware.psp.Psp.execute(Psp.java:800)

 

If I run a new bulkSync after it, it completes and uses far less memory, as well as all the following sync where there are no object creation.

 

My question :

Is it possible that there’s a memory leak somewhere in LDAP object creation ?

Would it be possible to configure some kind of batch size for object creation ? e.g. after 2000 objects created, restart the bulkSync, or clean up the memory, or ???

 

I believe this also crashes the PSP full sync cron in the Loader (if I let it create the 10 500 groups by itself) and requires a restart (I think it won’t show up in the diagnostic WS either)

 

Thanks

 

 

Sébastien Gagné,     | Analyste en informatique

514-343-6111 x33844  | Université de Montréal,

                     | Pavillon Roger-Gaudry, local X-100-11