Grouper Users - Open Discussion List

[Michael R Gettes <>]

Hi,

We run in a 12GB VM with -Xmx4096m on the loader and we are not using any 
swap space and 2.3.0 + patches (applied in december, i think).

After a long running loader job we see the error at the bottom of this mail.  
When we get this error, remaining loader jobs go bad and we have to restart 
the loader.  We are going to add memory.  But, should we be using a specific 
GC algorithm or taking some other steps for a large environment?  We 
currently have about 8M memberships (14M loader memberships) over 80K groups. 
 We have only been in production since June.  I am seeking any 
advice/guidance on what to do cuz I fear we are going to see this problem 
again.  We have a number of loader jobs managing groups of 100K to 230K.  We 
do run concurrent jobs on multiple servers (currently 2 cores for 2 loader 
job servers about to go to 4 cores).

I should also note: while we run with 4096m the loader nightly report shows 
heapsize as 3GB.
heapSize: 3 GB
heapMaxSize: 3 GB
heapSizeFree: 1 GB
version: 2.3.0, build date: 2016/04/16 05:37:21, env: PRODUCTION
os.name: Linux
os.arch: amd64
os.version: 2.6.32-642.6.2.el6.x86_64
java.version: 1.7.0_121
java.vendor: Oracle Corporation

Thank you!

/mrg

2017-01-09 08:56:48,254: [pool-1-thread-6229] ERROR 
GrouperLoaderType.syncGroupLogicForOneGroup(1634) -  - Error in job for 
group: Course:2171:UP:AA:044532:all
edu.internet2.middleware.grouper.internal.dao.GrouperDAOException: Problem in 
HibernateSession: null
        at 
edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch(HibernateSession.java:604)
        at 
edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:710)
        at 
edu.internet2.middleware.grouper.app.loader.db.Hib3GrouperLoaderLog.store(Hib3GrouperLoaderLog.java:730)
        at 
edu.internet2.middleware.grouper.app.loader.GrouperLoaderType.syncGroupLogicForOneGroup(GrouperLoaderType.java:1621)
        at 
edu.internet2.middleware.grouper.app.loader.GrouperLoaderType.access$200(GrouperLoaderType.java:118)
        at 
edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$11.callLogic(GrouperLoaderType.java:1542)
        at 
edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$11.callLogic(GrouperLoaderType.java:1538)
        at 
edu.internet2.middleware.grouper.util.GrouperCallable$1.callback(GrouperCallable.java:164)
        at 
edu.internet2.middleware.grouper.GrouperSession.callbackGrouperSession(GrouperSession.java:974)
        at 
edu.internet2.middleware.grouper.util.GrouperCallable.callLogicWithSessionIfExists(GrouperCallable.java:161)
        at 
edu.internet2.middleware.grouper.util.GrouperCallable.call(GrouperCallable.java:132)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOf(Arrays.java:2367)
        at 
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:117)
        at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:562)
        at java.lang.StringBuffer.append(StringBuffer.java:320)
        at java.io.StringWriter.write(StringWriter.java:94)
        at 
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.copy(ConfigPropertiesCascadeCommonUtils.java:5088)
        at 
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.copy(ConfigPropertiesCascadeCommonUtils.java:5073)
        at 
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.toString(ConfigPropertiesCascadeCommonUtils.java:5056)
        at 
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase$ConfigFile.retrieveContents(ConfigPropertiesCascadeBase.java:636)
        at 
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.filesNeedReloadingBasedOnContents(ConfigPropertiesCascadeBase.java:984)
        at 
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.retrieveFromConfigFileOrCache(ConfigPropertiesCascadeBase.java:926)
        at 
edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.retrieveConfig(ConfigPropertiesCascadeBase.java:89)
        at 
edu.internet2.middleware.grouper.cfg.GrouperConfig.retrieveConfig(GrouperConfig.java:179)
        at 
edu.internet2.middleware.grouper.hibernate.HibernateSession.isReadonlyMode(HibernateSession.java:118)
        at 
edu.internet2.middleware.grouper.hibernate.HibernateSession.<init>(HibernateSession.java:181)
        at 
edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSession(HibernateSession.java:467)
        at 
edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:659)
        ... 13 more