Grouper Users - Open Discussion List

["Hyzer, Chris" <>]

Im afraid there might be a leak and a daily bounce would be a temporary fix...  is that ok for now?  :)

---

From: <> on behalf of Michael R Gettes <>
Sent: Monday, January 9, 2017 4:40 PM
To: Mailing List
Subject: [grouper-users] GC overhead limit exceeded on 2.3.0 + patches

 

Hi,

We run in a 12GB VM with -Xmx4096m on the loader and we are not using any swap space and 2.3.0 + patches (applied in december, i think).

After a long running loader job we see the error at the bottom of this mail.  When we get this error, remaining loader jobs go bad and we have to restart the loader.  We are going to add memory.  But, should we be using a specific GC algorithm or taking some other steps for a large environment?  We currently have about 8M memberships (14M loader memberships) over 80K groups.  We have only been in production since June.  I am seeking any advice/guidance on what to do cuz I fear we are going to see this problem again.  We have a number of loader jobs managing groups of 100K to 230K.  We do run concurrent jobs on multiple servers (currently 2 cores for 2 loader job servers about to go to 4 cores).

I should also note: while we run with 4096m the loader nightly report shows heapsize as 3GB.
heapSize: 3 GB
heapMaxSize: 3 GB
heapSizeFree: 1 GB
version: 2.3.0, build date: 2016/04/16 05:37:21, env: PRODUCTION
os.name: Linux
os.arch: amd64
os.version: 2.6.32-642.6.2.el6.x86_64
java.version: 1.7.0_121
java.vendor: Oracle Corporation

Thank you!

/mrg

2017-01-09 08:56:48,254: [pool-1-thread-6229] ERROR GrouperLoaderType.syncGroupLogicForOneGroup(1634) -  - Error in job for group: Course:2171:UP:AA:044532:all
edu.internet2.middleware.grouper.internal.dao.GrouperDAOException: Problem in HibernateSession: null
        at edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch(HibernateSession.java:604)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:710)
        at edu.internet2.middleware.grouper.app.loader.db.Hib3GrouperLoaderLog.store(Hib3GrouperLoaderLog.java:730)
        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType.syncGroupLogicForOneGroup(GrouperLoaderType.java:1621)
        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType.access$200(GrouperLoaderType.java:118)
        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$11.callLogic(GrouperLoaderType.java:1542)
        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$11.callLogic(GrouperLoaderType.java:1538)
        at edu.internet2.middleware.grouper.util.GrouperCallable$1.callback(GrouperCallable.java:164)
        at edu.internet2.middleware.grouper.GrouperSession.callbackGrouperSession(GrouperSession.java:974)
        at edu.internet2.middleware.grouper.util.GrouperCallable.callLogicWithSessionIfExists(GrouperCallable.java:161)
        at edu.internet2.middleware.grouper.util.GrouperCallable.call(GrouperCallable.java:132)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOf(Arrays.java:2367)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:117)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:562)
        at java.lang.StringBuffer.append(StringBuffer.java:320)
        at java.io.StringWriter.write(StringWriter.java:94)
        at edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.copy(ConfigPropertiesCascadeCommonUtils.java:5088)
        at edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.copy(ConfigPropertiesCascadeCommonUtils.java:5073)
        at edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeCommonUtils.toString(ConfigPropertiesCascadeCommonUtils.java:5056)
        at edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase$ConfigFile.retrieveContents(ConfigPropertiesCascadeBase.java:636)
        at edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.filesNeedReloadingBasedOnContents(ConfigPropertiesCascadeBase.java:984)
        at edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.retrieveFromConfigFileOrCache(ConfigPropertiesCascadeBase.java:926)
        at edu.internet2.middleware.grouperClient.config.ConfigPropertiesCascadeBase.retrieveConfig(ConfigPropertiesCascadeBase.java:89)
        at edu.internet2.middleware.grouper.cfg.GrouperConfig.retrieveConfig(GrouperConfig.java:179)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.isReadonlyMode(HibernateSession.java:118)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.<init>(HibernateSession.java:181)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSession(HibernateSession.java:467)
        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:659)
        ... 13 more