Grouper Users - Open Discussion List

["Kediyal, Prashant" <>]

Hi Chris,

We have been experiencing a lot of problems where grouper web service is crashing almost once or twice every week.

Eventually we get the problem where grouper is unable to checkout jdbc connections. 

I tried to read the code at HibernateSession.java:571 and am unsure if the problem arises because connections aren’t getting closed; one of the typical error we see in the logs is:

2015-04-13 20:04:32,656: [TP-Processor19] WARN  WsStemLookup.retrieveStemIfNeeded(208) - < etc:wsusers:grouper-wsuser-ursa - 164.67.134.21 > - Invalid query: WsStemLookup[

  stemFindResult=INVALID_QUERY]

2015-04-13 20:04:32,661: [pool-1-thread-750] ERROR Member$2.call(3845) -  - Error updating member attributes: 

edu.internet2.middleware.grouper.internal.dao.GrouperDAOException: Problem in HibernateSession: HibernateSession: isNew: true, isReadonly: false, grouperTransactionType: READ_WRITE_NEW

        at edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch(HibernateSession.java:508)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:581)

        at edu.internet2.middleware.grouper.Member$2.call(Member.java:3827)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.NoSuchFieldError: INSTANCE

        at edu.internet2.middleware.grouper.hibernate.HibUtils.hibernateType(HibUtils.java:609)

        at edu.internet2.middleware.grouper.hibernate.HibUtils.hibernateTypes(HibUtils.java:635)

        at edu.internet2.middleware.grouper.hibernate.BySqlStatic.attachParams(BySqlStatic.java:298)

        at edu.internet2.middleware.grouper.hibernate.BySql.executeSql(BySql.java:143)

        at edu.internet2.middleware.grouper.Member$2$1.callback(Member.java:3839)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:571)

From: Chris Hyzer <>
Date: Tuesday, March 24, 2015 at 11:02 AM
To: Prashant Kediyal <>
Cc: "" <>
Subject: RE: grouper web service crash

I think add more memory.

 

There are limits in the config files for max number of records retrieved, that might help.

 

Hmmm, not sure why not check out JDBC connections, maybe need to change a database setting or a config settings to have a larger pool?

 

Thanks,

Chris

 

From: Kediyal, Prashant []
Sent: Tuesday, March 24, 2015 2:00 PM
To: Chris Hyzer
Cc:
Subject: grouper web service crash

 

Hi Chris,

 

We had an incident where grouper web service crashed. Tomcat logs showed that there was an out of heap memory exception.

We suspect that this was caused because an application was retrieving 10’s of thousands of groups information. Is there any way we 

could have configured grouper to fail more gracefully?

 

Secondly, the same application had typos in some web service requests and we saw invalid query being logged. Later on, grouper crashed

again because it could not check out a jdbc connection from the pool — had to restart tomcat again.

 

Any advice?

Prashant