Grouper Users - Open Discussion List

["Kediyal, Prashant" <>]

Many Thanks. Let us see.

From: Chris Hyzer <>
Date: Thursday, April 16, 2015 at 8:57 AM
To: Prashant Kediyal <>, Shilen Patel <>
Cc: "" <>
Subject: RE: grouper web service crash

1.       That is a superfluous warning, you can ignore it, or you can patch the WS manually and recompile so it doesn’t happen anymore

 

https://bugs.internet2.edu/jira/browse/GRP-1129

 

2.       I don’t know why you wouldn’t be able to get connections from the database.  If the DBA can monitor how many connections from where are being used, maybe it can point to a problem?   Maybe you can change logging from c3p0?  Maybe put that in its own file which is rotated and cant fill up the filesystem (it’s a LOT of output).  Maybe you need to increase the max in c3p0 to 50 instead of 16.   Maybe you need more app servers.

 

log4j.logger.com.mchange = DEBUG

 

3.       Regarding this: Caused by: java.lang.NoSuchFieldError: INSTANCE, When I google that it is a jar problem.  Ive never seen it before.  Was the whole stack in the email?  Since you have not used the Grouper jars but have pulled them in from maven, maybe something isn’t correct?  Nothing jumps out at me when I glance at the list (looking for duplicates).  Maybe you could try building Grouper WS without maven (the Grouper ant file) and see if it helps?  J   Note, improving jar management is on our roadmap for v2.3

 

Thanks,

Chris

 

 

 

From: [] On Behalf Of Kediyal, Prashant
Sent: Tuesday, April 14, 2015 8:28 AM
To: Chris Hyzer; Shilen Patel
Cc:
Subject: [grouper-users] Re: grouper web service crash

 

Hi Chris,

 

We have been experiencing a lot of problems where grouper web service is crashing almost once or twice every week.

 

Eventually we get the problem where grouper is unable to checkout jdbc connections. 

 

I tried to read the code at HibernateSession.java:571 and am unsure if the problem arises because connections aren’t getting closed; one of the typical error we see in the logs is:

 

2015-04-13 20:04:32,656: [TP-Processor19] WARN  WsStemLookup.retrieveStemIfNeeded(208) - < etc:wsusers:grouper-wsuser-ursa - 164.67.134.21 > - Invalid query: WsStemLookup[

  stemFindResult=INVALID_QUERY]

2015-04-13 20:04:32,661: [pool-1-thread-750] ERROR Member$2.call(3845) -  - Error updating member attributes: 

edu.internet2.middleware.grouper.internal.dao.GrouperDAOException: Problem in HibernateSession: HibernateSession: isNew: true, isReadonly: false, grouperTransactionType: READ_WRITE_NEW

        at edu.internet2.middleware.grouper.hibernate.HibernateSession._internal_hibernateSessionCatch(HibernateSession.java:508)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:581)

        at edu.internet2.middleware.grouper.Member$2.call(Member.java:3827)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.NoSuchFieldError: INSTANCE

        at edu.internet2.middleware.grouper.hibernate.HibUtils.hibernateType(HibUtils.java:609)

        at edu.internet2.middleware.grouper.hibernate.HibUtils.hibernateTypes(HibUtils.java:635)

        at edu.internet2.middleware.grouper.hibernate.BySqlStatic.attachParams(BySqlStatic.java:298)

        at edu.internet2.middleware.grouper.hibernate.BySql.executeSql(BySql.java:143)

        at edu.internet2.middleware.grouper.Member$2$1.callback(Member.java:3839)

        at edu.internet2.middleware.grouper.hibernate.HibernateSession.callbackHibernateSession(HibernateSession.java:571)

 

 

 

From: Chris Hyzer <>
Date: Tuesday, March 24, 2015 at 11:02 AM
To: Prashant Kediyal <>
Cc: "" <>
Subject: RE: grouper web service crash

 

I think add more memory.

 

There are limits in the config files for max number of records retrieved, that might help.

 

Hmmm, not sure why not check out JDBC connections, maybe need to change a database setting or a config settings to have a larger pool?

 

Thanks,

Chris

 

From: Kediyal, Prashant []
Sent: Tuesday, March 24, 2015 2:00 PM
To: Chris Hyzer
Cc:
Subject: grouper web service crash

 

Hi Chris,

 

We had an incident where grouper web service crashed. Tomcat logs showed that there was an out of heap memory exception.

We suspect that this was caused because an application was retrieving 10’s of thousands of groups information. Is there any way we 

could have configured grouper to fail more gracefully?

 

Secondly, the same application had typos in some web service requests and we saw invalid query being logged. Later on, grouper crashed

again because it could not check out a jdbc connection from the pool — had to restart tomcat again.

 

Any advice?

Prashant