perfSONAR User Q&A and Other Discussion

["Smith, Cody A." <>]

Andy,

 

The first command fails, as that directory already exists. ( It’s also owned properly by maddash:maddash )

On starting the maddash-server service, it runs momentarily and then dies.

Looking at the derby.log in the /var/lib/maddash directory reveals:

 

java.sql.SQLException: Database 'maddash' not found.

        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)

        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)

        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)

        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)

        at org.apache.derby.impl.jdbc.EmbedConnection.newSQLException(Unknown Source)

        at org.apache.derby.impl.jdbc.EmbedConnection.handleDBNotFound(Unknown Source)

        at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)

        at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)

        at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(Unknown Source)

        at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)

        at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)

        at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)

        at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:120)

        at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:143)

        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:148)

        at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)

        at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)

        at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)

        at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Caused by: ERROR XJ004: Database 'maddash' not found.

        at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)

        at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)

        ... 21 more

 

Thoughts?

 

Best,

Cody Smith

Performance Engineer

OARnet

 

From: Andrew Lake <>
Date: Monday, August 6, 2018 at 1:01 PM
To: "Smith, Cody A." <>, "" <>
Subject: Re: [perfsonar-user] Maddash Server fails to start.

 

Hi,

 

It sounds like maybe you didn’t include * in my in my rm command so the “data" directory got deleted instead of its contents. Run the following and you should hopefully be good:

 

mkdir /var/lib/maddash/data

chown -R maddash:maddash  /var/lib/maddash/data

/etc/init.d/maddash-server restart

 

Thanks,

Andy

 

 

 

 

 

On August 6, 2018 at 2:49:57 PM, Smith, Cody A. () wrote:

Andy,

 

The only thing that populates that directory ( momentarily, at that ) is the pid file. Here is the systemctl output after it fails.

 

maddash-server.se  rvice - MaDDash Scheduler and REST Server

   Loaded: loaded (/usr/lib/systemd/system/maddash-server.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Mon 2018-08-06 14:47:19 EDT; 1min 12s ago

  Process: 16047 ExecStart=/bin/sh -c /usr/lib/maddash/maddash-server/bin/startServer.sh /var/run/maddash/maddash-server.pid /usr/lib/maddash/maddash-server/target/maddash-server)

  Process: 16044 ExecStartPre=/bin/chown -R maddash:maddash /var/log/maddash (code=exited, status=0/SUCCESS)

  Process: 16042 ExecStartPre=/bin/chown maddash:maddash /var/run/maddash (code=exited, status=0/SUCCESS)

  Process: 16039 ExecStartPre=/bin/mkdir -p /var/run/maddash (code=exited, status=0/SUCCESS)

Main PID: 16051 (code=exited, status=1/FAILURE)

 

Thoughts // Suggestions?

 

Best,

Cody Smith

Performance Engineer

OARnet

 

From: Andrew Lake <>
Date: Monday, August 6, 2018 at 12:44 PM
To: "Smith, Cody A." <>, "" <>
Subject: Re: [perfsonar-user] Maddash Server fails to start.

 

Hi,

 

The empty directory is interesting, might be why it can'’t connect. MaDDash should be filling that directory when it starts up. Maybe a file permissions issue? Does the directory get filled if you run the following:

 

chown -R maddash:maddash /var/lib/maddash

/etc/init.d/maddash-server restart

 

Thanks,

Andy

 

 

 

On August 6, 2018 at 2:13:18 PM, Smith, Cody A. () wrote:

Andy,

 

Thanks for your reply. So, interestingly enough there’s nothing in that directory, even after rebooting the node.

Are there any other recommendations or things I can glance at?

 

Also, the node has a good amount of free disk space.

 

Thanks!

 

Best,

Cody Smith

Performance Engineer

OARnet

 

From: Andrew Lake <>
Date: Monday, August 6, 2018 at 11:36 AM
To: "Smith, Cody A." <>, "" <>
Subject: Re: [perfsonar-user] Maddash Server fails to start.

 

Hi Cody,

 

Are you able to restart MaDDash or does it immediately crash on boot? It looks like it is having problems connecting to the embedded database it uses to track the alert history. If maddash runs after a restart of the service, it may have just temporarily lost its connection for some reason.

 

If it won’t come back up, there are a couple things you can do. For one, you might want to double-check your disk isn't full since that can cause problems for the embedded database it uses. If the disk is fine, you will likely need to remove the data of the database. Doing so will NOT destroy any measurements since MaDDash does not store those. It will destroy the last 7 days of alert history (i.e. whether a check was red, orange, green, etc). The commands to do so are as follows:

 

/etc/init.d/maddash-server stop

rm -rf /var/lib/maddash/data/*

/etc/init.d/maddash-server start

 

Hope this helps,

Andy

 

 

 

 

On August 6, 2018 at 11:53:54 AM, Smith, Cody A. () wrote:

Hello,

 

Recently our Maddash server has ran into some complications. While the toolkit side of the node is exhibiting no issues, testing seemingly isn’t effected , and I believe I have found the cause of the issue.

 

# systemctl status maddash-server

maddash-server.service - MaDDash Scheduler and REST Server

                Loaded: loaded (/usr/lib/systemd/system/maddash-server.service; enabled; vendor preset: disabled)

                Active: failed (Result: exit-code) since Sun 2018-08-05 15:44:15 EDT; 18h ago

               Main PID: 1287 (code=exited, status=1/FAILURE)

 

So, looking into the /var/log/maddash/maddash-server.out file, we see this:

 

Initialization error: Connections could not be acquired from the underlying database!

java.lang.RuntimeException: Connections could not be acquired from the underlying database!

        at net.es.maddash.MaDDashGlobals.<init>(MaDDashGlobals.java:185)

        at net.es.maddash.MaDDashGlobals.getInstance(MaDDashGlobals.java:506)

        at net.es.maddash.Invoker.main(Invoker.java:55)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at com.simontuffs.onejar.Boot.run(Boot.java:340)

        at com.simontuffs.onejar.Boot.main(Boot.java:166)

 

Has anyone else ran into this issue? ( Or, it’s entirely possible this is a redherring. )

 

Best,

Cody Smith

Performance Engineer

OARnet

 

 

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user