perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

Thank you Andy, 

     That fixed the problem with Maddash not starting.  I almost did that yesterday, but only having a single host with Maddash on it, I didn't have another example of what the permissions SHOULD look like on that folder.  It was a mishmash of maddash and root owned files.

     However, I do have a new problem.  Now that Maddash is running, it's not able to run the checks to evaluate the status of the grid squares.  PS tests are all running and the data is being collected.  If you click on a maddash grid square, the graph is created and the data is current (and within bounds).  You can see it here:  http://ps-dashboard.perfsonar.kanren.net/maddash-webui

     However the grid is always in "unable to retrieve" data state.  The specific error in the web interface is:  " 500 Can't connect to ps-dashboard.perfsonar.kanren.net:80 (connect: Network is unreachable): 500 Can't connect to ps-dashboard.perfsonar.kanren.net:80 (connect: Network is unreachable) "  That hinted to me that maybe the host couldn't contact it's self on port 80 in general when trying to run the status checks.  however I can log in and use wget to grab a quick page from the localhost on port 80 via the hostname as indicated.

     So I looked at the logs and found this in the /var/log/maddash/meshconfig-guiagent.log:

2017/04/21 08:41:16 (12806) ERROR> GUIAgent.pm:193 perfSONAR_PS::MeshConfig::GUIAgent::__configure_guis - Problem with mesh configuration: 500 Can't connect to ps-dashboard.perfsonar.kanren.net:80 (connect: Network is unreachable): 500 Can't connect to ps-dashboard.perfsonar.kanren.net:80 (connect: Network is unreachable)

2017/04/21 08:47:48 (3310) ERROR> GUIAgent.pm:193 perfSONAR_PS::MeshConfig::GUIAgent::__configure_guis - Problem with mesh configuration: 500 Can't connect to ps-dashboard.perfsonar.kanren.net:80 (connect: Network is unreachable): 500 Can't connect to ps-dashboard.perfsonar.kanren.net:80 (connect: Network is unreachable)

That indicated to me that when the Maddash server started, the guiagent couldn't read the Mesh Config file at:  http://ps-dashboard.perfsonar.kanren.net/toolkit/KanREN.json  But again, it seems reachable from both remote hosts and the local dashboard host.  

Any thoughts?  hints as to where to look next?

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047

On Thu, Apr 20, 2017 at 6:16 PM, Andrew Lake <> wrote:

Hi Casey,

Is the directory /etc/maddash/maddash-server and it’s contents owned and readable by the maddash user? Not sure why it would have changed but the following should make it readable if its not:

chown -R maddash:maddash  /etc/maddash

chmod 755  /etc/maddash/maddash-server

chmod 644 /etc/maddash/maddash-server/maddash.yaml

Thanks,

Andy

On April 20, 2017 at 5:10:08 PM, Casey Russell () wrote:

Group,

     After waiting patiently for 4.0.0.1 I upgraded one of my 3.5.1 hosts today and promptly broke it (yay me!).  Fortunately most everything appears to be working except for Maddash. 

     The host is my Mesh Config creator/distributor, my Central MA for the Mesh, a full toolkit install and lives on a VM.  I upgraded it directly from 3.5.1 via yum and it seemed to go fairly well.  

     If I go to the toolkit page and log in, I still see the test results for the full mesh and the results are currently updating, so I'm faaairly confident that the previous problems reported (postgres, etc) aren't at fault.  But the Maddash server stops about 15 seconds after starting.  

[root@ps-dashboard crussell]# service maddash-server restart

Stopping maddash-server:                                   [FAILED]

Starting maddash-server:                                   [  OK  ]

[root@ps-dashboard crussell]# service maddash-server status

maddash-server (pid  8173) is running...

[root@ps-dashboard crussell]# service maddash-server status

maddash-server (pid  8173) is running...

[root@ps-dashboard crussell]# service maddash-server status

maddash-server dead but pid file exists

[root@ps-dashboard crussell]# 

The only hint I've found thus far in the logs comes from /var/log/maddash/maddash-server.out  (here is the end of the output after a restart of the service.)

JarClassLoader: Warning: com/mchange/v2/util/DoubleWeakHashMap$WKey.class in lib/c3p0-0.9.1.1.jar is hidden by lib/mchange-commons-java-0.2.11.jar (with different bytecode)

JarClassLoader: Warning: com/mchange/v2/util/DoubleWeakHashMap$WVal.class in lib/c3p0-0.9.1.1.jar is hidden by lib/mchange-commons-java-0.2.11.jar (with different bytecode)

JarClassLoader: Warning: com/mchange/v2/util/DoubleWeakHashMap.class in lib/c3p0-0.9.1.1.jar is hidden by lib/mchange-commons-java-0.2.11.jar (with different bytecode)

JarClassLoader: Warning: com/mchange/v2/util/ResourceClosedException.class in lib/c3p0-0.9.1.1.jar is hidden by lib/mchange-commons-java-0.2.11.jar (with different bytecode)

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

MLog initialization issue: slf4j found no binding or threatened to use its (dangerously silent) NOPLogger. We consider the slf4j library not found.

java.nio.file.AccessDeniedException: /etc/maddash/maddash-server

        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)

        at sun.nio.fs.UnixException.asIOException(UnixException.java:111)

        at sun.nio.fs.LinuxWatchService$Poller.implRegister(LinuxWatchService.java:263)

        at sun.nio.fs.AbstractPoller.processRequests(AbstractPoller.java:260)

        at sun.nio.fs.LinuxWatchService$Poller.run(LinuxWatchService.java:326)

        at java.lang.Thread.run(Thread.java:745)

Initialization error: /etc/maddash/maddash-server

java.lang.RuntimeException: /etc/maddash/maddash-server

        at net.es.maddash.MaDDashGlobals.<init>(MaDDashGlobals.java:196)

        at net.es.maddash.MaDDashGlobals.getInstance(MaDDashGlobals.java:506)

        at net.es.maddash.Invoker.main(Invoker.java:55)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at com.simontuffs.onejar.Boot.run(Boot.java:340)

        at com.simontuffs.onejar.Boot.main(Boot.java:166)

It looks like a permissions error, but I can't tell what (specifically) is failing.

Thoughts?  

P.S. and thank you in advance

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047