perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

For anyone who takes a look at this, I've restarted the 3 worst hosts, so that the dashboard will clear up, so you'll see data beginning to show up in our dashboard again, with a large gap.  The 3 hosts that were the worst up until this morning were ps-ksu, ps-psu, and ps-washburn.

Dashboard is at:  http://ps-dashboard.kanren.net/maddash-webui

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047

On Fri, Oct 13, 2017 at 9:55 AM, Casey Russell <> wrote:

Group,

     I mentioned it some time back, when I thought it was a problem with my 4 lower powered hosts running out of CPU, but I've been chasing it ever since and it's hitting my larger hosts as well.  Ever since I upgraded to 4.0 several months ago, I've had an issue where regularly, my hosts stop scheduling tests from the mesh.  My dashboard today shows a mess of hosts that failed to schedule tests last night some of them are on their second, (or more) continuous day.

     I can't figure out if this is a problem with the mesh config file or on the hosts (although since it's spread everywhere, even a newly installed CentOS7 host) I'm leaning toward some problem in the mesh config file.

     I'm not sure what to give you that will help, so below you'll find some diagnostic commands from an affected host this morning that is only running bandwidth tests, none of the latency tests scheduled.

Any ideas or help is appreciated.

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047

Since the latency tests were never scheduled, I don't have anything from the API to show you, the mesh config file is at:  

[root@ps-ksu-bw crussell]# pscheduler schedule

2017-10-13T09:47:54-05:00 - 2017-10-13T09:48:23-05:00  (Pending)

throughput --duration PT20S --source ps-fhsu-bw.perfsonar.kanren.net --ip-version 4 --dest ps-ksu-bw.perfsonar.kanren.net --parallel 1 (Run with tool 'iperf3')

https://localhost/pscheduler/tasks/75b82820-db17-4e89-ab92-4c736bc273f4/runs/05948429-c5bc-4f1e-8f78-2989c27bb954

2017-10-13T09:49:33-05:00 - 2017-10-13T09:49:52-05:00  (Pending)

throughput --bandwidth 920000000 --duration PT10S --source ps-esu-bw.perfsonar.kanren.net --ip-version 4 --dest ps-ksu-bw.perfsonar.kanren.net --parallel 1 --udp (Run with tool 'iperf3')

https://localhost/pscheduler/tasks/3b1c5c75-7c23-4cc2-a280-e5c288fcacd8/runs/2a53a85c-19c5-4810-8553-9d75ae2c0529

2017-10-13T09:52:08-05:00 - 2017-10-13T09:52:27-05:00  (Pending)

throughput --bandwidth 920000000 --duration PT10S --source ps-bryant-bw.perfsonar.kanren.net --ip-version 6 --dest ps-ksu-bw.perfsonar.kanren.net --parallel 1 --udp (Run with tool 'iperf3')

https://localhost/pscheduler/tasks/fae2a1a4-11d4-4cf6-ba76-70c1caa8189d/runs/bb1c3d43-7662-4774-8501-6aacdd79f5e4

2017-10-13T09:58:44-05:00 - 2017-10-13T09:59:03-05:00  (Pending)

throughput --bandwidth 920000000 --duration PT10S --source ps-bryant-bw.perfsonar.kanren.net --ip-version 4 --dest ps-ksu-bw.perfsonar.kanren.net --parallel 1 --udp (Run with tool 'iperf3')

https://localhost/pscheduler/tasks/c663e121-a11d-49b4-8f21-25d77ef2ff76/runs/26c16f16-7d61-4d82-ba57-36c55d154bb3

2017-10-13T10:07:36-05:00 - 2017-10-13T10:08:05-05:00  (Pending)

throughput --duration PT20S --source ps-ku-bw.perfsonar.kanren.net --ip-version 6 --dest ps-ksu-bw.perfsonar.kanren.net --parallel 1 (Run with tool 'iperf3')

https://localhost/pscheduler/tasks/b51ab890-6c4e-4e8f-acf8-422154073114/runs/59151f65-36b6-4298-bbe2-e7a67aef3713

2017-10-13T10:08:38-05:00 - 2017-10-13T10:08:57-05:00  (Pending)

throughput --bandwidth 920000000 --duration PT10S --source ps-ku-bw.perfsonar.kanren.net --ip-version 6 --dest ps-ksu-bw.perfsonar.kanren.net --parallel 1 --udp (Run with tool 'iperf3')

https://localhost/pscheduler/tasks/5fd54280-af27-4f9e-b1a9-b62462f3ce0c/runs/231f1eb1-5cda-4e39-9a3c-bfc18333dd06

2017-10-13T10:10:18-05:00 - 2017-10-13T10:10:47-05:00  (Pending)

throughput --duration PT20S --source ps-esu-bw.perfsonar.kanren.net --ip-version 6 --dest ps-ksu-bw.perfsonar.kanren.net --parallel 1 (Run with tool 'iperf3')

https://localhost/pscheduler/tasks/f38efff5-851f-4d4f-9388-155d25bfe288/runs/5a803842-539a-403a-ae31-7f2351d527cb

2017-10-13T10:10:49-05:00 - 2017-10-13T10:11:08-05:00  (Pending)

throughput --bandwidth 920000000 --duration PT10S --source ps-esu-bw.perfsonar.kanren.net --ip-version 6 --dest ps-ksu-bw.perfsonar.kanren.net --parallel 1 --udp (Run with tool 'iperf3')

https://localhost/pscheduler/tasks/efaf912b-a6ef-45f3-a54a-cf157fdd8734/runs/a4a72b62-362f-4b79-a0a6-fe19e841431e

2017-10-13T10:16:39-05:00 - 2017-10-13T10:17:08-05:00  (Pending)

throughput --duration PT20S --source ps-fhsu-bw.perfsonar.kanren.net --ip-version 6 --dest ps-ksu-bw.perfsonar.kanren.net --parallel 1 (Run with tool 'iperf3')

https://localhost/pscheduler/tasks/25f395da-0fdb-41e5-b2a1-ac2c260f7279/runs/a58da605-0bf2-4ada-97eb-f8753a029519

2017-10-13T10:36:46-05:00 - 2017-10-13T10:37:15-05:00  (Pending)

throughput --duration PT20S --source ps-esu-bw.perfsonar.kanren.net --ip-version 4 --dest ps-ksu-bw.perfsonar.kanren.net --parallel 1 (Run with tool 'iperf3')

https://localhost/pscheduler/tasks/48092447-bd16-4220-a849-34ca8a0889c8/runs/05e74fcc-a287-48cc-bd35-ff76c29f9ef5

[root@ps-ksu-bw crussell]# service pscheduler-runner status

runner (pid  13073) is running...

[root@ps-ksu-bw crussell]# service pscheduler-ticker status

ticker (pid  13071) is running...

[root@ps-ksu-bw crussell]# service pscheduler-archiver status

archiver (pid  13078) is running...

[root@ps-ksu-bw crussell]# service pscheduler-server status

pscheduler-server: unrecognized service

[root@ps-ksu-bw crussell]# service pscheduler-scheduler status

scheduler (pid  13090) is running...

[root@ps-ksu-bw crussell]# ps -ax | grep pscheduler

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ

 3448 pts/0    S+     0:00 grep pscheduler

 8236 ?        Ss     0:17 postgres: pscheduler pscheduler 127.0.0.1(41520) idle   

13071 ?        Sl     0:42 /usr/bin/python /usr/libexec/pscheduler/daemons/ticker --daemon --pid-file /var/run/pscheduler-ticker.pid --dsn @/etc/pscheduler/database/database-dsn

13073 ?        Sl    21:20 /usr/bin/python /usr/libexec/pscheduler/daemons/runner --daemon --pid-file /var/run/pscheduler-runner.pid --dsn @/etc/pscheduler/database/database-dsn

13075 ?        Ss     1:20 postgres: pscheduler pscheduler 127.0.0.1(48114) idle   

13076 ?        Ss     9:40 postgres: pscheduler pscheduler 127.0.0.1(48116) idle   

13078 ?        S     67:00 /usr/bin/python /usr/libexec/pscheduler/daemons/archiver --daemon --pid-file /var/run/pscheduler-archiver.pid --dsn @/etc/pscheduler/database/database-dsn

13079 ?        Ss   360:11 postgres: pscheduler pscheduler 127.0.0.1(48118) idle   

13081 ?        Ss     8:31 postgres: pscheduler pscheduler 127.0.0.1(48122) idle   

13083 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(48126) idle   

13090 ?        Sl    65:19 /usr/bin/python /usr/libexec/pscheduler/daemons/scheduler --daemon --pid-file /var/run/pscheduler-scheduler.pid --dsn @/etc/pscheduler/database/database-dsn

13108 ?        Ss   115:36 postgres: pscheduler pscheduler 127.0.0.1(48132) idle   

13114 ?        Ss     0:00 postgres: pscheduler pscheduler 127.0.0.1(48136) idle   

28737 ?        Ss     0:01 postgres: pscheduler pscheduler 127.0.0.1(55217) idle   

[root@ps-ksu-bw crussell]# 

[root@ps-ksu-bw crussell]# service perfsonar-meshconfig-agent

usage: /etc/init.d/perfsonar-meshconfig-agent (start|stop|restart|help)

start      - start perfSONAR MeshConfig Agent

stop       - stop perfSONAR MeshConfig Agent

restart    - restart perfSONAR MeshConfig Agent if running by sending a SIGHUP or start if 

             not running

status     - Indicates if the service is running

help       - this screen

[root@ps-ksu-bw crussell]# service perfsonar-meshconfig-agent restart

/etc/init.d/perfsonar-meshconfig-agent stop: perfSONAR MeshConfig Agent stopped

waiting...

/usr/lib/perfsonar/bin/perfsonar_meshconfig_agent --config=/etc/perfsonar/meshconfig-agent.conf --pidfile=/var/run/perfsonar-meshconfig-agent.pid --logger=/etc/perfsonar/meshconfig-agent-logger.conf --user=perfsonar --group=perfsonar

/etc/init.d/perfsonar-meshconfig-agent start: perfSONAR MeshConfig Agent started

[root@ps-ksu-bw crussell]# tail -n 50 /var/log/perfsonar/meshconfig-agent.log 

2017/10/12 20:10:55 (8826) INFO> perfsonar_meshconfig_agent:438 main:: - Added 3 new tasks, and deleted 0 old tasks

2017/10/12 21:10:10 (8826) INFO> perfsonar_meshconfig_agent:438 main:: - Added 1 new tasks, and deleted 0 old tasks

2017/10/13 03:10:37 (8826) INFO> perfsonar_meshconfig_agent:438 main:: - Added 2 new tasks, and deleted 0 old tasks

2017/10/13 04:10:40 (8826) WARN> perfsonar_meshconfig_agent:430 main:: - Problem determining which pscheduler to submit test to for deletion, skipping test throughput/iperf3(ps-ksu-bw.perfsonar.kanren.net->ps-fhsu-bw.perfsonar.kanren.net): 500 Internal Server Error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<html><head>

<title>500 Internal Server Error</title>

</head><body>

<h1>Internal Server Error</h1>

<p>The server encountered an internal error or

misconfiguration and was unable to complete

your request.</p>

<p>Please contact the server administrator at 

 root@localhost to inform them of the time this error occurred,

 and the actions you performed just before this error.</p>

<p>More information about this error may be available

in the server error log.</p>

</body></html>

2017/10/13 07:11:39 (8826) INFO> perfsonar_meshconfig_agent:438 main:: - Added 5 new tasks, and deleted 0 old tasks

2017/10/13 09:20:23 (8826) INFO> perfsonar_meshconfig_agent:438 main:: - Added 97 new tasks, and deleted 0 old tasks