perfSONAR User Q&A and Other Discussion

[Brian Candler <>]

I have installed three servers from 
pS-Toolkit-4.0-CentOS7-FullInstall-x86_64-2017Apr21.iso, but on all of 
them the front status page is showing pscheduler status "Not running"

I have configured some tests (latency and throughput) on one of the 
hosts pointing at the other two, but the front page says there are no 
test results. Some messages from /var/log/pscheduler/pscheduler.log are 
shown at the end of this message but I can't make much sense out of them.

Postgres seems to be fine (e.g. "sudo -u postgres psql pscheduler" and 
"\d" shows me tables)

Doing some tests suggested at 
http://docs.perfsonar.net/troubleshooting_overview.html I see that:

"pscheduler ping localhost" says it is alive.

"pscheduler task idle --duration PT2S" works.

I am at a bit of a loss now as to how to debug this. Any suggestions?  
As a starting point, under what circumstances does the front page 
display "Not running" for pscheduler?  And can you determine anything 
useful from the logs below?

May or may not be related: often when I go to the front page the 
"Loading..." spinner remains forever. But if I click outside it, it goes 
away.

Two of the machines are quite old with 3GB and 4GB RAM respectively, but 
one is a new Skull Canyon NUC with 32GB and SSD.

Thanks,

Brian.


Jun 13 22:01:02 perfsonar-core journal: runner INFO     50: With 
powstream: latencybg --data-ports 8760-9960 --dest y.y.y.y 
--packet-padding 0 --source x.x.x.x --ip-version 4 --packet-interval 0.1 
--duration PT86400S --packet-count 600
Jun 13 22:01:02 perfsonar-core journal: runner WARNING  42: Starting 
0:00:00.558099 later than scheduled
Jun 13 22:01:05 perfsonar-core journal: pscheduler-api INFO Started
Jun 13 22:01:06 perfsonar-core journal: pscheduler-api INFO Limits 
loaded from /etc/pscheduler/limits.conf
Jun 13 22:01:12 perfsonar-core journal: ticker WARNING  Queue maintainer 
got exception server closed the connection unexpectedly#012#011This 
probably means the server terminated abnormally#012#011before or while 
processing the request.
Jun 13 22:01:12 perfsonar-core journal: ticker WARNING  Queue maintainer 
got exception server closed the connection unexpectedly#012#011This 
probably means the server terminated abnormally#012#011before or while 
processing the request.
Jun 13 22:01:12 perfsonar-core journal: safe_run/scheduler ERROR    
Program threw an exception after 0:00:16.031096
Jun 13 22:01:12 perfsonar-core journal: safe_run/scheduler ERROR    
Exception: DatabaseError: server closed the connection 
unexpectedly#012#011This probably means the server terminated 
abnormally#012#011before or while processing the 
request.#012#012Traceback (most recent call last):#012  File 
"/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 41, in 
safe_run#012    function()#012  File 
"/usr/libexec/pscheduler/daemons/scheduler", line 660, in 
<lambda>#012    pscheduler.safe_run(lambda: main_program())#012  File 
"/usr/libexec/pscheduler/daemons/scheduler", line 544, in 
main_program#012    """)#012DatabaseError: server closed the connection 
unexpectedly#012#011This probably means the server terminated 
abnormally#012#011before or while processing the request.
Jun 13 22:01:12 perfsonar-core journal: safe_run/scheduler ERROR    
Waiting 2.0 seconds before restarting
Jun 13 22:01:12 perfsonar-core journal: safe_run/runner ERROR Program 
threw an exception after 0:00:16.040845
Jun 13 22:01:12 perfsonar-core journal: safe_run/runner ERROR Exception: 
DatabaseError: server closed the connection unexpectedly#012#011This 
probably means the server terminated abnormally#012#011before or while 
processing the request.#012#012Traceback (most recent call last):#012  
File "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 41, 
in safe_run#012    function()#012  File 
"/usr/libexec/pscheduler/daemons/runner", line 909, in <lambda>#012    
pscheduler.safe_run(lambda: main_program())#012  File 
"/usr/libexec/pscheduler/daemons/runner", line 817, in 
main_program#012    """, [refresh]);#012DatabaseError: server closed the 
connection unexpectedly#012#011This probably means the server terminated 
abnormally#012#011before or while processing the request.
Jun 13 22:01:12 perfsonar-core journal: safe_run/runner ERROR Waiting 
2.0 seconds before restarting
Jun 13 22:01:14 perfsonar-core journal: safe_run/scheduler ERROR    
Restarting
Jun 13 22:01:14 perfsonar-core journal: safe_run/runner ERROR Restarting
Jun 13 22:01:16 perfsonar-core journal: safe_run/ticker ERROR Program 
threw an exception after 0:00:19.574050
Jun 13 22:01:16 perfsonar-core journal: safe_run/ticker ERROR Exception: 
OperationalError: terminating connection due to administrator 
command#012server closed the connection unexpectedly#012#011This 
probably means the server terminated abnormally#012#011before or while 
processing the request.#012#012Traceback (most recent call last):#012  
File "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 41, 
in safe_run#012    function()#012  File 
"/usr/libexec/pscheduler/daemons/ticker", line 156, in <lambda>#012    
pscheduler.safe_run(lambda: main_program())#012  File 
"/usr/libexec/pscheduler/daemons/ticker", line 133, in 
main_program#012    cursor.execute("SELECT 
ticker()")#012OperationalError: terminating connection due to 
administrator command#012server closed the connection 
unexpectedly#012#011This probably means the server terminated 
abnormally#012#011before or while processing the request.
Jun 13 22:01:16 perfsonar-core journal: safe_run/ticker ERROR Waiting 
2.0 seconds before restarting
Jun 13 22:01:18 perfsonar-core journal: safe_run/ticker ERROR Restarting
Jun 13 22:01:51 perfsonar-core journal: runner INFO     62: Running 
https://perfsonar-core.example.com/pscheduler/tasks/e9b259b6-9d84-4aa5-a8f9-616192807bf1/runs/f2a8a95b-913d-4b2a-ac6b-d273135babba
Jun 13 22:01:51 perfsonar-core journal: runner INFO     62: With 
traceroute: trace --dest y.y.y.y --ip-version 4 --source x.x.x.x
Jun 13 22:01:51 perfsonar-core journal: runner ERROR    62: Failed to 
detect if system is paused and/or set missed state: terminating 
connection due to administrator command#012server closed the connection 
unexpectedly#012#011This probably means the server terminated 
abnormally#012#011before or while processing the request.
Jun 13 22:01:51 perfsonar-core journal: runner WARNING  Error committing 
DB transaction: connection already closed
Jun 13 22:01:51 perfsonar-core journal: runner ERROR    62: Failed to 
set running state: terminating connection due to administrator 
command#012server closed the connection unexpectedly#012#011This 
probably means the server terminated abnormally#012#011before or while 
processing the request.
Jun 13 22:01:51 perfsonar-core journal: runner WARNING  Error committing 
DB transaction: connection already closed
Jun 13 22:02:08 perfsonar-core journal: runner INFO     49: Running 
https://perfsonar-core.example.com/pscheduler/tasks/9e80fd96-48b5-484e-b09c-3e0948f1c72e/runs/e8903e4d-0e3f-4b1e-aece-936aea6e709c
Jun 13 22:02:08 perfsonar-core journal: runner INFO     49: With iperf3: 
throughput --duration PT20S --source x.x.x.x --ip-version 4 --dest 
z.z.z.z --parallel 1
Jun 13 22:02:30 perfsonar-core journal: runner INFO     50: Posted 
result to 
https://x.x.x.x/pscheduler/tasks/7daa09a3-979b-471c-a688-3a2776925ecc/runs/bc92b42c-6b74-4592-be45-abfd45066768
Jun 13 22:02:30 perfsonar-core journal: runner INFO     38: Posted 
result to 
https://x.x.x.x/pscheduler/tasks/082c6dc3-223e-45de-a820-2fec02688f5d/runs/ec991742-8030-45fa-bd0f-cc081da9b849
Jun 13 22:02:30 perfsonar-core journal: runner INFO     42: Posted 
result to 
https://x.x.x.x/pscheduler/tasks/5f0d75f9-fdf4-44e7-816f-5d286055800a/runs/ded286c1-1100-42c2-96a6-92e8d818b414
Jun 13 22:02:30 perfsonar-core journal: runner INFO     47: Posted 
result to 
https://x.x.x.x/pscheduler/tasks/c13a1a56-0d2e-432d-ac92-f673107598bf/runs/0360bdf3-cc4b-4130-bb89-002723fe5cd7
Jun 13 22:02:31 perfsonar-core journal: runner INFO     49: Run succeeded.