perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Casey Russell writes:

 

     I've seen similar errors mentioned before, but I'm having a hard time tracing these back to their origins.  I've started getting dozens (and maybe more than a hundred) emails a day from my single Centos7 machine complaining of a problem running a traceroute test (if I'm interpreting it correctly).  The line from the pscheduler.log is as follows:

 

Feb 21 07:55:18 ps-fhsu-bw journal: runner INFO     12356773: Run failed 1: Traceback (most recent call last):#012  File "/usr/libexec/pscheduler/commands/../classes/tool/traceroute/run", line 389, in <module>#012    ases = pscheduler.as_bulk_resolve(ips, threads=len(ips))#012  File "/usr/lib/python2.7/site-packages/pscheduler/psas.py", line 76, in as_bulk_resolve#012    processes=min(len(candidates), threads))#012  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 714, in __init__#012    Pool.__init__(self, processes, initializer, initargs)#012  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 152, in __init__#012    raise ValueError("Number of processes must be at least 1")#012ValueError: Number of processes must be at least 1

 

This was sufficient information for me to hunt down the problem.  It’s a bug in a library routine used by the traceroute tool to determine the AS numbers of IP addresses.  I’ve opened a ticket on it:  https://github.com/perfsonar/pscheduler/issues/588.  It’s an easy fix and will show up in 4.1 or the next bugfix release if we do one of those.

 

For future reference, the large number before the colon is an internal ID used by the database.  (Yes, your system has done 12.3 million runs since the database was last reset.)  Anything else in the log related to it will have that number prepended, so a quick application of grep will pull out any additional information.

 

When I hand-restart pscheduler-scheduler (and occasionally when I restart the machine), I get an error about not having permissions to the pscheduler-archiver.pid file.  That has been a problem that has come and gone ever since I upgraded this machine to CentOS7 6 or more months ago.  So it may or may not be related to this problem.

 

That’s unrelated.  At some point since pScheduler was introduced, we did an overhaul to the scripts that start and stop the services that included not using root to create the PID files.  In some cases the old, root-owned files got left around and caused this problem.  The workaround for the time being is to stop the service, kill off any additional copies you see in the process table, remove /var/run/pscheduler-xxx.pid (where xxx is the service name) and then start the service.  After that, things should be fine.  I’ve opened a ticket on that and will get some clean-up code into the next release:   https://github.com/perfsonar/pscheduler/issues/589.

 

--Mark