perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

Group,

     I've seen similar errors mentioned before, but I'm having a hard time tracing these back to their origins.  I've started getting dozens (and maybe more than a hundred) emails a day from my single Centos7 machine complaining of a problem running a traceroute test (if I'm interpreting it correctly).  The line from the pscheduler.log is as follows:

Feb 21 07:55:18 ps-fhsu-bw journal: runner INFO     12356773: Run failed 1: Traceback (most recent call last):#012  File "/usr/libexec/pscheduler/commands/../classes/tool/traceroute/run", line 389, in <module>#012    ases = pscheduler.as_bulk_resolve(ips, threads=len(ips))#012  File "/usr/lib/python2.7/site-packages/pscheduler/psas.py", line 76, in as_bulk_resolve#012    processes=min(len(candidates), threads))#012  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 714, in __init__#012    Pool.__init__(self, processes, initializer, initargs)#012  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 152, in __init__#012    raise ValueError("Number of processes must be at least 1")#012ValueError: Number of processes must be at least 1

     And the content of the emails is below my signature line.  Since this error in the pscheduler log doesn't contain a task ID, or run ID, I can't see what two hosts this traceroute tried to run between, but most of them seem to be running fine.  The lines before and after this in the pscheduler log seem to be pretty normal.  

     What relevant info in this log entry (or the email) should be pointing me toward the cause?  What log files should I be looking into?  

Additional info:

pscheduler-scheduler, archiver, and ticket seem to be running and in a healthy state when this occurs.

When I hand-restart pscheduler-scheduler (and occasionally when I restart the machine), I get an error about not having permissions to the pscheduler-archiver.pid file.  That has been a problem that has come and gone ever since I upgraded this machine to CentOS7 6 or more months ago.  So it may or may not be related to this problem.

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047

reason:         pool.py:152:__init__:ValueError: Number of processes must be at least 1
cmdline:        /usr/bin/python /usr/libexec/pscheduler/commands/../classes/tool/traceroute/run
executable:     /usr/libexec/pscheduler/classes/tool/traceroute/run
package:        pscheduler-tool-traceroute-1.0.2.3-1.el7.centos
component:      pscheduler-tool-traceroute
pid:            18360
hostname:       ps-fhsu-bw.perfsonar.kanren.net
count:          190
abrt_version:   2.1.11
analyzer:       Python
architecture:   x86_64
duphash:        9909e45e4eb84b8b18b22553792d3e7cc5692413
event_log:
kernel:         3.10.0-693.17.1.el7.x86_64
last_occurrence: 1519219707
os_release:     CentOS Linux release 7.4.1708 (Core)
pkg_arch:       noarch
pkg_epoch:      0
pkg_fingerprint: 868D DF2C 9D7B 9686
pkg_name:       pscheduler-tool-traceroute
pkg_release:    1.el7.centos
pkg_vendor:     (none)
pkg_version:    1.0.2.3
runlevel:       N 3
time:           Tue 20 Feb 2018 02:25:22 PM CST
type:           Python
uid:            1000
username:       pscheduler
uuid:           9909e45e4eb84b8b18b22553792d3e7cc5692413

backtrace:
:pool.py:152:__init__:ValueError: Number of processes must be at least 1
:
:Traceback (most recent call last):
:  File "/usr/libexec/pscheduler/commands/../classes/tool/traceroute/run", line 389, in <module>
:    ases = pscheduler.as_bulk_resolve(ips, threads=len(ips))
:  File "/usr/lib/python2.7/site-packages/pscheduler/psas.py", line 76, in as_bulk_resolve
:    processes=min(len(candidates), threads))
:  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 714, in __init__
:    Pool.__init__(self, processes, initializer, initargs)
:  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 152, in __init__
:    raise ValueError("Number of processes must be at least 1")
:ValueError: Number of processes must be at least 1
:
:Local variables in innermost frame:
:self: <multiprocessing.pool.ThreadPool object at 0x23fb190>
:initargs: ()
:processes: 0
:maxtasksperchild: None
:initializer: None
dso_list:
:python-libs-2.7.5-58.el7.x86_64
:python-pscheduler-1.3.2.3-1.el7.centos.noarch
environ:
:LANG=en_US.UTF-8
:SHELL=/bin/bash
:SHLVL=0
:PSCHEDULER_LOG_STATE=(dp0
:S'is_quiet'
:p1
:I00
:sS'facility'
:p2
:I20
:sS'last_level'
:p3
:I20
:sS'forced_debug'
:p4
:I00
:s.
:PWD=/
:LOGNAME=pscheduler
:USER=pscheduler
:HOME=/home/pscheduler
:PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
:PSCHEDULER_SAFERUN_STATE=(dp0
:S'current_backoff'
:p1
:F0.5
:sS'runs'
:p2
:I0
:sS'initial_backoff'
:p3
:F0.25
:s.
machineid:
:systemd=27794959ad5b4cfaafba9ccc90e5d6e6
:sosreport_uploader-dmidecode=ec83b527193d3178118634d43166162a6cec86f0004e6c992d3e458d1549ea79
os_info:
:NAME="CentOS Linux"
:VERSION="7 (Core)"
:ID="centos"
:ID_LIKE="rhel fedora"
:VERSION_ID="7"
:PRETTY_NAME="CentOS Linux 7 (Core)"
:ANSI_COLOR="0;31"
:CPE_NAME="cpe:/o:centos:centos:7"
:HOME_URL="https://www.centos.org/"
:BUG_REPORT_URL="https://bugs.centos.org/"
:
:CENTOS_MANTISBT_PROJECT="CentOS-7"
:CENTOS_MANTISBT_PROJECT_VERSION="7"
:REDHAT_SUPPORT_PRODUCT="centos"
:REDHAT_SUPPORT_PRODUCT_VERSION="7"
: