perfSONAR User Q&A and Other Discussion

[Brian Tierney <>]

Winnie:

It sounds like that host has a strange mix of perfSONAR 3.3 and perfSONAR 3.4 software on it.

Any chance you could wipe the disk and start over with a fresh 3.4 install?

On Tue, Feb 3, 2015 at 7:42 AM, Winnie Lacesso <> wrote:

Good afternoon,

Now having started the PingER service, its logs seem full of error:
root@lcgnetmon> tail /var/log/perfsonar/pinger.log
2015/02/03 15:03:24 (30993) ERROR> Remote.pm:373 perfSONAR_PS::Client::LS::Remote::getKey - Key not found.
2015/02/03 15:03:24 (30993) ERROR> Remote.pm:692 perfSONAR_PS::Client::LS::Remote::sendKeepalive - Key value not supplied.
2015/02/03 15:03:56 (30992) INFO> PingER.pm:287 perfSONAR_PS::Services::MP::PingER::registerLS - Registering PingER MP with LS
2015/02/03 15:03:56 (30992) INFO> PingER.pm:287 perfSONAR_PS::Services::MP::PingER::registerLS - Registering PingER MP with LS
2015/02/03 15:03:56 (30992) ERROR> Remote.pm:373 perfSONAR_PS::Client::LS::Remote::getKey - Key not found.
2015/02/03 15:03:56 (30992) ERROR> Remote.pm:692 perfSONAR_PS::Client::LS::Remote::sendKeepalive - Key value not supplied.
2015/02/03 15:04:24 (30993) INFO> PingER.pm:300 perfSONAR_PS::Services::MA::PingER::registerLS - Registering PingER MA with LS
2015/02/03 15:04:24 (30993) INFO> PingER.pm:300 perfSONAR_PS::Services::MA::PingER::registerLS - Registering PingER MA with LS
2015/02/03 15:04:25 (30993) ERROR> Remote.pm:373 perfSONAR_PS::Client::LS::Remote::getKey - Key not found.
2015/02/03 15:04:25 (30993) ERROR> Remote.pm:692 perfSONAR_PS::Client::LS::Remote::sendKeepalive - Key value not supplied.

What is this Key it's wanting, whence should it be got?

Also yesterday these services were started in the order that rc3.d would
start them (in a little "for" loop)
traceroute_ma
traceroute_master
traceroute_ondemand_mp
traceroute_scheduler

No obvious errors at that time: (lost output for traceroute_ma startup)
on traceroute_master
chkconfig traceroute_master on
service traceroute_master start
/opt/perfsonar_ps/traceroute_ma/bin/traceroute_master.pl --config=/opt/perfsonar_ps/traceroute_ma/etc/traceroute-master.conf --pidfile=/var/run/traceroute-master.pid --logger=/opt/perfsonar_ps/traceroute_ma/etc/traceroute-master_logger.conf --user=perfsonar --group=perfsonar
/etc/init.d/traceroute_master start: Traceroute Master started

on traceroute_ondemand_mp
chkconfig traceroute_ondemand_mp on
service traceroute_ondemand_mp start
/opt/perfsonar_ps/traceroute_ma/bin/daemon.pl --config=/opt/perfsonar_ps/traceroute_ma/etc/ondemand_mp-daemon.conf --pidfile=traceroute_ondemand_mp.pid --piddir=/var/run --logger=/opt/perfsonar_ps/traceroute_ma/etc/ondemand_mp-daemon_logger.conf --user=perfsonar --group=perfsonar
/etc/init.d/traceroute_ondemand_mp start: perfSONAR Traceroute On-Demand MP Service started

on traceroute_scheduler
chkconfig traceroute_scheduler on
service traceroute_scheduler start
/opt/perfsonar_ps/traceroute_ma/bin/traceroute_scheduler.pl --owmesh=/opt/perfsonar_ps/traceroute_ma/etc --pidfile=/var/run/traceroute-scheduler.pid --logger=/opt/perfsonar_ps/traceroute_ma/etc/traceroute-scheduler_logger.conf --user=perfsonar --group=perfsonar
/etc/init.d/traceroute_scheduler start: Traceroute Scheduler started

but the logs are full of errors:

root@lcgnetmon> tail -5 /var/log/perfsonar/traceroute_ondemand_mp.log
(timestamped today)
2015/02/03 15:03:15 (11289) DEBUG> daemon.pl:569 main::psService - Accept returned nothing, likely a timeout occurred or a child exited
2015/02/03 15:03:35 (11289) DEBUG> daemon.pl:569 main::psService - Accept returned nothing, likely a timeout occurred or a child exited
2015/02/03 15:03:55 (11289) DEBUG> daemon.pl:569 main::psService - Accept returned nothing, likely a timeout occurred or a child exited
2015/02/03 15:04:15 (11289) DEBUG> daemon.pl:569 main::psService - Accept returned nothing, likely a timeout occurred or a child exited
2015/02/03 15:04:35 (11289) DEBUG> daemon.pl:569 main::psService - Accept returned nothing, likely a timeout occurred or a child exited

But, our working Bandwidth box is logging exactly the same so all that noise
must be "normal noise"?! (SIGH, wish for _real_ info/data in logs?!)

Also from yesterday startup, errors in logs:

in traceroute_ma.log

2015/02/02 15:32:43 (11237) WARN> daemon.pl:272 main:: - Enabling echo service for each endpoint unless specified otherwise
2015/02/02 15:32:43 (11237) INFO> Traceroute.pm:111 perfSONAR_PS::Services::MA::Traceroute::init - Setting service access point to http://lcgnetmon.phy.bris.ac.uk:8086/perfSONAR_PS/services/tracerouteMA
2015/02/02 15:32:43 (11237) WARN> Traceroute.pm:126 perfSONAR_PS::Services::MA::Traceroute::init - Setting 'service_description' to 'perfSONAR_PS Traceroute MA at UKI-SOUTHGRID-BRIS-HEP'.
2015/02/02 15:32:43 (11237) WARN> Traceroute.pm:133 perfSONAR_PS::Services::MA::Traceroute::init - Setting 'service_name' to 'Traceroute MA'.
2015/02/02 15:32:43 (11237) WARN> Traceroute.pm:140 perfSONAR_PS::Services::MA::Traceroute::init - Setting 'service_type' to 'MA'.
2015/02/02 15:32:43 (11258) ERROR> SQL.pm:166 perfSONAR_PS::DB::SQL::openDB - Open error "DBI connect('traceroute_ma:localhost','perfsonar',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /opt/perfsonar_ps/traceroute_ma/bin/../lib/perfSONAR_PS/DB/SQL.pm line 163
".
2015/02/02 15:32:43 (11258) ERROR> SQL.pm:218 perfSONAR_PS::DB::SQL::query - Query error "Can't call method "prepare" on an undefined value at /opt/perfsonar_ps/traceroute_ma/bin/../lib/perfSONAR_PS/DB/SQL.pm line 212.
" on statement "SELECT year, month, day FROM DATES".
2015/02/02 15:32:43 (11258) ERROR> daemon.pl:670 main::registerLS - Problem running register LS: Can't use string ("-1") as an ARRAY ref while "strict refs" in use at /opt/perfsonar_ps/traceroute_ma/bin/../lib/perfSONAR_PS/Services/MA/Traceroute.pm line 757.

(note that mysql started right AFTER that, according to the order in rc3.d)

Again, our working Bandwidth box is logging exactly the same so all that noise
must be "normal errors"?! (SIGH, wish for _real_ info/data in logs?!)

in traceroute_scheduler.log:

2015/01/26 14:12:04 (1820) ERROR> TracerouteScheduler.pm:35 perfSONAR_PS::Services::MP::TracerouteScheduler::new - Error loading test config: Conf::must_get_val(ATTR=>NODE, ) undefined, called from /opt/perfsonar_ps/traceroute_ma/bin/../lib/perfSONAR_PS/Services/MP/TracerouteScheduler.pm:52
2015/01/26 14:52:03 (2006) ERROR> TracerouteScheduler.pm:35 perfSONAR_PS::Services::MP::TracerouteScheduler::new - Error loading test config: Conf::must_get_val(ATTR=>NODE, ) undefined, called from /opt/perfsonar_ps/traceroute_ma/bin/../lib/perfSONAR_PS/Services/MP/TracerouteScheduler.pm:52
2015/02/02 15:32:47 (11306) ERROR> TracerouteScheduler.pm:35 perfSONAR_PS::Services::MP::TracerouteScheduler::new - Error loading test config: Conf::must_get_val(ATTR=>NODE, ) undefined, called from /opt/perfsonar_ps/traceroute_ma/bin/../lib/perfSONAR_PS/Services/MP/TracerouteScheduler.pm:52

Again, our working Bandwidth box is logging exactly the same so all that noise
must be "normal errors"?! (SIGH, wish for _real_ info/data in logs?!)

Next to start is regular_testing & hope it doesn't implode again....

--

Brian Tierney, http://www.es.net/tierney
Energy Sciences Network (ESnet), Berkeley National Lab
http://fasterdata.es.net