perfSONAR User Q&A and Other Discussion

[Jonathon A Anderson <>]

Here’s some more logs, in case it’s useful. It looks to me like something is 
wrong (e.g., timeouts) but I don’t know enough about PS to know what, or how 
to fix it.

~jonathon


[rcops@sn-comp-perf40g
 ~]$ tail -c0 -F /var/log/perfsonar/*log
[...]

==> /var/log/perfsonar/owamp_bwctl.log <==
Jan 15 13:54:25 sn-comp-perf40g owampd[25651]: FILE=sapi.c, LINE=303, 
Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
([sn-hpcf-perf40g.colorado.edu]:44396)
Jan 15 13:54:25 sn-comp-perf40g owampd[25651]: FILE=sapi.c, LINE=466, 
ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
userid(nil):([sn-hpcf-perf40g.colorado.edu]:44396)
Jan 15 13:54:29 sn-comp-perf40g owampd[25653]: FILE=sapi.c, LINE=303, 
Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
([sn-hpcf-perf40g.colorado.edu]:44402)
Jan 15 13:54:29 sn-comp-perf40g owampd[25653]: FILE=sapi.c, LINE=466, 
ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
userid(nil):([sn-hpcf-perf40g.colorado.edu]:44402)
Jan 15 13:54:29 sn-comp-perf40g owampd[25655]: FILE=sapi.c, LINE=303, 
Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
([sn-hpcf-perf40g.colorado.edu]:44403)
Jan 15 13:54:29 sn-comp-perf40g owampd[25655]: FILE=sapi.c, LINE=466, 
ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
userid(nil):([sn-hpcf-perf40g.colorado.edu]:44403)
Jan 15 13:54:31 sn-comp-perf40g bwctld[25646]: FILE=endpoint.c, LINE=1310, 
PeerAgent: Peer cancelled test before expected

==> /var/log/perfsonar/traceroute_ondemand_mp.log <==
2016/01/15 13:54:36 (7542) DEBUG> daemon.pl:569 main::psService - Accept 
returned nothing, likely a timeout occurred or a child exited

==> /var/log/perfsonar/traceroute_scheduler.netlogger.log <==
ts=2016-01-15T20:54:50.166811Z 
event=org.perfSONAR.Services.MP.TracerouteScheduler.run.start 
guid=420f0bb3-362d-4624-8d85-0efec6a6708b
ts=2016-01-15T20:54:50.167085Z 
event=org.perfSONAR.Services.MP.TracerouteScheduler.run.end 
guid=420f0bb3-362d-4624-8d85-0efec6a6708b

==> /var/log/perfsonar/traceroute_ondemand_mp.log <==
2016/01/15 13:54:56 (7542) DEBUG> daemon.pl:569 main::psService - Accept 
returned nothing, likely a timeout occurred or a child exited
2016/01/15 13:55:16 (7542) DEBUG> daemon.pl:569 main::psService - Accept 
returned nothing, likely a timeout occurred or a child exited

==> /var/log/perfsonar/traceroute_scheduler.netlogger.log <==
ts=2016-01-15T20:55:20.167341Z 
event=org.perfSONAR.Services.MP.TracerouteScheduler.run.start 
guid=420f0bb3-362d-4624-8d85-0efec6a6708b
ts=2016-01-15T20:55:20.167599Z 
event=org.perfSONAR.Services.MP.TracerouteScheduler.run.end 
guid=420f0bb3-362d-4624-8d85-0efec6a6708b

==> /var/log/perfsonar/owamp_bwctl.log <==
Jan 15 13:55:26 sn-comp-perf40g owampd[25659]: FILE=sapi.c, LINE=303, 
Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
([sn-hpcf-perf40g.colorado.edu]:44427)
Jan 15 13:55:26 sn-comp-perf40g owampd[25659]: FILE=sapi.c, LINE=466, 
ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
userid(nil):([sn-hpcf-perf40g.colorado.edu]:44427)
Jan 15 13:55:29 sn-comp-perf40g owampd[25661]: FILE=sapi.c, LINE=303, 
Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
([sn-hpcf-perf40g.colorado.edu]:44431)
Jan 15 13:55:29 sn-comp-perf40g owampd[25661]: FILE=sapi.c, LINE=466, 
ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
userid(nil):([sn-hpcf-perf40g.colorado.edu]:44431)
Jan 15 13:55:29 sn-comp-perf40g owampd[25663]: FILE=sapi.c, LINE=303, 
Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
([sn-hpcf-perf40g.colorado.edu]:44432)
Jan 15 13:55:29 sn-comp-perf40g owampd[25663]: FILE=sapi.c, LINE=466, 
ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
userid(nil):([sn-hpcf-perf40g.colorado.edu]:44432)

==> /var/log/perfsonar/traceroute_ondemand_mp.log <==
2016/01/15 13:55:36 (7542) DEBUG> daemon.pl:569 main::psService - Accept 
returned nothing, likely a timeout occurred or a child exited
^C



> On Jan 15, 2016, at 1:10 PM, Jonathon A Anderson 
> <>
>  wrote:
> 
> Ignore that. I forgot I wasn’t logged in as root.
> 
> regular_testing restarted this time. I don’t see any indication of any 
> change (and, again, I tried rebooting the box before to no avail, which 
> would have restarted the service before ;)
> 
> ~jonathon
> 
> 
> [rcops@sn-comp-perf40g
>  ~]$ sudo service regular_testing restart
> [sudo] password for rcops: 
> /etc/init.d/regular_testing stop: perfSONAR Regular Testing stopped
> waiting...
> /opt/perfsonar_ps/regular_testing/bin/daemon 
> --config=/opt/perfsonar_ps/regular_testing/etc/regular_testing.conf 
> --pidfile=regular_testing.pid --piddir=/var/run 
> --logger=/opt/perfsonar_ps/regular_testing/etc/regular_testing-logger.conf 
> --user=perfsonar --group=perfsonar --daemonize
> /etc/init.d/regular_testing start: perfSONAR Regular Testing started
> 
> 
>> On Jan 15, 2016, at 1:08 PM, Jonathon A Anderson 
>> <>
>>  wrote:
>> 
>> Looks like regular_testing (and likely other services?) can’t start 
>> because it can’t create a pidfile in /var/run.
>> 
>> selinux is disabled, permissions look fine (the same as on a working 
>> perfsonar box, anyway) and the disk isn’t full.
>> 
>> Any thoughts on why pidfile write might be failing?
>> 
>> ~jonathon
>> 
>> 
>> [rcops@sn-comp-perf40g
>>  ~]$ tail -n0 -F /var/log/perfsonar/regular_testing.log &
>> [1] 24836
>> 
>> 
>> [rcops@sn-comp-perf40g
>>  ~]$ service regular_testing restart
>> 
>> 
>> /etc/init.d/regular_testing stop: perfSONAR Regular Testing (pid 12925?) 
>> not running
>> waiting...
>> /opt/perfsonar_ps/regular_testing/bin/daemon 
>> --config=/opt/perfsonar_ps/regular_testing/etc/regular_testing.conf 
>> --pidfile=regular_testing.pid --piddir=/var/run 
>> --logger=/opt/perfsonar_ps/regular_testing/etc/regular_testing-logger.conf 
>> --user=perfsonar --group=perfsonar --daemonize
>> Can't write pidfile: /var/run/regular_testing.pid
>> at /opt/perfsonar_ps/regular_testing/bin/daemon line 220
>>      main::lockPIDFile('/var/run', 'regular_testing.pid') called at 
>> /opt/perfsonar_ps/regular_testing/bin/daemon line 46
>> /etc/init.d/regular_testing start: perfSONAR Regular Testing could not be 
>> started
>> 
>> 
>> [rcops@sn-comp-perf40g
>>  ~]$ getenforce
>> Disabled
>> 
>> 
>> [rcops@sn-comp-perf40g
>>  ~]$ stat /var/run
>> File: `/var/run'
>> Size: 4096           Blocks: 8          IO Block: 4096   directory
>> Device: fd00h/64768d Inode: 1704766     Links: 32
>> Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
>> Access: 2016-01-14 20:08:43.121063972 -0700
>> Modify: 2016-01-15 03:52:23.869842497 -0700
>> Change: 2016-01-15 03:52:23.869842497 -0700
>> 
>> 
>> [rcops@sn-comp-perf40g
>>  ~]$ df -h
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/mapper/vg_sncompperf40g-lv_root
>>                      50G  3.8G   43G   8% /
>> tmpfs                 1.9G     0  1.9G   0% /dev/shm
>> /dev/sda1             477M  161M  291M  36% /boot
>> /dev/mapper/vg_sncompperf40g-lv_home
>>                     864G   73M  820G   1% /home
>> 
>> 
>> 
>>> On Jan 15, 2016, at 1:02 PM, Brian Tierney 
>>> <>
>>>  wrote:
>>> 
>>> 
>>> The next thing I'd try is to restart regular testing.
>>> 
>>> sudo /etc/init.d/regular_testing restart
>>> 
>>> and look at /var/log/perfsonar/regular_testing.log
>>> 
>>> On Fri, Jan 15, 2016 at 7:00 AM, Jonathon A Anderson 
>>> <>
>>>  wrote:
>>> Absolutely I am clicking “save." In fact, *before* I click “save,” the 
>>> change appears to be present in the interface; but when I click “save," 
>>> it reverts the view.
>>> 
>>> on one such host, clicking “save” causes the settings panel to go blank 
>>> and never recover without a “refresh.”
>>> 
>>> ~jonathon
>>> 
>>> 
>>>> On Jan 15, 2016, at 6:45 AM, Daniel Doyle 
>>>> <>
>>>>  wrote:
>>>> 
>>>> When you make changes, are you sure you are hitting "Save" after 
>>>> clicking the "Set" button on edit test parameters?
>>>> 
>>>> -Dan
>>>> 
>>>>> On Jan 14, 2016, at 10:37 PM, Jonathon A Anderson 
>>>>> <>
>>>>>  wrote:
>>>>> 
>>>>> I’m trying to change the tests through the perfsonar gui (e.g., change 
>>>>> the test description, or add a host) but the changes don’t persist. I’m 
>>>>> tailing /var/log/perfsonar/config_daemon.log looking for an error, but 
>>>>> there’s no output. Where else should I look?
>>>>> 
>>>>> ~jonathon
>>>>> 
>>>> 
>>>> Dan Doyle
>>>> GlobalNOC Software Developer
>>>> 1-812-856-3892
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Brian Tierney, http://www.es.net/tierney
>>> Energy Sciences Network (ESnet), Berkeley National Lab
>>> http://fasterdata.es.net
>>> 
>> 
>