perfSONAR User Q&A and Other Discussion

[Jonathon A Anderson <>]

I ended up getting this to work with a combination of

* run /opt/perfsonar_ps/mesh_config/bin/generate_configuration
* delete and re-create the test

Possibly a coincidence, but tests that continued to express bad behavior 
(requiring delete/re-create) seemed to all be tests that had a NaN value for 
"TOS Bits."

~jonathon


> On Jan 15, 2016, at 1:56 PM, Jonathon A Anderson 
> <>
>  wrote:
> 
> Here’s some more logs, in case it’s useful. It looks to me like something 
> is wrong (e.g., timeouts) but I don’t know enough about PS to know what, or 
> how to fix it.
> 
> ~jonathon
> 
> 
> [rcops@sn-comp-perf40g
>  ~]$ tail -c0 -F /var/log/perfsonar/*log
> [...]
> 
> ==> /var/log/perfsonar/owamp_bwctl.log <==
> Jan 15 13:54:25 sn-comp-perf40g owampd[25651]: FILE=sapi.c, LINE=303, 
> Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
> ([sn-hpcf-perf40g.colorado.edu]:44396)
> Jan 15 13:54:25 sn-comp-perf40g owampd[25651]: FILE=sapi.c, LINE=466, 
> ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
> userid(nil):([sn-hpcf-perf40g.colorado.edu]:44396)
> Jan 15 13:54:29 sn-comp-perf40g owampd[25653]: FILE=sapi.c, LINE=303, 
> Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
> ([sn-hpcf-perf40g.colorado.edu]:44402)
> Jan 15 13:54:29 sn-comp-perf40g owampd[25653]: FILE=sapi.c, LINE=466, 
> ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
> userid(nil):([sn-hpcf-perf40g.colorado.edu]:44402)
> Jan 15 13:54:29 sn-comp-perf40g owampd[25655]: FILE=sapi.c, LINE=303, 
> Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
> ([sn-hpcf-perf40g.colorado.edu]:44403)
> Jan 15 13:54:29 sn-comp-perf40g owampd[25655]: FILE=sapi.c, LINE=466, 
> ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
> userid(nil):([sn-hpcf-perf40g.colorado.edu]:44403)
> Jan 15 13:54:31 sn-comp-perf40g bwctld[25646]: FILE=endpoint.c, LINE=1310, 
> PeerAgent: Peer cancelled test before expected
> 
> ==> /var/log/perfsonar/traceroute_ondemand_mp.log <==
> 2016/01/15 13:54:36 (7542) DEBUG> daemon.pl:569 main::psService - Accept 
> returned nothing, likely a timeout occurred or a child exited
> 
> ==> /var/log/perfsonar/traceroute_scheduler.netlogger.log <==
> ts=2016-01-15T20:54:50.166811Z 
> event=org.perfSONAR.Services.MP.TracerouteScheduler.run.start 
> guid=420f0bb3-362d-4624-8d85-0efec6a6708b
> ts=2016-01-15T20:54:50.167085Z 
> event=org.perfSONAR.Services.MP.TracerouteScheduler.run.end 
> guid=420f0bb3-362d-4624-8d85-0efec6a6708b
> 
> ==> /var/log/perfsonar/traceroute_ondemand_mp.log <==
> 2016/01/15 13:54:56 (7542) DEBUG> daemon.pl:569 main::psService - Accept 
> returned nothing, likely a timeout occurred or a child exited
> 2016/01/15 13:55:16 (7542) DEBUG> daemon.pl:569 main::psService - Accept 
> returned nothing, likely a timeout occurred or a child exited
> 
> ==> /var/log/perfsonar/traceroute_scheduler.netlogger.log <==
> ts=2016-01-15T20:55:20.167341Z 
> event=org.perfSONAR.Services.MP.TracerouteScheduler.run.start 
> guid=420f0bb3-362d-4624-8d85-0efec6a6708b
> ts=2016-01-15T20:55:20.167599Z 
> event=org.perfSONAR.Services.MP.TracerouteScheduler.run.end 
> guid=420f0bb3-362d-4624-8d85-0efec6a6708b
> 
> ==> /var/log/perfsonar/owamp_bwctl.log <==
> Jan 15 13:55:26 sn-comp-perf40g owampd[25659]: FILE=sapi.c, LINE=303, 
> Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
> ([sn-hpcf-perf40g.colorado.edu]:44427)
> Jan 15 13:55:26 sn-comp-perf40g owampd[25659]: FILE=sapi.c, LINE=466, 
> ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
> userid(nil):([sn-hpcf-perf40g.colorado.edu]:44427)
> Jan 15 13:55:29 sn-comp-perf40g owampd[25661]: FILE=sapi.c, LINE=303, 
> Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
> ([sn-hpcf-perf40g.colorado.edu]:44431)
> Jan 15 13:55:29 sn-comp-perf40g owampd[25661]: FILE=sapi.c, LINE=466, 
> ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
> userid(nil):([sn-hpcf-perf40g.colorado.edu]:44431)
> Jan 15 13:55:29 sn-comp-perf40g owampd[25663]: FILE=sapi.c, LINE=303, 
> Connection to ([sn-comp-perf40g.colorado.edu]:861) from 
> ([sn-hpcf-perf40g.colorado.edu]:44432)
> Jan 15 13:55:29 sn-comp-perf40g owampd[25663]: FILE=sapi.c, LINE=466, 
> ControlSession([sn-comp-perf40g.colorado.edu]:861) accepted from 
> userid(nil):([sn-hpcf-perf40g.colorado.edu]:44432)
> 
> ==> /var/log/perfsonar/traceroute_ondemand_mp.log <==
> 2016/01/15 13:55:36 (7542) DEBUG> daemon.pl:569 main::psService - Accept 
> returned nothing, likely a timeout occurred or a child exited
> ^C
> 
> 
> 
>> On Jan 15, 2016, at 1:10 PM, Jonathon A Anderson 
>> <>
>>  wrote:
>> 
>> Ignore that. I forgot I wasn’t logged in as root.
>> 
>> regular_testing restarted this time. I don’t see any indication of any 
>> change (and, again, I tried rebooting the box before to no avail, which 
>> would have restarted the service before ;)
>> 
>> ~jonathon
>> 
>> 
>> [rcops@sn-comp-perf40g
>>  ~]$ sudo service regular_testing restart
>> [sudo] password for rcops: 
>> /etc/init.d/regular_testing stop: perfSONAR Regular Testing stopped
>> waiting...
>> /opt/perfsonar_ps/regular_testing/bin/daemon 
>> --config=/opt/perfsonar_ps/regular_testing/etc/regular_testing.conf 
>> --pidfile=regular_testing.pid --piddir=/var/run 
>> --logger=/opt/perfsonar_ps/regular_testing/etc/regular_testing-logger.conf 
>> --user=perfsonar --group=perfsonar --daemonize
>> /etc/init.d/regular_testing start: perfSONAR Regular Testing started
>> 
>> 
>>> On Jan 15, 2016, at 1:08 PM, Jonathon A Anderson 
>>> <>
>>>  wrote:
>>> 
>>> Looks like regular_testing (and likely other services?) can’t start 
>>> because it can’t create a pidfile in /var/run.
>>> 
>>> selinux is disabled, permissions look fine (the same as on a working 
>>> perfsonar box, anyway) and the disk isn’t full.
>>> 
>>> Any thoughts on why pidfile write might be failing?
>>> 
>>> ~jonathon
>>> 
>>> 
>>> [rcops@sn-comp-perf40g
>>>  ~]$ tail -n0 -F /var/log/perfsonar/regular_testing.log &
>>> [1] 24836
>>> 
>>> 
>>> [rcops@sn-comp-perf40g
>>>  ~]$ service regular_testing restart
>>> 
>>> 
>>> /etc/init.d/regular_testing stop: perfSONAR Regular Testing (pid 12925?) 
>>> not running
>>> waiting...
>>> /opt/perfsonar_ps/regular_testing/bin/daemon 
>>> --config=/opt/perfsonar_ps/regular_testing/etc/regular_testing.conf 
>>> --pidfile=regular_testing.pid --piddir=/var/run 
>>> --logger=/opt/perfsonar_ps/regular_testing/etc/regular_testing-logger.conf
>>>  --user=perfsonar --group=perfsonar --daemonize
>>> Can't write pidfile: /var/run/regular_testing.pid
>>> at /opt/perfsonar_ps/regular_testing/bin/daemon line 220
>>>     main::lockPIDFile('/var/run', 'regular_testing.pid') called at 
>>> /opt/perfsonar_ps/regular_testing/bin/daemon line 46
>>> /etc/init.d/regular_testing start: perfSONAR Regular Testing could not be 
>>> started
>>> 
>>> 
>>> [rcops@sn-comp-perf40g
>>>  ~]$ getenforce
>>> Disabled
>>> 
>>> 
>>> [rcops@sn-comp-perf40g
>>>  ~]$ stat /var/run
>>> File: `/var/run'
>>> Size: 4096          Blocks: 8          IO Block: 4096   directory
>>> Device: fd00h/64768d        Inode: 1704766     Links: 32
>>> Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
>>> Access: 2016-01-14 20:08:43.121063972 -0700
>>> Modify: 2016-01-15 03:52:23.869842497 -0700
>>> Change: 2016-01-15 03:52:23.869842497 -0700
>>> 
>>> 
>>> [rcops@sn-comp-perf40g
>>>  ~]$ df -h
>>> Filesystem            Size  Used Avail Use% Mounted on
>>> /dev/mapper/vg_sncompperf40g-lv_root
>>>                     50G  3.8G   43G   8% /
>>> tmpfs                 1.9G     0  1.9G   0% /dev/shm
>>> /dev/sda1             477M  161M  291M  36% /boot
>>> /dev/mapper/vg_sncompperf40g-lv_home
>>>                    864G   73M  820G   1% /home
>>> 
>>> 
>>> 
>>>> On Jan 15, 2016, at 1:02 PM, Brian Tierney 
>>>> <>
>>>>  wrote:
>>>> 
>>>> 
>>>> The next thing I'd try is to restart regular testing.
>>>> 
>>>> sudo /etc/init.d/regular_testing restart
>>>> 
>>>> and look at /var/log/perfsonar/regular_testing.log
>>>> 
>>>> On Fri, Jan 15, 2016 at 7:00 AM, Jonathon A Anderson 
>>>> <>
>>>>  wrote:
>>>> Absolutely I am clicking “save." In fact, *before* I click “save,” the 
>>>> change appears to be present in the interface; but when I click “save," 
>>>> it reverts the view.
>>>> 
>>>> on one such host, clicking “save” causes the settings panel to go blank 
>>>> and never recover without a “refresh.”
>>>> 
>>>> ~jonathon
>>>> 
>>>> 
>>>>> On Jan 15, 2016, at 6:45 AM, Daniel Doyle 
>>>>> <>
>>>>>  wrote:
>>>>> 
>>>>> When you make changes, are you sure you are hitting "Save" after 
>>>>> clicking the "Set" button on edit test parameters?
>>>>> 
>>>>> -Dan
>>>>> 
>>>>>> On Jan 14, 2016, at 10:37 PM, Jonathon A Anderson 
>>>>>> <>
>>>>>>  wrote:
>>>>>> 
>>>>>> I’m trying to change the tests through the perfsonar gui (e.g., change 
>>>>>> the test description, or add a host) but the changes don’t persist. 
>>>>>> I’m tailing /var/log/perfsonar/config_daemon.log looking for an error, 
>>>>>> but there’s no output. Where else should I look?
>>>>>> 
>>>>>> ~jonathon
>>>>>> 
>>>>> 
>>>>> Dan Doyle
>>>>> GlobalNOC Software Developer
>>>>> 1-812-856-3892
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Brian Tierney, http://www.es.net/tierney
>>>> Energy Sciences Network (ESnet), Berkeley National Lab
>>>> http://fasterdata.es.net
>>>> 
>>> 
>> 
>