perfSONAR User Q&A and Other Discussion

["Fligor, Debbie" <>]

Hi everyone,

I installed 2 new CentOs7 perfsonar systems with 100G cards last December, 
and finally got everything to align to finish getting them up and running 
last week.  I ran into a few problems. I solved one, and brute-forced the 
others, but would like help fixing them in a hopefully more sustainable way.  

The first one I hit was that after I configured some tests, nothing was 
showing up in test results. They were all throughput tests, so when I 
finished setting it up on Friday, I didn’t think much of it, expecting to 
find results yesterday when I got to work.  Still nothing at all.  I read 
through other posts on the mailing lists and started checking things.  the 
logs said pscheduler was running and seeing no errors, but esmond had nothing 
at all in the log - as if it was never being called. Clicking on the archive 
link under esmond and showed no data at all.

First thing I figured out was that even though the button for auto update was 
set, it hadn’t done any updates on either system. So I did by-hand yum update 
and it got everything up to the current versions as best I can tell (psconfig 
is 4.1.6-1el7).  So that’s my first ask - what else do I need to do for auto 
update to work?


Then I added some ping tests so that I should start seeing results sooner. I 
started getting errors when I changed tests, and would hit cancel, and try 
again, and fairly soon on that host clicking on the tests tab got me a 
spinning load circle that never resolved.  So I thought I’d corrupted a 
database or something on one of the two servers.  Still no test results on 
either.  I couldn’t find any errors in any of the logs you can see on the web 
page, so I tried some things that had helped others with similar errors who 
had posted their fixes, like cassandara cleanup, and nothing helped. plent of 
reboots as well. Convinced I had corrupted something I tried "yum erase 
perfsonar-toolkit” then “yum clean all” then “yum install perfsonar-toolkit”. 
 That didn’t fix my test configs not saving.  I got to the FAQ page and 
worked my way through all the log files in the list, and found this error 
repeating in /var/log/httpd/ssl_error_log:


[Mon Mar 04 21:58:42.441962 2019] [cgi:error] [pid 53339] [client 
107.152.10.165:19568] AH01215: [Mon Mar  4 21:58:42 2019] 
regular_testing.cgi: Config::General: Cannot create hashref from <target> 
because there is, referer: 
https://res-dtn.techservices.illinois.edu/toolkit/auth/admin/tests.cgi
[Mon Mar 04 21:58:42.442069 2019] [cgi:error] [pid 53339] [client 
107.152.10.165:19568] AH01215: [Mon Mar  4 21:58:42 2019] 
regular_testing.cgi: already a scalar option 'target' with value 
'psb-bw-100g.ncsa.illinois.edu', referer: 
https://res-dtn.techservices.illinois.edu/toolkit/auth/admin/tests.cgi
[Mon Mar 04 21:58:42.442156 2019] [cgi:error] [pid 53339] [client 
107.152.10.165:19568] AH01215: [Mon Mar  4 21:58:42 2019] 
regular_testing.cgi:  at 
/usr/lib/perfsonar/web-ng/root/admin/services/regular_testing.cgi line 38., 
referer: 
https://res-dtn.techservices.illinois.edu/toolkit/auth/admin/tests.cgi

Looking at the tests.cgi file I figured out it was unhappy with the 
/var/lib/perfsonar/toolkit/gui-tasks.conf  so after making a backup copy, I 
cleared the contents of the file (FWIW just removing the file gui-tasks.conf 
is bad. it doesn’t re-make it when you go into the tests tab, it just throws 
errors in the ssl_error_log about it not being there.) Success! now I could 
load the (blank) tests screen.  I could add a few tests, but then it started 
giving me problems saving again (but still loads). It would complain when it 
wanted to save the page of tests with Internal Server Error.  my brute force 
solution for this is if I hand-edit gui-tasks.conf and remove all the 
“target” lines where “target" isn’t a tag, it will then save.  Here’s an 
example test it’s putting in the file:

<test>
    local_interface   enp101s0
    <parameters>
        tool   iperf3,iperf
        test_ipv4_ipv6   1
        type   bwctl
        duration   20
        packet_tos_bits   0
        window_size   0
    </parameters>
    target   res-dtn.techservices.illinois.edu
    description   res-dtn
    <schedule>
        type   regular_intervals
        interval   7200
    </schedule>
</test>



and it doesn’t save until I remove the line:
    target   res-dtn.techservices.illinois.edu
(and all similar “target” lines)  It’s fine when “target" is in <> like this:

   <target>
        address   rvperf.routeviews.org
        <override_parameters>
            type   bwping
            force_ipv4   1
        </override_parameters>
        description   RouteViews, Eugene, Oregon, US
    </target>


I would like to be able to save tests from the web interface without having 
to edit gui-tasks.conf (not to mention I don’t actually know what I’m doing 
when I edit it).  What do I need to do?  There is less than a day’s data, so 
blowing everything away and starting again is fine, if that fixes this, but I 
ran out of steam for finding ideas on how to do that.  


And once I had that solved, the host that I had deleted and re-installed 
perfsonar-toolkit on started showing test results (on the dashboard and the 
esmond archive), but the other host did not. Rebooting didn’t make the tests 
show up.  So on that host I also uninstalled and reinstalled 
perfsonar-toolkit and now the tests results show up.  On this one I’m curios 
as to why this worked, but since it’s working and continued to update all 
night, I’m not looking for a better way to do it.  Now I’m moving on to 
system tuning.

One last minor question for anyone that got this far. How do I set the 
primary (default) interface so it’s what I want? one system is picking the 
100G, the other the 1 G copper port.  on both systems the 100G has the 
default route for v4 and v6.

-- 
-debbie
Debbie Fligor, n9dn       Lead Network Engineer @ Univ. of Il
email: