perfSONAR User Q&A and Other Discussion

[Winnie Lacesso <>]

Greetings Perfsonar Gurus!

A few q. 

I believe it is true that most sites (probably) have 2 boxes, 1 for Lat & 
1 for BW, but some sites use one for both. Is that correct?

Looking at the serviceTest/psGraph.cgi for some boxes, eg our BW box, some 
of the entries clearly say lat/latency eg
        ps-latency.scinet.utoronto.ca (142.150.19.61) 
        perfsonar-ps-latency.pic.es (193.109.172.189) 

Click on either our BW box on the line paired with the remote (lat) box, 
or the remote (lat) box => a window pops up naming both our BW box & the 
remote (presumably) lat box, but no data. 
Now on the same page are either clearly the same site's BW box
        perfsonar-ps-latency.pic.es (193.109.172.189) 
or what might be (a guess) the site's BW box (guess: "psb" = BW)
        psb01.pic.es (193.109.172.187) 

So am a bit curious why there might be BW tests from our BW box to a 
site's *pair* of perfsonar boxes, that is to both their BW & their Lat 
box.... like is that normal or a config error? (especially since there is 
no data in the graphs between our BW box & their Lat box).

Next, something happened early hours of 27 April. To quite a number of 
sites, our BW box appears to have been very fine till early hours of 27 
April. Then abruptly nothing.

A .7 bet of the cause is there was an overnight yum auto-update
Apr 27 04:06:46 Updated: perl-perfSONAR_PS-RegularTesting-3.4.2-4.pSPS.noarch

(Note: our Lat box updated at same time & seems fine)

It was noticed IIRC early May that very few graphs showed up which click 
on other hosts on BW's graphs page:

http://lcgnetmon02.phy.bris.ac.uk/serviceTest/psGraph.cgi

All the "we are talking to these hosts" show, but for most, click on it -> 
it cycles a bit -> then box shows with no data (or red dots = "Errors"). 

Our BW box was rebooted 6 May in hopes that a fresh restart of everything 
=>> bw box graphs (with data) again. But no. 

If eg choose t2ps-bandwidth.physics.ox.ac.uk (not too far from Bristol)  
or perfsonar-bw.tier2.hep.manchester.ac.uk, there is a graph, but only 
red-dot Error; hit "1m" => pretty clear it was ok till 27 April & stopped 
abruptly in the early hours, to both ox & mancs.

The same is true if one goes to that remote box eg
http://t2ps-bandwidth.physics.ox.ac.uk/serviceTest/psGraph.cgi
(I can't ATM get to mancs) & clicks on lcgnetmon02.phy.bris.ac.uk - all 
fine till wee hours of 27 April.

But on other boxen, eg hepsonar1.ph.liv.ac.uk no graph shows at all &  
clicking "Previous 1w" does nothing (eg the dates don't change to show the  
previous week). Same if choose eg "1m" - it not show anything.
(What does it mean, that the dates don't change - are those graphs 
"broken"?) 

To a *few* boxes out there, in one direction at least all seems well: 
psum02.aglt2.org, ps-bandwidth.scinet.utoronto.ca, perfsonar.na.infn.it 
But going to that remote box's page & click on lcgnetmon02.phy.bris = NO 
DATA 

So our Bw box IS ok, for a few hosts, in at least one direction 

So am unsure if the BW box lcgnetmon02 has so to speak "lost its (or  
some of its) data" for some connections, or just can't graph it or what.  
Or does this sound like a firewall issue? (I can't see how, but.....)

I took a look in the regular_testing.log coverine 27 April in the early 
hours but nothing stands out as "obviously, it went pear-shaped here"; 
it's full of WARN & ERROR both before 27 April & after.  
The owamp_bwctl.log covering the early hours of 27 April shows nothing 
different.

In /var/lib/perfsonar/regular_testing are LOTS of bwctl_* directories. 
Sorting by time shows lots timestamped 24 April** & then 28 April. No 27 
April. 
** & earlier back to Mar 29th, should delete them.... no, wait, shouldn't
the  software delete them itself?!

Can anyone advise how to debug what happened to our bw box's graphs/data  
please? Or, is this all Normal? 
(I'm not Looking for Work, our site's not been ticketed so, guess: it's 
not completely broken....)


Winnie Lacesso / Bristol University Particle Physics Computing Systems
HH Wills Physics Laboratory, Tyndall Avenue, Bristol, BS8 1TL, UK