perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Phil Reese writes:
    
    However, since Thursday. I've had quite a number of failed Throughput 
    tests happen between my hosts.  Till the upgrade, the every three hour 
    tests, would come back green (>=.9Gbs) with an occasional yellow 
    (<.9Gbps but more than .5Gbps).  Now it seems that during business hours 
    I'll have as many as 70 out of 484 tests fully fail, these mostly clear 
    over night, but come back the next day but not necessarily the same host 
    to host failures.

As Szymon mentioned, we saw some of that in Europe over the weekend, so 
you're not the only one.  Most of what I saw on those systems was the far end 
running past its scheduled time and getting cut off at the knees, which 
scuppers the whole measurement even if the near end collects a result.  Also 
worth mentioning is that a fraction of trace tests (and possibly others) are 
failing for the same reason, but those probably run often enough that the 
dashboards may cover up the failures.  The trace failures are actually a good 
thing because I can tinker with that in a single-system setting where there 
are fewer variables.

I'm not inclined to think it's something pScheduler code because almost 
nothing involved has changed.  Iperf3 hasn't seen a release since June, its 
pScheduler plugin hasn't been modified since September (pre-4.2.2), the 
runner service that oversees all of this hasn't changed in a year and the 
code in our internal library is even older than that.  That said, something 
has to be different to cause this behavior.
 
I've been working on this over the last couple of days.  Once I get to the 
bottom of it, there will be a patch.

--Mark