perfSONAR User Q&A and Other Discussion

[Dan Pritts <>]

Mark Feit wrote on 4/3/19 7:56 PM:

 

My guess on that would be that pScheduler on the AWS end did start iperf3 in server mode but it ran into a problem and failed.  There would be diagnostic information available in pScheduler to check on that.  When you run into problems, it’s a good idea to run “pscheduler result --diags RUN-URL” against the failed run to see what happened.  A “pscheduler troubleshoot” between the hosts would be good, too.  Sometimes it’s something as simple as the clocks not being in sync and the times where the client and server start don’t match up.

As you may recall from my previous note, iperf3 works great when I run it by hand.  And pscheduler will successfully run the test between the same two endpoints when I use IPv6.

pscheduler troubleshoot reports no issues.

pscheduler result returns:

[root@maccutil ~]# pscheduler result --diags https://localhost/pscheduler/tasks/eb45a8d5-953d-456e-8981-009eacf813bb/runs/4b1f75db-74f4-4b29-977e-e87e82c63ee3
2019-04-03T21:37:56-04:00 on localhost and perf0.us-east-1.icpsr.umich.edu with iperf3:
throughput --bandwidth 1000000 --ip-version 4 --dest perf0.us-east-1.icpsr.umich.edu --udp

Run did not complete: Failed

Diagnostics from localhost:
  No diagnostics.

Error from localhost:
  iperf3 returned an error: error - unable to connect to server: Connection refused

Diagnostics from perf0.us-east-1.icpsr.umich.edu:
  No diagnostics.

Error from perf0.us-east-1.icpsr.umich.edu:
  iperf3 returned an error: exiting

My eyes skimmed past the "iperf3 returned an error: exiting" it when I looked earlier, it's pretty vague. 

But you're right, it does run iperf3, which immediately exits.  I confirmed it with

perf record -e sched:sched_process_exec -a

...which unfortunately didn't record the full options.  Probably you can get perf to give you that, but I didn't dig further on that front.

I am unable to find anything in the logs that stated exactly what commands were being run.  The closest I get is

perf0 journal: runner INFO     55: With iperf3: throughput --bandwidth 1000000 --ip-version 6 --dest perf0.us-east-1.icpsr.umich.edu --udp
perf0 journal: runner INFO     56: With iperf3: throughput --bandwidth 1000000 --ip-version 4 --dest perf0.us-east-1.icpsr.umich.edu --udp

The iperf3 plugin doesn’t do anything special as far as binding to a particular address by default, so NAT (real 1:1 NAT, not cable modem-style PAT) shouldn’t have any effect on the server side.

ps told me that the iperf3 command that runs (successfully) for an ipv6 test was

/usr/bin/iperf3 -s -1 --json -p 5201 -B perf0.us-east-1.icpsr.umich.edu -6

Running the same command with a -4 suggests that it actually is attempting to bind to a particular address.  And it sends multiple "error"s in the json, and I bet pscheduler was only grabbing the last one.  If I'd seen "cannot assign requested address", well, I'd have KNOWN that it was a NAT problem. :)

# /usr/bin/iperf3 -s -1 --json -p 5201 -B perf0.us-east-1.icpsr.umich.edu -4
{
    "start":    {
        "connected":    [],
        "version":    "iperf 3.6",
        "system_info":    "Linux perf0.us-east-1.icpsr.umich.edu 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64"
    },
    "intervals":    [],
    "end":    {
    },
    "error":    "error - unable to start listener for connections: Cannot assign requested address",
    "error":    "exiting"
}

(Forward) DNS works for both names. 
 
I'm sure there was a good reason to bind to an address.  Multi-homed hosts or something like that.  Sigh.

This all points to a tool-by-tool analysis, like you say.  NBD, just a week or two project for an experienced engineer who's already got a million things on their plate. 

thanks
danno