perfSONAR User Q&A and Other Discussion

["Uhl, George D. (GSFC-423.0)[Arctic Slope Technical Services, Inc.]" <>]

Hello Ivan,

 

Resolving the FQDN of uhmanoa-tp.ps.uhnet.net on enpl-pt2-10g.eos.nasa.gov yields the following. 

# host uhmanoa-tp.ps.uhnet.net 

uhmanoa-tp.ps.uhnet.net has address 132.160.6.178

uhmanoa-tp.ps.uhnet.net has IPv6 address 2607:f278:4101:f2::2

 

There were five tests that began to fail after I assigned an IPv6 address to my test interface including the test to uhmanoa-tp.ps.uhnet.net.  All five of the FQDN’s were resolvable to an IPv4 and an IPV6 address by enpl-pt2-10g.eos.nasa.gov.

 

As of this morning the tests that were failing last week are now working.  This morning I unassigned the IPv6 address from my local host’s test interface and successfully ran throughput tests with FQDN’s that had been failing.  When I restored the IPv6 address to the interface and restarted the network service on my local host, enpl-pt2-10g.eos.nasa.gov, I reran the same throughput tests with FQDN’s again but this time with successful results!  The attached file shows successful tests using the same pscheduler command syntax I used in last week’s tests.

 

It's possible(probable) that I broke something last week when I assigned an IPv6 and its FQDN to my test interface although I don’t know what that might have been.  So please disregard my previous email and I’ll pretend I never brought it up.

 

Thanks,

George

 

From: "Garnizov, Ivan" <>
Date: Monday, December 9, 2019 at 4:05 AM
To: "George.D.Uhl" <>, "" <>
Subject: [EXTERNAL] AW: Dual stack throughput test failures

 

Hello George,

 

From what you have sent one can see:

 

Case 1:  “from uhmanoa-tp.ps.uhnet.net: Could not resolve host: enpl-pt2-10g.eos.nasa.gov”

Most likely your DNS resolution fails for this FQDN.

My take from here is that the message from pScheduler has ambiguity and it is not clear enough, which participant does the reporting and which has the failure.

 

Case 2: Success with IPv4 address in destination

Apparently the remote participant is live and running and pScheduler coordination / connectivity is present on the specified address.

 

Case 3: pscheduler is unable to use an IP address to bind to the lead interface…

Perhaps that can become a feature request…, but even if it did succeed and your DNS resolution of the destination still fails, you would get in the end the same failure as in case 1.

 

 

In order to be able to further assist you or open a case for a bug, please demonstrate, that you are able to resolve on the enpl-pt2-10g.eos.nasa.gov the hostname of uhmanoa-tp.ps.uhnet.net

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

Von: [mailto:] Im Auftrag von Uhl, George D. (GSFC-423.0)[Arctic Slope Technical Services, Inc.]
Gesendet: Freitag, 6. Dezember 2019 20:44
An:
Betreff: [perfsonar-user] FW: Dual stack throughput test failures

 

Forward to pS user list.

 

From: "George.D.Uhl" <>
Date: Friday, December 6, 2019 at 10:40 AM
To: "" <>
Subject: Dual stack throughput test failures

 

Hi,

 

I run a mesh with a dual stack local host that is failing to run ipv4 throughput tests to any non-agent dual stack remote host when the remote host shares a single hostname for IPv4 and IPv6.  I’ve attached some test cases showing conditions when tests work vs. when they fail.

 

Specifying ip-version as IPv4 in the pscheduler command line doesn’t seem to make a difference.  The local test node has an advanced configuration which includes multiple test interfaces, policy routing and a non-test interface that is used by the main routing table as the default path.  The issue appears that the non-agent remote host prefers using it’s IPv6 address when it translates the “--destination hostname” given to it by the local pscheduler during test setup/tool negotiation.  Thus the ip-version parameter is used to specify the protocol for testing only and isn’t used as part of the test setup.  Would it be possible to support a new feature that specidies the preferred protocol during test setup?

 

Thanks,

George

{\rtf1\ansi\ansicpg1252\cocoartf1561\cocoasubrtf600
{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fnil\fcharset0 Menlo-Regular;}
{\colortbl;\red255\green255\blue255;\red0\green0\blue0;\red0\green0\blue0;\red255\green255\blue255;
}
{\*\expandedcolortbl;;\cssrgb\c0\c0\c0;\csgray\c0;\csgray\c100000;
}
\margl1440\margr1440\vieww27200\viewh13480\viewkind0
\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0

\f0\b\fs24 \cf0 SCENARIO - two dual-stack perfsonar test nodes running ipv4 throughput tests.  The local test node has an advanced configuration which includes multiple test interfaces, policy routing and anon-test interface that is used to by the host's main routing table as the default path.
\b0 \
\
\

\b CASE 1 - specify an ipv4 test only using hostnames
\b0 \
\
\cf2 \ul \ulc2 LAST WEEK\'92S RESULT\ulnone \
# pscheduler task --lead-bind enpl-pt2-10g.eos.nasa.gov throughput --source enpl-pt2-10g.eos.nasa.gov --dest uhmanoa-tp.ps.uhnet.net --ip-version=4\
Submitting task...\
Failed to post task: Error getting tools from uhmanoa-tp.ps.uhnet.net: Could not resolve host: enpl-pt2-10g.eos.nasa.gov; Unknown error\
\
The 'pscheduler troubleshoot' command may be of use in problem\
diagnosis. 'pscheduler troubleshoot --help' for more information.\cf0 \
\
\ul TODAY\'92S RESULT\ulnone \
\pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\partightenfactor0

\f1\fs22 \cf3 \cb4 \CocoaLigature0 [root@enpl ~]# pscheduler task --lead-bind enpl-pt2-10g.eos.nasa.gov throughput --source enpl-pt2-10g.eos.nasa.gov --dest uhmanoa-tp.ps.uhnet.net --ip-version=4\
Submitting task...\
Task URL:\
https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/f3e9cccf-7f98-4177-a1b9-cb6542451f22\
Running with tool 'iperf3'\
Fetching first run...\
\
Next scheduled run:\
https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/f3e9cccf-7f98-4177-a1b9-cb6542451f22/runs/26c73b83-c90b-4114-818c-cb3862880eda\
Starts 2019-12-09T12:28:56-05 (~63 seconds)\
Ends   2019-12-09T12:29:15-05 (~18 seconds)\
Waiting for result...\
\
* Stream ID 5\
Interval       Throughput     Retransmits    Current Window \
0.0 - 1.0      187.57 Mbps    0              1.09 MBytes    \
1.0 - 2.0      901.75 Mbps    0              65.51 MBytes   \
2.0 - 3.0      1.52 Gbps      0              65.51 MBytes   \
3.0 - 4.0      1.52 Gbps      0              65.51 MBytes   \
4.0 - 5.0      1.51 Gbps      0              65.51 MBytes   \
5.0 - 6.0      1.52 Gbps      0              65.51 MBytes   \
6.0 - 7.0      1.52 Gbps      0              65.51 MBytes   \
7.0 - 8.0      1.52 Gbps      0              65.51 MBytes   \
8.0 - 9.0      1.52 Gbps      0              65.51 MBytes   \
9.0 - 10.0     1.51 Gbps      0              65.51 MBytes   \
\
Summary\
Interval       Throughput     Retransmits    \
0.0 - 10.0     1.32 Gbps      0\
\
No further runs scheduled.
\f0\fs24 \cf0 \cb1 \CocoaLigature1 \
\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
\cf0 \

\b CASE 2 - specify an IPv4 test using hostnames except for the destination which is an IPv4 address literal
\b0 \
\
\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
\cf2 \ul \ulc2 LAST WEEK\'92S RESULT\cf0 \ulnone \
\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
\cf0 # pscheduler task --lead-bind enpl-pt2-10g.eos.nasa.gov throughput --source enpl-pt2-10g.eos.nasa.gov --dest 132.160.6.178 --ip-version=4\
Submitting task...\
Task URL:\
https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/d9727824-d2ec-4543-afc0-a72a5eabcd22\
Running with tool 'iperf3'\
Fetching first run...\
\
Next scheduled run:\
https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/d9727824-d2ec-4543-afc0-a72a5eabcd22/runs/3917927e-7c5d-4e72-b27c-974ee90bf19d\
Starts 2019-12-06T07:59:50-05 (~36 seconds)\
Ends   2019-12-06T08:00:09-05 (~18 seconds)\
Waiting for result...\
\
* Stream ID 5\
Interval       Throughput     Retransmits    Current Window \
0.0 - 1.0      187.57 Mbps    0              1.13 MBytes    \
1.0 - 2.0      922.80 Mbps    0              65.71 MBytes   \
2.0 - 3.0      1.53 Gbps      0              65.71 MBytes   \
3.0 - 4.0      1.52 Gbps      0              65.71 MBytes   \
4.0 - 5.0      1.54 Gbps      0              65.71 MBytes   \
5.0 - 6.0      1.51 Gbps      0              65.71 MBytes   \
6.0 - 7.0      1.54 Gbps      0              65.71 MBytes   \
7.0 - 8.0      1.52 Gbps      0              65.71 MBytes   \
8.0 - 9.0      1.53 Gbps      0              65.71 MBytes   \
9.0 - 10.0     1.53 Gbps      0              65.71 MBytes   \
\
Summary\
Interval       Throughput     Retransmits    \
0.0 - 10.0     1.33 Gbps      0\
\
No further runs scheduled.\
\
\ul TODAY\'92S RESULT\ulnone \
\pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\partightenfactor0

\f1\fs22 \cf3 \cb4 \CocoaLigature0 # pscheduler task --lead-bind enpl-pt2-10g.eos.nasa.gov throughput --source enpl-pt2-10g.eos.nasa.gov --dest 132.160.6.178 --ip-version=4\
Submitting task...\
Task URL:\
https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/20ecff75-bd95-4aa9-b110-91963527718c\
Running with tool 'iperf3'\
Fetching first run...\
\
Next scheduled run:\
https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/20ecff75-bd95-4aa9-b110-91963527718c/runs/e81d0d4a-aca6-4f74-8fe6-f4f5d32f720e\
Starts 2019-12-09T12:36:58-05 (~300 seconds)\
Ends   2019-12-09T12:37:17-05 (~18 seconds)\
Waiting for result...\
\
* Stream ID 5\
Interval       Throughput     Retransmits    Current Window \
0.0 - 1.0      184.44 Mbps    0              1.09 MBytes    \
1.0 - 2.0      911.82 Mbps    0              65.73 MBytes   \
2.0 - 3.0      1.51 Gbps      0              65.73 MBytes   \
3.0 - 4.0      1.52 Gbps      0              65.73 MBytes   \
4.0 - 5.0      1.51 Gbps      0              65.73 MBytes   \
5.0 - 6.0      1.52 Gbps      0              65.73 MBytes   \
6.0 - 7.0      1.52 Gbps      0              65.73 MBytes   \
7.0 - 8.0      1.52 Gbps      0              65.73 MBytes   \
8.0 - 9.0      1.52 Gbps      0              65.73 MBytes   \
9.0 - 10.0     1.51 Gbps      0              65.73 MBytes   \
\
Summary\
Interval       Throughput     Retransmits    \
0.0 - 10.0     1.32 Gbps      0\
\
No further runs scheduled.\
\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0

\f0\fs24 \cf0 \cb1 \CocoaLigature1 \
\

\b CASE 3 - specify an ipv4 test using an IPv4 address literal as the lead-bind and with hostnames for the remaining parameters
\b0 \
\
\ul LAST WEEK\'92S RESULT\ulnone \
# pscheduler task --lead-bind 169.154.197.28 throughput --source enpl-pt2-10g.eos.nasa.gov --dest uhmanoa-tp.ps.uhnet.net --ip-version=4\
Submitting task...\
Failed to post task: Error getting tools from uhmanoa-tp.ps.uhnet.net: bind failed with errno 22: Invalid argument\
\
The 'pscheduler troubleshoot' command may be of use in problem\
diagnosis. 'pscheduler troubleshoot --help' for more information.\
\
\ul TODAY\'92S RESULT\ulnone \
\pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\partightenfactor0

\f1\fs22 \cf3 \cb4 \CocoaLigature0 # pscheduler task --lead-bind 169.154.197.28 throughput --source enpl-pt2-10g.eos.nasa.gov --dest uhmanoa-tp.ps.uhnet.net --ip-version=4\
Submitting task...\
Task URL:\
https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/7413f8fd-6ff3-48b3-97a8-4e96eba8e8fe\
Running with tool 'iperf3'\
Fetching first run...\
\
Next scheduled run:\
https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/7413f8fd-6ff3-48b3-97a8-4e96eba8e8fe/runs/45d7e72b-70a0-4bbb-ad6f-b6dca401ff31\
Starts 2019-12-09T12:43:24-05 (~142 seconds)\
Ends   2019-12-09T12:43:43-05 (~18 seconds)\
Waiting for result...\
\
* Stream ID 5\
Interval       Throughput     Retransmits    Current Window \
0.0 - 1.0      187.56 Mbps    0              1.21 MBytes    \
1.0 - 2.0      933.24 Mbps    0              65.66 MBytes   \
2.0 - 3.0      1.54 Gbps      0              65.66 MBytes   \
3.0 - 4.0      1.52 Gbps      0              65.66 MBytes   \
4.0 - 5.0      1.53 Gbps      0              65.66 MBytes   \
5.0 - 6.0      1.52 Gbps      0              65.66 MBytes   \
6.0 - 7.0      1.53 Gbps      0              65.66 MBytes   \
7.0 - 8.0      1.53 Gbps      0              65.66 MBytes   \
8.0 - 9.0      1.52 Gbps      0              65.66 MBytes   \
9.0 - 10.0     1.54 Gbps      0              65.66 MBytes   \
\
Summary\
Interval       Throughput     Retransmits    \
0.0 - 10.0     1.34 Gbps      0\
\
No further runs scheduled.\
}