perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi Charley,

Thanks for the additional info. The problem with rtt tests not properly looking-up AAAA records on DNS hosts is a known issue: https://github.com/perfsonar/pscheduler/issues/431. We actually fixed it in the source code a couple weeks ago but it has not yet been released. The plan is to include it as part of the 4.0.1 release in a few weeks. 

Thanks,

Andy

On July 1, 2017 at 12:44:10 PM, Charley Kneifel () wrote:

Hi Andy,

Both of the servers below should only be ping-able from v6 and are not open for performance tests.

These are both 4.0 systems (but different versions of Centos)

The build based on the v4 ISO is:  2606:a000:111c:406c:250:56ff:feb2:244a

The build based on an upgraded 3.5 is: 2606:a000:111c:406c:250:56ff:feb2:76b6

The config stanza from /etc/perfsonar/meshconfig-agent-tasks.conf is:

    <target>

        address   sdsc-pt1-v6.es.net
        <override_parameters>
            force_ipv6   1
            type   bwtraceroute
        </override_parameters>
        description   SDSC Node (IPV6)
    </target>

I can run a ping test to the IPv6 address

[charley@ps-node-c perfsonar]$ pscheduler task --debug rtt  --ip-version 6 --dest 2001:400:1101:1150::3
2017-07-01T12:11:43 Debug signal ignored; already not debugging
2017-07-01T12:11:43 Debug discontinued
2017-07-01T12:11:43 Assistance is from localhost
2017-07-01T12:11:43 Forcing default slip of PT5M
2017-07-01T12:11:43 Converting to spec via https://localhost/pscheduler/tests/rtt/spec
Submitting task...
2017-07-01T12:11:44 Fetching participant list
2017-07-01T12:11:44 Spec is: {"dest": "2001:400:1101:1150::3", "ip-version": 6, "schema": 1}
2017-07-01T12:11:45 Got participants: {u'participants': [None], u'null-reason': u'No source specified'}
2017-07-01T12:11:45 Lead is None
2017-07-01T12:11:45 Null reason is No source specified
2017-07-01T12:11:45 Pinging https://ps-node-c.kneifel.local/pscheduler/
2017-07-01T12:11:45 ps-node-c.kneifel.local is up
2017-07-01T12:11:45 Posting task to https://ps-node-c.kneifel.local/pscheduler/tasks
2017-07-01T12:11:45 Data is {"test": {"type": "rtt", "spec": {"dest": "2001:400:1101:1150::3", "ip-version": 6, "schema": 1}}, "schedule": {"slip": "PT5M"}, "schema": 1}
Task URL:
https://ps-node-c.kneifel.local/pscheduler/tasks/377d74c7-6b01-43c7-880c-8a5703a27355
2017-07-01T12:11:48 Posted https://ps-node-c.kneifel.local/pscheduler/tasks/377d74c7-6b01-43c7-880c-8a5703a27355
Running with tool 'ping'
Fetching first run...
2017-07-01T12:11:48 Fetching https://ps-node-c.kneifel.local/pscheduler/tasks/377d74c7-6b01-43c7-880c-8a5703a27355/runs/first
2017-07-01T12:11:49 Handing off: pscheduler watch --format text/plain --debug https://ps-node-c.kneifel.local/pscheduler/tasks/377d74c7-6b01-43c7-880c-8a5703a27355
2017-07-01T12:11:49 Debug signal ignored; already not debugging
2017-07-01T12:11:49 Debug discontinued
2017-07-01T12:11:49 Fetching https://ps-node-c.kneifel.local/pscheduler/tasks/377d74c7-6b01-43c7-880c-8a5703a27355

Next scheduled run:
https://ps-node-c.kneifel.local/pscheduler/tasks/377d74c7-6b01-43c7-880c-8a5703a27355/runs/6fb9065d-1a23-484a-84e5-08a05a0d2d57
Starts 2017-07-01T12:11:58-04:00 (~8 seconds)
Ends   2017-07-01T12:12:09-04:00 (~10 seconds)
Waiting for result...

1       sdsc-pt1-v6.es.net (2001:400:1101:1150::3)  64 Bytes  TTL 48  RTT 100.0000 ms
2       sdsc-pt1-v6.es.net (2001:400:1101:1150::3)  64 Bytes  TTL 48  RTT 101.0000 ms
3       sdsc-pt1-v6.es.net (2001:400:1101:1150::3)  64 Bytes  TTL 48  RTT 108.0000 ms
4       sdsc-pt1-v6.es.net (2001:400:1101:1150::3)  64 Bytes  TTL 48  RTT 106.0000 ms
5       sdsc-pt1-v6.es.net (2001:400:1101:1150::3)  64 Bytes  TTL 48  RTT 106.0000 ms

0% Packet Loss  RTT Min/Mean/Max/StdDev = 100.110000/104.620000/108.188000/3.125000 ms

No further runs scheduled.

But not to the name

[charley@ps-node-c perfsonar]$ pscheduler task --debug rtt  --ip-version 6 --dest sdsc-pt1-v6.es.net
2017-07-01T12:09:57 Debug signal ignored; already not debugging
2017-07-01T12:09:57 Debug discontinued
2017-07-01T12:09:57 Assistance is from localhost
2017-07-01T12:09:57 Forcing default slip of PT5M
2017-07-01T12:09:57 Converting to spec via https://localhost/pscheduler/tests/rtt/spec
Submitting task...
2017-07-01T12:09:58 Fetching participant list
2017-07-01T12:09:58 Spec is: {"dest": "sdsc-pt1-v6.es.net", "ip-version": 6, "schema": 1}
2017-07-01T12:09:59 Got participants: {u'participants': [None], u'null-reason': u'No source specified'}
2017-07-01T12:09:59 Lead is None
2017-07-01T12:09:59 Null reason is No source specified
2017-07-01T12:09:59 Pinging https://ps-node-c.kneifel.local/pscheduler/
2017-07-01T12:09:59 ps-node-c.kneifel.local is up
2017-07-01T12:09:59 Posting task to https://ps-node-c.kneifel.local/pscheduler/tasks
2017-07-01T12:09:59 Data is {"test": {"type": "rtt", "spec": {"dest": "sdsc-pt1-v6.es.net", "ip-version": 6, "schema": 1}}, "schedule": {"slip": "PT5M"}, "schema": 1}
Task URL:
https://ps-node-c.kneifel.local/pscheduler/tasks/88038988-ecc3-47dc-bf67-9d62ec218082
2017-07-01T12:10:02 Posted https://ps-node-c.kneifel.local/pscheduler/tasks/88038988-ecc3-47dc-bf67-9d62ec218082
Running with tool 'ping'
Fetching first run...
2017-07-01T12:10:02 Fetching https://ps-node-c.kneifel.local/pscheduler/tasks/88038988-ecc3-47dc-bf67-9d62ec218082/runs/first
2017-07-01T12:10:03 Handing off: pscheduler watch --format text/plain --debug https://ps-node-c.kneifel.local/pscheduler/tasks/88038988-ecc3-47dc-bf67-9d62ec218082
2017-07-01T12:10:03 Debug signal ignored; already not debugging
2017-07-01T12:10:03 Debug discontinued
2017-07-01T12:10:03 Fetching https://ps-node-c.kneifel.local/pscheduler/tasks/88038988-ecc3-47dc-bf67-9d62ec218082

Next scheduled run:
https://ps-node-c.kneifel.local/pscheduler/tasks/88038988-ecc3-47dc-bf67-9d62ec218082/runs/2bbbe7a4-ff9e-45d9-9ac4-8de329d9e3c0
Starts 2017-07-01T12:10:12-04:00 (~8 seconds)
Ends   2017-07-01T12:10:23-04:00 (~10 seconds)
Waiting for result...

Run failed.  The following errors were reported:
By ps-node-c.kneifel.local:
    Unable to resolve destination 'sdsc-pt1-v6.es.net'

No runs scheduled for this task.

Which indicates that the DNS query is failing - is there an explicit check for v6 addresses/QuadA records?

I have similar erors with the hostname for internal checks

Internal ping tests and throughput tests run with the IPv6 name, but not with the hostname

It sounds like I may have something incorrect in my IPv6 configuration, but ping6 and other services work fine.

I am able to run internal bandwidth tests internally when I use the V6 address directly:

[charley@ps-node-c perfsonar]$ pscheduler task --debug throughput  --ip-version 6 --dest 2606:a000:111c:406c:250:56ff:feb2:244a
2017-07-01T12:18:33 Debug signal ignored; already not debugging
2017-07-01T12:18:33 Debug discontinued
2017-07-01T12:18:33 Assistance is from localhost
2017-07-01T12:18:33 Forcing default slip of PT5M
2017-07-01T12:18:33 Converting to spec via https://localhost/pscheduler/tests/throughput/spec
Submitting task...
2017-07-01T12:18:33 Fetching participant list
2017-07-01T12:18:33 Spec is: {"dest": "2606:a000:111c:406c:250:56ff:feb2:244a", "ip-version": 6, "schema": 1}
2017-07-01T12:18:34 Got participants: {u'participants': [None], u'null-reason': u'No source specified'}
2017-07-01T12:18:34 Lead is None
2017-07-01T12:18:34 Null reason is No source specified
2017-07-01T12:18:34 Pinging https://ps-node-c.kneifel.local/pscheduler/
2017-07-01T12:18:34 ps-node-c.kneifel.local is up
2017-07-01T12:18:34 Posting task to https://ps-node-c.kneifel.local/pscheduler/tasks
2017-07-01T12:18:34 Data is {"test": {"type": "throughput", "spec": {"dest": "2606:a000:111c:406c:250:56ff:feb2:244a", "ip-version": 6, "schema": 1}}, "schedule": {"slip": "PT5M"}, "schema": 1}
Task URL:
https://ps-node-c.kneifel.local/pscheduler/tasks/a10da127-a23b-480c-8dbd-80d61627c8be
2017-07-01T12:18:38 Posted https://ps-node-c.kneifel.local/pscheduler/tasks/a10da127-a23b-480c-8dbd-80d61627c8be
Running with tool 'bwctliperf3'
Fetching first run...
2017-07-01T12:18:38 Fetching https://ps-node-c.kneifel.local/pscheduler/tasks/a10da127-a23b-480c-8dbd-80d61627c8be/runs/first
2017-07-01T12:18:40 Handing off: pscheduler watch --format text/plain --debug https://ps-node-c.kneifel.local/pscheduler/tasks/a10da127-a23b-480c-8dbd-80d61627c8be
2017-07-01T12:18:40 Debug signal ignored; already not debugging
2017-07-01T12:18:40 Debug discontinued
2017-07-01T12:18:40 Fetching https://ps-node-c.kneifel.local/pscheduler/tasks/a10da127-a23b-480c-8dbd-80d61627c8be

Next scheduled run:
https://ps-node-c.kneifel.local/pscheduler/tasks/a10da127-a23b-480c-8dbd-80d61627c8be/runs/f11838f8-3e50-43e2-a563-f9853ac47d6d
Starts 2017-07-01T12:18:48-04:00 (~7 seconds)
Ends   2017-07-01T12:19:59-04:00 (~70 seconds)
Waiting for result...

* Stream ID 16
Interval       Throughput     Retransmits    Current Window
0.0 - 1.0      11.70 Gbps     41             1.44 MBytes
1.0 - 2.0      16.60 Gbps     154            1.92 MBytes
2.0 - 3.0      15.01 Gbps     178            1.29 MBytes
3.0 - 4.0      14.34 Gbps     137            1.58 MBytes
4.0 - 5.0      14.26 Gbps     229            1.66 MBytes
5.0 - 6.0      15.39 Gbps     181            1.26 MBytes
6.0 - 7.0      16.29 Gbps     131            910.66 KBytes
7.0 - 8.0      15.07 Gbps     167            1.47 MBytes
8.0 - 9.0      14.69 Gbps     157            714.24 KBytes
9.0 - 10.0     16.65 Gbps     282            1.93 MBytes

Summary
Interval       Throughput     Retransmits
0.0 - 10.0     15.00 Gbps     1657

No further runs scheduled.

---

From: Andrew Lake <>
Sent: Friday, June 30, 2017 10:31 AM
To: ; Garnizov, Ivan (RRZE); Charley Kneifel
Subject: Re: [perfsonar-user] RE: perfSONAR v4 and IPv6

 

Hi Charley,

Can you send me the address of your perfSONAR server? I was going to pull up the pScheduler interface and see if I can find any references to your tests. I can also login on sdsc-pt1-v6.es.net and run some tests to get some more info if need. 

Thanks,

Andy

On June 30, 2017 at 10:03:41 AM, Charley Kneifel () wrote:

Hi Ivan,

 

It’s a 4.0 that was upgraded from 3.5 – it’s not a 3.5 system anymore.

 

Charley

 

From: Garnizov, Ivan (RRZE) [mailto:]
Sent: Friday, June 30, 2017 9:57 AM
To: Charley Kneifel <>;
Subject: RE: perfSONAR v4 and IPv6

 

Hi Charlie,

 

I am surprised you have found a problem with pS 3.5 on IPv6. It had been heavily used and tested in the past.

Do these servers participate in other tests, or are you testing only in between?

If you are testing (seeing problems) only in tests between these 2 servers, then it would make sense to diagnose a problem with backward compatibility.

                Please tell if you are able to run tests on the direction from pS 3.5 to pS 4.0.

If you are reporting here about problems with these 2 servers separately, I suggest you open 2 tracks, it might get very messy otherwise.

 

Please check the conf, where the tests get described and scheduled from.

For pS 3.5 /etc/perfsonar/regulartesting.conf

For pS 4.0 /etc/perfsonar/meshconfig-agent-tasks.conf

 

Both should have blocks that define your IPv6 tests, if these are missing, then nothing will be scheduled / executed as tests.

 

For the pS 4.0 here is the doc for running tests: http://docs.perfsonar.net/pscheduler_client_tasks.html

Example (please mind source and destination): pscheduler task --debug throughput --duration PT10S --source sdsc-pt1-v6.es.net --ip-version6 --dest ps-node-c --parallel 1

 

 

Regards,

Ivan Garnizov

 

GEANT SA1T2: pS deployments GN Operations

GEANT SA2T3: pS development team

GEANT SA3T5: eduPERT team

 

 

 

 

From: Charley Kneifel []
Sent: Freitag, 30. Juni 2017 14:40
To: Garnizov, Ivan (RRZE);
Subject: RE: perfSONAR v4 and IPv6

 

Hello Ivan

 

See below -

 

From: Garnizov, Ivan (RRZE) []
Sent: Friday, June 30, 2017 4:07 AM
To: Charley Kneifel <>;
Subject: RE: perfSONAR v4 and IPv6

 

Hi Charley,

 

Could you please clarify:

-          Which bundle do you use?

 

I have two builds – one based on the 4.X ISO (Centos 7) and another based on an upgrade 3.5 build (Centos 6.9) both have the same problem.

 

-          Do you have IPv4 tests working?

 

Yes –

 

-          How did you configure IPv6 tests (where)?

 

GUI – add a host with an IPv6 name and check the [] IPv6 checkbox and un-check the v4 checkbox.

  I have also directly used the IPv6 address.

 

-          Did you run command line tests for verification? Do you have results  you can share?

 

Can you send me an example – I have only run a command link ping test.

 

-          Any log events, that you find related and suspicious ?

 

Nothing that I noticed – but if there are particular places to look, please let me know.

 

 

Regards,

Ivan Garnizov

 

GEANT SA1T2: pS deployments GN Operations

GEANT SA2T3: pS development team

GEANT SA3T5: eduPERT team

 

 

From: [] On Behalf Of Charley Kneifel
Sent: Freitag, 30. Juni 2017 03:13
To:
Subject: [perfsonar-user] perfSONAR v4 and IPv6

 

Hello All,

 

I am unable to get IPv6 tests to work.

 

I can ping a remote IPv6 node:

 

ping6 sdsc-pt1-v6.es.net
PING sdsc-pt1-v6.es.net(sdsc-pt1-v6.es.net) 56 data bytes
64 bytes from sdsc-pt1-v6.es.net: icmp_seq=1 ttl=48 time=107 ms
64 bytes from sdsc-pt1-v6.es.net: icmp_seq=2 ttl=48 time=106 ms
64 bytes from sdsc-pt1-v6.es.net: icmp_seq=3 ttl=48 time=105 ms
64 bytes from sdsc-pt1-v6.es.net: icmp_seq=4 ttl=48 time=104 ms
^C
--- sdsc-pt1-v6.es.net ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3726ms
rtt min/avg/max/mdev = 104.495/105.886/107.681/1.283 ms

name services are resolving properly

[root@ps-node-c pscheduler]# nslookup -type=aaaa sdsc-pt1-v6.es.net
Server:         192.168.202.5
Address:        192.168.202.5#53

Non-authoritative answer:
sdsc-pt1-v6.es.net      has AAAA address 2001:400:1101:1150::3

 

I have added an IPv6 host to a PS4 node and confirmed that it is set for IPv6

 

I see no tests with the hostname (sdsc-pt1-v6) in the pscheduler logs and no tests with --ip-version=6 in the logs either.

 

I do have a private v4 address, but have a public V6 address that is exposed for icmpv6.

 

Anyone have any suggestions?

 

Charley