perfSONAR User Q&A and Other Discussion

[John-Paul Robinson <>]

For the sake of completeness, I also tested configuring my tests from my node2 which can reach the public address of node1.  This also failed due to problems which are likely related to the "behind the firewall IP" limitations of bwctl's use of IP addresses.

Below is the log message from btwctl on node1 (the node with the behind the firewall IP).  It appears that btwcl on  node2 (the one on which the throughput test is scheduled) is telling node1 to bind to a specific IP:port combination.   Node2 correctly resolved the hostname of node1 (as specified in the "Test Members" config)  to the external address of node1, as evidenced by the appearance of the 172.22.128.35 IP.  However, it looks like node2 is telling node1 to bind a listener using this IP address.  Obviously, node1 can't use the firewall's public address.  It should use it's own private address.   

Jul 10 11:08:06 perfsonar-cloud bwctld[14328]: FILE=sapi.c, LINE=353, Connection to (perfsonar-cloud.uabgrid.uab.edu:4823) from (172.20.0.41:3
8014)
Jul 10 11:08:06 perfsonar-cloud bwctld[14328]: FILE=sapi.c, LINE=510, ControlSession([perfsonar-cloud.uabgrid.uab.edu]:4823) accepted from use
rid(nil):([172.20.0.41]:38014)
Jul 10 11:08:06 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=791, Test Reservation Information: Current Time: 1405008486, Fuzz: 1.734863
, Reservation Start: 1405008488, Reservation End: 1405008513, Test Start Time: 1405008490
Jul 10 11:08:06 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=442, Reservation Status: time=1405008486 action="new" sender=172.22.128.35 re
ceiver=172.20.0.41 tool=iperf res_start=1405008488 res_end=1405008512 test_start=1405008490
Jul 10 11:08:06 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=822, Time Slots
Jul 10 11:08:06 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=827, Time Slot 1: 1405008488 to 1405008513: 1 reservations
Jul 10 11:08:06 perfsonar-cloud bwctld[14329]: FILE=capi.c, LINE=142, bind(): Cannot assign requested address
Jul 10 11:08:06 perfsonar-cloud bwctld[14329]: FILE=endpoint.c, LINE=935, Endpoint: Unable to connect to Peer([172.20.0.41]:6033): Cannot assign requested address
Jul 10 11:08:06 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=442, Reservation Status: time=1405008486 action="remove" sender=172.22.128.35 receiver=172.20.0.41 tool=iperf res_start=1405008488 res_end=1405008512 test_start=1405008490
Jul 10 11:08:08 perfsonar-cloud bwctld[14333]: FILE=sapi.c, LINE=353, Connection to (perfsonar-cloud.uabgrid.uab.edu:4823) from (172.20.0.41:47438)
Jul 10 11:08:08 perfsonar-cloud bwctld[14333]: FILE=sapi.c, LINE=510, ControlSession([perfsonar-cloud.uabgrid.uab.edu]:4823) accepted from userid(nil):([172.20.0.41]:47438)
Jul 10 11:08:08 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=791, Test Reservation Information: Current Time: 1405008488, Fuzz: 1.737305, Reservation Start: 1405008491, Reservation End: 1405008515, Test Start Time: 1405008492
Jul 10 11:08:08 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=442, Reservation Status: time=1405008488 action="new" sender=172.20.0.41 receiver=172.22.128.35 tool=iperf res_start=1405008491 res_end=1405008514 test_start=1405008492
Jul 10 11:08:08 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=822, Time Slots
Jul 10 11:08:08 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=827, Time Slot 1: 1405008491 to 1405008515: 1 reservations
Jul 10 11:08:08 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=791, Test Reservation Information: Current Time: 1405008488, Fuzz: 1.737305, Reservation Start: 1405008513, Reservation End: 1405008538, Test Start Time: 1405008515
Jul 10 11:08:08 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=442, Reservation Status: time=1405008488 action="reschedule" sender=172.20.0.41 receiver=172.22.128.35 tool=iperf res_start=1405008513 res_end=1405008537 test_start=1405008515
Jul 10 11:08:08 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=822, Time Slots
Jul 10 11:08:08 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=827, Time Slot 1: 1405008513 to 1405008538: 1 reservations
Jul 10 11:08:08 perfsonar-cloud bwctld[14333]: FILE=endpoint.c, LINE=319, bind([172.22.128.35]:6164): Cannot assign requested address
Jul 10 11:08:08 perfsonar-cloud bwctld[14333]: FILE=protocol.c, LINE=2056, _BWLWriteStopSession called in wrong state.
Jul 10 11:08:08 perfsonar-cloud bwctld[14306]: FILE=bwctld.c, LINE=442, Reservation Status: time=1405008488 action="remove" sender=172.20.0.41 receiver=172.22.128.35 tool=iperf res_start=1405008513 res_end=1405008537 test_start=1405008515
Jul 10 11:08:08 perfsonar-cloud bwctld[14333]: FILE=bwctld.c, LINE=1471, Control session terminated abnormally...

This looks like the same issue Aaron clarified.  Bwctl uses IP addresses rather than hostnames.  In this case it translates the target hostname to the node2 visible address and sends that address along on the bwctl channel to node1.  Unfortunately node1 can't use that address since that's the firewall's IP and not node1's private IP.

Anyway, it's clear there are issues going across firewall boundaries and the only way to change that would be to explicitly control what IP address gets sent along on the bwctl control channel.

If I were wishing for features, I'd say allow DNS names to be sent on the control channel and rely on the deploying site to create valid, unique names for specific interfaces on their perfsonar node.  This would ensure things work correctly if the site correctly manages its names for each interface.  For example, I could assign names like ps-mytag-ipv4-10g.my.domain or ps-mytag-ipv6-10g.my.domain or ps-mytag-1g.my.domain.  If I ensure that both the public DNS and privately visible namespace are configured correctly, I should be able to ensure that my tests are going to the exact interface I want them to go to.

Thanks,

~jpr

On 07/10/2014 11:10 AM, John-Paul Robinson wrote:

Darn.

So is there anyway to control what IP bwctl sends to a peer?   I haven't come to an full understanding of how this subsystem works.  The config file didn't have anything obvious.

Our production 10G interface will be on a public facing IP so we shouldn't run into this with that node.

I was hoping to offer a cloud-hosted device for testing, comparison and tuning.  If I could get it advertising it's reachable address I could work around this limitation. 

BTW, are there any other tools in the suite that will be sensitive to this limitation?  Does bwctl control the ping, traceroute and other tests or is that handled be distinct measurement systems?

Thanks for the clarifications,

~jpr

On 07/10/2014 09:30 AM, Aaron Brown wrote:

Hi John-Paul,

The bwctl protocol passes around IP addresses to avoid the ambiguity of having each side resolve a DNS name. e.g. a DNS name resolves to both v4 and v6 addresses so each side has a different view of who “hostA” is.

There will be a workaround available in the 3.4 series that will allow you to say “this host is behind a firewall”, and the bwctl tests (assuming they’re using iperf3), will work as expected. However, that option isn’t available for the 3.3.2 series.

Cheers,

Aaron

On Jul 10, 2014, at 9:24 AM, John-Paul Robinson <> wrote:

Could you clarify what you mean by an "ambiguously defined FQDN"?

If using DHCP to assign and address and relying on the discover_external_address process to define the hostame is ambiguous, wha is the correct way to define the hostname for a perfsonar node so that it advertises it's hostname rather than it's IP address. 

I've looked through the install docs  and FAQ many times but they appear to fall short in describing how to set the hostname and ensure that perfsonar advertises it's hostname to peers rather than it's IP address.

I can certainly use your work around in this specific case, but this setup is a precursor to our production configuration, so I'd like to solve it the correct way.

~jpr

On 07/10/2014 08:46 AM, Garnizov, Ivan wrote:

Hi John-Paul,

 

From what I see here you have node1 setup with an IP and an ambiguously defined FQDN. I fully understand your DNS trick, but am not sure if there is a configuration possibility for that in the toolkit. After all perfSONAR has to bind itself to a valid IP address and what is more you can imagine in your scenario how easily would then be possible to achieve a DoS attack with perfSONAR.

 

Anyway since you are in control of the forward DNS zone you are able to define where connections go for “external viewers” like node 2. My point is: as a workaround of the problem to just manage the measurements from node2 (schedule them from there), since there you are able to specify the “correct” address of node1 (by hostname or even by IP and then measurements will be automatically setup in both directions from the point of view of node2.

 

How does that sound?

 

Best regards,

Iva

 

 

From: John-Paul Robinson [] 
Sent: Thursday, July 10, 2014 2:00 PM
To: Garnizov, Ivan; 
Subject: Re: [perfsonar-user] using a hostname instead of IP address in bwctl tests

 

Hi Ivan,

Let me see if I can clarify.

Node1 and node2 both use private IP addresses but are on different networks.   Additionally node1 sits behind a firewall that maps its behind-the-firewall IP to an infront-of-the-firewall address that node2 can reach.  

Node1 gets its IP via DCHP and hostname via reverse-DNS.  Here is the discover_external_address.log:

2014/06/26 12:57:16 (1236) INFO> discover_external_address:147 main:: - 172.21.1.3 is ipv4
2014/06/26 12:57:16 (1236) INFO> discover_external_address:149 main:: - 172.21.1.3 has a DNS name: perfsonar-cloud.uabgrid.uab.edu
2014/06/26 12:57:16 (1236) INFO> discover_external_address:151 main:: - 172.21.1.3 isn't private or we're okay with private addresses
2014/06/26 12:57:16 (1236) INFO> discover_external_address:249 main:: - Selected eth0/perfsonar-cloud.uabgrid.uab.edu as the primary address
2014/06/26 12:57:16 (1236) INFO> discover_external_address:256 main:: - Selected eth0/perfsonar-cloud.uabgrid.uab.edu as the primary ipv4 address
2014/06/26 12:57:16 (1236) INFO> discover_external_address:267 main:: - No primary ipv6 address found

Node2 is statically assigned and doesn't have a DNS entry.  Here is its discover_external_address.log:

2014/07/09 17:29:53 (1817) INFO> discover_external_address:147 main:: - 172.20.0.41 is ipv4
2014/07/09 17:29:53 (1817) INFO> discover_external_address:204 main:: - 172.20.0.41 is IPv4
2014/07/09 17:29:53 (1817) INFO> discover_external_address:206 main:: - 172.20.0.41 isn't private or we're okay with private addresses
2014/07/09 17:29:53 (1817) INFO> discover_external_address:249 main:: - Selected em1/172.20.0.41 as the primary address
2014/07/09 17:29:53 (1817) INFO> discover_external_address:256 main:: - Selected em1/172.20.0.41 as the primary ipv4 address
2014/07/09 17:29:53 (1817) INFO> discover_external_address:267 main:: - No primary ipv6 address found

Node2 is able to resolve node1's "public" ip address via an entry in node2's /etc/hosts file.

[root@localhost ~]# ping -c 3 perfsonar-cloud.uabgrid.uab.edu
PING perfsonar-cloud.uabgrid.uab.edu (172.22.128.35) 56(84) bytes of data.
64 bytes from perfsonar-cloud.uabgrid.uab.edu (172.22.128.35): icmp_seq=1 ttl=62 time=0.775 ms
64 bytes from perfsonar-cloud.uabgrid.uab.edu (172.22.128.35): icmp_seq=2 ttl=62 time=0.759 ms
64 bytes from perfsonar-cloud.uabgrid.uab.edu (172.22.128.35): icmp_seq=3 ttl=62 time=0.706 ms

--- perfsonar-cloud.uabgrid.uab.edu ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.706/0.746/0.775/0.043 ms

I have a scheduled throughput test registered on node1 that measures throughput to node2.  Node2 is listed as a test member using it's IP address.  This test is producing throughput results for iperf tests from node1 to node2.  However, when the node2 to node1 iperf test is run by node2, I see this error in the node2 messages log:

Jul 10 06:49:13 localhost bwctld[12025]: FILE=sapi.c, LINE=353, Connection to (172.20.0.41:4823) from (perfsonar-cloud.uabgrid.uab.edu:58029)
Jul 10 06:49:13 localhost bwctld[12025]: FILE=sapi.c, LINE=510, ControlSession([172.20.0.41]:4823) accepted from userid(nil):([perfsonar-cloud
.uabgrid.uab.edu]:58029)
Jul 10 06:49:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=791, Test Reservation Information: Current Time: 1404992953, Fuzz: 2.521484, Reser
vation Start: 1404992957, Reservation End: 1404992983, Test Start Time: 1404992960
Jul 10 06:49:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=442, Reservation Status: time=1404992953 action="new" sender=172.21.1.3 receiver=172
.20.0.41 tool=iperf res_start=1404992957 res_end=1404992982 test_start=1404992960
Jul 10 06:49:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=822, Time Slots
Jul 10 06:49:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=827, Time Slot 1: 1404992957 to 1404992983: 1 reservations
Jul 10 06:49:13 localhost bwctld[12026]: FILE=endpoint.c, LINE=886, Connect from unknown addr, assuming NAT
Jul 10 06:49:13 localhost bwctld[12026]: FILE=sapi.c, LINE=353, Connection to (172.20.0.41:6093) from (perfsonar-cloud.uabgrid.uab.edu:40420)
Jul 10 06:49:13 localhost bwctld[12026]: FILE=sapi.c, LINE=510, ControlSession([172.20.0.41]:6093) accepted from userid(nil):([perfsonar-cloud.uabgrid.uab.edu]:40420)
Jul 10 06:49:42 localhost bwctld[7855]: FILE=bwctld.c, LINE=442, Reservation Status: time=1404992982 action="remove" sender=172.21.1.3 receiver=172.20.0.41 tool=iperf res_start=1404992957 res_end=1404992982 test_start=1404992960
Jul 10 06:50:13 localhost bwctld[12063]: FILE=sapi.c, LINE=353, Connection to (172.20.0.41:4823) from (perfsonar-cloud.uabgrid.uab.edu:40296)
Jul 10 06:50:13 localhost bwctld[12063]: FILE=sapi.c, LINE=510, ControlSession([172.20.0.41]:4823) accepted from userid(nil):([perfsonar-cloud.uabgrid.uab.edu]:40296)
Jul 10 06:50:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=791, Test Reservation Information: Current Time: 1404993013, Fuzz: 2.582031, Reservation Start: 1404993016, Reservation End: 1404993043, Test Start Time: 1404993019
Jul 10 06:50:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=442, Reservation Status: time=1404993013 action="new" sender=172.20.0.41 receiver=172.21.1.3 tool=iperf res_start=1404993016 res_end=1404993042 test_start=1404993019
Jul 10 06:50:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=822, Time Slots
Jul 10 06:50:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=827, Time Slot 1: 1404993016 to 1404993043: 1 reservations
Jul 10 06:50:13 localhost bwctld[12064]: FILE=endpoint.c, LINE=942, Endpoint: Unable to connect to Peer([172.21.1.3]:6150): Operation now in progress
Jul 10 06:50:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=442, Reservation Status: time=1404993013 action="remove" sender=172.20.0.41 receiver=172.21.1.3 tool=iperf res_start=1404993016 res_end=1404993042 test_start=1404993019    

Note host node2 is trying to reach node1 using it's behind-the-firewall address 172.21.1.3.  This address is unreachable and the connection fails.

My assumption is that node2 is being informed about node1, by the throughput test harness on node1, to find it's peer using an IP address.  

My preference would be for node1's throughput test harness to tell node2 to contact node1 using node1's DNS name.   That way when node2 resolves node1's name, it will get an IP address that is reachable from node2.

Hope this clarifies,

~jpr

On 07/10/2014 04:49 AM, Garnizov, Ivan wrote:

Hi John-Paul,

 

The perfSONAR suite does not apply restrictions on usage of hostnames. All you need is to make sure that each testing server you have is capable of RESOLVING the correct IP addresses (the visible ones) of the rest servers in the testing environment.

 

It seems also that you might be in the confusing situation, which applies to multi-homed servers, where special routing assignments must be set in order to provision the perfSONAR services on all interfaces simultaneously. Is that what you are trying to achieve?

 

From your explanation it does not become clear:

·         if these two hosts have real IP addresses or just private

·         if these hosts are being successfully resolved to the respective IPs correctly or not for each server.

·         which server is used to assign the measurements (assuming you are using the web interface)

 

Thus it is hard to answer the questions in your email!

 

 

Please also apply relevant information from the owamp/bwctl log file, which generally is /var/log/perfsonar/owamp_bwctl.log

 

Best regards,

Ivan

 

 

 

-----Original Message-----
From:  [] On Behalf Of John-Paul Robinson
Sent: Thursday, July 10, 2014 5:43 AM
To: 
Subject: [perfsonar-user] using a hostname instead of IP address in bwctl tests

 

Hi,

 

I'm trying to configure a perfsonar test environment and am having a hard time determining how to get my nodes to refer to each other by DNS name rather than IP address.

 

I have two nodes on different private networks and allow_internal_addresses is set on both.

 

My first node is configured by DHCP for it's IP and hostname.  It reflects its correct hostname in the /toolkit UI.  This node sits behind a firewall that exposes it's "public" IP to other nodes in the fabric.

My second node, which happens to be statically IP assigned, sits on another internal network.

 

The two nodes can reach each other when the using the correct destination IP addresses. (What about reaching each other by hostnames)  For example, when the second node uses the first nodes hostname which resolves to it's public IP and not the IP it uses behind the firewall.

 

My first node is set up to run scheduled throughput tests with the second node.  The node1->node2 tests succeed and data is being gathered for the tests.  However the node2->node1 reverse test that is triggered to run on node2 fails because it is being told to use node1's behind-the-firewall address.

 

 

I can't seem to find the documentation I need so the bwctl test scheduler tells node2 to use the DNS of node1 as the target address instead of the unreachable, behind-the-firewall IP address.

 

Is there such a configuration?

 

How could I be missing something that seems to be such an obvious config?

 

Thanks,

 

~jpr