perfSONAR User Q&A and Other Discussion

[Aaron Brown <>]

Hey John-Paul,

The IPs in the test are actually provided by the client to the server, so there’s two separate cases you’d run into:

1) when the client is run on the same machine as the server (i.e. behind the NAT)

2) when the client is run on a remove machine from the server

In case #1, the client needs to tell the remote server a different IP than it tells the loopback server. Though, there is a quasi-work around for this case where we would just have the client specify that it’s behind a NAT, and the remote server wouldn’t really need to know the actual IP of the client.

In case #2, the server needs to know “when a client tells you IP X, I really need to bind to IP Y”. 

In 3.4, the plan is to migrate the ping and traceroute into bwctl as well, so it’ll eventually run into this case.

Given that, what specifically are you looking to do for the cloud-hosting device. Is it meant to be a remote testing device (i.e. other toolkit hosts would test against it), or is it meant to be something where folks would configure tests on that device for testing to other sites?

Cheers,

Aaron

On Jul 10, 2014, at 11:10 AM, John-Paul Robinson <> wrote:

Darn.

So is there anyway to control what IP bwctl sends to a peer?   I haven't come to an full understanding of how this subsystem works.  The config file didn't have anything obvious.

Our production 10G interface will be on a public facing IP so we shouldn't run into this with that node.

I was hoping to offer a cloud-hosted device for testing, comparison and tuning.  If I could get it advertising it's reachable address I could work around this limitation. 

BTW, are there any other tools in the suite that will be sensitive to this limitation?  Does bwctl control the ping, traceroute and other tests or is that handled be distinct measurement systems?

Thanks for the clarifications,

~jpr

On 07/10/2014 09:30 AM, Aaron Brown wrote:

Hi John-Paul,

The bwctl protocol passes around IP addresses to avoid the ambiguity of having each side resolve a DNS name. e.g. a DNS name resolves to both v4 and v6 addresses so each side has a different view of who “hostA” is.

There will be a workaround available in the 3.4 series that will allow you to say “this host is behind a firewall”, and the bwctl tests (assuming they’re using iperf3), will work as expected. However, that option isn’t available for the 3.3.2 series.

Cheers,

Aaron

On Jul 10, 2014, at 9:24 AM, John-Paul Robinson <> wrote:

Could you clarify what you mean by an "ambiguously defined FQDN"?

If using DHCP to assign and address and relying on the discover_external_address process to define the hostame is ambiguous, wha is the correct way to define the hostname for a perfsonar node so that it advertises it's hostname rather than it's IP address. 

I've looked through the install docs  and FAQ many times but they appear to fall short in describing how to set the hostname and ensure that perfsonar advertises it's hostname to peers rather than it's IP address.

I can certainly use your work around in this specific case, but this setup is a precursor to our production configuration, so I'd like to solve it the correct way.

~jpr

On 07/10/2014 08:46 AM, Garnizov, Ivan wrote:

Hi John-Paul,

 

From what I see here you have node1 setup with an IP and an ambiguously defined FQDN. I fully understand your DNS trick, but am not sure if there is a configuration possibility for that in the toolkit. After all perfSONAR has to bind itself to a valid IP address and what is more you can imagine in your scenario how easily would then be possible to achieve a DoS attack with perfSONAR.

 

Anyway since you are in control of the forward DNS zone you are able to define where connections go for “external viewers” like node 2. My point is: as a workaround of the problem to just manage the measurements from node2 (schedule them from there), since there you are able to specify the “correct” address of node1 (by hostname or even by IP and then measurements will be automatically setup in both directions from the point of view of node2.

 

How does that sound?

 

Best regards,

Iva

 

 

From: John-Paul Robinson [] 
Sent: Thursday, July 10, 2014 2:00 PM
To: Garnizov, Ivan; 
Subject: Re: [perfsonar-user] using a hostname instead of IP address in bwctl tests

 

Hi Ivan,

Let me see if I can clarify.

Node1 and node2 both use private IP addresses but are on different networks.   Additionally node1 sits behind a firewall that maps its behind-the-firewall IP to an infront-of-the-firewall address that node2 can reach.  

Node1 gets its IP via DCHP and hostname via reverse-DNS.  Here is the discover_external_address.log:

2014/06/26 12:57:16 (1236) INFO> discover_external_address:147 main:: - 172.21.1.3 is ipv4
2014/06/26 12:57:16 (1236) INFO> discover_external_address:149 main:: - 172.21.1.3 has a DNS name: perfsonar-cloud.uabgrid.uab.edu
2014/06/26 12:57:16 (1236) INFO> discover_external_address:151 main:: - 172.21.1.3 isn't private or we're okay with private addresses
2014/06/26 12:57:16 (1236) INFO> discover_external_address:249 main:: - Selected eth0/perfsonar-cloud.uabgrid.uab.edu as the primary address
2014/06/26 12:57:16 (1236) INFO> discover_external_address:256 main:: - Selected eth0/perfsonar-cloud.uabgrid.uab.edu as the primary ipv4 address
2014/06/26 12:57:16 (1236) INFO> discover_external_address:267 main:: - No primary ipv6 address found

Node2 is statically assigned and doesn't have a DNS entry.  Here is its discover_external_address.log:

2014/07/09 17:29:53 (1817) INFO> discover_external_address:147 main:: - 172.20.0.41 is ipv4
2014/07/09 17:29:53 (1817) INFO> discover_external_address:204 main:: - 172.20.0.41 is IPv4
2014/07/09 17:29:53 (1817) INFO> discover_external_address:206 main:: - 172.20.0.41 isn't private or we're okay with private addresses
2014/07/09 17:29:53 (1817) INFO> discover_external_address:249 main:: - Selected em1/172.20.0.41 as the primary address
2014/07/09 17:29:53 (1817) INFO> discover_external_address:256 main:: - Selected em1/172.20.0.41 as the primary ipv4 address
2014/07/09 17:29:53 (1817) INFO> discover_external_address:267 main:: - No primary ipv6 address found

Node2 is able to resolve node1's "public" ip address via an entry in node2's /etc/hosts file.

[root@localhost ~]# ping -c 3 perfsonar-cloud.uabgrid.uab.edu
PING perfsonar-cloud.uabgrid.uab.edu (172.22.128.35) 56(84) bytes of data.
64 bytes from perfsonar-cloud.uabgrid.uab.edu (172.22.128.35): icmp_seq=1 ttl=62 time=0.775 ms
64 bytes from perfsonar-cloud.uabgrid.uab.edu (172.22.128.35): icmp_seq=2 ttl=62 time=0.759 ms
64 bytes from perfsonar-cloud.uabgrid.uab.edu (172.22.128.35): icmp_seq=3 ttl=62 time=0.706 ms

--- perfsonar-cloud.uabgrid.uab.edu ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.706/0.746/0.775/0.043 ms

I have a scheduled throughput test registered on node1 that measures throughput to node2.  Node2 is listed as a test member using it's IP address.  This test is producing throughput results for iperf tests from node1 to node2.  However, when the node2 to node1 iperf test is run by node2, I see this error in the node2 messages log:

Jul 10 06:49:13 localhost bwctld[12025]: FILE=sapi.c, LINE=353, Connection to (172.20.0.41:4823) from (perfsonar-cloud.uabgrid.uab.edu:58029)
Jul 10 06:49:13 localhost bwctld[12025]: FILE=sapi.c, LINE=510, ControlSession([172.20.0.41]:4823) accepted from userid(nil):([perfsonar-cloud
.uabgrid.uab.edu]:58029)
Jul 10 06:49:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=791, Test Reservation Information: Current Time: 1404992953, Fuzz: 2.521484, Reser
vation Start: 1404992957, Reservation End: 1404992983, Test Start Time: 1404992960
Jul 10 06:49:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=442, Reservation Status: time=1404992953 action="new" sender=172.21.1.3 receiver=172
.20.0.41 tool=iperf res_start=1404992957 res_end=1404992982 test_start=1404992960
Jul 10 06:49:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=822, Time Slots
Jul 10 06:49:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=827, Time Slot 1: 1404992957 to 1404992983: 1 reservations
Jul 10 06:49:13 localhost bwctld[12026]: FILE=endpoint.c, LINE=886, Connect from unknown addr, assuming NAT
Jul 10 06:49:13 localhost bwctld[12026]: FILE=sapi.c, LINE=353, Connection to (172.20.0.41:6093) from (perfsonar-cloud.uabgrid.uab.edu:40420)
Jul 10 06:49:13 localhost bwctld[12026]: FILE=sapi.c, LINE=510, ControlSession([172.20.0.41]:6093) accepted from userid(nil):([perfsonar-cloud.uabgrid.uab.edu]:40420)
Jul 10 06:49:42 localhost bwctld[7855]: FILE=bwctld.c, LINE=442, Reservation Status: time=1404992982 action="remove" sender=172.21.1.3 receiver=172.20.0.41 tool=iperf res_start=1404992957 res_end=1404992982 test_start=1404992960
Jul 10 06:50:13 localhost bwctld[12063]: FILE=sapi.c, LINE=353, Connection to (172.20.0.41:4823) from (perfsonar-cloud.uabgrid.uab.edu:40296)
Jul 10 06:50:13 localhost bwctld[12063]: FILE=sapi.c, LINE=510, ControlSession([172.20.0.41]:4823) accepted from userid(nil):([perfsonar-cloud.uabgrid.uab.edu]:40296)
Jul 10 06:50:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=791, Test Reservation Information: Current Time: 1404993013, Fuzz: 2.582031, Reservation Start: 1404993016, Reservation End: 1404993043, Test Start Time: 1404993019
Jul 10 06:50:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=442, Reservation Status: time=1404993013 action="new" sender=172.20.0.41 receiver=172.21.1.3 tool=iperf res_start=1404993016 res_end=1404993042 test_start=1404993019
Jul 10 06:50:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=822, Time Slots
Jul 10 06:50:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=827, Time Slot 1: 1404993016 to 1404993043: 1 reservations
Jul 10 06:50:13 localhost bwctld[12064]: FILE=endpoint.c, LINE=942, Endpoint: Unable to connect to Peer([172.21.1.3]:6150): Operation now in progress
Jul 10 06:50:13 localhost bwctld[7855]: FILE=bwctld.c, LINE=442, Reservation Status: time=1404993013 action="remove" sender=172.20.0.41 receiver=172.21.1.3 tool=iperf res_start=1404993016 res_end=1404993042 test_start=1404993019    

Note host node2 is trying to reach node1 using it's behind-the-firewall address 172.21.1.3.  This address is unreachable and the connection fails.

My assumption is that node2 is being informed about node1, by the throughput test harness on node1, to find it's peer using an IP address.  

My preference would be for node1's throughput test harness to tell node2 to contact node1 using node1's DNS name.   That way when node2 resolves node1's name, it will get an IP address that is reachable from node2.

Hope this clarifies,

~jpr

On 07/10/2014 04:49 AM, Garnizov, Ivan wrote:

Hi John-Paul,

 

The perfSONAR suite does not apply restrictions on usage of hostnames. All you need is to make sure that each testing server you have is capable of RESOLVING the correct IP addresses (the visible ones) of the rest servers in the testing environment.

 

It seems also that you might be in the confusing situation, which applies to multi-homed servers, where special routing assignments must be set in order to provision the perfSONAR services on all interfaces simultaneously. Is that what you are trying to achieve?

 

From your explanation it does not become clear:

·         if these two hosts have real IP addresses or just private

·         if these hosts are being successfully resolved to the respective IPs correctly or not for each server.

·         which server is used to assign the measurements (assuming you are using the web interface)

 

Thus it is hard to answer the questions in your email!

 

 

Please also apply relevant information from the owamp/bwctl log file, which generally is /var/log/perfsonar/owamp_bwctl.log

 

Best regards,

Ivan

 

 

 

-----Original Message-----
From:  [] On Behalf Of John-Paul Robinson
Sent: Thursday, July 10, 2014 5:43 AM
To: 
Subject: [perfsonar-user] using a hostname instead of IP address in bwctl tests

 

Hi,

 

I'm trying to configure a perfsonar test environment and am having a hard time determining how to get my nodes to refer to each other by DNS name rather than IP address.

 

I have two nodes on different private networks and allow_internal_addresses is set on both.

 

My first node is configured by DHCP for it's IP and hostname.  It reflects its correct hostname in the /toolkit UI.  This node sits behind a firewall that exposes it's "public" IP to other nodes in the fabric.

My second node, which happens to be statically IP assigned, sits on another internal network.

 

The two nodes can reach each other when the using the correct destination IP addresses. (What about reaching each other by hostnames)  For example, when the second node uses the first nodes hostname which resolves to it's public IP and not the IP it uses behind the firewall.

 

My first node is set up to run scheduled throughput tests with the second node.  The node1->node2 tests succeed and data is being gathered for the tests.  However the node2->node1 reverse test that is triggered to run on node2 fails because it is being told to use node1's behind-the-firewall address.

 

 

I can't seem to find the documentation I need so the bwctl test scheduler tells node2 to use the DNS of node1 as the target address instead of the unreachable, behind-the-firewall IP address.

 

Is there such a configuration?

 

How could I be missing something that seems to be such an obvious config?

 

Thanks,

 

~jpr