perfSONAR User Q&A and Other Discussion

[aplovich <>]

Hello,

I'm currently standing up a couple servers to test perfsonar, each server has a separate interface for bandwidth and latency checks.  However, I'm running into an issue with configuring multiple interfaces in that when I apply the ip rules, (per http://docs.perfsonar.net/manage_dual_xface.html) TCP connections fail over one of them.  I've narrowed the cause down to the ip rules since removing one of them allows traffic to pass normally.  I'm wondering if anybody has experienced a similar issue. 

Some background: both servers are virtual machines with vNICs in the same two subnets (one for bandwidth, one for latency).  The final environment will be setup on physical hardware, this environment is just for testing.  Oddly this doesn't affect ping: pings sourced from the problem interface work as expected.  Another oddity is that TCP connections over the latency interface work as expected even though it has the same ip rules as the bandwidth interface.  I've watched (via tcpdump) a telnet session to bwctld's port (4823) on another test server.  When the rules are applied I can see that the sending machine tries to send the initial SYN, but it never reaches the receiving server.  This makes it seem the issue is lower in the network stack than where tcpdump captures from.

I've tried removing the default route in the main routing table, removing the latency interface's rules but leaving the bandwidth interface's rules, changing the priorities of each interface's "source_route" table, swapping the VM's VMXnet3 vNICs for E1000s.

psonar-dev1 ~]$ ip rule

0:    from all lookup local
200:    from 146.139.146.67 lookup eth0_source_route <-- bandwidth interface (TCP connections are broken), removing just this rule allows connections to establish
200:    from 146.137.22.29 lookup eth1_source_route  <--- latency interface (TCP connections still work)
32766:    from all lookup main
32767:    from all lookup default

What tcpdump sees on the host with the above rules as it tries to connect to TCP 4823 on the other perfsonar server

psonar-dev1 ~]$ sudo tcpdump -n -i eth0 port 4823
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:01:03.819472 IP 146.139.146.67.56058 > 146.139.146.69.4823: Flags [S], seq 3795946227, win 64240, options [mss 1460,sackOK,TS val 599700 ecr 0,nop,wscale 11], length 0
11:01:04.818838 IP 146.139.146.67.56058 > 146.139.146.69.4823: Flags [S], seq 3795946227, win 64240, options [mss 1460,sackOK,TS val 600700 ecr 0,nop,wscale 11], length 0
11:01:06.818834 IP 146.139.146.67.56058 > 146.139.146.69.4823: Flags [S], seq 3795946227, win 64240, options [mss 1460,sackOK,TS val 602700 ecr 0,nop,wscale 11], length 0
11:01:10.818854 IP 146.139.146.67.56058 > 146.139.146.69.4823: Flags [S], seq 3795946227, win 64240, options [mss 1460,sackOK,TS val 606700 ecr 0,nop,wscale 11], length 0

What the receiving perfsonar server sees (nothing)

psonar-dev2 ~]$ sudo tcpdump -n -i eth0 port 4823
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

This same connection works over the latency interface:

psonar-dev1 ~]$ sudo tcpdump -n -i eth1 port 4823
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
11:05:19.023768 IP 146.137.22.29.34496 > 146.137.22.41.4823: Flags [S], seq 429553556, win 64240, options [mss 1460,sackOK,TS val 854899 ecr 0,nop,wscale 11], length 0
11:05:19.024293 IP 146.137.22.41.4823 > 146.137.22.29.34496: Flags [S.], seq 1359570240, ack 429553557, win 65160, options [mss 1460,sackOK,TS val 62332819 ecr 854899,nop,wscale 11], length 0
11:05:19.024332 IP 146.137.22.29.34496 > 146.137.22.41.4823: Flags [.], ack 1, win 32, options [nop,nop,TS val 854905 ecr 62332819], length 0
11:05:19.026903 IP 146.137.22.41.4823 > 146.137.22.29.34496: Flags [P.], seq 1:33, ack 1, win 32, options [nop,nop,TS val 62332822 ecr 854905], length 32
11:05:19.026915 IP 146.137.22.29.34496 > 146.137.22.41.4823: Flags [.], ack 33, win 32, options [nop,nop,TS val 854908 ecr 62332822], length 0

psonar-dev2 ~]$ sudo tcpdump -n -i eth1 port 4823
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
11:05:19.030753 IP 146.137.22.29.34496 > 146.137.22.41.4823: Flags [S], seq 429553556, win 64240, options [mss 1460,sackOK,TS val 854899 ecr 0,nop,wscale 11], length 0
11:05:19.030922 IP 146.137.22.41.4823 > 146.137.22.29.34496: Flags [S.], seq 1359570240, ack 429553557, win 65160, options [mss 1460,sackOK,TS val 62332819 ecr 854899,nop,wscale 11], length 0
11:05:19.031278 IP 146.137.22.29.34496 > 146.137.22.41.4823: Flags [.], ack 1, win 32, options [nop,nop,TS val 854905 ecr 62332819], length 0
11:05:19.033582 IP 146.137.22.41.4823 > 146.137.22.29.34496: Flags [P.], seq 1:33, ack 1, win 32, options [nop,nop,TS val 62332822 ecr 854905], length 32
11:05:19.033865 IP 146.137.22.29.34496 > 146.137.22.41.4823: Flags [.], ack 33, win 32, options [nop,nop,TS val 854908 ecr 62332822], length 0

Regards,

-- 
Tony Plovich ()
Infrastructure and Operation Services 
Argonne National Laboratory 
Phone: +1 630.252.2359