perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

Ivan (and group),

     I think I've resolved the problem myself.  It came down to the process I used for configuring my interfaces in the first place.  I had initially Put the base interface information (IP,SM Mask, DNS) for IPV4 and IPv6 in the interface configuration files at /etc/sysconfig/network-scripts/ifcfg-<name>

     Then I'd placed the Gateway information for the interfaces either in the /etc/sysconfig/network file, or in the routing table with a "route add" statement.  You can probably see from the methodology, I'm used to dealing with primarily single interface systems.  Once I deleted the Gateway information (IPv4 and IPv6) from the /etc/sysconfig/network file, and inserted it into the individual interface files at /etc/sysconfig/network-scripts/ifcfg-<name>.  I rebooted the system, ran the mod_interface_route script and things appear to be working now from a traffic standpoint. 

     I still don't see quite what I thought I'd see from the "ip -6 rule list" command, but traffic flows in and out correctly via both interfaces now (IPv4 and IPv6).

Thank you for your helpful prodding in the right direction.

Casey Russell

Network Engineer

Kansas Research and Education Network

2029 Becker Drive, Suite 282

Lawrence, KS  66047

(785)856-9820  ext 9809

On Mon, Jan 26, 2015 at 9:10 AM, Casey Russell <> wrote:

Ivan, 

>> The “magic” of /opt/perfsonar_ps/toolkit/scripts/mod_interface_route is not purposed to fix routing. The idea is to make server replies exit through the interface that has received inquiries.

Sure,  I never expected the script to have any effect on routing beyond right here at the host (Linux policy routing).  Just as you described, I expect the script to help the host determine which interface should respond to outside requests and which interface/gateway should be used to respond to those outside queries.

Looking back at my initial email, I see that the way I worded my email makes it look like I was testing OUT from both interfaces, but in fact, I was testing both out and inward with ICMP, and in both cases, that second interface failed to be able to talk to the outside world anywhere beyond it's gateway. 

I've included the Interface configuration info below, to clarify your question of whether or not the two interfaces are in the same subnet (they're not).  I also included more in-depth listing of the policy routing information that was created (showing the two source routing table entries).

em1       Link encap:Ethernet  HWaddr <stripped by me>
          inet addr:164.113.32.66  Bcast:164.113.32.71  Mask:255.255.255.248
          inet6 addr: 2001:49d0:23c0:8::26/64 Scope:Global
          inet6 addr: fe80::7a2b:cbff:fe0d:b9c8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:30183 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5600 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2128384 (2.0 MiB)  TX bytes:651400 (636.1 KiB)

p1p1      Link encap:Ethernet  HWaddr <stripped by me>
          inet addr:164.113.32.26  Bcast:164.113.32.31  Mask:255.255.255.248
          inet6 addr: 2001:49d0:23c0:3::26/64 Scope:Global
          inet6 addr: fe80::21b:21ff:fe8e:6304/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18291117 errors:0 dropped:0 overruns:0 frame:0
          TX packets:36827058 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1243119213 (1.1 GiB)  TX bytes:55062744850 (51.2 GiB)
          Memory:df3c0000-df3dffff


[root@ps-ksu-bw ~]# ip rule list
0:      from all lookup local
200:    from 164.113.32.26 lookup p1p1_source_route
200:    from 164.113.32.66 lookup em1_source_route
32766:  from all lookup main
32767:  from all lookup default
[root@ps-ksu-bw ~]# ip -6 rule list
0:      from all lookup local
32766:  from all lookup main

[root@ps-ksu-bw ~]# ip route list table p1p1_source_route
default via 164.113.32.30 dev p1p1
[root@ps-ksu-bw ~]# ip route list table em1_source_route
default via 164.113.32.70 dev em1

But remember, before we even get to the question of why the source routing for the second interface doesn't seem to be working, there is the question of why the two "mod_interface_route" commands ignored the IPv6 gateways altogether.

Thank you for your help.

Casey Russell

Network Engineer

Kansas Research and Education Network

2029 Becker Drive, Suite 282

Lawrence, KS  66047

(785)856-9820  ext 9809

On Mon, Jan 26, 2015 at 3:55 AM, Garnizov, Ivan (RRZE) <> wrote:

Hi Casey,

 

 

The “magic” of /opt/perfsonar_ps/toolkit/scripts/mod_interface_route is not purposed to fix routing. The idea is to make server replies exit through the interface that has received inquiries. Meaning that your tests should not be performed on the host itself, but from a remote host. In order to make sure that the source routing works, you should make dump on specific interface on the host you are configuring.

 

The “problem” you see there is probably because you have both interfaces in the same subnet, which means the routing is unable to determine, which is your desired path. It only sees that the second gateway is in the subnet of primary interface, therefore tries it through the primary interface, when initiating icmp tests.

 

Please share IP settings’ details in order to clear off this possibility.

 

Best regards,

Ivan

 

 

 

From: [mailto:] On Behalf Of Casey Russell
Sent: Saturday, January 24, 2015 3:28 PM
To:
Subject: [perfsonar-user] Dual Interface system

 

Group,

     I'm setting up a new dual-interface system (the first for us), and am using the instructions provided at http://docs.perfsonar.net/manage_dual_xface.html

     For clarity, the host will have 2 different physical interfaces in two separate subnets.  Each interface will have IPv4 and IPv4 connectivity.  The Basic IP and IPv6 setup goes simply enough, I get the device base connectivity through the primary interface.  And as you'd expect, the secondary interface can ping it's gateway, but can't talk to the outside world, because no policy (source routing) exists yet to give it a proper gateway.

     Then I ran the /opt/perfsonar_ps/toolkit/scripts/mod_interface_route script as suggested in the installation instructions.  I ran it once for each interface (modifying the examples to reflect the proper gateway info for my interfaces).  Each command ran cleanly (at least appeared to).  However, it appears to me that the script broke partway through executing each command and didn't properly build the routes for either interface. 

     After the commands are run.  Here is the output of "ip rule list"
[root@ps-ksu-bw iproute2]# ip rule list
0:      from all lookup local
200:    from 164.113.32.26 lookup p1p1_source_route
200:    from 164.113.32.66 lookup em1_source_route
32766:  from all lookup main
32767:  from all lookup default

     That looks good.  Looks like it installed the IPv4 gateway.  However, I still can only ping the gateway for that network from the second Linux interface (em1).  I have verified that other IPs in that subnet can talk to the world (to rule out routing issues or an ACL at the router).

     Also, if you look at the IPv6 table after the mod_interface_route script has been run for either interface, you see that it didn't appear to modify the IPv6 tables at all.  This is why I suspect it quietly broke mid-operation.

[root@ps-ksu-bw iproute2]# ip -6 rule list
0:      from all lookup local
32766:  from all lookup main

     Does anyone know if there is a debug or "verbose" switch for that mod_interface_route script?  (I tried the obvious -v --v)

Thank you,

Casey Russell

Network Engineer

Kansas Research and Education Network

2029 Becker Drive, Suite 282

Lawrence, KS  66047

(785)856-9820  ext 9809