perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

Ivan,

     On point one, this method (first alphabetical being used as the default system interface) wasn't a deal breaker in my situation, so it may not be for everyone, but it worked fine for us.

     On point two.  I took my cues from the PerfSONAR documentation when I put it in /etc/rc.local (although I believe they might mention the other method as well).  You almost certainly don't want to declare it in both places, as you said.  In our situation, our production machines run for months at a time and the only time their interfaces are reset are at reboot, so I wasn't all that concerned about the lack of a solution for resetting just one interface..  

     However, on hearing you describe it the way you have, you made me wonder.  It's not that much harder to set it up in the ifup/ifdown-local files.  I may just make a note to myself to go through and change that in the next several months during a future maintenance.

Casey Russell

Network Engineer

Kansas Research and Education Network

2029 Becker Drive, Suite 282

Lawrence, KS  66047

(785)856-9820  ext 9809

On Tue, Apr 14, 2015 at 10:46 AM, Garnizov, Ivan (RRZE) <> wrote:

Thanks Casey,

 

I have 2 major differences with your solution:

-          Putting all default GW statements in the ifcfg-* files ONLY means that you strictly rely on the OS method for selection of outgoing interface, which is the first name in alphabetical order. For me this breaks the operation of the system. It may require a workaround solution.

-          Running “mod_interface_route” commands from /etc/rc.local will not allow me to restore networking of the systems in cases with failed interfaces. I am setting the commands in the /sbin/ifup-local (ifdown-local) only so that they are repeated when the interface is brought back up. I am not putting them in rc.local as well, since this is going  to produce a conflict by running them twice on the initialisation of the system.

Please share how do you deal with this problem (re-initialisation of an interface)?

 

Best regards,

Ivan

 

From: Casey Russell [mailto:]
Sent: Tuesday, April 14, 2015 5:19 PM
To: Garnizov, Ivan (RRZE)
Cc:

Subject: Re: [perfsonar-user] Dual Interface system

 

Ivan,

     I can't reply for the developers and community, but I can reply from my experience.  I've modified my practices to do 3 things during my installs and have eliminated all of my routing problems in my install environment.  For reference, I am also installing dual interface systems.  And each interface is running both IPv4 and IPv6.

     First; where to put the Default gateway and other Interface settings.  I don't know if it's strictly required, but, in the end, I landed on putting all Default Gateway statements in the ifcfg-* files since it was the place most specific to the interface and subnet definition for each Interface.

     Second, the mod_interface_route script is broken on 3.4.1 systems and will NOT work for building the correct linux routing tables for v4 and v6.  It's been a couple of months now, but my recollection is that it builds v4 correctly, but botches v6.  The solution to this (early on) was to hand fix the mod_interface_route script on the affected machines.  Fortunately you already have a 3.4.2 machine or two to play with, so the faster fix would be to take the (fixed) file from one of those machines and move it to your 3.4.1 machine and run it again.  It should build the correct Linux routing table rules for both v4 and v6.

     Finally, after the mod_interface_route command was run, I had to reboot the system for reasons similar to what you're describing (although I don't remember if it's exactly the same).  Even though the routing rules were now built correctly, not all of the protocols on both interfaces worked correctly.  I restarted the network services, I unplugged the physical interface and plugged it back in.  Nothing but a system reboot actually allowed the system to come up with a clean configuration.  Make sure that you've added the appropriate mode_interface_route statements to your /etc/rc.local file before your reboot so that the routing changes are made permanent

     THAT'S WHY I'D ASK SOMEONE MORE KNOWLEDGABLE THAN ME to please respond to Ivan as well.  There's probably a way to do this without a reboot.  It just so happens in my environment that the reboot wasn't a big deal, so I finally tried it and it worked.  For me, that was good enough.

Casey Russell

Network Engineer

Kansas Research and Education Network

2029 Becker Drive, Suite 282

Lawrence, KS  66047

(785)856-9820  ext 9809

 

On Tue, Apr 14, 2015 at 9:40 AM, Garnizov, Ivan (RRZE) <> wrote:

Hi Casey, all,

 

I would like to bring up this topic again as there are some unclear to me configuration requirements, which I can’t seem  to find in the documentation.

 

Now currently I have physical server deployments with 3.4.1, which operate flawlessly with IPv4 and source routing regardless, if I define the default GW in the /etc/sysconfig/network.

BUT IPv6 source routing does not work at all and there is no indication on the routing table (ip -6 rule list command). Which leaves me with fully functional dual interface system on IPv4 and on IPv6 with only default interface.

 

I have played all the possible options (I could think of) and have the network reinitialized with “service network restart” only. IPv6 did not get registered and IPv6 source routing does not work there for me.

 

I decide to test the pS 3.4.2 on a VM system, which I can easily get offline access to. The system I installed just today. Now there is progress, but also a drawback.

I have configured a secondary interface with private IPv4 and IPv6 (real some imaginary) the interface is attached to nowhere.

My primary interface is also configured and is in my local subnet (not connecting through a gateway). Internet connectivity is present

Now when I configure the source routing using the command “mod_interface_route” for the secondary interface, I can see the route appear in the routing table (ip rule list) for IPv4 and IPv6.

BUT when I run the same command for my primary interface the system suddenly is locked for access from outside (I have tested with and without FW). I can operate through offline management interface only.

I get the proper entries in the routing table, but that seems not to be enough for proper communication, even on the same local subnet.

 

 

Now my question are:

 

1)      Has the source routing management been changed between pS 3.4.1 and pS 3.4.2? The release notes do not indicate it, but as Casey stated it for 3.4.1 and I have experienced it myself, the IPv6 routing table does not show indication that it is changed in 3.4.1, where in 3.4.2 changes are registered.

2)      What is the recommendation for the interface configuration for a multi-home system in terms of the following config files:

a.        /etc/sysconfig/network-scripts/ifcfg-*

b.      /etc/sysconfig/network

3)      If Casey is correct for the configuration of the system, then how would the sysadmin be able to define the default interface for outgoing requests? I know that the OS will choose one (I believe it is the first interface based on the name), but that is a very limiting factor for me.

4)      In fact I tried to reproduce the solution of Casey, but for IPv6 only, by removing the statement for a IPv6 GW in /etc/sysconfig/network, but keeping the one for IPv4 (I am sitting on this branch). I also removed the statements for IPv6GW in the ifcfg-* files and still, IPv6 source routing did not work for me. The only thing I avoided doing is reboot of the server for various reasons.

 

 

Please help with this case. I would prefer to be able to resolve the source routing case with pS 3.4.1, since that will allow me to work on the other services awaiting transitioning to newer versions.

 

Best regards,

Ivan

 

 

 

 

 

From: [mailto:] On Behalf Of Casey Russell
Sent: Tuesday, January 27, 2015 12:05 AM
To:
Subject: Re: [perfsonar-user] Dual Interface system

 

Ivan (and group),

     I think I've resolved the problem myself.  It came down to the process I used for configuring my interfaces in the first place.  I had initially Put the base interface information (IP,SM Mask, DNS) for IPV4 and IPv6 in the interface configuration files at /etc/sysconfig/network-scripts/ifcfg-<name>

     Then I'd placed the Gateway information for the interfaces either in the /etc/sysconfig/network file, or in the routing table with a "route add" statement.  You can probably see from the methodology, I'm used to dealing with primarily single interface systems.  Once I deleted the Gateway information (IPv4 and IPv6) from the /etc/sysconfig/network file, and inserted it into the individual interface files at /etc/sysconfig/network-scripts/ifcfg-<name>.  I rebooted the system, ran the mod_interface_route script and things appear to be working now from a traffic standpoint. 

     I still don't see quite what I thought I'd see from the "ip -6 rule list" command, but traffic flows in and out correctly via both interfaces now (IPv4 and IPv6).

 

Thank you for your helpful prodding in the right direction.

Casey Russell

Network Engineer

Kansas Research and Education Network

2029 Becker Drive, Suite 282

Lawrence, KS  66047

(785)856-9820  ext 9809

 

On Mon, Jan 26, 2015 at 9:10 AM, Casey Russell <> wrote:

Ivan, 

>> The “magic” of /opt/perfsonar_ps/toolkit/scripts/mod_interface_route is not purposed to fix routing. The idea is to make server replies exit through the interface that has received inquiries.

Sure,  I never expected the script to have any effect on routing beyond right here at the host (Linux policy routing).  Just as you described, I expect the script to help the host determine which interface should respond to outside requests and which interface/gateway should be used to respond to those outside queries.

Looking back at my initial email, I see that the way I worded my email makes it look like I was testing OUT from both interfaces, but in fact, I was testing both out and inward with ICMP, and in both cases, that second interface failed to be able to talk to the outside world anywhere beyond it's gateway. 

I've included the Interface configuration info below, to clarify your question of whether or not the two interfaces are in the same subnet (they're not).  I also included more in-depth listing of the policy routing information that was created (showing the two source routing table entries).

em1       Link encap:Ethernet  HWaddr <stripped by me>
          inet addr:164.113.32.66  Bcast:164.113.32.71  Mask:255.255.255.248
          inet6 addr: 2001:49d0:23c0:8::26/64 Scope:Global
          inet6 addr: fe80::7a2b:cbff:fe0d:b9c8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:30183 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5600 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2128384 (2.0 MiB)  TX bytes:651400 (636.1 KiB)

p1p1      Link encap:Ethernet  HWaddr <stripped by me>
          inet addr:164.113.32.26  Bcast:164.113.32.31  Mask:255.255.255.248
          inet6 addr: 2001:49d0:23c0:3::26/64 Scope:Global
          inet6 addr: fe80::21b:21ff:fe8e:6304/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18291117 errors:0 dropped:0 overruns:0 frame:0
          TX packets:36827058 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1243119213 (1.1 GiB)  TX bytes:55062744850 (51.2 GiB)
          Memory:df3c0000-df3dffff


[root@ps-ksu-bw ~]# ip rule list
0:      from all lookup local
200:    from 164.113.32.26 lookup p1p1_source_route
200:    from 164.113.32.66 lookup em1_source_route
32766:  from all lookup main
32767:  from all lookup default
[root@ps-ksu-bw ~]# ip -6 rule list
0:      from all lookup local
32766:  from all lookup main

[root@ps-ksu-bw ~]# ip route list table p1p1_source_route
default via 164.113.32.30 dev p1p1
[root@ps-ksu-bw ~]# ip route list table em1_source_route
default via 164.113.32.70 dev em1

But remember, before we even get to the question of why the source routing for the second interface doesn't seem to be working, there is the question of why the two "mod_interface_route" commands ignored the IPv6 gateways altogether.

 

Thank you for your help.

Casey Russell

Network Engineer

Kansas Research and Education Network

2029 Becker Drive, Suite 282

Lawrence, KS  66047

(785)856-9820  ext 9809

 

On Mon, Jan 26, 2015 at 3:55 AM, Garnizov, Ivan (RRZE) <> wrote:

Hi Casey,

 

 

The “magic” of /opt/perfsonar_ps/toolkit/scripts/mod_interface_route is not purposed to fix routing. The idea is to make server replies exit through the interface that has received inquiries. Meaning that your tests should not be performed on the host itself, but from a remote host. In order to make sure that the source routing works, you should make dump on specific interface on the host you are configuring.

 

The “problem” you see there is probably because you have both interfaces in the same subnet, which means the routing is unable to determine, which is your desired path. It only sees that the second gateway is in the subnet of primary interface, therefore tries it through the primary interface, when initiating icmp tests.

 

Please share IP settings’ details in order to clear off this possibility.

 

Best regards,

Ivan

 

 

 

From: [mailto:] On Behalf Of Casey Russell
Sent: Saturday, January 24, 2015 3:28 PM
To:
Subject: [perfsonar-user] Dual Interface system

 

Group,

     I'm setting up a new dual-interface system (the first for us), and am using the instructions provided at http://docs.perfsonar.net/manage_dual_xface.html

     For clarity, the host will have 2 different physical interfaces in two separate subnets.  Each interface will have IPv4 and IPv4 connectivity.  The Basic IP and IPv6 setup goes simply enough, I get the device base connectivity through the primary interface.  And as you'd expect, the secondary interface can ping it's gateway, but can't talk to the outside world, because no policy (source routing) exists yet to give it a proper gateway.

     Then I ran the /opt/perfsonar_ps/toolkit/scripts/mod_interface_route script as suggested in the installation instructions.  I ran it once for each interface (modifying the examples to reflect the proper gateway info for my interfaces).  Each command ran cleanly (at least appeared to).  However, it appears to me that the script broke partway through executing each command and didn't properly build the routes for either interface. 

     After the commands are run.  Here is the output of "ip rule list"
[root@ps-ksu-bw iproute2]# ip rule list
0:      from all lookup local
200:    from 164.113.32.26 lookup p1p1_source_route
200:    from 164.113.32.66 lookup em1_source_route
32766:  from all lookup main
32767:  from all lookup default

     That looks good.  Looks like it installed the IPv4 gateway.  However, I still can only ping the gateway for that network from the second Linux interface (em1).  I have verified that other IPs in that subnet can talk to the world (to rule out routing issues or an ACL at the router).

     Also, if you look at the IPv6 table after the mod_interface_route script has been run for either interface, you see that it didn't appear to modify the IPv6 tables at all.  This is why I suspect it quietly broke mid-operation.

[root@ps-ksu-bw iproute2]# ip -6 rule list
0:      from all lookup local
32766:  from all lookup main

     Does anyone know if there is a debug or "verbose" switch for that mod_interface_route script?  (I tried the obvious -v --v)

Thank you,

Casey Russell

Network Engineer

Kansas Research and Education Network

2029 Becker Drive, Suite 282

Lawrence, KS  66047

(785)856-9820  ext 9809