perfSONAR User Q&A and Other Discussion

[Eli Dart <>]

Hi all,

The place that the 8972 number often comes from is that 8972 is the size one must specify on the Linux ping command line in order to get a 9000-byte IP packet on the wire.  The size argument for ping specifies the data payload size, minus headers. So, adding 8 bytes of ICMP header and 20 bytes of IP header, we get 8972 + 8 + 20 = 9000.

In general, as Matt said, the R&E networks specify an MTU of 9000 bytes for peering connections and site connections when jumbo frames are supported. That means a 9000-byte IP packet, including headers. Another way to think about it is that the Ethernet frame data payload size is 9000 bytes, so depending on how your router counts the Ethernet frame, VLAN tags, and the Ethernet checksum, you might have to specify an MTU of 9018, 9014, or 9022, or something else.

For hosts, an MTU setting of 9000 bytes in the interface config (i.e. the argument to ifconfig) should do the right thing, so long as everything else in the broadcast domain is also set to the same MTU.

Eli

On Thu, Aug 11, 2016 at 3:30 PM, Matthew J Zekauskas <> wrote:

The convention in higher ed has been to use 9000 flat for the edge MTU when configuring "Jumbo frames".  See this fairly old statement for some background: <http://noc.net.internet2.edu/i2network/maps-documentation/policy-statements.html#Jumbo Frames>  MTU is the total packet length with headers.    The 8972 value is payload without headers to reach 9000 on the wire, so you don't want to set interface MTU to 8972 or the packets you can receive would max out at 8972 instead of 9000.

More pointers including some of the downsides and things to watch out for are in this fasterdata article: <http://fasterdata.es.net/network-tuning/mtu-issues/>

The switches (and everything else in that particular broadcast domain) need to also be at 9000 to have seamless operation.   Switches and intermediate nodes/routers may need to be higher -- say the 9000 byte frame is packaged in a 802.1q VLAN, you need to add an extra 8 bytes for VLAN headers.   I believe we just set link MTUs to be as high as they can be, and the particular (sub)interface MTU to be 9000.  (Some of this is in the Internet2 statement.)

If switch (or router) ports are not configured for large frames, then they will report "giant" errors if they receive a jumbo frame.

--Matt

On 8/11/16 6:11 PM, Jennewein, Douglas M wrote:

Hi, folks

This article got us thinking about the MTU configuration on our PerfSONAR nodes.

https://www.mylesgray.com/hardware/test-jumbo-frames-working/ 

At the OS level, should our nodes use 8972 instead of 9000?

Can this cause “giant” errors on the switch ports the PerfSonar nodes are connected to?

-- 
Doug Jennewein

Director of Research Computing

The University of South Dakota

--

Eli Dart, Network Engineer                          NOC: (510) 486-7600

ESnet Science Engagement Group                           (800) 333-7638

Lawrence Berkeley National Laboratory