perfSONAR User Q&A and Other Discussion

[Joe Metzger <>]

Mark,
The primary advantages of using large MTUs in my opinion are:

  The MTU is a parameter in most TCP recovery algorithms, so they 
  can recover up to 4x faster with 9K packets vs 1.5K packets.

  Many application servers are actually cpu bound, and the load
  imposed by the interrupt stream generated by large bandwidth
  inbound flows gates performance.

Or, if you want other opinions, google should find dozens of
presentations, and papers that explain why 1500 bytes was a 
good choice when networks were 10,000 times slower and is way
to small now.  :-)

As far as the choice of 9000, it is the value selected by the
JET community.  The JET is a subcommittee of the LSN.
(http://www.nitrd.gov/subcommittee/lsn.aspx)

Note that the position of this committee within the federal advisory
structure was not important for this decision.  Most of the operators
were getting requests from their constituents to support large mtu's.
The operators of the JET networks agreed that life would be better
for all of us if we all agreed on the same definition of a large mtu.

We picked a number that would allow end applications to easily transfer
8K of application data in a packet with lots of room for all sorts of 
protocol overhead. (Ethernet, VLANS, MPLS, IPSEC, IP, TCP, etc.)
Another constraint was that all of our equipment needed to support
it, and so it had to be below the floor supported by all of the
vendors common at the time.


As far as what goes faster, I have seen applications, TCP stacks, and
hardware that is sensitive to MTUs.  Some of it is the simple smooth
curves that are what you would expect based on calcuated overhead
rates. Some systems are limited by bits/second while others are limited
by packets/second.  But there are also other strange corner cases
or bugs where one particular packet size exercises corner cases
and gets significantly worse performance than packets a couple of bytes
bigger or smaller. These are typically classified as bugs and change
with software versions. :-)


--Joe


On Oct 25, 2010, at 9:31 AM, Mark Boolootian wrote:

> 
> I received a question from one of the folks here on campus on
> selection of MTU size.  Does anyone have any guidance/empirical
> data that addresses this?
> 
> ---
> 
>  Assuming your gear supports it, have you ever read anything that
>  suggests some jumbo frame sizes are better for hardware than others? In
>  a high performance arena?
> 
>  Aka, does 8192 (8K) make chipsets go faster than a non-byte aligned
>  jumbo frame? (9000) Particularly if your application takes full
>  advantage of the larger packet sizes?
> 
>  I know 9000 is "standard" but why is it standard? Its an odd number in a
>  "power of 2" world so I must be missing something.
> 
>  Does a chipset work harder moving zillions of packets that are 9000
>  bytes compared to 8192 bytes?
> 

Joe Metzger