Transport protocols and bulk file transfer

[stanislav shalunov <>]

Brian Tierney 
<>
 writes:

> The only thing I disagree with is the statement on CPU requirements
> of UDP vs TCP.  I've heard from Bill Allcock of ANL that using UDT
> inside of GridFTP instead of TCP used 2x the CPU (maybe this is due
> to the UDT implementation, not UDP).  I've done a bit of testing
> that shows that UDP CPU is higher too.  I think you really only see
> the difference at speeds great than 500 Mbps.

Brian,

Thank you for pointing that out.  In fact, we had a discussion that
touched on this before, and somehow I didn't incorporate the reasoning
in the document.  The sentence ``In fact, there seems to be little
reason to believe today that kernel implementation has inherent
efficiency advantages.'' is now removed.  In the discussion subsection
of the section where user- vs kernel-space approaches are discussed,
the following paragraph is added:

        A kernel-space implementation has an inherent efficiency
        advantage over user-space implementations: with a user-space
        approach, each packet\footnote{This is only true for packets
        that are not fragmented.  See section~\ref{sec-pmtud} for
        discussion of fragmentation.} requires a pair of context
        switches (in this case, transitions between user and kernel
        mode and \textit{vice versa}); with a kernel-space approach,
        the number of context switches per byte can be reduced, since
        the I/O block size can be made larger than the packet size.
        The increase of CPU use with user-space tools might or might
        not matter depending on its size.  It can be difficult to
        compare CPU cycles use by different programs, as
        implementation efficiency and degree of optimization might not
        be identical.  For the purposes of this design exploration,
        it, thus, appears reasonable to try to estimate the loss of
        efficiency with user-space tools.  Assume that a pair of
        context switches takes 4\,$\mu$s; this is a reasonable number
        for a CPU on a computer than might be equipped with a 1-Gb/s
        network interface.  Assume that the MTU is 1500\,B.  For the
        link to become fully saturated, about $86,000$ packets/second
        are needed, so the overhead associated with context switches
        of a user-space tool might account for approximately 35\% of
        the available CPU capacity.  While not, by any means,
        insignificant, this amount of CPU use might be appropriate for
        some applications, especially if the CPU would have been
        partially idle otherwise.

In addition, an item ``can use less CPU'' is added to the summary for
a kernel-space approach.

Does this seem reasonable?  I'm talking about 1Gb/s because, for
10Gb/s, a faster CPU would be used.  I lack access to a machine on
which to measure context switch time for these higher-end CPUs.

> But despite the higher CPU, I think a user-space, UDP based solution
> is  the way to go.

Yep.  At least while exploring the solutions.


-- 
Stanislav Shalunov              http://www.internet2.edu/~shalunov/

This message is designed to be viewed in boustrophedon.