Transport protocols and bulk file transfer

[stanislav shalunov <>]

                     Bulk Transport Protocol Call
                       Friday, February 4, 2005

Attendees:
Stanislav Shalunov 
Guy Almes
Matt Zekauskas
Susan Evett (scribe)

Call started at 12:01 p.m. EST. 

Agenda
1. Review of previous action items

2. Discussion with Guy Almes, NSF

1. Review of Previous Action Items 

a. Stanislav to submit a request for a track session at the upcoming
   Internet2 spring member meeting by 1/28 (done).

b. Stanislav to develop a charter and side materials to request
   Internet2 working group status.

c. Susan and Stanislav to pursue a side session at the Internet2
   spring member meeting for the Transport group. (session requested)

d. Stanislav: incorporate Lisong's TCP Westwood description.

e. Stanislav: address Bartek's comments about MaxNet and XCP. 

f. Stanislav: draft conclusions section for design space survey.

g. All: look at the requirements section of the document to ensure it
   captures the intent of the group.  Send comments to the mailing
   list.  (A simple ``I think this is what we want to build'' is OK,
   too.)

h. Steven: contact the authors of the SCTP RFC to discuss models of
   partial reliability other than timed reliability.

i. Steven: write up API specification.

2. Discussion with Guy Almes, NSF

Guy reviewed the draft paper and shared comments; he felt the paper
switches, in the middle, from analytical (transport issues) to
practical applications issues (see section 2.8 -- adding a
user interface).  Guy felt that socket-layer protocols do not, in
general, have GUIs.  Stanislav noted that, when TCP was first
introduced, the two programs to use it were the de facto
user-interface.  Guy agreed but said that this should be clarified in
the paper -- what is the purpose of the application?  Stanislav talked
about the need for e-VLBI and ImmSeg applications to use this
very-large-very-fast protocol.

Section 2.8.

Question 1: provide/not provide a user-interface? This is yet to be
determined.  Might be useful for some of the tools, but not a GUI, most
likely.

Question 2: explicit vs. implicit - Guy felt it would be interesting,
statistically, to measure the correlation between congestion in one
direction with congestion in the other (probably higher than a priori
congestion but not 'known').  Guy agreed that the paper's approach to
XCP is correct but felt that SourceQuench has some advantages that are
not as highly valued as he feels they could be.  He is not arguing FOR
it but wants it to be, possibly, more reviewed.  Guy asked if it was
the case that every explicit approach requires router reconfiguration?
Stanislav said that is what makes it 'explicit'; Guy argued that
removing the buffer requirement would make routers cheaper but adding
complications to the routing instructions might make it more
expensive.  The final approach should not make routers more expensive
or complicated to deploy.  Guy asked if the group had attempted to
identify which of the approaches require ALL routers in the path to be
modified and which ones require SOME routers to be modified.  He
suggested bringing this item out; which of the approaches (such as XCP
and modified-XCP) would benefit by having a specific router
(preferably the bottleneck router) to be modified.  Stanislav said
that determining which routers support an approach is fairly easy.

Section 2.1.

Question 3: routers vs. switches - If you are doing explicit, why is
it important for all the routers/switches, preferably including the
bottleneck link, to support the approach.

Question 4: Buffers - switches with large buffers cost a lot of money.
Matt asked what about switches with Gig and Mb ports?  Stanislav felt
there usually was shared memory.  Guy noted that, by definition, a
switch with ports that are an order of magnitude off, are at the head
of a bottleneck link.

Section 2.2

Question 5: Capacity / # of flows - is it practical for the router to
know how many flows?  Stanislav felt that is not a big deal (routers
can know how many flows they serve); but really this is a matter of
provisioning -- this has to be estimated BEFORE you design the router,
not every RTT.  Guy commented that we're living in a world where all
the router designers feel they need to install large buffers in all
H-S links; Stanislav said could be changing -- the designers are
waiting for more data but they are paying close attention to it.  Guy
asked if they are changing their buffer and memory size based on this
paper?  Stanislav felt that, at this point, second-tier manufacturers
might use the paper to justify already small buffers.

Section 2.2.2.

Question 6: Guy can't understand the problem statement. Consider re-phrasing.

Section 2.2.3.

Question 7: Using combined distributions - packet trains, etc.  Guy
felt this section would be a place for expansion of the paper (if a
longer paper were required) -- there is a lot of material that could
be included here.  Guy noted that section 2.2.1 was loss-based, 2.2.2
being delay-based, and 2.2.3 as being packet train inter-arrival
based.

Section 2.2.

Question 8: Stanislav asked what Guy's feeling was on doing pure
delay-based congestion control that notices when it is running against
loss-based; when you are at a bottleneck router running against a
loss-based flow, you'd have to switch to loss-based or you'd never get
anywhere.  Guy notes that you have to do something other than what
pure-Reno based TCP does on restart.  He feels that this is the best
conclusion he can think of; he suggested thinking more about the
packet-train idea (trust in the host applications instead of the
routers or kernels).  Might require software modifications to the
kernel to allow user applications to run, for example.  Driver and NIC
layers -- might want them to do some things differently.  Readable TSC
counter is visible to user-space; might want to put the congestion
control up in user-space, which would require a kernel modification.
Stanislav argued that, if he has to get into kernel modification, it
would break ease of installation and use requirement.

Section 2.3.

Guy thinks that Bob Grossman's group has shed some light on the issues
in 2.3 so they should get some credit.  Add citation to their work
(SABUL).

Section 2.4.

Guy noted that section 2.4 never mentioned UDP after the reference to
user-space approach.  Guy suggested adding a small paragraph on the
possibility of allowing protocols to set congestion control in user
space - this is not currently possible in TCP.  (This would have to
specify that this topics is out of the scope of this paper and drop
it.)

Section 2.4.?

Question 8: From a perf pov, is it ever wise to send a packet that is
larger than the MTU?  Stanislav mentioned the famous NFS example.  Guy
recalled an example at the FMM99 in Seattle where Dave Richardson's
demos sent packets that intentionally would be fragmented and
reassembled.  Guy mentioned that this is a design decision -- it is
almost always a bad idea but in some examples, it is useful to
particular applications.

Section 2.5

Guy feels that network performance suffers sometimes because a burst
of packets hits a switch, some are dropped, and although there is not
any real 'congestion' per se, pacing / rate-based approach is worth
considering. Rate-based and pacing may be different issues but they
should be explored, especially pacing.

Multi-core CPU situation should permit OS-designs that would,
historically, be a bad idea.  Such as: designing OS to support
busy-wait processes, wait times, fine-grained sleep or timing
services.

Section 2.7

Computers don't come with unlimited memory -- you can open a number of
TCP connections but your computer will not crash/violate the spec;
instead, the performance will continue to degrade.  Guy asked if
Stanislav was contemplating protocols / file transfers that do not
have that property.  Big memory of a TCP approach is the ability to
resend any packet needed.  Guy suggested intelligent API design -- or
user-space area.  Leave it to the core to do caching; in the RAM
address space -- I have a pointer to it (in virtual memory) and it
won't go away until I get acknowledgment.

Guy noted that the paper doesn't address the new approach: Improve TCP
performance by opening up several streams for concurrent transmission.
The multi-stream idea is horrible when applied to Reno (because of the
self-synchronization phenomenon) - one can conceive of a transport
technology that does NOT self-synchronize. Consider the best Section
2.2.2 approach and then combine in multi-stream.

ACTION ITEMS:

a. Stanislav to develop a charter and side materials to request
   Internet2 working group status.

b. Stanislav: incorporate Lisong's TCP Westwood description.

c. Stanislav: address Bartek's comments about MaxNet and XCP.

d. Stanislav: draft conclusions section for design space survey.

e. All: look at the requirements section of the document to ensure it
   captures the intent of the group.  Send comments to the mailing
   list.  (A simple ``I think this is what we want to build'' is OK,
   too.)

f. Steven: contact the authors of the SCTP RFC to discuss models of
   partial reliability other than timed reliability.

g. Steven: write up API specification.

h. Stanislav to modify the draft based on discussion items with Guy
   Almes.

Next call is at the same time on 2004-02-11.
Call ended at 1:54 pm EST.