Transport protocols and bulk file transfer

[stanislav shalunov <>]

        Transport Project Discussion Conference Call
        2004-10-29
        3:00-4:00PM US/Eastern time

Participants:

                Bob Grossman - University of Illinois-Chicago
                Xinwei Hong - University of Illinois-Chicago
                David Hanley - University of Illinois-Chicago
                David Lapsley - MIT Haystack Observatory
                Shawn McKee - University of Michigan
                Steven Senger - University of Wisconsin at La Crosse
                Injong Rhee - University of North Carolina
                Martin Swany - University of Delaware
                Stanislav Shalunov - Internet2

Scribe:         Terri Saarinen - Internet2

Agenda:

                Introductions - 5minutes
                Agenda Bashing - 5 minutes
                Tentative Work Plan - 5 minutes
                Prospective Applications - 15 minutes
                Mode of Participation - 15 minutes
                Problem Factorization - 15 minutes
                Action Item Summary - 2 minutes

(Discussion prior to start of call and clarification that this is an
open source effort.)

Introductions:

Stanislav Shalunov: Internet2; trying to work with community to turn
the excellent research in transport protocols into a user tool to
solve actual problems.

David Lapsley: Haystack; user of tool for eVLBI data transport; easy
to use, easy to install and reliable.

Bob Grossman: University of Illinois at Chicago; interested in working
together.  UDT source forge just released by Yunhong Gu.  Heading off
to teach a class, but David Hanley and Xinwei Hong will remain on the
call.  Want to participate in this effort.

Steven Senger: University of Wisconsin at La Crosse; collaborating
with Parvati Dev and has written a couple of UDP-based protocols and
would like to both cooperate and learn from this project.

Shawn McKee: University of Michigan; physicist, interested in moving
large amounts of data, would like to come up with workable solutions.
Stas asked what tools Shawn uses for large scale transfer?  Shawn:
GridFTP and Iperf for testing.  Storage resource brokers and managers,
Grid FTP good, people open multiple streams, also Grid NFS funded by
NMI.

(Procedural note: Stas asked participants to identify themselves
before speaking.)

Martin Swany: University of Delaware; interested, eventually thinking
about moving large data sets, has some ideas and wants to think about
and hear what others are doing.  Interest is more on the researcher
side than user side.

Injong Rhee: North Carolina State University; worked on BIC-TCP
protocol, interested in design for high speed, long distance network.
He wants to contribute ideas and code.

Agenda Comments?  (none)

Prospective Applications:

Killer application moving file from one point to another.

Important to understand application before hand so APIs are designed
right.

What are the high level applications other than file transfer that
bulk transfer would support?

David: intensive data rates, say 4096 megabits/sec in near future.
Real-time streams, timing, synchronization, the application can
tolerate 5% loss and still produce meaningful results.

Injong asked if David was seeing losses of 5-10%?  Sometimes.

David L. mentioned fiber links at 1gig/sec.

Reliability and not high bandwidth?

David L. said reliability or high bandwidth depends on the
application.  It it▓s a real time experiment, then there is a concern
for high bandwidth flow and still get good results.  He gave further
explanation about telescopes generating data at a high rate.  Aime for
real time application is to keep correlator fed at constant rate.  Is
link shared?  Currently a shared IP - Abilene to get access to
telescope.  May have dedicated links in the future.

Stas said that the tool could contain congestion control algorithm,
descendent rate, correlator consumption, etc, what should happen?

David L. talked about two data streams 512 megabit/sec into the
correlator.  If 90% available bandwidth, then congestion control
algorithm knows what is available and discards 10% of data stream.
Filler frames to indicate loss, but still able to process the request.

Stas said there needs to be a way to tell tool to not try to deliver
all data but take data at a fixed rate and deliver what can be
delivered.  Tool would be given minimum requirements, and if falls
below, then transmission would not be useful and would yield link to
others.

David L. agreed.  Or specify nominal rate, and tolerate 10% loss =
threshold, and only take action if 10% loss.  10% loss within a fair
rate.  If in common rate is greater than fair rate, then 10% of data
could be discarded.  What about bandwidth condition control?  If less
than 90%, then you have to take other action.  Shut down, save til
later, or store.  TFRC (TCP Friendly Rate Control) or TCP
(Transmission Control Protocol)?  TFRC would be nice.  David L.:
application can buffer up to about 5 seconds worth of data.

Stas noted a 10Gb/s TCP connection with RTT=100ms and MTU=1500B would
have oscillation period of 90 minutes.  To completely smooth that out,
quite a bit more than a 5 sec buffer is needed.

Other Applications?

Steve: highly interaction, variable data rates, implemented his own
transport to make it work.  Can tolerate modest loss sometimes, but
other times he needs to recover it all.  Need to maintain activity
between client rate and server.  He doesn▓t care about underlying
protocol.  What if rate is lower than you want to send?  Steve would
want information out of API and would change the amount of demand
(codec, or resolution) at user lever.  Stas asked if data transmits
and returns current desired rate it wants to be fed?  Yes, Steve could
deal with that.  Interactive applications, spikey traffic, but not
bulk transfer.  He wants to provide feedback to the user; what they
can expect out of the application.

Shawn: Mostly disk-to-disk bulk file transfer.  Open files across
network of the future.  Depends on network infrastructure, virtual
circuits, need to utilize infrastructure for access to data source.
Video conferences.  Monitoring applications agent-based.  Bulk moved
is in file storage to storage over or to CPU to do application. Disk
to processors.  Depends if pre-staging data.  Output streams, but not
where the processors are.

Other requirements for API?

Steve: a couple of apps have multiple individual streams transported.
Congestion control is important.  Do streams sum to more than the size
of the pipe?  Steve said that if end-point pipe size is unknown, then
fighting self is a possibility.

Stas sees work proceeding by first working on a prototype tool by
first quarter or March 2005, testing ideas in congestion control
algorithm to prove ideas can work or if changes are necessary.  And
then re-doing what▓s necessary and releasing as a production tool.
Agreement that this is the right track.

Mode of Participation: What would you contribute?

Shawn: testing role, some node access, some places to store large
transfer at end of test.

David L.: testing resources, large data nodes, ideas and advice.
Code? Some existing, free to look it over, but few new coding
resources available.

Stas: wants outcome to be open source and usable by anyone interested.
Internet2 likes BSD license.  Agreement that this is the appropriate
way to go.  No one is saying ``why not GPL?''

Injong: can contribute ideas and coding - some done by himself, some
done by grad students.

Injong asked about the availability of grant funding for this effort.
Stas explained that so far, the intent is to get this project moving
forward.  Once credible and a work in progress, then grant funding
will be sought as tool applies to complete community of high-end
network users.  **Let Stas know if your support needs external
funding.  Injong agrees with approach to work unfunded at the
beginning, then apply for funding to continue and finish effort.

Martin: can contribute self and a part of a grad student for
short-term, applying for funding to continue and finish effort.

David H.: has code base that does a large amount of what▓s needed and
he▓ll give it to the project.  Open source GPL (GNU public license) -
can probably release it.

Stas said GPL may be difficult since Internet2 is a consortium with
commercial companies as partners.  Internet2 intellectual property
framework says BSD.  **David H. will investigate.  
()
**Stas will add David to the mail group, others interested may also
self subscribe.

Steve: would like to contribute however he can.  Testing, machine
access at Stanford and Michigan, contributing code if no licensing
problems.

Problem Factorization:  Different dimensions of design space
independent of each other.

1.      Congestion Control
2.      Kernel space vs. User Space
3.      Protocol and Framing
4.      Window based vs. Rate based
5.      Explicit Congestion Feedback vs. Implementation
6.      Reno-friendly vs. better proforming
7.      State mostly kept at the sender vs. the receiver

Thoughts?

David: great idea to capture and document.  May turn into an Internet
draft down the road.

Will any factorization be a problem?  Loss or delay, do you care?

David L.: yes, there are user considerations.  VLBI runs mixed use for
production transfer.  Maintenance and installation becomes an issue,
especially on kernel if it is remote.

Shawn: wants something that works.  If it▓s difficult to get all
required framework to run, then it▓s harder to get adoption.  Ease of
use is important.  Customized kernel makes it tough.  If it works, he
does not really care about other details.

Are there other design choices that need to be spelled out?  Loss
based congestion control algorithm. The rest is explicitly mentioned.
Stas thinks it would be good to have a design paper.

Xinwei asked if transportation instrument needs to be on the kernel
side?  File system interface.  David L. said that ideally if it▓s the
same interface, then there is no change to existing software. He
agrees with others value in layering out design space and
understanding options.

Action Item Review:

    * Stas will review and edit minutes and then will post them to the list.

    * Stas will discuss future call agenda.

    * Stas will begin outline for design document.

    * David H. will investigation possible licensing conflicts and
      conditions for UDT.

    * All to keep eyes open for relevant funding opportunities.

Stas thanked folks for joining the call and agreeing to participate.
Call ended at 3:59PM.

-- 
Stanislav Shalunov              http://www.internet2.edu/~shalunov/

"Nuclear war would really set back cable [television]."  -- Ted Turner