NDT-DEV email list created

[Tiziana Refice <>]

On Fri, Apr 4, 2014 at 12:52 PM, Sebastian Kostuch <> wrote:

Hi
I have performed upload/download tests on clients available in repo (connecting to localhost server).
They all were run under same conditions (Ubuntu 12.04 Kernel 3.9.1 patched with Web10G-0.9-patch) and you
can see results in attached file. I will list here only average results (upload[Mb/s]/download[Mb/s])

C                                    Java     Flash (version 12.0.0.77)
17311.23/16152.68       16631.00/13752.13        417.31/111.32

I also noticed that in results I had very small PC buffer size (around 250KB) when compared to
Java one (around 3MB) which I think limits these speeds.
So I suppose that the goal is to try to determine whether it is flash technology restriction or maybe code issue.
If these speeds will remain far lower than Java/C ones then would it make sense to use current flash
client instead of java applet? Or should it be dropped for some time when maybe some newer version will
improve these results?

Any idea if multi-thread Flash would help here?

Regards
Sebastian Kostuch

On 27.03.2014 23:02, Will Hawkins wrote:

I have submitted a patch to NDT to close the listening sockets as soon
as the client connects (for both s2c and c2s tests).

I deployed code with that fix to one of the testing servers and
confirmed that it does what is expected. It does *not* change the
results. In other words, the URLLoader in Flash does not appear to be
attempting to either a) pipeline or b) use multiple threads to fetch
content over HTTP. This is what we expected, but it's good to confirm.

I will let you know when the code is merged.

The next step is to guarantee that there is no local caching being done
by the URLLoader. I doubt there is, but we should be sure just in case
(Thanks to Matt for suggesting that we test this!).

Will

On 03/21/2014 04:56 PM, Matt Mathis wrote:

Sounds reasonable.  I wonder if this approach holds up if the client
attempts to use multiple threads?  Could you tell if there was a second
connection attempted, but it was hung?

 From your description, I bet the ndt server does not close the listen
port once it has accept()'d the data connection.   It probably should.
  (The the second request would get an immediate error rather than an
indeterminant wait).

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our
services to speak in defiance of unjust governments.   We treat privacy
and security as matters of life and death, because for some users, they are.


On Fri, Mar 21, 2014 at 1:43 PM, Will Hawkins
< <mailto:>>
wrote:



     On 03/21/2014 03:20 PM, Will Hawkins wrote:
     > Excellent question. My hunch is that it will do this single-threaded
     > (although asynchronously). I am going to check in to this and I
     will let
     > you know asap.
     >
     > Will
     >
     > On 03/21/2014 03:05 PM, Matt Mathis wrote:
     >> Did you check to see how many concurrent connections it used?
     The risk
     >> with this approach is that it is entirely legal for the URL loading
     >> machinery to use multiple concurrent connections.
     >>

     Although there does not seem to be a way to determine if the runtime is
     using concurrent connections, I think that we can infer that it is NOT.

     The server allows for exactly one connection for each of the ephemeral
     ports that it uses for a particular test with a particular client. I
     just ran a S2C test and, concurrent with the test, attempted to connect
     to the same port (using nc). Although the connection appeared to have
     succeeded, it was not able to "receive" any of the test data.

     It would seem, then, that even if the URLLoader was attempting to make
     concurrent connections, it would not make the test results any better.
     Would you agree with that?

     Will
     >> Thanks,
     >> --MM--
     >> The best way to predict the future is to create it.  - Alan Kay
     >>
     >> Privacy matters!  We know from recent events that people are
     using our
     >> services to speak in defiance of unjust governments.   We treat
     privacy
     >> and security as matters of life and death, because for some
     users, they are.
     >>
     >>
     >> On Fri, Mar 21, 2014 at 11:48 AM, Will Hawkins
     >> <
     <mailto:>
     <mailto:
     <mailto:>>>
     >> wrote:
     >>
     >>     Hello there!
     >>
     >>     I will send you the full results of the speed testing that
     we've done as
     >>     a separate email. We too have noticed that the Flash client
     does not
     >>     seem to be as accurate as data rates increase.
     >>
     >>     However, I've made some progress today and thought I'd use this
     >>     opportunity to share:
     >>
     >>     First, nothing about the upload test has changed. I've left
     that alone.
     >>     This is all about the download test (S2C).
     >>
     >>     I changed the NDT server to spit out a valid, minimal set of HTTP
     >>     response headers before it starts sending random data. Then,
     on the
     >>     Flash client side, I've treated that data like an HTTP
     request and used
     >>     a built-in URL loading class. I thought this might get around
     some of
     >>     the event-driven I/O latency that seems to be a problem (see
     follow-up
     >>     email).
     >>
     >>     Indeed there is some progress. From my desk to the testing
     server at
     >>     nuq, I was able to get about 42 mbps download speeds. This is
     >>     approximately 12 mbps higher than I was able to get with the
     >>     socket-based version.
     >>
     >>     More importantly, I got similar results between the Flash
     client and the
     >>     Java client. Both report around 42 mbps download and around
     80% receiver
     >>     limited. They are eerily similar, actually. I will keep
     testing to see
     >>     what I find but I wanted to send this out.
     >>
     >>     Will
     >>
     >>     On 03/21/2014 02:35 PM, Matt Mathis wrote:
     >>     > Circumstantial evidence suggests that the flash runtime event
     >>     processing
     >>     > may be synchronized to the display refresh rate....  It
     would be
     >>     > interesting to know it changing the display settings
     changed the
     >>     maximum
     >>     > performance.
     >>     >
     >>     > Thanks,
     >>     > --MM--
     >>     > The best way to predict the future is to create it.  - Alan Kay
     >>     >
     >>     > Privacy matters!  We know from recent events that people
     are using our
     >>     > services to speak in defiance of unjust governments.   We treat
     >>     privacy
     >>     > and security as matters of life and death, because for some
     users,
     >>     they are.
     >>     >
     >>     >
     >>     > On Fri, Mar 21, 2014 at 11:19 AM, Tiziana Refice
     >>     < <mailto:>
     <mailto: <mailto:>>
     >>     > <mailto: <mailto:>
     <mailto: <mailto:>>>> wrote:
     >>     >
     >>     >     Will and Matt have been doing some experiments about that
     >>     recently.
     >>     >
     >>     >     Will, could you share the results with Sebastian?
     >>     >
     >>     >
     >>     >     On Fri, Mar 21, 2014 at 10:17 AM, Sebastian Kostuch
     >>     >     < <mailto:>
     <mailto: <mailto:>>
     >>     <mailto:
     <mailto:> <mailto:
     <mailto:>>>> wrote:
     >>     >
     >>     >         I think that when connecting to localhost speed
     should be much
     >>     >         more higher (unfortunately I am not able to achieve
     better
     >>     >         results connecting to another server). I have just
     noticed
     >>     that
     >>     >         when using C/Java client results are in fact way better
     >>     >         (around 14Gb/s for upload and almost 9Gb/s
     download). So it
     >>     >         seems like flash client is somehow restricted
     (tested on
     >>     >         different flash version and results are the same).
     Could also
     >>     >         someone confirm that on higher speeds flash client
     doesn't
     >>     >         give similar speeds compared to Java/C ones?
     >>     >
     >>     >         Regards,
     >>     >         SK
     >>     >
     >>     >
     >>     >         On 21.03.2014 10 <tel:21.03.2014%2010>
     <tel:21.03.2014%2010>
     >>     <tel:21.03.2014%2010>:51, Tiziana Refice wrote:
     >>     >>         Ideally, it would be better to test this change at
     much
     >>     higher
     >>     >>         speed (i.e., as close to 1Gbps as possible).
     >>     >>         Would it be possible for you to make such test?
     >>     >>
     >>     >>
     >>     >>         On Wed, Mar 19, 2014 at 12:01 PM, Sebastian Kostuch
     >>     >>         <
     <mailto:> <mailto:
     <mailto:>>
     >>     <mailto:
     <mailto:> <mailto:
     <mailto:>>>> wrote:
     >>     >>
     >>     >>             Hi Tiziana,
     >>     >>             I have performed some simple test which was
     executing
     >>     >>             flash client 10 times on my changes (with
     timer set
     >>     to 1s)
     >>     >>             and on newest trunk version of client. Both
     were made
     >>     >>             toward localhost NDT server (newest web10g) to
     >>     achieve big
     >>     >>             speed. Here are results (download/upload in Mb/s):
     >>     >>
     >>     >>             *trunk version*           *version from
     Issue131 branch*
     >>     >>             110.9/439.0                128.1/486.0
     >>     >>             121.8/456.8                122.2/471.9
     >>     >>             104.1/471.9                123.6/452.7
     >>     >>             120.5/478.7                126.4/451.0
     >>     >>             123.2/515.1                113.3/468.9
     >>     >>             120.8/462.7                130.8/476.8
     >>     >>             121.0/433.3                116.7/442.5
     >>     >>             109.1/473.7                115.6/442.8
     >>     >>             128.0/469.3                112.5/467.6
     >>     >>             118.8/469.0                126.5/496.2
     >>     >>
     >>     >>             average:
     >>     >>             117.82/466.95            121.57/465.64
     >>     >>
     >>     >>             These results are very similar to each other
     so it seems
     >>     >>             that these new events won't impact accuracy of
     test.
     >>     Is it
     >>     >>             valuable proof for you?
     >>     >>
     >>     >>             Regards,
     >>     >>             Sebastian Kostuch
     >>     >>
     >>     >>
     >>     >>             On 17.03.2014 11:40, Tiziana Refice wrote:
     >>     >>>             I am concerned this change (which adds new
     events that
     >>     >>>             will fire during the execution of the test)
     will affect
     >>     >>>             the accuracy of the test.
     >>     >>>             I would prefer to have test data that show
     that this is
     >>     >>>             not the case, especially for very high speeds
     (e.g.,
     >>     >>>             900+Mbps). See last email by Matt Mathis to
     ndt-dev on
     >>     >>>             this topic for details on the issue.
     >>     >>>
     >>     >>>
     >>     >>>
     >>     >>>             On Wed, Mar 12, 2014 at 12:17 PM,
     >>     < <mailto:>
     <mailto: <mailto:>>
     >>     >>>             <mailto:
     <mailto:>
     >>     <mailto: <mailto:>>>> wrote:
     >>     >>>
     >>     >>>                 Revision: 1012
     >>     >>>                 Author:  
     <mailto:>
     >>     <mailto: <mailto:>>
     >>     >>>                 <mailto:
     <mailto:>
     >>     <mailto: <mailto:>>>
     >>     >>>                 Date:     Wed Mar 12 12:17:39 2014 UTC
     >>     >>>                 Log:      Download/upload speed in flash
     client is
     >>     >>>                 now being updated during test duration
     >>     >>>
     >>     >>>
     http://code.google.com/p/ndt/source/detail?r=1012
     >>     >>>
     >>     >>>                 Modified:
     >>     >>>
      /branches/Issue131/flash-client/src/TestC2S.as
     >>     >>>
      /branches/Issue131/flash-client/src/TestS2C.as
     >>     >>>
     >>     >>>                 =======================================
     >>     >>>                 ---
     /branches/Issue131/flash-client/src/TestC2S.as
     >>     >>>                    Fri Feb 21 07:41:20 2014 UTC
     >>     >>>                 +++
     /branches/Issue131/flash-client/src/TestC2S.as
     >>     >>>                    Wed Mar 12 12:17:39 2014 UTC
     >>     >>>                 @@ -30,6 +30,8 @@
     >>     >>>                     * This class performs the
     Client-to-Server
     >>     >>>                 throughput test.
     >>     >>>                     */
     >>     >>>                    public class TestC2S {
     >>     >>>                 +    private const SPEED_UPDATE_TIMER:int =
     >>     1000; // ms
     >>     >>>                 +
     >>     >>>                      // Valid values for _testStage.
     >>     >>>                      private static const
     PREPARE_TEST1:int = 0;
     >>     >>>                      private static const
     PREPARE_TEST2:int = 1;
     >>     >>>                 @@ -45,12 +47,14 @@
     >>     >>>                      private var _c2sTestSuccess:Boolean;
     >>     >>>                      // Time to send data to server on
     the C2S
     >>     socket.
     >>     >>>                      private var _c2sTestDuration:Number;
     >>     >>>                 +    private var _c2sTestStartTime:Number;
     >>     >>>                      private var _ctlSocket:Socket;
     >>     >>>                      private var _c2sSocket:Socket;
     >>     >>>                      private var _c2sSendCount:int;
     >>     >>>                      // Bytes not sent from last send
     operation on
     >>     >>>                 the C2S socket.
     >>     >>>                      private var _c2sBytesNotSent:int;
     >>     >>>                      private var _c2sTimer:Timer;
     >>     >>>                 +    private var _speedUpdateTimer:Timer;
     >>     >>>                      private var _dataToSend:ByteArray;
     >>     >>>                      private var _msg:Message;
     >>     >>>                      private var _serverHostname:String;
     >>     >>>                 @@ -64,6 +68,7 @@
     >>     >>>
     >>     >>>                        _c2sTestSuccess = true;  //
     Initially the
     >>     test
     >>     >>>                 has not failed.
     >>     >>>                        _c2sTestDuration = 0;
     >>     >>>                 +      _c2sTestStartTime = 0;
     >>     >>>                        _dataToSend = new ByteArray();
     >>     >>>                        _c2sSendCount = 0;
     >>     >>>                        _c2sBytesNotSent = 0;
     >>     >>>                 @@ -175,6 +180,8 @@
     >>     >>>
     >>     >>>                        _c2sTimer = new
     >>     Timer(NDTConstants.C2S_DURATION);
     >>     >>>
      _c2sTimer.addEventListener(TimerEvent.TIMER,
     >>     >>>                 onC2STimeout);
     >>     >>>                 +      _speedUpdateTimer = new
     >>     Timer(SPEED_UPDATE_TIMER);
     >>     >>>                 +
     >>     >>>
     >>      _speedUpdateTimer.addEventListener(TimerEvent.TIMER,
     >>     >>>                 onSpeedUpdate);
     >>     >>>                        _msg = new Message();
     >>     >>>                        _testStage = START_TEST;
     >>     >>>                        TestResults.appendDebugMsg("C2S test:
     >>     >>>                 START_TEST stage.");
     >>     >>>                 @@ -244,6 +251,16 @@
     >>     >>>
      TestResults.appendDebugMsg("Timeout for
     >>     >>>                 sending data on C2S socket.");
     >>     >>>                        closeC2SSocket();
     >>     >>>                      }
     >>     >>>                 +
     >>     >>>                 +    private function
     >>     onSpeedUpdate(e:TimerEvent):void {
     >>     >>>                 +      _c2sTestDuration = getTimer() -
     >>     _c2sTestStartTime;
     >>     >>>                 +      _c2sBytesNotSent =
     _c2sSocket.bytesPending;
     >>     >>>                 +      var c2sByteSent:Number = (
     >>     >>>                 +          _c2sSendCount *
     >>     >>>                 NDTConstants.PREDEFINED_BUFFER_SIZE
     >>     >>>                 +          +
     (NDTConstants.PREDEFINED_BUFFER_SIZE -
     >>     >>>                 _c2sBytesNotSent));
     >>     >>>                 +
     >>     >>>                 +
      TestResults.ndt_test_results::c2sSpeed =
     >>     >>>                 (c2sByteSent * NDTConstants.BYTES2BITS);
     >>     >>>                 +    }
     >>     >>>
     >>     >>>                      private function startTest():void {
     >>     >>>                        if (!_msg.readHeader(_ctlSocket))
     >>     >>>                 @@ -267,9 +284,10 @@
     >>     >>>
      removeCtlSocketOnReceivedDataListener();
     >>     >>>
     >>     >>>                        _c2sTimer.start();
     >>     >>>                 +      _speedUpdateTimer.start();
     >>     >>>                        // Record start time right before
     it starts
     >>     >>>                 sending data, to be as
     >>     >>>                        // accurate as possible.
     >>     >>>                 -      _c2sTestDuration = getTimer();
     >>     >>>                 +      _c2sTestStartTime = getTimer();
     >>     >>>
     >>     >>>                        _testStage = SEND_DATA;
     >>     >>>                        TestResults.appendDebugMsg("C2S test:
     >>     >>>                 SEND_DATA stage.");
     >>     >>>                 @@ -288,9 +306,11 @@
     >>     >>>                      private function closeC2SSocket():void {
     >>     >>>                        // Record end time right after it
     stops
     >>     >>>                 sending data, to be as accurate as
     >>     >>>                        // possible.
     >>     >>>                 -      _c2sTestDuration = getTimer() -
     >>     _c2sTestDuration;
     >>     >>>                 +      _c2sTestDuration = getTimer() -
     >>     _c2sTestStartTime;
     >>     >>>                        TestResults.appendDebugMsg(
     >>     >>>                            "C2S test lasted " +
     _c2sTestDuration + "
     >>     >>>                 msec.");
     >>     >>>                 +      _speedUpdateTimer.stop();
     >>     >>>                 +
     >>     >>>
     >>      _speedUpdateTimer.removeEventListener(TimerEvent.TIMER,
     >>     >>>                 onSpeedUpdate);
     >>     >>>                        _c2sTimer.stop();
     >>     >>>
     >>     >>>
      _c2sTimer.removeEventListener(TimerEvent.TIMER,
     >>     >>>                 onC2STimeout);
     >>     >>>
     >>     >>>                 =======================================
     >>     >>>                 ---
     /branches/Issue131/flash-client/src/TestS2C.as
     >>     >>>                    Fri Feb 21 07:41:20 2014 UTC
     >>     >>>                 +++
     /branches/Issue131/flash-client/src/TestS2C.as
     >>     >>>                    Wed Mar 12 12:17:39 2014 UTC
     >>     >>>                 @@ -31,6 +31,7 @@
     >>     >>>                    public class TestS2C {
     >>     >>>                      // Timer for single read operation.
     >>     >>>                      private const READ_TIMEOUT:int =
     15000; //
     >>     15sec
     >>     >>>                 +    private const SPEED_UPDATE_TIMER:int =
     >>     1000; // ms
     >>     >>>
     >>     >>>                      // Valid values for _testStage.
     >>     >>>                      private static const
     PREPARE_TEST1:int = 0;
     >>     >>>                 @@ -48,17 +49,18 @@
     >>     >>>                      private var _ctlSocket:Socket;
     >>     >>>                      private var _msg:Message;
     >>     >>>                      private var _readTimer:Timer;
     >>     >>>                 +    private var _speedUpdateTimer:Timer;
     >>     >>>                      private var _s2cByteCount:int;
     >>     >>>                      private var _s2cSocket:Socket;
     >>     >>>                      private var _s2cTimer:Timer;
     >>     >>>                      // Time to send data to client on
     the S2C
     >>     socket.
     >>     >>>                      private var _s2cTestDuration:Number;
     >>     >>>                 +    private var _s2cTestStartTime:Number;
     >>     >>>                      private var _s2cTestSuccess:Boolean;
     >>     >>>                      private var _serverHostname:String;
     >>     >>>                      private var _testStage:int;
     >>     >>>                      private var _web100VarResult:String;
     >>     >>>
     >>     >>>                 -
     >>     >>>                      public function
     TestS2C(ctlSocket:Socket,
     >>     >>>                 serverHostname:String,
     >>     >>>
     >>      callerObj:NDTPController) {
     >>     >>>                        _callerObj = callerObj;
     >>     >>>                 @@ -67,6 +69,7 @@
     >>     >>>
     >>     >>>                        _s2cTestSuccess = true;  //
     Initially the
     >>     test
     >>     >>>                 has not failed.
     >>     >>>                        _s2cTestDuration = 0;
     >>     >>>                 +      _s2cTestStartTime = 0;
     >>     >>>                        _s2cByteCount = 0;
     >>     >>>                        _web100VarResult = "";
     >>     >>>                      }
     >>     >>>                 @@ -172,6 +175,8 @@
     >>     >>>                        }
     >>     >>>                        _readTimer = new Timer(READ_TIMEOUT);
     >>     >>>
      _readTimer.addEventListener(TimerEvent.TIMER,
     >>     >>>                 onS2CTimeout);
     >>     >>>                 +      _speedUpdateTimer = new
     >>     Timer(SPEED_UPDATE_TIMER);
     >>     >>>                 +
     >>     >>>
     >>      _speedUpdateTimer.addEventListener(TimerEvent.TIMER,
     >>     >>>                 onSpeedUpdate);
     >>     >>>                        _s2cTimer = new
     >>     Timer(NDTConstants.S2C_DURATION);
     >>     >>>
      _s2cTimer.addEventListener(TimerEvent.TIMER,
     >>     >>>                 onS2CTimeout);
     >>     >>>                        _msg = new Message();
     >>     >>>                 @@ -234,6 +239,13 @@
     >>     >>>                        _readTimer.start();
     >>     >>>                        receiveData();
     >>     >>>                      }
     >>     >>>                 +
     >>     >>>                 +    private function
     >>     onSpeedUpdate(e:TimerEvent):void {
     >>     >>>                 +      _s2cTestDuration = getTimer() -
     >>     _s2cTestStartTime;
     >>     >>>                 +
      TestResults.ndt_test_results::s2cSpeed =
     >>     >>>                 _s2cByteCount
     >>     >>>                 +
           *
     >>     >>>                 NDTConstants.BYTES2BITS
     >>     >>>                 +
           /
     >>     >>>                 _s2cTestDuration;
     >>     >>>                 +    }
     >>     >>>
     >>     >>>                      private function startTest():void {
     >>     >>>                        if (!_msg.readHeader(_ctlSocket))
     >>     >>>                 @@ -258,9 +270,10 @@
     >>     >>>
     >>     >>>                        _readTimer.start();
     >>     >>>                        _s2cTimer.start();
     >>     >>>                 +      _speedUpdateTimer.start();
     >>     >>>                        // Record start time right before
     it starts
     >>     >>>                 receiving data, to be as
     >>     >>>                        // accurate as possible.
     >>     >>>                 -      _s2cTestDuration = getTimer();
     >>     >>>                 +      _s2cTestStartTime = getTimer();
     >>     >>>
     >>     >>>                        _testStage = RECEIVE_DATA;
     >>     >>>                        TestResults.appendDebugMsg("S2C test:
     >>     >>>                 RECEIVE_DATA stage.");
     >>     >>>                 @@ -290,9 +303,11 @@
     >>     >>>                        // Record end time right after it
     stops
     >>     >>>                 receiving data, to be as accurate
     >>     >>>                        // as possible.
     >>     >>>                        _s2cTimer.stop();
     >>     >>>                 -      _s2cTestDuration = getTimer() -
     >>     _s2cTestDuration;
     >>     >>>                 +      _s2cTestDuration = getTimer() -
     >>     _s2cTestStartTime;
     >>     >>>                        TestResults.appendDebugMsg(
     >>     >>>                            "S2C test lasted " +
     _s2cTestDuration + "
     >>     >>>                 msec.");
     >>     >>>                 +      _speedUpdateTimer.stop();
     >>     >>>                 +
     >>     >>>
     >>      _speedUpdateTimer.removeEventListener(TimerEvent.TIMER,
     >>     >>>                 onSpeedUpdate);
     >>     >>>                        _readTimer.stop();
     >>     >>>
     >>     >>>
      _readTimer.removeEventListener(TimerEvent.TIMER,
     >>     >>>                 onS2CTimeout);
     >>     >>>
     >>     >>>
      _s2cTimer.removeEventListener(TimerEvent.TIMER,
     >>     >>>                 onS2CTimeout);
     >>     >>>                 @@ -485,7 +500,6 @@
     >>     >>>
      removeCtlSocketOnReceivedDataListener();
     >>     >>>
     >>     >>>                        TestResults.appendDebugMsg("S2C test:
     >>     END_TEST
     >>     >>>                 stage.");
     >>     >>>                 -
     >>     >>>                        if (_s2cTestSuccess)
     >>     >>>                           TestResults.appendDebugMsg(
     >>     >>>
     >>     ResourceManager.getInstance().getString(
     >>     >>>
     >>     >>>
     >>     >>
     >>     >>
     >>     >
     >>     >
     >>     >
     >>
     >>
     >