NDT-DEV email list created

[Don Slaunwhite <>]

Hi Jordan,

It seems we may have multiple email threads going on about this issue. I'll 
respond with details on the other thread. 8) But to help clarify for the 
people here.

We were running a Windows 2012 Server  as the Amazon VM. At first we chose 
one with "Low" network bandwidth, but we also created another with "High" 
network bandwidth. On those Windows machines we have testing software which 
runs our IPT test through the Chrome browser. It basically just goes flash 
then websocket etc. The times between runs is roughly 30 seconds and we let 
it run overnight.

We did see slower values coming from the Toronto site, so we ran the same 
tests to Calgary and Montreal. Those sites gave us much better results (in 
both Flash and Websocket) but still Flash was significantly faster. I'm not 
sure if it is our implementation of the Flash/websockets client or what. 

For instance in Montreal

Flash Upload - 304Mbps
Flash Download - 220Mbps
Websocket Upload - 45Mbps
Webscoket Download - 230Mbps

And in Calgary

Flash Upload - 616Mbps
Flash Download - 542Mbps
Websocket Upload - 44Mbps
Webscoket Download - 472Mbps

So even on other servers we are definitely seeing a degradation of Websocket 
upload. Now perhaps it truly is something with the VM. But even then it seems 
a bit odd. We have no firewall/anti-virus on. What sort of things have you 
seen gone wrong with VM testing?

We did do some external real life testing via our employee's home systems 
(not through VPN etc) and we still saw slower Websocket speeds. (But not 
anything of this magnitude of difference.)

For example:
 
(about 200 tests from around Ottawa)

20.8 Mbps - Flash Chrome Download Avg
3.034 Mpbs  - Flash Chrome Upload Avg
21.2 Mpbs - Websocket Chrome Download Avg
2.56 Mpbs - Websocket Chrome Upload Avg

So not as big a difference, but still large enough to wonder. Then combined 
with the VM testing above.

Do you have a dataset from your testing that we could look at? Honestly if 
you have a good sample size of data that we can feel comfortable with, then 
we can start looking at what exactly on our implementation is going wrong. 
(if it is indeed our implementation.)

Thanks,
Don


-----Original Message-----
From: Jordan McCarthy 
[mailto:]
 
Sent: July-07-15 3:53 PM
To: Don Slaunwhite
Cc: 
;
 

Subject: Re: [ndt-dev] Websocket Client - Upload Speed Problem

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi everybody,
        We've also been monitoring the performance characteristics of the 
WebSockets client closely, both before and after the client's official 
publication in NDT 3.7.0, and we haven't been able to reproduce the disparity 
that CIRA has encountered.  We've run Websocket-based tests from several 
different browser combinations on a variety of operating systems and 
consumer-grade connections, during various times of the day, and haven't 
encountered any appreciable differences (within any given 
machine/connection/time of day combination).  Additionally, for the sake of 
thoroughness we've run C-client tests from the same connections, and the 
numbers we got from the C client runs were pretty comparable with what we 
were getting out of all of the Websockets tests.

Don: could you tell us a little bit more about your testing methodology?  I'm 
guessing you spun up a Linux VM, and used X-forwarding to get access to an 
instance of the browser running on the VM?

Off the top of my head that sounds reasonable, but we've definitely seen 
weird artifacts introduced by running tests out of VM environments, so 
perhaps that could be throwing things off somewhat.

Jordan

Jordan McCarthy
Open Technology Institute @ New America
Public Key: 0xC08D8042 | 4A61 3D39 4125 127D 65EA DDC2 BFBD A2E9 C08D 80
42

On 07/06/2015 02:45 PM, Don Slaunwhite wrote:
> Hi Everyone,
> 
> 
> 
> My name is Don Slaunwhite and I’m a Product Manager at CIRA. We have 
> been utilizing the NDT tests as part of our Internet Performance Test 
> up here in Canada.
> 
> 
> 
> We have been working on transitioning to the Websocket client with our 
> test, but we have been seeing some very different results in upload 
> speeds as compared to the flash client.
> 
> 
> 
> We did a lot of internal/external testing and in every case the upload 
> speeds for the websocket version were lower (most times
> significantly) than our current flash client. The download speeds are 
> comparable, with websocket usually coming in a bit faster
> 
> 
> 
> For example we setup a VM at Amazon to run some (hopefully!) 
> controlled tests. Using Chrome and Firefox.
> 
> 
> 
> Chrome Averages based on ~200 tests
> 
> Flash 19.3Mpbs Upload
> 
> Flash 49.8Mpbs Download
> 
> Websocket  9.3Mpbs Upload
> 
> Websocket  54.3Mpbs Download
> 
> 
> 
> Firefox Averages based on ~300 tests
> 
> Flash 27.4 Mpbs Upload
> 
> Flash 50.1 Mpbs Download
> 
> Websocket  11.1 Mpbs Upload
> 
> Websocket  57.2 Mpbs Download
> 
> 
> 
> In each case the websocket upload is significantly lower. I’m trying 
> to determine if this is expected behaviour with the websocket code. If 
> not what possible items might be causing this type of speed 
> degradation.
> 
> 
> 
> We are running with client versions 3.7.0 (Flash has a buffer size of 
> 32K) against mlab servers in Toronto.
> 
> 
> 
> I realize there will be new functionality/capability with the multiple 
> stream releases, but right now I’d like to try and focus on one major 
> change at a time, so any ideas on speed differences between Flash and 
> Websocket using just 3.7.0 would be really helpful.
> 
> 
> 
> Thanks,
> 
> Don
> 
> 
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBCgAGBQJVnC4sAAoJEL+9ounAjYBCfVgH/2q3PGodloBkPZoa6dW5nTmx
pLRAitSZwD8DS12VP2Wdy9zWNhmDExJuCVtRVQo9jF+ZwPqghh7U+ZpGRqWvFYdq
XOUYxwUzRlN4fkVF43k+huGdrfGrG5Guz+zkkiVKAD/4Z1vLB6tknVUFyo5gOXs5
WcchPM8Hi/8V1x4i+nVY+FiwiVqJBDqG2EJXDPqMP/G60kguJGra2PhlljNl7j8t
sM0X+jyzQQzuUTruBHvQFES0TDPtS+AO07eft2JWUqdt6PcPYQt1NcBn8WJ+b/Ks
JF6KKBlG+vm0pJt7nuCflIgXDMe7CW885WhMf+rMGC5GByDa+rzxATHCS9TZANE=
=Ai6W
-----END PGP SIGNATURE-----