NDT-DEV email list created

[Will Hawkins <>]

Mr. Carlson,

Thank you for adding your insight on this. I certainly appreciate having
the chance to work with people as respected in the measurement world as
you and Matt.

I agree that the goal should be not to tell the user that they are
incapable of choosing a good browser. In fact, I don't actually like
Chrome (shh, don't tell anyone!). I also agree that, in the case where
the user is running a "bad" browser/os combination, the measurements
made with this Flash client will be indicative of general performance on
their connection when running Flash applications (with the caveat that
HTTP-based network i/o is significantly faster than socket-based i/o
because it is implemented better by the runtime across platforms). As a
final note of agreement, I concur that NDT is more than just a "speed
test".

The way that we are going to actually implement this is with a series of
configurable knobs. The person embedding the Flash client in their HTML
will have the option to treat an incompatible browser/os combination as
an error or a warning. They deployer will get to handle that condition
in a callback and display an appropriate message (so that they could say
something in "big bold letters", which I liked!).

If that type of scenario is acceptable to you, the decision is more
about the appropriate default (error or warning). We are close to
finishing up an implementation. Would you be willing to review that when
it's ready and see if it meets your expectations?

Thanks again for adding your thoughts! I was hoping that you'd weigh in.
We have Matt M on a separate thread adding his thoughts as well so I am
confident that we will come to a solid resolution.

Will

On 07/09/2014 08:40 PM, Richard Carlson wrote:
> Sebastian/Will;
> 
> I'm not sure I agree with this idea.  While I agree that letting the
> user know that the test results are bad due to the Flash runtime
> problem, I'm not at all convinced that the proper response is to say
> 'use Chrome'.  It just seems a bit strong to me, like you are saying the
> user is too stupid to run a good browser.
> 
> I'd prefer to let the test run and put in big bold letters across the
> results page or in some other manner indicate that the limiting factor
> is the client's flash runtime.  That way the user is being informed and
> the logged test results can also contain the same notation so they can
> be discarded if necessary.
> 
> After all, they will get that performance when running other flash apps,
> so it would be good for them to know what the limits are.
> 
> The whole purpose of the NDT system is to provide advanced diagnostic
> messages, not just run speed tests.  Showing the user that the test
> results are limited and explaining why would be much better than telling
> them to 'trust me and replace your browser's flash runtime'.
> 
> So for me, the real question is can you make a simple and intuitive
> image or text that clearly shows that the problem is the flash runtime
> and not the NDT service.
> 
> Rich
> 
> On 07/08/2014 03:09 AM, Sebastian Kostuch wrote:
>> Hi Will,
>> IMHO it would be good to disable possibility of running test when
>> we detect that flash client is being run from within not supported
>> platform. With this there won't be faked results noted on server
>> side's statistics.
>> Also looking forward to any further information
>> regarding new release :).
>>
>> Regards
>> Sebastian
>>
>> On 03.07.2014 02:40, Will Hawkins wrote:
>>> Sebastian and Aaron (and everyone, really),
>>>
>>> We at MLab are in the final throes of getting our platform ready for the
>>> Flash client. In so doing, we've gotten deep into the weeds on the
>>> performance/measurement implications of the Flash runtime.
>>>
>>> I wanted to present you with our findings and get your input.
>>>
>>> On all platforms (Windows, Mac, Linux), running the Flash client in
>>> Google Chrome works a-ok. On Windows, running the Flash client in all
>>> other browsers work great too. On Linux and Mac, however, that is not
>>> the case.
>>>
>>> On Mac and Linux where the user runs the NDT Flash client under Adobe's
>>> runtime (i.e., browsers other than Chrome [even Chromium does not
>>> work]), there is not enough network i/o fidelity to produce measurements
>>> for connections with > ~ 50mbps throughput (on the s2c side).
>>>
>>> In other words, Google's independent implementation of the Flash runtime
>>> performs well enough on all platforms, but Adobe's implementation
>>> performs well enough only on Windows.
>>>
>>> Because MLab collects lots of data, we don't want our results to be
>>> spoiled by tests whose results are artificially limited. We are
>>> attempting to work out a way to "encourage" users to run the test in the
>>> environment where we will get the most accurate results. To do so, we've
>>> settled on the following plan:
>>>
>>> 1.  When the flash client detects that it is being run within a
>>> non-optimal environment, display the following:  "Unfortunately, this
>>> test cannot run within your current browser environment.  Please run
>>> the test from within Google Chrome, or download M-Lab's Firefox
>>> Extension."
>>>
>>> 2.  Disallow the flash client from running on any platform that
>>> identifies itself as something other than Windows / Chrome.
>>>
>>> What do people think about this plan? Reasonable?
>>>
>>> As always, it's great working with everyone in NDT land. I look forward
>>> to continuing the work :-)
>>>
>>> Will
>>>
>>> On 06/18/2014 03:06 AM, Sebastian Kostuch wrote:
>>>> Hi Will
>>>> indeed these changes results in higher speeds (I was able to get ~760
>>>> mbps
>>>> which is quite similar to yours). Also I have checked flash client as
>>>> background
>>>> for new JS UI and both speeds are being updated correctly in real-time.
>>>> This is
>>>> really great work! Thanks for this. I have merged these changes into
>>>> trunk so
>>>> they would be also present in new release.
>>>> Also as you probably have noticed I have fixed compilation error when
>>>> using
>>>> web100 so I closed appropriated issue related to this.
>>>>
>>>> Regards
>>>> Sebastian
>>>>
>>>> On 17.06.2014 22:52, Will Hawkins wrote:
>>>>> Hello Sebastian and Aaron (and MLab'ers CC'd),
>>>>>
>>>>> We found a way to juice the Flash client and achieve the performance
>>>>> necessary to accurately measure high-speed network connections.
>>>>>
>>>>> The change is in r1082.
>>>>>
>>>>> The solution was counterintuitive (at least for me). Instead of
>>>>> attempting to read data from the socket as quickly as possible, the
>>>>> TestS2C class just gets out of the way for the duration of the
>>>>> test. It
>>>>> registers only for the "socket close" event. The runtime has the
>>>>> necessary speed to keep up with the network.
>>>>>
>>>>> The only problem with this method is keeping the user interface
>>>>> up-to-date on progress throughout the test. Sebastian, this is where I
>>>>> need you to double check. I believe that the code in r1082 properly
>>>>> updates _s2cByteCount in the appropriate spot for those real-time
>>>>> updates (specifically onSpeedUpdate). But, please double check!
>>>>>
>>>>> We were able to measure a connection at ~ 775 mbps using the Flash
>>>>> client, but it would be great to have as many people test this as
>>>>> possible. To do so, you can use a pre-compiled version at
>>>>>
>>>>> http://files.opentechinstitute.org/~hawkinsw/flash/test.html
>>>>>
>>>>> On a separate, but related note, please also review r1083 for
>>>>> inclusion
>>>>> in the NDT release. This change allows the client to re-query JS for
>>>>> the
>>>>> test's hostname just before launching a test. This gives the wrapper
>>>>> page the opportunity to do a AJAX query to determine the test's
>>>>> hostname. This is important for MLab because we do dynamic server
>>>>> selection using a web-based API.
>>>>>
>>>>> I am eager to hear whether you have success with these changes or
>>>>> not. I
>>>>> hope you do!
>>>>>
>>>>> Will
>>>>>
>>>>> On 06/17/2014 10:53 AM, Will Hawkins wrote:
>>>>>> Just FYI: I believe that we have happened on to an actual solution
>>>>>> for
>>>>>> the "limit" to the performance of the S2C test. I am going to
>>>>>> implement
>>>>>> it today and do some testing.
>>>>>>
>>>>>> If possible, could we hold out on doing a release for a little while
>>>>>> pending this investigation?
>>>>>>
>>>>>> Will
>>>>>>
>>>>>> On 06/16/2014 01:50 AM, Sebastian Kostuch wrote:
>>>>>>> Hi Will,
>>>>>>> what you mentioned seems reasonable. Would it be possible to merge
>>>>>>> these
>>>>>>> needed
>>>>>>> changes into trunk in nearest time so we could make new release for
>>>>>>> example tomorrow
>>>>>>> or the day after tomorrow?
>>>>>>>
>>>>>>> Regards
>>>>>>> Sebastian
>>>>>>>
>>>>>>> On 13.06.2014 17:36, Will Hawkins wrote:
>>>>>>>> Sebastian,
>>>>>>>>
>>>>>>>> Thanks for sending this out! We (at MLab) are very interested in
>>>>>>>> deploying a new version of the code so this 'release' looks like a
>>>>>>>> good place to fit together! :-) In other words, we (again, MLab)
>>>>>>>> are
>>>>>>>> definitely going to do what you said in 3!
>>>>>>>>
>>>>>>>> I am out of the office, but I wanted to quickly respond to point 2.
>>>>>>>> See below!
>>>>>>>>
>>>>>>>> Thanks again for driving this process forward!!
>>>>>>>>
>>>>>>>> Will
>>>>>>>>
>>>>>>>> On 6/13/14, 4:47 AM, Sebastian Kostuch wrote:
>>>>>>>>> Hello,
>>>>>>>>> NDT has been updated recently with some major changes (like
>>>>>>>>> MSG_EXTENDED_LOGIN,
>>>>>>>>> json support, new JS UI) so as I have mentioned in some earlier
>>>>>>>>> mail
>>>>>>>>> that would be probably
>>>>>>>>> good time to make new release. I have few questions related to
>>>>>>>>> this:
>>>>>>>>> 1. New JS UI would fit well within this new release, however
>>>>>>>>> changes
>>>>>>>>> related to this are still
>>>>>>>>> on work branches and need review. Any feedback would be very
>>>>>>>>> appreciated. These are
>>>>>>>>> following issues: 129, 132, 133.
>>>>>>>>> 2. Is Issue144 (URLLoader to flash) supposed to be contained
>>>>>>>>> also in
>>>>>>>>> nearest release version?
>>>>>>>>> If yes, then how does progress with work on this look like? If I
>>>>>>>>> recall
>>>>>>>>> correctly then you, Will,
>>>>>>>>> mentioned that they are well tested and thus almost ready, right?
>>>>>>>> Some of these are still up for debate. I am going to forward an
>>>>>>>> email
>>>>>>>> that I just sent to the MLab team with an update on the conundrum
>>>>>>>> around the HTTP test to give you a sense about why they are up for
>>>>>>>> debate.
>>>>>>>>
>>>>>>>> But, there are also changes in that branch that *need* to be
>>>>>>>> incorporated into anything that gets released. For instance, r1070.
>>>>>>>> There are parts of r1064 that will have to be integrated no matter
>>>>>>>> what (any of the changes that have to do with splitting start
>>>>>>>> test and
>>>>>>>> finalize test stages).
>>>>>>>>
>>>>>>>> Does that make sense?
>>>>>>>>
>>>>>>>>> 3. Along with new release on googlecode what do you think about
>>>>>>>>> pushing
>>>>>>>>> these changes also
>>>>>>>>> on M-Lab site? They bring many new features at all.
>>>>>>>>>
>>>>>>>>> Looking forward to your answers and I hope that next week could be
>>>>>>>>> celebrated with making
>>>>>>>>> new release of NDT :).
>>>>>>>>>
>>>>>>>>> Kind regards
>>>>>>>>> Sebastian Kostuch
>>>>>>>>>
>>>>
>>
>>