NDT-DEV email list created

[Will Hawkins <>]

On 06/26/2014 09:13 AM, Aaron Brown wrote:
> Hey Will,
> 
> On Jun 25, 2014, at 9:51 PM, Will Hawkins 
> <>
>  wrote:
> 
>> I am happy to create a branch for this, but I think that might be
>> overkill. Here's what I added
>>
>> Index: src/Main.as
>> ===================================================================
>> --- src/Main.as      (revision 1112)
>> +++ src/Main.as      (working copy)
>> @@ -45,6 +45,14 @@
>>     private function init(e:Event = null):void {
>>       removeEventListener(Event.ADDED_TO_STAGE, init);
>>
>> +      var selfUrl:String = this.loaderInfo.url;
>> +      if (selfUrl.indexOf("http://";, 0) == 0) {
>> +        selfUrl = selfUrl.replace("http://";, "");
>> +        selfUrl = selfUrl.replace(/:[0-9]+/, "");
>> +        selfUrl = selfUrl.replace(/\/.+$/, "");
>> +        server_hostname = selfUrl;
>> +      }
>> +
>>       // Set the properties of the SWF from HTML tags.
>>       NDTUtils.initializeFromHTML(this.root.loaderInfo.parameters);
> 
> Looking at the parsing there, it will definitely fail on IPv6 addresses. It 
> looks like actionscript has a URL library 
> (http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/mx/utils/URLUtil.html)
>  that would probably work for returning the address to test against.
> 

Oh, I get it. You don't think that two quickly-hacked-together regexes
covers all possible cases? You'd like me to use a library?

NO. :-)

Just kidding, of course. I was actually hoping that loaderInfo.url
returned some type of object that I could easy inquire about the
hostname. But, it seems like the URL library is the way to go. Standby
for another rev.

Will


>> This will make it default to test against an NDT server running on the
>> same host from which the swf was loaded. This does not preclude the user
>> from overriding that value with their JS callback.
>>
>> The other, perhaps easier, option is to do something similar in the JS
>> that hosts the Flash applet. JS would parse window.location and then
>> return that value in the getNDTServer() callback.
>>
>> Let me know which route you prefer!
> 
> My general preference would be to encode that default into the Flash client 
> to make it as simple as possible to embed the client in webpages.
> 
> Cheers,
> Aaron
> 
> 
>> Will
>>
>> On 06/25/2014 02:22 PM, Aaron Brown wrote:
>>> Hey Will,
>>>
>>> On Jun 25, 2014, at 1:59 PM, Will Hawkins 
>>> <>
>>>  wrote:
>>>
>>>> Hey Aaron!
>>>>
>>>> I am working on this -- I promise :-)
>>>
>>> Excellent, no rush :)
>>>
>>>> Just to get some context, are you assuming that we would include an HTML
>>>> wrapper for the Flash client and invoke the wrapped version to actually
>>>> do the self testing? I'm just trying to figure out which of the
>>>> Actionscript facilities I'll have access to in order to resolve this
>>>> dilemma.
>>>
>>> That’s what I’m thinking. Similar to how the java applet works (e.g. 
>>> http://desk146.internet2.edu:7123/), you’d go to a webpage and the flash 
>>> client would be embedded in the page. You could then just hit “start”, 
>>> and it’d do a test from your machine to the server that served it.
>>>
>>> Cheers,
>>> Aaron
>>>
>>>>
>>>> Will
>>>>
>>>> On 06/24/2014 10:13 AM, Aaron Brown wrote:
>>>>> Hey Will,
>>>>>
>>>>> I’ve got the RPM working with the existing fakewww/Applet.
>>>>>
>>>>> I’d like to include the flash client in the RPM if possible (otherwise, 
>>>>> there’s not much benefit to the average RPM install for the new 
>>>>> release). Unfortunately, it doesn’t look like there’s a good way to 
>>>>> zero-configuration install (i.e. have it by default test back to the 
>>>>> server it was served from) without using something like PHP (which 
>>>>> would require swapping over to using Apache to serve the clients). 
>>>>> Would it be possible to include a setting for the flash client to say 
>>>>> “the server to use is the address of the server you downloaded the SWF 
>>>>> from”?
>>>>>
>>>>> Cheers,
>>>>> Aaron
>>>>>
>>>>> On Jun 20, 2014, at 8:02 PM, Will Hawkins 
>>>>> <>
>>>>>  wrote:
>>>>>
>>>>>> Please let me know if/how I can help preparing the repo for RPM
>>>>>> creation. I'm happy to give it a try on my own, but if there is
>>>>>> expertise elsewhere (Aaron or Sebastian), then I am happy to take a 
>>>>>> back
>>>>>> seat.
>>>>>>
>>>>>> As for testers, we (MLab) can definitely find a few places to test. I
>>>>>> know that we've been running trunk versions of the server on several
>>>>>> test nodes for a while and haven't had any problems (besides the ones
>>>>>> that I've submitted patches for).
>>>>>>
>>>>>> Will
>>>>>>
>>>>>> On 06/20/2014 09:54 AM, Aaron Brown wrote:
>>>>>>> Hey Sebastian,
>>>>>>>
>>>>>>> I’m untagging the release since I can guarantee we’ll need to make 
>>>>>>> changes to the codebase for the NDT RPMs, and those are a requirement 
>>>>>>> for outside testing since a very small percentage of NDT servers out 
>>>>>>> there are compiled from source. I’ll commit some of the RPM bits, and 
>>>>>>> then if you could go through, and make sure that it gets updated as 
>>>>>>> needed for the changes (new web UI, etc), that’d be good. The goal of 
>>>>>>> the NDT RPMs is that the administrators can do “yum install 
>>>>>>> ndt-server”, start NDT and it just works. They don’t have to do any 
>>>>>>> hand configuration to get an NDT server up-and-running.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Aaron
>>>>>>>
>>>>>>> On Jun 20, 2014, at 5:15 AM, Sebastian Kostuch 
>>>>>>> <>
>>>>>>>  wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>> Will, I have merged your last fixes into trunk so they will be 
>>>>>>>> contained within
>>>>>>>> new release :).
>>>>>>>> Aaron, keeping in mind what you have said I have changed version 
>>>>>>>> numbers
>>>>>>>> to be release candidate and tagged this. Such tagged version is now 
>>>>>>>> ready
>>>>>>>> for being tested by outside folks. I hope that test stage will 
>>>>>>>> perform smoothly
>>>>>>>> and we would release new NDT version ASAP!
>>>>>>>> Also I have updated wiki pages to match new protocol changes.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Sebastian Kostuch
>>>>>>>>
>>>>>>>> On 18.06.2014 23:00, Will Hawkins wrote:
>>>>>>>>> Aaron,
>>>>>>>>>
>>>>>>>>> Thanks for the information! We (M-Lab) are especially interested in
>>>>>>>>>
>>>>>>>>> a) having the code "checked" to make sure that the measurements are
>>>>>>>>> still valid
>>>>>>>>>
>>>>>>>>> and
>>>>>>>>>
>>>>>>>>> b) having a release to deploy on the platform.
>>>>>>>>>
>>>>>>>>> Having said that, what is the best way to prepare a release 
>>>>>>>>> candidate to
>>>>>>>>> start the process rolling? Are you able to help verify the 
>>>>>>>>> measurements
>>>>>>>>> made with the new version of the tool? Will you be able to help,
>>>>>>>>> generally, with the release process (since, I mean, you're 
>>>>>>>>> awesome!)?
>>>>>>>>>
>>>>>>>>> Thanks again for your email!
>>>>>>>>> Will
>>>>>>>>>
>>>>>>>>> On 06/18/2014 04:46 PM, Aaron Brown wrote:
>>>>>>>>>> Hi Sebastian,
>>>>>>>>>>
>>>>>>>>>> To do a full release, NDT will need to go through a release 
>>>>>>>>>> candidate process to ensure that outside folks have a chance to 
>>>>>>>>>> test it before something gets labeled as “released”. Beyond that, 
>>>>>>>>>> NDT will need verified to make sure that something didn’t break 
>>>>>>>>>> with existing clients (important since this is a less common 
>>>>>>>>>> use-case). The upgrade process needs to be verified as well to 
>>>>>>>>>> make sure that it doesn’t break an existing installation when 
>>>>>>>>>> someone goes to upgrade.
>>>>>>>>>>
>>>>>>>>>> As part of the upgrade procedure above, the RPMs will need updated 
>>>>>>>>>> with any changes in the codebase since that’s the primary method 
>>>>>>>>>> folks use to install NDT (the vast majority of NDT installations 
>>>>>>>>>> in the wild are installed as part of the perfSONAR Performance 
>>>>>>>>>> Toolkit). This procedure will need tested to make sure that both 
>>>>>>>>>> new installs, and existing installs work as expected.
>>>>>>>>>>
>>>>>>>>>> Once there’s a final release, the NDT RPMs will get added to the 
>>>>>>>>>> Internet2 RPM repository (where the Toolkit users get their RPMs 
>>>>>>>>>> from), and the source tarball will be added to the Internet2 site 
>>>>>>>>>> at software.internet2.edu). Presumably, this link could be used as 
>>>>>>>>>> canonical for NDT since any linked location is arbitrary from the 
>>>>>>>>>> end user perspective.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Aaron
>>>>>>>>>>
>>>>>>>>>> On Jun 18, 2014, at 6:04 AM, Sebastian Kostuch 
>>>>>>>>>> <>
>>>>>>>>>>  wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi again,
>>>>>>>>>>> new NDT version is coming soon, however there are last changes to 
>>>>>>>>>>> be performed
>>>>>>>>>>> before this. I have merged all changes to release into trunk, 
>>>>>>>>>>> tested and fixed individual
>>>>>>>>>>> small bugs. Next step will be to update our wiki pages to match 
>>>>>>>>>>> new version which I will
>>>>>>>>>>> be working on in nearest time.
>>>>>>>>>>> However I wanted to ask two last questions to you:
>>>>>>>>>>> 1. So far new release package has been published in "downloads" 
>>>>>>>>>>> section of google code
>>>>>>>>>>> project but recently Google has disabled this option. Therefore 
>>>>>>>>>>> should we place such archive
>>>>>>>>>>> in some external source like Google drive where users could 
>>>>>>>>>>> directly download this (and update
>>>>>>>>>>> wiki with such link)? Or maybe just publish link to release 
>>>>>>>>>>> tagged revision?
>>>>>>>>>>>
>>>>>>>>>>> 2. Also what about updating NDT links/software on another sites? 
>>>>>>>>>>> I know that it will be performed
>>>>>>>>>>> on M-lab but what about Internet2 website?
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Sebastian
>>>>>>>>>>>
>>>>>>>>>>> On 18.06.2014 09:06, Sebastian Kostuch wrote:
>>>>>>>>>>>> Hi Will
>>>>>>>>>>>> indeed these changes results in higher speeds (I was able to get 
>>>>>>>>>>>> ~760 mbps
>>>>>>>>>>>> which is quite similar to yours). Also I have checked flash 
>>>>>>>>>>>> client as background
>>>>>>>>>>>> for new JS UI and both speeds are being updated correctly in 
>>>>>>>>>>>> real-time. This is
>>>>>>>>>>>> really great work! Thanks for this. I have merged these changes 
>>>>>>>>>>>> into trunk so
>>>>>>>>>>>> they would be also present in new release.
>>>>>>>>>>>> Also as you probably have noticed I have fixed compilation error 
>>>>>>>>>>>> when using
>>>>>>>>>>>> web100 so I closed appropriated issue related to this.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Sebastian
>>>>>>>>>>>>
>>>>>>>>>>>> On 17.06.2014 22:52, Will Hawkins wrote:
>>>>>>>>>>>>> Hello Sebastian and Aaron (and MLab'ers CC'd),
>>>>>>>>>>>>>
>>>>>>>>>>>>> We found a way to juice the Flash client and achieve the 
>>>>>>>>>>>>> performance
>>>>>>>>>>>>> necessary to accurately measure high-speed network connections.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The change is in r1082.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The solution was counterintuitive (at least for me). Instead of
>>>>>>>>>>>>> attempting to read data from the socket as quickly as possible, 
>>>>>>>>>>>>> the
>>>>>>>>>>>>> TestS2C class just gets out of the way for the duration of the 
>>>>>>>>>>>>> test. It
>>>>>>>>>>>>> registers only for the "socket close" event. The runtime has the
>>>>>>>>>>>>> necessary speed to keep up with the network.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The only problem with this method is keeping the user interface
>>>>>>>>>>>>> up-to-date on progress throughout the test. Sebastian, this is 
>>>>>>>>>>>>> where I
>>>>>>>>>>>>> need you to double check. I believe that the code in r1082 
>>>>>>>>>>>>> properly
>>>>>>>>>>>>> updates _s2cByteCount in the appropriate spot for those 
>>>>>>>>>>>>> real-time
>>>>>>>>>>>>> updates (specifically onSpeedUpdate). But, please double check!
>>>>>>>>>>>>>
>>>>>>>>>>>>> We were able to measure a connection at ~ 775 mbps using the 
>>>>>>>>>>>>> Flash
>>>>>>>>>>>>> client, but it would be great to have as many people test this 
>>>>>>>>>>>>> as
>>>>>>>>>>>>> possible. To do so, you can use a pre-compiled version at
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://files.opentechinstitute.org/~hawkinsw/flash/test.html
>>>>>>>>>>>>>
>>>>>>>>>>>>> On a separate, but related note, please also review r1083 for 
>>>>>>>>>>>>> inclusion
>>>>>>>>>>>>> in the NDT release. This change allows the client to re-query 
>>>>>>>>>>>>> JS for the
>>>>>>>>>>>>> test's hostname just before launching a test. This gives the 
>>>>>>>>>>>>> wrapper
>>>>>>>>>>>>> page the opportunity to do a AJAX query to determine the test's
>>>>>>>>>>>>> hostname. This is important for MLab because we do dynamic 
>>>>>>>>>>>>> server
>>>>>>>>>>>>> selection using a web-based API.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am eager to hear whether you have success with these changes 
>>>>>>>>>>>>> or not. I
>>>>>>>>>>>>> hope you do!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Will
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 06/17/2014 10:53 AM, Will Hawkins wrote:
>>>>>>>>>>>>>> Just FYI: I believe that we have happened on to an actual 
>>>>>>>>>>>>>> solution for
>>>>>>>>>>>>>> the "limit" to the performance of the S2C test. I am going to 
>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>> it today and do some testing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If possible, could we hold out on doing a release for a little 
>>>>>>>>>>>>>> while
>>>>>>>>>>>>>> pending this investigation?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Will
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 06/16/2014 01:50 AM, Sebastian Kostuch wrote:
>>>>>>>>>>>>>>> Hi Will,
>>>>>>>>>>>>>>> what you mentioned seems reasonable. Would it be possible to 
>>>>>>>>>>>>>>> merge these
>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>> changes into trunk in nearest time so we could make new 
>>>>>>>>>>>>>>> release for
>>>>>>>>>>>>>>> example tomorrow
>>>>>>>>>>>>>>> or the day after tomorrow?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>> Sebastian
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 13.06.2014 17:36, Will Hawkins wrote:
>>>>>>>>>>>>>>>> Sebastian,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for sending this out! We (at MLab) are very 
>>>>>>>>>>>>>>>> interested in
>>>>>>>>>>>>>>>> deploying a new version of the code so this 'release' looks 
>>>>>>>>>>>>>>>> like a
>>>>>>>>>>>>>>>> good place to fit together! :-) In other words, we (again, 
>>>>>>>>>>>>>>>> MLab) are
>>>>>>>>>>>>>>>> definitely going to do what you said in 3!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am out of the office, but I wanted to quickly respond to 
>>>>>>>>>>>>>>>> point 2.
>>>>>>>>>>>>>>>> See below!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks again for driving this process forward!!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Will
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 6/13/14, 4:47 AM, Sebastian Kostuch wrote:
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>> NDT has been updated recently with some major changes (like
>>>>>>>>>>>>>>>>> MSG_EXTENDED_LOGIN,
>>>>>>>>>>>>>>>>> json support, new JS UI) so as I have mentioned in some 
>>>>>>>>>>>>>>>>> earlier mail
>>>>>>>>>>>>>>>>> that would be probably
>>>>>>>>>>>>>>>>> good time to make new release. I have few questions related 
>>>>>>>>>>>>>>>>> to this:
>>>>>>>>>>>>>>>>> 1. New JS UI would fit well within this new release, 
>>>>>>>>>>>>>>>>> however changes
>>>>>>>>>>>>>>>>> related to this are still
>>>>>>>>>>>>>>>>> on work branches and need review. Any feedback would be very
>>>>>>>>>>>>>>>>> appreciated. These are
>>>>>>>>>>>>>>>>> following issues: 129, 132, 133.
>>>>>>>>>>>>>>>>> 2. Is Issue144 (URLLoader to flash) supposed to be 
>>>>>>>>>>>>>>>>> contained also in
>>>>>>>>>>>>>>>>> nearest release version?
>>>>>>>>>>>>>>>>> If yes, then how does progress with work on this look like? 
>>>>>>>>>>>>>>>>> If I recall
>>>>>>>>>>>>>>>>> correctly then you, Will,
>>>>>>>>>>>>>>>>> mentioned that they are well tested and thus almost ready, 
>>>>>>>>>>>>>>>>> right?
>>>>>>>>>>>>>>>> Some of these are still up for debate. I am going to forward 
>>>>>>>>>>>>>>>> an email
>>>>>>>>>>>>>>>> that I just sent to the MLab team with an update on the 
>>>>>>>>>>>>>>>> conundrum
>>>>>>>>>>>>>>>> around the HTTP test to give you a sense about why they are 
>>>>>>>>>>>>>>>> up for
>>>>>>>>>>>>>>>> debate.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But, there are also changes in that branch that *need* to be
>>>>>>>>>>>>>>>> incorporated into anything that gets released. For instance, 
>>>>>>>>>>>>>>>> r1070.
>>>>>>>>>>>>>>>> There are parts of r1064 that will have to be integrated no 
>>>>>>>>>>>>>>>> matter
>>>>>>>>>>>>>>>> what (any of the changes that have to do with splitting 
>>>>>>>>>>>>>>>> start test and
>>>>>>>>>>>>>>>> finalize test stages).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Does that make sense?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 3. Along with new release on googlecode what do you think 
>>>>>>>>>>>>>>>>> about pushing
>>>>>>>>>>>>>>>>> these changes also
>>>>>>>>>>>>>>>>> on M-Lab site? They bring many new features at all.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Looking forward to your answers and I hope that next week 
>>>>>>>>>>>>>>>>> could be
>>>>>>>>>>>>>>>>> celebrated with making
>>>>>>>>>>>>>>>>> new release of NDT :).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Kind regards
>>>>>>>>>>>>>>>>> Sebastian Kostuch
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>
>