NDT-DEV email list created

[Aaron Brown <>]

Hey Will,

On Jun 25, 2014, at 1:59 PM, Will Hawkins 
<>
 wrote:

> Hey Aaron!
> 
> I am working on this -- I promise :-)

Excellent, no rush :)

> Just to get some context, are you assuming that we would include an HTML
> wrapper for the Flash client and invoke the wrapped version to actually
> do the self testing? I'm just trying to figure out which of the
> Actionscript facilities I'll have access to in order to resolve this
> dilemma.

That’s what I’m thinking. Similar to how the java applet works (e.g. 
http://desk146.internet2.edu:7123/), you’d go to a webpage and the flash 
client would be embedded in the page. You could then just hit “start”, and 
it’d do a test from your machine to the server that served it.

Cheers,
Aaron

> 
> Will
> 
> On 06/24/2014 10:13 AM, Aaron Brown wrote:
>> Hey Will,
>> 
>> I’ve got the RPM working with the existing fakewww/Applet.
>> 
>> I’d like to include the flash client in the RPM if possible (otherwise, 
>> there’s not much benefit to the average RPM install for the new release). 
>> Unfortunately, it doesn’t look like there’s a good way to 
>> zero-configuration install (i.e. have it by default test back to the 
>> server it was served from) without using something like PHP (which would 
>> require swapping over to using Apache to serve the clients). Would it be 
>> possible to include a setting for the flash client to say “the server to 
>> use is the address of the server you downloaded the SWF from”?
>> 
>> Cheers,
>> Aaron
>> 
>> On Jun 20, 2014, at 8:02 PM, Will Hawkins 
>> <>
>>  wrote:
>> 
>>> Please let me know if/how I can help preparing the repo for RPM
>>> creation. I'm happy to give it a try on my own, but if there is
>>> expertise elsewhere (Aaron or Sebastian), then I am happy to take a back
>>> seat.
>>> 
>>> As for testers, we (MLab) can definitely find a few places to test. I
>>> know that we've been running trunk versions of the server on several
>>> test nodes for a while and haven't had any problems (besides the ones
>>> that I've submitted patches for).
>>> 
>>> Will
>>> 
>>> On 06/20/2014 09:54 AM, Aaron Brown wrote:
>>>> Hey Sebastian,
>>>> 
>>>> I’m untagging the release since I can guarantee we’ll need to make 
>>>> changes to the codebase for the NDT RPMs, and those are a requirement 
>>>> for outside testing since a very small percentage of NDT servers out 
>>>> there are compiled from source. I’ll commit some of the RPM bits, and 
>>>> then if you could go through, and make sure that it gets updated as 
>>>> needed for the changes (new web UI, etc), that’d be good. The goal of 
>>>> the NDT RPMs is that the administrators can do “yum install ndt-server”, 
>>>> start NDT and it just works. They don’t have to do any hand 
>>>> configuration to get an NDT server up-and-running.
>>>> 
>>>> Cheers,
>>>> Aaron
>>>> 
>>>> On Jun 20, 2014, at 5:15 AM, Sebastian Kostuch 
>>>> <>
>>>>  wrote:
>>>> 
>>>>> Hi
>>>>> Will, I have merged your last fixes into trunk so they will be 
>>>>> contained within
>>>>> new release :).
>>>>> Aaron, keeping in mind what you have said I have changed version numbers
>>>>> to be release candidate and tagged this. Such tagged version is now 
>>>>> ready
>>>>> for being tested by outside folks. I hope that test stage will perform 
>>>>> smoothly
>>>>> and we would release new NDT version ASAP!
>>>>> Also I have updated wiki pages to match new protocol changes.
>>>>> 
>>>>> Regards
>>>>> Sebastian Kostuch
>>>>> 
>>>>> On 18.06.2014 23:00, Will Hawkins wrote:
>>>>>> Aaron,
>>>>>> 
>>>>>> Thanks for the information! We (M-Lab) are especially interested in
>>>>>> 
>>>>>> a) having the code "checked" to make sure that the measurements are
>>>>>> still valid
>>>>>> 
>>>>>> and
>>>>>> 
>>>>>> b) having a release to deploy on the platform.
>>>>>> 
>>>>>> Having said that, what is the best way to prepare a release candidate 
>>>>>> to
>>>>>> start the process rolling? Are you able to help verify the measurements
>>>>>> made with the new version of the tool? Will you be able to help,
>>>>>> generally, with the release process (since, I mean, you're awesome!)?
>>>>>> 
>>>>>> Thanks again for your email!
>>>>>> Will
>>>>>> 
>>>>>> On 06/18/2014 04:46 PM, Aaron Brown wrote:
>>>>>>> Hi Sebastian,
>>>>>>> 
>>>>>>> To do a full release, NDT will need to go through a release candidate 
>>>>>>> process to ensure that outside folks have a chance to test it before 
>>>>>>> something gets labeled as “released”. Beyond that, NDT will need 
>>>>>>> verified to make sure that something didn’t break with existing 
>>>>>>> clients (important since this is a less common use-case). The upgrade 
>>>>>>> process needs to be verified as well to make sure that it doesn’t 
>>>>>>> break an existing installation when someone goes to upgrade.
>>>>>>> 
>>>>>>> As part of the upgrade procedure above, the RPMs will need updated 
>>>>>>> with any changes in the codebase since that’s the primary method 
>>>>>>> folks use to install NDT (the vast majority of NDT installations in 
>>>>>>> the wild are installed as part of the perfSONAR Performance Toolkit). 
>>>>>>> This procedure will need tested to make sure that both new installs, 
>>>>>>> and existing installs work as expected.
>>>>>>> 
>>>>>>> Once there’s a final release, the NDT RPMs will get added to the 
>>>>>>> Internet2 RPM repository (where the Toolkit users get their RPMs 
>>>>>>> from), and the source tarball will be added to the Internet2 site at 
>>>>>>> software.internet2.edu). Presumably, this link could be used as 
>>>>>>> canonical for NDT since any linked location is arbitrary from the end 
>>>>>>> user perspective.
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> Aaron
>>>>>>> 
>>>>>>> On Jun 18, 2014, at 6:04 AM, Sebastian Kostuch 
>>>>>>> <>
>>>>>>>  wrote:
>>>>>>> 
>>>>>>>> Hi again,
>>>>>>>> new NDT version is coming soon, however there are last changes to be 
>>>>>>>> performed
>>>>>>>> before this. I have merged all changes to release into trunk, tested 
>>>>>>>> and fixed individual
>>>>>>>> small bugs. Next step will be to update our wiki pages to match new 
>>>>>>>> version which I will
>>>>>>>> be working on in nearest time.
>>>>>>>> However I wanted to ask two last questions to you:
>>>>>>>> 1. So far new release package has been published in "downloads" 
>>>>>>>> section of google code
>>>>>>>> project but recently Google has disabled this option. Therefore 
>>>>>>>> should we place such archive
>>>>>>>> in some external source like Google drive where users could directly 
>>>>>>>> download this (and update
>>>>>>>> wiki with such link)? Or maybe just publish link to release tagged 
>>>>>>>> revision?
>>>>>>>> 
>>>>>>>> 2. Also what about updating NDT links/software on another sites? I 
>>>>>>>> know that it will be performed
>>>>>>>> on M-lab but what about Internet2 website?
>>>>>>>> 
>>>>>>>> Regards
>>>>>>>> Sebastian
>>>>>>>> 
>>>>>>>> On 18.06.2014 09:06, Sebastian Kostuch wrote:
>>>>>>>>> Hi Will
>>>>>>>>> indeed these changes results in higher speeds (I was able to get 
>>>>>>>>> ~760 mbps
>>>>>>>>> which is quite similar to yours). Also I have checked flash client 
>>>>>>>>> as background
>>>>>>>>> for new JS UI and both speeds are being updated correctly in 
>>>>>>>>> real-time. This is
>>>>>>>>> really great work! Thanks for this. I have merged these changes 
>>>>>>>>> into trunk so
>>>>>>>>> they would be also present in new release.
>>>>>>>>> Also as you probably have noticed I have fixed compilation error 
>>>>>>>>> when using
>>>>>>>>> web100 so I closed appropriated issue related to this.
>>>>>>>>> 
>>>>>>>>> Regards
>>>>>>>>> Sebastian
>>>>>>>>> 
>>>>>>>>> On 17.06.2014 22:52, Will Hawkins wrote:
>>>>>>>>>> Hello Sebastian and Aaron (and MLab'ers CC'd),
>>>>>>>>>> 
>>>>>>>>>> We found a way to juice the Flash client and achieve the 
>>>>>>>>>> performance
>>>>>>>>>> necessary to accurately measure high-speed network connections.
>>>>>>>>>> 
>>>>>>>>>> The change is in r1082.
>>>>>>>>>> 
>>>>>>>>>> The solution was counterintuitive (at least for me). Instead of
>>>>>>>>>> attempting to read data from the socket as quickly as possible, the
>>>>>>>>>> TestS2C class just gets out of the way for the duration of the 
>>>>>>>>>> test. It
>>>>>>>>>> registers only for the "socket close" event. The runtime has the
>>>>>>>>>> necessary speed to keep up with the network.
>>>>>>>>>> 
>>>>>>>>>> The only problem with this method is keeping the user interface
>>>>>>>>>> up-to-date on progress throughout the test. Sebastian, this is 
>>>>>>>>>> where I
>>>>>>>>>> need you to double check. I believe that the code in r1082 properly
>>>>>>>>>> updates _s2cByteCount in the appropriate spot for those real-time
>>>>>>>>>> updates (specifically onSpeedUpdate). But, please double check!
>>>>>>>>>> 
>>>>>>>>>> We were able to measure a connection at ~ 775 mbps using the Flash
>>>>>>>>>> client, but it would be great to have as many people test this as
>>>>>>>>>> possible. To do so, you can use a pre-compiled version at
>>>>>>>>>> 
>>>>>>>>>> http://files.opentechinstitute.org/~hawkinsw/flash/test.html
>>>>>>>>>> 
>>>>>>>>>> On a separate, but related note, please also review r1083 for 
>>>>>>>>>> inclusion
>>>>>>>>>> in the NDT release. This change allows the client to re-query JS 
>>>>>>>>>> for the
>>>>>>>>>> test's hostname just before launching a test. This gives the 
>>>>>>>>>> wrapper
>>>>>>>>>> page the opportunity to do a AJAX query to determine the test's
>>>>>>>>>> hostname. This is important for MLab because we do dynamic server
>>>>>>>>>> selection using a web-based API.
>>>>>>>>>> 
>>>>>>>>>> I am eager to hear whether you have success with these changes or 
>>>>>>>>>> not. I
>>>>>>>>>> hope you do!
>>>>>>>>>> 
>>>>>>>>>> Will
>>>>>>>>>> 
>>>>>>>>>> On 06/17/2014 10:53 AM, Will Hawkins wrote:
>>>>>>>>>>> Just FYI: I believe that we have happened on to an actual 
>>>>>>>>>>> solution for
>>>>>>>>>>> the "limit" to the performance of the S2C test. I am going to 
>>>>>>>>>>> implement
>>>>>>>>>>> it today and do some testing.
>>>>>>>>>>> 
>>>>>>>>>>> If possible, could we hold out on doing a release for a little 
>>>>>>>>>>> while
>>>>>>>>>>> pending this investigation?
>>>>>>>>>>> 
>>>>>>>>>>> Will
>>>>>>>>>>> 
>>>>>>>>>>> On 06/16/2014 01:50 AM, Sebastian Kostuch wrote:
>>>>>>>>>>>> Hi Will,
>>>>>>>>>>>> what you mentioned seems reasonable. Would it be possible to 
>>>>>>>>>>>> merge these
>>>>>>>>>>>> needed
>>>>>>>>>>>> changes into trunk in nearest time so we could make new release 
>>>>>>>>>>>> for
>>>>>>>>>>>> example tomorrow
>>>>>>>>>>>> or the day after tomorrow?
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Sebastian
>>>>>>>>>>>> 
>>>>>>>>>>>> On 13.06.2014 17:36, Will Hawkins wrote:
>>>>>>>>>>>>> Sebastian,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for sending this out! We (at MLab) are very interested in
>>>>>>>>>>>>> deploying a new version of the code so this 'release' looks 
>>>>>>>>>>>>> like a
>>>>>>>>>>>>> good place to fit together! :-) In other words, we (again, 
>>>>>>>>>>>>> MLab) are
>>>>>>>>>>>>> definitely going to do what you said in 3!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I am out of the office, but I wanted to quickly respond to 
>>>>>>>>>>>>> point 2.
>>>>>>>>>>>>> See below!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks again for driving this process forward!!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Will
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 6/13/14, 4:47 AM, Sebastian Kostuch wrote:
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>> NDT has been updated recently with some major changes (like
>>>>>>>>>>>>>> MSG_EXTENDED_LOGIN,
>>>>>>>>>>>>>> json support, new JS UI) so as I have mentioned in some 
>>>>>>>>>>>>>> earlier mail
>>>>>>>>>>>>>> that would be probably
>>>>>>>>>>>>>> good time to make new release. I have few questions related to 
>>>>>>>>>>>>>> this:
>>>>>>>>>>>>>> 1. New JS UI would fit well within this new release, however 
>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>> related to this are still
>>>>>>>>>>>>>> on work branches and need review. Any feedback would be very
>>>>>>>>>>>>>> appreciated. These are
>>>>>>>>>>>>>> following issues: 129, 132, 133.
>>>>>>>>>>>>>> 2. Is Issue144 (URLLoader to flash) supposed to be contained 
>>>>>>>>>>>>>> also in
>>>>>>>>>>>>>> nearest release version?
>>>>>>>>>>>>>> If yes, then how does progress with work on this look like? If 
>>>>>>>>>>>>>> I recall
>>>>>>>>>>>>>> correctly then you, Will,
>>>>>>>>>>>>>> mentioned that they are well tested and thus almost ready, 
>>>>>>>>>>>>>> right?
>>>>>>>>>>>>> Some of these are still up for debate. I am going to forward an 
>>>>>>>>>>>>> email
>>>>>>>>>>>>> that I just sent to the MLab team with an update on the 
>>>>>>>>>>>>> conundrum
>>>>>>>>>>>>> around the HTTP test to give you a sense about why they are up 
>>>>>>>>>>>>> for
>>>>>>>>>>>>> debate.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> But, there are also changes in that branch that *need* to be
>>>>>>>>>>>>> incorporated into anything that gets released. For instance, 
>>>>>>>>>>>>> r1070.
>>>>>>>>>>>>> There are parts of r1064 that will have to be integrated no 
>>>>>>>>>>>>> matter
>>>>>>>>>>>>> what (any of the changes that have to do with splitting start 
>>>>>>>>>>>>> test and
>>>>>>>>>>>>> finalize test stages).
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Does that make sense?
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 3. Along with new release on googlecode what do you think 
>>>>>>>>>>>>>> about pushing
>>>>>>>>>>>>>> these changes also
>>>>>>>>>>>>>> on M-Lab site? They bring many new features at all.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Looking forward to your answers and I hope that next week 
>>>>>>>>>>>>>> could be
>>>>>>>>>>>>>> celebrated with making
>>>>>>>>>>>>>> new release of NDT :).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Kind regards
>>>>>>>>>>>>>> Sebastian Kostuch
>>>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>