NDT-DEV email list created

[Will Hawkins <>]

Please let me know if/how I can help preparing the repo for RPM
creation. I'm happy to give it a try on my own, but if there is
expertise elsewhere (Aaron or Sebastian), then I am happy to take a back
seat.

As for testers, we (MLab) can definitely find a few places to test. I
know that we've been running trunk versions of the server on several
test nodes for a while and haven't had any problems (besides the ones
that I've submitted patches for).

Will

On 06/20/2014 09:54 AM, Aaron Brown wrote:
> Hey Sebastian,
> 
> I’m untagging the release since I can guarantee we’ll need to make changes 
> to the codebase for the NDT RPMs, and those are a requirement for outside 
> testing since a very small percentage of NDT servers out there are compiled 
> from source. I’ll commit some of the RPM bits, and then if you could go 
> through, and make sure that it gets updated as needed for the changes (new 
> web UI, etc), that’d be good. The goal of the NDT RPMs is that the 
> administrators can do “yum install ndt-server”, start NDT and it just 
> works. They don’t have to do any hand configuration to get an NDT server 
> up-and-running.
> 
> Cheers,
> Aaron
> 
> On Jun 20, 2014, at 5:15 AM, Sebastian Kostuch 
> <>
>  wrote:
> 
>> Hi
>> Will, I have merged your last fixes into trunk so they will be contained 
>> within
>> new release :).
>> Aaron, keeping in mind what you have said I have changed version numbers
>> to be release candidate and tagged this. Such tagged version is now ready
>> for being tested by outside folks. I hope that test stage will perform 
>> smoothly
>> and we would release new NDT version ASAP!
>> Also I have updated wiki pages to match new protocol changes.
>>
>> Regards
>> Sebastian Kostuch
>>
>> On 18.06.2014 23:00, Will Hawkins wrote:
>>> Aaron,
>>>
>>> Thanks for the information! We (M-Lab) are especially interested in
>>>
>>> a) having the code "checked" to make sure that the measurements are
>>> still valid
>>>
>>> and
>>>
>>> b) having a release to deploy on the platform.
>>>
>>> Having said that, what is the best way to prepare a release candidate to
>>> start the process rolling? Are you able to help verify the measurements
>>> made with the new version of the tool? Will you be able to help,
>>> generally, with the release process (since, I mean, you're awesome!)?
>>>
>>> Thanks again for your email!
>>> Will
>>>
>>> On 06/18/2014 04:46 PM, Aaron Brown wrote:
>>>> Hi Sebastian,
>>>>
>>>> To do a full release, NDT will need to go through a release candidate 
>>>> process to ensure that outside folks have a chance to test it before 
>>>> something gets labeled as “released”. Beyond that, NDT will need 
>>>> verified to make sure that something didn’t break with existing clients 
>>>> (important since this is a less common use-case). The upgrade process 
>>>> needs to be verified as well to make sure that it doesn’t break an 
>>>> existing installation when someone goes to upgrade.
>>>>
>>>> As part of the upgrade procedure above, the RPMs will need updated with 
>>>> any changes in the codebase since that’s the primary method folks use to 
>>>> install NDT (the vast majority of NDT installations in the wild are 
>>>> installed as part of the perfSONAR Performance Toolkit). This procedure 
>>>> will need tested to make sure that both new installs, and existing 
>>>> installs work as expected.
>>>>
>>>> Once there’s a final release, the NDT RPMs will get added to the 
>>>> Internet2 RPM repository (where the Toolkit users get their RPMs from), 
>>>> and the source tarball will be added to the Internet2 site at 
>>>> software.internet2.edu). Presumably, this link could be used as 
>>>> canonical for NDT since any linked location is arbitrary from the end 
>>>> user perspective.
>>>>
>>>> Cheers,
>>>> Aaron
>>>>
>>>> On Jun 18, 2014, at 6:04 AM, Sebastian Kostuch 
>>>> <>
>>>>  wrote:
>>>>
>>>>> Hi again,
>>>>> new NDT version is coming soon, however there are last changes to be 
>>>>> performed
>>>>> before this. I have merged all changes to release into trunk, tested 
>>>>> and fixed individual
>>>>> small bugs. Next step will be to update our wiki pages to match new 
>>>>> version which I will
>>>>> be working on in nearest time.
>>>>> However I wanted to ask two last questions to you:
>>>>> 1. So far new release package has been published in "downloads" section 
>>>>> of google code
>>>>> project but recently Google has disabled this option. Therefore should 
>>>>> we place such archive
>>>>> in some external source like Google drive where users could directly 
>>>>> download this (and update
>>>>> wiki with such link)? Or maybe just publish link to release tagged 
>>>>> revision?
>>>>>
>>>>> 2. Also what about updating NDT links/software on another sites? I know 
>>>>> that it will be performed
>>>>> on M-lab but what about Internet2 website?
>>>>>
>>>>> Regards
>>>>> Sebastian
>>>>>
>>>>> On 18.06.2014 09:06, Sebastian Kostuch wrote:
>>>>>> Hi Will
>>>>>> indeed these changes results in higher speeds (I was able to get ~760 
>>>>>> mbps
>>>>>> which is quite similar to yours). Also I have checked flash client as 
>>>>>> background
>>>>>> for new JS UI and both speeds are being updated correctly in 
>>>>>> real-time. This is
>>>>>> really great work! Thanks for this. I have merged these changes into 
>>>>>> trunk so
>>>>>> they would be also present in new release.
>>>>>> Also as you probably have noticed I have fixed compilation error when 
>>>>>> using
>>>>>> web100 so I closed appropriated issue related to this.
>>>>>>
>>>>>> Regards
>>>>>> Sebastian
>>>>>>
>>>>>> On 17.06.2014 22:52, Will Hawkins wrote:
>>>>>>> Hello Sebastian and Aaron (and MLab'ers CC'd),
>>>>>>>
>>>>>>> We found a way to juice the Flash client and achieve the performance
>>>>>>> necessary to accurately measure high-speed network connections.
>>>>>>>
>>>>>>> The change is in r1082.
>>>>>>>
>>>>>>> The solution was counterintuitive (at least for me). Instead of
>>>>>>> attempting to read data from the socket as quickly as possible, the
>>>>>>> TestS2C class just gets out of the way for the duration of the test. 
>>>>>>> It
>>>>>>> registers only for the "socket close" event. The runtime has the
>>>>>>> necessary speed to keep up with the network.
>>>>>>>
>>>>>>> The only problem with this method is keeping the user interface
>>>>>>> up-to-date on progress throughout the test. Sebastian, this is where I
>>>>>>> need you to double check. I believe that the code in r1082 properly
>>>>>>> updates _s2cByteCount in the appropriate spot for those real-time
>>>>>>> updates (specifically onSpeedUpdate). But, please double check!
>>>>>>>
>>>>>>> We were able to measure a connection at ~ 775 mbps using the Flash
>>>>>>> client, but it would be great to have as many people test this as
>>>>>>> possible. To do so, you can use a pre-compiled version at
>>>>>>>
>>>>>>> http://files.opentechinstitute.org/~hawkinsw/flash/test.html
>>>>>>>
>>>>>>> On a separate, but related note, please also review r1083 for 
>>>>>>> inclusion
>>>>>>> in the NDT release. This change allows the client to re-query JS for 
>>>>>>> the
>>>>>>> test's hostname just before launching a test. This gives the wrapper
>>>>>>> page the opportunity to do a AJAX query to determine the test's
>>>>>>> hostname. This is important for MLab because we do dynamic server
>>>>>>> selection using a web-based API.
>>>>>>>
>>>>>>> I am eager to hear whether you have success with these changes or 
>>>>>>> not. I
>>>>>>> hope you do!
>>>>>>>
>>>>>>> Will
>>>>>>>
>>>>>>> On 06/17/2014 10:53 AM, Will Hawkins wrote:
>>>>>>>> Just FYI: I believe that we have happened on to an actual solution 
>>>>>>>> for
>>>>>>>> the "limit" to the performance of the S2C test. I am going to 
>>>>>>>> implement
>>>>>>>> it today and do some testing.
>>>>>>>>
>>>>>>>> If possible, could we hold out on doing a release for a little while
>>>>>>>> pending this investigation?
>>>>>>>>
>>>>>>>> Will
>>>>>>>>
>>>>>>>> On 06/16/2014 01:50 AM, Sebastian Kostuch wrote:
>>>>>>>>> Hi Will,
>>>>>>>>> what you mentioned seems reasonable. Would it be possible to merge 
>>>>>>>>> these
>>>>>>>>> needed
>>>>>>>>> changes into trunk in nearest time so we could make new release for
>>>>>>>>> example tomorrow
>>>>>>>>> or the day after tomorrow?
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Sebastian
>>>>>>>>>
>>>>>>>>> On 13.06.2014 17:36, Will Hawkins wrote:
>>>>>>>>>> Sebastian,
>>>>>>>>>>
>>>>>>>>>> Thanks for sending this out! We (at MLab) are very interested in
>>>>>>>>>> deploying a new version of the code so this 'release' looks like a
>>>>>>>>>> good place to fit together! :-) In other words, we (again, MLab) 
>>>>>>>>>> are
>>>>>>>>>> definitely going to do what you said in 3!
>>>>>>>>>>
>>>>>>>>>> I am out of the office, but I wanted to quickly respond to point 2.
>>>>>>>>>> See below!
>>>>>>>>>>
>>>>>>>>>> Thanks again for driving this process forward!!
>>>>>>>>>>
>>>>>>>>>> Will
>>>>>>>>>>
>>>>>>>>>> On 6/13/14, 4:47 AM, Sebastian Kostuch wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>> NDT has been updated recently with some major changes (like
>>>>>>>>>>> MSG_EXTENDED_LOGIN,
>>>>>>>>>>> json support, new JS UI) so as I have mentioned in some earlier 
>>>>>>>>>>> mail
>>>>>>>>>>> that would be probably
>>>>>>>>>>> good time to make new release. I have few questions related to 
>>>>>>>>>>> this:
>>>>>>>>>>> 1. New JS UI would fit well within this new release, however 
>>>>>>>>>>> changes
>>>>>>>>>>> related to this are still
>>>>>>>>>>> on work branches and need review. Any feedback would be very
>>>>>>>>>>> appreciated. These are
>>>>>>>>>>> following issues: 129, 132, 133.
>>>>>>>>>>> 2. Is Issue144 (URLLoader to flash) supposed to be contained also 
>>>>>>>>>>> in
>>>>>>>>>>> nearest release version?
>>>>>>>>>>> If yes, then how does progress with work on this look like? If I 
>>>>>>>>>>> recall
>>>>>>>>>>> correctly then you, Will,
>>>>>>>>>>> mentioned that they are well tested and thus almost ready, right?
>>>>>>>>>> Some of these are still up for debate. I am going to forward an 
>>>>>>>>>> email
>>>>>>>>>> that I just sent to the MLab team with an update on the conundrum
>>>>>>>>>> around the HTTP test to give you a sense about why they are up for
>>>>>>>>>> debate.
>>>>>>>>>>
>>>>>>>>>> But, there are also changes in that branch that *need* to be
>>>>>>>>>> incorporated into anything that gets released. For instance, r1070.
>>>>>>>>>> There are parts of r1064 that will have to be integrated no matter
>>>>>>>>>> what (any of the changes that have to do with splitting start test 
>>>>>>>>>> and
>>>>>>>>>> finalize test stages).
>>>>>>>>>>
>>>>>>>>>> Does that make sense?
>>>>>>>>>>
>>>>>>>>>>> 3. Along with new release on googlecode what do you think about 
>>>>>>>>>>> pushing
>>>>>>>>>>> these changes also
>>>>>>>>>>> on M-Lab site? They bring many new features at all.
>>>>>>>>>>>
>>>>>>>>>>> Looking forward to your answers and I hope that next week could be
>>>>>>>>>>> celebrated with making
>>>>>>>>>>> new release of NDT :).
>>>>>>>>>>>
>>>>>>>>>>> Kind regards
>>>>>>>>>>> Sebastian Kostuch
>>>>>>>>>>>
>>>>>>
>>>>
>>
>