Skip to Content.
Sympa Menu

ndt-users - Re: Revisiting: Parsing web100srv.log in sync with client output?

Subject: ndt-users list created

List archive

Re: Revisiting: Parsing web100srv.log in sync with client output?


Chronological Thread 
  • From: Tom Throckmorton <>
  • To: Seth Peery <>
  • Cc: 'Richard Carlson' <>,
  • Subject: Re: Revisiting: Parsing web100srv.log in sync with client output?
  • Date: Thu, 05 Feb 2009 23:54:12 -0500
  • Organization: MCNC
On 2/5/09 5:25 PM, Tom Throckmorton wrote:
> On 2/5/09 3:34 PM, Seth Peery wrote:
>> Fellow NDT-users,
>>
>> I'm hoping to resurrect an archived discussion on the parsing of
>> web100srv.log. The messages below, from 2006, were the most recent that
>> I could find that addressed a method of figuring out which log entries
>> corresponded to what NDT variables. I'm wondering if there has been any
>> additional development in this area.
>>
>> I developed the "Community Broadband Map" tool at Virginia Tech
>> (http://www.ecorridors.vt.edu/maps/broadbandmap.php), and I've merged
>> NDT with a number of other web-based maps since the original project.
>> At the time, I did not pull all the variables from the applet into the
>> PHP that does the DB insert when a user runs a speed test - just
>> timestamp, c2sspd and s2cspd. Now I'm hoping to retroactively parse
>> web100srv.log into a format that can be imported into a database and
>> then associated with the existing entries by making a key out of the IP
>> address and timestamp, which should give me a unique identifier.
>>
>> We've got the development resources here to construct a parser for this
>> log file if need be (which could then of course be shared with the
>> community), but before I embarked on this, I wanted to see if anyone has
>> a) some documentation of the order of the variable names in the log
>> file beyond the comments in web100srv.c
>> b) an active effort to develop a parser with which we might
>> collaborate
>
> *waves northward*
>
> Hello Seth,
>
> I've thought about doing something similar before, and have my eyes on
> the code added in ndt-3.4.x that provided db logging (which Rich refers
> to in the original thread you included) - see ndt-$ver/src/ndt_odbc.*
> I've not yet tried it myself...

So, I just took a crack at this on a test NDT instance and was able to
get it working w/out a lot of pain. I think this will get you going in
the right direction - enabling this records a _lot_ of variables per test:

> mysql> describe ndt_test_results;
> +---------------------+---------+------+-----+---------+-------+
> | Field | Type | Null | Key | Default | Extra |
> +---------------------+---------+------+-----+---------+-------+
> | spds1 | text | YES | | NULL | |
> | spds2 | text | YES | | NULL | |
> | spds3 | text | YES | | NULL | |
> | spds4 | text | YES | | NULL | |
> | runave1 | float | YES | | NULL | |
> | runave2 | float | YES | | NULL | |
> | runave3 | float | YES | | NULL | |
> | runave4 | float | YES | | NULL | |
> | cputimelog | text | YES | | NULL | |
> | snaplog | text | YES | | NULL | |
> | c2s_snaplog | text | YES | | NULL | |
> | hostName | text | YES | | NULL | |
> | testPort | int(11) | YES | | NULL | |
> | date | text | YES | | NULL | |
> | rmt_host | text | YES | | NULL | |
> | s2c2spd | int(11) | YES | | NULL | |
> | s2cspd | int(11) | YES | | NULL | |
> | c2sspd | int(11) | YES | | NULL | |
> | Timeouts | int(11) | YES | | NULL | |
> | SumRTT | int(11) | YES | | NULL | |
> | CountRTT | int(11) | YES | | NULL | |
> | PktsRetrans | int(11) | YES | | NULL | |
> | FastRetran | int(11) | YES | | NULL | |
> | DataPktsOut | int(11) | YES | | NULL | |
> | AckPktsOut | int(11) | YES | | NULL | |
> | CurrentMSS | int(11) | YES | | NULL | |
> | DupAcksIn | int(11) | YES | | NULL | |
> | AckPktsIn | int(11) | YES | | NULL | |
> | MaxRwinRcvd | int(11) | YES | | NULL | |
> | Sndbuf | int(11) | YES | | NULL | |
> | MaxCwnd | int(11) | YES | | NULL | |
> | SndLimTimeRwin | int(11) | YES | | NULL | |
> | SndLimTimeCwnd | int(11) | YES | | NULL | |
> | SndLimTimeSender | int(11) | YES | | NULL | |
> | DataBytesOut | int(11) | YES | | NULL | |
> | SndLimTransRwin | int(11) | YES | | NULL | |
> | SndLimTransCwnd | int(11) | YES | | NULL | |
> | SndLimTransSender | int(11) | YES | | NULL | |
> | MaxSsthresh | int(11) | YES | | NULL | |
> | CurrentRTO | int(11) | YES | | NULL | |
> | CurrentRwinRcvd | int(11) | YES | | NULL | |
> | link | int(11) | YES | | NULL | |
> | mismatch | int(11) | YES | | NULL | |
> | bad_cable | int(11) | YES | | NULL | |
> | half_duplex | int(11) | YES | | NULL | |
> | congestion | int(11) | YES | | NULL | |
> | c2sdata | int(11) | YES | | NULL | |
> | c2sack | int(11) | YES | | NULL | |
> | s2cdata | int(11) | YES | | NULL | |
> | s2cack | int(11) | YES | | NULL | |
> | CongestionSignals | int(11) | YES | | NULL | |
> | PktsOut | int(11) | YES | | NULL | |
> | MinRTT | int(11) | YES | | NULL | |
> | RcvWinScale | int(11) | YES | | NULL | |
> | autotune | int(11) | YES | | NULL | |
> | CongAvoid | int(11) | YES | | NULL | |
> | CongestionOverCount | int(11) | YES | | NULL | |
> | MaxRTT | int(11) | YES | | NULL | |
> | OtherReductions | int(11) | YES | | NULL | |
> | CurTimeoutCount | int(11) | YES | | NULL | |
> | AbruptTimeouts | int(11) | YES | | NULL | |
> | SendStall | int(11) | YES | | NULL | |
> | SlowStart | int(11) | YES | | NULL | |
> | SubsequentTimeouts | int(11) | YES | | NULL | |
> | ThruBytesAcked | int(11) | YES | | NULL | |
> | minPeak | int(11) | YES | | NULL | |
> | maxPeak | int(11) | YES | | NULL | |
> | peaks | int(11) | YES | | NULL | |
> +---------------------+---------+------+-----+---------+-------+
> 68 rows in set (0.01 sec)


Anyone out there using/used this feature care to comment on it's
stability and/or any known issues?

-tt


--
Tom Throckmorton
Sr. Network Management Engineer
MCNC - Advanced Services Development
3021 Cornwallis Road
Research Triangle Park, NC 27709
919.248.1448

"Connecting North Carolina's future today"

Archive powered by MHonArc 2.6.16.

Top of Page