ndt-users - Re: Error running web100 3.5.0
Subject: ndt-users list created
List archive
- From: Chris Welti <>
- To: Richard Carlson <>
- Cc: Galuschka Christoph <>, "" <>
- Subject: Re: Error running web100 3.5.0
- Date: Wed, 02 Sep 2009 09:05:41 +0200
- Organization: SWITCH
yes, I can confirm that:
>>> send_msg: type=5, len=19
pcap_loop exited SIOCGSTAMP: Bad file descriptor
Pkt-Pair data collection ended, waiting for signal to terminate process
Signal 11 received by process 25555
Signal 17 received by process 25442
Protocol error!
>>> send_msg: type=7, len=61
S2C throughput test FAILED!
Finished testing C2S = 91.38 Mbps, S2C = 0.00 Mbps
Also there seemed to be alot of problems reading the variables, but that
might be a problem of the web100 kernel?
libweb100: warning: accessing depricated variable AckPktsIn
Variable 0 (AckPktsIn): web100_snap_read(): invalid arguments
libweb100: warning: accessing depricated variable AckPktsOut
Variable 1 (AckPktsOut): web100_snap_read(): invalid arguments
Variable 2 (BytesRetrans): web100_snap_read(): invalid arguments
Variable 3 (CongAvoid): web100_snap_read(): invalid arguments
Variable 4 (CongestionOverCount): web100_snap_read(): invalid arguments
Variable 5 (CongestionSignals): web100_snap_read(): invalid arguments
Variable 6 (CountRTT): web100_snap_read(): invalid arguments
Variable 7 (CurCwnd): web100_snap_read(): invalid arguments
>>> send_msg: type=5, len=13
Variable 9 (CurRTO): web100_snap_read(): invalid arguments
Variable 10 (CurRwinRcvd): web100_snap_read(): invalid arguments
Variable 11 (CurRwinSent): web100_snap_read(): invalid arguments
Variable 12 (CurSsthresh): web100_snap_read(): invalid arguments
Variable 13 (CwndRestores) not found in KIS
Variable 14 (DSACKDups): web100_snap_read(): invalid arguments
Variable 15 (DataBytesIn): web100_snap_read(): invalid arguments
Variable 16 (DataBytesOut): web100_snap_read(): invalid arguments
Variable 17 (DataPktsIn): web100_snap_read(): invalid arguments
Variable 18 (DataPktsOut): web100_snap_read(): invalid arguments
Variable 19 (DupAcksIn): web100_snap_read(): invalid arguments
Variable 20 (ECNEnabled): web100_snap_read(): invalid arguments
Variable 21 (FastRetran): web100_snap_read(): invalid arguments
Variable 22 (MaxCaCwnd) not found in KIS
Variable 23 (MaxCwnd): web100_snap_read(): invalid arguments
Variable 24 (MaxMSS): web100_snap_read(): invalid arguments
Variable 25 (MaxRTO): web100_snap_read(): invalid arguments
Variable 26 (MaxRTT): web100_snap_read(): invalid arguments
Variable 27 (MaxRwinRcvd): web100_snap_read(): invalid arguments
Variable 28 (MaxRwinSent): web100_snap_read(): invalid arguments
Variable 29 (MaxSsthresh): web100_snap_read(): invalid arguments
Variable 30 (MaxSaCwnd) not found in KIS
Variable 31 (MinMSS): web100_snap_read(): invalid arguments
Variable 32 (MinRTO): web100_snap_read(): invalid arguments
Variable 33 (MinRTT): web100_snap_read(): invalid arguments
Variable 34 (MinRwinRcvd): web100_snap_read(): invalid arguments
Variable 35 (MinRwinSent): web100_snap_read(): invalid arguments
Variable 36 (NagleEnabled): web100_snap_read(): invalid arguments
Variable 37 (OtherReductions): web100_snap_read(): invalid arguments
Variable 38 (PktsIn): web100_snap_read(): invalid arguments
Variable 39 (PktsOut): web100_snap_read(): invalid arguments
Variable 40 (PktsRetrans): web100_snap_read(): invalid arguments
>>> send_msg: type=5, len=17
Variable 42 (RcvWinScale): web100_snap_read(): invalid arguments
Variable 43 (SACKEnabled): web100_snap_read(): invalid arguments
Variable 44 (SACKsRcvd): web100_snap_read(): invalid arguments
Variable 45 (SendStall): web100_snap_read(): invalid arguments
Variable 46 (SlowStart): web100_snap_read(): invalid arguments
Variable 47 (SampleRTT): web100_snap_read(): invalid arguments
Variable 48 (SmoothedRTT): web100_snap_read(): invalid arguments
>>> send_msg: type=5, len=18
Variable 50 (SndWinScale): web100_snap_read(): invalid arguments
Variable 51 (SndLimTimeRwin): web100_snap_read(): invalid arguments
Variable 52 (SndLimTimeCwnd): web100_snap_read(): invalid arguments
Variable 53 (SndLimTimeSender): web100_snap_read(): invalid arguments
Variable 54 (SndLimTransRwin): web100_snap_read(): invalid arguments
Variable 55 (SndLimTransCwnd): web100_snap_read(): invalid arguments
Variable 56 (SndLimTransSender): web100_snap_read(): invalid arguments
Variable 57 (SndLimBytesRwin): web100_snap_read(): invalid arguments
Variable 58 (SndLimBytesCwnd): web100_snap_read(): invalid arguments
Variable 59 (SndLimBytesSender): web100_snap_read(): invalid arguments
Variable 60 (SubsequentTimeouts): web100_snap_read(): invalid arguments
Variable 61 (SumRTT): web100_snap_read(): invalid arguments
Variable 62 (Timeouts): web100_snap_read(): invalid arguments
Variable 63 (TimestampsEnabled): web100_snap_read(): invalid arguments
Variable 64 (WinScaleRcvd): web100_snap_read(): invalid arguments
Variable 65 (WinScaleSent): web100_snap_read(): invalid arguments
Variable 66 (DupAcksOut): web100_snap_read(): invalid arguments
Variable 67 (StartTimeUsec): web100_snap_read(): invalid arguments
Variable 68 (Duration): web100_snap_read(): invalid arguments
Maybe that's also the reason that the test fails, since it can't read the
databytes in correctly.
I've included the whole server and client output as a text file, in case you
need more info.
Chris
Richard Carlson wrote:
> Hi Chris;
>
> Great. and can you confirm that you are getting test sig 11 faults by
> running the web100srv in debug mode (web100srv -ddddd).
>
> Rich
>
> On Sep 1, 2009, at 10:36 AM, Chris Welti wrote:
>
>> Hi Rich,
>>
>> It seems to be using 0.9.8-5:
>>
>> :
>> root@lsmp2[sbin];
>> dpkg -l | grep libpcap
>> ii libpcap-dev
>> 0.9.8-5 development library for libpcap
>> (transitional package)
>> ii libpcap0.8
>> 0.9.8-5 system interface for user-level packet capture
>> ii libpcap0.8-dev
>> 0.9.8-5 development library and header files for
>> libpcap0.8
>>
>> Regards,
>> Chris
>>
>> Richard Carlson wrote:
>>> Hi Chris;
>>>
>>> I pulled the 3.5.6 tarball down and tested it on 2 systems. It ran
>>> properly.
>>>
>>> Looking again at the trace Christoph sent I see the following:
>>>
>>> 690876 kbps outbound
>>>>>> send_msg: type=5, len=6
>>> Signal USR1(10) sent to child [3744]
>>> Signal 10 received by process 3744
>>> Received SIGUSRx signal terminating data collection loop for pid=3744
>>> Sending pkt-pair data back to parent on pipe 8, 9
>>> fwd.saddr = f006c0a:3451, rev.saddr = dd70b0a:3002
>>> 01:02:46.369960 10.108.0.15:3451 --> 10.11.215.13:3002 Collected
>>> pkt-pair data max = 334203
>>> 01:02:46.369960 10.11.215.13:3002 --> 10.108.0.15:3451 Collected
>>> pkt-pair data max = 81670
>>> 128 bytes read ' 1 0 0 0 19 12738 129915 1642 108553 334203 0 6705
>>> 219.82 0 0 0 1 0 7' from monitor pipe
>>> 128 bytes read ' 0 0 1 0 3 31 49754 81670 19693 11227 1 0 581.71 21 1
>>> 162358 0 1 7' from monitor pipe
>>>>>> send_msg: type=6, len=0
>>> Signal 11 received by process 3744
>>> Signal 17 received by process 3741
>>> <------------------------->
>>> <-- S2C throughput test -->
>>> [ test results and debugging info followed by]
>>> Signal USR2(12) sent to child [3746]
>>> Signal 12 received by process 3746
>>> Received SIGUSRx signal terminating data collection loop for pid=3746
>>> Sending pkt-pair data back to parent on pipe 8, 9
>>> fwd.saddr = dd70b0a:3003, rev.saddr = f006c0a:3461
>>> 01:02:56.724367 10.11.215.13:3003 --> 10.108.0.15:3461 Collected
>>> pkt-pair data max = 18667
>>> 01:02:56.724367 10.108.0.15:3461 --> 10.11.215.13:3003 Collected
>>> pkt-pair data max = 65475
>>> Read ' 1 0 0 0 4 661 18667 6971 5501 5721 0 5377 976.37 0 0 0 1 0 7'
>>> from monitor pipe
>>> Read ' 0 0 0 1 367 9334 40681 26321 35413 65475 39990 34285 663.83
>>> 39864 40036 171967 0 39990 7' from monitor pipe
>>> 550764 kbps inbound
>>> [ followed by the test results being sent to the client and ]
>>> Signal 11 received by process 3746
>>> Signal 17 received by process 3741
>>> Protocol error!
>>>>>> send_msg: type=7, len=61
>>> S2C throughput test FAILED!
>>>
>>> When I run tests on my servers I don't get the "Signal 11 receive by
>>> process xxxx" message. Instead I see the lines
>>> pcap_loop exited recvfrom: Bad file descriptor
>>> Pkt-Pair data collection ended, waiting for signal to terminate process
>>>
>>> Signal 11 is a SIGSEGV (invalid memory reference) and I did make changes
>>> to the pcap routines to automatically detect which interface the server
>>> is using.
>>>
>>> According to yum I have 0.9.8-3.fc10_x86_64 on my development system and
>>> 0.9.4 on my production server.
>>> Given the code runs on my servers, I need to ask what version of the
>>> pcap lib development package are you using?
>>>
>>> Rich
>>>
>>> On Sep 1, 2009, at 7:39 AM, Chris Welti wrote:
>>>
>>>> Hi Rich,
>>>>
>>>> I've just upgraded one of our NDT servers from 3.4.4 to 3.5.6.
>>>> I'm observing the same behaviour as listed in this thread:
>>>>
>>>> ** Starting test 1 of 1 **
>>>> Connected to: lsmp2 -- Using IPv4 address
>>>> Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
>>>> checking for firewalls . . . . . . . . . . . . . . . . . . . Done
>>>> running 10s outbound test (client-to-server [C2S]) . . . . . 91.78Mb/s
>>>> running 10s inbound test (server-to-client [S2C]) . . . . . . 85.01Mb/s
>>>> S2C throughput test: Received wrong type of the message
>>>> ERROR MSG: Server (S2C throughput test): Invalid S2C throughput
>>>> received
>>>> S2C throughput test FAILED!
>>>> Server unable to determine bottleneck link type.
>>>> Information: Other network traffic is congesting the link
>>>>
>>>> There are test results for both C2S and S2C tests, but an error is
>>>> reported anyway.
>>>> On the server there are also error messages (Protocol error! S2C
>>>> throughput test FAILED!)
>>>>
>>>> Anyway, I've tried to install all the 3.5. versions before and it
>>>> seems to me that this bug was introduced with v3.5.3.
>>>> Using v3.5.2 or v3.5.1 it works as a charm on the same system (debian,
>>>> lenny, 2.6.26-web100)
>>>> v3.5.3, v3.5.4, v3.5.5 and v3.5.6 all produce the same error output as
>>>> above.
>>>>
>>>> Maybe that helps... Regards,
>>>> Chris
>>>>
>>>> Galuschka Christoph wrote:
>>>>> Hello Richard,
>>>>>
>>>>> I'm currently running the tests with 2.6.30.5 and IE6. JAVA - if
>>>>> relevant - is 1.6.11.
>>>>> The funny thing also is, I do get results in the webbrowser, the
>>>>> client
>>>>> just doesn't finish correctly.
>>>>>
>>>>> I will repair the WAIT-comment thing on monday (sorry, I'm not the
>>>>> best
>>>>> in C)
>>>>>
>>>>> Regards
>>>>> Christoph
>>>>>
>>>>> -----------------------------------------
>>>>> Ing. Christoph Galuschka
>>>>>
>>>>> TIWAG-Tiroler Wasserkraft AG
>>>>> Bereich IT/Betrieb und Services
>>>>> Eduard-Wallnöfer-Platz 2
>>>>> 6010 Innsbruck
>>>>> T: +43 (0)50607 21832
>>>>> F: +43 (0)50607 41832
>>>>> www.tiroler-wasserkraft.at <http://www.tiroler-wasserkraft.at>
>>>>> -----------------------------------------
>>>>> Firmenbuchgericht Innsbruck, FN 44133b
>>>>> Sitz der Gesellschaft: Innsbruck
>>>>> DVR: 0164089
>>>>>
>>>>>>
>>>>>
>>>>> Richard Carlson
>>>>>
>>>>>
>>>>> <mailto:>
>>>>> 1000 Oakbrook Dr
>>>>> Ann Arbor, MI 48104
>>>>>
>>>>> P: 734-352-7043
>>>>> C: 630-251-4572
>>>>>
>>>>
>>>
>>> Richard Carlson
>>>
>>>
>>> <mailto:>
>>> 1000 Oakbrook Dr
>>> Ann Arbor, MI 48104
>>>
>>> P: 734-352-7043
>>> C: 630-251-4572
>>>
>>
>
> Richard Carlson
>
> 1000 Oakbrook Dr
> Ann Arbor, MI 48104
>
> P: 734-352-7043
> C: 630-251-4572
>
Web-Client Output:
==================
TCP/Web100 Network Diagnostic Tool v5.5.4
click START to begin
** Starting test 1 of 1 **
Connected to: lsmp2 -- Using IPv4 address
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
checking for firewalls . . . . . . . . . . . . . . . . . . . Done
running 10s outbound test (client-to-server [C2S]) . . . . . 91.37Mb/s
running 10s inbound test (server-to-client [S2C]) . . . . . . 85.91Mb/s
S2C throughput test: Received wrong type of the message
ERROR MSG: Server (S2C throughput test): Invalid S2C throughput received
S2C throughput test FAILED!
Server unable to determine bottleneck link type.
Information: Other network traffic is congesting the link
click START to re-test
Client Statistics:
==================
WEB100 Kernel Variables:
Client: localhost/127.0.0.1
CurMSS: 1448
X_Rcvbuf: 174760
X_Sndbuf: 4194304
AckPktsIn: 41862
AckPktsOut: 0
BytesRetrans: 159280
CongAvoid: 0
CongestionOverCount: 0
CongestionSignals: 24
CountRTT: 40434
CurCwnd: 49232
CurRTO: 204
CurRwinRcvd: 524280
CurRwinSent: 17920
CurSsthresh: 46336
DSACKDups: 0
DataBytesIn: 0
DataBytesOut: 109500832
DataPktsIn: 0
DataPktsOut: 75468
DupAcksIn: 1429
ECNEnabled: 0
FastRetran: 24
MaxCwnd: 317112
MaxMSS: 1448
MaxRTO: 216
MaxRTT: 20
MaxRwinRcvd: 524280
MaxRwinSent: 17920
MaxSsthresh: 222992
MinMSS: 1448
MinRTO: 204
MinRTT: 0
MinRwinRcvd: 519832
MinRwinSent: 17896
NagleEnabled: 1
OtherReductions: 24
PktsIn: 41862
PktsOut: 75468
PktsRetrans: 110
RcvWinScale: 8
SACKEnabled: 3
SACKsRcvd: 1526
SendStall: 0
SlowStart: 0
SampleRTT: 4
SmoothedRTT: 4
SndWinScale: 3
SndLimTimeRwin: 0
SndLimTimeCwnd: 1161224
SndLimTimeSender: 9038092
SndLimTransRwin: 0
SndLimTransCwnd: 3718
SndLimTransSender: 3719
SndLimBytesRwin: 0
SndLimBytesCwnd: 89539984
SndLimBytesSender: 19960848
SubsequentTimeouts: 0
SumRTT: 212452
Timeouts: 0
TimestampsEnabled: 1
WinScaleRcvd: 3
WinScaleSent: 8
DupAcksOut: 0
StartTimeUsec: 20077
Duration: 10199462
c2sData: -1
c2sAck: -1
s2cData: -1
s2cAck: -1
half_duplex: 0
link: 100
congestion: 1
bad_cable: 0
mismatch: 0
spd: 85.89
bw: 117.90
loss: 0.000318016
avgrtt: 5.25
waitsec: 0.00
timesec: 10.00
order: 0.0341
rwintime: 0.0000
sendtime: 0.8861
cwndtime: 0.1139
rwin: 3.9999
swin: 32.0000
cwin: 2.4194
rttsec: 0.005254
Sndbuf: 4194304
aspd: 0.00000
CWND-Limited: 228.12
minCWNDpeak: 40544
maxCWNDpeak: 295392
CWNDpeaks: 22
The theoretical network limit is 117.9 Mbps
The NDT server has a 2048.0 KByte buffer which limits the throughput to
6090.59 Mbps
Your PC/Workstation has a 511.0 KByte buffer which limits the throughput to
761.30 Mbps
The network based flow control limits the throughput to 460.48 Mbps
Client Data reports link is 'System Fault', Client Acks report link is
'System Fault'
Server Data reports link is 'System Fault', Server Acks report link is
'System Fault'
More details:
=============
WEB100 Kernel Variables:
Client: localhost/127.0.0.1
CurMSS: 1448
X_Rcvbuf: 174760
X_Sndbuf: 4194304
AckPktsIn: 41862
AckPktsOut: 0
BytesRetrans: 159280
CongAvoid: 0
CongestionOverCount: 0
CongestionSignals: 24
CountRTT: 40434
CurCwnd: 49232
CurRTO: 204
CurRwinRcvd: 524280
CurRwinSent: 17920
CurSsthresh: 46336
DSACKDups: 0
DataBytesIn: 0
DataBytesOut: 109500832
DataPktsIn: 0
DataPktsOut: 75468
DupAcksIn: 1429
ECNEnabled: 0
FastRetran: 24
MaxCwnd: 317112
MaxMSS: 1448
MaxRTO: 216
MaxRTT: 20
MaxRwinRcvd: 524280
MaxRwinSent: 17920
MaxSsthresh: 222992
MinMSS: 1448
MinRTO: 204
MinRTT: 0
MinRwinRcvd: 519832
MinRwinSent: 17896
NagleEnabled: 1
OtherReductions: 24
PktsIn: 41862
PktsOut: 75468
PktsRetrans: 110
RcvWinScale: 8
SACKEnabled: 3
SACKsRcvd: 1526
SendStall: 0
SlowStart: 0
SampleRTT: 4
SmoothedRTT: 4
SndWinScale: 3
SndLimTimeRwin: 0
SndLimTimeCwnd: 1161224
SndLimTimeSender: 9038092
SndLimTransRwin: 0
SndLimTransCwnd: 3718
SndLimTransSender: 3719
SndLimBytesRwin: 0
SndLimBytesCwnd: 89539984
SndLimBytesSender: 19960848
SubsequentTimeouts: 0
SumRTT: 212452
Timeouts: 0
TimestampsEnabled: 1
WinScaleRcvd: 3
WinScaleSent: 8
DupAcksOut: 0
StartTimeUsec: 20077
Duration: 10199462
c2sData: -1
c2sAck: -1
s2cData: -1
s2cAck: -1
half_duplex: 0
link: 100
congestion: 1
bad_cable: 0
mismatch: 0
spd: 85.89
bw: 117.90
loss: 0.000318016
avgrtt: 5.25
waitsec: 0.00
timesec: 10.00
order: 0.0341
rwintime: 0.0000
sendtime: 0.8861
cwndtime: 0.1139
rwin: 3.9999
swin: 32.0000
cwin: 2.4194
rttsec: 0.005254
Sndbuf: 4194304
aspd: 0.00000
CWND-Limited: 228.12
minCWNDpeak: 40544
maxCWNDpeak: 295392
CWNDpeaks: 22
The theoretical network limit is 117.9 Mbps
The NDT server has a 2048.0 KByte buffer which limits the throughput to
6090.59 Mbps
Your PC/Workstation has a 511.0 KByte buffer which limits the throughput to
761.30 Mbps
The network based flow control limits the throughput to 460.48 Mbps
Client Data reports link is 'System Fault', Client Acks report link is
'System Fault'
Server Data reports link is 'System Fault', Server Acks report link is
'System Fault'
:
root@lsmp2[ndt-3.5.6];
/usr/local/sbin/web100srv -a --ipv4 -i eth0 -ddddd
ANL/Internet2 NDT ver 3.5.6
Variables file = /usr/local/ndt/web100_variables
log file = /usr/local/ndt/web100srv.log
Admin file = /usr/local/ndt/admin.html
Debug level set to 5
Send buffer initialized to 4194304, Receive buffer initialized to 174760
server ready on port 3001
web100_init() read 69 variables from file
Web100 variables line received
Initial counter Values Totalcnt = 0, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 1, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 1, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 2, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 2, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 3, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 3, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 4, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [3, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 4, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 5, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [3, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 5, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 6, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [3, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 6, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 7, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [4, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 7, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 8, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [4, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 8, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 9, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [4, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 9, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 10, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 10, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 11, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [6, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 11, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 12, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [7, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 12, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 13, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [7, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 13, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 14, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [7, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 14, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 15, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [8, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 15, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 16, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [9, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 16, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 17, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [10, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 17, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 18, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [11, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 18, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 19, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [12, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 19, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 20, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [12, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 20, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 21, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [12, 0, 1, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 21, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 22, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [12, 0, 1, 0, 0, 1, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 22, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 23, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [12, 0, 1, 0, 0, 2, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 23, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 24, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [13, 0, 1, 0, 0, 2, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Signal 17 received by process 25244
successfully locked '/tmp/view.string' for updating
sending '86147,0,972950,36,24,0,0,13,0,1,0,0,2,8,0,0,0,0,0,0,0,0,0,Sep 1
12:19:38,Sep 1 15:32:35' to tmp file
Generated iflist with device=eth0 and if_speed=7
Generated iflist with device=eth1 and if_speed=7
nothing in queue
Timer not running, waiting for new connection
New connection received, waiting for accept() to complete
New connection received from [130.59.4.118].
Parent process spawned child = 25442
Parent thinks pipe() returned fd0=7, fd1=8
<<< recv_msg: type=2, len=1
initialize_tests returned old/new client = 0, test_suite = 1 8 2 4
Incrementing waiting variable now = 1
Telling client 25442 testing will begin now
>>> send_msg: type=1, len=1
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
Child thinks pipe() returned fd0=7, fd1=8 for pid=0
extracting test_suite ' 1 8 2 4' and t_opts 'f' from buff 'go 15 1 8 2 4'
Child process 25442 started
>>> send_msg: type=2, len=6
run_test() routine, asking for test_suite = 1 8 2 4
>>> send_msg: type=2, len=8
Starting test suite:
> Middlebox test
> Simple firewall test
> C2S throughput test
> S2C throughput test
<-- Middlebox test -->
KillHung(): returned non-0 response, nothing to kill or kill failed
Send buffer initialized to 4194304, Receive buffer initialized to 174760
-- port: 3003
>>> send_msg: type=3, len=4
Looking for Web100 data on socketid 7
Server: 130.59.35.42;3003 Client: 130.59.4.118;50492 1444;8;8;
Sending 1444 Byte packets over the network
Setting Cwnd Limit to 2888 octets
Signal 17 received by process 25442
now = 1251874507 Process started at 1251874504, run time = 3
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
>>> send_msg: type=5, len=35
<<< recv_msg: type=5, len=17
CWND limited throughput = 228 kbps (228.1216865326454)
>>> send_msg: type=6, len=0
<-------------------->
<-- Simple firewall test -->
Send buffer initialized to 4194304, Receive buffer initialized to 174760
-- port: 58595
-- time: 1
>>> send_msg: type=3, len=7
<<< recv_msg: type=5, len=5
-- oport: 50493
>>> send_msg: type=4, len=0
Send buffer initialized to 4194304, Receive buffer initialized to 174760
<<< recv_msg: type=5, len=20
>>> send_msg: type=5, len=1
now = 1251874510 Process started at 1251874504, run time = 6
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
SIGALRM was caught
SIGALRM was caught
Unable to create connect socket.
>>> send_msg: type=6, len=0
<-------------------------->
<-- C2S throughput test -->
Send buffer initialized to 4194304, Receive buffer initialized to 174760
-- port: 3002
listening for Inet connection on testOptions->c2ssockfd, fd=3
Sending 'GO' signal, to tell client to head for the next test
>>> send_msg: type=3, len=4
C2S test Child thinks pipe() returned fd0=9, fd1=10
Opening network interface 'eth0' for packet-pair timing
pcap_open_live() returned pointer 0x8bcf1a8
installing pkt filter for 'host 130.59.4.118 and port 50495'
Initial pkt src data = 806a6e4
C2S test Parent thinks pipe() returned fd0=9, fd1=10
now = 1251874513 Process started at 1251874504, run time = 9
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
>>> send_msg: type=4, len=0
New IPv4 packet trace started -- initializing counters
now = 1251874516 Process started at 1251874504, run time = 12
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
now = 1251874519 Process started at 1251874504, run time = 15
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
now = 1251874522 Process started at 1251874504, run time = 18
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
91376 kbps outbound
>>> send_msg: type=5, len=5
Signal USR1(10) sent to child [25513]
Signal 10 received by process 25513
Received SIGUSRx signal terminating data collection loop for pid=25513
Sending pkt-pair data back to parent on pipe 9, 10
fwd.saddr = 0:0, rev.saddr = 0:0
00:55:13.953325 0.0.0.0:0 --> 0.0.0.0:0 No data Packets collected
Collected pkt-pair data max = 0
128 bytes read ' -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0.00 0 0 0 0 0 0' from
monitor pipe
00:55:13.953325 0.0.0.0:0 --> 0.0.0.0:0 No data Packets collected
Collected pkt-pair data max = 0
128 bytes read ' -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0.00 0 0 0 0 0 0' from
monitor pipe
>>> send_msg: type=6, len=0
pcap_loop exited SIOCGSTAMP: Bad file descriptor
Pkt-Pair data collection ended, waiting for signal to terminate process
Signal 11 received by process 25513
Signal 17 received by process 25442
<------------------------->
<-- S2C throughput test -->
Send buffer initialized to 4194304, Receive buffer initialized to 174760
-- port: 3003
>>> send_msg: type=3, len=4
waiting for data on testOptions->s2csockfd
S2C test Child thinks pipe() returned fd0=9, fd1=10
Opening network interface 'eth0' for packet-pair timing
pcap_open_live() returned pointer 0x8bd80d0
installing pkt filter for 'host 130.59.4.118 and port 50496'
Initial pkt src data = 806a6e4
Signal 17 received by process 25442
>>> send_msg: type=4, len=0
New IPv4 packet trace started -- initializing counters
now = 1251874525 Process started at 1251874504, run time = 21
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
now = 1251874528 Process started at 1251874504, run time = 24
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
now = 1251874531 Process started at 1251874504, run time = 27
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
now = 1251874534 Process started at 1251874504, run time = 30
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
>>> send_msg: type=5, len=23
sent 111820800 bytes to client in 10.13 seconds
Buffer control counters Total = 13650, new data = 0, Draining Queue = 0
Signal USR2(12) sent to child [25555]
Signal 12 received by process 25555
Received SIGUSRx signal terminating data collection loop for pid=25555
Sending pkt-pair data back to parent on pipe 9, 10
fwd.saddr = 0:0, rev.saddr = 0:0
00:55:24.091838 0.0.0.0:0 --> 0.0.0.0:0 No data Packets collected
Collected pkt-pair data max = 0
Read ' -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0.00 0 0 0 0 0 0' from monitor
pipe
00:55:24.091838 0.0.0.0:0 --> 0.0.0.0:0 No data Packets collected
Collected pkt-pair data max = 0
Read ' -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0.00 0 0 0 0 0 0' from monitor
pipe
88333 kbps inbound
libweb100: warning: accessing depricated variable AckPktsIn
Variable 0 (AckPktsIn): web100_snap_read(): invalid arguments
libweb100: warning: accessing depricated variable AckPktsOut
Variable 1 (AckPktsOut): web100_snap_read(): invalid arguments
Variable 2 (BytesRetrans): web100_snap_read(): invalid arguments
Variable 3 (CongAvoid): web100_snap_read(): invalid arguments
Variable 4 (CongestionOverCount): web100_snap_read(): invalid arguments
Variable 5 (CongestionSignals): web100_snap_read(): invalid arguments
Variable 6 (CountRTT): web100_snap_read(): invalid arguments
Variable 7 (CurCwnd): web100_snap_read(): invalid arguments
>>> send_msg: type=5, len=13
Variable 9 (CurRTO): web100_snap_read(): invalid arguments
Variable 10 (CurRwinRcvd): web100_snap_read(): invalid arguments
Variable 11 (CurRwinSent): web100_snap_read(): invalid arguments
Variable 12 (CurSsthresh): web100_snap_read(): invalid arguments
Variable 13 (CwndRestores) not found in KIS
Variable 14 (DSACKDups): web100_snap_read(): invalid arguments
Variable 15 (DataBytesIn): web100_snap_read(): invalid arguments
Variable 16 (DataBytesOut): web100_snap_read(): invalid arguments
Variable 17 (DataPktsIn): web100_snap_read(): invalid arguments
Variable 18 (DataPktsOut): web100_snap_read(): invalid arguments
Variable 19 (DupAcksIn): web100_snap_read(): invalid arguments
Variable 20 (ECNEnabled): web100_snap_read(): invalid arguments
Variable 21 (FastRetran): web100_snap_read(): invalid arguments
Variable 22 (MaxCaCwnd) not found in KIS
Variable 23 (MaxCwnd): web100_snap_read(): invalid arguments
Variable 24 (MaxMSS): web100_snap_read(): invalid arguments
Variable 25 (MaxRTO): web100_snap_read(): invalid arguments
Variable 26 (MaxRTT): web100_snap_read(): invalid arguments
Variable 27 (MaxRwinRcvd): web100_snap_read(): invalid arguments
Variable 28 (MaxRwinSent): web100_snap_read(): invalid arguments
Variable 29 (MaxSsthresh): web100_snap_read(): invalid arguments
Variable 30 (MaxSaCwnd) not found in KIS
Variable 31 (MinMSS): web100_snap_read(): invalid arguments
Variable 32 (MinRTO): web100_snap_read(): invalid arguments
Variable 33 (MinRTT): web100_snap_read(): invalid arguments
Variable 34 (MinRwinRcvd): web100_snap_read(): invalid arguments
Variable 35 (MinRwinSent): web100_snap_read(): invalid arguments
Variable 36 (NagleEnabled): web100_snap_read(): invalid arguments
Variable 37 (OtherReductions): web100_snap_read(): invalid arguments
Variable 38 (PktsIn): web100_snap_read(): invalid arguments
Variable 39 (PktsOut): web100_snap_read(): invalid arguments
Variable 40 (PktsRetrans): web100_snap_read(): invalid arguments
>>> send_msg: type=5, len=17
Variable 42 (RcvWinScale): web100_snap_read(): invalid arguments
Variable 43 (SACKEnabled): web100_snap_read(): invalid arguments
Variable 44 (SACKsRcvd): web100_snap_read(): invalid arguments
Variable 45 (SendStall): web100_snap_read(): invalid arguments
Variable 46 (SlowStart): web100_snap_read(): invalid arguments
Variable 47 (SampleRTT): web100_snap_read(): invalid arguments
Variable 48 (SmoothedRTT): web100_snap_read(): invalid arguments
>>> send_msg: type=5, len=18
Variable 50 (SndWinScale): web100_snap_read(): invalid arguments
Variable 51 (SndLimTimeRwin): web100_snap_read(): invalid arguments
Variable 52 (SndLimTimeCwnd): web100_snap_read(): invalid arguments
Variable 53 (SndLimTimeSender): web100_snap_read(): invalid arguments
Variable 54 (SndLimTransRwin): web100_snap_read(): invalid arguments
Variable 55 (SndLimTransCwnd): web100_snap_read(): invalid arguments
Variable 56 (SndLimTransSender): web100_snap_read(): invalid arguments
Variable 57 (SndLimBytesRwin): web100_snap_read(): invalid arguments
Variable 58 (SndLimBytesCwnd): web100_snap_read(): invalid arguments
Variable 59 (SndLimBytesSender): web100_snap_read(): invalid arguments
Variable 60 (SubsequentTimeouts): web100_snap_read(): invalid arguments
Variable 61 (SumRTT): web100_snap_read(): invalid arguments
Variable 62 (Timeouts): web100_snap_read(): invalid arguments
Variable 63 (TimestampsEnabled): web100_snap_read(): invalid arguments
Variable 64 (WinScaleRcvd): web100_snap_read(): invalid arguments
Variable 65 (WinScaleSent): web100_snap_read(): invalid arguments
Variable 66 (DupAcksOut): web100_snap_read(): invalid arguments
Variable 67 (StartTimeUsec): web100_snap_read(): invalid arguments
Variable 68 (Duration): web100_snap_read(): invalid arguments
>>> send_msg: type=5, len=17
>>> send_msg: type=5, len=14
>>> send_msg: type=5, len=21
>>> send_msg: type=5, len=13
>>> send_msg: type=5, len=23
>>> send_msg: type=5, len=22
>>> send_msg: type=5, len=16
>>> send_msg: type=5, len=15
Variable 8 (CurMSS): web100_snap_read(): invalid arguments
>>> send_msg: type=5, len=12
>>> send_msg: type=5, len=20
>>> send_msg: type=5, len=19
>>> send_msg: type=5, len=19
Variable 13 (CwndRestores) not found in KIS
>>> send_msg: type=5, len=13
>>> send_msg: type=5, len=15
>>> send_msg: type=5, len=24
>>> send_msg: type=5, len=14
>>> send_msg: type=5, len=19
>>> send_msg: type=5, len=16
>>> send_msg: type=5, len=14
>>> send_msg: type=5, len=15
Variable 22 (MaxCaCwnd) not found in KIS
>>> send_msg: type=5, len=16
>>> send_msg: type=5, len=13
>>> send_msg: type=5, len=12
>>> send_msg: type=5, len=11
>>> send_msg: type=5, len=20
>>> send_msg: type=5, len=19
>>> send_msg: type=5, len=20
Variable 30 (MaxSaCwnd) not found in KIS
>>> send_msg: type=5, len=13
>>> send_msg: type=5, len=12
>>> send_msg: type=5, len=10
>>> send_msg: type=5, len=20
>>> send_msg: type=5, len=19
>>> send_msg: type=5, len=16
>>> send_msg: type=5, len=20
>>> send_msg: type=5, len=14
>>> send_msg: type=5, len=15
>>> send_msg: type=5, len=17
Variable 41 (X_Rcvbuf): web100_snap_read(): invalid arguments
>>> send_msg: type=5, len=15
>>> send_msg: type=5, len=15
>>> send_msg: type=5, len=16
>>> send_msg: type=5, len=13
>>> send_msg: type=5, len=13
>>> send_msg: type=5, len=13
>>> send_msg: type=5, len=15
Variable 49 (X_Sndbuf): web100_snap_read(): invalid arguments
>>> send_msg: type=5, len=15
>>> send_msg: type=5, len=18
>>> send_msg: type=5, len=24
>>> send_msg: type=5, len=26
>>> send_msg: type=5, len=19
>>> send_msg: type=5, len=22
>>> send_msg: type=5, len=24
>>> send_msg: type=5, len=19
>>> send_msg: type=5, len=26
>>> send_msg: type=5, len=28
>>> send_msg: type=5, len=22
>>> send_msg: type=5, len=15
>>> send_msg: type=5, len=12
>>> send_msg: type=5, len=21
>>> send_msg: type=5, len=16
>>> send_msg: type=5, len=16
>>> send_msg: type=5, len=14
>>> send_msg: type=5, len=21
>>> send_msg: type=5, len=19
pcap_loop exited SIOCGSTAMP: Bad file descriptor
Pkt-Pair data collection ended, waiting for signal to terminate process
Signal 11 received by process 25555
Signal 17 received by process 25442
Protocol error!
>>> send_msg: type=7, len=61
S2C throughput test FAILED!
Finished testing C2S = 91.38 Mbps, S2C = 0.00 Mbps
Client --> Server data detects link = System Fault
Client <-- Server Ack's detect link = System Fault
Server --> Client data detects link = System Fault
Server <-- Client Ack's detect link = System Fault
CWND limited test = 228.12 while unlimited = 0.00
Better throughput when CWND is limited, may be duplex mismatch
>>> send_msg: type=8, len=46
>>> send_msg: type=8, len=75
>>> send_msg: type=8, len=85
>>> send_msg: type=8, len=78
>>> send_msg: type=8, len=81
>>> send_msg: type=8, len=53
>>> send_msg: type=9, len=0
now = 1251874537 Process started at 1251874504, run time = 33
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
Opened
'/usr/local/ndt/serverdata/2009/09/02/20090902T06:55:04.153056000Z_130.59.4.118:50491.meta'
metadata log file
Updating admin_view variables: Total count = 24
Initial counter Values Totalcnt = 24, Total Mismatch = 0, Total Bad Cables = 0
Updated counter values Totalcnt = 25, Total Mismatch = 0, Total Bad Cables = 0
Individual counts = [14, 0, 1, 0, 0, 2, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Signal 17 received by process 25442
successfully locked '/tmp/view.string' for updating
sending '91375,0,972950,0,25,0,0,14,0,1,0,0,2,8,0,0,0,0,0,0,0,0,0,Sep 2
06:55:04al (6) received,Sep 2 06:55:04al (6) received' to tmp file
Successfully returned from run_test() routine
Signal 17 received by process 25244
now = 1251874539 Process started at 1251874504, run time = 35
Select exited with rc = -1
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Received SIGCHLD signal for active web100srv process [25244]
wait3() returned 0 for PID=25442
wexitstatus = '0'
Attempting to clean up child 25442, head pid = 25442
Child process 25442 causing head pointer modification
Removing Child from head, decrementing waiting now = 0
Timer not running, waiting for new connection
- Re: Error running web100 3.5.0, Chris Welti, 09/01/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/01/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/01/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/01/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/01/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/02/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/02/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/02/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/02/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/02/2009
- AW: Error running web100 3.5.0, Galuschka Christoph, 09/02/2009
- Re: AW: Error running web100 3.5.0, Richard Carlson, 09/04/2009
- AW: AW: Error running web100 3.5.0, Galuschka Christoph, 09/07/2009
- Re: AW: Error running web100 3.5.0, Chris Welti, 09/11/2009
- Re: AW: Error running web100 3.5.0, Richard Carlson, 09/11/2009
- RE: AW: Error running web100 3.5.0, Galuschka Christoph, 09/11/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/02/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/02/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/02/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/01/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/01/2009
Archive powered by MHonArc 2.6.16.