ndt-users - Re: Error running web100 3.5.0
Subject: ndt-users list created
List archive
- From: Richard Carlson <>
- To: Chris Welti <>
- Cc: Galuschka Christoph <>, "" <>
- Subject: Re: Error running web100 3.5.0
- Date: Wed, 2 Sep 2009 10:00:42 -0500
Hi Chris;
Drat. OK, please run the server in debug mode level 6 (-dddddd) and send me that output. That may tell me something.
Rich
On Sep 2, 2009, at 9:00 AM, Chris Welti wrote:
Hi Rich,
I've added the if statement and recompiled, but that didn't seem to have any positive effect; I still get the same error message in the client.
Any other ideas?
Chris
Richard Carlson wrote:
Hi Chris & Christoph;
I'm not sure why I can't duplicate this bug, but in looking at the code
this is a possible solution.
Please manually edit the src/web100-pcap.c file and scroll down to line
922. You should see this snipit of code
log_println(5, "Pkt-Pair data collection ended, waiting for signal to
terminate process");
pcap_freealldevs(alldevs);
Please add the if statement so the snipitt look like this
log_println(5, "Pkt-Pair data collection ended, waiting for signal to
terminate process");
if (alldevs != NULL)
pcap_freealldevs(alldevs);
The log provides shows that this message is being printed just before
the fault is detected. If the alldevs value is NULL, then the attempt
to free it will cause this fault.
As for the messages about the web100 variables. The web100 variables
are grouped into a couple of major categories (read only and tuning are
2 of these categories). The NDT code collects variables in both of
these categories. When the s2c test completes the server walks through
the list of variables (in the tcpbw100.variables file) and retrieves the
values. There are 2 passes made to retrieve the values one pass through
the readonly category and a 2nd pass through the tuning category. Since
I pass the complete list to each category, you see lots of warnings when
I ask for a variable from the wrong category. To illustrate this
Variable 3 (CongAvoid): web100_snap_read(): invalid arguments
The CongAvoid variable is in the 'read' category. This error is
generated when the code looks for it in the 'tune' category. I'll put
this on the to-do list, but it will be a low priority so it's not a
functional problem, it just looks wrong when you turn on debugging.
Rich
On Sep 2, 2009, at 2:05 AM, Chris Welti wrote:
yes, I can confirm that:
pcap_loop exited SIOCGSTAMP: Bad file descriptorsend_msg: type=5, len=19
Pkt-Pair data collection ended, waiting for signal to terminate process
Signal 11 received by process 25555
Signal 17 received by process 25442
Protocol error!
S2C throughput test FAILED!send_msg: type=7, len=61
Finished testing C2S = 91.38 Mbps, S2C = 0.00 Mbps
Also there seemed to be alot of problems reading the variables, but
that might be a problem of the web100 kernel?
libweb100: warning: accessing depricated variable AckPktsIn
Variable 0 (AckPktsIn): web100_snap_read(): invalid arguments
libweb100: warning: accessing depricated variable AckPktsOut
Variable 1 (AckPktsOut): web100_snap_read(): invalid arguments
Variable 2 (BytesRetrans): web100_snap_read(): invalid arguments
Variable 3 (CongAvoid): web100_snap_read(): invalid arguments
Variable 4 (CongestionOverCount): web100_snap_read(): invalid arguments
Variable 5 (CongestionSignals): web100_snap_read(): invalid arguments
Variable 6 (CountRTT): web100_snap_read(): invalid arguments
Variable 7 (CurCwnd): web100_snap_read(): invalid arguments
Variable 9 (CurRTO): web100_snap_read(): invalid argumentssend_msg: type=5, len=13
Variable 10 (CurRwinRcvd): web100_snap_read(): invalid arguments
Variable 11 (CurRwinSent): web100_snap_read(): invalid arguments
Variable 12 (CurSsthresh): web100_snap_read(): invalid arguments
Variable 13 (CwndRestores) not found in KIS
Variable 14 (DSACKDups): web100_snap_read(): invalid arguments
Variable 15 (DataBytesIn): web100_snap_read(): invalid arguments
Variable 16 (DataBytesOut): web100_snap_read(): invalid arguments
Variable 17 (DataPktsIn): web100_snap_read(): invalid arguments
Variable 18 (DataPktsOut): web100_snap_read(): invalid arguments
Variable 19 (DupAcksIn): web100_snap_read(): invalid arguments
Variable 20 (ECNEnabled): web100_snap_read(): invalid arguments
Variable 21 (FastRetran): web100_snap_read(): invalid arguments
Variable 22 (MaxCaCwnd) not found in KIS
Variable 23 (MaxCwnd): web100_snap_read(): invalid arguments
Variable 24 (MaxMSS): web100_snap_read(): invalid arguments
Variable 25 (MaxRTO): web100_snap_read(): invalid arguments
Variable 26 (MaxRTT): web100_snap_read(): invalid arguments
Variable 27 (MaxRwinRcvd): web100_snap_read(): invalid arguments
Variable 28 (MaxRwinSent): web100_snap_read(): invalid arguments
Variable 29 (MaxSsthresh): web100_snap_read(): invalid arguments
Variable 30 (MaxSaCwnd) not found in KIS
Variable 31 (MinMSS): web100_snap_read(): invalid arguments
Variable 32 (MinRTO): web100_snap_read(): invalid arguments
Variable 33 (MinRTT): web100_snap_read(): invalid arguments
Variable 34 (MinRwinRcvd): web100_snap_read(): invalid arguments
Variable 35 (MinRwinSent): web100_snap_read(): invalid arguments
Variable 36 (NagleEnabled): web100_snap_read(): invalid arguments
Variable 37 (OtherReductions): web100_snap_read(): invalid arguments
Variable 38 (PktsIn): web100_snap_read(): invalid arguments
Variable 39 (PktsOut): web100_snap_read(): invalid arguments
Variable 40 (PktsRetrans): web100_snap_read(): invalid arguments
Variable 42 (RcvWinScale): web100_snap_read(): invalid argumentssend_msg: type=5, len=17
Variable 43 (SACKEnabled): web100_snap_read(): invalid arguments
Variable 44 (SACKsRcvd): web100_snap_read(): invalid arguments
Variable 45 (SendStall): web100_snap_read(): invalid arguments
Variable 46 (SlowStart): web100_snap_read(): invalid arguments
Variable 47 (SampleRTT): web100_snap_read(): invalid arguments
Variable 48 (SmoothedRTT): web100_snap_read(): invalid arguments
Variable 50 (SndWinScale): web100_snap_read(): invalid argumentssend_msg: type=5, len=18
Variable 51 (SndLimTimeRwin): web100_snap_read(): invalid arguments
Variable 52 (SndLimTimeCwnd): web100_snap_read(): invalid arguments
Variable 53 (SndLimTimeSender): web100_snap_read(): invalid arguments
Variable 54 (SndLimTransRwin): web100_snap_read(): invalid arguments
Variable 55 (SndLimTransCwnd): web100_snap_read(): invalid arguments
Variable 56 (SndLimTransSender): web100_snap_read(): invalid arguments
Variable 57 (SndLimBytesRwin): web100_snap_read(): invalid arguments
Variable 58 (SndLimBytesCwnd): web100_snap_read(): invalid arguments
Variable 59 (SndLimBytesSender): web100_snap_read(): invalid arguments
Variable 60 (SubsequentTimeouts): web100_snap_read(): invalid arguments
Variable 61 (SumRTT): web100_snap_read(): invalid arguments
Variable 62 (Timeouts): web100_snap_read(): invalid arguments
Variable 63 (TimestampsEnabled): web100_snap_read(): invalid arguments
Variable 64 (WinScaleRcvd): web100_snap_read(): invalid arguments
Variable 65 (WinScaleSent): web100_snap_read(): invalid arguments
Variable 66 (DupAcksOut): web100_snap_read(): invalid arguments
Variable 67 (StartTimeUsec): web100_snap_read(): invalid arguments
Variable 68 (Duration): web100_snap_read(): invalid arguments
Maybe that's also the reason that the test fails, since it can't read
the databytes in correctly.
I've included the whole server and client output as a text file, in
case you need more info.
Chris
Richard Carlson wrote:
Hi Chris;
Great. and can you confirm that you are getting test sig 11 faults by
running the web100srv in debug mode (web100srv -ddddd).
Rich
On Sep 1, 2009, at 10:36 AM, Chris Welti wrote:
Hi Rich,
It seems to be using 0.9.8-5:
:
root@lsmp2[sbin];
dpkg -l | grep libpcap
ii libpcap-dev
0.9.8-5 development library for libpcap
(transitional package)
ii libpcap0.8
0.9.8-5 system interface for user-level packet
capture
ii libpcap0.8-dev
0.9.8-5 development library and header files for
libpcap0.8
Regards,
Chris
Richard Carlson wrote:
Hi Chris;
I pulled the 3.5.6 tarball down and tested it on 2 systems. It ran
properly.
Looking again at the trace Christoph sent I see the following:
690876 kbps outbound
Signal USR1(10) sent to child [3744]send_msg: type=5, len=6
Signal 10 received by process 3744
Received SIGUSRx signal terminating data collection loop for pid=3744
Sending pkt-pair data back to parent on pipe 8, 9
fwd.saddr = f006c0a:3451, rev.saddr = dd70b0a:3002
01:02:46.369960 10.108.0.15:3451 --> 10.11.215.13:3002 Collected
pkt-pair data max = 334203
01:02:46.369960 10.11.215.13:3002 --> 10.108.0.15:3451 Collected
pkt-pair data max = 81670
128 bytes read ' 1 0 0 0 19 12738 129915 1642 108553 334203 0 6705
219.82 0 0 0 1 0 7' from monitor pipe
128 bytes read ' 0 0 1 0 3 31 49754 81670 19693 11227 1 0 581.71 21 1
162358 0 1 7' from monitor pipe
Signal 11 received by process 3744send_msg: type=6, len=0
Signal 17 received by process 3741
<------------------------->
<-- S2C throughput test -->
[ test results and debugging info followed by]
Signal USR2(12) sent to child [3746]
Signal 12 received by process 3746
Received SIGUSRx signal terminating data collection loop for pid=3746
Sending pkt-pair data back to parent on pipe 8, 9
fwd.saddr = dd70b0a:3003, rev.saddr = f006c0a:3461
01:02:56.724367 10.11.215.13:3003 --> 10.108.0.15:3461 Collected
pkt-pair data max = 18667
01:02:56.724367 10.108.0.15:3461 --> 10.11.215.13:3003 Collected
pkt-pair data max = 65475
Read ' 1 0 0 0 4 661 18667 6971 5501 5721 0 5377 976.37 0 0 0 1 0 7'
from monitor pipe
Read ' 0 0 0 1 367 9334 40681 26321 35413 65475 39990 34285 663.83
39864 40036 171967 0 39990 7' from monitor pipe
550764 kbps inbound
[ followed by the test results being sent to the client and ]
Signal 11 received by process 3746
Signal 17 received by process 3741
Protocol error!
S2C throughput test FAILED!send_msg: type=7, len=61
When I run tests on my servers I don't get the "Signal 11 receive by
process xxxx" message. Instead I see the lines
pcap_loop exited recvfrom: Bad file descriptor
Pkt-Pair data collection ended, waiting for signal to terminate
process
Signal 11 is a SIGSEGV (invalid memory reference) and I did make
changes
to the pcap routines to automatically detect which interface the
server
is using.
According to yum I have 0.9.8-3.fc10_x86_64 on my development
system and
0.9.4 on my production server.
Given the code runs on my servers, I need to ask what version of the
pcap lib development package are you using?
Rich
On Sep 1, 2009, at 7:39 AM, Chris Welti wrote:
Hi Rich,
I've just upgraded one of our NDT servers from 3.4.4 to 3.5.6.
I'm observing the same behaviour as listed in this thread:
** Starting test 1 of 1 **
Connected to: lsmp2 -- Using IPv4 address
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
checking for firewalls . . . . . . . . . . . . . . . . . . . Done
running 10s outbound test (client-to-server [C2S]) . . . . .
91.78Mb/s
running 10s inbound test (server-to-client [S2C]) . . . . . .
85.01Mb/s
S2C throughput test: Received wrong type of the message
ERROR MSG: Server (S2C throughput test): Invalid S2C throughput
received
S2C throughput test FAILED!
Server unable to determine bottleneck link type.
Information: Other network traffic is congesting the link
There are test results for both C2S and S2C tests, but an error is
reported anyway.
On the server there are also error messages (Protocol error! S2C
throughput test FAILED!)
Anyway, I've tried to install all the 3.5. versions before and it
seems to me that this bug was introduced with v3.5.3.
Using v3.5.2 or v3.5.1 it works as a charm on the same system
(debian,
lenny, 2.6.26-web100)
v3.5.3, v3.5.4, v3.5.5 and v3.5.6 all produce the same error
output as
above.
Maybe that helps... Regards,
Chris
Galuschka Christoph wrote:
Hello Richard,
I'm currently running the tests with 2.6.30.5 and IE6. JAVA - if
relevant - is 1.6.11.
The funny thing also is, I do get results in the webbrowser, the
client
just doesn't finish correctly.
I will repair the WAIT-comment thing on monday (sorry, I'm not the
best
in C)
Regards
Christoph
-----------------------------------------
Ing. Christoph Galuschka
TIWAG-Tiroler Wasserkraft AG
Bereich IT/Betrieb und Services
Eduard-Wallnöfer-Platz 2
6010 Innsbruck
T: +43 (0)50607 21832
F: +43 (0)50607 41832
www.tiroler-wasserkraft.at <http://www.tiroler-wasserkraft.at>
-----------------------------------------
Firmenbuchgericht Innsbruck, FN 44133b
Sitz der Gesellschaft: Innsbruck
DVR: 0164089
Richard Carlson
<mailto:>
1000 Oakbrook Dr
Ann Arbor, MI 48104
P: 734-352-7043
C: 630-251-4572
Richard Carlson
<mailto:>
1000 Oakbrook Dr
Ann Arbor, MI 48104
P: 734-352-7043
C: 630-251-4572
Richard Carlson
1000 Oakbrook Dr
Ann Arbor, MI 48104
P: 734-352-7043
C: 630-251-4572
Web-Client Output:
==================
TCP/Web100 Network Diagnostic Tool v5.5.4
click START to begin
** Starting test 1 of 1 **
Connected to: lsmp2 -- Using IPv4 address
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
checking for firewalls . . . . . . . . . . . . . . . . . . . Done
running 10s outbound test (client-to-server [C2S]) . . . . . 91.37Mb/s
running 10s inbound test (server-to-client [S2C]) . . . . . . 85.91Mb/s
S2C throughput test: Received wrong type of the message
ERROR MSG: Server (S2C throughput test): Invalid S2C throughput received
S2C throughput test FAILED!
Server unable to determine bottleneck link type.
Information: Other network traffic is congesting the link
click START to re-test
Client Statistics:
==================
WEB100 Kernel Variables:
Client: localhost/127.0.0.1
CurMSS: 1448
X_Rcvbuf: 174760
X_Sndbuf: 4194304
AckPktsIn: 41862
AckPktsOut: 0
BytesRetrans: 159280
CongAvoid: 0
CongestionOverCount: 0
CongestionSignals: 24
CountRTT: 40434
CurCwnd: 49232
CurRTO: 204
CurRwinRcvd: 524280
CurRwinSent: 17920
CurSsthresh: 46336
DSACKDups: 0
DataBytesIn: 0
DataBytesOut: 109500832
DataPktsIn: 0
DataPktsOut: 75468
DupAcksIn: 1429
ECNEnabled: 0
FastRetran: 24
MaxCwnd: 317112
MaxMSS: 1448
MaxRTO: 216
MaxRTT: 20
MaxRwinRcvd: 524280
MaxRwinSent: 17920
MaxSsthresh: 222992
MinMSS: 1448
MinRTO: 204
MinRTT: 0
MinRwinRcvd: 519832
MinRwinSent: 17896
NagleEnabled: 1
OtherReductions: 24
PktsIn: 41862
PktsOut: 75468
PktsRetrans: 110
RcvWinScale: 8
SACKEnabled: 3
SACKsRcvd: 1526
SendStall: 0
SlowStart: 0
SampleRTT: 4
SmoothedRTT: 4
SndWinScale: 3
SndLimTimeRwin: 0
SndLimTimeCwnd: 1161224
SndLimTimeSender: 9038092
SndLimTransRwin: 0
SndLimTransCwnd: 3718
SndLimTransSender: 3719
SndLimBytesRwin: 0
SndLimBytesCwnd: 89539984
SndLimBytesSender: 19960848
SubsequentTimeouts: 0
SumRTT: 212452
Timeouts: 0
TimestampsEnabled: 1
WinScaleRcvd: 3
WinScaleSent: 8
DupAcksOut: 0
StartTimeUsec: 20077
Duration: 10199462
c2sData: -1
c2sAck: -1
s2cData: -1
s2cAck: -1
half_duplex: 0
link: 100
congestion: 1
bad_cable: 0
mismatch: 0
spd: 85.89
bw: 117.90
loss: 0.000318016
avgrtt: 5.25
waitsec: 0.00
timesec: 10.00
order: 0.0341
rwintime: 0.0000
sendtime: 0.8861
cwndtime: 0.1139
rwin: 3.9999
swin: 32.0000
cwin: 2.4194
rttsec: 0.005254
Sndbuf: 4194304
aspd: 0.00000
CWND-Limited: 228.12
minCWNDpeak: 40544
maxCWNDpeak: 295392
CWNDpeaks: 22
The theoretical network limit is 117.9 Mbps
The NDT server has a 2048.0 KByte buffer which limits the throughput
to 6090.59 Mbps
Your PC/Workstation has a 511.0 KByte buffer which limits the
throughput to 761.30 Mbps
The network based flow control limits the throughput to 460.48 Mbps
Client Data reports link is 'System Fault', Client Acks report link is
'System Fault'
Server Data reports link is 'System Fault', Server Acks report link is
'System Fault'
More details:
=============
WEB100 Kernel Variables:
Client: localhost/127.0.0.1
CurMSS: 1448
X_Rcvbuf: 174760
X_Sndbuf: 4194304
AckPktsIn: 41862
AckPktsOut: 0
BytesRetrans: 159280
CongAvoid: 0
CongestionOverCount: 0
CongestionSignals: 24
CountRTT: 40434
CurCwnd: 49232
CurRTO: 204
CurRwinRcvd: 524280
CurRwinSent: 17920
CurSsthresh: 46336
DSACKDups: 0
DataBytesIn: 0
DataBytesOut: 109500832
DataPktsIn: 0
DataPktsOut: 75468
DupAcksIn: 1429
ECNEnabled: 0
FastRetran: 24
MaxCwnd: 317112
MaxMSS: 1448
MaxRTO: 216
MaxRTT: 20
MaxRwinRcvd: 524280
MaxRwinSent: 17920
MaxSsthresh: 222992
MinMSS: 1448
MinRTO: 204
MinRTT: 0
MinRwinRcvd: 519832
MinRwinSent: 17896
NagleEnabled: 1
OtherReductions: 24
PktsIn: 41862
PktsOut: 75468
PktsRetrans: 110
RcvWinScale: 8
SACKEnabled: 3
SACKsRcvd: 1526
SendStall: 0
SlowStart: 0
SampleRTT: 4
SmoothedRTT: 4
SndWinScale: 3
SndLimTimeRwin: 0
SndLimTimeCwnd: 1161224
SndLimTimeSender: 9038092
SndLimTransRwin: 0
SndLimTransCwnd: 3718
SndLimTransSender: 3719
SndLimBytesRwin: 0
SndLimBytesCwnd: 89539984
SndLimBytesSender: 19960848
SubsequentTimeouts: 0
SumRTT: 212452
Timeouts: 0
TimestampsEnabled: 1
WinScaleRcvd: 3
WinScaleSent: 8
DupAcksOut: 0
StartTimeUsec: 20077
Duration: 10199462
c2sData: -1
c2sAck: -1
s2cData: -1
s2cAck: -1
half_duplex: 0
link: 100
congestion: 1
bad_cable: 0
mismatch: 0
spd: 85.89
bw: 117.90
loss: 0.000318016
avgrtt: 5.25
waitsec: 0.00
timesec: 10.00
order: 0.0341
rwintime: 0.0000
sendtime: 0.8861
cwndtime: 0.1139
rwin: 3.9999
swin: 32.0000
cwin: 2.4194
rttsec: 0.005254
Sndbuf: 4194304
aspd: 0.00000
CWND-Limited: 228.12
minCWNDpeak: 40544
maxCWNDpeak: 295392
CWNDpeaks: 22
The theoretical network limit is 117.9 Mbps
The NDT server has a 2048.0 KByte buffer which limits the throughput
to 6090.59 Mbps
Your PC/Workstation has a 511.0 KByte buffer which limits the
throughput to 761.30 Mbps
The network based flow control limits the throughput to 460.48 Mbps
Client Data reports link is 'System Fault', Client Acks report link is
'System Fault'
Server Data reports link is 'System Fault', Server Acks report link is
'System Fault'
:
root@lsmp2[ndt-3.5.6];
/usr/local/sbin/web100srv -a --ipv4 -i eth0
-ddddd
ANL/Internet2 NDT ver 3.5.6
Variables file = /usr/local/ndt/web100_variables
log file = /usr/local/ndt/web100srv.log
Admin file = /usr/local/ndt/admin.html
Debug level set to 5
Send buffer initialized to 4194304, Receive buffer initialized to 174760
server ready on port 3001
web100_init() read 69 variables from file
Web100 variables line received
Initial counter Values Totalcnt = 0, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 1, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 1, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 2, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 2, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 3, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 3, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 4, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [3, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 4, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 5, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [3, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 5, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 6, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [3, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 6, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 7, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [4, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 7, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 8, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [4, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 8, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 9, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [4, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 9, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 10, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [5, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 10, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 11, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [6, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 11, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 12, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [7, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 12, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 13, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [7, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 13, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 14, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [7, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 14, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 15, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [8, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 15, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 16, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [9, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 16, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 17, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [10, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 17, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 18, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [11, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 18, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 19, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [12, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 19, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 20, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [12, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 20, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 21, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [12, 0, 1, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 21, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 22, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [12, 0, 1, 0, 0, 1, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 22, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 23, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [12, 0, 1, 0, 0, 2, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Web100 variables line received
Initial counter Values Totalcnt = 23, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 24, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [13, 0, 1, 0, 0, 2, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Signal 17 received by process 25244
successfully locked '/tmp/view.string' for updating
sending
'86147,0,972950,36,24,0,0,13,0,1,0,0,2,8,0,0,0,0,0,0,0,0,0,Sep 1
12:19:38,Sep 1 15:32:35' to tmp file
Generated iflist with device=eth0 and if_speed=7
Generated iflist with device=eth1 and if_speed=7
nothing in queue
Timer not running, waiting for new connection
New connection received, waiting for accept() to complete
New connection received from [130.59.4.118].
Parent process spawned child = 25442
Parent thinks pipe() returned fd0=7, fd1=8
<<< recv_msg: type=2, len=1
initialize_tests returned old/new client = 0, test_suite = 1 8 2 4
Incrementing waiting variable now = 1
Telling client 25442 testing will begin now
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0send_msg: type=1, len=1
Waiting for new connection, timer running
Child thinks pipe() returned fd0=7, fd1=8 for pid=0
extracting test_suite ' 1 8 2 4' and t_opts 'f' from buff 'go 15 1 8 2 4'
Child process 25442 started
run_test() routine, asking for test_suite = 1 8 2 4send_msg: type=2, len=6
Starting test suite:send_msg: type=2, len=8
Middlebox test<-- Middlebox test -->
Simple firewall test
C2S throughput test
S2C throughput test
KillHung(): returned non-0 response, nothing to kill or kill failed
Send buffer initialized to 4194304, Receive buffer initialized to 174760
-- port: 3003
Looking for Web100 data on socketid 7send_msg: type=3, len=4
Server: 130.59.35.42;3003 Client: 130.59.4.118;50492 1444;8;8;
Sending 1444 Byte packets over the network
Setting Cwnd Limit to 2888 octets
Signal 17 received by process 25442
now = 1251874507 Process started at 1251874504, run time = 3
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
<<< recv_msg: type=5, len=17send_msg: type=5, len=35
CWND limited throughput = 228 kbps (228.1216865326454)
<-------------------->send_msg: type=6, len=0
<-- Simple firewall test -->
Send buffer initialized to 4194304, Receive buffer initialized to 174760
-- port: 58595
-- time: 1
<<< recv_msg: type=5, len=5send_msg: type=3, len=7
-- oport: 50493
send_msg: type=4, len=0
Send buffer initialized to 4194304, Receive buffer initialized to 174760
<<< recv_msg: type=5, len=20
now = 1251874510 Process started at 1251874504, run time = 6send_msg: type=5, len=1
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
SIGALRM was caught
SIGALRM was caught
Unable to create connect socket.
<-------------------------->send_msg: type=6, len=0
<-- C2S throughput test -->
Send buffer initialized to 4194304, Receive buffer initialized to 174760
-- port: 3002
listening for Inet connection on testOptions->c2ssockfd, fd=3
Sending 'GO' signal, to tell client to head for the next test
C2S test Child thinks pipe() returned fd0=9, fd1=10send_msg: type=3, len=4
Opening network interface 'eth0' for packet-pair timing
pcap_open_live() returned pointer 0x8bcf1a8
installing pkt filter for 'host 130.59.4.118 and port 50495'
Initial pkt src data = 806a6e4
C2S test Parent thinks pipe() returned fd0=9, fd1=10
now = 1251874513 Process started at 1251874504, run time = 9
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
New IPv4 packet trace started -- initializing counterssend_msg: type=4, len=0
now = 1251874516 Process started at 1251874504, run time = 12
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
now = 1251874519 Process started at 1251874504, run time = 15
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
now = 1251874522 Process started at 1251874504, run time = 18
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
91376 kbps outbound
Signal USR1(10) sent to child [25513]send_msg: type=5, len=5
Signal 10 received by process 25513
Received SIGUSRx signal terminating data collection loop for pid=25513
Sending pkt-pair data back to parent on pipe 9, 10
fwd.saddr = 0:0, rev.saddr = 0:0
00:55:13.953325 0.0.0.0:0 --> 0.0.0.0:0 No data Packets collected
Collected pkt-pair data max = 0
128 bytes read ' -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0.00 0 0 0 0 0
0' from monitor pipe
00:55:13.953325 0.0.0.0:0 --> 0.0.0.0:0 No data Packets collected
Collected pkt-pair data max = 0
128 bytes read ' -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0.00 0 0 0 0 0
0' from monitor pipe
pcap_loop exited SIOCGSTAMP: Bad file descriptorsend_msg: type=6, len=0
Pkt-Pair data collection ended, waiting for signal to terminate process
Signal 11 received by process 25513
Signal 17 received by process 25442
<------------------------->
<-- S2C throughput test -->
Send buffer initialized to 4194304, Receive buffer initialized to 174760
-- port: 3003
waiting for data on testOptions->s2csockfdsend_msg: type=3, len=4
S2C test Child thinks pipe() returned fd0=9, fd1=10
Opening network interface 'eth0' for packet-pair timing
pcap_open_live() returned pointer 0x8bd80d0
installing pkt filter for 'host 130.59.4.118 and port 50496'
Initial pkt src data = 806a6e4
Signal 17 received by process 25442
New IPv4 packet trace started -- initializing counterssend_msg: type=4, len=0
now = 1251874525 Process started at 1251874504, run time = 21
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
now = 1251874528 Process started at 1251874504, run time = 24
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
now = 1251874531 Process started at 1251874504, run time = 27
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
now = 1251874534 Process started at 1251874504, run time = 30
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
sent 111820800 bytes to client in 10.13 secondssend_msg: type=5, len=23
Buffer control counters Total = 13650, new data = 0, Draining Queue = 0
Signal USR2(12) sent to child [25555]
Signal 12 received by process 25555
Received SIGUSRx signal terminating data collection loop for pid=25555
Sending pkt-pair data back to parent on pipe 9, 10
fwd.saddr = 0:0, rev.saddr = 0:0
00:55:24.091838 0.0.0.0:0 --> 0.0.0.0:0 No data Packets collected
Collected pkt-pair data max = 0
Read ' -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0.00 0 0 0 0 0 0' from
monitor pipe
00:55:24.091838 0.0.0.0:0 --> 0.0.0.0:0 No data Packets collected
Collected pkt-pair data max = 0
Read ' -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0.00 0 0 0 0 0 0' from
monitor pipe
88333 kbps inbound
libweb100: warning: accessing depricated variable AckPktsIn
Variable 0 (AckPktsIn): web100_snap_read(): invalid arguments
libweb100: warning: accessing depricated variable AckPktsOut
Variable 1 (AckPktsOut): web100_snap_read(): invalid arguments
Variable 2 (BytesRetrans): web100_snap_read(): invalid arguments
Variable 3 (CongAvoid): web100_snap_read(): invalid arguments
Variable 4 (CongestionOverCount): web100_snap_read(): invalid arguments
Variable 5 (CongestionSignals): web100_snap_read(): invalid arguments
Variable 6 (CountRTT): web100_snap_read(): invalid arguments
Variable 7 (CurCwnd): web100_snap_read(): invalid arguments
Variable 9 (CurRTO): web100_snap_read(): invalid argumentssend_msg: type=5, len=13
Variable 10 (CurRwinRcvd): web100_snap_read(): invalid arguments
Variable 11 (CurRwinSent): web100_snap_read(): invalid arguments
Variable 12 (CurSsthresh): web100_snap_read(): invalid arguments
Variable 13 (CwndRestores) not found in KIS
Variable 14 (DSACKDups): web100_snap_read(): invalid arguments
Variable 15 (DataBytesIn): web100_snap_read(): invalid arguments
Variable 16 (DataBytesOut): web100_snap_read(): invalid arguments
Variable 17 (DataPktsIn): web100_snap_read(): invalid arguments
Variable 18 (DataPktsOut): web100_snap_read(): invalid arguments
Variable 19 (DupAcksIn): web100_snap_read(): invalid arguments
Variable 20 (ECNEnabled): web100_snap_read(): invalid arguments
Variable 21 (FastRetran): web100_snap_read(): invalid arguments
Variable 22 (MaxCaCwnd) not found in KIS
Variable 23 (MaxCwnd): web100_snap_read(): invalid arguments
Variable 24 (MaxMSS): web100_snap_read(): invalid arguments
Variable 25 (MaxRTO): web100_snap_read(): invalid arguments
Variable 26 (MaxRTT): web100_snap_read(): invalid arguments
Variable 27 (MaxRwinRcvd): web100_snap_read(): invalid arguments
Variable 28 (MaxRwinSent): web100_snap_read(): invalid arguments
Variable 29 (MaxSsthresh): web100_snap_read(): invalid arguments
Variable 30 (MaxSaCwnd) not found in KIS
Variable 31 (MinMSS): web100_snap_read(): invalid arguments
Variable 32 (MinRTO): web100_snap_read(): invalid arguments
Variable 33 (MinRTT): web100_snap_read(): invalid arguments
Variable 34 (MinRwinRcvd): web100_snap_read(): invalid arguments
Variable 35 (MinRwinSent): web100_snap_read(): invalid arguments
Variable 36 (NagleEnabled): web100_snap_read(): invalid arguments
Variable 37 (OtherReductions): web100_snap_read(): invalid arguments
Variable 38 (PktsIn): web100_snap_read(): invalid arguments
Variable 39 (PktsOut): web100_snap_read(): invalid arguments
Variable 40 (PktsRetrans): web100_snap_read(): invalid arguments
Variable 42 (RcvWinScale): web100_snap_read(): invalid argumentssend_msg: type=5, len=17
Variable 43 (SACKEnabled): web100_snap_read(): invalid arguments
Variable 44 (SACKsRcvd): web100_snap_read(): invalid arguments
Variable 45 (SendStall): web100_snap_read(): invalid arguments
Variable 46 (SlowStart): web100_snap_read(): invalid arguments
Variable 47 (SampleRTT): web100_snap_read(): invalid arguments
Variable 48 (SmoothedRTT): web100_snap_read(): invalid arguments
Variable 50 (SndWinScale): web100_snap_read(): invalid argumentssend_msg: type=5, len=18
Variable 51 (SndLimTimeRwin): web100_snap_read(): invalid arguments
Variable 52 (SndLimTimeCwnd): web100_snap_read(): invalid arguments
Variable 53 (SndLimTimeSender): web100_snap_read(): invalid arguments
Variable 54 (SndLimTransRwin): web100_snap_read(): invalid arguments
Variable 55 (SndLimTransCwnd): web100_snap_read(): invalid arguments
Variable 56 (SndLimTransSender): web100_snap_read(): invalid arguments
Variable 57 (SndLimBytesRwin): web100_snap_read(): invalid arguments
Variable 58 (SndLimBytesCwnd): web100_snap_read(): invalid arguments
Variable 59 (SndLimBytesSender): web100_snap_read(): invalid arguments
Variable 60 (SubsequentTimeouts): web100_snap_read(): invalid arguments
Variable 61 (SumRTT): web100_snap_read(): invalid arguments
Variable 62 (Timeouts): web100_snap_read(): invalid arguments
Variable 63 (TimestampsEnabled): web100_snap_read(): invalid arguments
Variable 64 (WinScaleRcvd): web100_snap_read(): invalid arguments
Variable 65 (WinScaleSent): web100_snap_read(): invalid arguments
Variable 66 (DupAcksOut): web100_snap_read(): invalid arguments
Variable 67 (StartTimeUsec): web100_snap_read(): invalid arguments
Variable 68 (Duration): web100_snap_read(): invalid arguments
Variable 8 (CurMSS): web100_snap_read(): invalid argumentssend_msg: type=5, len=17
send_msg: type=5, len=14
send_msg: type=5, len=21
send_msg: type=5, len=13
send_msg: type=5, len=23
send_msg: type=5, len=22
send_msg: type=5, len=16
send_msg: type=5, len=15
Variable 13 (CwndRestores) not found in KISsend_msg: type=5, len=12
send_msg: type=5, len=20
send_msg: type=5, len=19
send_msg: type=5, len=19
Variable 22 (MaxCaCwnd) not found in KISsend_msg: type=5, len=13
send_msg: type=5, len=15
send_msg: type=5, len=24
send_msg: type=5, len=14
send_msg: type=5, len=19
send_msg: type=5, len=16
send_msg: type=5, len=14
send_msg: type=5, len=15
Variable 30 (MaxSaCwnd) not found in KISsend_msg: type=5, len=16
send_msg: type=5, len=13
send_msg: type=5, len=12
send_msg: type=5, len=11
send_msg: type=5, len=20
send_msg: type=5, len=19
send_msg: type=5, len=20
Variable 41 (X_Rcvbuf): web100_snap_read(): invalid argumentssend_msg: type=5, len=13
send_msg: type=5, len=12
send_msg: type=5, len=10
send_msg: type=5, len=20
send_msg: type=5, len=19
send_msg: type=5, len=16
send_msg: type=5, len=20
send_msg: type=5, len=14
send_msg: type=5, len=15
send_msg: type=5, len=17
Variable 49 (X_Sndbuf): web100_snap_read(): invalid argumentssend_msg: type=5, len=15
send_msg: type=5, len=15
send_msg: type=5, len=16
send_msg: type=5, len=13
send_msg: type=5, len=13
send_msg: type=5, len=13
send_msg: type=5, len=15
pcap_loop exited SIOCGSTAMP: Bad file descriptorsend_msg: type=5, len=15
send_msg: type=5, len=18
send_msg: type=5, len=24
send_msg: type=5, len=26
send_msg: type=5, len=19
send_msg: type=5, len=22
send_msg: type=5, len=24
send_msg: type=5, len=19
send_msg: type=5, len=26
send_msg: type=5, len=28
send_msg: type=5, len=22
send_msg: type=5, len=15
send_msg: type=5, len=12
send_msg: type=5, len=21
send_msg: type=5, len=16
send_msg: type=5, len=16
send_msg: type=5, len=14
send_msg: type=5, len=21
send_msg: type=5, len=19
Pkt-Pair data collection ended, waiting for signal to terminate process
Signal 11 received by process 25555
Signal 17 received by process 25442
Protocol error!
S2C throughput test FAILED!send_msg: type=7, len=61
Finished testing C2S = 91.38 Mbps, S2C = 0.00 Mbps
Client --> Server data detects link = System Fault
Client <-- Server Ack's detect link = System Fault
Server --> Client data detects link = System Fault
Server <-- Client Ack's detect link = System Fault
CWND limited test = 228.12 while unlimited = 0.00
Better throughput when CWND is limited, may be duplex mismatch
now = 1251874537 Process started at 1251874504, run time = 33send_msg: type=8, len=46
send_msg: type=8, len=75
send_msg: type=8, len=85
send_msg: type=8, len=78
send_msg: type=8, len=81
send_msg: type=8, len=53
send_msg: type=9, len=0
Timer expired while waiting for a new connection
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Waiting for new connection, timer running
Opened
'/usr/local/ndt/serverdata/ 2009/09/02/20090902T06:55:04.153056000Z_130.59.4.118:50491.meta'
metadata log file
Updating admin_view variables: Total count = 24
Initial counter Values Totalcnt = 24, Total Mismatch = 0, Total Bad
Cables = 0
Updated counter values Totalcnt = 25, Total Mismatch = 0, Total Bad
Cables = 0
Individual counts = [14, 0, 1, 0, 0, 2, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Signal 17 received by process 25442
successfully locked '/tmp/view.string' for updating
sending '91375,0,972950,0,25,0,0,14,0,1,0,0,2,8,0,0,0,0,0,0,0,0,0,Sep
2 06:55:04al (6) received,Sep 2 06:55:04al (6) received' to tmp file
Successfully returned from run_test() routine
Signal 17 received by process 25244
now = 1251874539 Process started at 1251874504, run time = 35
Select exited with rc = -1
Queue pointer = 25442, testing = 1, waiting = 1, zombie_check = 0
Received SIGCHLD signal for active web100srv process [25244]
wait3() returned 0 for PID=25442
wexitstatus = '0'
Attempting to clean up child 25442, head pid = 25442
Child process 25442 causing head pointer modification
Removing Child from head, decrementing waiting now = 0
Timer not running, waiting for new connection
Richard Carlson
1000 Oakbrook Dr
Ann Arbor, MI 48104
P: 734-352-7043
C: 630-251-4572
Richard Carlson
1000 Oakbrook Dr
Ann Arbor, MI 48104
P: 734-352-7043
C: 630-251-4572
- Re: Error running web100 3.5.0, Chris Welti, 09/01/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/01/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/01/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/01/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/01/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/02/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/02/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/02/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/02/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/02/2009
- AW: Error running web100 3.5.0, Galuschka Christoph, 09/02/2009
- Re: AW: Error running web100 3.5.0, Richard Carlson, 09/04/2009
- AW: AW: Error running web100 3.5.0, Galuschka Christoph, 09/07/2009
- Re: AW: Error running web100 3.5.0, Chris Welti, 09/11/2009
- Re: AW: Error running web100 3.5.0, Richard Carlson, 09/11/2009
- RE: AW: Error running web100 3.5.0, Galuschka Christoph, 09/11/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/02/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/02/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/02/2009
- Re: Error running web100 3.5.0, Richard Carlson, 09/01/2009
- Re: Error running web100 3.5.0, Chris Welti, 09/01/2009
- AW: Error running web100 3.5.0, Galuschka Christoph, 09/02/2009
Archive powered by MHonArc 2.6.16.