ndt-users list created

Greetings NDT Community:

I have been working on this issue with my equipment for some time.  To 
provide some background:

I was approached last month to move our bandwidth tester back to real 
hardware because of complaints of inaccurate reports on the 1GB network 
connections.  The virtual hardware was running NDT 3.4.4a, and aside from 
'reporting inaccurate throughput', seemed to run as expected.

I moved to real hardware two weeks ago.  Since then, I continue to notice 
issues with testing the 1GB connections.  However, with the newer versions of 
NDT (3.5.0 and above), I am seeing 'Protocol Errors' when the S2C test is 
attempted.

Here is my current system information:

speedtest:~/web100src # uname -a
Linux speedtest 2.6.31.8-web100-WEB100-V1 #1 SMP PREEMPT Wed Feb 3 10:48:27 
EST 2010 i686 i686 i386 GNU/Linux
speedtest:~/web100src # dmesg|grep Web100
[    0.563046] Web100 2.5.26 200911182032 net100: Initialization successful

I am running NDT 3.5.0, because this is what seems to behave the most 
reliably for this particular system.


Here are the steps I have attempted to resolve the matter myself:

1.  Uncommenting wait(NULL); in testoptions.c 
2.  v3.5.0
3.  v3.5.6
4.  v3.5.10
5.  v3.5.13
6.  v3.5.14
7.  ethtool to limit packet coalescing on tx and rx

Using ethtool to limit the coalescing helped a little, because now the 
problem is happening about 80% of the time.

I am open to any suggestions why this is happening or how to address it.  I 
do *NOT* see this happening on the 10/100 connections.  It is only happening 
on the clients with 1GB connections.

Heather Sanders.

P.S.

For your viewing pleasure, I have ran the web100srv in debug level 6.  Here 
is the output, when the S2C produces the error:
----------------------------------------------------------------------
Waiting for new connection, timer runningChild thinks pipe() returned fd0=5, 
fd1=6 for pid=0                            

Got 'go' signal from parent, ready to start testing
Child process 475 started                          
>>> send_msg: type=2, len=6                        
<<< recv_msg: type=2, len=1                        
>>> send_msg: type=2, len=7                        
Starting test suite:                               
 > Middlebox test                                  
 > Simple firewall test                            
 > C2S throughput test                             
 > S2C throughput test                             
 <-- Middlebox test -->                            
KillHung(): returned non-0 response, nothing to kill or kill failed
  -- port: 3003                                                    
>>> send_msg: type=3, len=4                                        
Middlebox test, Port 3003 waiting for incoming connection (fd=3)   
Set MSS to 536, Receiving Window size set to 262142KB              
Sending Window size set to 262142KB                                
Looking for Web100 data on socketid 5                              
Server: 155.138.1.49;3003 Client: 155.138.8.12;2298 1456;6;6;      
Sending 1456 Byte packets over the network                         
Setting Cwnd Limit to 2912 octets                                  
Signal 17 received by process 475                                  
now = 1265297898 Process started at 1265297895, run time = 3       
Timer expired while waiting for a new connection                   
Queue pointer = 475, testing = 1, waiting = 1                      
Waiting for new connection, timer running                          
now = 1265297901 Process started at 1265297895, run time = 6       
Timer expired while waiting for a new connection                   
Queue pointer = 475, testing = 1, waiting = 1                      
Waiting for new connection, timer running                          
>>> send_msg: type=5, len=35                                       
<<< recv_msg: type=5, len=17                                       
CWND limited throughput = 95759 kbps (95758.65915288129)           
>>> send_msg: type=6, len=0                                        
 <-------------------->                                            
 <-- Simple firewall test -->                                      
  -- port: 59639                                                   
  -- time: 1                                                       
>>> send_msg: type=3, len=7                                        
<<< recv_msg: type=5, len=4                                        
  -- oport: 2302                                                   
>>> send_msg: type=4, len=0                                        
>>> send_msg: type=5, len=20                                       
<<< recv_msg: type=5, len=20                                       
>>> send_msg: type=5, len=1                                        
>>> send_msg: type=6, len=0                                        
 <-------------------------->                                      
 <-- C2S throughput test -->                                       
  -- port: 3002                                                    
listening for Inet connection on testOptions->c2ssockfd, fd=3      
Sending 'GO' signal, to tell client to head for the next test      
>>> send_msg: type=3, len=4                                        
C2S test Child thinks pipe() returned fd0=7, fd1=8                 
C2S test calling init_pkttrace() with pd=0xbfce34bc                
Opening network interface 'eth0' for packet-pair timing            
pcap_open_live() returned pointer 0x80844b8                        
installing pkt filter for 'host 155.138.8.12 and port 2305'        
Initial pkt src data = 8068484                                     
C2S test Parent thinks pipe() returned fd0=7, fd1=8                
now = 1265297904 Process started at 1265297895, run time = 9       
Timer expired while waiting for a new connection                   
Queue pointer = 475, testing = 1, waiting = 1                      
Waiting for new connection, timer running                          
>>> send_msg: type=4, len=0                                        
New packet trace started -- initializing counters                  
now = 1265297907 Process started at 1265297895, run time = 12      
Timer expired while waiting for a new connection                   
Queue pointer = 475, testing = 1, waiting = 1                      
Waiting for new connection, timer running                          
now = 1265297910 Process started at 1265297895, run time = 15      
Timer expired while waiting for a new connection                   
Queue pointer = 475, testing = 1, waiting = 1                      
Waiting for new connection, timer running                          
now = 1265297913 Process started at 1265297895, run time = 18      
Timer expired while waiting for a new connection                   
Queue pointer = 475, testing = 1, waiting = 1                      
Waiting for new connection, timer running                          
922788 kbps outbound                                               
>>> send_msg: type=5, len=6                                        
Signal USR1(10) sent to child [478]                                
Signal 10 received by process 478                                  
DEBUG, caught SIGUSR1, setting sig1 flag to force exit             
Received SIGUSRx signal terminating data collection loop for pid=478
Sending pkt-pair data back to parent on pipe 7, 8                   
fwd.saddr = c088a9b:2305, rev.saddr = 31018a9b:3002                 
09:38:26.001989   155.138.8.12:2305 --> 155.138.1.49:3002 Collected pkt-pair 
data max = 386195
wrote 128 bytes: link counters are '  0 0 9 15 34 81 69392 386195 319950 
16272 20 6 890.92 0 0 0 1 0'
#$#$#$#$ pcap routine says window increases = 0, decreases = 0, no change = 0 
                       
09:38:26.001989   155.138.1.49:3002 --> 155.138.8.12:2305 Collected pkt-pair 
data max = 125066       
wrote 128 bytes: link counters are '  0 0 1 12 21 38 41309 121803 125066 
19862 108 0 825.17 47530 35374 225316 0 107'
#$#$#$#$ pcap routine says window increases = 47530, decreases = 35374, no 
change = 225316                           
128 bytes read '  0 0 9 15 34 81 69392 386195 319950 16272 20 6 890.92 0 0 0 
1 0' from monitor pipe                  
128 bytes read '  0 0 1 12 21 38 41309 121803 125066 19862 108 0 825.17 47530 
35374 225316 0 107' from monitor pipe  
>>> send_msg: type=6, len=0                                                   
>>>                                        
Finished reading pkt-pair data from network, process 478 should terminate now 
                                       
Signal 11 received by process 478                                             
                                       
DEBUG, caught SIGSEGV signal and terminated process (478)                     
                                       
Signal 17 received by process 475                                             
                                       
 <------------------------->                                                  
                                       
 <-- S2C throughput test -->                                                  
                                       
  -- port: 3003                                                               
                                       
>>> send_msg: type=3, len=4                                                   
>>>                                        
waiting for data on testOptions->s2csockfd                                    
                                       
S2C test Child thinks pipe() returned fd0=7, fd1=8                            
                                       
S2C test calling init_pkttrace() with pd=0xbfce34b0                           
                                       
Opening network interface 'eth0' for packet-pair timing                       
                                       
pcap_open_live() returned pointer 0x8085190                                   
                                       
installing pkt filter for 'host 155.138.8.12 and port 2313'                   
                                       
Initial pkt src data = 8068484                                                
                                       
Signal 17 received by process 475                                             
                                       
>>> send_msg: type=4, len=0                                                   
>>>                                        
New packet trace started -- initializing counters                             
                                       
now = 1265297916 Process started at 1265297895, run time = 21                 
                                       
Timer expired while waiting for a new connection                              
                                       
Queue pointer = 475, testing = 1, waiting = 1                                 
                                       
Waiting for new connection, timer running                                     
                                       
now = 1265297919 Process started at 1265297895, run time = 24                 
                                       
Timer expired while waiting for a new connection                              
                                       
Queue pointer = 475, testing = 1, waiting = 1                                 
                                       
Waiting for new connection, timer running                                     
                                       
now = 1265297922 Process started at 1265297895, run time = 27                 
                                       
Timer expired while waiting for a new connection                              
                                       
Queue pointer = 475, testing = 1, waiting = 1                                 
                                       
Waiting for new connection, timer running                                     
                                       
now = 1265297925 Process started at 1265297895, run time = 30                 
                                       
Timer expired while waiting for a new connection                              
                                       
Queue pointer = 475, testing = 1, waiting = 1                                 
                                       
Waiting for new connection, timer running                                     
                                       
>>> send_msg: type=5, len=23                                                  
>>>                                        
sent 984743936 bytes to client in 10.00 seconds                               
                                       
Buffer control counters Total = 120208, new data = 0, Draining Queue = 0      
                                       
Signal USR2(12) sent to child [480]                                           
                                       
Signal 12 received by process 480                                             
                                       
DEBUG, caught SIGUSR2, setting sig2 flag to force exit                        
                                       
Received SIGUSRx signal terminating data collection loop for pid=480          
                                       
Sending pkt-pair data back to parent on pipe 7, 8                             
                                       
fwd.saddr = 31018a9b:3003, rev.saddr = c088a9b:2313                           
                                       
09:38:36.283596   155.138.1.49:3003 --> 155.138.8.12:2313 Collected pkt-pair 
data max = 28638                        
wrote 128 bytes: link counters are '  0 0 1 20 60 96 20251 5270 6182 28638 1 
793 2772.94 0 0 0 1 0'                  
#$#$#$#$ pcap routine says window increases = 0, decreases = 0, no change = 0 
                                       
09:38:36.283596   155.138.8.12:2313 --> 155.138.1.49:3003 Collected pkt-pair 
data max = 134636                       
wrote 128 bytes: link counters are '  0 0 1 33 53 1519 19503 6630 134636 
92000 691 1113 2056.62 885 1622 253672 0 690'
#$#$#$#$ pcap routine says window increases = 885, decreases = 1622, no 
change = 253672                               
Read '  0 0 1 20 60 96 20251 5270 6182 28638 1 793 2772.94 0 0 0 1 0' from 
monitor pipe                               
Read '  0 0 1 33 53 1519 19503 6630 134636 92000 691 1113 2056.62 885 1622 
253672 0 690' from monitor pipe            
787710 kbps inbound                                                           
                                        
libweb100: warning: accessing depricated variable AckPktsIn                   
                                        
Variable 0 (AckPktsIn): web100_snap_read(): invalid arguments                 
                                        
libweb100: warning: accessing depricated variable AckPktsOut                  
                                        
Variable 1 (AckPktsOut): web100_snap_read(): invalid arguments                
                                        
Variable 2 (BytesRetrans): web100_snap_read(): invalid arguments              
                                        
Variable 3 (CongAvoid): web100_snap_read(): invalid arguments                 
                                        
Variable 4 (CongestionOverCount): web100_snap_read(): invalid arguments       
                                        
Variable 5 (CongestionSignals): web100_snap_read(): invalid arguments         
                                        
Variable 6 (CountRTT): web100_snap_read(): invalid arguments                  
                                        
Variable 7 (CurCwnd): web100_snap_read(): invalid arguments                   
                                        
>>> send_msg: type=5, len=13                                                  
>>>                                         
CurMSS: 1460                                                                  
                                        
Variable 9 (CurRTO): web100_snap_read(): invalid arguments                    
                                        
Variable 10 (CurRwinRcvd): web100_snap_read(): invalid arguments              
                                        
Variable 11 (CurRwinSent): web100_snap_read(): invalid arguments              
                                        
Variable 12 (CurSsthresh): web100_snap_read(): invalid arguments              
                                        
Variable 13 (CwndRestores) not found in KIS                                   
                                        
Variable 14 (DSACKDups): web100_snap_read(): invalid arguments                
                                        
Variable 15 (DataBytesIn): web100_snap_read(): invalid arguments              
                                        
Variable 16 (DataBytesOut): web100_snap_read(): invalid arguments             
                                        
Variable 17 (DataPktsIn): web100_snap_read(): invalid arguments               
                                        
Variable 18 (DataPktsOut): web100_snap_read(): invalid arguments              
                                        
Variable 19 (DupAcksIn): web100_snap_read(): invalid arguments                
                                        
Variable 20 (ECNEnabled): web100_snap_read(): invalid arguments               
                                        
Variable 21 (FastRetran): web100_snap_read(): invalid arguments               
                                        
Variable 22 (MaxCaCwnd) not found in KIS                                      
                                        
Variable 23 (MaxCwnd): web100_snap_read(): invalid arguments                  
                                        
Variable 24 (MaxMSS): web100_snap_read(): invalid arguments                   
                                        
Variable 25 (MaxRTO): web100_snap_read(): invalid arguments                   
                                        
Variable 26 (MaxRTT): web100_snap_read(): invalid arguments                   
                                        
Variable 27 (MaxRwinRcvd): web100_snap_read(): invalid arguments              
                                        
Variable 28 (MaxRwinSent): web100_snap_read(): invalid arguments              
                                        
Variable 29 (MaxSsthresh): web100_snap_read(): invalid arguments              
                                        
Variable 30 (MaxSaCwnd) not found in KIS                                      
                                        
Variable 31 (MinMSS): web100_snap_read(): invalid arguments                   
                                        
Variable 32 (MinRTO): web100_snap_read(): invalid arguments                   
                                        
Variable 33 (MinRTT): web100_snap_read(): invalid arguments                   
                                        
Variable 34 (MinRwinRcvd): web100_snap_read(): invalid arguments              
                                        
Variable 35 (MinRwinSent): web100_snap_read(): invalid arguments              
                                        
Variable 36 (NagleEnabled): web100_snap_read(): invalid arguments             
                                        
Variable 37 (OtherReductions): web100_snap_read(): invalid arguments          
                                        
Variable 38 (PktsIn): web100_snap_read(): invalid arguments                   
                                        
Variable 39 (PktsOut): web100_snap_read(): invalid arguments                  
                                        
Variable 40 (PktsRetrans): web100_snap_read(): invalid arguments              
                                        
>>> send_msg: type=5, len=17                                                  
>>>                                         
X_Rcvbuf: 262142                                                              
                                        
Variable 42 (RcvWinScale): web100_snap_read(): invalid arguments              
                                        
Variable 43 (SACKEnabled): web100_snap_read(): invalid arguments              
                                        
Variable 44 (SACKsRcvd): web100_snap_read(): invalid arguments                
                                        
Variable 45 (SendStall): web100_snap_read(): invalid arguments                
                                        
Variable 46 (SlowStart): web100_snap_read(): invalid arguments                
                                        
Variable 47 (SampleRTT): web100_snap_read(): invalid arguments                
                                        
Variable 48 (SmoothedRTT): web100_snap_read(): invalid arguments              
                                        
>>> send_msg: type=5, len=17                                                  
>>>                                         
X_Sndbuf: 262142                                                              
                                        
Variable 50 (SndWinScale): web100_snap_read(): invalid arguments              
                                        
Variable 51 (SndLimTimeRwin): web100_snap_read(): invalid arguments           
                                        
Variable 52 (SndLimTimeCwnd): web100_snap_read(): invalid arguments           
                                        
Variable 53 (SndLimTimeSender): web100_snap_read(): invalid arguments         
                                        
Variable 54 (SndLimTransRwin): web100_snap_read(): invalid arguments          
                                        
Variable 55 (SndLimTransCwnd): web100_snap_read(): invalid arguments          
                                        
Variable 56 (SndLimTransSender): web100_snap_read(): invalid arguments        
                                        
Variable 57 (SndLimBytesRwin): web100_snap_read(): invalid arguments          
                                        
Variable 58 (SndLimBytesCwnd): web100_snap_read(): invalid arguments          
                                        
Variable 59 (SndLimBytesSender): web100_snap_read(): invalid arguments        
                                        
Variable 60 (SubsequentTimeouts): web100_snap_read(): invalid arguments       
                                        
Variable 61 (SumRTT): web100_snap_read(): invalid arguments                   
                                        
Variable 62 (Timeouts): web100_snap_read(): invalid arguments                 
                                        
Variable 63 (TimestampsEnabled): web100_snap_read(): invalid arguments        
                                        
Variable 64 (WinScaleRcvd): web100_snap_read(): invalid arguments             
                                        
Variable 65 (WinScaleSent): web100_snap_read(): invalid arguments             
                                        
Variable 66 (DupAcksOut): web100_snap_read(): invalid arguments               
                                        
Variable 67 (StartTimeUsec): web100_snap_read(): invalid arguments            
                                        
Variable 68 (Duration): web100_snap_read(): invalid arguments                 
                                        
>>> send_msg: type=5, len=18                                                  
>>>                                         
AckPktsIn: 256179                                                             
                                        
>>> send_msg: type=5, len=14                                                  
>>>                                         
AckPktsOut: 0                                                                 
                                        
>>> send_msg: type=5, len=16                                                  
>>>                                         
BytesRetrans: 0                                                               
                                        
>>> send_msg: type=5, len=13                                                  
>>>                                         
CongAvoid: 0                                                                  
                                        
>>> send_msg: type=5, len=23                                                  
>>>                                         
CongestionOverCount: 0                                                        
                                        
>>> send_msg: type=5, len=21                                                  
>>>                                         
CongestionSignals: 0                                                          
                                        
>>> send_msg: type=5, len=17                                                  
>>>                                         
CountRTT: 255489                                                              
                                        
>>> send_msg: type=5, len=15                                                  
>>>                                         
CurCwnd: 99280                                                                
                                        
Variable 8 (CurMSS): web100_snap_read(): invalid arguments                    
                                        
>>> send_msg: type=5, len=12                                                  
>>>                                         
CurRTO: 201                                                                   
                                        
>>> send_msg: type=5, len=19                                                  
>>>                                         
CurRwinRcvd: 65700                                                            
                                        
>>> send_msg: type=5, len=18                                                  
>>>                                         
CurRwinSent: 5888                                                             
                                        
>>> send_msg: type=5, len=19                                                  
>>>                                         
CurSsthresh: 23360                                                            
                                        
Variable 13 (CwndRestores) not found in KIS                                   
                                        
>>> send_msg: type=5, len=13                                                  
>>>                                         
DSACKDups: 0                                                                  
                                        
>>> send_msg: type=5, len=15                                                  
>>>                                         
DataBytesIn: 0                                                                
                                        
>>> send_msg: type=5, len=24                                                  
>>>                                         
DataBytesOut: 985970196                                                       
                                        
>>> send_msg: type=5, len=14                                                  
>>>                                         
DataPktsIn: 0                                                                 
                                        
>>> send_msg: type=5, len=20                                                  
>>>                                         
DataPktsOut: 674483                                                           
                                        
>>> send_msg: type=5, len=15                                                  
>>>                                         
DupAcksIn: 656                                                                
                                        
>>> send_msg: type=5, len=14                                                  
>>>                                         
ECNEnabled: 0                                                                 
                                        
>>> send_msg: type=5, len=14                                                  
>>>                                         
FastRetran: 0                                                                 
                                        
Variable 22 (MaxCaCwnd) not found in KIS                                      
                                        
>>> send_msg: type=5, len=15                                                  
>>>                                         
MaxCwnd: 99280                                                                
                                        
>>> send_msg: type=5, len=13                                                  
>>>                                         
MaxMSS: 1460                                                                  
                                        
>>> send_msg: type=5, len=12                                                  
>>>                                         
MaxRTO: 232                                                                   
                                        
>>> send_msg: type=5, len=11                                                  
>>>                                         
MaxRTT: 37                                                                    
                                        
>>> send_msg: type=5, len=19                                                  
>>>                                         
MaxRwinRcvd: 65700                                                            
                                        
>>> send_msg: type=5, len=18                                                  
>>>                                         
MaxRwinSent: 5888                                                             
                                        
>>> send_msg: type=5, len=19                                                  
>>>                                         
MaxSsthresh: 23360                                                            
                                        
Variable 30 (MaxSaCwnd) not found in KIS                                      
                                        
>>> send_msg: type=5, len=13                                                  
>>>                                         
MinMSS: 1460                                                                  
                                        
>>> send_msg: type=5, len=12                                                  
>>>                                         
MinRTO: 201                                                                   
                                        
>>> send_msg: type=5, len=10                                                  
>>>                                         
MinRTT: 0                                                                     
                                        
>>> send_msg: type=5, len=17                                                  
>>>                                         
MinRwinRcvd: 204                                                              
                                        
>>> send_msg: type=5, len=18                                                  
>>>                                         
MinRwinSent: 5840                                                             
                                        
>>> send_msg: type=5, len=16                                                  
>>>                                         
NagleEnabled: 1                                                               
                                        
>>> send_msg: type=5, len=19                                                  
>>>                                         
OtherReductions: 0                                                            
                                        
>>> send_msg: type=5, len=15                                                  
>>>                                         
PktsIn: 256179                                                                
                                        
>>> send_msg: type=5, len=16                                                  
>>>                                         
PktsOut: 674483                                                               
                                        
>>> send_msg: type=5, len=15                                                  
>>>                                         
PktsRetrans: 0                                                                
                                        
Variable 41 (X_Rcvbuf): web100_snap_read(): invalid arguments                 
                                        
>>> send_msg: type=5, len=15                                                  
>>>                                         
RcvWinScale: 6                                                                
                                        
>>> send_msg: type=5, len=15                                                  
>>>                                         
SACKEnabled: 3                                                                
                                        
>>> send_msg: type=5, len=13                                                  
>>>                                         
SACKsRcvd: 0                                                                  
                                        
>>> send_msg: type=5, len=13                                                  
>>>                                         
SendStall: 0                                                                  
                                        
>>> send_msg: type=5, len=13                                                  
>>>                                         
SlowStart: 0                                                                  
                                        
>>> send_msg: type=5, len=13                                                  
>>>                                         
SampleRTT: 0                                                                  
                                        
>>> send_msg: type=5, len=15                                                  
>>>                                         
SmoothedRTT: 1                                                                
                                        
Variable 49 (X_Sndbuf): web100_snap_read(): invalid arguments                 
                                        
>>> send_msg: type=5, len=15                                                  
>>>                                         
SndWinScale: 2                                                                
                                        
>>> send_msg: type=5, len=24                                                  
>>>                                         
SndLimTimeRwin: 1354121                                                       
                                        
>>> send_msg: type=5, len=22                                                  
>>>                                         
SndLimTimeCwnd: 17074                                                         
                                        
>>> send_msg: type=5, len=26                                                  
>>>                                         
SndLimTimeSender: 8707290                                                     
                                        
>>> send_msg: type=5, len=23                                                  
>>>                                         
SndLimTransRwin: 37439                                                        
                                        
>>> send_msg: type=5, len=21                                                  
>>>                                         
SndLimTransCwnd: 417                                                          
                                        
>>> send_msg: type=5, len=25                                                  
>>>                                         
SndLimTransSender: 37855                                                      
                                        
>>> send_msg: type=5, len=27                                                  
>>>                                         
SndLimBytesRwin: 763507300                                                    
                                        
>>> send_msg: type=5, len=25                                                  
>>>                                         
SndLimBytesCwnd: 6458940                                                      
                                        
>>> send_msg: type=5, len=29                                                  
>>>                                         
SndLimBytesSender: 216003956                                                  
                                        
>>> send_msg: type=5, len=22                                                  
>>>                                         
SubsequentTimeouts: 0                                                         
                                        
>>> send_msg: type=5, len=15                                                  
>>>                                         
SumRTT: 150490                                                                
                                        
>>> send_msg: type=5, len=12                                                  
>>>                                         
Timeouts: 0                                                                   
                                        
>>> send_msg: type=5, len=21                                                  
>>>                                         
TimestampsEnabled: 0                                                          
                                        
>>> send_msg: type=5, len=16                                                  
>>>                                         
WinScaleRcvd: 2                                                               
                                        
>>> send_msg: type=5, len=16                                                  
>>>                                         
WinScaleSent: 6                                                               
                                        
>>> send_msg: type=5, len=14                                                  
>>>                                         
DupAcksOut: 0                                                                 
                                        
>>> send_msg: type=5, len=22                                                  
>>>                                         
StartTimeUsec: 207054                                                         
                                        
>>> send_msg: type=5, len=19                                                  
>>>                                         
Duration: 10081036                                                            
                                        
Finished reading pkt-pair data from network, process 480 should terminate now 
                                        
Signal 11 received by process 480                                             
                                        
DEBUG, caught SIGSEGV signal and terminated process (480)                     
                                        
Signal 17 received by process 475                                             
                                        
Protocol error!                                                               
                                        
>>> send_msg: type=7, len=61                                                  
>>>                                         
S2C throughput test FAILED!                                                   
                                        
Finished testing C2S = 922.79 Mbps, S2C = 0.00 Mbps                           
                                        
Client --> Server data detects link = Gigabit Ethernet                        
                                        
Client <-- Server Ack's detect link = OC-48                                   
                                        
Server --> Client data detects link = 10 Gigabit Enet                         
                                        
Server <-- Client Ack's detect link = OC-48                                   
                                        
CWND limited test = 95758.66 while unlimited = 0.00                           
                                        
Better throughput when CWND is limited, may be duplex mismatch                
                                        
>>> send_msg: type=8, len=42                                                  
>>>                                         
>>> send_msg: type=8, len=76                                                  
>>>                                         
>>> send_msg: type=8, len=89                                                  
>>>                                         
>>> send_msg: type=8, len=77                                                  
>>>                                         
>>> send_msg: type=8, len=82                                                  
>>>                                         
>>> send_msg: type=8, len=46                                                  
>>>                                         
>>> send_msg: type=9, len=0                                                   
>>>                                         
Successfully returned from run_test() routine                                 
                                        
Signal 17 received by process 353                                             
                                        
now = 1265297926 Process started at 1265297895, run time = 31                 
                                        
Select exited with rc = -1                                                    
                                        
Queue pointer = 475, testing = 1, waiting = 1                                 
                                        
Received SIGCHLD signal for active web100srv process [353]                    
                                        
wait3() returned 0 for PID=475                                                
                                        
wexitstatus = '0'                                                             
                                        
Attempting to clean up child 475, head pid = 475                              
                                        
Child process 475 causing head pointer modification                           
                                        
Removing Child from head, decrementing waiting now = 0                        
                                        
Timer not running, waiting for new connection                                 
                                        
New connection received, waiting for accept() to complete                     
                                        
New connection received from [155.138.8.12].                                  
                                        
Parent process spawned child = 486                                            
                                        
Parent thinks pipe() returned fd0=5, fd1=6                                    
                                        
Incrementing waiting variable now = 1                                         
                                        
Telling client 486 testing will begin now                                     
                                        
>>> send_msg: type=1, len=1                                                   
>>>                                         
Queue pointer = 486, testing = 1, waiting = 1                                 
                                        
Waiting for new connection, timer running                                     
                                        
Child thinks pipe() returned fd0=5, fd1=6 for pid=0                           
                                        
Got 'go' signal from parent, ready to start testing                           
                                        
Child process 486 started                                                     
                                        
>>> send_msg: type=2, len=6                                                   
>>>                                         
<<< recv_msg: type=2, len=1                                                   
                                        
>>> send_msg: type=2, len=7                                                   
>>>                                         
Starting test suite:                                                          
                                        
 > Middlebox test                                                             
 >                                         
 > Simple firewall test                                                       
 >                                         
 > C2S throughput test                                                        
 >                                         
 > S2C throughput test                                                        
 >                                         
 <-- Middlebox test -->                                                       
                                        
KillHung(): returned non-0 response, nothing to kill or kill failed           
                                        
  -- port: 3003                                                               
                                        
>>> send_msg: type=3, len=4                                                   
>>>                                         
Middlebox test, Port 3003 waiting for incoming connection (fd=3)              
                                        
Set MSS to 536, Receiving Window size set to 262142KB                         
                                        
Sending Window size set to 262142KB                                           
                                        
Looking for Web100 data on socketid 5                                         
                                        
Server: 155.138.1.49;3003 Client: 155.138.8.12;2327 1456;6;6;                 
                                        
Sending 1456 Byte packets over the network                                    
                                        
Setting Cwnd Limit to 2912 octets                                             
                                        
Signal 17 received by process 486                                             
                                        
now = 1265297937 Process started at 1265297934, run time = 3                  
                                        
Timer expired while waiting for a new connection                              
                                        
Queue pointer = 486, testing = 1, waiting = 1                                 
                                        
Waiting for new connection, timer running                                     
                                        
now = 1265297940 Process started at 1265297934, run time = 6                  
                                        
Timer expired while waiting for a new connection                              
                                        
Queue pointer = 486, testing = 1, waiting = 1                                 
                                        
Waiting for new connection, timer running                                     
                                        
>>> send_msg: type=5, len=35                                                  
>>>                                         
<<< recv_msg: type=5, len=17                                                  
                                        
CWND limited throughput = 95905 kbps (95904.76204326487)                      
                                        
>>> send_msg: type=6, len=0                                                   
>>>                                         
 <-------------------->                                                       
                                        
 <-- Simple firewall test -->                                                 
                                        
  -- port: 48244                                                              
                                        
  -- time: 1                                                                  
                                        
>>> send_msg: type=3, len=7                                                   
>>>                                         
<<< recv_msg: type=5, len=4                                                   
                                        
  -- oport: 2331                                                              
                                        
>>> send_msg: type=4, len=0                                                   
>>>                                         
>>> send_msg: type=5, len=20                                                  
>>>                                         
<<< recv_msg: type=5, len=20                                                  
                                        
>>> send_msg: type=5, len=1                                                   
>>>                                         
>>> send_msg: type=6, len=0                                                   
>>>                                         
 <-------------------------->                                                 
                                        
 <-- C2S throughput test -->                                                  
                                        
  -- port: 3002                                                               
                                        
listening for Inet connection on testOptions->c2ssockfd, fd=3                 
                                        
Sending 'GO' signal, to tell client to head for the next test                 
                                        
>>> send_msg: type=3, len=4                                                   
>>>                                         
C2S test Child thinks pipe() returned fd0=7, fd1=8                            
                                        
C2S test calling init_pkttrace() with pd=0xbfce34bc                           
                                        
Opening network interface 'eth0' for packet-pair timing                       
                                        
pcap_open_live() returned pointer 0x8084568                                   
                                        
installing pkt filter for 'host 155.138.8.12 and port 2334'                   
                                        
Initial pkt src data = 8068484                                                
                                        
C2S test Parent thinks pipe() returned fd0=7, fd1=8                           
                                        
now = 1265297943 Process started at 1265297934, run time = 9                  
                                        
Timer expired while waiting for a new connection                              
                                        
Queue pointer = 486, testing = 1, waiting = 1                                 
                                        
Waiting for new connection, timer running                                     
                                        
>>> send_msg: type=4, len=0                                                   
>>>                                         
New packet trace started -- initializing counters                             
                                        
now = 1265297946 Process started at 1265297934, run time = 12                 
                                        
Timer expired while waiting for a new connection  
Queue pointer = 486, testing = 1, waiting = 1