perfSONAR User Q&A and Other Discussion

[Joseph Bernard <>]

I replaced the Broadcom with an Intel Gigabit CT Desktop Adapter card and
got the same results.  I then noticed that the version of iperf on CentOS
5.7 I've been using is 2.0.4 instead of 2.0.5.  I loaded up PS-PS and
replaced 2.0.5 with 2.0.4.  I am now getting 950Mb/s on UDP.  I was using
2.0.5 on my Macs for testing.  I compiled 2.0.4 and went from 800Mb/s to
900Mb/s. So it looks like iperf is the problem here.

Thanks,
Joseph



On 2/14/12 6:55 PM, "Michael Sinatra" 
<>
 wrote:

>The problem is that Broadcoms aren't uniform, not that they're uniformly
>poor.  I agree that Intels are uniformly better (mainly because they're
>uniform), but when Broadcom bought the Tigon II chipset from Alteon,
>they were getting the best ethernet chipset at the time.  (The initial
>Intel gigabit offering was junk at that time.)
>
>The 5721 is part of the 5700 series and it's sometimes called the "Tigon
>III" and I believe it still has 2 R4000 CPUs onboard.  (Remember when
>you and about 100 other users logged into an entire computer running a
>single R4000?)
>
>The problem is that Broadcom created a bunch of scaled down versions of
>the chipset that just plain suck.  Some of them can't even do jumbo
>frames.  *However* I don't think the 5721 falls into that camp.  I think
>the 5721 is a fairly upscale version that was the (one of?) the first
>Broadcoms to support PCIe.  (Note that this is not part of the NetXtreme
>II chips--those will generally tend to use a different driver.)
>
>This is a long-winded version of saying that there is really no excuse
>for not getting full gigabit line-rate on this particular chip.  I have
>saturated many a gig link using the "good" Broadcoms.  You just have to
>remember to stay away from the "bad" ones.  I agree, though, that it's
>often just easier to go with any Intel gigE card made in the last 5 years.
>
>So from the standpoint of the original question, this is not likely to
>be a hardware limitation, but either a driver bug in the older kernels
>or some other software issue.
>
>michael
>
>On 2/14/12 2:41 PM, Gerry Creager wrote:
>> You're getting pretty good results for a Broadcom NIC. I tend to spec
>> Intel exclusively because of uniform poor performance with Broadcoms..
>> 
>> gerry
>> 
>> On 02/14/2012 01:43 PM, Joseph Bernard wrote:
>>> Broadcom NetXtreme BCM5721
>>>
>>>
>>> On 2/14/12 2:30 PM, "Brian 
>>> Tierney"<>
>>>   wrote:
>>>
>>>>
>>>>
>>>> interesting.
>>>>
>>>> what type of NIC is in this host?
>>>>
>>>>
>>>> On Feb 14, 2012, at 11:20 AM, Joseph Bernard wrote:
>>>>
>>>>> One more thing I've noticed.  When I do TCP in PS-PS, the server side
>>>>> CPU
>>>>> gets to 80%.  When I do UDP, the CPU only gets to 45%.  On CentOS
>>>>>5.7,
>>>>> the
>>>>> CPU goes to 20% on TCP and 99% on UDP.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Joseph
>>>>>
>>>>>
>>>>> On 2/14/12 11:30 AM, "Joseph 
>>>>> Bernard"<>
>>>>>   wrote:
>>>>>
>>>>>> I used the -w option and got closer to 700Mb/s. I even put it at
>>>>>> 64M to
>>>>>> see if it made a difference.  I decided to install CentOS 5.7 i386
>>>>>>and
>>>>>> got
>>>>>> 950Mb/s on both TCP and UDP, and the limit with the -w option is 256
>>>>>> Kbyte.  Now I'm not sure if PS-PS is slower or if CentOS 5.7 is less
>>>>>> accurate.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Joseph
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2/13/12 5:13 PM, "Brian 
>>>>>> Tierney"<>
>>>>>>   wrote:
>>>>>>
>>>>>>>
>>>>>>> Did you use the -w option with UDP?
>>>>>>>
>>>>>>> TCP autotunes, UDP does not, and to go fast UDP needs more buffer
>>>>>>> space
>>>>>>> than the default.
>>>>>>>
>>>>>>> Try adding "-w 4M" to both sender and receiver.
>>>>>>>
>>>>>>> On Feb 13, 2012, at 1:35 PM, Joseph Bernard wrote:
>>>>>>>
>>>>>>>> I loaded 3.2.1.1 on 2 workstations and got 950Mb/s on TCP and
>>>>>>>> 600Mb/s
>>>>>>>> on
>>>>>>>> UDP tests.  They both have 1Gb/s cards.  I thought maybe the
>>>>>>>>network
>>>>>>>> cards
>>>>>>>> weren't up to snuff, so I got 2 HP servers, loaded them up, and
>>>>>>>>did
>>>>>>>> the
>>>>>>>> same tests.  I got the same results.  The workstations/servers are
>>>>>>>> connected via crossover.  I even tried a brand new cable to be
>>>>>>>>sure
>>>>>>>> it
>>>>>>>> wasn't bad materials.
>>>>>>>>
>>>>>>>> When I used my iMac and Macbook Pro from 2009 running Snow
>>>>>>>> Leopard, I
>>>>>>>> was
>>>>>>>> able to get 800Mb/s running iperf.  Is there something I am
>>>>>>>>missing?
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Joseph Bernard
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>