perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Hi Roy;

Unfortunately I can't.  I am CCing Eric Pouyoul from ESnet who may have more 
insight.  

-jason

On Mar 27, 2014, at 9:55 AM, Roy Hockett 
<>
 wrote:

> Azher and Jason,
>       Can you say a bit more on the  thread competition issue?  How have 
> you seen this manifest itself?
> What was changed in the Ivy Bridge 2690 V2 processor to fix this issue?
> 
> Thanks,
> -Roy Hockett
> 
> Network Architect,
> ITS Communications Systems and Data Centers
> University of Michigan
> Tel: (734) 763-7325
> Fax: (734) 615-1727
> email: 
> 
> 
> On Mar 20, 2014, at 2:07 PM, Azher Mughal wrote:
> 
>> Hi Roy, All,
>> 
>> Regarding 100GE NICs, I heard from a vendor (netronome) that their NIC 
>> will occupy multiple Gen3 x8 slots :) rather than x16 (in case of 
>> Mellanox).
>> 
>> For processor as Jason mentioned, Ivy Bridge 2690 v2 should be used rather 
>> than the SandyBridge version as it has fixes on thread competition.
>> 
>> I agree my documentation is getting a little older, will try to update in 
>> coming weeks.
>> 
>> Cheers
>> -Azher
>> 
>> 
>> 
>> On 3/20/2014 10:50 AM, Roy Hockett wrote:
>>> Mark,
>>> I built and tested two 40GE boxes for SC13, with mixed results.
>>> 
>>> Keys that I found were the following:
>>> 
>>> Single 40Gbps connected Server
>>> ------------------------------------------------
>>> Sandy/Ivy Bridge Family processor
>>> - Support for PCIe V3
>>> - Minimum x8 lane slot
>>> 
>>> Dual 40Gbps connected server
>>> ------------------------------------------------
>>> Sandy/Ivy Bridge Family processor
>>> - 2 CPU each with PCIe v3 slots from each CPU.
>>> - Support for PCIe V3
>>> - Minimum x8 lane slot
>>> 
>>> I had boxes that had both PCI slots connected to the same CPU and ran
>>> into issues when I tried to go above 60Gbps of traffic with nuttcp, which
>>> is why i recommend making sure the PCI slots are connected to different 
>>> CPUs to avoid this issue.
>>> 
>>> Network Card
>>> ---------------------
>>> 40GE card was ConnectX®-3 VPI.  I know there are a couple others out, but
>>> have not been able to test them.
>>> 
>>> I am told by venters that 100Gbps cards will be PCIe v3 16 lanes.
>>> Be careful and look at the server architecture layout like the attached, 
>>> so you
>>> can see how the PCI3 slots are connected to the CPU, whether they are 8 
>>> lanes
>>> or 16 lanes, even though the connector is x16.
>>> 
>>> If you are doing Disk to Disk tests, the RAID controller and Disks are 
>>> important.
>>> 
>>> Many builtin RADI controllers have limitations, so make sure you under 
>>> stand them.
>>> 
>>> I know LSI have 12Gbps RAID controller that uses a PCIe v3 8 lane slot, 
>>> so even
>>> if you aren't purchasing this now, it might be good to have an extra PCIe 
>>> slot so
>>> you can use this if you want.
>>> 
>>> 
>>> Azher Mughal, did this back in 2011, but the information is still 
>>> relevant.
>>> 
>>> http://supercomputing.caltech.edu/archive/sc11/40gekit_setup.html
>>> 
>>> 
>>> <Mail Attachment.png>
>>> 
>>> 
>>> Thanks,
>>> -Roy Hockett
>>> 
>>> Network Architect,
>>> ITS Communications Systems and Data Centers
>>> University of Michigan
>>> Tel: (734) 763-7325
>>> Fax: (734) 615-1727
>>> email: 
>>> 
>>> 
>>> On Mar 20, 2014, at 12:08 PM, Christopher A Konger 
>>> <>
>>>  wrote:
>>> 
>>>> Joe Breen at Utah and Conan Moore at Colorado also are building/testing 
>>>> 40G pS boxes.
>>>> 
>>>> I have CC'd them on this in case you want to reach out directly (I think 
>>>> they have already conferred with Jason).
>>>> 
>>>> Chris Konger / 864-656-8140
>>>> 
>>>> From: 
>>>> 
>>>>  
>>>> [mailto:]
>>>>  On Behalf Of Jason Zurawski
>>>> Sent: Thursday, March 20, 2014 8:44 AM
>>>> To: Mark Gardner
>>>> Cc: 
>>>> 
>>>> Subject: Re: [perfsonar-user] 40-100 GigE Hardware Recommendations?
>>>> 
>>>> Hey Mark;
>>>> 
>>>> I spec'ed one out last fall, but didn't purchase or do any testing.  It 
>>>> uses a 40G card that we have tested on ESnet DTNs (Mellanox MCX353A-TCBT 
>>>> ConnectX3), Ivy Bridge motherboard/processor combo, and lots of memory.  
>>>> Spec is below, use at your own risk of course :)
>>>> 
>>>> Thanks;
>>>> 
>>>> -jason
>>>> 
>>>> 
>>>> 
>>>> On Mar 20, 2014, at 8:22 AM, Mark Gardner 
>>>> <>
>>>>  wrote:
>>>> 
>>>>> Has anyone built a perfSONAR node intended to support 40-100 Gbps 
>>>>> Ethernet?
>>>>> 
>>>>> We do not have that level of connectivity yet but are planning for the
>>>>> future. I think I have sufficient funds that I can purchase beefier
>>>>> hardware now to support those rates in the future. Does anyone have a
>>>>> spec that they would be willing to share?
>>>>> 
>>>>> Mark
>>>>> -- 
>>>>> Mark Gardner
>>>>> Network Research Manager
>>>>> Virginia Tech