perfSONAR User Q&A and Other Discussion

[Roy Hockett <>]

Mark,

I built and tested two 40GE boxes for SC13, with mixed results.

Keys that I found were the following:

Single 40Gbps connected Server

------------------------------------------------

Sandy/Ivy Bridge Family processor

- Support for PCIe V3

- Minimum x8 lane slot

Dual 40Gbps connected server

------------------------------------------------

Sandy/Ivy Bridge Family processor

- 2 CPU each with PCIe v3 slots from each CPU.

- Support for PCIe V3

- Minimum x8 lane slot

I had boxes that had both PCI slots connected to the same CPU and ran

into issues when I tried to go above 60Gbps of traffic with nuttcp, which

is why i recommend making sure the PCI slots are connected to different 

CPUs to avoid this issue.

Network Card

---------------------

40GE card was ConnectX®-3 VPI.  I know there are a couple others out, but

have not been able to test them.

I am told by venters that 100Gbps cards will be PCIe v3 16 lanes.

Be careful and look at the server architecture layout like the attached, so you

can see how the PCI3 slots are connected to the CPU, whether they are 8 lanes

or 16 lanes, even though the connector is x16.

If you are doing Disk to Disk tests, the RAID controller and Disks are important.

Many builtin RADI controllers have limitations, so make sure you under stand them.

I know LSI have 12Gbps RAID controller that uses a PCIe v3 8 lane slot, so even

if you aren't purchasing this now, it might be good to have an extra PCIe slot so

you can use this if you want.

Azher Mughal, did this back in 2011, but the information is still relevant.

http://supercomputing.caltech.edu/archive/sc11/40gekit_setup.html

Thanks,
-Roy Hockett

Network Architect,
ITS Communications Systems and Data Centers
University of Michigan
Tel: (734) 763-7325
Fax: (734) 615-1727
email:

On Mar 20, 2014, at 12:08 PM, Christopher A Konger <> wrote:

Joe Breen at Utah and Conan Moore at Colorado also are building/testing 40G pS boxes.

 

I have CC'd them on this in case you want to reach out directly (I think they have already conferred with Jason).

 

Chris Konger / 864-656-8140

 

From:  [mailto:perfsonar-] On Behalf Of Jason Zurawski
Sent: Thursday, March 20, 2014 8:44 AM
To: Mark Gardner
Cc: 
Subject: Re: [perfsonar-user] 40-100 GigE Hardware Recommendations?

 

Hey Mark;

I spec'ed one out last fall, but didn't purchase or do any testing.  It uses a 40G card that we have tested on ESnet DTNs (Mellanox MCX353A-TCBT ConnectX3), Ivy Bridge motherboard/processor combo, and lots of memory.  Spec is below, use at your own risk of course :)

Thanks;

-jason

On Mar 20, 2014, at 8:22 AM, Mark Gardner <> wrote:

> Has anyone built a perfSONAR node intended to support 40-100 Gbps Ethernet?
> 
> We do not have that level of connectivity yet but are planning for the
> future. I think I have sufficient funds that I can purchase beefier
> hardware now to support those rates in the future. Does anyone have a
> spec that they would be willing to share?
> 
> Mark
> -- 
> Mark Gardner
> Network Research Manager
> Virginia Tech
> --