Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] Recommendations of perfsonar server platform

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] Recommendations of perfsonar server platform


Chronological Thread 
  • From: Joe Metzger <>
  • To: Yannis Mitsos <>
  • Cc: Nina Jeliazkova <>,
  • Subject: Re: [perfsonar-user] Recommendations of perfsonar server platform
  • Date: Wed, 15 Dec 2010 15:46:10 -0600

Yannis,
Thanks for your feedback.

I was involved in another discussion this morning that really illustrated the
tradeoffs we have to make when working on different scales.

At one end of the continuum you have residential broadband type situations, where
most of the locations on your network have extremely limited equipment and capacity,
and you are looking to establish very general or coarse usability measurements.
In this environment, a measurement probe needs to be very in-expensive,
possibly just a small application or service that runs in the background on a
cable modem, or wireless router.

At the other end of the continuum, you have networks like ESnet, Internet2 & GEANT
where all of the nodes include big expensive equipment that are connected by multiple
10GE links. In this environment we are concerned with highly accurate measurement
tools that can help us debug subtle problems in high bandwidth circuits, some of which
are very expensive.

So it makes sense for me to deploy several rackmount servers in
a pop where I have $500,000 worth of routing & switching equipment.  Clearly this
doesn't make sense in all environments.

The recent changes in the perfSONAR-PS toolkit packaging methods should
make it significantly easier to use those tools in an imbeded environment
running a standard redhat OS.  Of course the host administrative type
stuff in there will probably need to be tweaked to deal with differing
device names, and enabling the 'right' set of services.  But this
should be fairly straight forward.

--Joe


On Dec 15, 2010, at 2:18 PM, Yannis Mitsos wrote:

Dear Joe,

Thank you for your enlightening response. It is obvious that I had a different scenario in my mind. Namely, how can put cheap (?) probes all around our network. Bear in mind that according to our policy, for each customer we provide a L2 CPE, operated by us to monitor the status of the fibre up to our customer's premises. Planting a probe (I was thinking for instance a stick-like embedded system) in each CPS would provide us very comprehensive and accurate data for the whole network.

Now, talking about porting, allow me to clarify something: I was considering standard embedded platforms that are equipped with typical components and may run *standard* linux distribution. Not embedded systems that we would have to port the kernel or re-compile the applications. 

From your words, I understand that there is no substantial problem in having perfsonar-ps executed in a "weaker" platform. I will try to elaborate a little bit more in my free time and if I come-up with concrete results, I will update the list. 

Best Regards,

Yannis

On Dec 14, 2010, at 17:37 , Joe Metzger wrote:

Yannis,
Somebody has suggested this idea at least twice a year for the last 2-3 years.
The 'green' aspects of this are worth considering, and it might improve the value
of perfSONAR to communities concerned with inexpensive low bandwidth networks.

The reason it hasn't taken off is that the primary perfSONAR contributors are
concerned with improving the ability measuring across very high performance
wide area networks, and diagnose problems on paths across multiple wide area
networks. The small, cheap 'plug computers' have limited processing power
which could reduce the accuracy of the measurements they take, and make
diagnosing problems more difficult. Also, porting perfSONAR tools to
this type of platform and supporting it would require resources that we
don't have at this time, and the personnel costs of porting & maintaining it
greatly outweigh the potential capital costs savings.

So, my response has always been that yes, I think it is a great idea. And
the perfSONAR collaboration will gladly provide you all the help we can,
if you are willing to invest the time and resources to do this.

I suspect that the 'green computing' and energy efficiency issues will heat
up and the capabilities of the plug computers will continue to grow to the
point where it will make economical or political sense for the principal 
contributors to invest the resources to port and maintain perfSONAR on that
type of platform in a couple of years.

--Joe



On Dec 14, 2010, at 3:15 AM, Yannis Mitsos wrote:

Dear Nina,

On Dec 14, 2010, at 11:02 , Nina Jeliazkova wrote:



On 14 December 2010 10:34, Yannis Mitsos <> wrote:
Dear All,

Speaking about hardware requirements, I was wondering if one has considered "porting" perfsonar-ps to a compact embedded platform. Instead of having big servers scattered around the network, a small and cheap device might be connected directly to an ethernet port of a CPE switch providing a good view of the network performance.


Can't help replying ... a wonderful idea, although I wonder if the current XML protocol will fit well in an embedded platform. 
Can you please elaborate on the latter. How is the XML correlated to the hardware? When I refer to embedded platform, I have in my mind hardware that is fully supported by the Linux kernel (no uClinux stuff) with adequate memory. 

Best regards,

Yannis 


Best regards,
Nina Jeliazkova
 

Regards,

Yannis

On Dec 8, 2010, at 21:49 , Alan Whinery wrote:

Since I hadn't actually done much with them yet, I was keeping to myself, but I just recently bought and deployed 4 of these as PerfSONAR throughput machines:

 PowerEdge R210 Chassis w/up to 2 Cabled HDs and Quad-Pack LED Diagnostics, Unit Price $1,074.96
 Operating System No Operating System
 Memory 4GB Memory (4x1GB), 1333MHz, Single Ranked UDIMM
 Processor Intel® Xeon® X3440, 2.53 GHz, 8M Cache, Turbo, HT
 Network Adapter Intel PRO 1000PT 1GbE Single Port NIC, PCIe-1
  Also separate built-in NIC for IPMI
 Hard Drive 500GB 7.2k RPM Serial ATA 3Gbps 3.5-in Cabled Hard Drive
 3Yr Basic Hardware Warranty Repair: 5x10 HW-Only, 5x10 NBD Onsite

 I don't buy optical drives for machines that are never going to see a CD or DVD after initial install. I would like to see flash devices (i.e. SATA or IDE SATA CF adapters) edge out optical. I have to buy a plane ticket to get to most of my devices.

Although they are intended as 10GbE nodes, I haven't bought 10 GbE for these yet, as most locations aren't tooled for it.  Performance test seems to be a niche dominated by Myricom and Chelsio, at the moment. Our two existing NICs are Myri.

========================

Also considered HP Proliant, which turned out to be more expensive for very similar machine.
========================

I also have 10 of these coming for use as PerfSONAR delay-oriented machines (and possible CAIDA Ark fillers):

Super Micro 1RU Chassis SYS-5015A-EHF -
    Intel® Atom™ D510 processor
(DMI)
    2x Intel 82574L Gigabit LAN
    1 PCI-e x4 slot
    real RS-232 serial port (critical for CDMA timing)
    IPMI
   Crucial 2GB Kit SO-DIMM DDR2 667MHz unbuffered
    ST31000520AS 1 TB Low Power HDD
    200 watt power supply.
I required the PCI-e slot as a fall-back in case I don't like the DMI-connected Intel LAN
Total $530 each, incl all taxes and fees -- through open public bid.

Although they haven't actually arrived yet. I'm nose deep in hard drives and RAM, which has limited entertainment, with no computers attached. Maybe in time for Christmas!

===================================
My current operational 10 GbE bwctl/NDT/NPAD box, which I have seen push between 7 & 8 Gbit/sec to Los Angeles:
Intel MB with dual Intel(R) Pentium(R) D CPU 3.20GHz 1 M cache
1 GB RAM
30 GB HDD
Myri-10G Dual-Protocol NIC (10G-PCIE-8A)
Which I didn't buy, but a colleague's desktop that he left behind when he moved to another university.
==============================

 -Alan

On 12/8/2010 6:26 AM, Buraglio, Nicholas D wrote:
No problem.  Here is the brief synopsis of what I ordered:

PowerEdge R210 Chassis
Intel Xeon X3430, 2.4 GHz, 8M Cache
4GB Memory 
500Gig Disk
DVD-ROM
3Yr Basic Hardware Warranty Repair: 5x10 HW-Only, 5x10 NBD Onsite

Total cost was a bit under $1100 per box and it should meet or exceed the requirements.  I'm also going to test this config with an Intel 10G card that I already have just to see how well it performs.   
For what it's worth, I purposely priced these out for cost savings over redundancy so that I could get more of them and instrument our networks in more locations.   The small footprint was nice as well since rack space is always a premium commodity.  

nb

---
Nick Buraglio   
Network Engineer
University of Illinois CITES / ICCN
GPG key 0x2E5B44F4
Phone: 217.689.4254


On Dec 7, 2010, at 8:12 PM, Thomas Tam wrote:

Nicks,

Thanks for the information! It would be much appreciated if you could share your hardware config.  

Regards,
-Thomas

On 10-12-07 5:16 PM, "Buraglio, Nicholas D"  wrote:

We're in the middle of doing this both on our campus and on our statewide optical network, so I'm by no means an expert, and only experienced in that I have a handful of test boxes out there.  I'm more in the same type of situation as you, "new deployment" status. 
I've started with a handful of 1G boxes to add to the few cobbled together test boxes we already have and plan to add a handful of 10G boxes specifically for throughput testing.  I used the ESnet KB as well as the page Jason posted for a starting point for my 1G hardware build:  http://fasterdata.es.net/ps_hardware.html
I've priced out a handful of slightly customized (more RAM, better processor) Dell PowerEdge R210, which are pretty inexpensive, have a small footprint and exceed the minimum requirements for a 1G perfSonar box.  Generally speaking, I've always stuck with Intel or Myricom 10G cards for my other applications (Intel has the best support for what I've used them for in the past, which is mostly pf_ring packet capture stuff). 
I'd be more than happy to share my hardware config if you're interested.

nb


---
Nick Buraglio  
Network Engineer
University of Illinois CITES / ICCN
GPG key 0x2E5B44F4
Phone: 217.689.4254


On Dec 7, 2010, at 2:15 PM, Thomas Tam wrote:

Hi

I am planning to deploy couple of perfonser servers in our network. At first, we are looking into deploying a 1GE connection to the network, but will migrate to a 10GE connection in a few months. I would love to hear recommendations of what server platforms and 10GE interface cards to be used from experienced perfsonar users.

Your information is greatly appreciated.

Best regards,
-Thomas
CANARIE NOC

      

    





Joe Metzger





Joe Metzger






Archive powered by MHonArc 2.6.16.

Top of Page