Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] 100G testing configuration

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] 100G testing configuration


Chronological Thread 
    < Chronological >      < Thread >
  • From: "Curtis, Bruce" <>
  • To: "ELGER, NATHAN" <>
  • Cc: "" <>
  • Subject: Re: [perfsonar-user] 100G testing configuration
  • Date: Fri, 19 Jun 2020 20:00:12 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ndsu.edu; dmarc=pass action=none header.from=ndsu.edu; dkim=pass header.d=ndsu.edu; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5BnlyD5TSWUjAQk5yvN0y9Y75SSEQYwHF2hLR9hG2jc=; b=L+r/NEgZ3BAWT/sRHABPFUqoBg7Ne7Ri2dVG6pd1sK/QS+4IdDh0JBQ5mzfg4xb1TDz1j7HDderiQ3+Whm6jeO3cLLfYqJof3IZfufRIOE1bfFCL0gbFx7/DsmExITG+h0ubalC1zI8/NRyw/COr7fR2Gv/q6/QLUu/0gjGXs6kTGd/FddxVITIfmW/R8rUTMZg+gI/bf5nYRXP9ecyxgpfdykPO6WvBcKwzzrw9UeFiLMGqyro+aoqV80deXlZa05k1W5Lxiw39VEWJY0UM69aMTA5yycodM42lq3zjwfJrcBE6NgSbtvIDjNhqNs4jDQHAxFahMM/Fvi4h1NRX0A==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QkYKYSbfMZFwr/LMM9v2Qb0aDbRZYeODk5QkubgARhO4XgynNpItSw1H4KnZlxTJuRYJBVkr7zAe4sBp2D0y3y9j9u0LwTrBFvvdr+Kn4fEq9dd2O1ioHOSXvRsH/3EJN2uolneVib/0hlixw0UxwckIGGfw2p17/K0TcvaIjrhawhxLfIyLU6W9MfxQ4pze+Q+lTGQiGEHxCVfywWTeW2jgM4kY8fS5mUn4n813jMg4TpRet/j6Ksh5XbTBH5enOVF+updlAq85BLiPtTD3ls7lgWSRRIaGN70QCHUFYqSZvp1zrC8/Ld+s549plskLyb9yCO09LUkSKFdwAZ7NBA==


On Jun 16, 2020, at 1:23 PM, ELGER, NATHAN <> wrote:

Hello there,

We've recently purchased a few 100G hosts to use for perfsonar testing across our campus 100G network as well as external perfsonar hosts in the R&E space. These are high spec Dell machines using mellanox connectX-6 adapters configured for ethernet. I have loaded the latest perfsonar images and configured the servers with the recommended esnet tuning parameters for 100g testing. I have also applied bios tuning for HPC workloads as recommended by Dell, and updated the kernel to change the congestion control algorithm to BBR

My issue is that with any scheduled throughput tests to 100G hosts, such as TACC, we are only seeing around 20gbit forward with very asymmetrical speeds back to us- around 5gbit.

  You are using BBR to transmit.

Do you know what the other perfsonar nodes are using when transmitting to you?
If they use Cubic and you use BBR that could explain the asymmetry.

Cubic does better with multiple parallel sessions.  Are your scheduled tests configured to run multiple parallel iperf3 streams?


Some tests to 40gbit hosts at other institutions have even worse asymmetry- a test to gatech has about 200mbit back to us and we should both be going over I2. Even with a scheduled throughput test between my 2 perfsonar hosts in the same switch, I only see ~15gbit between them. I have scripted 8 parallel iperf3 tests and although there are a lot of retransmissions, I do see a total of around 50-60gbit for the testing period of 30 seconds.

Are these expected numbers for a 100G test over pscheduler? Is there some configuration I might be missing, or a better method for testing external 100G hosts? Our previous perfsonar machines were 10gbit, just to log network health between our HPC clusters and a few regional hosts, so I am somewhat a novice at really pushing this 100G network. 

I really appreciate any pointers! 

Nathan Elger
UofSC Research Computing
HPC systems architect
1301 Gervais St. ste 750

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user

Bruce Curtis
Network Engineer  /  Information Technology
NORTH DAKOTA STATE UNIVERSITY
phone: 701.231.8527






Archive powered by MHonArc 2.6.19.

Top of Page