Skip to Content.
Sympa Menu

perfsonar-user - [perfsonar-user] Failing throughput tests

Subject: perfSONAR User Q&A and Other Discussion

List archive

[perfsonar-user] Failing throughput tests


Chronological Thread 
    < Chronological >      < Thread >
  • From: "Smith, Sebastian" <>
  • To: "" <>
  • Subject: [perfsonar-user] Failing throughput tests
  • Date: Mon, 16 Dec 2024 23:48:18 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=seattlechildrens.org; dmarc=pass action=none header.from=seattlechildrens.org; dkim=pass header.d=seattlechildrens.org; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=F8CzWP6RQohWEQuyC2aQEXXdldizjIs2xiydH3HQNTU=; b=fnJ8AHbgZ955g4LAf3Fkq1bPvBxZ5T37AQyxnEUrKVYjf/yi5uE7QR3rUN3btBPSGbeFgaSVRf0njYIMT9tsZ2uCjqSd/e97XDdEYQllCeir9FvAngWhGcfIHXdoQ5LyduJ8s68UEDbzabLnMtgK3lt7mdaLwk1BgqR6t2aeMJ0XAQ+gSum9UhTgIEkRIy4PP0ZTDaOTlSfocHoMALyafmLFDhC8wROyJuzcOWD4r+6rjhHDekDCBjqhOwjlAmSX/c2nOZX6kVRNkRGiGDb8ppv6KK5J1T7edfLp20atmN2WahycCKq7yctjfAB0UtbYANcXYBx6QFIWIvuq9xL3UQ==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BknYQfy2nkQBRI28+BhdGsyjRS/8NOYJpGvUf3iE56jTclEulNb0OrSqsGDFvOrYMLkioF5dqN64u4s+6fPoojDubEO+1NnXfajOVdoBV8UwJ2QYZvkTIc66hy4kr/ljevELTYdq9c7dG+RJTMVy7X4LGygZhG+kY162MQLKcuV0GwBt2TArKfompCgflz55JDlS3XuTEm2Chj877LlREDNflTR6ZF+Dez6sv1q8cCj7d01sCwWIaLrQvJ2sWVgHtvDo3XJcmje6vdQloCE34BYzAI8m6UcZlFaxxB+KrvfV+CKc3vlKW5THG+HSmaJklSW7Oz6u8LKyIIdqNpjTBw==
  • Msip_labels: MSIP_Label_046da4d3-ba20-4986-879c-49e262eff745_Enabled=True;MSIP_Label_046da4d3-ba20-4986-879c-49e262eff745_SiteId=9f693e63-5e9e-4ced-98a4-8ab28f9d0c2d;MSIP_Label_046da4d3-ba20-4986-879c-49e262eff745_SetDate=2024-12-16T23:13:05.1267714Z;MSIP_Label_046da4d3-ba20-4986-879c-49e262eff745_Name=Internal;MSIP_Label_046da4d3-ba20-4986-879c-49e262eff745_ContentBits=0;MSIP_Label_046da4d3-ba20-4986-879c-49e262eff745_Method=Standard

Hi,

 

We are deploying 10 perfSONAR nodes in a mesh model throughout our networks to help optimize performance.  Servers are running Ubuntu v22.04 and perfSONAR toolkit v5.1.3.  iperf3 throughput tests run well for a couple of days, then start failing.  I’ve copied some common warnings we’re seeing from runner and archiver in pscheduler.log below.  Can you offer any advice on how to stabilize throughput tests?

 

iperf3 tests run.  I think the results aren’t being archived.

Initially, we were planning to use a central archive, but we retreated on this idea due to complexity of managing firewalls and routing in some of our network zones – and we’ll be moving nodes around the network pretty rapidly searching for bottlenecks.  I believe the central archive configuration has been reverted on all nodes.

 

When throughput tests fail, we’re seeing errors from `pscheduler troubleshoot`.  Restarting logstash seems to clear the troubleshooter errors:
```
root@pdlperf5:/home/perfadmin# pscheduler troubleshoot

Performing basic troubleshooting of pdlperf5.

 

pdlperf5:

 

  Checking that host "pdlperf5" resolves... 127.0.1.1

  Measuring MTU... 65535 (Local)

  Looking for pScheduler... OK.

  Fetching API level... 6

  Checking clock... OK.

  Exercising API... Archivers... Contexts... Tests... Tools... OK.

  Fetching service status... OK.

  Checking services... Ticker... Scheduler... Runner... Archiver... OK.

  Checking limits... OK.

  Last run scheduled... 7 seconds ago

  Last run completed... Never

  Idle test.... 8 seconds... Pending, probably missed... Failed.

 

Did not get a result: Resource not found.

```

```

# pscheduler monitor error

0d 00:02:02      Failed       throughput --source 172.31.192.55 --source-node 172.31.192.55 --dest 172.31.192.53 --dest-node 172.31.192.53 --duration PT20S --ip-version 4

```

```

# pscheduler.log runner warning

Dec 16 15:10:20 pdlperf2 runner WARNING  72241: Unable to retrieve run https://172.31.192.53/pscheduler/tasks/3835a167-e9e2-4c90-bfe0-1278b8894361/runs/49ac3c9e-a4e1-49b5-ba66-7935118d06c6: 404: Resource not found.

 ```

```

# pscheduler.log archive warning

Dec 16 15:08:03 pdlperf2 archiver WARNING  58018: Failed to archive https://pdlperf2/pscheduler/tasks/e20eed17-e4fd-4f05-90de-b1ffe895c794/runs/d247fca1-3fcf-447a-aabb-ed1cbe4d099a to http: Failed to put result: 401: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

Dec 16 15:08:03 pdlperf2 archiver WARNING  <html><head>

Dec 16 15:08:03 pdlperf2 archiver WARNING  <title>401 Unauthorized</title>

Dec 16 15:08:03 pdlperf2 archiver WARNING  </head><body>

Dec 16 15:08:03 pdlperf2 archiver WARNING  <h1>Unauthorized</h1>

Dec 16 15:08:03 pdlperf2 archiver WARNING  <p>This server could not verify that you

Dec 16 15:08:03 pdlperf2 archiver WARNING  are authorized to access the document

Dec 16 15:08:03 pdlperf2 archiver WARNING  requested.  Either you supplied the wrong

Dec 16 15:08:03 pdlperf2 archiver WARNING  credentials (e.g., bad password), or your

Dec 16 15:08:03 pdlperf2 archiver WARNING  browser doesn't understand how to supply

Dec 16 15:08:03 pdlperf2 archiver WARNING  the credentials required.</p>

Dec 16 15:08:03 pdlperf2 archiver WARNING  </body></html>

```

 

Thanks for your time!

 

Sebastian Smith

Seattle Children’s | Enterprise Analytics

Systems Engineer, Principal

Email:

Phone: 1-206-395-4312

Web: https://seattlechildrens.org

 

-- 

 

CONFIDENTIALITY NOTICE: This e-mail, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

Archive powered by MHonArc 2.6.24.

Top of Page