perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

Mark,

     It may be worth noting, that all of our hosts are dual-nic/dual-stack hosts where there is a single server, with a latency NIC, and a Bandwidth NIC.  They also run IPv4 and IPv6 on those interfaces.  So it's entirely possible there's something about they way we've built the mesh that causes the problem with the clash of the UUID.

A single host might look like this:

DNS NAME 1 (latency) NIC 1

ps-ku-lt.perfsonar.kanren.net. 85034 IN A       164.113.32.57

ps-ku-lt.perfsonar.kanren.net. 85122 IN AAAA    2001:49d0:23c0:7::57

DNS NAME 2 (bandwidth) NIC 2

ps-ku-bw.perfsonar.kanren.net. 300 IN   A       164.113.32.145

ps-ku-bw.perfsonar.kanren.net. 300 IN   AAAA    2001:49d0:23c0:2::18

And as I sent directly to you, when tests do get scheduled, they appear to run just fine.  The graphs eventually fill in the dead spots after 24 hours or so when tests get re-scheduled and so my longer term graphs (further out that 24 hours) look more or less normal

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047

On Thu, Mar 22, 2018 at 6:53 PM, Mark Feit <> wrote:

Casey Russell writes:

 

     We've got a large mesh config, and for some time now (months) the owamp tests have not been scheduling reliably.  What I mean by that is tonight when the mesh config agent runs on them, somewhere around 30-40% of the latency tests in the mesh will fail to schedule (one way).  The same test, in the other direction between those hosts will probably schedule fine.  24 hours later, when it runs again, most of those will re-schedule just fine, but a new 30-40% fail to schedule.  

…

2018/03/21 04:21:34 (22276) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test throughput(ps-fhsu-bw.perfsonar.kanren.net->ps-ksu-bw.perfsonar.kanren.net), continuing with rest of config: 500 INTERNAL SERVER ERROR: Error while tasking ps-ksu-bw.perfsonar.kanren.net: Unable to post task to ps-ksu-bw.perfsonar.kanren.net: Task already exists.  All participants must be on separate systems.

 

2018/03/18 23:16:27 (30529) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test throughput(ps-ku-bw.perfsonar.kanren.net->ps-bryant-bw.perfsonar.kanren.net), continuing with rest of config: 500 INTERNAL SERVER ERROR: Error while tasking ps-bryant-bw.perfsonar.kanren.net: Unable to post task to ps-bryant-bw.perfsonar.kanren.net: Task already exists.  All participants must be on separate systems.

 

That’s a known error, but not expected under these circumstances.  Let me think aloud for a minute:

 

Meshconfig submits the task to the first participant (“A”), which assigns it an identifier.  A then submits the task to the second participant (“B”) under the same identifier.  The usual case is that B doesn’t have a task with that identifier and everything goes to plan.  If the task already exists on B, it will be rejected by B and, in turn, A with the error you see.  There are two things that can cause this to happen:  One is tasks from different systems having the same identifier.  The identifiers are version 4 (random) UUIDs.  The other is when the task has parameters that put both participants on the same system (e.g., pscheduler task throughput --dest localhost).  That makes A and B the same machine, and when A tries to task itself (as if it were B), it complains that the task is a duplicate.

 

I suspect that the latter is what’s going on here, because happenstance collisions in a 128-bit space should be exceedingly rare.  To the best of my knowledge, we’re not seeing this anywhere else, even at sites running large meshes.  This could be a case of something really weird happening network- or DNS-wise, but it’s also possible that Meshconfig is doing something silly like trying to task the wrong system.  Andy’s our resident guru on that subject and us out until next week, but I may have a peek at the Meshconfig sources to see if I can spot anything obvious.

 

 

2018/03/21 04:21:35 (22276) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test latencybg(ps-fhsu-lt.perfsonar.kanren.net->ps-esu-lt.perfsonar.kanren.net), continuing with rest of config: 500 Internal Server Error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

…

 

For this one, you’ll probably find an error with the same timestamp (plus or minus a bit) in the Apache logs.

 

Can I take it that you’re not having trouble with the tasks that do get scheduled?

 

--Mark