Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] OWAMP tests not scheduling reliably in mesh

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] OWAMP tests not scheduling reliably in mesh


Chronological Thread 
  • From: Mark Feit <>
  • To: Casey Russell <>, "" <>
  • Subject: Re: [perfsonar-user] OWAMP tests not scheduling reliably in mesh
  • Date: Thu, 22 Mar 2018 23:53:19 +0000
  • Accept-language: en-US
  • Authentication-results: kanren.net; dkim=none (message not signed) header.d=none;kanren.net; dmarc=none action=none header.from=internet2.edu;
  • Ironport-phdr: 9a23:OwJo4BRg3hdvYxW6yra+6jTSR9psv+yvbD5Q0YIujvd0So/mwa67ZBeEt8tkgFKBZ4jH8fUM07OQ7/i7HzRYqb+681k6OKRWUBEEjchE1ycBO+WiTXPBEfjxciYhF95DXlI2t1uyMExSBdqsLwaK+i764jEdAAjwOhRoLerpBIHSk9631+ev8JHPfglEnjWwba98IRmssQndqtQdjJd/JKo21hbHuGZDdf5MxWNvK1KTnhL86dm18ZV+7SleuO8v+tBZX6nicKs2UbJXDDI9M2Ao/8LrrgXMTRGO5nQHTGoblAdDDhXf4xH7WpfxtTb6tvZ41SKHM8D6Uaw4VDK/5KpwVhTmlDkIOCI48GHPi8x/kqRboA66pxdix4LYeZyZOOZicq/Ye94RWGhPUdtLVyFZAo2ycZYBD/YPM+hboYnypVoOogexCgS3Huzj1jpIi2Xq0aEm0eksFxzN0gw6H9IJtXTZtNL7O70IUeC20aLGzSvMb/JK2Tzg74XIdx4hru+NXbJsasfRyE8vFx/bgVWKr4zqIS+V2voXv2eF8uVgSPuihmg6oA9/pTivw90jiojPho8NyVDL7yN5wJwrKt2+UkJ7Z8CrEIdWuiqHNIV2WtsvT39ytyom17ELvIO3cDUXxJkiyR7SZOCLf5SN7x/hSumcLi13iXdgdb6hmxq+7FCsxvDgWsS3ylpGsClInsPSun0C0xHf8NWLRuZg8ku51zaAyQPe5v1BLE0xlafUN4ItzqA1m5cTq0vOGjH5lUDygaKYa0op+e2l5Pr6bbr8qJ+RMZJ/hBvkPaQ0gMO/BPw1MggQUGif/uSxzKXt8FH+TrlWgPA6i7TUv5LEKcgCoa62GBFa3pwk6xaiEzepy9MYnWQBLF1YYh6Hl5LpO1bSIP/mEfi/n1WskDBtx/zcOb3hH4nNLnzEkLfmfrZx8VJTyA02zdxH5pJUDK8OIO7rV0PvrtPUEgI1Pgmpz+r6Fdlw040eVG2TDqOFNa7fs0GH6+01LOSJYYIZpirxJ+U96/7rl3A5mFsdfaez3ZsQbXC1BvFmI0uHbnrtntcMCmYKvgwiTOP0kl2CVyBcZ2qsU64m+D40FZ+mAZ/ZRo+xmLyBwDu7HppOa2BeFF+MC3nod56DW/cKci2SONZtkiEfVbe/UY8szhWutA7hy7p7NerY5DcUtZPl1Nhp+eLTjxcy+iJoD8iDyW2CUXx7nn5bDwMxiYt2ukFsgm2eyrN1h/gQQddJ+uhSXwM+HZ3VyfZ3DZb0Vx6XOp/DUFu8TM6hBzgrC88qzsUmYkBhFs+kgwyZmSemHvVdw6SGHpIv9aTVxT3sPMtn43fAyKQ7iVQ6GI1COXDw1YBl8A2GIofTkA23mqe2PfAZ0iPM6E+Cy3aDpkdVTFQ2XKnYCyNMLnDKpMj0sxuRB4SlDq4qZ04YkZaP
  • Spamdiagnosticoutput: 1:0

Casey Russell writes:

 

     We've got a large mesh config, and for some time now (months) the owamp tests have not been scheduling reliably.  What I mean by that is tonight when the mesh config agent runs on them, somewhere around 30-40% of the latency tests in the mesh will fail to schedule (one way).  The same test, in the other direction between those hosts will probably schedule fine.  24 hours later, when it runs again, most of those will re-schedule just fine, but a new 30-40% fail to schedule.  

…

2018/03/21 04:21:34 (22276) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test throughput(ps-fhsu-bw.perfsonar.kanren.net->ps-ksu-bw.perfsonar.kanren.net), continuing with rest of config: 500 INTERNAL SERVER ERROR: Error while tasking ps-ksu-bw.perfsonar.kanren.net: Unable to post task to ps-ksu-bw.perfsonar.kanren.net: Task already exists.  All participants must be on separate systems.

 

2018/03/18 23:16:27 (30529) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test throughput(ps-ku-bw.perfsonar.kanren.net->ps-bryant-bw.perfsonar.kanren.net), continuing with rest of config: 500 INTERNAL SERVER ERROR: Error while tasking ps-bryant-bw.perfsonar.kanren.net: Unable to post task to ps-bryant-bw.perfsonar.kanren.net: Task already exists.  All participants must be on separate systems.

 

That’s a known error, but not expected under these circumstances.  Let me think aloud for a minute:

 

Meshconfig submits the task to the first participant (“A”), which assigns it an identifier.  A then submits the task to the second participant (“B”) under the same identifier.  The usual case is that B doesn’t have a task with that identifier and everything goes to plan.  If the task already exists on B, it will be rejected by B and, in turn, A with the error you see.  There are two things that can cause this to happen:  One is tasks from different systems having the same identifier.  The identifiers are version 4 (random) UUIDs.  The other is when the task has parameters that put both participants on the same system (e.g., pscheduler task throughput --dest localhost).  That makes A and B the same machine, and when A tries to task itself (as if it were B), it complains that the task is a duplicate.

 

I suspect that the latter is what’s going on here, because happenstance collisions in a 128-bit space should be exceedingly rare.  To the best of my knowledge, we’re not seeing this anywhere else, even at sites running large meshes.  This could be a case of something really weird happening network- or DNS-wise, but it’s also possible that Meshconfig is doing something silly like trying to task the wrong system.  Andy’s our resident guru on that subject and us out until next week, but I may have a peek at the Meshconfig sources to see if I can spot anything obvious.

 

 

2018/03/21 04:21:35 (22276) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test latencybg(ps-fhsu-lt.perfsonar.kanren.net->ps-esu-lt.perfsonar.kanren.net), continuing with rest of config: 500 Internal Server Error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

…

 

For this one, you’ll probably find an error with the same timestamp (plus or minus a bit) in the Apache logs.

 

Can I take it that you’re not having trouble with the tasks that do get scheduled?

 

--Mark

 


Archive powered by MHonArc 2.6.19.

Top of Page