perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Onno Zweers writes:

 

[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl

…

Gave up after too many scheduling conflicts.

 

Every so often when two pSchedulers are trying to arrange a run, the time slot they were planning to use gets consumed by something else in the middle of the process.  The usual process is to start over and try again, and most times it gets resolved because everyone trying to grab schedule time backs off for a random interval.  Sometimes these conflicts happen too many times in a row and the scheduler is forced to give up.

 

I pulled the schedule from both machines and found that the one at SURF is fairly crowded, which makes this a more-likely occurrence.

 

 

[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl

…

Unable to fetch runs: No such run.

 

This happens when the CLI is trying to fetch the first run for a task and the API says it can’t find one.  It can happen if the scheduling process takes too long and is something of a side effect of what I described above. 

 

There’s been a latent bug in the scheduler where the API makes a run that isn’t fully-scheduled yet available and the CLI grabs it before a conflict causes it to be deleted.  This became a more-visible problem in 4.4.0 when we made part of the scheduling process more efficient.   The upcoming 4.4.1 includes an interlock to prevent it, and you’ll get the “too many conflicts” message.

 

 

 

[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl

…

Run failed.

 

Error:

… 

  iperf3: error - unable to send control message: Bad file descriptor

…

Diagnostics from perfsonar-bandwidth.grid.surfsara.nl:

  numactl -N 0 /usr/bin/iperf3 -p 5201 -B 2001:610:108:203a::32 -c 2a07:8504:120:e068::72 -t 10 --json --rsa-public-key-path /run/pscheduler-server/runner/tmp/tmp80cl8f6t/tmp0xaouzpa/public-key --username H0ZOxEmIfGkflgY1leYT

 

Error from luisteraar.nikhef.nl:

  iperf3 returned an error: 

  

  Process took too long to run.

 

 

Without a lot more detail, it’s hard to say precisely why this is happening, but my guess is that it has something to do with IPv6 at SURF.  But if I can pontificate:

 

We’ve seen instances where a program tries to do a DNS lookup and takes an excessively-long time to do it because of a misconfiguration.  That forces pScheduler to kill it so it doesn’t bleed over into any other run’s time slot.  Unfortunately, there isn’t a way for us to get good diagnostics in that situation.  I ran the same task forced to by adding --ip-version 4 to the command line results in a successful test.  Forced IPv6 resulted in the same failure, which leads me to believe there might be a misconfiguration on the network somewhere.  I’d have a look at IPv6 DNS, firewalls and routing, in that order.

 

Hope that helps.

 

--Mark