perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Petr Šesták writes:

 

Aug 05 10:26:13 ps01-l.farm.particle.cz runner[2405952]: runner WARNING

39765: Unable to retrieve run

https://130.184.244.8/pscheduler/tasks/6c36ec1f-e51a-4d6a-bb9f-540162709dcf/runs/c9b8849a-9e85-4765-b4c7-d6ae4e18cc1e:

400: Operation timed out after 14500 milliseconds with 0 bytes received

 

The host at the other end (dmz-perf.uark.edu) or the network between it and your system has problems.  I’ll cover that below.

 

Aug 05 10:27:54 ps01-l.farm.particle.cz scheduler[2398130]: scheduler

INFO     23438: Posting non-starting run at 2024-08-05T19:21:56+02:00

for task e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51: Gave up after too many

scheduling conflicts.

 

This happens when, after repeated attempts to schedule a run where a conflict arose during the scheduling process, the scheduler elected to stop trying.  It should be a very-rare event.

 

This task was still in your system, so I pulled it out through the API and determined that the other end is the system at UArk.  After running a couple of tests and other checks, I noticed that its clock is not synchronized and has drifted by nearly nine minutes:

 

$ pscheduler clock ps01-l.farm.particle.cz  130.184.244.8

clock      2024-08-07T15:05:34.982705+02:00 synchronized   ps01-l.farm.particle.cz

clock      2024-08-07T08:56:50.204645-04:00 unsynchronized 130.184.244.8

difference PT8M44.77806S unsafe

 

That will make it unable to properly run tests involving multiple pSchedulers (i.e., throughput).    The clock on your system is in correct sync (checked against one of mine).  The UArk system also appears to have some intermittent problems being able to schedule and run tests.   Based that, I’d say your system is fine and that you should disregard any errors that are a result of tasks involving the UArk system.

 

 

Aug 03 19:50:33 ps01-l.farm.particle.cz archiver[1055237]: archiver

WARNING  3204: Failed to archive

https://ps01-l.farm.particle.cz/pscheduler/tasks/e24f7abc-21d2-4cb3-88dc-67d30cc38989/runs/74f58e57-178b-4112-9485-13dff578d041

to http: Possibly-permanent archiver error: Program failed to start

after 3 tries: Timed out waiting for read

 

This task is no longer in your system, so I wasn’t able look at what it was trying to do.  If you can point me at something from the last 48 hours, I can take a look at it.

 

Hope that helps.

 

--Mark