perfSONAR User Q&A and Other Discussion

["Garnizov, Ivan (RRZE)" <>]

Hello Pete,

 

Just by looking at the dashboard you shared it is impossible to identify your system. The failures are all over the place. From the messages applied I can only deduce that your host is: “perfsonar-1850.frgp.net”. Having so many failures on the dashboard means, that there are problems not only on your system, but there are problems on the remote systems as well. Meaning you most likely won’t be able to deal with all the problems you manage to diagnose.

 

The messages you applied signify problems on coordination of tasks from the remote side towards your host. From the mesh dashboard it appears you have lots of different problems.

 

Immediately from your logs it is apparent that a remote system does not allow your system to negotiate a test. Meaning your system is identified as a “non-friendly host” and cannot even submit a request for a test with “ps-svl-10g.cenic.net”. The resolution for this failure requires a discussion with the remote admin, so that his Limits config allows your system inquiries.

“2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(ps-svl-10g.cenic.net->perfsonar-1850.frgp.net), continuing with rest of config: 403 FORBIDDEN: Task forbidden by limits:

Hints:

  requester: 129.19.165.2

  server: 137.164.28.121

Identified as everybody

Classified as default

Application: Defaults applied to non-friendly hosts

  Group 1: Limit 'innocuous-tests' failed: Test type not in list

  Group 1: Want any, 0/1 passed, 1/1 failed: FAIL

  Group 1: Failed; stopping here.

  Application FAILS

Proposal does not meet limits

”

 

For the rest of the test failures I would suggest to verify initial communication with make a test with each remote system through pS (pScheduler) not the direct measurement tool:

·         pscheduler troubleshoot

·         pscheduler troubleshoot --host <host>

·         pscheduler task --debug trace --dest perfsonar-1850.frgp.net --source <host>

 

The steps above will not resolve the problems on the dashboard, but you have so many of issues, which require us to establish a solid base.

On top of these review the /var/log/pscheduler/pscheduler.log

There you will find information on service failures or communication problems, which need to be addressed in order to restore operation of your system

 

 

Regards,

Ivan Garnizov

 

GEANT SA1T2: pS deployments GN Operations

GEANT SA2T3: pS development team

GEANT SA3T5: eduPERT team

 

Jubiläumsjahr 2018 - IT in Bewegung

Das RRZE - der IT-Dienstleister der FAU

www.50-jahre.rrze.fau.de

 

Von: [mailto:] Im Auftrag von Pete Siemsen
Gesendet: Dienstag, 14. August 2018 02:21
An:
Betreff: [perfsonar-user] meshconfig problems

 

Ever since I upgraded to 4.0.2, my (the FRGP's) participation in the Quilt mesh at http://quiltmesh.onenet.net/maddash-webui/ has been problematic. My whole row is orange. Embarrassing. I've appended the last 25 lines from /var/log/perfsonar/meshconfig-agent.log file. These messages mystify me because I can ping and/or traceroute to most of the hosts that appear in these error messages, like these

 

ps-grand-bw.perfsonar.kanren.net

kc-core-psr.mo.more.net

 

perfsonar-1850$ tail --lines=25 meshconfig-agent.log

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(ps-svl-10g.cenic.net->perfsonar-1850.frgp.net), continuing with rest of config: 403 FORBIDDEN: Task forbidden by limits:

Hints:

  requester: 129.19.165.2

  server: 137.164.28.121

Identified as everybody

Classified as default

Application: Defaults applied to non-friendly hosts

  Group 1: Limit 'innocuous-tests' failed: Test type not in list

  Group 1: Want any, 0/1 passed, 1/1 failed: FAIL

  Group 1: Failed; stopping here.

  Application FAILS

Proposal does not meet limits

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(ps-grand-bw.perfsonar.kanren.net->perfsonar-1850.frgp.net), continuing with rest of config: 500 timeout: timeout

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(ps-grand-lt.perfsonar.kanren.net->perfsonar-1850.frgp.net), continuing with rest of config: 500 timeout: timeout

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(nmon-aa.mich.net->perfsonar-1850.frgp.net), continuing with rest of config: 500 timeout: timeout

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(kc-core-psr.mo.more.net->perfsonar-1850.frgp.net), continuing with rest of config: 500 timeout: timeout

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(latency.eugn-perfsonar.nero.net->perfsonar-1850.frgp.net), continuing with rest of config: 500 timeout: timeout

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(bandwidth.eugn-perfsonar.nero.net->perfsonar-1850.frgp.net), continuing with rest of config: 500 timeout: timeout

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(perfsonar-1850.frgp.net->web100.pnw-gigapop.net), continuing with rest of config: 500 INTERNAL SERVER ERROR: Unable to determine participants: Process took too long to run.

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(web100.pnw-gigapop.net->perfsonar-1850.frgp.net), continuing with rest of config: 500 INTERNAL SERVER ERROR: Unable to determine participants: Process took too long to run.

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(perfsonar.unl.edu->perfsonar-1850.frgp.net), continuing with rest of config: 500 timeout: timeout

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(perfsonar-msn.wiscnet.net->perfsonar-1850.frgp.net), continuing with rest of config: 500 timeout: timeout

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(perfsonar-1850.frgp.net->noctuidae.cns.vt.edu), continuing with rest of config: 500 INTERNAL SERVER ERROR: Unable to determine participants: Process took too long to run.

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(noctuidae.cns.vt.edu->perfsonar-1850.frgp.net), continuing with rest of config: 500 INTERNAL SERVER ERROR: Unable to determine participants: Process took too long to run.

2018/08/13 18:06:47 (26630) WARN> perfsonar_meshconfig_agent:430 main:: - Problem adding test trace(psonar.arc.vt.edu->perfsonar-1850.frgp.net), continuing with rest of config: 500 timeout: timeout

 

Any clue appreciated :-)

 

-- Pete