Skip to Content.
Sympa Menu

perfsonar-user - [perfsonar-user] mesh config, bidirectional tests and storing results locally - can't wrap my head around what changed

Subject: perfSONAR User Q&A and Other Discussion

List archive

[perfsonar-user] mesh config, bidirectional tests and storing results locally - can't wrap my head around what changed


Chronological Thread 
  • From: Casey Russell <>
  • To: "" <>
  • Subject: [perfsonar-user] mesh config, bidirectional tests and storing results locally - can't wrap my head around what changed
  • Date: Wed, 2 Aug 2017 17:38:28 -0500
  • Ironport-phdr: 9a23:YaRUfRfumeXnzA1qDkNyuMj6lGMj4u6mDksu8pMizoh2WeGdxcW9bB7h7PlgxGXEQZ/co6odzbGH4+a4ASQp2tWoiDg6aptCVhsI2409vjcLJ4q7M3D9N+PgdCcgHc5PBxdP9nC/NlVJSo6lPwWB6nK94iQPFRrhKAF7Ovr6GpLIj8Swyuu+54Dfbx9GiTe5Yr5+Ngm6oRnMvcQKnIVuLbo8xAHUqXVSYeRWwm1oJVOXnxni48q74YBu/SdNtf8/7sBMSar1cbg2QrxeFzQmLns65Nb3uhnZTAuA/WUTX2MLmRdVGQfF7RX6XpDssivms+d2xSeXMdHqQb0yRD+t4b1rSBv1gykZMTA3/nzchshpgK9bpR6soQF0zYzJb4GPLPdwfq3Tc9AHS2RfQslcTDZODp+mYoYVE+YNIeRVoo/grFUOtxu+AgysCfvxxzBSnX/5w6072Pk9HwHbxwwgBMwBsHDQrN7oM6odTfq1zLTTzTXYcfxW3TP95ZPLch87p/GMR6x/cczLxUkpCQzFkkydpIr4ND2b0eQNtnKU7+tmVe+3jG4nrR9+oj6xyccwkIXJgJwaykzc+SV9wYY1I964R1Bmbt6lFptcrT2VN4xzQs86X2Fptic6yqEauZGlZigG0ogoxxnaa/CfdYiH+AnjW/yLLTd3g3JlZqqzhxOo/kmvxe38V8a030xSripCitnArHYN2ALP6sSfSfty5EGh2SyR2ADV8O1EJ147lbbdJpU8wbAwjoIevVrCEyPqmkj7iaGWe0Y/9eS07unqbanqqoOAOIJxlg7yLqQjl8m6DOgmLAQBQm6W8vmm2rL55032WrBKg+U2kqbHtJDaItwWpqujDA9U1oYv8gi/DzS63NgBkngLMkxJdw+dg4jmPFHOJ//4DfOhjFi2jDhrwPXGMqXgApXLMHfDjK/scah85kJAygc+yN5f6pFPBb0dJf/+VVP9uMDEARI8LwO43+bqBdB4248AR26AH7eVMKbIvl+J4uIvLfOMZIgQuDvlN/cl/ePujWQimVADeamp2YAaaHOiEfRgOUWWf3zsjs0HEWgUogoyVPbqh0GaUT5Pe3ayWLox5iklB4K8A4fDXYetgLqb0yehB5FWe3tGBU6WEXrzc4WEWuwMaD6JIsN/iDAEVL6hS5M/2hG0sg/11aZnIvTO9iIGqJ3jyYs92+qG3wk/7zJvCMKUySSQVGxutmIOWzIs2q1j+wpwxkrJmfxgjuZWDttV7ukMTxw3L7bdyfB3Edb/RliHc9uUHgWIWNKjVBo4Vd8gi+UTeF1wH9Hq2hvZwjGxDrsRv7+CAoY59OTa0mSndJU18GrPyKR01wpuecBIL2Dzw/Ny
Group,

     One of our hosts participates in a larger regional mesh, about the time of the upgrades to PS 4.0, many hosts in the mesh "went yellow" meaning no results are found for tests in that grid square.  I've just recently begun looking into why that is.  While I can tell that it's either an inability to store results on my local host, or an inability of the dashboard host to read them.  I can't wrap my head around what changed in 4.0 and what I need to do to fix it.  A number of the hosts in the mesh just seemed to work through the conversion just fine although according to the JSON, they're running the same tests and storing the same *(flipped) results from remote hosts.

     I suspect it's related to the thread I've copied in below between George Uhl and Andrew Lake from Back in April, but even having read it, I can't quite grasp why some hosts in the mesh are working and some aren't.  The mesh is at: http://ps.onenet.net/maddash-webui/index.cgi?grid=Quilt%20Latency  My host is ps-bryant-bw.perfsonar.kanren.net.  You can see the entire horizontal row (save one host) is yellow.  That's the row where all the results should be stored in/retrieved from the local MA on my machine.

I've verified that IPtables isn't blocking access to esmond from off-network (port 443/80), and I've tried adding IP based authentication for the remote hosts.  Both to no effect.

Any suggestions appreciated.  

Sincerely,
Casey Russell
Network Engineer
KanREN
phone785-856-9809
2029 Becker Drive, Suite 282
Lawrence, Kansas 66047
linkedin twitter twitter

On Thu, Apr 27, 2017 at 9:01 AM, Andrew Lake <> wrote:




On April 26, 2017 at 7:00:48 PM, Uhl, George D. (GSFC-423.0)[SGT INC] () wrote:

Thanks for the clarification, Andy.  So back in the 3.5.1 days I had identified pS nodes that I don’t manage as no_agent hosts with the intention of having my managed pS nodes initiate the bi-directional tests and send the bi-directional results the central MA.  Is that feature no longer available in pS 4.0?

You can still do no_agent with force bidirectional and your host will still initiate the test (i.e. be responsible for creating the pscheduler task) but in the caseof throughput, traceroute and ping tests the source is always the be one that sends it to the archiver regardless of the initator. Your OWAMP tests (latency and latencybg in pscheduler terminology) still work the same way since we can use the —flip option to have the local side be the only pscheduler participant and thus responsible for the archiving even when it is not the source. 





Thanks,
George

From: Andrew Lake <>
Date: Wednesday, April 26, 2017 at 4:27 PM
To: George Uhl <>, "" <>
Subject: Re: [perfsonar-user] mesh tests fail to archive results from reverse path tests

Hi George,

Sorry for the delay. The source of the test is always responsible for archiving and is the side that has all the info about whether it succeeded or not. If you swap mcln-ps.maxgigapop.net into the URL you should see what you are after:

2017-04-25T03:53:08-05:00 on mcln-ps.maxgigapop.net and enpl-pt2-10g.eos.nasa.gov with iperf3:

throughput --duration PT30S --source mcln-ps.maxgigapop.net --ip-version 4 --dest enpl-pt2-10g.eos.nasa.gov --window-size 1310720 --parallel 1

* Stream ID 4
Interval       Throughput     Retransmits    Current Window 
0.0 - 1.0      9.03 Gbps      0              2.03 MBytes    
1.0 - 2.0      8.87 Gbps      0              2.03 MBytes    
2.0 - 3.0      8.70 Gbps      0              2.03 MBytes    
3.0 - 4.0      8.92 Gbps      0              2.03 MBytes    
4.0 - 5.0      8.81 Gbps      0              2.03 MBytes    
5.0 - 6.0      8.50 Gbps      0              2.03 MBytes    
6.0 - 7.0      8.05 Gbps      0              2.03 MBytes    
7.0 - 8.0      7.79 Gbps      0              2.03 MBytes    
8.0 - 9.0      7.49 Gbps      0              2.03 MBytes    
9.0 - 10.0     8.43 Gbps      0              2.03 MBytes    
10.0 - 11.0    8.64 Gbps      0              2.03 MBytes    
11.0 - 12.0    8.46 Gbps      0              2.03 MBytes    
12.0 - 13.0    8.10 Gbps      0              2.03 MBytes    
13.0 - 14.0    7.80 Gbps      0              2.03 MBytes    
14.0 - 15.0    7.23 Gbps      0              2.03 MBytes    
15.0 - 16.0    7.04 Gbps      0              2.03 MBytes    
16.0 - 17.0    7.10 Gbps      0              2.03 MBytes    
17.0 - 18.0    6.99 Gbps      0              2.03 MBytes    
18.0 - 19.0    7.26 Gbps      0              2.03 MBytes    
19.0 - 20.0    7.32 Gbps      0              2.03 MBytes    
20.0 - 21.0    7.38 Gbps      0              2.03 MBytes    
21.0 - 22.0    7.37 Gbps      0              2.03 MBytes    
22.0 - 23.0    7.29 Gbps      0              2.03 MBytes    
23.0 - 24.0    7.21 Gbps      0              2.03 MBytes    
24.0 - 25.0    7.11 Gbps      0              2.03 MBytes    
25.0 - 26.0    7.07 Gbps      0              2.03 MBytes    
26.0 - 27.0    7.06 Gbps      0              2.03 MBytes    
27.0 - 28.0    6.99 Gbps      0              2.03 MBytes    
28.0 - 29.0    7.07 Gbps      0              2.03 MBytes    
29.0 - 30.0    7.07 Gbps      0              2.03 MBytes    

Summary
Interval       Throughput     Retransmits    
0.0 - 30.0     7.74 Gbps      0

Archivings:

  To esmond, Finished
    2017-04-25T03:53:53-05:00 400: Invalid JSON returned
    2017-04-25T03:54:58-05:00 400: Invalid JSON returned
    2017-04-25T04:04:05-05:00 400: Invalid JSON returned
    2017-04-25T05:06:55-05:00 400: Invalid JSON returned
    2017-04-25T06:07:27-05:00 400: Invalid JSON returned
    2017-04-25T07:07:32-05:00 400: Invalid JSON returned
    2017-04-25T08:12:35-05:00 400: Invalid JSON returned
    2017-04-25T09:13:20-05:00 400: Invalid JSON returned
    2017-04-25T10:16:57-05:00 400: Invalid JSON returned
    2017-04-25T11:17:02-05:00 400: Invalid JSON returned
    2017-04-25T12:17:08-05:00 400: Invalid JSON returned
    2017-04-25T13:17:13-05:00 400: Invalid JSON returned
    2017-04-25T14:18:37-05:00 400: Invalid JSON returned
    2017-04-25T15:22:30-05:00 Archiver permanently abandoned registering test after 14 attempt(s): 400: Invalid JSON returned


Does archive.eos.nasa.gov allow mcln-ps.maxgigapop.net to connect to it on port 443? Having the source be responsible for the archiving is a change from 3.5 and a result of some of the architectural changes.

Thanks,
Andy




On April 25, 2017 at 3:09:01 PM, Uhl, George D. (GSFC-423.0)[SGT INC] () wrote:

Since the pS 4.0 upgrade I’ve noticed that some tests results are not getting archived to a my central archive.  In these cases I manage one of the hosts and I test to a no_agent host.  It’s the test results sourced from the no_agent host that fail to be archived.  Drilling down into the pscheduler results on my managed host shows tests in both directions run successfully but only the managed->no_agent test results get archived.  Nothing obvious in my meshconfig-agent-tasks.conf file stands out to me that would indicate a cause. 

From my managed host:

# pscheduler result --archivings https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/bb0dd703-6559-4543-83d6-b3844bba516a/runs/356db092-d154-4a30-b728-1a256431635c

2017-04-25T06:53:59-04:00 on enpl-pt2-10g.eos.nasa.gov and mcln-ps.maxgigapop.net with iperf3:


throughput --duration PT30S --source enpl-pt2-10g.eos.nasa.gov --ip-version 4 --dest mcln-ps.maxgigapop.net --window-size 1310720 --parallel 1


* Stream ID 4

Interval       Throughput     Retransmits    Current Window 

0.0 - 1.0      6.19 Gbps      0              2.03 MBytes    

1.0 - 2.0      6.29 Gbps      0              2.03 MBytes    

2.0 - 3.0      6.28 Gbps      0              2.03 MBytes    

3.0 - 4.0      6.22 Gbps      0              2.03 MBytes    

4.0 - 5.0      6.15 Gbps      0              2.03 MBytes    

5.0 - 6.0      6.08 Gbps      0              2.03 MBytes    

6.0 - 7.0      5.89 Gbps      0              2.03 MBytes    

7.0 - 8.0      5.56 Gbps      0              2.03 MBytes    

8.0 - 9.0      5.06 Gbps      0              2.03 MBytes    

9.0 - 10.0     4.65 Gbps      0              2.03 MBytes    

10.0 - 11.0    4.38 Gbps      0              2.03 MBytes    

11.0 - 12.0    4.23 Gbps      0              2.03 MBytes    

12.0 - 13.0    4.25 Gbps      0              2.03 MBytes    

13.0 - 14.0    4.39 Gbps      0              2.03 MBytes    

14.0 - 15.0    6.29 Gbps      0              2.03 MBytes    

15.0 - 16.0    6.67 Gbps      0              2.03 MBytes    

16.0 - 17.0    6.64 Gbps      0              2.03 MBytes    

17.0 - 18.0    6.64 Gbps      0              2.03 MBytes    

18.0 - 19.0    6.68 Gbps      0              2.03 MBytes    

19.0 - 20.0    6.67 Gbps      0              2.03 MBytes    

20.0 - 21.0    6.66 Gbps      0              2.03 MBytes    

21.0 - 22.0    6.63 Gbps      0              2.03 MBytes    

22.0 - 23.0    6.63 Gbps      0              2.03 MBytes    

23.0 - 24.0    6.65 Gbps      0              2.03 MBytes    

24.0 - 25.0    6.68 Gbps      0              2.03 MBytes    

25.0 - 26.0    6.65 Gbps      0              2.03 MBytes    

26.0 - 27.0    6.66 Gbps      0              2.03 MBytes    

27.0 - 28.0    6.68 Gbps      0              2.03 MBytes    

28.0 - 29.0    6.66 Gbps      0              2.03 MBytes    

29.0 - 30.0    6.66 Gbps      0              2.03 MBytes    


Summary

Interval       Throughput     Retransmits    

0.0 - 30.0     6.06 Gbps      0


Archivings:


  To esmond, Finished

    2017-04-25T06:54:40-04:00 Succeeded




From the no_agent host:

# pscheduler result  --archivings https://enpl-pt2-10g.eos.nasa.gov/pscheduler/tasks/36bb4f1d-dd27-4e5b-8975-36d36b014af2/runs/2819c8ac-23be-480f-8ef5-30760ea0e5c4

2017-04-25T04:53:08-04:00 on mcln-ps.maxgigapop.net and enpl-pt2-10g.eos.nasa.gov with iperf3:


throughput --duration PT30S --source mcln-ps.maxgigapop.net --ip-version 4 --dest enpl-pt2-10g.eos.nasa.gov --window-size 1310720 --parallel 1


* Stream ID 4

Interval       Throughput     Retransmits    Current Window 

0.0 - 1.0      9.03 Gbps      0              2.03 MBytes    

1.0 - 2.0      8.87 Gbps      0              2.03 MBytes    

2.0 - 3.0      8.70 Gbps      0              2.03 MBytes    

3.0 - 4.0      8.92 Gbps      0              2.03 MBytes    

4.0 - 5.0      8.81 Gbps      0              2.03 MBytes    

5.0 - 6.0      8.50 Gbps      0              2.03 MBytes    

6.0 - 7.0      8.05 Gbps      0              2.03 MBytes    

7.0 - 8.0      7.79 Gbps      0              2.03 MBytes    

8.0 - 9.0      7.49 Gbps      0              2.03 MBytes    

9.0 - 10.0     8.43 Gbps      0              2.03 MBytes    

10.0 - 11.0    8.64 Gbps      0              2.03 MBytes    

11.0 - 12.0    8.46 Gbps      0              2.03 MBytes    

12.0 - 13.0    8.10 Gbps      0              2.03 MBytes    

13.0 - 14.0    7.80 Gbps      0              2.03 MBytes    

14.0 - 15.0    7.23 Gbps      0              2.03 MBytes    

15.0 - 16.0    7.04 Gbps      0              2.03 MBytes    

16.0 - 17.0    7.10 Gbps      0              2.03 MBytes    

17.0 - 18.0    6.99 Gbps      0              2.03 MBytes    

18.0 - 19.0    7.26 Gbps      0              2.03 MBytes    

19.0 - 20.0    7.32 Gbps      0              2.03 MBytes    

20.0 - 21.0    7.38 Gbps      0              2.03 MBytes    

21.0 - 22.0    7.37 Gbps      0              2.03 MBytes    

22.0 - 23.0    7.29 Gbps      0              2.03 MBytes    

23.0 - 24.0    7.21 Gbps      0              2.03 MBytes    

24.0 - 25.0    7.11 Gbps      0              2.03 MBytes    

25.0 - 26.0    7.07 Gbps      0              2.03 MBytes    

26.0 - 27.0    7.06 Gbps      0              2.03 MBytes    

27.0 - 28.0    6.99 Gbps      0              2.03 MBytes    

28.0 - 29.0    7.07 Gbps      0              2.03 MBytes    

29.0 - 30.0    7.07 Gbps      0              2.03 MBytes    


Summary

Interval       Throughput     Retransmits    

0.0 - 30.0     7.74 Gbps      0


Archivings:


    This task had no archivings.







Archive powered by MHonArc 2.6.19.

Top of Page