perfSONAR User Q&A and Other Discussion

[Jack Wallis <>]

Hi

 

Thanks for the checks to run. It has the below:

checks:

  Dashboard_1_-_Dashboard_1_-_latency_task_-_Loss-forward:

    added_by_psconfig: 1

    checkInterval: 1800

    description: Queries an esmond MA for one-way loss data and alerts on response

    name: Packet Loss

    params:

      command: /usr/lib/nagios/plugins/check_loss.pl -s %row.map.address -r 2700 -t 60 -d %col.map.address -u %maUrl -w 0.001 -c 0.1

      graphUrl: /perfsonar-graphs/graphWidget.cgi?displaysetsrc=%row.displayset&url="%maUrl&dest=%col.map.address&reverseurl=%maUrlReverse&source=%row.map.address&displaysetdest=%col.displayset

      maUrl:

        Hex89:

          default: http://10.149.196.250/esmond/perfsonar/archive

        Sal1:

          default: http://10.149.196.250/esmond/perfsonar/archive

        TCYMan:

          default: http://10.149.196.250/esmond/perfsonar/archive

    retryAttempts: 1

    retryInterval: 300

    timeout: 60

    type: net.es.maddash.checks.PSNagiosCheck

 

I run the command manually (filling in the variables with source/destination and measurement archive url) I get the below:

 

./check_loss.pl -u  https://10.149.196.250/esmond/perfsonar/archive -d 10.149.196.2 -w 0.001 -c 0.1 -s 10.149.196.14 -r 2700 -t 60

PS_CHECK_LOSS UNKNOWN - Unable to find any tests with data in the given time range where source is 10.149.196.14 and destination is 10.149.196.2

 

So it looks to me like its not able to read the information from the archive location.  If I browse the Esmond webinterface and set the below, I get some information.

The only thing that strikes me as a little odd is the ‘time-updated’ field on all the entries appears to be null.  Could that cause an issue with the check_log

 

http://10.149.196.250/esmond/perfsonar/archive/?source=10.149.196.2&destination=10.149.196.14

HTTP 200 OK

Allow: GET, POST, HEAD, OPTIONS

Content-Type: application/json

Vary: Accept

 

[

    {

        "url": "http://85.159.135.151/esmond/perfsonar/archive/c5b03a1f94ab44adb8e30ab75f4410bb/",

        "metadata-key": "c5b03a1f94ab44adb8e30ab75f4410bb",

        "subject-type": "point-to-point",

        "event-types": [

            {

                "summaries": [],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c5b03a1f94ab44adb8e30ab75f4410bb/packet-retransmits/base",

                "event-type": "packet-retransmits"

            },

            {

                "summaries": [],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c5b03a1f94ab44adb8e30ab75f4410bb/pscheduler-run-href/base",

                "event-type": "pscheduler-run-href"

            },

            {

                "summaries": [

                    {

                        "summary-type": "average",

                        "uri": "/esmond/perfsonar/archive/c5b03a1f94ab44adb8e30ab75f4410bb/throughput/averages/86400",

                        "time-updated": null,

                        "summary-window": "86400"

                    }

                ],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c5b03a1f94ab44adb8e30ab75f4410bb/throughput/base",

                "event-type": "throughput"

            },

            {

                "summaries": [],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c5b03a1f94ab44adb8e30ab75f4410bb/failures/base",

                "event-type": "failures"

            },

            {

                "summaries": [],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c5b03a1f94ab44adb8e30ab75f4410bb/throughput-subintervals/base",

                "event-type": "throughput-subintervals"

            },

            {

                "summaries": [],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c5b03a1f94ab44adb8e30ab75f4410bb/packet-retransmits-subintervals/base",

                "event-type": "packet-retransmits-subintervals"

            }

        ],

        "source": "10.149.196.2",

        "destination": "10.149.196.14",

        "measurement-agent": "10.149.196.2",

        "tool-name": "pscheduler/iperf3",

        "input-source": "10.149.196.2",

        "input-destination": "10.149.196.14",

        "ip-transport-protocol": "tcp",

        "pscheduler-reference-psconfig-created-by-user-agent": "psconfig-pscheduler-agent",

        "pscheduler-reference-psconfig-created-by-uuid": "8bc86ef3-aec5-45ee-b4ce-96bd478e42cc",

        "pscheduler-test-type": "throughput",

        "time-duration": "39.0",

        "uri": "/esmond/perfsonar/archive/c5b03a1f94ab44adb8e30ab75f4410bb/",

        "metadata-count-total": 2,

        "metadata-previous-page": null,

        "metadata-next-page": null

    },

    {

        "url": "http://85.159.135.151/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/",

        "metadata-key": "c6f5b3624e91480da703b78616969b5e",

        "subject-type": "point-to-point",

        "event-types": [

            {

                "summaries": [],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/time-error-estimates/base",

                "event-type": "time-error-estimates"

            },

            {

                "summaries": [],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-duplicates/base",

                "event-type": "packet-duplicates"

            },

            {

                "summaries": [],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/pscheduler-run-href/base",

                "event-type": "pscheduler-run-href"

            },

            {

                "summaries": [],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/histogram-ttl/base",

                "event-type": "histogram-ttl"

            },

            {

                "summaries": [

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-count-sent/aggregations/300",

                        "time-updated": null,

                        "summary-window": "300"

                    },

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-count-sent/aggregations/3600",

                        "time-updated": null,

                        "summary-window": "3600"

                    },

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-count-sent/aggregations/86400",

                        "time-updated": null,

                        "summary-window": "86400"

                    }

                ],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-count-sent/base",

                "event-type": "packet-count-sent"

            },

            {

                "summaries": [],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-reorders/base",

                "event-type": "packet-reorders"

            },

            {

                "summaries": [

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-count-lost/aggregations/300",

                        "time-updated": null,

                        "summary-window": "300"

                    },

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-count-lost/aggregations/3600",

                        "time-updated": null,

                        "summary-window": "3600"

                    },

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-count-lost/aggregations/86400",

                        "time-updated": null,

                        "summary-window": "86400"

                    }

                ],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-count-lost/base",

                "event-type": "packet-count-lost"

            },

            {

                "summaries": [

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/histogram-owdelay/aggregations/300",

                        "time-updated": null,

                        "summary-window": "300"

                    },

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/histogram-owdelay/aggregations/3600",

                        "time-updated": null,

                        "summary-window": "3600"

                    },

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/histogram-owdelay/aggregations/86400",

                        "time-updated": null,

                        "summary-window": "86400"

                    },

                    {

                        "summary-type": "statistics",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/histogram-owdelay/statistics/0",

                        "time-updated": null,

                        "summary-window": "0"

                    },

                    {

                        "summary-type": "statistics",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/histogram-owdelay/statistics/300",

                        "time-updated": null,

                        "summary-window": "300"

                    },

                    {

                        "summary-type": "statistics",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/histogram-owdelay/statistics/3600",

                        "time-updated": null,

                        "summary-window": "3600"

                    },

                    {

                        "summary-type": "statistics",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/histogram-owdelay/statistics/86400",

                        "time-updated": null,

                        "summary-window": "86400"

                    }

                ],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/histogram-owdelay/base",

                "event-type": "histogram-owdelay"

            },

            {

                "summaries": [],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/failures/base",

                "event-type": "failures"

            },

            {

                "summaries": [

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-loss-rate/aggregations/300",

                        "time-updated": null,

                        "summary-window": "300"

                    },

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-loss-rate/aggregations/3600",

                        "time-updated": null,

                        "summary-window": "3600"

                    },

                    {

                        "summary-type": "aggregation",

                        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-loss-rate/aggregations/86400",

                        "time-updated": null,

                        "summary-window": "86400"

                    }

                ],

                "time-updated": null,

                "base-uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/packet-loss-rate/base",

                "event-type": "packet-loss-rate"

            }

        ],

        "source": "10.149.196.2",

        "destination": "10.149.196.14",

        "measurement-agent": "10.149.196.2",

        "tool-name": "pscheduler/powstream",

        "input-source": "10.149.196.2",

        "input-destination": "10.149.196.14",

        "pscheduler-reference-psconfig-created-by-user-agent": "psconfig-pscheduler-agent",

        "pscheduler-reference-psconfig-created-by-uuid": "8bc86ef3-aec5-45ee-b4ce-96bd478e42cc",

        "pscheduler-test-type": "latencybg",

        "sample-size": "600",

        "time-duration": "86400.0",

        "time-probe-interval": "0.1",

        "uri": "/esmond/perfsonar/archive/c6f5b3624e91480da703b78616969b5e/"

    }

]

 

 

 

Jack Wallis​

Core Network Specialist

T & SfB: +44 1612 680 003

|

M: +44 7393 014 765

|

E:

Company information 
GCI Network Solutions Limited (registered in England & Wales with Company No.  04082862 and VAT No. 199872429, registered office address: Global House, 2 Crofton Close, Lincoln LN3 4NT). GCI refers to the group wide GCI organisation, and may refer to GCI Network Solutions Limited or one or more of the other subsidiary companies of GCI Managed Services Group Limited, each of which is a separate legal entity.  
 
Confidentiality
This message is intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to us, and immediately and permanently delete it. Do not use, copy or disclose the information contained in this message or in any attachment.
 
Privacy Policy and Terms of Use 
 
For information about how we process data and monitor communications please see our Privacy Policy and for terms of use please see our Acceptable Use Policy 
 

From: <> On Behalf Of Valentin Vidic
Sent: 11 March 2019 13:33
To:
Subject: Re: [perfsonar-user] Maddash Dashboard Grid is Down

 

On Fri, Mar 08, 2019 at 04:56:42PM +0000, Jack Wallis wrote:
> We've recently installed the Perfsonar Centralized Management package on an
> Ubuntu 18.04.2 LTS server, and are having an issue seeing the test results
> sent by our perfsonar test-points to the Maddash server/Centralized
> Measurement Archive.
>
> In the dashboard is just shows an orange box on the grid, when hovering over
> it shows 'unable to find any tests with data within the given time range where
> source is x.x.x.x and destination is y.y.y.y', as well as saying one issue is
> effecting the entire grid, and that is apparently due to the configuration.
> I've followed all the suggested troubleshooting steps:

Can you check if the /etc/maddash/maddash-server/maddash.yaml contains the
appropriate check commands like:

checks:
Staging_Testbed_-_Ping_Between_Testbeds_-_Ping_Loss-forward:
added_by_psconfig: 1
checkInterval: 1800
description: Queries an esmond MA for bidirectional loss data and alerts on response
name: Loss
params:
command: /usr/lib64/nagios/plugins/check_ping_loss.pl -4 -d %col.map.address -t 60 -u %maUrl -s %row.map.address -c 0.1 -w 0.001 -r 2700
graphUrl: /perfsonar-graphs/graphWidget.cgi?displaysetdest=%col.displayset&source=%row.map.address&dest=%col.map.address&reverseurl=%maUrlReverse&displaysetsrc=%row.displayset&url="%maUrl
You can than test if the command works properly when run manually from the
command line.

--
Valentin Vidic
Computer Systems Engineer - Expert
Department of Computer Infrastructure and Services
Croatian Academic and Research Network - CARNET
Josipa Marohnica 5, HR-10000 Zagreb, Croatia
tel: +385 1 6661 714, fax. +385 1 6661 635
gsm: +385 91 2480 919
www.CARNet.hr

---

This email originated outside of the GCI corporate network. Please treat any attachments or hyperlinks with caution. If you suspect this to be malicious please report it to GCI Internal IT.

---