perfSONAR User Q&A and Other Discussion

[Carl Hayter <>]

I'm having a bit of a weird issue with maddash.  We have 8 perfsonar
boxes (updated to 3.5) and I installed maddash one one
of them, created a mesh configuration and configured all the machines
to pull it down, created the maddash config, and crossed my fingers...

Funny thing is, on the dashboard, the OWAMP tests show up fine.  The
throughput tests all stay orange and say 'Unable to find any tests
with data in the given time range ...'.

Looking at individual boxes, the tests in the mesh config are running
and collecting data.  If I go to hpc-perfsonar.usc.edu I can view
the graph for the test against phe-perfsonar.usc.edu, and if I go
to phe-perfsonar.usc.edu I can view the graph for the test against
hpc-perfsonar.usc.edu.

On the maddash page, the HPC - PHE cells are orange and have the error
message, if I click on the cells I do see a graph with the correct
test data.

http://hpc-perfsonar.usc.edu/maddash-webui/index.cgi?dashboard=USC%20TEN-II

http://hpc-perfsonar.usc.edu/maddash-webui/details.cgi?uri=/maddash/grids/USC+TEN-II+-+USC+TEN-II+Throughput+Testing/phe-perfsonar.usc.edu/hpc-perfsonar.usc.edu/Throughput

http://hpc-perfsonar.usc.edu/maddash-webui/details.cgi?uri=/maddash/grids/USC+TEN-II+-+USC+TEN-II+Throughput+Testing/hpc-perfsonar.usc.edu/phe-perfsonar.usc.edu/Throughput+Alternate+MA

If I do a `curl http://phe-perfsonar.usc.edu/esmond/perfsonar/archive/`
I get a bunch of JSON data, same for hpc-perfsonar.usc.edu.

If I run the `check_throughput.pl` nagios script (I think this is how the
maddash data is acquired...), I get seemingly normal data.

~~~ on hpc-perfsonar.usc.edu ~~~

H=hpc-perfsonar.usc.edu
P=phe-perfsonar.usc.edu
C='/opt/perfsonar_ps/nagios/bin/check_throughput.pl -p tcp --tool 
bwctl/iperf3 -w 7: -c 20: -r 86400'

# PHE->HPC from PHE ma
$C -s $P -d $H -a $P -u http://$P/esmond/perfsonar/archive
PS_CHECK_THROUGHPUT CRITICAL - Average throughput is 9.840Gbps | Count=9;; 
Min=9.60168;; Max=9.90942;; Average=9.84042555555556;; 
Standard_Deviation=0.121737880414458;;

# HPC->PHE from PHE ma
$C -s $H -d $P -a $P -u http://$P/esmond/perfsonar/archive
PS_CHECK_THROUGHPUT CRITICAL - Average throughput is 9.273Gbps | Count=7;; 
Min=5.64209;; Max=9.9051;; Average=9.27343285714286;; 
Standard_Deviation=1.60169745801878;;

# HPC->PHE from HPC ma
$C -s $H -d $P -a $H -u http://$H/esmond/perfsonar/archive
PS_CHECK_THROUGHPUT CRITICAL - Average throughput is 9.816Gbps | Count=4;; 
Min=9.76832;; Max=9.83687;; Average=9.8162625;; 
Standard_Deviation=0.0322087476473979;;

# PHE->HPC from HPC ma
$C -s $P -d $H -a $H -u http://$H/esmond/perfsonar/archive
PS_CHECK_THROUGHPUT CRITICAL - Average throughput is 9.542Gbps | Count=4;; 
Min=8.44071;; Max=9.9097;; Average=9.54223;; 
Standard_Deviation=0.734346699182354;;

~~~

So, how/why does my dashboard show the OWAMP results correctly but not
the Throughput results?

I've attached the 'usc-tenii.conf' (mesh config file), 'usc-tenii.json'
(via /opt/perfsonar_ps/mesh_config/bin/generate_configuration), and
'maddash.yaml' (via 
/opt/perfsonar_ps_mesh_config/bin/generate_gui_configuration).

Any ideas?

----
Carl Hayter
ITS - Data Network Operations
University of Southern California

description USC TEN-II

<administrator>
        name    Chris Ho
        email   

</administrator>

<organization>
        description     USC TEN-II

        <site>
                <host>
                        description     HPC
                        address hpc-perfsonar.usc.edu
                        <measurement_archive>
                                type perfsonarbuoy/bwctl
                                read_url        
http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type perfsonarbuoy/owamp
                                read_url        
http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type traceroute
                                read_url        
http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                </host>
                <host>
                        description     ZNI
                        address zni-perfsonar.usc.edu
                        <measurement_archive>
                                type perfsonarbuoy/bwctl
                                read_url        
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type perfsonarbuoy/owamp
                                read_url        
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type traceroute
                                read_url        
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                </host>
                <host>
                        description     PHE
                        address phe-perfsonar.usc.edu
                        <measurement_archive>
                                type perfsonarbuoy/bwctl
                                read_url        
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type perfsonarbuoy/owamp
                                read_url        
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type traceroute
                                read_url        
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                </host>
                <host>
                        description     RRI
                        address rri-perfsonar.usc.edu
                        <measurement_archive>
                                type perfsonarbuoy/bwctl
                                read_url        
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type perfsonarbuoy/owamp
                                read_url        
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type traceroute
                                read_url        
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                </host>
                <host>
                        description     VHE
                        address vhe-perfsonar.usc.edu
                        <measurement_archive>
                                type perfsonarbuoy/bwctl
                                read_url        
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type perfsonarbuoy/owamp
                                read_url        
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type traceroute
                                read_url        
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                </host>
                <host>
                        description     ACB
                        address acb-perfsonar.usc.edu
                        <measurement_archive>
                                type perfsonarbuoy/bwctl
                                read_url        
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type perfsonarbuoy/owamp
                                read_url        
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type traceroute
                                read_url        
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                </host>
                <host>
                        description     SCB
                        address scb-perfsonar.usc.edu
                        <measurement_archive>
                                type perfsonarbuoy/bwctl
                                read_url        
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type perfsonarbuoy/owamp
                                read_url        
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type traceroute
                                read_url        
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                </host>
                <host>
                        description     DETER
                        address deter-perfsonar.usc.edu
                        <measurement_archive>
                                type perfsonarbuoy/bwctl
                                read_url        
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type perfsonarbuoy/owamp
                                read_url        
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                        <measurement_archive>
                                type traceroute
                                read_url        
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
                                write_url       
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
                        </measurement_archive>
                </host>
        </site>
</organization>

<test_spec bwctl_test>
        type    perfsonarbuoy/bwctl
        tool    bwctl/iperf3
        protocol        tcp
        interval        21600 # 6h
        duration        30
        force_bidirectional 1
        random_start_percentage 10
        omit_interval   5 # ignore first few results
</test_spec>

<test_spec owamp_test>
        type    perfsonarbuoy/owamp
        packet_interval 0.1
        loss_threshold  10
        session_count   18000 # refresh session 1/2 hr 18000 packets
        sample_count 600
        packet_padding  0
        bucket_width    0
        force_bidirectional     1
</test_spec>

<test_spec traceroute_test>
        type    traceroute
        test_interval   600
        protocol        udp
        timeout 30
        pause   0
        waittime        5
        packet_size     40
</test_spec>

<group tenii_bwctl_group>
        type    mesh
        member  hpc-perfsonar.usc.edu
        member  zni-perfsonar.usc.edu
        member  phe-perfsonar.usc.edu
        member  rri-perfsonar.usc.edu
        member  vhe-perfsonar.usc.edu
        member  acb-perfsonar.usc.edu
        member  scb-perfsonar.usc.edu
        member  deter-perfsonar.usc.edu
</group>

<group tenii_owamp_group>
        type    mesh
        member  hpc-perfsonar.usc.edu
        member  zni-perfsonar.usc.edu
        member  phe-perfsonar.usc.edu
        member  rri-perfsonar.usc.edu
        member  vhe-perfsonar.usc.edu
        member  acb-perfsonar.usc.edu
        member  scb-perfsonar.usc.edu
        member  deter-perfsonar.usc.edu
</group>

<test>
        description USC TEN-II Throughput Testing
        group   tenii_bwctl_group
        test_spec       bwctl_test
</test>

<test>
        description USC TEN-II OWAMP Testing
        group   tenii_owamp_group
        test_spec       owamp_test
</test>

<test>
        description USC TEN-II Traceroute Testing
        group   tenii_bwctl_group
        test_spec       traceroute_test
</test>

Attachment: usc-tenii.json
Description: usc-tenii.json

checks:
  USC_TEN-II_-_USC_TEN-II_OWAMP_Testing_-_Loss:
    checkInterval: 1800
    critical_description: Loss rate is >= 0.01
    description: Loss from %row to %col
    excludeChecks:
      acb-perfsonar.usc.edu:
        - acb-perfsonar.usc.edu
      deter-perfsonar.usc.edu:
        - deter-perfsonar.usc.edu
      hpc-perfsonar.usc.edu:
        - hpc-perfsonar.usc.edu
      phe-perfsonar.usc.edu:
        - phe-perfsonar.usc.edu
      rri-perfsonar.usc.edu:
        - rri-perfsonar.usc.edu
      scb-perfsonar.usc.edu:
        - scb-perfsonar.usc.edu
      vhe-perfsonar.usc.edu:
        - vhe-perfsonar.usc.edu
      zni-perfsonar.usc.edu:
        - zni-perfsonar.usc.edu
    id: USC_TEN-II_-_USC_TEN-II_OWAMP_Testing_-_Loss
    name: Loss
    ok_description: Loss rate is <= 0
    params:
      command: /opt/perfsonar_ps/nagios/bin/check_owdelay.pl -u %maUrl -w 0 
-c 0.01 -r 900 -l -p -s %row -d %col -a %row
      graphUrl:
        acb-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%row
        deter-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%row
        hpc-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%row
        phe-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%row
        rri-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%row
        scb-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%row
        vhe-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%row
        zni-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%row
      maUrl:
        acb-perfsonar.usc.edu:
          default: http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
        deter-perfsonar.usc.edu:
          default: http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
        hpc-perfsonar.usc.edu:
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
        phe-perfsonar.usc.edu:
          default: http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
        rri-perfsonar.usc.edu:
          default: http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
        scb-perfsonar.usc.edu:
          default: http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
        vhe-perfsonar.usc.edu:
          default: http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
        zni-perfsonar.usc.edu:
          default: http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
    retryAttempts: 1
    retryInterval: 600
    timeout: 60
    type: net.es.maddash.checks.PSNagiosCheck
    warning_description: Loss rate is >= 0
  USC_TEN-II_-_USC_TEN-II_OWAMP_Testing_-_Loss_Alternate_MA:
    checkInterval: 1800
    critical_description: Loss rate is >= 0.01
    description: Loss from %row to %col
    excludeChecks:
      acb-perfsonar.usc.edu:
        - acb-perfsonar.usc.edu
      deter-perfsonar.usc.edu:
        - deter-perfsonar.usc.edu
      hpc-perfsonar.usc.edu:
        - hpc-perfsonar.usc.edu
      phe-perfsonar.usc.edu:
        - phe-perfsonar.usc.edu
      rri-perfsonar.usc.edu:
        - rri-perfsonar.usc.edu
      scb-perfsonar.usc.edu:
        - scb-perfsonar.usc.edu
      vhe-perfsonar.usc.edu:
        - vhe-perfsonar.usc.edu
      zni-perfsonar.usc.edu:
        - zni-perfsonar.usc.edu
    id: USC_TEN-II_-_USC_TEN-II_OWAMP_Testing_-_Loss_Alternate_MA
    name: Loss Alternate MA
    ok_description: Loss rate is <= 0
    params:
      command: /opt/perfsonar_ps/nagios/bin/check_owdelay.pl -u %maUrl -w 0 
-c 0.01 -r 900 -l -p -s %row -d %col -a %col
      graphUrl:
        acb-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%col
        deter-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%col
        hpc-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%col
        phe-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%col
        rri-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%col
        scb-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%col
        vhe-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%col
        zni-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&agent=%col
      maUrl:
        acb-perfsonar.usc.edu:
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        deter-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        hpc-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
        phe-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        rri-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        scb-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        vhe-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        zni-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
    retryAttempts: 1
    retryInterval: 600
    timeout: 60
    type: net.es.maddash.checks.PSNagiosCheck
    warning_description: Loss rate is >= 0
  USC_TEN-II_-_USC_TEN-II_Throughput_Testing_-_Throughput:
    checkInterval: 300
    critical_description: Throughput <= 20000Mbps
    description: Throughput from %row to %col
    excludeChecks:
      acb-perfsonar.usc.edu:
        - acb-perfsonar.usc.edu
      deter-perfsonar.usc.edu:
        - deter-perfsonar.usc.edu
      hpc-perfsonar.usc.edu:
        - hpc-perfsonar.usc.edu
      phe-perfsonar.usc.edu:
        - phe-perfsonar.usc.edu
      rri-perfsonar.usc.edu:
        - rri-perfsonar.usc.edu
      scb-perfsonar.usc.edu:
        - scb-perfsonar.usc.edu
      vhe-perfsonar.usc.edu:
        - vhe-perfsonar.usc.edu
      zni-perfsonar.usc.edu:
        - zni-perfsonar.usc.edu
    id: USC_TEN-II_-_USC_TEN-II_Throughput_Testing_-_Throughput
    name: Throughput
    ok_description: Throughput >= 7000Mbps
    params:
      command: '/opt/perfsonar_ps/nagios/bin/check_throughput.pl -u %maUrl -w 
7: -c 20: -r 86400 -s %row -d %col -a %row -p tcp --tool "bwctl/iperf3"'
      graphUrl:
        acb-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%row
        deter-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%row
        hpc-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%row
        phe-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%row
        rri-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%row
        scb-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%row
        vhe-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%row
        zni-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%row
      maUrl:
        acb-perfsonar.usc.edu:
          default: http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
        deter-perfsonar.usc.edu:
          default: http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
        hpc-perfsonar.usc.edu:
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
        phe-perfsonar.usc.edu:
          default: http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
        rri-perfsonar.usc.edu:
          default: http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
        scb-perfsonar.usc.edu:
          default: http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
        vhe-perfsonar.usc.edu:
          default: http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
        zni-perfsonar.usc.edu:
          default: http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
    retryAttempts: 3
    retryInterval: 600
    timeout: 60
    type: net.es.maddash.checks.PSNagiosCheck
    warning_description: Throughput < 7000Mbps
  USC_TEN-II_-_USC_TEN-II_Throughput_Testing_-_Throughput_Alternate_MA:
    checkInterval: 300
    critical_description: Throughput <= 20000Mbps
    description: Throughput from %row to %col
    excludeChecks:
      acb-perfsonar.usc.edu:
        - acb-perfsonar.usc.edu
      deter-perfsonar.usc.edu:
        - deter-perfsonar.usc.edu
      hpc-perfsonar.usc.edu:
        - hpc-perfsonar.usc.edu
      phe-perfsonar.usc.edu:
        - phe-perfsonar.usc.edu
      rri-perfsonar.usc.edu:
        - rri-perfsonar.usc.edu
      scb-perfsonar.usc.edu:
        - scb-perfsonar.usc.edu
      vhe-perfsonar.usc.edu:
        - vhe-perfsonar.usc.edu
      zni-perfsonar.usc.edu:
        - zni-perfsonar.usc.edu
    id: USC_TEN-II_-_USC_TEN-II_Throughput_Testing_-_Throughput_Alternate_MA
    name: Throughput Alternate MA
    ok_description: Throughput >= 7000Mbps
    params:
      command: '/opt/perfsonar_ps/nagios/bin/check_throughput.pl -u %maUrl -w 
7: -c 20: -r 86400 -s %row -d %col -a %col -p tcp --tool "bwctl/iperf3"'
      graphUrl:
        acb-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%col
        deter-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%col
        hpc-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%col
        phe-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%col
        rri-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%col
        scb-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%col
        vhe-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%col
        zni-perfsonar.usc.edu:
          default: 
/serviceTest/graphWidget.cgi?url=%maUrl&source=%row&dest=%col&protocol=tcp&tool=bwctl/iperf3&agent=%col
      maUrl:
        acb-perfsonar.usc.edu:
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        deter-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        hpc-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
        phe-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        rri-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        scb-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        vhe-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          zni-perfsonar.usc.edu: 
http://zni-perfsonar.usc.edu/esmond/perfsonar/archive
        zni-perfsonar.usc.edu:
          acb-perfsonar.usc.edu: 
http://acb-perfsonar.usc.edu/esmond/perfsonar/archive
          default: http://hpc-perfsonar.usc.edu/esmond/perfsonar/archive
          deter-perfsonar.usc.edu: 
http://deter-perfsonar.usc.edu/esmond/perfsonar/archive
          phe-perfsonar.usc.edu: 
http://phe-perfsonar.usc.edu/esmond/perfsonar/archive
          rri-perfsonar.usc.edu: 
http://rri-perfsonar.usc.edu/esmond/perfsonar/archive
          scb-perfsonar.usc.edu: 
http://scb-perfsonar.usc.edu/esmond/perfsonar/archive
          vhe-perfsonar.usc.edu: 
http://vhe-perfsonar.usc.edu/esmond/perfsonar/archive
    retryAttempts: 3
    retryInterval: 600
    timeout: 60
    type: net.es.maddash.checks.PSNagiosCheck
    warning_description: Throughput < 7000Mbps
dashboards:
  -
    added_by_mesh_agent: 1
    grids:
      -
        name: 'USC TEN-II - USC TEN-II Throughput Testing'
      -
        name: 'USC TEN-II - USC TEN-II OWAMP Testing'
    name: USC TEN-II
database: /var/lib/maddash/
grids:
  -
    added_by_mesh_agent: 1
    checks:
      - USC_TEN-II_-_USC_TEN-II_Throughput_Testing_-_Throughput
      - USC_TEN-II_-_USC_TEN-II_Throughput_Testing_-_Throughput_Alternate_MA
    colOrder: alphabetical
    columnAlgorithm: all
    columns: USC_TEN-II_-_USC_TEN-II_Throughput_Testing
    excludeSelf: 1
    name: 'USC TEN-II - USC TEN-II Throughput Testing'
    rowOrder: alphabetical
    rows: USC_TEN-II_-_USC_TEN-II_Throughput_Testing
    statusLabels:
      critical: Throughput <= 20000Mbps
      notrun: Check has not yet run
      ok: Throughput >= 7000Mbps
      unknown: Unable to retrieve data
      warning: Throughput < 7000Mbps
  -
    added_by_mesh_agent: 1
    checks:
      - USC_TEN-II_-_USC_TEN-II_OWAMP_Testing_-_Loss
      - USC_TEN-II_-_USC_TEN-II_OWAMP_Testing_-_Loss_Alternate_MA
    colOrder: alphabetical
    columnAlgorithm: all
    columns: USC_TEN-II_-_USC_TEN-II_OWAMP_Testing
    excludeSelf: 1
    name: 'USC TEN-II - USC TEN-II OWAMP Testing'
    rowOrder: alphabetical
    rows: USC_TEN-II_-_USC_TEN-II_OWAMP_Testing
    statusLabels:
      critical: Loss rate is >= 0.01
      notrun: Check has not yet run
      ok: Loss rate is <= 0
      unknown: Unable to retrieve data
      warning: Loss rate is >= 0
groupMembers:
  -
    added_by_mesh_agent: yes
    id: hpc-perfsonar.usc.edu
    label: HPC
  -
    added_by_mesh_agent: yes
    id: zni-perfsonar.usc.edu
    label: ZNI
  -
    added_by_mesh_agent: yes
    id: phe-perfsonar.usc.edu
    label: PHE
  -
    added_by_mesh_agent: yes
    id: rri-perfsonar.usc.edu
    label: RRI
  -
    added_by_mesh_agent: yes
    id: vhe-perfsonar.usc.edu
    label: VHE
  -
    added_by_mesh_agent: yes
    id: acb-perfsonar.usc.edu
    label: ACB
  -
    added_by_mesh_agent: yes
    id: scb-perfsonar.usc.edu
    label: SCB
  -
    added_by_mesh_agent: yes
    id: deter-perfsonar.usc.edu
    label: DETER
groups:
  USC_TEN-II_-_USC_TEN-II_OWAMP_Testing:
    - zni-perfsonar.usc.edu
    - rri-perfsonar.usc.edu
    - hpc-perfsonar.usc.edu
    - phe-perfsonar.usc.edu
    - deter-perfsonar.usc.edu
    - scb-perfsonar.usc.edu
    - vhe-perfsonar.usc.edu
    - acb-perfsonar.usc.edu
  USC_TEN-II_-_USC_TEN-II_Throughput_Testing:
    - zni-perfsonar.usc.edu
    - rri-perfsonar.usc.edu
    - hpc-perfsonar.usc.edu
    - phe-perfsonar.usc.edu
    - deter-perfsonar.usc.edu
    - scb-perfsonar.usc.edu
    - vhe-perfsonar.usc.edu
    - acb-perfsonar.usc.edu
http:
  port: 8881
serverHost: localhost