perfSONAR User Q&A and Other Discussion

["Pennington, Mike" <>]

Here are the results so far, it’s been stuck on NEAR-TERM SCHEDULE for a while now.  I can jump on video pretty much any time today if you are so game.

 

pscheduler diags

                       pScheduler System Diagnostic Dump

 

           Generated for root @ ps-10G-dual-core-hartford.cen.ct.gov

                          Wed Dec 13 10:34:18 EST 2017

 

 

-------------------------------------------------------------------------------

                                INSTALLED TESTS

-------------------------------------------------------------------------------

 

[

    {

        "description": "Measure the clock difference between hosts",

        "href": "https://localhost/pscheduler/tests/clock",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "clock",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Measure DNS transaction time",

        "href": "https://localhost/pscheduler/tests/dns",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "dns",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Consume time in the background",

        "href": "https://localhost/pscheduler/tests/idle",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "idle",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Consume time in the background - NOT FOR PRODUCTION",

        "href": "https://localhost/pscheduler/tests/idlebgm",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "idlebgm",

        "scheduling-class": "background-multi",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Consume time exclusively - NOT FOR PRODUCTION",

        "href": "https://localhost/pscheduler/tests/idleex",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "idleex",

        "scheduling-class": "exclusive",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Measure network latency between hosts",

        "href": "https://localhost/pscheduler/tests/latency",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "latency",

        "scheduling-class": "normal",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Run one-way latency tests in the background",

        "href": "https://localhost/pscheduler/tests/latencybg",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "latencybg",

        "scheduling-class": "background-multi",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Measure the round trip time between hosts",

        "href": "https://localhost/pscheduler/tests/rtt",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "rtt",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Test communication between two hosts using TCP",

        "href": "https://localhost/pscheduler/tests/simplestream",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "simplestream",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Measure network throughput between hosts",

        "href": "https://localhost/pscheduler/tests/throughput",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "throughput",

        "scheduling-class": "exclusive",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Trace the path between IP hosts",

        "href": "https://localhost/pscheduler/tests/trace",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "trace",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    }

]

 

 

-------------------------------------------------------------------------------

                               INSTALLED TOOLS

-------------------------------------------------------------------------------

 

[

    {

        "description": "Measure the clock difference between hosts",

        "href": "https://localhost/pscheduler/tests/clock",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "clock",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Measure DNS transaction time",

        "href": "https://localhost/pscheduler/tests/dns",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "dns",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Consume time in the background",

        "href": "https://localhost/pscheduler/tests/idle",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "idle",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Consume time in the background - NOT FOR PRODUCTION",

        "href": "https://localhost/pscheduler/tests/idlebgm",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "idlebgm",

        "scheduling-class": "background-multi",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Consume time exclusively - NOT FOR PRODUCTION",

        "href": "https://localhost/pscheduler/tests/idleex",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "idleex",

        "scheduling-class": "exclusive",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Measure network latency between hosts",

        "href": "https://localhost/pscheduler/tests/latency",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "latency",

        "scheduling-class": "normal",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Run one-way latency tests in the background",

        "href": "https://localhost/pscheduler/tests/latencybg",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "latencybg",

        "scheduling-class": "background-multi",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Measure the round trip time between hosts",

        "href": "https://localhost/pscheduler/tests/rtt",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "rtt",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Test communication between two hosts using TCP",

        "href": "https://localhost/pscheduler/tests/simplestream",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "simplestream",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Measure network throughput between hosts",

        "href": "https://localhost/pscheduler/tests/throughput",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "throughput",

        "scheduling-class": "exclusive",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Trace the path between IP hosts",

        "href": "https://localhost/pscheduler/tests/trace",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "trace",

        "scheduling-class": "background",

        "schema": 1,

        "version": "1.0"

    }

]

 

 

-------------------------------------------------------------------------------

                              TOOL CONFIGURATIONS

-------------------------------------------------------------------------------

 

nuttcp.conf:

 

###

# Configuration file for pScheduler nuttcp tool throughput tests

#

[nuttcp]

##

# Path to the nuttcp command. Default is /usr/bin/nuttcp

#

#nuttcp_cmd = /usr/bin/nuttcp

 

##

# The control port where nuttcp server will listen

# when it is the receiving endpoint. Default is 5001

#

#server_port = 5001

 

##

# The first data port where nuttcp server will listen

# when it is the receiving endpoint. Parallel tests will

# use one additional port above this per parallelization.

# Default is 5101

#

#data_port_start = 5101

 

 

iperf2.conf:

 

###

# Configuration file for pScheduler iperf2 tool throughput tests

#

[iperf2]

##

# Path to the iperf2 command. Default is /usr/bin/iperf

#

# iperf2_cmd = /usr/bin/iperf

 

##

# The port where iperf2 server will listen when this hos

# is the receiving endpoint. Default is 5001

#

#server_port = 5001

 

 

iperf3.conf:

 

###

# Configuration file for pScheduler iperf3 tool throughput tests

#

[iperf3]

##

# Path to the iperf3 command. Default is /usr/bin/iperf3

#

# iperf3_cmd = /usr/bin/iperf3

 

##

# The port where iperf3 server will listen when this hos

# is the receiving endpoint. Default is 5201

#

#server_port = 5201

 

 

owping.conf:

 

###

# Configuration file for pScheduler latency tests

#

[owping]

##

# Path to the owping command. Default is /usr/bin/owping

#

# owping_cmd = /usr/bin/owping

 

##

# Indicates that this server does not run an owamp-server so if a request comes in where

# it is the server role, then an error should be thrown

#

# disable_server = 0

 

##

# The port(s) where owamp-server is listening. The default is 861. It can be a single number

# or a range in the form of "N-M". The latter would be a case where you have multiple

# owamp-servers running

#

#control_ports = 861

 

 

powstream.conf:

 

###

# Configuration file for pScheduler latency tests

#

[powstream]

##

# Path to the powstream command. Default is /usr/bin/powstream

#

# powstream_cmd = /usr/bin/powstream

 

##

# Path to the owstats command used to read result files. Default is /usr/bin/owstats

#

# owstats_cmd = /usr/bin/owstats

 

##

# Path to the pkill command used to clean-up old tests

#

# pkill_cmd = /usr/bin/pkill

 

##

# Indicates the log level that powstream should use when it runs.

# Valid values: NONE, FATAL, WARN, INFO, DEBUG, ALL

#

# log_level = INFO

 

##

# Indicates the data directory that powstream use to store result files that will be read

# by the pscheduler tool to publish result. Default is /var/lib/pscheduler/tool/powstream

#

# data_dir = /var/lib/pscheduler/tool/powstream

 

##

# Indicates that data files should not be removed after powstream runs is set to 1.

# Otherwise they are removed as soon as pScheduler processes them. The default behavior is

# to remove them. If they are not removed they can use considerable disk space over time.

# This should only be used for advanced application that want the ability to access the raw

# files on disk.

#

# keep_data_files = 0

 

##

# The port(s) where owamp-server is listening. The default is 861. It can be a single number

# or a range in the form of "N-M". The latter would be a case where you have multiple

# owamp-servers running

#

#control_ports = 861

 

 

 

 

-------------------------------------------------------------------------------

                              INSTALLED ARCHIVERS

-------------------------------------------------------------------------------

 

[

    {

        "description": "Discard results - NOT FOR PRODUCTION",

        "href": "https://localhost/pscheduler/archivers/bitbucket",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "bitbucket",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Send results to Esmond",

        "href": "https://localhost/pscheduler/archivers/esmond",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "esmond",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Dummy archiver that some of the time - NOT FOR PRODUCTION",

        "href": "https://localhost/pscheduler/archivers/failer",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "failer",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Send a raw JSON result to a HTTP server",

        "href": "https://localhost/pscheduler/archivers/http",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "http",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Send a JSON result to RabbitMQ",

        "href": "https://localhost/pscheduler/archivers/rabbitmq",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "rabbitmq",

        "schema": 1,

        "version": "1.0"

    },

    {

        "description": "Send a raw JSON result to Syslog",

        "href": "https://localhost/pscheduler/archivers/syslog",

        "maintainer": {

            "email": "",

            "href": "http://www.perfsonar.net",

            "name": "perfSONAR Development Team"

        },

        "name": "syslog",

        "schema": 1,

        "version": "1.0"

    }

]

 

 

-------------------------------------------------------------------------------

                                  CLOCK STATE

-------------------------------------------------------------------------------

 

{

    "offset": 4.6014785766601562e-05,

    "reference": "secondary reference (2) from 198.124.252.90",

    "source": "ntp",

    "synchronized": true,

    "time": "2017-12-13T10:34:22.217827-05:00"

}

 

 

-------------------------------------------------------------------------------

                               NEAR-TERM SCHEDULE

-------------------------------------------------------------------------------

 

 

Mike Pennington

Network Engineer

Connecticut Education Network (CEN)

860-622-4566

www.cen.ct.gov

 

From: Mark Feit [mailto:]
Sent: Wednesday, December 13, 2017 7:37 AM
To: Pennington, Mike <>;
Subject: Re: [perfsonar-user] Best way to troubleshoot HTTP/HTTPS flapping?

 

Pennington, Mike writes:

 

One of our Perfsonar nodes has been acting funky over the past week or so.  Our monitoring shows HTTP/HTTPS going up and down at random times during the day, I’ve confirmed it is actually going down as I cannot access perfsonar-hartford.cen.ct.gov during those times.  It can be anywhere from a few minutes and has lasted up to 30 minutes.  I’ve rebooted multiple times, power cycled it, yum is all up to date, etc.  Tests and results seem fine, it just seems to be HTTP/HTTPS access.  I’ve looked through some of the httpd logs but nothing is jumping out at me. 

 

Based on what I can see on your system from the outside, Apache is up and pScheduler is up and answering queries that don’t have anything to do with the schedule.  The latter leads me to suspect that something in the database is jammed up, and I’d like to get a handle on what that is.

 

If you’d be so kind as to run “pscheduler diags” as root and send me the results off-list, I’ll have an initial look at what’s going on.  Alternately, if you want to jump on video sometime today, drop me a line and we can troubleshoot it live.

 

--Mark