Skip to Content.
Sympa Menu

perfsonar-user - RE: [perfsonar-user] Best way to troubleshoot HTTP/HTTPS flapping?

Subject: perfSONAR User Q&A and Other Discussion

List archive

RE: [perfsonar-user] Best way to troubleshoot HTTP/HTTPS flapping?


Chronological Thread 
  • From: "Pennington, Mike" <>
  • To: "Garnizov, Ivan (RRZE)" <>, Mark Feit <>, "" <>
  • Subject: RE: [perfsonar-user] Best way to troubleshoot HTTP/HTTPS flapping?
  • Date: Fri, 15 Dec 2017 12:19:17 +0000
  • Accept-language: en-US
  • Authentication-results: spf=none (sender IP is ) ;
  • Ironport-phdr: 9a23:4JK3NRceFwlgxSQ5802XQ+pxlGMj4u6mDksu8pMizoh2WeGdxc25ZBeN2/xhgRfzUJnB7Loc0qyK6/mmATRIyK3CmUhKSIZLWR4BhJdetC0bK+nBN3fGKuX3ZTcxBsVIWQwt1Xi6NU9IBJS2PAWK8TW94jEIBxrwKxd+KPjrFY7OlcS30P2594HObwlSizexfa5+IA+qoQnNq8IbnZZsJqEtxxXTv3BGYf5WxWRmJVKSmxbz+MK994N9/ipTpvws6ddOXb31cKokQ7NYCi8mM30u683wqRbDVwqP6WACXWgQjxFFHhLK7BD+Xpf2ryv6qu9w0zSUMMHqUbw5Xymp4rx1QxH0ligIKz858HnWisNuiqJbvAmhrAF7z4LNfY2ZKOZycqbbcNwUX2pBWttaWTJHDI2ycoADC/MNMOZdo4T7ulAArwaxBRO0Ce3v1DFGm2P206M53eQvEw7G3RAvEd0Bv3nPsNX1LbseXfypwKXUzTjOae5d1zfn6IjPdxAsuf+AU6hufsrXz0kvFgXFjlqMqYziOTOZzvkGvW+a7+pjS+2jhXMspgZ0ojiu2sgsipPGiZwRylze7SV23oA1Jdu5SE59bt6kDJxQtyeBOotrXswiRGdouCAgxrIYpJG7YTEHxZI6zBDRbPyHdpKH4hPlVOuJPTh4mGhpeL24hxas6USgyvH8VtWs0FZPtCVFk9/MtmwX2BzS7MiIVOF98Vmk2TmV1gDT7vlELl4ularVMJ4hxbgwmYAJsUjaAyD5hlv2g7GSdkUj4uin9f7nYrTgpp+AKYB0kAf+PboymsChG+g3LBUBX3WD9eSkyrLj50P5QK5Njv0wiKbZqorWJcIFqa6lBQJez4Ut6w6nAju4ytsUgWQLIVdYdB6akoTkPl/OLf/kAfujhlmsnilkyv/YMrDkB5jALGbPnbP5cbt/9kJQ0hQ/wN5f6p5OF70NPv3+VlXsuNHXEhM0NRG/zfz9B9VnzIweXHqCAq+HP6PWtl+F/vohLvWLaoMJpTrxMuUo6PzwgXMgnl8SZrem0YEQaHClAvRpOEKZYWfqgtgcC2sKphA+TOv2iFKcTTFTe3eyX6U65j0hD4KmEJvDRoSqgLyG3ye3BIFZZmdDClyUEHfocZuLW+sUZS6MOMBtjiEIWL28R4M8yB2jsQH6xrV7IubI/yAYu4zs2ddr6+DWkBw/9yJ4AsCF32GIVW50n2cIRzEs3KB4pExw0lKD0alkjPxYD9Nc+fdJXh0iNZHCyex6DdHyWgTOfteVT1amRM6rASwtQdI2xd8OZVxxFMu/gRDexyaqGaMal6SXBJwo9aLRx33xJ8hjxHbHyKkslFgmQs1TNWK4nK5/8RbcB5LXn0mDiamlaLkc1jbX9Gif1WqOoF1YUAloXKXdQ38ffEXWrc/+5kPEVbCuE60rMhZaycGcMKtHcdnpjVRdRPf/I9TSfXi9m2a2BRaU2LyMdozqdHsB3CnDEkQLjR0c/WvVfTQ5U22OqnjYDS5pCxanQl3l96E+kkmJYw5+h1WLcURnkbW84BgUl/uCY/0aw/QItXFl4xB9BlX18d/XF5LUogRsfbl0YNUh7U1B2H6D8QFxI8rkZ+p6i0QQaANxtlmryg56EK1Bl9QntnUn0FA0JK6FmhsVbz6C04v3PLTNb3Tp8QqHaqjK10vY3crMvKoD9aJrhU/kuVTjLkck9XRh3MVS1T/Uw5TJAUBaBZfwWUY++hw8v7zcZCg76J/T01VxOq6ytHnP189/V7ht8QqpY9oKaPDMLwT1CcBPQpH2deE=
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:99

So far so good since yesterday. Let's see if it continues smooth throughout
the day now that I've wrote this :)

Mike Pennington
Network Engineer
Connecticut Education Network (CEN)
860-622-4566

www.cen.ct.gov
________________________________________
From:


[]
on behalf of Pennington, Mike
[]
Sent: Thursday, December 14, 2017 2:49 PM
To: Garnizov, Ivan (RRZE); Mark Feit;

Subject: RE: [perfsonar-user] Best way to troubleshoot HTTP/HTTPS flapping?

After some reseating, firmware upgrades, BIOS upgrade, etc…the diag came out
100% clean, no reported issues. So we’ll see how it goes the next 24 hours.

Mike Pennington
Network Engineer
Connecticut Education Network (CEN)
860-622-4566

www.cen.ct.gov<http://www.cen.ct.gov>

From:


[mailto:]
On Behalf Of Pennington, Mike
Sent: Thursday, December 14, 2017 10:13 AM
To: Garnizov, Ivan (RRZE)
<>;
Mark Feit
<>;


Subject: RE: [perfsonar-user] Best way to troubleshoot HTTP/HTTPS flapping?

Here ya go –

Installed Packages
python27-mod_wsgi.x86_64 3.4-12.sc1.el6
@Internet2

As Mark said, we think it’s a hardware issue. No Dell support, so going to
try a few things I found via Google. Lots of firmware updates and then we’ll
see what happens. I’ll update the group after I do some more troubleshooting.

Thanks Ivan/Mark!

Mike Pennington
Network Engineer
Connecticut Education Network (CEN)
860-622-4566
<mailto:>
www.cen.ct.gov<http://www.cen.ct.gov>

From: Garnizov, Ivan (RRZE)
[mailto:]
Sent: Thursday, December 14, 2017 3:50 AM
To: Pennington, Mike
<<mailto:>>;
Mark Feit
<<mailto:>>;

<mailto:>
Subject: RE: [perfsonar-user] Best way to troubleshoot HTTP/HTTPS flapping?

Hello Mike,

Actually your httpd logs are full with errors.
Dec 10 04:29:14 2017] [error] [client 64.251.58.166] mod_wsgi (pid=17014):
Exception occurred processing WSGI script
'/var/www/pscheduler-server/pscheduler-api-server.wsgi'.
Dec 10 04:29:14 2017] [error] [client 64.251.58.166] IOError: failed to write
data
Dec 10 04:32:03 2017] [error] [client 2001:400:2001:1158::3] Premature end of
script headers: pscheduler-api-server.wsgi
Dec 10 04:32:03 2017] [error] [client 127.0.0.1] Premature end of script
headers: pscheduler-api-server.wsgi
Dec 10 04:37:36 2017] [error] [client 2607:f460:a001:9::2] Premature end of
script headers: pscheduler-api-server.wsgi
Dec 10 04:37:36 2017] [error] [client 2607:f460:a001:9::2] Premature end of
script headers: pscheduler-api-server.wsgi
Dec 10 04:45:57 2017] [error] [client 2607:f460:a001:9::2] Premature end of
script headers: pscheduler-api-server.wsgi
Dec 10 04:45:57 2017] [error] [client 2607:f7b8:eeee:eeee:260:ddff:fe44:a2e6]
Premature end of script headers: pscheduler-api-server.wsgi

Please share the version of the python27-mod_wsgi package.

yum list python27-mod_wsgi

Regards,
Ivan Garnizov

GEANT SA1T2: pS deployments GN Operations
GEANT SA2T3: pS development team
GEANT SA3T5: eduPERT team

From:
<mailto:>

[mailto:]
On Behalf Of Pennington, Mike
Sent: Mittwoch, 13. Dezember 2017 16:45
To: Mark Feit;
<mailto:>
Subject: RE: [perfsonar-user] Best way to troubleshoot HTTP/HTTPS flapping?

Here are the results so far, it’s been stuck on NEAR-TERM SCHEDULE for a
while now. I can jump on video pretty much any time today if you are so game.

pscheduler diags
pScheduler System Diagnostic Dump

Generated for root @ ps-10G-dual-core-hartford.cen.ct.gov
Wed Dec 13 10:34:18 EST 2017


-------------------------------------------------------------------------------
INSTALLED TESTS
-------------------------------------------------------------------------------

[
{
"description": "Measure the clock difference between hosts",
"href": "https://localhost/pscheduler/tests/clock";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "clock",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
},
{
"description": "Measure DNS transaction time",
"href": "https://localhost/pscheduler/tests/dns";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "dns",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
},
{
"description": "Consume time in the background",
"href": "https://localhost/pscheduler/tests/idle";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "idle",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
},
{
"description": "Consume time in the background - NOT FOR PRODUCTION",
"href": "https://localhost/pscheduler/tests/idlebgm";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "idlebgm",
"scheduling-class": "background-multi",
"schema": 1,
"version": "1.0"
},
{
"description": "Consume time exclusively - NOT FOR PRODUCTION",
"href": "https://localhost/pscheduler/tests/idleex";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "idleex",
"scheduling-class": "exclusive",
"schema": 1,
"version": "1.0"
},
{
"description": "Measure network latency between hosts",
"href": "https://localhost/pscheduler/tests/latency";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "latency",
"scheduling-class": "normal",
"schema": 1,
"version": "1.0"
},
{
"description": "Run one-way latency tests in the background",
"href": "https://localhost/pscheduler/tests/latencybg";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "latencybg",
"scheduling-class": "background-multi",
"schema": 1,
"version": "1.0"
},
{
"description": "Measure the round trip time between hosts",
"href": "https://localhost/pscheduler/tests/rtt";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "rtt",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
},
{
"description": "Test communication between two hosts using TCP",
"href": "https://localhost/pscheduler/tests/simplestream";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "simplestream",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
},
{
"description": "Measure network throughput between hosts",
"href": "https://localhost/pscheduler/tests/throughput";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "throughput",
"scheduling-class": "exclusive",
"schema": 1,
"version": "1.0"
},
{
"description": "Trace the path between IP hosts",
"href": "https://localhost/pscheduler/tests/trace";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "trace",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
}
]


-------------------------------------------------------------------------------
INSTALLED TOOLS
-------------------------------------------------------------------------------

[
{
"description": "Measure the clock difference between hosts",
"href": "https://localhost/pscheduler/tests/clock";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "clock",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
},
{
"description": "Measure DNS transaction time",
"href": "https://localhost/pscheduler/tests/dns";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "dns",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
},
{
"description": "Consume time in the background",
"href": "https://localhost/pscheduler/tests/idle";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "idle",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
},
{
"description": "Consume time in the background - NOT FOR PRODUCTION",
"href": "https://localhost/pscheduler/tests/idlebgm";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "idlebgm",
"scheduling-class": "background-multi",
"schema": 1,
"version": "1.0"
},
{
"description": "Consume time exclusively - NOT FOR PRODUCTION",
"href": "https://localhost/pscheduler/tests/idleex";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "idleex",
"scheduling-class": "exclusive",
"schema": 1,
"version": "1.0"
},
{
"description": "Measure network latency between hosts",
"href": "https://localhost/pscheduler/tests/latency";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "latency",
"scheduling-class": "normal",
"schema": 1,
"version": "1.0"
},
{
"description": "Run one-way latency tests in the background",
"href": "https://localhost/pscheduler/tests/latencybg";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "latencybg",
"scheduling-class": "background-multi",
"schema": 1,
"version": "1.0"
},
{
"description": "Measure the round trip time between hosts",
"href": "https://localhost/pscheduler/tests/rtt";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "rtt",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
},
{
"description": "Test communication between two hosts using TCP",
"href": "https://localhost/pscheduler/tests/simplestream";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "simplestream",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
},
{
"description": "Measure network throughput between hosts",
"href": "https://localhost/pscheduler/tests/throughput";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "throughput",
"scheduling-class": "exclusive",
"schema": 1,
"version": "1.0"
},
{
"description": "Trace the path between IP hosts",
"href": "https://localhost/pscheduler/tests/trace";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "trace",
"scheduling-class": "background",
"schema": 1,
"version": "1.0"
}
]


-------------------------------------------------------------------------------
TOOL CONFIGURATIONS
-------------------------------------------------------------------------------

nuttcp.conf:

###
# Configuration file for pScheduler nuttcp tool throughput tests
#
[nuttcp]
##
# Path to the nuttcp command. Default is /usr/bin/nuttcp
#
#nuttcp_cmd = /usr/bin/nuttcp

##
# The control port where nuttcp server will listen
# when it is the receiving endpoint. Default is 5001
#
#server_port = 5001

##
# The first data port where nuttcp server will listen
# when it is the receiving endpoint. Parallel tests will
# use one additional port above this per parallelization.
# Default is 5101
#
#data_port_start = 5101


iperf2.conf:

###
# Configuration file for pScheduler iperf2 tool throughput tests
#
[iperf2]
##
# Path to the iperf2 command. Default is /usr/bin/iperf
#
# iperf2_cmd = /usr/bin/iperf

##
# The port where iperf2 server will listen when this hos
# is the receiving endpoint. Default is 5001
#
#server_port = 5001


iperf3.conf:

###
# Configuration file for pScheduler iperf3 tool throughput tests
#
[iperf3]
##
# Path to the iperf3 command. Default is /usr/bin/iperf3
#
# iperf3_cmd = /usr/bin/iperf3

##
# The port where iperf3 server will listen when this hos
# is the receiving endpoint. Default is 5201
#
#server_port = 5201


owping.conf:

###
# Configuration file for pScheduler latency tests
#
[owping]
##
# Path to the owping command. Default is /usr/bin/owping
#
# owping_cmd = /usr/bin/owping

##
# Indicates that this server does not run an owamp-server so if a request
comes in where
# it is the server role, then an error should be thrown
#
# disable_server = 0

##
# The port(s) where owamp-server is listening. The default is 861. It can be
a single number
# or a range in the form of "N-M". The latter would be a case where you have
multiple
# owamp-servers running
#
#control_ports = 861


powstream.conf:

###
# Configuration file for pScheduler latency tests
#
[powstream]
##
# Path to the powstream command. Default is /usr/bin/powstream
#
# powstream_cmd = /usr/bin/powstream

##
# Path to the owstats command used to read result files. Default is
/usr/bin/owstats
#
# owstats_cmd = /usr/bin/owstats

##
# Path to the pkill command used to clean-up old tests
#
# pkill_cmd = /usr/bin/pkill

##
# Indicates the log level that powstream should use when it runs.
# Valid values: NONE, FATAL, WARN, INFO, DEBUG, ALL
#
# log_level = INFO

##
# Indicates the data directory that powstream use to store result files that
will be read
# by the pscheduler tool to publish result. Default is
/var/lib/pscheduler/tool/powstream
#
# data_dir = /var/lib/pscheduler/tool/powstream

##
# Indicates that data files should not be removed after powstream runs is set
to 1.
# Otherwise they are removed as soon as pScheduler processes them. The
default behavior is
# to remove them. If they are not removed they can use considerable disk
space over time.
# This should only be used for advanced application that want the ability to
access the raw
# files on disk.
#
# keep_data_files = 0

##
# The port(s) where owamp-server is listening. The default is 861. It can be
a single number
# or a range in the form of "N-M". The latter would be a case where you have
multiple
# owamp-servers running
#
#control_ports = 861




-------------------------------------------------------------------------------
INSTALLED ARCHIVERS
-------------------------------------------------------------------------------

[
{
"description": "Discard results - NOT FOR PRODUCTION",
"href": "https://localhost/pscheduler/archivers/bitbucket";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "bitbucket",
"schema": 1,
"version": "1.0"
},
{
"description": "Send results to Esmond",
"href": "https://localhost/pscheduler/archivers/esmond";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "esmond",
"schema": 1,
"version": "1.0"
},
{
"description": "Dummy archiver that some of the time - NOT FOR
PRODUCTION",
"href": "https://localhost/pscheduler/archivers/failer";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "failer",
"schema": 1,
"version": "1.0"
},
{
"description": "Send a raw JSON result to a HTTP server",
"href": "https://localhost/pscheduler/archivers/http";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "http",
"schema": 1,
"version": "1.0"
},
{
"description": "Send a JSON result to RabbitMQ",
"href": "https://localhost/pscheduler/archivers/rabbitmq";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "rabbitmq",
"schema": 1,
"version": "1.0"
},
{
"description": "Send a raw JSON result to Syslog",
"href": "https://localhost/pscheduler/archivers/syslog";,
"maintainer": {
"email":
"<mailto:>",
"href": "http://www.perfsonar.net";,
"name": "perfSONAR Development Team"
},
"name": "syslog",
"schema": 1,
"version": "1.0"
}
]


-------------------------------------------------------------------------------
CLOCK STATE
-------------------------------------------------------------------------------

{
"offset": 4.6014785766601562e-05,
"reference": "secondary reference (2) from 198.124.252.90",
"source": "ntp",
"synchronized": true,
"time": "2017-12-13T10:34:22.217827-05:00"
}


-------------------------------------------------------------------------------
NEAR-TERM SCHEDULE
-------------------------------------------------------------------------------


Mike Pennington
Network Engineer
Connecticut Education Network (CEN)
860-622-4566
<mailto:>
www.cen.ct.gov<http://www.cen.ct.gov>

From: Mark Feit
[mailto:]
Sent: Wednesday, December 13, 2017 7:37 AM
To: Pennington, Mike
<<mailto:>>;

<mailto:>
Subject: Re: [perfsonar-user] Best way to troubleshoot HTTP/HTTPS flapping?

Pennington, Mike writes:

One of our Perfsonar nodes has been acting funky over the past week or so.
Our monitoring shows HTTP/HTTPS going up and down at random times during the
day, I’ve confirmed it is actually going down as I cannot access
perfsonar-hartford.cen.ct.gov during those times. It can be anywhere from a
few minutes and has lasted up to 30 minutes. I’ve rebooted multiple times,
power cycled it, yum is all up to date, etc. Tests and results seem fine, it
just seems to be HTTP/HTTPS access. I’ve looked through some of the httpd
logs but nothing is jumping out at me.

Based on what I can see on your system from the outside, Apache is up and
pScheduler is up and answering queries that don’t have anything to do with
the schedule. The latter leads me to suspect that something in the database
is jammed up, and I’d like to get a handle on what that is.

If you’d be so kind as to run “pscheduler diags” as root and send me the
results off-list, I’ll have an initial look at what’s going on. Alternately,
if you want to jump on video sometime today, drop me a line and we can
troubleshoot it live.

--Mark




Archive powered by MHonArc 2.6.19.

Top of Page