perfSONAR User Q&A and Other Discussion

["Uhl, George D. (GSFC-423.0)[SGT INC]" <>]

Hi Mark,

 

I’m having difficulty in developing a jq transform script that will swap the source/dest address pairs as a result of a single-ended throughput test in the reverse direction.  I can write a jq script that operates on the single-ended perfsonar node’s json generated by the psconfig-pscheduler-agent.   For example,

 

"transform" : {

     "script" : "if (.tasks[].test.type=\"throughput\" and .tasks[].test.spec.reverse and .tasks[].test.spec.\"single-ended\") then .tasks[].test.spec.source as $tmp | .tasks[].test.spec.source = .tasks[].test.spec.dest | .tasks[].test.spec.dest = $tmp else null end"

},

 

But this transform script only swaps the source/dest addresses within the test specification (as verified through jq.play).  Having a transform script that operates on test spec parameters contained inside an archive.data object doesn’t make sense since archiving is a post-test operation.   I can’t find anything in the archiver object reference that allows me to swap the source/dest address pair post-test.  All I found was the following Esmond reference.

The esmond archiver submits measurement results to the esmond time series database using specialized translations of results for throughput, latency, trace and rtt tests into a format used by earlier versions of perfSONAR. If it does not recognize a test it will store the raw JSON of the pscheduler result in the pscheduler-raw event type.

 

I’m a bit confused about which data the transform script can modify with respect to swapping source/dest addressing when the archiver writes data to Esmond.  From the archiver reference all I see is the following:

url - The URL for the esmond server which will collect the result.

_auth-token - Optional. The authorization token to be used when submitting the result. Note that the _ prefix indicates that this value is considered a secret and will not be supplied if the task specification is retrieved from pScheduler via its REST API. If not specified, IP authentication is assumed.

measurement-agent - Optional. The name of the pScheduler host that produced the result. If not specified, defaults to the endpoint pscheduler deemed the lead.

retry-policy - Optional. Describes how to retry failed attempts to submit the measurement to esmond before giving up. The default behavior is to try once and then give up.

data-formatting-policy - Optional. Indicates how the record should be stored. Valid values are:

·         prefer-mapped - This is the default. It means that if test is type throughput, latency, trace and rtt than store using the traditional metadata and event types. If it does not recognize the result it will store as a pscheduler-raw record.

·         mapped-and-raw - Store both a mapped type and a raw record. Will not store either if not a recognized type that can be mapped.

·         mapped-only - Only store a mapped type and do not store anything if it is not a known type

·         raw-only - Only store a pscheduler-raw record regardless of test type.

summaries - Optional. A list of objects containing an event-type, summary-type and summary-window. If not specified, defaults to a standard set of summaries used by perfSONAR. See the esmond documentation for more details on summaries.

verify-ssl - Optional. Defaults to false. If enabled, check SSL certificate of esmond server against list of known certificate authorities (CAs). See the requests documentation for more details on environment variables and other options for specifying path to CA store.

 

Help!

-George

From: Mark Feit <>
Date: Wednesday, August 14, 2019 at 3:58 PM
To: "George.D.Uhl" <>, "" <>
Subject: Re: [perfsonar-user] [non-nasa source] [EXTERNAL] FW: Defining single-ended throughput tests inside a psconfig template

 

"Uhl, George D. (GSFC-423.0)[SGT INC]" writes:

 

I’m remembering what Mark Feit said a few months ago:

 

The JSON that goes into a pSConfig test template is the same JSON that the test spec that “pscheduler task --export throughput …” would produce.  Any parameter available for that test (or any other) is available for pSConfig, too.  The throughput test has a “reverse” option (see “pscheduler task throughput --help”) that can be put into a template:

 

{

    "dest": "{% address[1] %}",

    "reverse": true

}

 

This answers the question about defining single-ended tests in a psconfig template.  However will the results of a single-ended test be extractable from Esmond via a maddash dashboard or would I need to use a different Nagios check than the “check_throughput.pl”?

 

The result format for each test is standardized, so single- and dual-ended will be indistinguishable.

 

As I recall, the reverse option in a psconfig defined throughput test did not produce reverse direction throughput results from “check_throughput.pl” because the source and destination test nodes as defined in the test remain unchanged in the reverse test.

 

That’s correct.  The result format doesn’t repeat the test parameters and the Esmond archiver maps the source and destination as-is.

 

There are a couple of ways around it.  One is to give the Esmond archiver a switch that makes it swap the source and destination when the parameters say it was a reverse test.  (Making this a default behavior would violate the principle of least astonishment.)  The other would be to put a jq transform into your archive spec for throughput tests that does the equivalent.  The latter would be a good test bed for this since it can be done with a config change rather than us having to release code before we’re sure it’s a good thing.

 

--Mark