perfSONAR User Q&A and Other Discussion

["Uhl, George D. (GSFC-423.0)[SGT INC]" <>]

Hi Mark,

 

I’m still unable to get the single-ended A-to-B reverse throughput test results to appear in my archives as a B-to-A forward throughput test after including your script in my psconfig template.  There isn’t enough information in /var/log/pscheduler.log to shine a light on what’s happening.  I did try running this with pscheduler on the CLI in debug and archive mode but I don’t see the transformation taking place.  I tried it again but instead of archiving to Esmond, I used syslog and it appears in this case that the transform script might have worked.  Is there a way to track the Esmond archive processing in more detail? 

 

Sep 12 18:03:01 esdis-ps TRANSFORM: {"task": {"test": {"type": "throughput", "spec": {"dest": ",<host A>", "source": "<host B>", "ip-version": 4, "mss": 1448, "schema": 2}}, "tool": "iperf3", "href": "https://esdis-ps/pscheduler/tasks/3a315705-0583-4cbb-ba22-d90dd5c24bd7", "detail": {"exclusive": true, "runs": 1, "added": "2019-09-12T14:02:40-04", "participant": 0, "cli": ["--mss", "1448", "--source", "<host A>", "--ip-version", "4", "--dest", "<host B>", "--single-ended", "--reverse"]

 

 

Thanks,

George

 

From: Mark Feit <>
Date: Tuesday, September 10, 2019 at 12:02 PM
To: "George.D.Uhl" <>, "" <>
Subject: Re: [perfsonar-user] [non-nasa source] [EXTERNAL] FW: Defining single-ended throughput tests inside a psconfig template

 

Uhl, George D. (GSFC-423.0)[SGT INC] writes:

 

I’m having difficulty in developing a jq transform script that will swap the source/dest address pairs as a result of a single-ended throughput test in the reverse direction.  I can write a jq script that operates on the single-ended perfsonar node’s json generated by the psconfig-pscheduler-agent.   For example,

 

"transform" : {

     "script" : "if (.tasks[].test.type=\"throughput\" and .tasks[].test.spec.reverse and .tasks[].test.spec.\"single-ended\") then .tasks[].test.spec.source as $tmp | .tasks[].test.spec.source = .tasks[].test.spec.dest | .tasks[].test.spec.dest = $tmp else null end"

},

 

But this transform script only swaps the source/dest addresses within the test specification (as verified through jq.play).  Having a transform script that operates on test spec parameters contained inside an archive.data object doesn’t make sense since archiving is a post-test operation.   I can’t find anything in the archiver object reference that allows me to swap the source/dest address pair post-test.

 

If I understand correctly, you’d like to make an A-to-B reverse throughput test appear in your archives as a B-to-A forward throughput test.  This is something you can do when you archive the runs.

 

Archiving happens in two stages:  first, the result is transformed if requested (or just left alone if not) and then whatever comes out of that process is sent to the archiver plugin.  The archiver plugins don’t know that transforms exist; they just operate on what pScheduler hands them.  (There’s a design reason for this that I won’t go into here.)  The Esmond plugin is a bit of an odd duck in that it has expectations about what the input will look like, so there’s a little less latitude in what you can change than there is with, say, syslog.  But what you’re doing is just fine since it would still result in something the Esmond plugin will understand.

 

I set up a quick test on a couple of my development systems and came up with this variation on what you provided as an archive specification:

 

{  

   "transform" : {

        "script" : [

            "if (.test.type == \"throughput\")",

            "   and (.test.spec.reverse)",

            "   and (.test.spec.\"single-ended\")",

            "then",

            "    .task.test.spec.source as $tmp",

            "    | .task.test.spec.source = .task.test.spec.dest",

            "    | .task.test.spec.dest = $tmp",

            "    | del(.task.test.spec.reverse)",

            "    | del(.task.test.spec.\"single-ended\")",

            "else",

            "    .",

            "end"

        ]

    },

    "archiver": "esmond",

    "data": { ... Esmond plugin parameters ... }

}

 

A few things of note:

 

The script has been split into an array of strings.  This feature became available in 4.1 and makes them easier to read and edit.

After the source and destination are swapped, the reverse and single-ended parameters are removed to avoid confusion about which way the test actually ran.

 

I corrected a mistake in your original jq script, which used = instead of == for the equality test.  That would have ended up behaving as an assignment and always evaluated to true, which is probably not what you wanted.

 

Instead of the else clause being a null (which would result in the result being dropped entirely), I let it pass the original JSON through to be archived.

 

 

I’m a bit confused about which data the transform script can modify with respect to swapping source/dest addressing when the archiver writes data to Esmond.

 

The transform is handed everything pScheduler knows about the task, the run and the result and can read or alter any of it.  (I notice that we didn’t spell out exactly what’s provided in the documentation; that will be remedied in the next bugfix release.)

 

It does not see any of the data object passed to the archiver plugin, so you won’t see any reference to transformations in those.

 

HTH.

 

--Mark