perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

(Sorry, this ran very long, but I hope it adequately answers your questions 
and addresses your concerns.)

Brian Candler writes:

    In the pf3.x model, you could have a "full fat" perfsonar node 
    (including scheduler, archiver) which periodically ran tests to a 
    stateless remote node (bwctl-server, owamp-server, iperf3).  The remote 
    node didn't store anything, and never initiated tests on its own behalf.

Any self-scheduling behavior you see is not part of pScheduler.  You can 
install just pScheduler and it will happily sit and do absolutely nothing 
until it’s asked to do something.  The only thing it does autonomously is the 
scheduling and execution of tasks that repeat, and even those orders have to 
come from the outside.


    If the pf4.x model we have pscheduler, which has a SQL database plus 
    pluggable tests and archivers.

4.0 was structured to introduce pScheduler while keeping enough things from 
3.5 that the transition wasn’t a total disruption to existing users.  If 
you’re looking at perfSONAR from the familiar 3.5 perspective, it doesn’t 
look like that much has changed.  There’s considerably more under the hood 
than just plug-ins and there’s a lot more in the pipeline.  The database is 
an implementation detail, not a feature.  Much of what’s new is outlined in 
the talk I gave at TechEx last September:  
http://meetings.internet2.edu/2016-technology-exchange/detail/10004321. 


    Here's what I don't understand. Let's call the node which decides it's 
    time to run a test "A", and the one which receives an incoming 
    connection and responds "B"

pScheduler has nomenclature for those things; let me use that instead:   The 
nodes involved in a test are called “participants.”   Participants are 
numbered from zero, and each has a specific role defined by the test.  For 
example, the throughput test’s default behavior has the sending end as 
participant 0 and the receiving end as participant 1.  (A pleasant side 
effect of pScheduler’s extensible design is that it can support tests that 
have any number of participants.  When someone invents a test that involves 
five participants, all that needs to be done to support it is write the 
plugins.)  Participant 0 is designated the “lead participant” and is given 
additional administrative responsibilities which include scheduling runs with 
the other participants, collecting and merging the results and doing the 
archiving.  Non-lead participants are otherwise passive, answering questions 
about when they can schedule runs of tasks and, of course, carrying out their 
ends of the tests.

    
    Question 1: when "A" runs a test, are the archivers invoked at both A 
    and B ends?  Is there a way to configure B with a null archiver which 
    doesn't store the results?

Archiving is done by the lead participant.  (We have an open request to add a 
per-task knob to determine which participant(s) do it, and that’s being 
considered for a future release.)  You may be seeing behaviors that look like 
everything is being archived on both ends, but that’s a function of what 
meshconfig is asking pScheduler to do.  Again, pScheduler does nothing 
without being asked.

There is a feature that allows the administrator of a node to force 
pScheduler to send every result of every run where it’s the lead to an 
archiver, but nothing that ships with pScheduler or the toolkit enables that 
behavior.  (We wrote that feature so projects like PuNDIT can skim off and 
save trace results without having to include directives in the tasks.  They 
used to require installation of a modified version of perfSONAR for that to 
work.  Now they just install an archiver plugin and one file to make it 
happen.)


    Question 2: assuming "A" uses esmond as archiver, what's the postgres 
    database used for?  Is it just storing the configuration of scheduled 
    tests?  Or is it storing test history/logs, in addition to the results 
    stored in esmond?

pScheduler’s database, which is separate from esmond’s, plays two roles.

The first is as a store for anything that needs to persist, which includes 
the inventory of available tests, tools and archivers, tasks, the schedule of 
each run, the results (for as long as we keep them), queues of results that 
need to be archived and a handful of other administrative items.  Everything 
pScheduler knows will persist in a consistent state across software upgrades, 
system crashes and reboots, which means the single ad hoc test you scheduled 
for next Wednesday at 3:15 will still run as if nothing had happened.

The other is doing almost 100% of the heavy lifting involved in dealing with 
that data.  Almost everything pScheduler does is accomplished by querying the 
database, sometimes in complex ways.  We’re not using it as a fancy 
replacement for flat files or data structures in memory.  The first question 
that gets asked about anything in pScheduler is whether or not the database 
can do the grunt work for us, and the answer is almost always “yes.”

Making the database do something new is a matter of defining the relational 
calculus, which can take hours or days instead of the weeks or months it 
would take to develop, test, debug and maintain hand-rolled code of 
comparable quality and ability.  I’ve done it both ways, and this way results 
in a much-better product in a lot less time.  Keep in mind that pScheduler 
was developed from nothing in 18 months into a much-larger superset of the 
features in a project that evolved over about 15 years.  Some of that time is 
an investment that will pay dividends later in the form of higher reliability 
and less time spent on maintenance and new development.  That was one of many 
factors that influenced our design decisions.

     
    Question 3: is it in principle possible to write a databaseless 
    pscheduler responder daemon which passively acts on test requests like 
    bwctl did? 

With sufficient time and budget, sure.  I wouldn’t recommend it if you intend 
to make it a 100% work-alike with 100% of the features.

If your concern here is PostgreSQL’s weight, spend some time with pScheduler 
as we shipped it before declaring it unsuitable and writing your own.  I’ve 
been using PostgreSQL in my projects for 20 years, 30 if you count its 
previous incarnations, and have contributed code to it.  It’s a very solid 
system maintained by people who know the material cold.  Because of the 
current version’s roots in the 1990s, it does a very good job in keeping its 
use of resources down to a minimum in its out-of-the-box configuration.  It 
can be tuned to run with a smaller footprint (with trade-offs), but our 
experience so far has been that it’s not necessary, even on systems that 
don’t come anywhere close to meeting our recommended minimums.

If your concern is the hassle of installing everything and removing it later, 
Brian Tierney is developing a Docker container that runs the whole test point 
that could very easily be stripped down to just pScheduler.  (There’s a 
pScheduler-only container in the sources under scripts/docker, but I don’t 
recommend it for production.) 


    (I did try looking for documentation on the pscheduler protocol, but 
    didn't find it)

That document hasn’t been released yet but will probably be out with 4.1.  
The short summary is that it’s a REST API with a reasonably-simple data 
model.  There is extensive documentation of the JSON involved is in with the 
source code (see the docs directory) that will be converted from LaTeX into 
something else at some point and added to docs.perfsonar.net.  I can build 
and email you a set of the PDFs if you’d like.


    Question 4: owamp has its own control protocol on well-known port (861). 
    Is a perfsonar 4.x node able to run owamp tests to a remote node which 
    is running owamp-server but *not* running pscheduler or bwctl?

It is, but that’s because the latency test was designed to be 
single-participant (as are rtt and trace).  I’d have preferred that the test 
be two-participant so the OWAMP tool plugin only starts owampd when needed, 
but I didn’t make that decision and there are valid reasons for it to work 
the way it does.

If I can pick a nomenclature nit:  pScheduler doesn’t run OWAMP tests, it 
runs latency tests and can use the OWAMP tool to make the measurement.  This 
is an important distinction, because someone might write a TWAMP tool plugin 
that runs the same tests.


    In the past, I've been able to add bwctl+owamp to an existing server 
    with very low overhead. I used it both as an endpoint for interactive 
    testing (bwctl at command line) and scheduled testing (from a full 
    perfsonar node).  It's a shame if the switch to pscheduler protocol will 
    mean this isn't possible any more.

pScheduler’s design and implementation are a result of what members of the 
community told us they about what they wanted out of perfSONAR and where the 
development team saw needs that might arise in the future.  If we misread the 
community, we need to hear about it at the same volume as we heard about all 
the things that led us to the current design.  I got a lot of positive 
feedback on the feature set after TechEx and very little comment about the 
weight of it after having made it very plain that there’s a RDBMS behind the 
scenes.

If BWCTL suits your needs, we’re not going seek out and destroy every copy of 
the source code we can find.  The license doesn’t prevent anyone from 
continuing to build, maintain, use or distribute it.  The same goes for 
pScheduler’s BWCTL backward-compatibility plugins, although we do have plans 
to remove a couple of things that had to be hacked into pScheduler  that 
allow them to work.  BWCTL wouldn’t be the first project to be abandoned and 
picked up by the segment of the community that needs it, nor will it be the 
last.

The bottom line is that perfSONAR can’t be all things to all people.  We have 
to make decisions and compromises that we think will serve the organizations 
providing our development resources (Internet2, GÉANT, ESNet, Indiana 
University and the newly-signed-on University of Michigan) and as much of the 
rest of the community as we can.  In this case, we made the decision -- and 
not lightly -- that moving on from BWCTL was the best way to get us there.  

--Mark