perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Murilo Vetter writes:

 

Does anyone know how was made the integration of perfSONAR measurements to Grafana?

 

I found something in this document:

https://indico.cern.ch/event/637013/contributions/2739243/attachments/1539101/2412730/HEPiX_WLCG-neworking_fall2017.pdf

 

Shawn can speak directly to what OSG/WLCG is doing, but I can give you some more-general information.

 

Historically, the only method for getting perfSONAR data into other systems has been to send it to Esmond and write programs to pull it from there.

 

One of the things pScheduler brought with it from day one is archiver plugins that can send results as they’re produced.  MeshConfig had been reprogrammed to use the Esmond plugin, as does the out-of-the-box pSConfig template.  However, there are also plugins that will send results to syslog, HTTP(S), RabbitMQ and, if you’re a glutton for punishment, SNMP traps.  A plugin for Kafka is in the pipeline for release with 4.2.0.  We’ve tried to cover as many of the common cases as we could, but it’s easy to write new ones for things we haven’t.

 

All of the plugins receive a blob of JSON that includes the result and as much information about the task and run as possible.  There’s a mechanism for the JSON to be transformed using a jq script before it’s handed to the plugin, so if you need special formatting, that can be done by pScheduler on your perfSONAR nodes and you don’t have to run a separate system to do it.  (Some of this is discussed in a webinar we did on jq  last year:  https://youtu.be/FrT6R75M3BE?t=2678.)

 

I did some throwaway some experimentation last year that sent data into ElasticSearch using the HTTP plugin and pulled it for display using Grafana.  It wasn’t much and the data probably wasn’t very accurate, but it was enough to prove the concept.  This is how it looked:  https://slack-files.com/T073A2Z0D-FEM1QA4LC-04a7cd4665.

 

The development team is in the early stages of looking at ways to improve the visualization component of perfSONAR.  What we have now is mostly home-grown code that requires changing to add new types of tests which, as we found out with MeshConfig, isn’t such a good thing.  Expect to hear more about this at TechEx in December.

 

Hope that’s useful.

 

--Mark