perfSONAR User Q&A and Other Discussion

["Uhl, George D. (GSFC-423.0)[SGT INC]" <>]

Thanks Andy, that worked.

-George

From: Andrew Lake <>
Date: Friday, September 28, 2018 at 3:01 PM
To: George Uhl <>
Cc: perfsonar-user <>
Subject: Re: [perfsonar-user] perfSONAR 4.1 is now available

Hi George,

It looks like it is complaining that your old mesh file does not have "write_url" defined in the <measurement_archive> blocks. If you add those in there (you should just be able to copy the read_url) I think you will be in business. According to the old MeshConfig documentation that was a required field, obviously it was not doing a great job of enforcing that rule though as you were bale to get away with it up until now. 

Thanks,
Andy

On Fri, Sep 28, 2018 at 11:05 AM Uhl, George D. (GSFC-423.0)[SGT INC] <> wrote:

Hi Andy,

I’m getting the errors show below for both json files that I use for pS.  Just to clarify things a little more, the latency/throughput tests are failing are from my 4.1.1 server to remote sites although reverse tests are working.  The exception to this is for tests with no-agent hosts – tests there are failing in both directions.  Traceroutes are “mostly” working but some have gone inexplicably missing. My dashboard is also reflecting strange behavior but then it’s the old maddash server I built several years ago.  I’m in the process of migrating to a new archive/dashboard server based on 4.1.1 so I’m hoping that will clear up any dashboard abnormalities.

Thanks,

George

# psconfig translate https://archive.eos.nasa.gov/ps-mesh/international.json

Error retrieving configuration. Encountered the following error:

   No task objects returned.

MeshConfig JSON translator encountered the following errors:

MeshConfig JSON is not valid. Encountered the following validation errors:

   Node: /measurement_archives/0/write_url

   Error: Missing property.

   Node: /measurement_archives/1/write_url

   Error: Missing property.

   Node: /measurement_archives/2/write_url

   Error: Missing property.

# psconfig translate https://archive.eos.nasa.gov/ps-mesh/eosdis.json

Error retrieving configuration. Encountered the following error:

   No task objects returned.

MeshConfig JSON translator encountered the following errors:

MeshConfig JSON is not valid. Encountered the following validation errors:

   Node: /measurement_archives/0/write_url

   Error: Missing property.

   Node: /measurement_archives/1/write_url

   Error: Missing property.

   Node: /measurement_archives/2/write_url

   Error: Missing property.

From: Andrew Lake <>
Date: Friday, September 28, 2018 at 9:48 AM
To: George Uhl <>
Cc: perfsonar-user <>
Subject: Re: [perfsonar-user] perfSONAR 4.1 is now available

Hi George,

If you run "psconfig translate https://archive.eos.nasa.gov/ps-mesh/international.json" do you get any errors back? or does it return some JSON? Its possible there is some item in the configuration it doesn't like. 

Thanks,
Andy

On Thu, Sep 27, 2018 at 2:36 PM Uhl, George D. (GSFC-423.0)[SGT INC] <> wrote:

Hi Andy,

I actually have tests missing.  When I run “psconfig remote list”, I get:

# psconfig remote list

=== pScheduler Agent ===

[

   {

      "url" : "https://archive.eos.nasa.gov/ps-mesh/eosdis.json"

   },

   {

      "url" : "https://archive.eos.nasa.gov/ps-mesh/international.json"

   }

]

These URL’s point to the correct json files that were generated in the old mesh configuration-file-to-json conversion using the old /usr/lib/perfsonar/bin/build_json script.  If I understand psconfig correctly, it can read the old json files and generate pscheduler tasks but this is no longer happening in my case since the upgrade to 4.1.1.   Since the upgrade, /var/log/perfsonar/psconfig-pscheduler-agent.log is reporting:

2018/08/19 05:29:38 ERROR pid=192237 prog=perfSONAR_PS::PSConfig::BaseAgent::_process_psconfig line=371 guid=D420178C-A370-11E8-8BCA-95FD32D31611 config_url=psconfig-pscheduler-agent.log config_s

rc=remote msg=Error loading psconfig: No task objects returned.

2018/08/19 05:29:38 ERROR pid=192237 prog=perfSONAR_PS::PSConfig::BaseAgent::_process_psconfig line=371 guid=D420178C-A370-11E8-8BCA-95FD32D31611 config_url=https://archive.eos.nasa.gov/ps-mesh/international.json c

_onfig_src_=remote msg=Error loading psconfig: No task objects returned.

Thanks,

George

From: Andrew Lake <>
Date: Thursday, September 27, 2018 at 1:53 PM
To: George Uhl <>
Cc: perfsonar-user <>
Subject: Re: [perfsonar-user] perfSONAR 4.1 is now available

Hi George,

The issue originally reported in this thread was just that tests not created by the Toolkit GUI were not showing up any longer in the Toolkit GUI, but they were still in place and running. This is a known change in behavior. Is this the issue you are seeing or is it that you have actual tests missing?

Also, there should not be anything in pscheduler.d unless you are using the toolkit GUI. If you want to make sure it is still pointing at your meshes you can run the command "psconfig remote list". Also, this is a good resource to get you started in troubleshooting the issue: http://docs.perfsonar.net/psconfig_pscheduler_agent.html#troubleshooting

Thanks,
Andy

On Thu, Sep 27, 2018 at 11:20 AM Uhl, George D. (GSFC-423.0)[SGT INC] <> wrote:

Hi,

I’ve had similar issues as Andreas with a perfsonar testpoint node (not a toolkit) that was automagically upgraded from pS 4.0 to 4.1 on Sunday, Aug 19 .  The effect was that all latency and throughput tasks ceased to/from the impacted server after the migration.  I’ve cleared my plate of other tasks and I now have time to focus on this issue. I’ve included the migration log file and the current state of psconfig.  The directory pscheduler.d is currently empty.  Any ideas?

Thanks,

George

# cat /var/log/perfsonar/psconfig-pscheduler-migrate.log

[Sun Aug 19 05:29:21 UTC 2018] Migration started

/etc/perfsonar/meshconfig-agent.conf exists, running 'psconfig pscheduler-translate --save-requesting-agent -o /tmp/tmp.fTVEmN76K7' /etc/perfsonar/meshconfig-agent.conf

Translation successful. Result saved in file /tmp/tmp.fTVEmN76K7.

/etc/perfsonar/psconfig/pscheduler-agent.json exists, backing-up to /etc/perfsonar/psconfig/pscheduler-agent.json.Sun_Aug_19_05_29_21_UTC_2018

Moving to translated file to /etc/perfsonar/psconfig/pscheduler-agent.json

Backing up /etc/perfsonar/meshconfig-agent.conf to /etc/perfsonar/meshconfig-agent.conf.Sun_Aug_19_05_29_21_UTC_2018

/etc/perfsonar/meshconfig-agent-tasks.conf exists, running 'psconfig translate --save-global-archives -o /tmp/tmp.pLznnErm8o /etc/perfsonar/meshconfig-agent-tasks.conf'

Error retrieving configuration. Encountered the following error:

   malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "<test>\n    added_by...") at /usr/share/perl5/vendor_perl/JSON.pm line 171.

meshconfig-agent-tasks.conf translator encountered the following errors:

Nothing to convert. This is not an error if all tests contain added_by_mesh. Ignore any errors above about malformed JSON string.

psconfig translate did not work, but it's probably not an error. It will only translate if there are tasks from the webui or created manually. Any global archives were still converted if this was the case.

Copying /etc/perfsonar/meshconfig-agent-tasks.conf to /var/lib/perfsonar/toolkit/gui-tasks.conf

Backing up /etc/perfsonar/meshconfig-agent-tasks.conf to /etc/perfsonar/meshconfig-agent-tasks.conf.Sun_Aug_19_05_29_21_UTC_2018

/var/lib/perfsonar/meshconfig/client_uuid exists, migrating

Moving to /var/lib/perfsonar/meshconfig/client_uuid to /var/lib/perfsonar/psconfig/client_uuid

/var/lib/perfsonar/meshconfig/psc_tracker exists, migrating

Moving to /var/lib/perfsonar/meshconfig/psc_tracker to /var/lib/perfsonar/psconfig/psc_tracker

[Sun Aug 19 05:29:21 UTC 2018] Migration complete

#

# psconfig pscheduler-stats

Agent Last Run Start Time: 2018/09/27 14:00:40

Agent Last Run End Time: 2018/09/27 14:00:41

Agent Last Run Process ID (PID): 89438

Agent Last Run Log GUID: B7EAD438-C25D-11E8-A932-F35A34D31611

Total tasks managed by agent: 0

#

# psconfig pscheduler-tasks

{

   "tasks" : []

}

#

# ls -la psconfig/pscheduler.d

total 4

drwxr-xr-x. 2 perfsonar perfsonar    6 Aug 19 05:29 .

drwxr-xr-x. 5 root      root      4096 Sep 14 06:57 ..

#

 

From: <> on behalf of Antoine Delvaux <>
Date: Thursday, August 16, 2018 at 8:42 AM
To: Andreas Haupt <>, Shawn McKee <>
Cc: perfsonar-user <>
Subject: Re: [perfsonar-user] perfSONAR 4.1 is now available

And here are a few further psconfig command examples that you may find useful:

psconfig pscheduler-stats
psconfig pscheduler-tasks | jq .tasks[].test
psconfig pscheduler-tasks | jq .tasks[].test.type

If you want to know more about JSON filtering with JQ, I invite you to watch Mark's webinar on our youtube channel: https://www.youtube.com/watch?v=FrT6R75M3BE

Thanks,

Antoine.

> Le 16 août 2018 à 12:37, Andrew Lake <> a écrit :
>
> Hi,
>
> This is not a bug, but a known change in behavior of the Toolkit UI. We actually discussed this a few times internally during the development process and have an open issue for a future release on it here: https://github.com/perfsonar/toolkit/issues/310
>
> Basically the Toolkit UI is in a state where it needs some refactoring on the backend as it was very closely tied to the old “meshconfig-agent-tasks.conf” format. We didn't feel we had time for that refactoring this release and a casualty of that decision is that the tests not setup through UI are not displayed on the configuration page. They ARE still displayed in the result listing on the front page assuming you are storing the results in the local measurement archive.
>
> It will be possible to restore the old behavior in the future as the tests setup through the UI are very clearly separated from your tests that come from other pSConfig templates on the system. In the meantime, if you need a detailed listing of all the tasks pSConfig will try to setup, whether they be from those configured through the UI or read rom remote templates, then you can run the command “psconfig pscheduler-tasks”. 
>
> Thanks,
> Andy
>
>
>
> On August 16, 2018 at 8:09:30 AM, Andreas Haupt () wrote:
>
>> Hi Szymon,
>>
>> On Thu, 2018-08-16 at 13:44 +0200, Szymon Trocha wrote:
>> > Hi Andreas,
>> > Thank you for reporting. Is there anything suspicious in /var/log/perfsonar/psconfig-maddash-migrate.log ?
>>
>> Well, yes - but it also tell me, the error is no "real error":
>>
>>
>> [perfson1] /root # cat /var/log/perfsonar/psconfig-pscheduler-migrate.log
>> [Thu Aug 16 05:59:29 CEST 2018] Migration started
>>
>> /etc/perfsonar/meshconfig-agent.conf exists, running 'psconfig pscheduler-translate --save-requesting-agent -o /tmp/tmp.w9AS61N6Ye' /etc/perfsonar/meshconfig-agent.conf
>> Translation successful. Result saved in file /tmp/tmp.w9AS61N6Ye.
>> /etc/perfsonar/psconfig/pscheduler-agent.json exists, backing-up to /etc/perfsonar/psconfig/pscheduler-agent.json.Thu_Aug_16_05_59_29_CEST_2018
>> Moving to translated file to /etc/perfsonar/psconfig/pscheduler-agent.json
>> Backing up /etc/perfsonar/meshconfig-agent.conf to /etc/perfsonar/meshconfig-agent.conf.Thu_Aug_16_05_59_29_CEST_2018
>>
>> /etc/perfsonar/meshconfig-agent-tasks.conf exists, running 'psconfig translate --save-global-archives -o /tmp/tmp.C5490oLOUv /etc/perfsonar/meshconfig-agent-tasks.conf'
>> Error retrieving configuration. Encountered the following error:
>>
>> malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "<test>\n added_by...") at /usr/share/perl5/vendor_perl/JSON.pm line 171.
>>
>>
>> meshconfig-agent-tasks.conf translator encountered the following errors:
>>
>> Nothing to convert. This is not an error if all tests contain added_by_mesh. Ignore any errors above about malformed JSON string.
>> psconfig translate did not work, but it's probably not an error. It will only translate if there are tasks from the webui or created manually. Any global archives were still converted if this was the case.
>> Copying /etc/perfsonar/meshconfig-agent-tasks.conf to /var/lib/perfsonar/toolkit/gui-tasks.conf
>> Backing up /etc/perfsonar/meshconfig-agent-tasks.conf to /etc/perfsonar/meshconfig-agent-tasks.conf.Thu_Aug_16_05_59_29_CEST_2018
>>
>> /var/lib/perfsonar/meshconfig/client_uuid exists, migrating
>> Moving to /var/lib/perfsonar/meshconfig/client_uuid to /var/lib/perfsonar/psconfig/client_uuid
>>
>> /var/lib/perfsonar/meshconfig/psc_tracker exists, migrating
>> Moving to /var/lib/perfsonar/meshconfig/psc_tracker to /var/lib/perfsonar/psconfig/psc_tracker
>>
>> [Thu Aug 16 05:59:30 CEST 2018] Migration complete
>> [perfson1] /root # 
>>
>>
>> Cheers,
>> Andreas
>> -- 
>> | Andreas Haupt | E-Mail:
>> | DESY Zeuthen | WWW:  http://www-zeuthen.desy.de/~ahaupt
>> | Platanenallee 6 | Phone: +49/33762/7-7359
>> | D-15738 Zeuthen | Fax: +49/33762/7-7216

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user