perfSONAR User Q&A and Other Discussion

[Kathy Benninger <>]

I have a perfSONAR Toolkit v4.1.2-1.el7 server that is not initiating tests. 
The homepage says "No data available in table" and displays no test results. 
The pS does participate in testing initiated by other perfSONARs.

It was built via a clean "Full Install", with ps-migrate backup/restore to 
retain historical test results.

The pS is pointing to a meshconfig file that is used by, and working with, 
other hosts:

[benninge@perfsonar-xsede ~]$  psconfig remote list
=== pScheduler Agent ===
[
   {
      "url" : "https://psarch.psc.xsede.org/xsede_mesh.json";
   }
]

but doesn't seem to know what to do with the file:

[benninge@perfsonar-xsede ~]$ psconfig pscheduler-tasks
{
   "tasks" : []
}

/var/log/perfsonar/psconfig-pscheduler-agent.log is full of the following:

2018/09/27 12:31:18 INFO pid=2590 prog=main:: line=176 
guid=2473202A-C27B-11E8-BA88-1D1AFF374669 msg=Running agent...
2018/09/27 12:31:19 INFO pid=2590 prog=main:: line=178 
guid=2473202A-C27B-11E8-BA88-1D1AFF374669 msg=Agent completed running
2018/09/27 12:31:19 INFO pid=2590 prog=main:: line=189 
guid=2473202A-C27B-11E8-BA88-1D1AFF374669 msg=Time until next record refresh 
is 3599 seconds
2018/09/27 13:31:19 INFO pid=2590 prog=main:: line=176 
guid=86D3FEEE-C283-11E8-BA88-1D1AFF374669 msg=Running agent...
2018/09/27 13:31:20 INFO pid=2590 prog=main:: line=178 
guid=86D3FEEE-C283-11E8-BA88-1D1AFF374669 msg=Agent completed running
2018/09/27 13:31:20 INFO pid=2590 prog=main:: line=189 
guid=86D3FEEE-C283-11E8-BA88-1D1AFF374669 msg=Time until next record refresh 
is 3599 seconds
2018/09/27 14:31:20 INFO pid=2590 prog=main:: line=176 
guid=E935A2CE-C28B-11E8-BA88-1D1AFF374669 msg=Running agent...
2018/09/27 14:31:22 INFO pid=2590 prog=main:: line=178 
guid=E935A2CE-C28B-11E8-BA88-1D1AFF374669 msg=Agent completed running
2018/09/27 14:31:22 INFO pid=2590 prog=main:: line=189 
guid=E935A2CE-C28B-11E8-BA88-1D1AFF374669 msg=Time until next record refresh 
is 3598 seconds
2018/09/27 15:31:22 INFO pid=2590 prog=main:: line=176 
guid=4C0F6D00-C294-11E8-BA88-1D1AFF374669 msg=Running agent...
2018/09/27 15:31:23 INFO pid=2590 prog=main:: line=178 
guid=4C0F6D00-C294-11E8-BA88-1D1AFF374669 msg=Agent completed running
2018/09/27 15:31:23 INFO pid=2590 prog=main:: line=189 
guid=4C0F6D00-C294-11E8-BA88-1D1AFF374669 msg=Time until next record refresh 
is 3599 seconds
2018/09/27 16:31:23 INFO pid=2590 prog=main:: line=176 
guid=AE771FB2-C29C-11E8-BA88-1D1AFF374669 msg=Running agent...
2018/09/27 16:31:25 INFO pid=2590 prog=main:: line=178 
guid=AE771FB2-C29C-11E8-BA88-1D1AFF374669 msg=Agent completed running
2018/09/27 16:31:25 INFO pid=2590 prog=main:: line=189 
guid=AE771FB2-C29C-11E8-BA88-1D1AFF374669 msg=Time until next record refresh 
is 3598 seconds

Thanks for any help!
Kathy


On 9/27/2018 2:36 PM, Uhl, George D. (GSFC-423.0)[SGT INC] wrote:
> Hi Andy,
>
> I actually have tests missing.  When I run “psconfig remote list”, I get:
>
> # psconfig remote list
>
> === pScheduler Agent ===
>
> [
>
>     {
>
>        "url" : "https://archive.eos.nasa.gov/ps-mesh/eosdis.json";
>
>     },
>
>     {
>
>        "url" : "https://archive.eos.nasa.gov/ps-mesh/international.json";
>
>     }
>
> ]
>
>
> These URL’s point to the correct json files that were generated in the old 
> mesh configuration-file-to-json conversionusing the old 
> /usr/lib/perfsonar/bin/build_json script.  If I understand psconfig 
> correctly, it can read the old json files and generate pscheduler tasks but 
> this is no longer happening in my case since the upgrade to 4.1.1.   Since 
> the upgrade, /var/log/perfsonar/psconfig-pscheduler-agent.log is reporting:
>
> 2018/08/19 05:29:38 ERROR pid=192237 
> prog=perfSONAR_PS::PSConfig::BaseAgent::_process_psconfig line=371 
> guid=D420178C-A370-11E8-8BCA-95FD32D31611 
> config_url=psconfig-pscheduler-agent.log config_s
>
> rc=remote msg=Error loading psconfig: No task objects returned.
>
> 2018/08/19 05:29:38 ERROR pid=192237 
> prog=perfSONAR_PS::PSConfig::BaseAgent::_process_psconfig line=371 
> guid=D420178C-A370-11E8-8BCA-95FD32D31611 
> config_url=https://archive.eos.nasa.gov/ps-mesh/international.json c
>
> onfig_src=remote msg=Error loading psconfig: No task objects returned.
>
>
> Thanks,
> George
>
> From: Andrew Lake 
> <
>  
> <mailto:>>
> Date: Thursday, September 27, 2018 at 1:53 PM
> To: George Uhl 
> <
>  
> <mailto:>>
> Cc: perfsonar-user < 
> <mailto:>>
> Subject: Re: [perfsonar-user] perfSONAR 4.1 is now available
>
> Hi George,
>
> The issue originally reported in this thread was just that tests not created 
> by the Toolkit GUI were not showing up any longer in the Toolkit GUI, but 
> they were still in place and running. This is a known change in behavior. Is 
> this the issue you are seeing or is it that you have actual tests missing?
>
> Also, there should not be anything in pscheduler.d unless you are using the 
> toolkit GUI. If you want to make sure it is still pointing at your meshes 
> you can run the command "psconfig remote list". Also, this is a good 
> resource to get you started in troubleshooting the issue: 
> http://docs.perfsonar.net/psconfig_pscheduler_agent.html#troubleshooting
>
> Thanks,
> Andy
>
>
>
>
>
> On Thu, Sep 27, 2018 at 11:20 AM Uhl, George D. (GSFC-423.0)[SGT INC] 
> < <mailto:>> wrote:
>
>     Hi,
>
>     I’ve had similar issues as Andreas with a perfsonar testpoint node (not
>     a toolkit) that was automagically upgraded from pS 4.0 to 4.1 on Sunday,
>     Aug 19 .  The effect was that all latency and throughput tasks ceased
>     to/from the impacted server after the migration.  I’ve cleared my plate
>     of other tasks and I now have time to focus on this issue. I’ve included
>     the migration log file and the current state of psconfig.  The directory
>     pscheduler.d is currently empty.  Any ideas?
>
>     Thanks,
>     George
>
>     # cat /var/log/perfsonar/psconfig-pscheduler-migrate.log
>
>     [Sun Aug 19 05:29:21 UTC 2018] Migration started
>
>
>     /etc/perfsonar/meshconfig-agent.conf exists, running 'psconfig
>     pscheduler-translate --save-requesting-agent -o /tmp/tmp.fTVEmN76K7'
>     /etc/perfsonar/meshconfig-agent.conf
>
>     Translation successful. Result saved in file /tmp/tmp.fTVEmN76K7.
>
>     /etc/perfsonar/psconfig/pscheduler-agent.json exists, backing-up to
>     /etc/perfsonar/psconfig/pscheduler-agent.json.Sun_Aug_19_05_29_21_UTC_2018
>
>     Moving to translated file to /etc/perfsonar/psconfig/pscheduler-agent.json
>
>     Backing up /etc/perfsonar/meshconfig-agent.conf to
>     /etc/perfsonar/meshconfig-agent.conf.Sun_Aug_19_05_29_21_UTC_2018
>
>
>     /etc/perfsonar/meshconfig-agent-tasks.conf exists, running 'psconfig
>     translate --save-global-archives -o /tmp/tmp.pLznnErm8o
>     /etc/perfsonar/meshconfig-agent-tasks.conf'
>
>     Error retrieving configuration. Encountered the following error:
>
>
>         malformed JSON string, neither tag, array, object, number, string or
>     atom, at character offset 0 (before "<test>\n    added_by...") at
>     /usr/share/perl5/vendor_perl/JSON.pm line 171.
>
>
>
>     meshconfig-agent-tasks.conf translator encountered the following errors:
>
>
>     Nothing to convert. This is not an error if all tests contain
>     added_by_mesh. Ignore any errors above about malformed JSON string.
>
>     psconfig translate did not work, but it's probably not an error. It will
>     only translate if there are tasks from the webui or created manually.
>     Any global archives were still converted if this was the case.
>
>     Copying /etc/perfsonar/meshconfig-agent-tasks.conf to
>     /var/lib/perfsonar/toolkit/gui-tasks.conf
>
>     Backing up /etc/perfsonar/meshconfig-agent-tasks.conf to
>     /etc/perfsonar/meshconfig-agent-tasks.conf.Sun_Aug_19_05_29_21_UTC_2018
>
>
>     /var/lib/perfsonar/meshconfig/client_uuid exists, migrating
>
>     Moving to /var/lib/perfsonar/meshconfig/client_uuid to
>     /var/lib/perfsonar/psconfig/client_uuid
>
>
>     /var/lib/perfsonar/meshconfig/psc_tracker exists, migrating
>
>     Moving to /var/lib/perfsonar/meshconfig/psc_tracker to
>     /var/lib/perfsonar/psconfig/psc_tracker
>
>
>     [Sun Aug 19 05:29:21 UTC 2018] Migration complete
>
>     #
>
>
>     # psconfig pscheduler-stats
>
>     Agent Last Run Start Time: 2018/09/27 14:00:40
>
>     Agent Last Run End Time: 2018/09/27 14:00:41
>
>     Agent Last Run Process ID (PID): 89438
>
>     Agent Last Run Log GUID: B7EAD438-C25D-11E8-A932-F35A34D31611
>
>     Total tasks managed by agent: 0
>
>     #
>
>
>     # psconfig pscheduler-tasks
>
>     {
>
>         "tasks" : []
>
>     }
>
>     #
>
>
>
>     # ls -la psconfig/pscheduler.d
>
>     total 4
>
>     drwxr-xr-x. 2 perfsonar perfsonar    6 Aug 19 05:29 .
>
>     drwxr-xr-x. 5 root      root      4096 Sep 14 06:57 ..
>
>     #
>
>
>
>
>     From: 
> <
>     
> <mailto:>>
>  on behalf of Antoine
>     Delvaux 
> <
>     
> <mailto:>>
>     Date: Thursday, August 16, 2018 at 8:42 AM
>     To: Andreas Haupt 
> <
>     
> <mailto:>>,
>  Shawn McKee 
> <
>     
> <mailto:>>
>     Cc: perfsonar-user 
> <
>     
> <mailto:>>
>     Subject: Re: [perfsonar-user] perfSONAR 4.1 is now available
>
>     And here are a few further psconfig command examples that you may find
>     useful:
>
>     psconfig pscheduler-stats
>     psconfig pscheduler-tasks | jq .tasks[].test
>     psconfig pscheduler-tasks | jq .tasks[].test.type
>
>     If you want to know more about JSON filtering with JQ, I invite you to
>     watch Mark's webinar on our youtube channel:
>     https://www.youtube.com/watch?v=FrT6R75M3BE
>
>     Thanks,
>
>     Antoine.
>
>     > Le 16 août 2018 à 12:37, Andrew Lake 
> <
>  
> <mailto:>>
>  a écrit :
>     > 
>     > Hi,
>     > 
>     > This is not a bug, but a known change in behavior of the Toolkit UI. We actually discussed this a few times internally during the development process and have an open issue for a future release on it here:  https://github.com/perfsonar/toolkit/issues/310
>     > 
>     > Basically the Toolkit UI is in a state where it needs some refactoring on the backend as it was very closely tied to the old “meshconfig-agent-tasks.conf” format. We didn't feel we had time for that refactoring this release and a casualty of that decision  is that the tests not setup through UI are not displayed on the
>     configuration page. They ARE still displayed in the result listing on
>     the front page assuming you are storing the results in the local
>     measurement archive.
>     > 
>     > It will be possible to restore the old behavior in the future as the tests setup through the UI are very clearly separated from your tests that come from other pSConfig templates on the system. In the meantime, if you need a detailed listing of all the tasks  pSConfig will try to setup, whether they be from those configured
>     through the UI or read rom remote templates, then you can run the
>     command “psconfig pscheduler-tasks”.
>     > 
>     > Thanks,
>     > Andy
>     > 
>     > 
>     > 
>     > On August 16, 2018 at 8:09:30 AM, Andreas Haupt ( <mailto:>) wrote:
>     > 
>     >> Hi Szymon, 
>     >> 
>     >> On Thu, 2018-08-16 at 13:44 +0200, Szymon Trocha wrote: 
>     >> > Hi Andreas, 
>     >> > Thank you for reporting. Is there anything suspicious in /var/log/perfsonar/psconfig-maddash-migrate.log ?
>     >> 
>     >> Well, yes - but it also tell me, the error is no "real error": 
>     >> 
>     >> 
>     >> [perfson1] /root # cat /var/log/perfsonar/psconfig-pscheduler-migrate.log 
>     >> [Thu Aug 16 05:59:29 CEST 2018] Migration started 
>     >> 
>     >> /etc/perfsonar/meshconfig-agent.conf exists, running 'psconfig pscheduler-translate --save-requesting-agent -o /tmp/tmp.w9AS61N6Ye' /etc/perfsonar/meshconfig-agent.conf
>     >> Translation successful. Result saved in file /tmp/tmp.w9AS61N6Ye. 
>     >> /etc/perfsonar/psconfig/pscheduler-agent.json exists, backing-up to /etc/perfsonar/psconfig/pscheduler-agent.json.Thu_Aug_16_05_59_29_CEST_2018
>     >> Moving to translated file to /etc/perfsonar/psconfig/pscheduler-agent.json 
>     >> Backing up /etc/perfsonar/meshconfig-agent.conf to /etc/perfsonar/meshconfig-agent.conf.Thu_Aug_16_05_59_29_CEST_2018
>     >> 
>     >> /etc/perfsonar/meshconfig-agent-tasks.conf exists, running 'psconfig translate --save-global-archives -o /tmp/tmp.C5490oLOUv /etc/perfsonar/meshconfig-agent-tasks.conf'
>     >> Error retrieving configuration. Encountered the following error: 
>     >> 
>     >> malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "<test>\n added_by...") at /usr/share/perl5/vendor_perl/JSON.pm line 171.
>     >> 
>     >> 
>     >> meshconfig-agent-tasks.conf translator encountered the following errors: 
>     >> 
>     >> Nothing to convert. This is not an error if all tests contain added_by_mesh. Ignore any errors above about malformed JSON string.
>     >> psconfig translate did not work, but it's probably not an error. It 
> will only translate if there are tasks from the webui or created manually. Any 
> global archives were still converted if this was the case.
>     >> Copying /etc/perfsonar/meshconfig-agent-tasks.conf to 
> /var/lib/perfsonar/toolkit/gui-tasks.conf
>     >> Backing up /etc/perfsonar/meshconfig-agent-tasks.conf to 
> /etc/perfsonar/meshconfig-agent-tasks.conf.Thu_Aug_16_05_59_29_CEST_2018
>     >> 
>     >> /var/lib/perfsonar/meshconfig/client_uuid exists, migrating 
>     >> Moving to /var/lib/perfsonar/meshconfig/client_uuid to /var/lib/perfsonar/psconfig/client_uuid
>     >> 
>     >> /var/lib/perfsonar/meshconfig/psc_tracker exists, migrating 
>     >> Moving to /var/lib/perfsonar/meshconfig/psc_tracker to /var/lib/perfsonar/psconfig/psc_tracker
>     >> 
>     >> [Thu Aug 16 05:59:30 CEST 2018] Migration complete 
>     >> [perfson1] /root #  
>     >> 
>     >> 
>     >> Cheers, 
>     >> Andreas 
>     >> --  
>     >> | Andreas Haupt | E-Mail:  <mailto:>
>     >> | DESY Zeuthen | WWW:  http://www-zeuthen.desy.de/~ahaupt 
> <http://www-zeuthen.desy.de/%7Eahaupt>
>     >> | Platanenallee 6 | Phone: +49/33762/7-7359 
>     >> | D-15738 Zeuthen | Fax: +49/33762/7-7216 
>
>     --
>     To unsubscribe from this list:
>     https://lists.internet2.edu/sympa/signoff/perfsonar-user