perfSONAR User Q&A and Other Discussion

[Johann Hugo <jhugo@sanren.ac.za>]

Hi all

Since the upgrade a couple of my toolkit servers are not getting their pscheduler tasks any more. Other servers in the mesh are fine. I fixed limits.conf on all of them. Here are the outputs on one of the problematic servers.

[root@ps-14-ldn-tele pscheduler.d]# psconfig pscheduler-stats
Agent Last Run Start Time: 2023/04/21 11:26:28
Agent Last Run End Time: 2023/04/21 11:26:31
Agent Last Run Process ID (PID): 6043
Agent Last Run Log GUID: 982BCE5E-E026-11ED-AAE4-70F37E5E890A
Total tasks managed by agent: 0

[root@ps-14-ldn-tele pscheduler.d]# psconfig remote list
=== pScheduler Agent ===
[
   {
      "url" : "http://perf-pwa.sanren.ac.za/pub/config/sanren_mesh_2019.json",
      "configure-archives" : true
   }
]

[root@ps-14-ldn-tele pscheduler.d]# psconfig validate http://perf-pwa.sanren.ac.za/pub/config/sanren_mesh_2019.json
Loading template ...... OK
Validating JSON schema ...... OK
Verifying object references ...... OK
pScheduler Validation (Quick) ...... OK

pSConfig JSON is valid

[root@ps-14-ldn-tele pscheduler.d]# pscheduler  troubleshoot
Performing basic troubleshooting of ps-14-ldn-tele.

ps-14-ldn-tele:

  Measuring MTU... 65535 (Local)
  Looking for pScheduler... OK.
  Fetching API level... 5
  Checking clock... OK.
  Exercising API... Archivers... Contexts... Tests... Tools... OK.
  Fetching service status... OK.
  Checking services... Ticker... Scheduler... Runner... Archiver... OK.
  Checking limits... OK.
  Idle test.... 6 seconds... Finished... Checking archiving... OK.

pScheduler appears to be functioning normally.

I can see some tasks are available on the web interface:

https://ps-14-ldn-tele-10g.perfsonar.ac.za/pscheduler/tasks

I don't see anything funny in the log files:

[root@ps-14-ldn-tele pscheduler]# tail -F /var/log/perfsonar/psconfig-pscheduler-agent.log
2023/04/21 11:35:03 INFO pid=6375 prog=main:: line=166 msg=Starting pSConfig pScheduler agent
2023/04/21 11:35:03 INFO pid=6375 prog=main:: line=176 guid=CAE52010-E027-11ED-9B71-BCF47E5E890A msg=Running agent...
2023/04/21 11:35:05 INFO pid=6375 prog=main:: line=178 guid=CAE52010-E027-11ED-9B71-BCF47E5E890A msg=Agent completed running
2023/04/21 11:35:05 INFO pid=6375 prog=main:: line=189 guid=CAE52010-E027-11ED-9B71-BCF47E5E890A msg=Time until next record refresh is 3598 seconds
2023/04/21 12:35:05 INFO pid=6375 prog=main:: line=176 guid=2E4ED44A-E030-11ED-9B71-BCF47E5E890A msg=Running agent...
2023/04/21 12:35:08 INFO pid=6375 prog=main:: line=178 guid=2E4ED44A-E030-11ED-9B71-BCF47E5E890A msg=Agent completed running
2023/04/21 12:35:08 INFO pid=6375 prog=main:: line=189 guid=2E4ED44A-E030-11ED-9B71-BCF47E5E890A msg=Time until next record refresh is 3597 seconds
2023/04/21 13:35:08 INFO pid=6375 prog=main:: line=176 guid=91957E34-E038-11ED-9B71-BCF47E5E890A msg=Running agent...
2023/04/21 13:35:10 INFO pid=6375 prog=main:: line=178 guid=91957E34-E038-11ED-9B71-BCF47E5E890A msg=Agent completed running
2023/04/21 13:35:10 INFO pid=6375 prog=main:: line=189 guid=91957E34-E038-11ED-9B71-BCF47E5E890A msg=Time until next record refresh is 3598 seconds

[root@ps-14-ldn-tele pscheduler]# tail -F /var/log/perfsonar/psconfig-pscheduler-agent-transactions.log
2023/04/21 12:35:08 INFO guid=2E4ED44A-E030-11ED-9B71-BCF47E5E890A action="create" msg=Creating tasks
2023/04/21 12:35:08 INFO guid=2E4ED44A-E030-11ED-9B71-BCF47E5E890A action="create" msg=No tasks to create
2023/04/21 12:35:08 INFO guid=2E4ED44A-E030-11ED-9B71-BCF47E5E890A action="create" msg=Done creating tasks
2023/04/21 13:35:09 INFO guid=91957E34-E038-11ED-9B71-BCF47E5E890A action="list" url="https://localhost/pscheduler msg=Getting task list from https://localhost/pscheduler
2023/04/21 13:35:10 INFO guid=91957E34-E038-11ED-9B71-BCF47E5E890A action="delete" msg=Deleting tasks
2023/04/21 13:35:10 INFO guid=91957E34-E038-11ED-9B71-BCF47E5E890A action="delete" msg=No tasks marked for deletion
2023/04/21 13:35:10 INFO guid=91957E34-E038-11ED-9B71-BCF47E5E890A action="delete" msg=Done deleting tasks
2023/04/21 13:35:10 INFO guid=91957E34-E038-11ED-9B71-BCF47E5E890A action="create" msg=Creating tasks
2023/04/21 13:35:10 INFO guid=91957E34-E038-11ED-9B71-BCF47E5E890A action="create" msg=No tasks to create
2023/04/21 13:35:10 INFO guid=91957E34-E038-11ED-9B71-BCF47E5E890A action="create" msg=Done creating tasks

Any ideas ?

Regards

Johann

--

SANReN Engineer

South African National Research Network (SANReN)

National Integrated Cyber Infrastructure System (NICIS)

CSIR NextGen Enterprises and Institutions Cluster

Office: 012 841 2066Email: jhugo@sanren.ac.za, jhugo@csir.co.zaWebsite: www.sanren.ac.za / www.csir.co.za