perfSONAR User Q&A and Other Discussion

["Garnizov, Ivan" <>]

Hello Johann,

 

Please tell, where did you fetch this Docker image of pS Testpoint from?

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

From: Johann Hugo [mailto:]
Sent: Tuesday, June 7, 2022 3:00 PM
To: Garnizov, Ivan (RRZE) <>
Cc:
Subject: Re: [perfsonar-user] Docker testpoint is not downloading the meshconfig file

 

Hi Ivan

 

The JSON file is not the problem. It's accessible and valid when I verify it by hand:

[root@ps-100-100g /]# psconfig validate http://perf-pwa.sanren.ac.za/pub/config/sanren_mesh_2019.json?format=psconfig
Loading template ...... OK
Validating JSON schema ...... OK
Verifying object references ...... OK
pScheduler Validation (Quick) ...... OK
pSConfig JSON is valid

 

The problem is that the pSConfig PScheduler Agent fails and the config file is never fetched.

[root@ps-100-100g /]# systemctl status psconfig*
● psconfig-pscheduler-agent.service - pSConfig PScheduler Agent
   Loaded: loaded (/usr/lib/systemd/system/psconfig-pscheduler-agent.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2022-06-07 12:09:08 SAST; 2h 44min ago
  Process: 160 ExecStart=/usr/lib/perfsonar/bin/psconfig_pscheduler_agent --config=/etc/perfsonar/psconfig/pscheduler-agent.json --logger=/etc/perfsonar/psconfig/pscheduler-agent-logger.conf --pidfile=/var/run/psconfig-pscheduler-agent.pid --user=perfsonar --group=perfsonar (code=exited, status=0/SUCCESS)
 Main PID: 380 (code=exited, status=25)

Jun 07 12:09:08 ps-100-100g systemd[1]: Starting pSConfig PScheduler Agent...
Jun 07 12:09:08 ps-100-100g systemd[1]: Started pSConfig PScheduler Agent.
Jun 07 12:09:08 ps-100-100g systemd[1]: psconfig-pscheduler-agent.service: main process exited, code=exited, status=25/n/a
Jun 07 12:09:08 ps-100-100g systemd[1]: Unit psconfig-pscheduler-agent.service entered failed state.
Jun 07 12:09:08 ps-100-100g systemd[1]: psconfig-pscheduler-agent.service failed.

 

Regards

Johann

 

On Tue, Jun 7, 2022 at 1:04 PM Garnizov, Ivan <> wrote:

Hi Johann,

 

Please check the current state with these commands:

 

-          curl http://perf-pwa.sanren.ac.za/pub/config/sanren_mesh_2019.json?format=psconfig

-          psconfig validate http://perf-pwa.sanren.ac.za/pub/config/sanren_mesh_2019.json?format=psconfig

 

I believe these should give you an idea of what’s wrong on the system.

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

From: [mailto:] On Behalf Of Johann Hugo
Sent: Tuesday, June 7, 2022 10:48 AM
To:
Subject: [perfsonar-user] Docker testpoint is not downloading the meshconfig file

 

Hi all

 

My docker testpoint is not downloading it's meshconfig file (verified with tcpdump on both the docker host and on my PWA server)

 

[root@ps-100-100g /]# psconfig remote list
=== pScheduler Agent ===
[
   {
      "bind-address" : "155.232.40.194",
      "url" : "http://perf-pwa.sanren.ac.za/pub/config/sanren_mesh_2019.json?format=psconfig"
   }
]

 

/var/log/perfsonar/psconfig-pscheduler-agent-tasks.log is empty

/var/log/perfsonar/psconfig-pscheduler-agent-transactions.log is empty

/var/log/perfsonar/psconfig-pscheduler-agent.log has the following error:
2022/06/07 07:46:27 ERROR pid=387 prog=main::__ANON__ line=131 msg=Died: /etc/perfsonar/psconfig/pscheduler.d watcher creation failed at /usr/lib/perfsonar/bin/psconfig_pscheduler_agent line 160

No errors in /var/log/messages

 

DNS inside the container is working fine and it can resolve perf-pwa.sanren.ac.za. 

Only forward DNS is configured for the container and it's working fine

[root@ps-100-100g /]# host ps-100-100g.perfsonar.ac.za
ps-100-100g.perfsonar.ac.za has address 155.232.40.194

 

ntpd is running on the docker host. 

lsregistration is working fine

The mesh config file is built on a PWA server. All my other servers + Maddash are happy with the mesh config file

I can download the meshconfig with "psconfig translate http://perf-pwa.sanren.ac.za/pub/config/sanren_mesh_2019.json?format=psconfig" and it looks fine.

 

My server setup:

Ubuntu 20.04.4, with a dual 100g Ethernet interface card

DNS + ntpd is running on the host. 

The container uses MACVLAN networking, on its own dedicated Ethernet adapter and it starts up using docker-compose

The testpoint image is set to image: perfsonar/testpoint:latest

The timezone is configured inside docker-compose.yml

 

Any ideas on how to debug this ?

 

Regards

Johann

 

--

SANReN Engineer

South African National Research Network (SANReN)

National Integrated Cyber Infrastructure System (NICIS)

CSIR NextGen Enterprises and Institutions Cluster

Office: 012 841 2066Email: , Website: www.sanren.ac.za / www.csir.co.za