perfsonar-user - RE: [perfsonar-user] Tests are not scheduling on pscheduler

Subject: perfSONAR User Q&A and Other Discussion

List archive

RE: [perfsonar-user] Tests are not scheduling on pscheduler

From: "Garnizov, Ivan" <>
To: Murilo Vetter <>
Cc: Mark Feit <>, perfsonar-user <>
Subject: RE: [perfsonar-user] Tests are not scheduling on pscheduler
Date: Wed, 3 Apr 2019 12:24:42 +0000

Hi Murillo,

I understand your problem now.

You have your container constructed without the psconfig configuration inside and thus the system operates without the knowledge of the mesh configuration, which is added during the operation of the pS container….

In your case, I would suggest to prepare the psconfig configuration for the meshes with the building of the container. Thus when started with the container psconfig will automatically parse it and configure pscheduler.

There might be other mean to restart a service without the use of the ‘systemctl’ command. Of course I can’t recommend that, but I guess in your case, there is nothing to loose.

Best regards,

Ivan

From: Murilo Vetter [mailto:]
Sent: Wednesday, April 3, 2019 1:31 PM
To: Garnizov, Ivan (RRZE) <>
Cc: Ezra D Kissel <>; Mark Feit <>; perfsonar-user <>
Subject: Re: Tests are not scheduling on pscheduler

Hi Ivan,

About the suggested action:

1) psconfig remotes is configured as expected, as we can see bellow:

[root@monipe-to-perfsonar /]# psconfig remote list
=== pScheduler Agent ===
[
   {
      "url" : "http://monipe-central.rnp.br:8000/pub/config/matriz",
      "configure-archives" : true
   }
]

2) About restart the service, as I am running in a container, I could not restart it, as we can see bellow:

[root@monipe-to-perfsonar /]# systemctl status psconfig-pscheduler-agent
Failed to get D-Bus connection: Operation not permitted
[root@monipe-to-perfsonar /]# systemctl restart psconfig-pscheduler-agent
Failed to get D-Bus connection: Operation not permitted
[root@monipe-to-perfsonar /]# systemctl stop psconfig-pscheduler-agent
Failed to get D-Bus connection: Operation not permitted

Is there another way to check? I tried to restart the container with no success.

I wounder if would have a log that would show the problem in setting the hosts as responsible to run the tests and show that in this host would have a mismatch.

Regards,

Murilo Vetter
Analista REMEP-FLN/PoP-SC
PoP-SC - Ponto de Presença da RNP em Santa Catarina
REMEP-FLN - Rede Metropolitana de Educação e Pesquisa da Região de Florianópolis
RNP - Rede Nacional de Ensino e Pesquisa
UFSC - Universidade Federal de Santa Catarina
SETIC - Superintendência de Governança Eletrônica e Tecnologia da Informação e Comunicação
Fundações de Apoio FEESC e FEPESE
--/--

www.pop-sc.rnp.br
remep.pop-sc.rnp.br
CHAMADOS:
+55(48)3721-3000
INOC: 11242*100

De: "Ivan Garnizov" <>
Para: "Murilo Vetter" <>
Cc: "Ezra D Kissel" <>, "Mark Feit" <>, "perfsonar-user" <>
Enviadas: Quarta-feira, 3 de abril de 2019 7:00:29
Assunto: RE: Tests are not scheduling on pscheduler

Hello Murilo,

Your response does not include confirmation / statements about the suggested actions.

It is not clear now, if the shared results are after a successful restart or not.

The /var/lib/perfsonar/psconfig/psc_tracker will be populated after valid mesh configuration had been identified and parsed by psconfig.

Best regards,

Ivan

Von: Murilo Vetter []
Gesendet: Dienstag, 2. April 2019 18:25
An: Garnizov, Ivan (RRZE) <>
Cc: Ezra D Kissel <>; Mark Feit <>; perfsonar-user <>
Betreff: Re: AW: Tests are not scheduling on pscheduler

Hi list,

I am running in containers both hosts (one that it is working and another that it is not working).

In the broken one, it appears the following lines in the log:

2019/04/01 08:23:56 INFO guid=A346FB38-5470-11E9-BDC4-F723A98EDC83 action="list" url="https://localhost/pscheduler" msg=Getting task list from https://localhost/psched
uler
2019/04/01 08:24:02 INFO guid=A346FB38-5470-11E9-BDC4-F723A98EDC83 action="delete" msg=Deleting tasks
2019/04/01 08:24:02 INFO guid=A346FB38-5470-11E9-BDC4-F723A98EDC83 action="delete" msg=No tasks marked for deletion
2019/04/01 08:24:02 INFO guid=A346FB38-5470-11E9-BDC4-F723A98EDC83 action="delete" msg=Done deleting tasks
2019/04/01 08:24:02 INFO guid=A346FB38-5470-11E9-BDC4-F723A98EDC83 action="create" msg=Creating tasks
2019/04/01 08:24:02 INFO guid=A346FB38-5470-11E9-BDC4-F723A98EDC83 action="create" msg=No tasks to create
2019/04/01 08:24:02 INFO guid=A346FB38-5470-11E9-BDC4-F723A98EDC83 action="create" msg=Done creating tasks
--

No error message that could tell me what it is wrong. Any clue how to debug it?

When is the moment that the file /var/lib/perfsonar/psconfig/psc_tracker is written?

Regards,

www.pop-sc.rnp.br
remep.pop-sc.rnp.br
CHAMADOS:
+55(48)3721-3000
INOC: 11242*100

De: "Ivan Garnizov" <>
Para: "Murilo Vetter" <>, "Ezra D Kissel" <>
Cc: "Mark Feit" <>, "perfsonar-user" <>
Enviadas: Terça-feira, 2 de abril de 2019 12:25:18
Assunto: AW: Tests are not scheduling on pscheduler

Hello Murilo,

It may sound strange, but did you restart the “psconfig-pscheduler-agent” service?

Also your prompt looks strange. It appears like you are on a different system running those commands.

Anyway please make sure you have on the problematic host the psconfig remotes configured

psconfig remote list
=== pScheduler Agent ===
[
   {
      "url" : "http://monipe-central.rnp.br:8000/pub/config/matriz",
      "configure-archives" : true
   }
]

and then restart the service.

Please check again the /var/log/perfsonar/psconfig-pscheduler-agent-transactions.log

There should be messages hinting the mesh configuration in question is being checked.

Regards,

Ivan Garnizov

GEANT WP6T3: pS development team

GEANT WP6T1: pS deployments GN Operations

GEANT WP9T2:Software governance in GEANT

Von: [] Im Auftrag von Murilo Vetter
Gesendet: Montag, 1. April 2019 22:46
An: Kissel, Ezra D <>
Cc: Mark Feit <>; perfsonar-user <>
Betreff: Re: [perfsonar-user] Tests are not scheduling on pscheduler

Hi Ezra,

In the mentioned file no errors information:

2019/04/01 13:25:16 INFO guid=B9C249C4-549A-11E9-B3C8-E524A98EDC83 action="delete" msg=Deleting tasks
2019/04/01 13:25:16 INFO guid=B9C249C4-549A-11E9-B3C8-E524A98EDC83 action="delete" msg=No tasks marked for deletion
2019/04/01 13:25:16 INFO guid=B9C249C4-549A-11E9-B3C8-E524A98EDC83 action="delete" msg=Done deleting tasks
2019/04/01 13:25:16 INFO guid=B9C249C4-549A-11E9-B3C8-E524A98EDC83 action="create" msg=Creating tasks
2019/04/01 13:25:16 INFO guid=B9C249C4-549A-11E9-B3C8-E524A98EDC83 action="create" msg=No tasks to create
2019/04/01 13:25:16 INFO guid=B9C249C4-549A-11E9-B3C8-E524A98EDC83 action="create" msg=Done creating tasks
2019/04/01 14:25:16 INFO guid=1D833F88-54A3-11E9-B3C8-E524A98EDC83 action="list" url="https://localhost/pscheduler" msg=Getting task list from https://localhost/pscheduler
2019/04/01 14:25:19 INFO guid=1D833F88-54A3-11E9-B3C8-E524A98EDC83 action="delete" msg=Deleting tasks
2019/04/01 14:25:19 INFO guid=1D833F88-54A3-11E9-B3C8-E524A98EDC83 action="delete" msg=No tasks marked for deletion
2019/04/01 14:25:19 INFO guid=1D833F88-54A3-11E9-B3C8-E524A98EDC83 action="delete" msg=Done deleting tasks
2019/04/01 14:25:19 INFO guid=1D833F88-54A3-11E9-B3C8-E524A98EDC83 action="create" msg=Creating tasks
2019/04/01 14:25:19 INFO guid=1D833F88-54A3-11E9-B3C8-E524A98EDC83 action="create" msg=No tasks to create
2019/04/01 14:25:19 INFO guid=1D833F88-54A3-11E9-B3C8-E524A98EDC83 action="create" msg=Done creating tasks
2019/04/01 15:25:19 INFO guid=812B4A46-54AB-11E9-B3C8-E524A98EDC83 action="list" url="https://localhost/pscheduler" msg=Getting task list from https://localhost/pscheduler
2019/04/01 15:25:21 INFO guid=812B4A46-54AB-11E9-B3C8-E524A98EDC83 action="delete" msg=Deleting tasks
2019/04/01 15:25:21 INFO guid=812B4A46-54AB-11E9-B3C8-E524A98EDC83 action="delete" msg=No tasks marked for deletion
2019/04/01 15:25:21 INFO guid=812B4A46-54AB-11E9-B3C8-E524A98EDC83 action="delete" msg=Done deleting tasks
2019/04/01 15:25:21 INFO guid=812B4A46-54AB-11E9-B3C8-E524A98EDC83 action="create" msg=Creating tasks
2019/04/01 15:25:21 INFO guid=812B4A46-54AB-11E9-B3C8-E524A98EDC83 action="create" msg=No tasks to create
2019/04/01 15:25:21 INFO guid=812B4A46-54AB-11E9-B3C8-E524A98EDC83 action="create" msg=Done creating tasks
2019/04/01 16:25:22 INFO guid=E44FDD0A-54B3-11E9-B3C8-E524A98EDC83 action="list" url="https://localhost/pscheduler" msg=Getting task list from https://localhost/pscheduler
2019/04/01 16:25:25 INFO guid=E44FDD0A-54B3-11E9-B3C8-E524A98EDC83 action="delete" msg=Deleting tasks
2019/04/01 16:25:25 INFO guid=E44FDD0A-54B3-11E9-B3C8-E524A98EDC83 action="delete" msg=No tasks marked for deletion
2019/04/01 16:25:25 INFO guid=E44FDD0A-54B3-11E9-B3C8-E524A98EDC83 action="delete" msg=Done deleting tasks
2019/04/01 16:25:25 INFO guid=E44FDD0A-54B3-11E9-B3C8-E524A98EDC83 action="create" msg=Creating tasks
2019/04/01 16:25:25 INFO guid=E44FDD0A-54B3-11E9-B3C8-E524A98EDC83 action="create" msg=No tasks to create
2019/04/01 16:25:25 INFO guid=E44FDD0A-54B3-11E9-B3C8-E524A98EDC83 action="create" msg=Done creating tasks

I tried to change the configuration of /etc/perfsonar/psconfig/pscheduler-agent.json adding the attribute pscheduler-assist-server but it didn't help.

On thing that the followoing file /var/lib/perfsonar/psconfig/psc_tracker is empty like it:

{
"archives" : {},
"leads" : {}
}
--

In another host that the scheduling is working, it looks like this:

{
   "archives" : {
      "UmrpleC0xJUosfKOeTb7qw" : "UmrpleC0xJUosfKOeTb7qw",
      "SIFdxrCaheQUgBzXkZY5NA" : "SIFdxrCaheQUgBzXkZY5NA"
   },
   "leads" : {
      "https://monipe-ro-banda.rnp.br/pscheduler" : {
         "success_time" : 1554150625
      },
      "https://monipe-ro-atraso.rnp.br/pscheduler" : {
         "success_time" : 1554121653
      }
   }
}

What is going on with the problem host? Any clue?

Regards,

www.pop-sc.rnp.br
remep.pop-sc.rnp.br
CHAMADOS:
+55(48)3721-3000
INOC: 11242*100

De: "Kissel, Ezra D" <>
Para: "Murilo Vetter" <>
Cc: "Mark Feit" <>, "perfsonar-user" <>
Enviadas: Segunda-feira, 1 de abril de 2019 17:08:40
Assunto: RE: Tests are not scheduling on pscheduler

Hi Murilo,

Are there any errors present in /var/log/perfsonar/psconfig-pscheduler-agent-transactions.log ?

My similar issue a while back was due to host resolution, which required some “helper” configuration. See

https://lists.internet2.edu/sympa/arc/perfsonar-user/2019-01/msg00022.html

- ezra

From: <> On Behalf Of Murilo Vetter
Sent: Monday, April 1, 2019 3:57 PM
To: Murilo Vetter <>
Cc: Mark Feit <>; perfsonar-user <>
Subject: Re: [perfsonar-user] Tests are not scheduling on pscheduler

Hi list,

I would like to understand how the perfSONAR hosts know when to schedule tasks based on psconfig remote agent.

I am trying to debug a wierd behavior that the host appears on the configuration of PWA and in the following json, but it does not schedule on pscheduler.

[root@monipe-to-perfsonar /]# psconfig remote list
=== pScheduler Agent ===
[
   {
      "url" : "http://monipe-central.rnp.br:8000/pub/config/matriz",
      "configure-archives" : true
   }
]

If I try to troubleshoot the host, it looks it is ok.

[root@monipe-to-perfsonar /]# pscheduler troubleshoot --host monipe-to-atraso.rnp.br
Performing basic troubleshooting of monipe-to-atraso.rnp.br.

monipe-to-atraso.rnp.br:

Checking path MTU... 65520+
Checking for pScheduler... OK.
Checking clock... OK.
Idle test.... 12 seconds.... Checking archiving... OK.

pScheduler appears to be functioning normally.
[root@monipe-to-perfsonar /]# pscheduler troubleshoot --host monipe-to-banda.rnp.br
Performing basic troubleshooting of monipe-to-banda.rnp.br.

monipe-to-banda.rnp.br:

Checking path MTU... 65520+
Checking for pScheduler... OK.
Checking clock... OK.
Idle test.... 13 seconds.... Checking archiving... OK.

pScheduler appears to be functioning normally.

It is resolving the server name:

[root@monipe-to-perfsonar /]# hostname -A
monipe-to-atraso.rnp.br monipe-to-banda.rnp.br monipe-to-perfsonar

What am I missing? Is there a way to troubleshoot it in a better way?

Regards,

www.pop-sc.rnp.br
remep.pop-sc.rnp.br
CHAMADOS:
+55(48)3721-3000
INOC: 11242*100

De: "Murilo Vetter" <>
Para: "Mark Feit" <>
Cc: "perfsonar-user" <>
Enviadas: Sexta-feira, 29 de março de 2019 17:43:52
Assunto: Re: [perfsonar-user] Tests are not scheduling on pscheduler

Hi Guys,

It is strange that is is not scheduling.

I am using hostnames to try to schedule. Should it works with hostnames or just IP address?

Some commands below:

[root@monipe-ba-perfsonar tmp]# psconfig remote list
=== pScheduler Agent ===
[
   {
      "url" : "http://monipe-central.rnp.br:8000/pub/config/matriz"
   }
]

[root@monipe-ba-perfsonar tmp]# psconfig pscheduler-tasks
{
"tasks" : []
}

Any help?

Regards,

www.pop-sc.rnp.br
remep.pop-sc.rnp.br
CHAMADOS:
+55(48)3721-3000
INOC: 11242*100

De: "Mark Feit" <>
Para: "Murilo Vetter" <>, "perfsonar-user" <>
Enviadas: Quinta-feira, 28 de março de 2019 18:45:52
Assunto: Re: [perfsonar-user] Tests are not scheduling on pscheduler

Murilo Vetter writes:

I am trying to schedule some tests from the host: monipe-ba-atraso.rnp.br / monipe-ba-banda.rnp.br ( 2 net interfaces in this host, 1 for ping/trace/owamp and 1 for troughput ).

I am getting problems since it does not put on schedule.

I already check this issues:

- Hostname:

[root@monipe-ba-perfsonar httpd]# hostname -A
monipe-ba-atraso.rnp.br monipe-ba-banda.rnp.br monipe-ba-perfsonar

- Troubleshooting:

[root@monipe-ba-perfsonar httpd]# pscheduler troubleshoot monipe-ba-atraso.rnp.br
…

If I understand this correctly, you’re trying to troubleshoot from the local host to itself. That will (and did) work fine for the single-host idle tests. Once it tries to do something multi-participant (simplestream, throughput), it’s going to fail because the same machine can’t be both ends of a multi-participant test even if they’re different interfaces. I was able to run tasks to and from that host with no problem, although I did notice that either the system or the network between here and there was a bit sluggish.

localhost and monipe-ba-atraso.rnp.br:

Checking path MTU... 65520+
Checking timekeeping... OK.
Simple stream test... Failed.
Unable to post task: Error while tasking monipe-ba-atraso.rnp.br: Unable to post task to monipe-ba-atraso.rnp.br: Internal problem; see system logs.

Unless there’s some other problem causing that, you should be seeing a message that says “Task already exists. All participants must be on separate systems.” Early versions of pScheduler choked like that, but it was corrected before anything went into production.

Is there anything of interest in the logs on that system?

--Mark

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user

Re: [perfsonar-user] Tests are not scheduling on pscheduler, Murilo Vetter, 04/01/2019
- RE: [perfsonar-user] Tests are not scheduling on pscheduler, Kissel, Ezra D, 04/01/2019
  - Re: [perfsonar-user] Tests are not scheduling on pscheduler, Murilo Vetter, 04/01/2019
    - AW: [perfsonar-user] Tests are not scheduling on pscheduler, Garnizov, Ivan, 04/02/2019
      - Re: [perfsonar-user] Tests are not scheduling on pscheduler, Murilo Vetter, 04/02/2019
- <Possible follow-up(s)>
- RE: [perfsonar-user] Tests are not scheduling on pscheduler, Garnizov, Ivan, 04/03/2019
  - Re: [perfsonar-user] Tests are not scheduling on pscheduler, Murilo Vetter, 04/03/2019
    - RE: [perfsonar-user] Tests are not scheduling on pscheduler, Garnizov, Ivan, 04/03/2019
      - Re: [perfsonar-user] Tests are not scheduling on pscheduler, Murilo Vetter, 04/03/2019
        
        Re: [perfsonar-user] Tests are not scheduling on pscheduler, Andrew Lake, 04/03/2019
        
        Re: [perfsonar-user] Tests are not scheduling on pscheduler, Murilo Vetter, 04/03/2019
        
        Re: [perfsonar-user] Tests are not scheduling on pscheduler, Andrew Lake, 04/04/2019
        Re: [perfsonar-user] Tests are not scheduling on pscheduler, Murilo Vetter, 04/04/2019
        Re: [perfsonar-user] Tests are not scheduling on pscheduler, Murilo Vetter, 04/04/2019
        Re: [perfsonar-user] Tests are not scheduling on pscheduler, Andrew Lake, 04/04/2019
        Re: [perfsonar-user] Tests are not scheduling on pscheduler, Murilo Vetter, 04/04/2019
        Re: [perfsonar-user] Tests are not scheduling on pscheduler, Murilo Vetter, 04/05/2019
        Re: [perfsonar-user] Tests are not scheduling on pscheduler, Murilo Vetter, 04/05/2019

List archive

RE: [perfsonar-user] Tests are not scheduling on pscheduler