perfSONAR User Q&A and Other Discussion

[Zhi-Wei Lu <>]

Andy helped me found the latest problem.  My mesh configuration file has missing write_url for a number of pinger tests.

The command

psconfig translate https://ucdhost/UCD-mesh.json

has zero tests configured for the server.

After I fixed the write_url on the mesh configuration file, my tests started to work again.

 

I do need to drop all esmond tables in the postgresql database before running the restore script.

 

Thanks again Andy!

 

Zhi-Wei Lu

IET-CR-Network Operations Center

University of California, Davis

(530) 752-0155

 

From: <> On Behalf Of Zhi-Wei Lu
Sent: Thursday, September 06, 2018 12:35 PM
To: Andrew Lake <>
Cc:
Subject: Re: [perfsonar-user] Archiving to esmond failed after CentOS6 -> CentOS7 migration

 

I have dropped all tables in esmond (postgresql) and have run the migrate-restore script one more time.

 

Now I see runs for traceroute but not for ping or throughput in the log file:

 

Sep  6 12:27:08 mammoth-owamp-v4 journal: runner INFO     1575231: Running https://mammoth-v4.noc.ucdavis.edu/pscheduler/tasks/9951d807-57ed-47f1-b5dc-924e88256601/runs/91fec1cf-4642-48db-be78-0f46243fa2a7

Sep  6 12:27:08 mammoth-owamp-v4 journal: runner INFO     1575231: With traceroute: trace --dest mach1-v4.noc.ucdavis.edu --length 40 --source mammoth-v4.noc.ucdavis.edu --hops 64

Sep  6 12:27:08 mammoth-owamp-v4 journal: runner INFO     1575231: Run succeeded.

Sep  6 12:29:22 mammoth-owamp-v4 journal: runner INFO     1575387: Running https://mammoth-v4.noc.ucdavis.edu/pscheduler/tasks/30d677d6-fc6c-4dfd-8edc-8be365a846f0/runs/642479c3-2f9e-4eeb-b654-fdc49e074023

Sep  6 12:29:22 mammoth-owamp-v4 journal: runner INFO     1575387: With traceroute: trace --dest fumee-v4.noc.ucdavis.edu --length 40 --source mammoth-v4.noc.ucdavis.edu --hops 64

Sep  6 12:29:22 mammoth-owamp-v4 journal: runner INFO     1575387: Run succeeded.

 

The system /etc/perfsonar/psconfig/pscheduler-agent.json

 

{

   "remotes" : [

      {

         "url" : "https://ucdhost/UCD-mesh.json"

      }

   ]

}

 

The UCD-mesh defines runs: throughtput, ping, and traceroute.  What daemons do I need to restart again?

 

Thank you.

 

Zhi-Wei Lu

 

IET-CR-Network Operations Center

University of California, Davis

(530) 752-0155

---

From: <> on behalf of Zhi-Wei Lu <>
Sent: Thursday, September 6, 2018 9:26:36 AM
To: Andrew Lake
Cc:
Subject: Re: [perfsonar-user] Archiving to esmond failed after CentOS6 -> CentOS7 migration

 

Thanks, Andy!

 

It turns out that the "authtoken_token" in the esmond database has the "NEW" value rather than the migrated value.

I looked at the backup and restore scripts and examined the postgresql database.  I re-"restore" the database, but it

didn't replace the "new" value.  I had to drop the authtoken_token table (one table only) and re-run the restore script one more time.

It then restored the "old" authtoken values. After that I ran the scirpt again and restarted a few daemon last night,

 

/usr/lib/perfsonar/scripts/system_environment/configure_esmond --force

 

In the /var/log/pscheduler/pscheduler.log, we can see successful runs,

...

Sep  6 09:18:39 mammoth-owamp-v4 journal: runner INFO     1575526: Running https://mammoth-v4.noc.ucdavis.edu/pscheduler/tasks/dbc2f80b-d785-4366-94fd-bc64d2a042ba/runs/5e0f34cc-1f84-4499-93ce-1774a0df7f1c

Sep  6 09:18:39 mammoth-owamp-v4 journal: runner INFO     1575526: With traceroute: trace --dest melange-v4.noc.ucdavis.edu --length 40 --source mammoth-v4.noc.ucdavis.edu --hops 64

Sep  6 09:18:39 mammoth-owamp-v4 journal: runner INFO     1575526: Run succeeded.

...

 

However, I do not see any new data in the web-gui

 

http://mammoth.noc.ucdavis.edu/toolkit/

 

Maybe, there are still inconsistency in the postgresql esmond database with cassandra data sources (mixing with new values which were not restore properly).  I will try to drop all esmond tables and re-run restore script.

 

Thanks again Andy!

 

Zhi-Wei Lu

IET-CR-Network Operations Center

University of California, Davis

(530) 752-0155

---

From: Andrew Lake <>
Sent: Thursday, September 6, 2018 6:06:44 AM
To: Zhi-Wei Lu
Cc:
Subject: Re: [perfsonar-user] Archiving to esmond failed after CentOS6 -> CentOS7 migration

 

Hi,

 

I think maybe one of the steps you have done has gotten things working again, possible it took some time to propagate. If I look at your newest tasks they seem to be archiving successfully to localhost. For example see:

 

https://mammoth-owamp-v4.noc.ucdavis.edu/pscheduler/tasks/44d685eb-40c6-48eb-90c9-fc5dd427e3cc/runs/first?pretty

 

Notice the first JSON blob labelled "archivings" and how it indicates it succeeded in the "diags" section. 

 

Andy

 

 

 

 

On Wed, Sep 5, 2018 at 1:46 PM, Zhi-Wei Lu <> wrote:

Hi all,

 

I have just upgraded and migrated a few of my CentOS 6 perfsonar servers to CentOS 7.  I don’t see any new tests recorded.  I then noticed error messages in /var/log/ pscheduler/pscheduler.log

 

Sep  5 10:28:37 mammoth-owamp-v4 journal: runner INFO     1575513: With ping: rtt --count 10 --dest grx-v4.noc.ucdavis.edu --interval PT1S --source mammoth-owamp-v4.noc.ucdavis.edu --length 1000

Sep  5 10:28:46 mammoth-owamp-v4 journal: runner INFO     1575513: Run succeeded.

Sep  5 10:28:47 mammoth-owamp-v4 journal: archiver WARNING  1473429: Failed to archive https://localhost/pscheduler/tasks/44d685eb-40c6-48eb-90c9-fc5dd427e3cc/runs/79c06354-dd20-48c1-afe4-e6bf4fdd0ae2 to esmond: 401: Invalid token.

Sep  5 10:29:02 mammoth-owamp-v4 journal: archiver WARNING  1472536: Failed to archive https://localhost/pscheduler/tasks/c2a13eb2-b022-443d-95b9-42fa62c1859c/runs/44381ff1-71c4-4875-922f-c9cc9b253c29 to esmond: 401: Invalid token.

Sep  5 10:29:47 mammoth-owamp-v4 journal: archiver WARNING  1473429: Failed to archive https://localhost/pscheduler/tasks/44d685eb-40c6-48eb-90c9-fc5dd427e3cc/runs/79c06354-dd20-48c1-afe4-e6bf4fdd0ae2 to esmond: 401: Invalid token.

Sep  5 10:34:47 mammoth-owamp-v4 journal: archiver WARNING  1473429: Failed to archive https://localhost/pscheduler/tasks/44d685eb-40c6-48eb-90c9-fc5dd427e3cc/runs/79c06354-dd20-48c1-afe4-e6bf4fdd0ae2 to esmond: Archiver permanently abandoned registering test after 3 attempt(s): 401: Invalid token.

Sep  5 10:34:47 mammoth-owamp-v4 journal: archiver WARNING  1473429: Gave up archiving https://localhost/pscheduler/tasks/44d685eb-40c6-48eb-90c9-fc5dd427e3cc/runs/79c06354-dd20-48c1-afe4-e6bf4fdd0ae2 to esmond

 

I guess that “localhost” probably should be the real host name, is that right? As I have a few hostnames for the server, which one I should use?  How do I configure perfsonar to use that name?

 

Thank you!

 

Zhi-Wei Lu

IET-CR-Network Operations Center

University of California, Davis

(530) 752-0155

 

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user