Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] maddash-01.nordu.net

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] maddash-01.nordu.net


Chronological Thread 
    < Chronological >      < Thread >
  • From: Robert Lageano <>
  • To: Christopher Walker <>, Joachim Hunosøe <>, Tristan Sullivan <>
  • Cc: "" <>
  • Subject: Re: [perfsonar-user] maddash-01.nordu.net
  • Date: Mon, 25 Sep 2023 04:19:27 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=anu.edu.au; dmarc=pass action=none header.from=anu.edu.au; dkim=pass header.d=anu.edu.au; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zxmq9yN71I/h+yDvICiS2bX9nVmxTAlRynCNI77qui0=; b=jkeK5dsqboYGNo9vb0yNgMDMPFi25zYizmBI3QzxYUW/Op/Tn8SfoaSrbSCRmyH/egDue3JpjxpQXClTQZ1uPZgCOfV2KzXsfNAbjKjYkhV4wjCgLU/Nk16gKG0rG08g7Q78jxNlaGjvsTffjW5Uh9paDelZJZrwtshJvPxl2kna1sgn4cUEys6mf7EEWvZWsU3JUjq20sMKXuUe+lOlEK55QPEosSMUV6dNJ4ZJpefpPGQjnoGmZrlIRIaWZAVv7gShmBizl3IXg+8nZl/p2CSb8Fpc6BaDSooLWvypSuGKK0eam2c5YSZtXPTHQOIHprE+6pYQwRJouAeya8QF3Q==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TCimpllyt+Jg8aUaRO+MkAMYP5+N0UOaHwnX9yYpYyHAmvNexCt+cy06QfYGajiAUhbWwBfn7orN18TWNYU0cBXqIsqeYzmtPoSAxM7rqEdNnVLZZKvzGonz1EQQGhlAAIo8gyc8OZjDNMV6Hp+cZjfcgKzhA5gNwUSUSayw1lzkRwx/pOfhHn59AxGg8EaqCFc6N/oWw4k9Bbeqrr/PyaxyY+8X6u/0rkGxe5+s8q5GFBb5lDyCFK5HdZVkGoXQXB6O8eMZ04ZnrCVJFQnDBKAb7rFbXCxGp37QVxrdmsfHr+iVDcTj+77kIdLOSLhVTyepjGS93hMysg9TiKJTUg==
  • Msip_labels:
Hi Tristan,

I've just tried increasing the number of shards as per your instructions. However, that still hasn't solved our issues here. Pscheduler-archiver continues to crash even with the increased number of shards. Would you have any other solution for that?

Sep 25 00:42:45 perfsonar python3[38575]: archiver WARNING  5681: Failed to archive https://perfsonar/pscheduler/tasks/e047c1db-b3bb-4c62-bf2c-e5>
Sep 25 00:42:45 perfsonar python3[38575]: archiver WARNING  <html><head>
Sep 25 00:42:45 perfsonar python3[38575]: archiver WARNING  <title>503 Service Unavailable</title>
Sep 25 00:42:45 perfsonar python3[38575]: archiver WARNING  </head><body>
Sep 25 00:42:45 perfsonar python3[38575]: archiver WARNING  <h1>Service Unavailable</h1>
Sep 25 00:42:45 perfsonar python3[38575]: archiver WARNING  <p>The server is temporarily unable to service your
Sep 25 00:42:45 perfsonar python3[38575]: archiver WARNING  request due to maintenance downtime or capacity
Sep 25 00:42:45 perfsonar python3[38575]: archiver WARNING  problems. Please try again later.</p>
Sep 25 00:42:45 perfsonar python3[38575]: archiver WARNING  </body></html>
Sep 25 00:42:45 perfsonar python3[38575]: archiver WARNING  5681: Gave up archiving https://perfsonar/pscheduler/tasks/e047c1db-b3bb-4c62-bf2c-e5>

Whenever I restart pscheduler, it seems to be happy again but it crashes after a few minutes. 

Cheers,

Rob
-----------------------------------------------------------------------------
Robert Lageano
Senior IT Network Security Specialist
National Computational Infrastructure (NCI)
The Australian National University
143 Ward Road
Acton, ACT, 2601
M: +61 452 535 678

Want the latest news from NCI? nci.org.au

Find out more about NCI: YouTube | Facebook | Twitter | LinkedIn | Podcast






From: <> on behalf of Tristan Sullivan <>
Sent: Saturday, September 23, 2023 3:48 AM
To: Christopher Walker <>; Joachim Hunosøe <>
Cc: <>
Subject: Re: [perfsonar-user] maddash-01.nordu.net
 
You don't often get email from . Learn why this is important
Hi,

In case anyone with the shards problem wants to revive their host before the next release, a workaround is to increase the maximum number of shards:

curl -k --netrc-file pw -X PUT https://localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d '{"persistent": {"cluster.max_shards_per_node": "2000"}}'

Where pw is a file in the directory from which you run the command, in the following format:

machine localhost login admin password {password}

The password can be found in /etc/perfsonar/opensearch/auth_setup.out.

Tristan

From: <> on behalf of Joachim Hunosøe <>
Sent: September 22, 2023 10:36 AM
To: Christopher Walker <>
Cc: <>
Subject: Re: [perfsonar-user] maddash-01.nordu.net
 
Hi Chris

Alright, ah yeah holy moly the opensciencegrid ain’t looking good,  thank you so much for the info, it helps calm down again. After many frustrating hours.

Then I’m just looking forward for the next release :-) 

Best regards 
Joachim Hunosøe
System Engineer
+45 5374 7719


On 22 Sep 2023, at 15.12, Christopher Walker <> wrote:


Hi Joachim, 
 
Much of the WLCG/OSG mesh is down too, so it may well not be a problem with your config:

I know some/many hosts have an issue with number of shards - https://github.com/perfsonar/archive/issues/18 - which I think I've heard will be fixed in the next release of perfsonar. 

Chris 


From: <> on behalf of Joachim Hunosøe <>
Sent: Tuesday, September 19, 2023 2:32 PM
To: <>
Subject: [perfsonar-user] maddash-01.nordu.net
 
Hey guys

I had a mesh before my summer vacation with 5/6 host working 100% 
And when I get back to work Im lucky if I see 3/6 sometime 2/6 
Pretty unstable Maddash of some reason 

Psconfig are can be found here



Do one have some spare time to zoom about this mesh with me. 
So I can have an extra pair of eyes looking at it. 

My eyes start to cross each other 


Best regards 
Joachim Hunosøe 



Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under company number. 05747339, VAT number GB 197 0632 86. Jisc’s registered office is: 4 Portwall Lane, Bristol, BS1 6NB. T 0203 697 5800.

Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 02881024, VAT number GB 197 0632 86. The registered office is: 4 Portwall Lane, Bristol, BS1 6NB. T 0203 697 5800.

Jisc Commercial Limited is a wholly owned Jisc subsidiary and a company limited by shares which is registered in England under company number 09316933, VAT number GB 197 0632 86. The registered office is: 4 Portwall Lane, Bristol, BS1 6NB. T 0203 697 5800.

For more details on how Jisc handles your data see our privacy notice here: https://www.jisc.ac.uk/website/privacy-notice


Archive powered by MHonArc 2.6.24.

Top of Page