Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] Opensearch running out of shards (1000/1000)

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] Opensearch running out of shards (1000/1000)


Chronological Thread 
    < Chronological >      < Thread >
  • From: Tim Chown <>
  • To: Tim Chown <>
  • Cc: "" <>
  • Subject: Re: [perfsonar-user] Opensearch running out of shards (1000/1000)
  • Date: Wed, 13 Sep 2023 14:19:04 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=jisc.ac.uk; dmarc=pass action=none header.from=jisc.ac.uk; dkim=pass header.d=jisc.ac.uk; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3HTjygBWipiHfC85VT2CsdmXcf/V7BOE4W3YASA7B2g=; b=fIBZX5g+nbGHUwRX4hBnvYdkE/W4/JlwRPi7Wq4dssapqK2ZBBRnGkNs/gmESQl3cXd2kHwnTGnh6ZBaIYD9rxt7nh+i2R7zA92r4J7kRe81C49Doup0drtm8hPEdMQpy4yah6Wg2QlC1cTaQOwz1o48i3oZLzLxOKzo+2PDKktK4YY880HePCheOc81/Nk1ef3SqPUqHG65BWUbM8d/vgwDjysjQEEBMhTxPrVhgh5Hkd/v1jyR+DWA4o1z4BxwzwsYIB2nLkfn/3niy6CEHpeSeraqOSJbdBJwpmJQ9Ub1PZjsd0rKY3H+5T1eECsYOBC+U9Q1ON5BDHf/NzxBBg==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=i4XV0U1NNXCspYoYE5RtQ1tzxZA0YT8LXzvvrbbN/DlpAHK+O3nkzB26t6iv80HKjxFeyzMiuSjkSkRSyVvvHCgDKGAJ1DwXB2lEtORxLSYSidqSfcHm64Y708/RZvr/+ZW03BGFkLdKoypUM3oQPj4I66VvNvMCKSAc2LYSMy90Eo9Ynftd0dnxlfP+c+6ufUqD+kjyMtLECXA0PSfBALiW0cgDrlWYBHZkLwje86g0xW1SHZcaHahLFWQsZajGJ+xV5S808hOSVd0aUc7VDnjxMW1C3+c4iv8TSdkFU2TrTbnfKnp68ixuNB2sf91LjGwfZepXGpbE9lN0J8Y6Xw==
Hi,

Scratch that, the email of 30th August from Andy answers the question, about the index lifecycle management policy issue resulting in too many new shards being created.

Hopefully 5.0.5 coming soon!

Cheers,
Tim

On 13 Sep 2023, at 15:09, Tim Chown <> wrote:

Hi,

We’re seeing servers running out of shards and thus perfSONAR falling over, and thus some rather barren maddish views.

An example error:

" "ip"=>"2001:630:1:112:0:0:0:3"}}, "reference"=>{"psconfig"=>{"created-by"=>{"uuid"=>"61E213D2-F410-11ED-B798-E28714F07E7B", "user-agent"=>"psconfig-pscheduler-agent"}}}, "id"=>"dc9c56e1-eaeb-4751-998f-7ce6d3c8c623"}], :response=>{"index"=>{"_index"=>"pscheduler_latencybg-2023.09.13", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}}}}”

It looks like a latency test is trying to save a result, but wants 2 more shards and opensearch is capped at 1000/1000. That seems a lot of shards - is it creating a lot of new indices which in turn need more shards?

We’ve seen this on multiple systems.  The error above is from http://ps-london-bw.perf.ja.net/toolkit/ which only has 67 tests running, with 100G interfaces.

Is this a known problem? I can’t immediately find an open issue mentioning shards.

Thanks,
Tim


Archive powered by MHonArc 2.6.24.

Top of Page