perfsonar-user - Re: [perfsonar-user] Opensearch taking a lot of space in /var

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] Opensearch taking a lot of space in /var

From: Tim Chown <>
To: "Yamamoto, Miguel" <>, Mark Feit <>, "" <>
Subject: Re: [perfsonar-user] Opensearch taking a lot of space in /var
Date: Thu, 5 Sep 2024 15:44:06 +0000
Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=jisc.ac.uk; dmarc=pass action=none header.from=jisc.ac.uk; dkim=pass header.d=jisc.ac.uk; arc=none
Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1yNeMH0CdiEF4vskxwZe48QyZZwPOJBBUBDFO5BiTE8=; b=YxdPBHVmdN4NzGihvcSynOYGnM2Jl6mH92Im404saEUXHpXc99+AE5Y4mI1+P5qxSoJwO4i6b/OLYAks+wiQDx1P5d2335MaL11BPLHRUXBPCHHe26+LuaesqoQepU7kPNWLMbDf9Bh3iNAemu5jSbwXWedqEFAp3HQn/kmr1je1cH0HlY8kw8eAvy/lj1PJ0WNV9JC7rufQoIB6WhlyZ9k8H+DofEVdCttS/LW4r3vUvuqEjbrJalZmNjoi/H1xg/Jfqqxy6g/xFiWenvFgX8GPyDnM5zMlr6uVTVldvHtUU0zIOfGEX0NMO9iTGa8tVUaKsQ0E92oUODS/2q3KHQ==
Arc-seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=m3ZXcuML0XmSufXUWbH1xnrVppkzr0grHu4HRvyxV1D+77y24KIR/A5mUW58rVaNGMO7Uz1IuS3VroAIvbgjSHZFOZRvuY6BpL5Ql2Ea6tXBtCFDG7E0r76kt6TiilJVD9koszqYkhjkmgNn9z8lSUa0D5ZeV/nqPx9B2HJm0yCZWfh4LBbZgpSBLwthEp/4BL7YL8IaK0nOakLa9eOyHyH+QU2YpljBsQevdbvZRn9DcPVQYHkKVKnYLMfZ/GVYwlY8hXkA4aQ95tb6EsOez9XKEzj3zPRpyw6uI+KsTdwmp+WIq4ztub6xPG5+6mV/gO24K8JbcIjqDGMZJ5UExQ==
Msip_labels: MSIP_Label_190374fc-c2b5-4c8e-bee8-6305ebc1550a_Enabled=True;MSIP_Label_190374fc-c2b5-4c8e-bee8-6305ebc1550a_SiteId=48f9394d-8a14-4d27-82a6-f35f12361205;MSIP_Label_190374fc-c2b5-4c8e-bee8-6305ebc1550a_SetDate=2024-09-05T15:44:05.9596180Z;MSIP_Label_190374fc-c2b5-4c8e-bee8-6305ebc1550a_ContentBits=0;MSIP_Label_190374fc-c2b5-4c8e-bee8-6305ebc1550a_Method=Privileged

Hi,

It seems for us that the bulk of the data is host information data rather than measurement data. Very useful to have, but not needed for longer-term reflection.

Is it possible to have varied preservation settings, e.g., the host information kept for say 1-2 weeks and the measurement data for 6-12 months?

Tim

From: <> on behalf of "Yamamoto, Miguel" <>
Date: Wednesday, 4 September 2024 at 20:42
To: Mark Feit <>, <>
Subject: [perfsonar-user] Opensearch taking a lot of space in /var

Hi Mark,

I am not sure if this is an ongoing issue, but Opensearch is continuously taking up a lot of space in /var. We could expand the drive, but I wanted to ask if there is a better way to handle the increasing disk utilization of OpenSearch. Our aim is to not always increase the drive as a workaround.

Current version: 5.1.2
The system has been up for about 2-3 months now

This is from the central archive machine and reading data coming from multiple testpoints (about 10+)

36G /var/lib/opensearch/nodes/0/indices/ZyA-IwTlQXGS2Z8gRKslaw/0/index

37G /var/lib/opensearch/nodes/0/indices/ZyA-IwTlQXGS2Z8gRKslaw/0

37G /var/lib/opensearch/nodes/0/indices/ZyA-IwTlQXGS2Z8gRKslaw

3.3G /var/lib/opensearch/nodes/0/indices/BbSkqaH9QVy0AlbUL4VG-g/0/index

3.3G /var/lib/opensearch/nodes/0/indices/BbSkqaH9QVy0AlbUL4VG-g/0

3.3G /var/lib/opensearch/nodes/0/indices/BbSkqaH9QVy0AlbUL4VG-g

1.8G /var/lib/opensearch/nodes/0/indices/u9-EbLT9R6Op5EZX0DVrcg/0/index

1.8G /var/lib/opensearch/nodes/0/indices/u9-EbLT9R6Op5EZX0DVrcg/0

1.8G /var/lib/opensearch/nodes/0/indices/u9-EbLT9R6Op5EZX0DVrcg

43G /var/lib/opensearch/nodes/0/indices

43G /var/lib/opensearch/nodes/0

43G /var/lib/opensearch/nodes

43G /var/lib/opensearch

44G /var/lib

1.1G /var/log/opensearch

1.2G /var/log

45G /var/

Thanks,
Miguel

[perfsonar-user] Opensearch taking a lot of space in /var, Yamamoto, Miguel, 09/04/2024
- Re: [perfsonar-user] Opensearch taking a lot of space in /var, Tim Chown, 09/05/2024

List archive

Re: [perfsonar-user] Opensearch taking a lot of space in /var