perfSONAR User Q&A and Other Discussion

[Olav Kvittem <>]

Den 05.09.2024 17:44, skrev Tim Chown (via perfsonar-user Mailing List):

Hi,

 

It seems for us that the bulk of the data is host information data rather than measurement data.  Very useful to have, but not needed for longer-term reflection.

 

Is it possible to have varied preservation settings, e.g., the host information kept for say 1-2 weeks and the measurement data for 6-12 months?

Not to the concrete question, but ES is generous on memory and disk space. (Each record has even a copy of itself : the src-attribute.)

In microdep we have implemented a temporal topology database that keeps information about previous configs, 

so that presenting historic data is presented on a historic topology. 

So be careful not to throw away info about probing topology ?

It could be a good idea to implement an aggregation function that could make summary records for time periods like days, weeks, months and years and then expire the more detailed records as needed (RRD-like but more flexible).

cheers

 Olav

 

Tim

 

From: on behalf of "Yamamoto, Miguel"
Date: Wednesday, 4 September 2024 at 20:42
To: Mark Feit ,
Subject: [perfsonar-user] Opensearch taking a lot of space in /var

Hi Mark,

I am not sure if this is an ongoing issue, but Opensearch is continuously taking up a lot of space in /var. We could expand the drive, but I wanted to ask if there is a better way to handle the increasing disk utilization of OpenSearch.  Our aim is to not always increase the drive as a workaround.

Current version: 5.1.2
The system has been up for about 2-3 months now

This is from the central archive machine and reading data coming from multiple testpoints (about 10+)

36G       /var/lib/opensearch/nodes/0/indices/ZyA-IwTlQXGS2Z8gRKslaw/0/index

37G       /var/lib/opensearch/nodes/0/indices/ZyA-IwTlQXGS2Z8gRKslaw/0

37G       /var/lib/opensearch/nodes/0/indices/ZyA-IwTlQXGS2Z8gRKslaw

3.3G      /var/lib/opensearch/nodes/0/indices/BbSkqaH9QVy0AlbUL4VG-g/0/index

3.3G      /var/lib/opensearch/nodes/0/indices/BbSkqaH9QVy0AlbUL4VG-g/0

3.3G      /var/lib/opensearch/nodes/0/indices/BbSkqaH9QVy0AlbUL4VG-g

1.8G      /var/lib/opensearch/nodes/0/indices/u9-EbLT9R6Op5EZX0DVrcg/0/index

1.8G      /var/lib/opensearch/nodes/0/indices/u9-EbLT9R6Op5EZX0DVrcg/0

1.8G      /var/lib/opensearch/nodes/0/indices/u9-EbLT9R6Op5EZX0DVrcg

43G       /var/lib/opensearch/nodes/0/indices

43G       /var/lib/opensearch/nodes/0

43G       /var/lib/opensearch/nodes

43G       /var/lib/opensearch

44G       /var/lib

1.1G      /var/log/opensearch

1.2G      /var/log

45G       /var/

Thanks,
Miguel

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user