perfsonar-user - [perfsonar-user] Observation of resource utilization of perfSONAR version 5
Subject: perfSONAR User Q&A and Other Discussion
List archive
- From: Syed Asif Shah <>
- To: Wilson Cheung <>
- Subject: [perfsonar-user] Observation of resource utilization of perfSONAR version 5
- Date: Thu, 17 Aug 2023 15:03:43 +0000
- Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fnal.gov; dmarc=pass action=none header.from=fnal.gov; dkim=pass header.d=fnal.gov; arc=none
- Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dG//V0ApDk1VfUYQR7IveqIRf57dvS4pt4uX5Cf6KSY=; b=chk4az1ReUTuF7HsUCQXLX4iqKB0AZdL8m+s27TW6OtDjiuYDB3RWOxIEl9OyY3C3uBBleEog2B2GPOoW+JioTTO/F6IcmiiNxYET1FqJTIHABg0qG0ddue+Wdczn/4Wb1EjjTgSI9DH5sMoQIM89p2NPmokKt8oqjvfudaMdiTzajZtCCK9H3dmQk6GTVP+5p59fwvVEVlWelp6tMtfggFw4QmycKT3UrvI5X7vlnXkw4O0g3f/3jnzi12Bwjpou7ECQdpJY/mWtETWu6Xr5v6vkNBj/NQD4G6zWrWc24ThLMmG69w+HJ5hfwaUXUPpbMHAYXrQK0fwXrNMKEhOew==
- Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UcH1uFWSrHZHtcxHGetOWtJmj9/T9rsIVLGSfFZp7/UBZe7tnlP8hDg1c4ywjPQV99M45xQegRzdnttV+7jCxlTRAfN7sURVlWPpuyJOkr0HiqviCsT+6n44kD2PT0c8VaYEaOYZa72O8BrK/FsloBi+FXCjeG/4I8urlhfPCUzdbLaAZVon3Mg7oVkWJ870T7gN0+KwaPO3K09YOT/H4l7kyGdKxxX+2QL2PmIJA+e7UqUotDb6tuO7r2TXA7hKPTpxgS4n8tGzGxP9uunz2IWw+6LZaADEIE/hOtXAmv6k+puWEJcjJ13/RkZV2owOwvHMNITDFQZumlMrAtfe0Q==
- Msip_labels:
Hi,
I would like to share some of my observations and experience with community and developers about the latest perfSONAR version (5.0.4).
- Resources utilization by new version 5.0.4:
- It has been observed that the new version of perfsonar is utilizing huge RAM/Virtual Memory resources on server. If your available free primary memory is not enough (~10G) then you cannot start OpenSearch or pscheduler services. It also utilizing more than 18G virtual memory.
- - perfsonar services are stressing all server's core to 100% time by time and having huge utilizations cpu resources.
- No enough Memory on PS Nodes & Assignment of max/min memory to JVM for OpenSearch:
- It is recommended by perfsonar community that we should have to assign 50% of total memory in JVM configuration to smoothly start OpenSearch service. But, if your PS nodes have less memory available and other processes are already occupying more than 50% memory than this trick will not work and we cannot start OpenSearch service. What we have to do first check how much remaining memory is available then assign max/min memory to OpenSearch JVM configurations. Keep in mind that if your server's available memory is more than 10~12G then you can start OpenSearch service successfully.
Let me explain my real experience, We upgraded our PS server with latest version and it has total 12GB RAM. I was unable to start OpenSearch service even I assigned 50% memory in JVM configuration (/etc/opensearch/jvm.options). The reason I found that my other processes are already occupying ~10G out of 12G RAM on my server. Similarly, we have another PS server but it is with 32G RAM, and when I configured JVM with 10G min/max then I was able to successfully start my all services on server.
Conclusion: I observed that current version of perfsonar is utilizing huge hardware resources on our servers and we need to be carefully configure the PS nodes specially if our current system specs are not enough. It might be possible that as developers
point of view the current resources utilization is normal, but if it is not then I hope they will also take a look into the higher utilization of resources issues.
Thank you
--
Asif
Networking R&D,
FNAL
- [perfsonar-user] Observation of resource utilization of perfSONAR version 5, Syed Asif Shah, 08/17/2023
Archive powered by MHonArc 2.6.24.