perfsonar-user - Re: [perfsonar-user] opensearch issue

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] opensearch issue

From: Eric Simich <>
To: Andrew Lake <>
Cc:
Subject: Re: [perfsonar-user] opensearch issue
Date: Mon, 7 Aug 2023 12:54:53 -0700

Andrew,

Thanks for your prompt response, I appreciate it!

When you run “systemctl restart opensearch” does it start?

No.

Assuming it doesn’t, anything of use in journalctl -xe related to opensearch?

Yes.

root@hq-pson01:~# journalctl -xe --no-pager | grep -iE "search"
Aug 07 19:31:22 hq-pson01 logstash[576157]: [2023-08-07T19:31:22,773][ERROR][logstash.agent ] Failed to execute action {:action=">LogStash::PipelineAction::Create/pipeline_id:pscheduler," :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: Cannot evaluate `${opensearch_output_host}`. Replacement variable `opensearch_output_host` is not defined in a Logstash secret store or an environment entry and there is no default value given.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:392)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}

How about var/log/opensearch/opensearch.log?

Attached to email.

The /etc/opensearch.yml

root@hq-pson01:/etc/opensearch# less opensearch.yml | grep -v "#"
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3

plugins.security.ssl.transport.pemcert_filepath: node.pem
plugins.security.ssl.transport.pemkey_filepath: node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node.pem
plugins.security.ssl.http.pemkey_filepath: node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_default_init_securityindex: false
plugins.security.authcz.admin_dn:
- CN=admin,OU=Archive,O=perfSONAR
plugins.security.nodes_dn:
- CN=localhost,OU=Archive,O=perfSONAR
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.allow_unsafe_democertificates: false
plugins.security.ssl.transport.enforce_hostname_verification: false
discovery.type: single-node

psconfig configuration
root@hq-pson01:/etc/psconfig# psconfig published
[skeleton.json]
Local File: /usr/lib/perfsonar/web-psconfig/skeleton.json
Web URL: https://localhost/psconfig/skeleton.json

attached skeleton.json

On Mon, Aug 7, 2023 at 11:47 AM Andrew Lake <> wrote:

Hi Eric,

When you run “systemctl restart opensearch” does it start? Assuming it doesn’t, anything of use in journalctl -xe related to opensearch? How about var/log/opensearch/opensearch.log?

Thanks,
Andy

On August 7, 2023 at 12:08:32 PM, Eric Simich () wrote:

Support,

I'm running into an issue after installing the Perfsonar toolkit. I installed the toolkit per the bundle install instructions. Wherein, the opensearch service fails to load. But the toolkit webUI indicates the service is running. The toolkit webUI indicates pscheduler is not running, but systemctl shows the service is running. See attached screenshot. Maddash is also running, but I cannot access the webUI - 404 not found.

I'd like to get this perfsonar toolkit node running, but it has not been able to operate as a turn-key solution after installing the packages and following the guide.

pson01:~$ sudo systemctl list-units --state failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● opensearch.service loaded failed failed OpenSearch

● opensearch.service - OpenSearch
Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
Active: failed (Result: timeout) since Sun 2023-07-23 07:32:46 UTC; 2 weeks 1 days ago
Docs: https://opensearch.org/
Main PID: 825 (code=killed, signal=KILL)

pscheduler-scheduler.service - pScheduler server - scheduler
Loaded: loaded (/lib/systemd/system/pscheduler-scheduler.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-08-07 15:00:07 UTC; 27min ago
Process: 516494 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/scheduler (code=exited, status=0/SUCCESS)
Process: 516505 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/scheduler (code=exited, status=0/SUCCESS)
Process: 516507 ExecStartPre=/bin/chmod 755 /var/run/pscheduler-server/scheduler (code=exited, status=0/SUCCESS)
Process: 516508 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/scheduler/tmp (code=exited, status=0/SUCCESS)
Process: 516509 ExecStartPre=/bin/chmod 700 /var/run/pscheduler-server/scheduler/tmp (code=exited, status=0/SUCCESS)
Process: 516510 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/scheduler/tmp (code=exited, status=0/SUCCESS)
Process: 516511 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/scheduler.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/scheduler.conf); echo OPTIONS=$opts > /var/run/pscheduler->
Main PID: 516519 (python3)
Tasks: 1 (limit: 4662)
Memory: 21.3M
CGroup: /system.slice/pscheduler-scheduler.service
└─516519 python3 /usr/lib/pscheduler/daemons/scheduler --dsn @/etc/pscheduler/database/database-dsn --pid-file /var/run/pscheduler-server/scheduler/pid

Aug 07 15:00:07 hq-pson01 systemd[1]: Starting pScheduler server - scheduler...
Aug 07 15:00:07 hq-pson01 systemd[1]: Started pScheduler server - scheduler.
Aug 07 15:00:07 hq-pson01 python3[516519]: scheduler INFO Started

Jul 23 07:31:06 hq-pson01 systemd-entrypoint[825]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Jul 23 07:31:06 hq-pson01 systemd-entrypoint[825]: WARNING: System::setSecurityManager will be removed in a future release
Jul 23 07:31:13 hq-pson01 systemd-entrypoint[825]: WARNING: A terminally deprecated method in java.lang.System has been called
Jul 23 07:31:13 hq-pson01 systemd-entrypoint[825]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.6.0.jar)
Jul 23 07:31:13 hq-pson01 systemd-entrypoint[825]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Jul 23 07:31:13 hq-pson01 systemd-entrypoint[825]: WARNING: System::setSecurityManager will be removed in a future release
Jul 23 07:32:00 hq-pson01 systemd[1]: opensearch.service: start operation timed out. Terminating.
Jul 23 07:32:46 hq-pson01 systemd[1]: opensearch.service: Main process exited, code=killed, status=9/KILL
Jul 23 07:32:46 hq-pson01 systemd[1]: opensearch.service: Failed with result 'timeout'.
Jul 23 07:32:46 hq-pson01 systemd[1]: Failed to start OpenSearch.

maddash-server.service - MaDDash Scheduler and REST Server
Loaded: loaded (/lib/systemd/system/maddash-server.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2023-07-23 07:30:45 UTC; 2 weeks 1 days ago
Main PID: 926 (java)
Tasks: 57 (limit: 4662)
Memory: 280.1M
CGroup: /system.slice/maddash-server.service
└─926 java -jar /usr/share/maddash/maddash-server/target/maddash-server.one-jar.jar -c /etc/maddash/maddash-server/maddash.yaml -l /etc/maddash/maddash-server/log4j.properties

Jul 23 07:30:45 hq-pson01 systemd[1]: Starting MaDDash Scheduler and REST Server...
Jul 23 07:30:45 hq-pson01 systemd[1]: Started MaDDash Scheduler and REST Server.
root@hq-pson01:~#

apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2023-07-23 07:30:46 UTC; 2 weeks 1 days ago
Docs: https://httpd.apache.org/docs/2.4/
Process: 324347 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
Main PID: 1046 (apache2)
Tasks: 246 (limit: 4662)
Memory: 382.3M
CGroup: /system.slice/apache2.service
├─ 1046 /usr/sbin/apache2 -k start
├─324369 /usr/sbin/apache2 -k start
├─324370 pscheduler-api -k start
├─324371 pscheduler-api -k start
├─324372 pscheduler-api -k start
├─324373 pscheduler-api -k start
├─324374 pscheduler-api -k start
├─324375 pscheduler-api -k start
├─324376 pscheduler-api -k start
├─324377 pscheduler-api -k start
├─324378 pscheduler-api -k start
├─324379 pscheduler-api -k start
├─324380 /usr/sbin/apache2 -k start
└─324381 /usr/sbin/apache2 -k start

Aug 07 00:13:10 hq-pson01 apache2[324375]: pscheduler-api[324375] INFO Started
Aug 07 00:13:10 hq-pson01 apache2[324375]: pscheduler-api[324375] INFO Limits loaded from /etc/pscheduler/limits.conf
Aug 07 00:13:11 hq-pson01 apache2[324376]: pscheduler-api[324376] INFO Started
Aug 07 00:13:11 hq-pson01 apache2[324376]: pscheduler-api[324376] INFO Limits loaded from /etc/pscheduler/limits.conf
Aug 07 00:23:09 hq-pson01 apache2[324370]: pscheduler-api[324370] INFO Started
Aug 07 00:23:10 hq-pson01 apache2[324370]: pscheduler-api[324370] INFO Limits loaded from /etc/pscheduler/limits.conf
Aug 07 00:23:10 hq-pson01 apache2[324372]: pscheduler-api[324372] INFO Started
Aug 07 00:23:11 hq-pson01 apache2[324372]: pscheduler-api[324372] INFO Limits loaded from /etc/pscheduler/limits.conf
Aug 07 00:23:11 hq-pson01 apache2[324379]: pscheduler-api[324379] INFO Started
Aug 07 00:23:11 hq-pson01 apache2[324379]: pscheduler-api[324379] INFO Limits loaded from /etc/pscheduler/limits.conf

Sys info:
hq-pson01:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Sincerely,
Eric Simich
--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user

[2023-07-23T06:51:23,977][INFO ][o.o.n.Node               ] [hq-pson01] version[2.6.0], pid[814], build[deb/7203a5af21a8a009aece1474446b437a3c674db6/2023-02-24T18:58:38.730482464Z], OS[Linux/5.4.0-153-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.6/17.0.6+10]
[2023-07-23T06:51:24,096][INFO ][o.o.n.Node               ] [hq-pson01] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2023-07-23T06:51:24,096][INFO ][o.o.n.Node               ] [hq-pson01] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms2g, -Xmx2g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-8551770097472595461, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/opensearch, -XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Djna.tmpdir=/var/lib/opensearch/tmp, -XX:MaxDirectMemorySize=1073741824, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/etc/opensearch, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]
[2023-07-23T06:51:27,779][WARN ][stderr                   ] [hq-pson01] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2023-07-23T06:51:27,779][WARN ][stderr                   ] [hq-pson01] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2023-07-23T06:51:27,783][WARN ][stderr                   ] [hq-pson01] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2023-07-23T06:51:27,840][INFO ][o.o.s.s.t.SSLConfig      ] [hq-pson01] SSL dual mode is disabled
[2023-07-23T06:51:27,840][INFO ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] OpenSearch Config path is /etc/opensearch
[2023-07-23T06:51:29,076][INFO ][o.o.s.s.DefaultSecurityKeyStore] [hq-pson01] JVM supports TLSv1.3
[2023-07-23T06:51:29,081][INFO ][o.o.s.s.DefaultSecurityKeyStore] [hq-pson01] Config directory is /etc/opensearch/, from there the key- and truststore files are resolved relatively
[2023-07-23T06:51:30,800][INFO ][o.o.s.s.DefaultSecurityKeyStore] [hq-pson01] TLS Transport Client Provider : JDK
[2023-07-23T06:51:30,801][INFO ][o.o.s.s.DefaultSecurityKeyStore] [hq-pson01] TLS Transport Server Provider : JDK
[2023-07-23T06:51:30,802][INFO ][o.o.s.s.DefaultSecurityKeyStore] [hq-pson01] TLS HTTP Provider             : JDK
[2023-07-23T06:51:30,803][INFO ][o.o.s.s.DefaultSecurityKeyStore] [hq-pson01] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2023-07-23T06:51:30,804][INFO ][o.o.s.s.DefaultSecurityKeyStore] [hq-pson01] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
[2023-07-23T06:51:30,831][INFO ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] Clustername: opensearch
[2023-07-23T06:51:30,852][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] Directory /etc/opensearch has insecure file permissions (should be 0700)
[2023-07-23T06:51:30,853][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] Directory /etc/opensearch/jvm.options.d has insecure file permissions (should be 0700)
[2023-07-23T06:51:30,857][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] Directory /etc/opensearch/opensearch-performance-analyzer has insecure file permissions (should be 0700)
[2023-07-23T06:51:30,858][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-performance-analyzer/performance-analyzer.properties has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,858][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-performance-analyzer/supervisord.conf has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,859][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-performance-analyzer/rca.conf has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,859][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,859][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-performance-analyzer/rca_idle_cluster_manager.conf has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,860][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-performance-analyzer/log4j2.xml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,860][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-performance-analyzer/rca_cluster_manager.conf has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,861][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-performance-analyzer/plugin-stats-metadata has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,861][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-performance-analyzer/agent-stats-metadata has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,861][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] Directory /etc/opensearch/opensearch-security has insecure file permissions (should be 0700)
[2023-07-23T06:51:30,862][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-security/opensearch.yml.example has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,862][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-security/roles.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,862][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-security/audit.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,866][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-security/config.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,866][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-security/tenants.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,867][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-security/nodes_dn.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,867][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-security/roles_mapping.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,867][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-security/action_groups.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,867][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-security/allowlist.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,868][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-security/whitelist.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,873][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-security/internal_users.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,874][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] Directory /etc/opensearch/opensearch-reports-scheduler has insecure file permissions (should be 0700)
[2023-07-23T06:51:30,874][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-reports-scheduler/reports-scheduler.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,876][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] Directory /etc/opensearch/opensearch-observability has insecure file permissions (should be 0700)
[2023-07-23T06:51:30,877][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-observability/observability.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,877][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/node.pem has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,877][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/root-ca.pem has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,878][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] Directory /etc/opensearch/opensearch-notifications-core has insecure file permissions (should be 0700)
[2023-07-23T06:51:30,878][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-notifications-core/notifications-core.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,878][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/root-ca.srl has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,879][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/root-ca.der has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,879][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/admin.pem has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,882][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/log4j2.properties has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,882][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] Directory /etc/opensearch/opensearch-notifications has insecure file permissions (should be 0700)
[2023-07-23T06:51:30,883][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch-notifications/notifications.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,890][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/jvm.options has insecure file permissions (should be 0600)
[2023-07-23T06:51:30,891][WARN ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] File /etc/opensearch/opensearch.yml has insecure file permissions (should be 0600)
[2023-07-23T06:51:33,998][INFO ][o.o.p.c.PluginSettings   ] [hq-pson01] Trying to create directory /dev/shm/performanceanalyzer/.
[2023-07-23T06:51:33,998][INFO ][o.o.p.c.PluginSettings   ] [hq-pson01] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2023-07-23T06:51:35,933][INFO ][o.o.i.r.ReindexPlugin    ] [hq-pson01] ReindexPlugin reloadSPI called
[2023-07-23T06:51:35,935][INFO ][o.o.i.r.ReindexPlugin    ] [hq-pson01] Unable to find any implementation for RemoteReindexExtension
[2023-07-23T06:51:36,077][INFO ][o.o.j.JobSchedulerPlugin ] [hq-pson01] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2023-07-23T06:51:36,194][INFO ][o.o.j.JobSchedulerPlugin ] [hq-pson01] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-07-23T06:51:36,195][INFO ][o.o.j.JobSchedulerPlugin ] [hq-pson01] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-07-23T06:51:36,320][INFO ][o.o.j.JobSchedulerPlugin ] [hq-pson01] Loaded scheduler extension: observability, index: .opensearch-observability-job
[2023-07-23T06:51:36,341][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [aggs-matrix-stats]
[2023-07-23T06:51:36,341][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [analysis-common]
[2023-07-23T06:51:36,342][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [geo]
[2023-07-23T06:51:36,342][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [ingest-common]
[2023-07-23T06:51:36,342][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [ingest-geoip]
[2023-07-23T06:51:36,342][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [ingest-user-agent]
[2023-07-23T06:51:36,342][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [lang-expression]
[2023-07-23T06:51:36,343][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [lang-mustache]
[2023-07-23T06:51:36,343][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [lang-painless]
[2023-07-23T06:51:36,343][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [mapper-extras]
[2023-07-23T06:51:36,343][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [opensearch-dashboards]
[2023-07-23T06:51:36,343][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [parent-join]
[2023-07-23T06:51:36,343][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [percolator]
[2023-07-23T06:51:36,344][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [rank-eval]
[2023-07-23T06:51:36,344][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [reindex]
[2023-07-23T06:51:36,344][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [repository-url]
[2023-07-23T06:51:36,344][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [systemd]
[2023-07-23T06:51:36,344][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded module [transport-netty4]
[2023-07-23T06:51:36,345][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-alerting]
[2023-07-23T06:51:36,345][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-anomaly-detection]
[2023-07-23T06:51:36,345][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-asynchronous-search]
[2023-07-23T06:51:36,345][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-cross-cluster-replication]
[2023-07-23T06:51:36,345][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-geospatial]
[2023-07-23T06:51:36,346][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-index-management]
[2023-07-23T06:51:36,346][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-job-scheduler]
[2023-07-23T06:51:36,346][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-knn]
[2023-07-23T06:51:36,346][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-ml]
[2023-07-23T06:51:36,346][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-neural-search]
[2023-07-23T06:51:36,346][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-notifications]
[2023-07-23T06:51:36,347][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-notifications-core]
[2023-07-23T06:51:36,347][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-observability]
[2023-07-23T06:51:36,347][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-performance-analyzer]
[2023-07-23T06:51:36,347][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-reports-scheduler]
[2023-07-23T06:51:36,347][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-security]
[2023-07-23T06:51:36,348][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-security-analytics]
[2023-07-23T06:51:36,348][INFO ][o.o.p.PluginsService     ] [hq-pson01] loaded plugin [opensearch-sql]
[2023-07-23T06:51:36,452][INFO ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2023-07-23T06:51:36,560][INFO ][o.o.e.NodeEnvironment    ] [hq-pson01] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [91.4gb], net total_space [96.7gb], types [ext4]
[2023-07-23T06:51:36,560][INFO ][o.o.e.NodeEnvironment    ] [hq-pson01] heap size [2gb], compressed ordinary object pointers [true]
[2023-07-23T06:51:37,070][INFO ][o.o.n.Node               ] [hq-pson01] node name [hq-pson01], node ID [AW9MvwLURSSNWbcAtN4VOw], cluster name [opensearch], roles [ingest, remote_cluster_client, data, cluster_manager]
[2023-07-23T06:51:46,947][WARN ][o.o.s.c.Salt             ] [hq-pson01] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2023-07-23T06:51:47,092][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Message routing enabled: true
[2023-07-23T06:51:47,198][INFO ][o.o.s.f.SecurityFilter   ] [hq-pson01] <NONE> indices are made immutable.
[2023-07-23T06:51:48,159][INFO ][o.o.a.b.ADCircuitBreakerService] [hq-pson01] Registered memory breaker.
[2023-07-23T06:51:49,554][INFO ][o.o.m.b.MLCircuitBreakerService] [hq-pson01] Registered ML memory breaker.
[2023-07-23T06:51:49,554][INFO ][o.o.m.b.MLCircuitBreakerService] [hq-pson01] Registered ML disk breaker.
[2023-07-23T06:51:49,555][INFO ][o.o.m.b.MLCircuitBreakerService] [hq-pson01] Registered ML native memory breaker.
[2023-07-23T06:51:49,901][INFO ][o.r.Reflections          ] [hq-pson01] Reflections took 144 ms to scan 1 urls, producing 13 keys and 33 values 
[2023-07-23T06:51:52,625][INFO ][o.o.t.NettyAllocator     ] [hq-pson01] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=256kb, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=1mb}]
[2023-07-23T06:51:52,784][INFO ][o.o.d.DiscoveryModule    ] [hq-pson01] using discovery type [single-node] and seed hosts providers [settings]
[2023-07-23T06:51:54,090][WARN ][o.o.g.DanglingIndicesState] [hq-pson01] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-07-23T06:51:55,164][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [hq-pson01] PerformanceAnalyzer Enabled: true
[2023-07-23T06:51:55,212][INFO ][o.o.n.Node               ] [hq-pson01] initialized
[2023-07-23T06:51:55,212][INFO ][o.o.n.Node               ] [hq-pson01] starting ...
[2023-07-23T06:51:55,739][INFO ][o.o.t.TransportService   ] [hq-pson01] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2023-07-23T06:51:56,444][INFO ][o.o.c.c.Coordinator      ] [hq-pson01] cluster UUID [7yY80OU7RtKoyJKJfLzKZA]
[2023-07-23T06:51:56,722][INFO ][o.o.c.s.MasterService    ] [hq-pson01] elected-as-cluster-manager ([1] nodes joined)[{hq-pson01}{AW9MvwLURSSNWbcAtN4VOw}{RwctLc2hQb6KXG_JTpuQvQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 4, version: 41, delta: cluster-manager node changed {previous [], current [{hq-pson01}{AW9MvwLURSSNWbcAtN4VOw}{RwctLc2hQb6KXG_JTpuQvQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
[2023-07-23T06:51:56,899][INFO ][o.o.c.s.ClusterApplierService] [hq-pson01] cluster-manager node changed {previous [], current [{hq-pson01}{AW9MvwLURSSNWbcAtN4VOw}{RwctLc2hQb6KXG_JTpuQvQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 4, version: 41, reason: Publication{term=4, version=41}
[2023-07-23T06:51:56,917][INFO ][o.o.a.c.ADClusterEventListener] [hq-pson01] Cluster is not recovered yet.
[2023-07-23T06:51:56,953][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [hq-pson01] Detected cluster change event for destination migration
[2023-07-23T06:51:57,049][INFO ][o.o.i.i.ManagedIndexCoordinator] [hq-pson01] Cache cluster manager node onClusterManager time: 1690095117048
[2023-07-23T06:51:57,093][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [hq-pson01] Config override setting update called with empty string. Ignoring.
[2023-07-23T06:51:57,118][INFO ][o.o.d.PeerFinder         ] [hq-pson01] setting findPeersInterval to [1s] as node commission status = [true] for local node [{hq-pson01}{AW9MvwLURSSNWbcAtN4VOw}{RwctLc2hQb6KXG_JTpuQvQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]
[2023-07-23T06:51:57,215][INFO ][o.o.h.AbstractHttpServerTransport] [hq-pson01] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2023-07-23T06:51:57,215][INFO ][o.o.n.Node               ] [hq-pson01] started
[2023-07-23T06:51:57,260][INFO ][o.o.c.s.ClusterSettings  ] [hq-pson01] updating [plugins.index_state_management.metadata_migration.status] from [0] to [1]
[2023-07-23T06:51:57,260][INFO ][o.o.i.i.ManagedIndexCoordinator] [hq-pson01] Canceling metadata moving job because of cluster setting update.
[2023-07-23T06:51:57,260][INFO ][o.o.c.s.ClusterSettings  ] [hq-pson01] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2023-07-23T06:51:57,280][INFO ][o.o.a.c.HashRing         ] [hq-pson01] Node added: [AW9MvwLURSSNWbcAtN4VOw]
[2023-07-23T06:51:57,294][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [hq-pson01] Detected cluster change event for destination migration
[2023-07-23T06:51:57,301][INFO ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] Node started
[2023-07-23T06:51:57,301][INFO ][o.o.a.c.HashRing         ] [hq-pson01] Add data node to AD version hash ring: AW9MvwLURSSNWbcAtN4VOw
[2023-07-23T06:51:57,302][INFO ][o.o.s.c.ConfigurationRepository] [hq-pson01] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2023-07-23T06:51:57,305][INFO ][o.o.a.c.HashRing         ] [hq-pson01] All nodes with known AD version: {AW9MvwLURSSNWbcAtN4VOw=ADNodeInfo{version=2.6.0, isEligibleDataNode=true}}
[2023-07-23T06:51:57,305][INFO ][o.o.a.c.HashRing         ] [hq-pson01] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2023-07-23T06:51:57,305][INFO ][o.o.a.c.HashRing         ] [hq-pson01] Build AD version hash ring successfully
[2023-07-23T06:51:57,315][INFO ][o.o.a.c.ADDataMigrator   ] [hq-pson01] Start migrating AD data
[2023-07-23T06:51:57,316][INFO ][o.o.a.c.ADDataMigrator   ] [hq-pson01] AD job index doesn't exist, no need to migrate
[2023-07-23T06:51:57,317][INFO ][o.o.a.c.ADClusterEventListener] [hq-pson01] Init AD version hash ring successfully
[2023-07-23T06:51:57,323][INFO ][o.o.s.c.ConfigurationRepository] [hq-pson01] Background init thread started. Install default config?: false
[2023-07-23T06:51:57,338][INFO ][o.o.s.OpenSearchSecurityPlugin] [hq-pson01] 0 OpenSearch Security modules loaded so far: []
[2023-07-23T06:51:57,436][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [hq-pson01] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6e237a23] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-07-23T06:51:57,439][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [hq-pson01] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6e237a23] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-07-23T06:51:57,439][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [hq-pson01] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6e237a23] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-07-23T06:51:57,439][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [hq-pson01] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6e237a23] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-07-23T06:51:57,439][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [hq-pson01] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6e237a23] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-07-23T06:51:57,447][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [hq-pson01] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6e237a23] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-07-23T06:51:57,448][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [hq-pson01] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6e237a23] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-07-23T06:51:57,448][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [hq-pson01] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6e237a23] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-07-23T06:51:57,448][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [hq-pson01] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6e237a23] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-07-23T06:51:57,448][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [hq-pson01] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@6e237a23] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-07-23T06:51:57,529][INFO ][o.o.g.GatewayService     ] [hq-pson01] recovered [4] indices into cluster_state
[2023-07-23T06:51:57,860][ERROR][o.o.i.i.ManagedIndexCoordinator] [hq-pson01] Failed to get ISM policies with templates: Failed to execute phase [query], all shards failed
[2023-07-23T06:51:58,067][INFO ][o.o.p.PluginsService     ] [hq-pson01] PluginService:onIndexModule index:[.opensearch-observability/Y7vdVdG4Q1KjInbOXQ-8LA]
[2023-07-23T06:51:58,678][INFO ][o.o.p.PluginsService     ] [hq-pson01] PluginService:onIndexModule index:[.opendistro_security/KkaJmVYbTUybCusGdypaHg]
[2023-07-23T06:51:58,712][INFO ][o.o.p.PluginsService     ] [hq-pson01] PluginService:onIndexModule index:[security-auditlog-2023.07.20/dd23XOniSeO8laNjMqLubA]
[2023-07-23T06:51:58,766][INFO ][o.o.p.PluginsService     ] [hq-pson01] PluginService:onIndexModule index:[.opendistro-ism-config/SwHqIDGFReaZRfrcuimttw]
[2023-07-23T06:51:58,777][INFO ][o.o.j.JobSchedulerPlugin ] [hq-pson01] JobSweeper started listening to operations on index .opendistro-ism-config
[2023-07-23T06:51:59,000][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [hq-pson01] Detected cluster change event for destination migration
[2023-07-23T06:51:59,021][INFO ][o.o.p.PluginsService     ] [hq-pson01] PluginService:onIndexModule index:[.opensearch-observability/Y7vdVdG4Q1KjInbOXQ-8LA]
[2023-07-23T06:51:59,197][INFO ][o.o.o.i.ObservabilityIndex] [hq-pson01] observability:Index .opensearch-observability update mapping Acknowledged
[2023-07-23T06:51:59,199][INFO ][o.o.o.i.ObservabilityTracesIndex] [hq-pson01] observability:createMappingTemplate sso_trace_template API called
[2023-07-23T06:51:59,286][INFO ][o.o.p.PluginsService     ] [hq-pson01] PluginService:onIndexModule index:[validate-template-mfud_ggotji-ggnmz1iqjq/UEMJacmySo2IimT1DJDaeA]
[2023-07-23T06:51:59,429][INFO ][o.o.c.m.MetadataIndexTemplateService] [hq-pson01] updating index template [sso_trace_template] for index patterns [sso_traces-*-*]
[2023-07-23T06:51:59,597][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [hq-pson01] Detected cluster change event for destination migration
[2023-07-23T06:51:59,607][INFO ][o.o.o.i.ObservabilityTracesIndex] [hq-pson01] observability:Mapping Template sso_trace_template creation Acknowledged
[2023-07-23T06:51:59,608][INFO ][o.o.o.i.ObservabilityMetricsIndex] [hq-pson01] observability:createMappingTemplate sso_metric_template API called
[2023-07-23T06:51:59,609][INFO ][o.o.c.r.a.AllocationService] [hq-pson01] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.opensearch-observability][0], [.opendistro_security][0], [security-auditlog-2023.07.20][0], [.opendistro-ism-config][0]]]).
[2023-07-23T06:51:59,714][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [hq-pson01] Detected cluster change event for destination migration
[2023-07-23T06:51:59,727][INFO ][o.o.p.PluginsService     ] [hq-pson01] PluginService:onIndexModule index:[validate-template-wwlj-vdwt_qorxbok2ej-g/OcaWWS7fRsy8uw9mgq3uHA]
[2023-07-23T06:51:59,795][INFO ][o.o.c.m.MetadataIndexTemplateService] [hq-pson01] updating index template [sso_metric_template] for index patterns [sso_metrics-*-*]
[2023-07-23T06:51:59,842][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [hq-pson01] Detected cluster change event for destination migration
[2023-07-23T06:51:59,851][INFO ][o.o.o.i.ObservabilityMetricsIndex] [hq-pson01] observability:Mapping Template sso_metric_template creation Acknowledged
[2023-07-23T06:52:14,338][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing on REST API is enabled.
[2023-07-23T06:52:14,372][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2023-07-23T06:52:14,374][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing on Transport API is enabled.
[2023-07-23T06:52:14,374][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2023-07-23T06:52:14,374][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing of request body is enabled.
[2023-07-23T06:52:14,374][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Bulk requests resolution is disabled during request auditing.
[2023-07-23T06:52:14,374][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Index resolution is enabled during request auditing.
[2023-07-23T06:52:14,375][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Sensitive headers auditing is enabled.
[2023-07-23T06:52:14,375][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing requests from kibanaserver users is disabled.
[2023-07-23T06:52:14,639][WARN ][o.o.s.a.r.AuditMessageRouter] [hq-pson01] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2023-07-23T06:52:14,657][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing of external configuration is disabled.
[2023-07-23T06:52:14,657][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing of internal configuration is enabled.
[2023-07-23T06:52:14,657][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing only metadata information for read request is enabled.
[2023-07-23T06:52:14,657][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing will watch {} for read requests.
[2023-07-23T06:52:14,657][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing read operation requests from kibanaserver users is disabled.
[2023-07-23T06:52:14,661][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing only metadata information for write request is enabled.
[2023-07-23T06:52:14,663][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing diffs for write requests is disabled.
[2023-07-23T06:52:14,663][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing write operation requests from kibanaserver users is disabled.
[2023-07-23T06:52:14,663][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Auditing will watch <NONE> for write requests.
[2023-07-23T06:52:14,672][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] .opendistro_security is used as internal security index.
[2023-07-23T06:52:14,680][INFO ][o.o.s.a.i.AuditLogImpl   ] [hq-pson01] Internal index used for posting audit logs is null
[2023-07-23T06:52:14,756][INFO ][o.o.s.c.ConfigurationRepository] [hq-pson01] Hot-reloading of audit configuration is enabled
[2023-07-23T06:52:14,772][INFO ][o.o.s.c.ConfigurationRepository] [hq-pson01] Node 'hq-pson01' initialized
[2023-07-23T06:53:02,799][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector OSMetrics is still in progress, so skipping this Interval
[2023-07-23T06:53:09,501][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector ThreadPoolMetrics is still in progress, so skipping this Interval
[2023-07-23T06:53:11,318][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector OSMetrics is still in progress, so skipping this Interval
[2023-07-23T06:53:12,437][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector NodeStatsMetrics is still in progress, so skipping this Interval
[2023-07-23T06:53:12,457][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector NetworkInterfaceCollector is still in progress, so skipping this Interval
[2023-07-23T06:53:12,457][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector CacheConfigMetrics is still in progress, so skipping this Interval
[2023-07-23T06:53:12,458][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector NodeDetails is still in progress, so skipping this Interval
[2023-07-23T06:53:12,458][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector ShardIndexingPressureMetricsCollector is still in progress, so skipping this Interval
[2023-07-23T06:53:12,458][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector ShardsStateCollector is still in progress, so skipping this Interval
[2023-07-23T06:53:12,502][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector AdmissionControlMetricsCollector is still in progress, so skipping this Interval
[2023-07-23T06:53:16,800][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector ClusterManagerServiceEventMetrics is still in progress, so skipping this Interval
[2023-07-23T06:53:19,779][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector ThreadPoolMetrics is still in progress, so skipping this Interval
[2023-07-23T06:53:21,773][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector ClusterManagerServiceEventMetrics is still in progress, so skipping this Interval
[2023-07-23T06:53:25,527][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector NodeDetails is still in progress, so skipping this Interval
[2023-07-23T06:53:29,406][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector CacheConfigMetrics is still in progress, so skipping this Interval
[2023-07-23T06:53:30,819][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector AdmissionControlMetricsCollector is still in progress, so skipping this Interval
[2023-07-23T06:54:04,707][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector NodeStatsMetrics is still in progress, so skipping this Interval
[2023-07-23T06:54:09,992][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector NetworkInterfaceCollector is still in progress, so skipping this Interval
[2023-07-23T06:54:17,351][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector DisksCollector is still in progress, so skipping this Interval
[2023-07-23T06:54:26,790][INFO ][o.o.p.c.ScheduledMetricCollectorsExecutor] [hq-pson01] Collector DisksCollector is still in progress, so skipping this Interval
[2023-07-23T07:31:13,413][INFO ][o.o.n.Node               ] [hq-pson01] version[2.6.0], pid[825], build[deb/7203a5af21a8a009aece1474446b437a3c674db6/2023-02-24T18:58:38.730482464Z], OS[Linux/5.4.0-153-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.6/17.0.6+10]
[2023-07-23T07:31:13,440][INFO ][o.o.n.Node               ] [hq-pson01] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2023-07-23T07:31:13,441][INFO ][o.o.n.Node               ] [hq-pson01] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms2g, -Xmx2g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-1752518876644756195, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/opensearch, -XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Djna.tmpdir=/var/lib/opensearch/tmp, -XX:MaxDirectMemorySize=1073741824, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/etc/opensearch, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]

Attachment: skeleton.json
Description: application/json

[perfsonar-user] opensearch issue, Eric Simich, 08/07/2023
- Re: [perfsonar-user] opensearch issue, Andrew Lake, 08/07/2023
  - Re: [perfsonar-user] opensearch issue, Eric Simich, 08/07/2023
    - Re: [perfsonar-user] opensearch issue, Eric Simich, 08/17/2023

List archive

Re: [perfsonar-user] opensearch issue