perfSONAR User Q&A and Other Discussion

[Eric Simich <>]

Any movement on this ?

On Mon, Aug 7, 2023 at 12:54 PM Eric Simich <> wrote:

Andrew,

Thanks for your prompt response, I appreciate it!

When you run “systemctl restart opensearch” does it start? 

No.

Assuming it doesn’t, anything of use in journalctl -xe related to opensearch? 

Yes.

root@hq-pson01:~# journalctl -xe --no-pager | grep -iE "search"
Aug 07 19:31:22 hq-pson01 logstash[576157]: [2023-08-07T19:31:22,773][ERROR][logstash.agent           ] Failed to execute action {:action=">LogStash::PipelineAction::Create/pipeline_id:pscheduler," :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: Cannot evaluate `${opensearch_output_host}`. Replacement variable `opensearch_output_host` is not defined in a Logstash secret store or an environment entry and there is no default value given.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:392)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}

How about var/log/opensearch/opensearch.log?

Attached to email.

The /etc/opensearch.yml

root@hq-pson01:/etc/opensearch# less opensearch.yml | grep -v "#"
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch


plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3


plugins.security.ssl.transport.pemcert_filepath: node.pem
plugins.security.ssl.transport.pemkey_filepath: node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node.pem
plugins.security.ssl.http.pemkey_filepath: node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_default_init_securityindex: false
plugins.security.authcz.admin_dn:
  - CN=admin,OU=Archive,O=perfSONAR
plugins.security.nodes_dn:
  - CN=localhost,OU=Archive,O=perfSONAR
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.allow_unsafe_democertificates: false
plugins.security.ssl.transport.enforce_hostname_verification: false
discovery.type: single-node

psconfig configuration
root@hq-pson01:/etc/psconfig# psconfig published
[skeleton.json]
    Local File: /usr/lib/perfsonar/web-psconfig/skeleton.json
    Web URL: https://localhost/psconfig/skeleton.json

attached skeleton.json

On Mon, Aug 7, 2023 at 11:47 AM Andrew Lake <> wrote:

Hi Eric,

When you run “systemctl restart opensearch” does it start? Assuming it doesn’t, anything of use in journalctl -xe related to opensearch? How about var/log/opensearch/opensearch.log?

Thanks,

Andy

On August 7, 2023 at 12:08:32 PM, Eric Simich () wrote:

Support,

I'm running into an issue after installing the Perfsonar toolkit. I installed the toolkit per the bundle install instructions. Wherein, the opensearch service fails to load. But the toolkit webUI indicates the service is running. The toolkit webUI indicates pscheduler is not running, but systemctl shows the service is running. See attached screenshot. Maddash is also running, but I cannot access the webUI - 404 not found.

I'd like to get this perfsonar toolkit node running, but it has not been able to operate as a turn-key solution after installing the packages and following the guide.

pson01:~$ sudo systemctl list-units --state failed
  UNIT                  LOAD   ACTIVE SUB    DESCRIPTION                          
● opensearch.service    loaded failed failed OpenSearch            

● opensearch.service - OpenSearch
     Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Sun 2023-07-23 07:32:46 UTC; 2 weeks 1 days ago
       Docs: https://opensearch.org/
   Main PID: 825 (code=killed, signal=KILL)

pscheduler-scheduler.service - pScheduler server - scheduler
     Loaded: loaded (/lib/systemd/system/pscheduler-scheduler.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-08-07 15:00:07 UTC; 27min ago
    Process: 516494 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/scheduler (code=exited, status=0/SUCCESS)
    Process: 516505 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/scheduler (code=exited, status=0/SUCCESS)
    Process: 516507 ExecStartPre=/bin/chmod 755 /var/run/pscheduler-server/scheduler (code=exited, status=0/SUCCESS)
    Process: 516508 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/scheduler/tmp (code=exited, status=0/SUCCESS)
    Process: 516509 ExecStartPre=/bin/chmod 700 /var/run/pscheduler-server/scheduler/tmp (code=exited, status=0/SUCCESS)
    Process: 516510 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/scheduler/tmp (code=exited, status=0/SUCCESS)
    Process: 516511 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/scheduler.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/scheduler.conf); echo OPTIONS=$opts > /var/run/pscheduler->
   Main PID: 516519 (python3)
      Tasks: 1 (limit: 4662)
     Memory: 21.3M
     CGroup: /system.slice/pscheduler-scheduler.service
             └─516519 python3 /usr/lib/pscheduler/daemons/scheduler --dsn @/etc/pscheduler/database/database-dsn --pid-file /var/run/pscheduler-server/scheduler/pid

Aug 07 15:00:07 hq-pson01 systemd[1]: Starting pScheduler server - scheduler...
Aug 07 15:00:07 hq-pson01 systemd[1]: Started pScheduler server - scheduler.
Aug 07 15:00:07 hq-pson01 python3[516519]: scheduler INFO     Started

Jul 23 07:31:06 hq-pson01 systemd-entrypoint[825]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Jul 23 07:31:06 hq-pson01 systemd-entrypoint[825]: WARNING: System::setSecurityManager will be removed in a future release
Jul 23 07:31:13 hq-pson01 systemd-entrypoint[825]: WARNING: A terminally deprecated method in java.lang.System has been called
Jul 23 07:31:13 hq-pson01 systemd-entrypoint[825]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.6.0.jar)
Jul 23 07:31:13 hq-pson01 systemd-entrypoint[825]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Jul 23 07:31:13 hq-pson01 systemd-entrypoint[825]: WARNING: System::setSecurityManager will be removed in a future release
Jul 23 07:32:00 hq-pson01 systemd[1]: opensearch.service: start operation timed out. Terminating.
Jul 23 07:32:46 hq-pson01 systemd[1]: opensearch.service: Main process exited, code=killed, status=9/KILL
Jul 23 07:32:46 hq-pson01 systemd[1]: opensearch.service: Failed with result 'timeout'.
Jul 23 07:32:46 hq-pson01 systemd[1]: Failed to start OpenSearch.

maddash-server.service - MaDDash Scheduler and REST Server
     Loaded: loaded (/lib/systemd/system/maddash-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-07-23 07:30:45 UTC; 2 weeks 1 days ago
   Main PID: 926 (java)
      Tasks: 57 (limit: 4662)
     Memory: 280.1M
     CGroup: /system.slice/maddash-server.service
             └─926 java -jar /usr/share/maddash/maddash-server/target/maddash-server.one-jar.jar -c /etc/maddash/maddash-server/maddash.yaml -l /etc/maddash/maddash-server/log4j.properties

Jul 23 07:30:45 hq-pson01 systemd[1]: Starting MaDDash Scheduler and REST Server...
Jul 23 07:30:45 hq-pson01 systemd[1]: Started MaDDash Scheduler and REST Server.
root@hq-pson01:~# 

apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-07-23 07:30:46 UTC; 2 weeks 1 days ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 324347 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
   Main PID: 1046 (apache2)
      Tasks: 246 (limit: 4662)
     Memory: 382.3M
     CGroup: /system.slice/apache2.service
             ├─  1046 /usr/sbin/apache2 -k start
             ├─324369 /usr/sbin/apache2 -k start
             ├─324370 pscheduler-api    -k start
             ├─324371 pscheduler-api    -k start
             ├─324372 pscheduler-api    -k start
             ├─324373 pscheduler-api    -k start
             ├─324374 pscheduler-api    -k start
             ├─324375 pscheduler-api    -k start
             ├─324376 pscheduler-api    -k start
             ├─324377 pscheduler-api    -k start
             ├─324378 pscheduler-api    -k start
             ├─324379 pscheduler-api    -k start
             ├─324380 /usr/sbin/apache2 -k start
             └─324381 /usr/sbin/apache2 -k start

Aug 07 00:13:10 hq-pson01 apache2[324375]: pscheduler-api[324375] INFO     Started
Aug 07 00:13:10 hq-pson01 apache2[324375]: pscheduler-api[324375] INFO     Limits loaded from /etc/pscheduler/limits.conf
Aug 07 00:13:11 hq-pson01 apache2[324376]: pscheduler-api[324376] INFO     Started
Aug 07 00:13:11 hq-pson01 apache2[324376]: pscheduler-api[324376] INFO     Limits loaded from /etc/pscheduler/limits.conf
Aug 07 00:23:09 hq-pson01 apache2[324370]: pscheduler-api[324370] INFO     Started
Aug 07 00:23:10 hq-pson01 apache2[324370]: pscheduler-api[324370] INFO     Limits loaded from /etc/pscheduler/limits.conf
Aug 07 00:23:10 hq-pson01 apache2[324372]: pscheduler-api[324372] INFO     Started
Aug 07 00:23:11 hq-pson01 apache2[324372]: pscheduler-api[324372] INFO     Limits loaded from /etc/pscheduler/limits.conf
Aug 07 00:23:11 hq-pson01 apache2[324379]: pscheduler-api[324379] INFO     Started
Aug 07 00:23:11 hq-pson01 apache2[324379]: pscheduler-api[324379] INFO     Limits loaded from /etc/pscheduler/limits.conf

Sys info:

hq-pson01:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Sincerely,

Eric Simich

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user