perfSONAR User Q&A and Other Discussion

[Kathy Benninger <>]

    Sorry for the prior incomplete message I just sent! Here's the rest
    of what I meant to include.
    
    There is a set of errors repeating over and over in the log files
    (sample, following). Please let me know if I should send this output
    some other way.
    
    [root@ps100g-10g logstash]# grep -i error logstash-plain.log | less
    [2023-05-10T00:00:04,560][INFO ][logstash.runner          ] JVM
    bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC,
    -XX:CMSInitiatingOccupancyFraction=75,
    -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true,
    -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true,
    -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0,
    -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError,
    -Djava.security.egd=file:/dev/urandom,
    -Dlog4j2.isThreadContextMapInheritable=true]
    [2023-05-10T00:00:07,135][INFO ][logstash.filters.ruby.script] Test
    run complete
{:script_path=>"/usr/lib/perfsonar/logstash/ruby/pscheduler_proxy_normalize.rb",
    :results=>{:passed=>0, :failed=>0, :errored=>0}}
    [2023-05-10T00:00:07,201][INFO ][logstash.filters.ruby.script] Test
    run complete
{:script_path=>"/usr/lib/perfsonar/logstash/ruby/pscheduler_test_checksum.rb",
    :results=>{:passed=>0, :failed=>0, :errored=>0}}
    [2023-05-10T00:00:07,229][INFO ][logstash.filters.ruby.script] Test
    run complete
{:script_path=>"/usr/lib/perfsonar/logstash/ruby/pscheduler_normalize_endpoints.rb",
    :results=>{:passed=>0, :failed=>0, :errored=>0}}
    [2023-05-10T00:00:07,237][INFO ][logstash.filters.ruby.script] Test
    run complete
{:script_path=>"/usr/lib/perfsonar/logstash/ruby/pscheduler_iso8601_duration.rb",
    :results=>{:passed=>0, :failed=>0, :errored=>0}}
    [2023-05-10T00:00:07,277][INFO ][logstash.filters.ruby.script] Test
    run complete
{:script_path=>"/usr/lib/perfsonar/logstash/ruby/pscheduler_histogram.rb",
    :results=>{:passed=>0, :failed=>0, :errored=>0}}
    [2023-05-10T00:00:07,280][INFO ][logstash.filters.ruby.script] Test
    run complete
{:script_path=>"/usr/lib/perfsonar/logstash/ruby/pscheduler_histogram.rb",
    :results=>{:passed=>0, :failed=>0, :errored=>0}}
    [2023-05-10T00:00:07,286][INFO ][logstash.filters.ruby.script] Test
    run complete
{:script_path=>"/usr/lib/perfsonar/logstash/ruby/pscheduler_iso8601_duration.rb",
    :results=>{:passed=>0, :failed=>0, :errored=>0}}
    [2023-05-10T00:00:07,293][INFO ][logstash.filters.ruby.script] Test
    run complete
{:script_path=>"/usr/lib/perfsonar/logstash/ruby/pscheduler_iso8601_duration.rb",
    :results=>{:passed=>0, :failed=>0, :errored=>0}}
    [2023-05-10T00:00:07,300][INFO ][logstash.filters.ruby.script] Test
    run complete
{:script_path=>"/usr/lib/perfsonar/logstash/ruby/pscheduler_iso8601_duration.rb",
    :results=>{:passed=>0, :failed=>0, :errored=>0}}
    [2023-05-10T00:00:07,302][INFO ][logstash.filters.ruby.script] Test
    run complete
{:script_path=>"/usr/lib/perfsonar/logstash/ruby/pscheduler_trace.rb",
    :results=>{:passed=>0, :failed=>0, :errored=>0}}
    [2023-05-10T00:00:07,364][ERROR][logstash.plugins.registry] Unable
    to load plugin. {:type=>"output", :name=>"opensearch"}
    [2023-05-10T00:00:07,371][ERROR][logstash.agent           ] Failed
    to execute action
    {:action=">LogStash::PipelineAction::Create/pipeline_id:pscheduler,"
    :exception=>"Java::JavaLang::IllegalStateException",
    :message=>"Unable to configure plugins: (PluginLoadingError)
    Couldn't find any output plugin named 'opensearch'. Are you sure
    this is correct? Trying to load the opensearch output plugin
    resulted in this error: Unable to load the requested plugin named
    opensearch of type output. The plugin is not installed.",
:backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)",
"org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)",
"org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)",
"org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)",
"org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)",
"org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)",
"org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)",
"usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48)",
"org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)",
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)",
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)",
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)",
    "org.jruby.RubyClass.newInstance(RubyClass.java:939)",
"org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)",
    "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)",
"usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)",
"usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)",
"org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)",
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)",
    "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)",
"usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:392)",
"org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)",
    "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)",
    "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)",
    "org.jruby.runtime.Block.call(Block.java:139)",
    "org.jruby.RubyProc.call(RubyProc.java:318)",
    "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)",
    "java.base/java.lang.Thread.run(Thread.java:829)"]}
    [2023-05-10T00:00:07,376][ERROR][logstash.agent           ] An
    exception happened when converging configuration
    {:exception=>LogStash::Error, :message=>"Don't know how to
    handle `Java::JavaLang::IllegalStateException` for
    `PipelineAction::Create<pscheduler>`"}
    [2023-05-10T00:00:07,381][FATAL][logstash.runner          ] An
    unexpected error occurred! {:error=>#<LogStash::Error: Don't
    know how to handle `Java::JavaLang::IllegalStateException` for
    `PipelineAction::Create<pscheduler>`>,
    :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in
    `create'", "org/logstash/execution/ConvergeResultExt.java:60:in
    `add'",
    "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:405:in
    `block in converge_state'"]}
    [2023-05-10T00:00:07,385][FATAL][org.logstash.Logstash    ] Logstash
    stopped processing because of an error: (SystemExit) exit
    
    Thanks for your help!
    
    Kathy

On 5/10/2023 5:27 PM, Kathy Benninger wrote:

      Hi Andy,
      
      "/var/log/logstash/logstash-plain.log” has many repeating error
      messages, just since midnight. The error file has
      
      [2023-05-10T00:00:07,364][ERROR][logstash.plugins.registry] Unable
      to load plugin. {:type=>"output", :name=>"opensearch"}
      
      logstash.runner and logstash.agent

On 5/10/2023 10:56 AM, Andrew Lake wrote:

Hi Kathy,

Anything interesting in "/var/log/logstash/logstash-plain.log”?Interesting would include frequent errors or anything like that. Possible its doing something unexpected.

Thanks,

Andy 

On May 9, 2023 at 9:37:00 PM, Kathy Benninger () wrote:

Greetings!

I have a bare-metal server that was up to date and running fine with the
version 4 pS-Toolkit. After the 5.0 upgrade, the perfSONAR is reporting high
CPU utilization by logstash. All testing is stopped (Actually pscheduler
testing doesn't run and I believe I also have to fix the limits.conf issue,
though I never modified the limits file.).

The OS is CentOS Linux release 7.9.2009 (Core)

Example utilization:

[benninge@ps100g-10g ~]$ top -H
top - 13:41:53 up  1:55,  2 users,  load average: 3.33, 3.11, 3.10
Threads: 542 total,   6 running, 536 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.4 us,  2.0 sy, 72.1 ni, 25.5 id,  0.0 wa,  0.0 hi, 0.0 si,  0.0 st
KiB Mem : 47884028 total, 19617196 free, 27611884 used,   654948 buff/cache
KiB Swap: 52428796 total, 52428796 free,        0 used. 19800300 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM TIME+ COMMAND
 42145 logstash  39  19 3889836 456264  14888 R 91.4  1.0 0:02.76 java
 42158 logstash  39  19 3889836 456264  14888 R 71.2  1.0 0:02.15 C2
CompilerThre
 42153 logstash  39  19 3889836 456264  14888 R 70.2  1.0 0:02.12 C2
CompilerThre
 42154 logstash  39  19 3889836 456264  14888 R 46.4  1.0 0:01.40 C1
CompilerThre
 42147 logstash  39  19 3889836 456264  14888 R  6.6  1.0 0:00.20 CMS Main
Thread
 42146 logstash  39  19 3889836 456264  14888 S  1.7  1.0 0:00.05 GC Thread#0
  7325 benninge  20   0  121236   2792   1452 S  0.7  0.0 0:53.08 htop
 40457 benninge  20   0  160652   2712   1568 R  0.7  0.0 0:01.20 top
 42162 logstash  39  19 3889836 456264  14888 S  0.7  1.0 0:00.02 GC Thread#2
     9 root      20   0       0      0      0 S  0.3  0.0 0:11.79 rcu_sched
  1387 opensea+  20   0   26.6g  23.1g  23432 S  0.3 50.7 0:01.52 G1 Service
  1540 opensea+  20   0   26.6g  23.1g  23432 S  0.3 50.7 0:03.36
pa-collectors-t
  4261 opensea+  20   0   26.6g  23.1g  23432 S  0.3 50.7 0:00.25
opensearch[ps10
  6144 opensea+  20   0   26.6g  23.1g  23432 S  0.3 50.7 0:00.02
opensearch[ps10
  3054 root      20   0  638544  15176   5096 S  0.3  0.0 0:03.51 f2b/f.sshd
 24256 root      20   0  638544  15176   5096 S  0.3  0.0 0:02.45 f2b/observer
  3184 perfson+  20   0  313844  27952   5276 S  0.3  0.1 0:33.51 python3
  7148 benninge  20   0  157500   2552   1188 S  0.3  0.0 0:00.72 sshd
 10376 benninge  20   0  157500   2560   1204 S  0.3  0.0 0:00.35 sshd
 42148 logstash  39  19 3889836 456264  14888 S  0.3  1.0 0:00.01 VM Thread
 42161 logstash  39  19 3889836 456264  14888 S  0.3  1.0 0:00.01 GC Thread#1
 42163 logstash  39  19 3889836 456264  14888 S  0.3  1.0 0:00.01 GC Thread#3

I saw that Cas D'Angelo had reported a similar problem on 24-Apr-2023, but I
do not see a resolution. This node was not an archive host, but it was a
standalone host storing its own measurements.

Thanks for any guidance!

Kathy Benninger
Pittsburgh Supercomputing Center

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user