perfSONAR User Q&A and Other Discussion

[Daniel Manzo <>]

Hi Andy,

 

It appears that changing the value works, as when I run the command to start Cassandra, the status says that it is running. However, after refreshing the web interface, the status goes back to ‘cassandra dead but pid file exists.’ Here are the last few entries in cassandra.log:

 

INFO 14:52:20,488 Enqueuing flush of Memtable-local@1730688340(84/840 serialized/live bytes, 4 ops)

INFO 14:52:20,488 Writing Memtable-local@1730688340(84/840 serialized/live bytes, 4 ops)

INFO 14:52:20,502 Completed flushing /var/lib/cassandra/data/system/local/system-local-jb-24-Data.db (116 bytes) for commitlog position ReplayPosition(segmentId=1465325538284, position=190643)

INFO 14:52:20,503 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-24-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-21-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-23-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-22-Data.db')]

INFO 14:52:20,556 Enqueuing flush of Memtable-local@1384610825(10091/100910 serialized/live bytes, 259 ops)

INFO 14:52:20,557 Writing Memtable-local@1384610825(10091/100910 serialized/live bytes, 259 ops)

INFO 14:52:20,580 Completed flushing /var/lib/cassandra/data/system/local/system-local-jb-26-Data.db (5286 bytes) for commitlog position ReplayPosition(segmentId=1465325538284, position=202635)

INFO 14:52:20,605 Compacted 4 sstables to [/var/lib/cassandra/data/system/local/system-local-jb-25,].  6,249 bytes to 5,709 (~91% of original) in 96ms = 0.056714MB/s.  4 total partitions merged to 1.  Partition merge counts were {4:1, }

INFO 14:52:20,623 Node localhost/127.0.0.1 state jump to normal

 

Thanks,

Daniel

 

From: Andrew Lake [mailto:]
Sent: Tuesday, June 07, 2016 2:03 PM
To: ; Daniel Manzo
Subject: RE: [perfsonar-user] RHEL 6.8 web interface not retrieving results

 

Hi,

 

Looks like the lack of memory is preventing cassandra from starting-up. You’re not supposed to run it with less than 4GB but usually it boots. You should be able to override it’s memory allocations to get it to boot for your testing. You can do so by opening the file /etc/cassandra/conf/cassandra-env.sh and uncommenting the “MAX_HEAP_SIZE” properties and HEAP_NEWSIZE, then setting them to values like the following:

 

MAX_HEAP_SIZE=“512M"

HEAP_NEWSIZE=“128M”

 

You may have to experiment with the values, but hopefully that gets you running. I wouldn’t recommend doing that in production or for a very large number of measurements, but should be fine for testing.

 

Thanks,

Andy

 

 

 

 

On June 7, 2016 at 1:04:54 PM, Daniel Manzo () wrote:

Okay so here’s the last 6 errors from system.log:

 

INFO [main] 2016-06-07 10:07:13,509 StorageService.java (line 1522) Node localhost/127.0.0.1 state jump to normal

INFO [main] 2016-06-07 10:07:13,662 Server.java (line 156) Starting listening for CQL clients on localhost/127.0.0.1:9042...

INFO [main] 2016-06-07 10:07:13,742 ThriftServer.java (line 99) Using TFramedTransport with a max frame size of 15728640 bytes.

INFO [main] 2016-06-07 10:07:13,743 ThriftServer.java (line 118) Binding thrift service to localhost/127.0.0.1:9160

INFO [main] 2016-06-07 10:07:13,754 TServerCustomFactory.java (line 47) Using synchronous/threadpool thrift server on localhost : 9160

INFO [Thread-2] 2016-06-07 10:07:13,757 ThriftServer.java (line 135) Listening for thrift clients...

 

Here is what is in cassandra.log:

 

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000d6800000, 104857600, 0) failed; error='Cannot allocate memory' (errno=12)

#

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (malloc) failed to allocate 104857600 bytes for committing reserved memory.

# An error report file with more information is saved as:

# /tmp/jvm-5972/hs_error.log

 

Also, the OS is currently on a VM with only 1 GB of RAM, simply to test out the perfSONAR environment before installing on a physical server.

 

Thanks,

Daniel

From: Andrew Lake [mailto:]
Sent: Tuesday, June 07, 2016 12:56 PM
To: ; Garnizov, Ivan (RRZE); Daniel Manzo
Subject: RE: [perfsonar-user] RHEL 6.8 web interface not retrieving results

 

Hi,

 

Cassandra not running is the root of the problem. Both sets of errors should correct themselves once we get that figured out. The “cassandra dead but pid file exists” I believe is just a warning and not what actually prevents it from starting. Is there anything in /var/log/cassandra/cassandra.log or /var/log/cassandra/system.log?

 

Thanks,

Andy

On June 7, 2016 at 12:50:26 PM, Daniel Manzo () wrote:

Hi guys,

 

This is the latest error in /var/log/esmond/esmond.log:

2016-06-07 12:39:45,164 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

 

I’ve tried to restart Cassandra, but the status command always comes back with, “cassandra dead but pid file exists.”

 

Also here is the latest error reported in /var/log/perfsonar/regulartesting.log:

2016/06/07 12:39:45 (5204) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 500 Internal Server Error at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122.

 

Not sure what to do for the error in regulartesting.log

 

Thanks,

Daniel

 

From: Garnizov, Ivan (RRZE) [mailto:]
Sent: Tuesday, June 07, 2016 12:10 PM
To: Andrew Lake; ; Daniel Manzo
Subject: RE: [perfsonar-user] RHEL 6.8 web interface not retrieving results

 

Hi Daniel,

 

Please note that also your regular_testing is with “not running” status.

You have to make sure this service is running in order to have the results submitted and collected.

There is a log for the service on /var/log/perfsonar/regulartesting.log

 

Try restarting the service and report back the errors, if any.

 

Regards,

Ivan

 

From: [] On Behalf Of Andrew Lake
Sent: Dienstag, 7. Juni 2016 17:46
To: ; Daniel Manzo
Subject: Re: [perfsonar-user] RHEL 6.8 web interface not retrieving results

 

Hi,

 

When it shows up, you might want to look in /var/log/httpd/error_log or in any of the files under /var/log/esmond/. That’s usually caused by your measurement archive not running or encountering an error. One of those logs should give some indication as to why.

 

Thanks,

Andy

 

 

 

On June 7, 2016 at 11:33:38 AM, Daniel Manzo () wrote:

Hi,

 

I am currently running the latest version of the perfSONAR toolkit on RHEL 6.8 (specifically Red Hat Enterprise Linux Server release 6.8 (Santiago)). I have configured tests through the web interface and allowed adequate time for them to finish, however I am getting an error on the toolkit home page that states either, “Error loading test listing: Internal Server Error” or “Error loading detailed test summary data: Internal Server Error”. There is no consistency as when these show up, but there is consistently one of these showing up under test results.

 

Here is my screen:

 

 

Best regards,

 

Daniel Manzo

Scientific Computing Competence Ctr

_______________________

 

Bayer: Science For A Better Life

 

Bayer Business and Technology Services LLC

BBTS-IAO-COMM TECH NAIT College Program

Building B16

Pittsburgh, PA 15205-9741, USA

Tel:        412-777-2171

Mobile:   412-525-8332

E-mail:   

Web:      http://www.bayer.com