perfSONAR User Q&A and Other Discussion

[Kent Myers <>]

Hi Daniel, Andy and Ivan;

 

I’m reaching out about a perfSONAR issue Daniel inquired about that is dated a couple years ago. I just wanted to make sure you’re aware, as the issue encountered may affect current builds of perfSONAR.

 

I ran into your inquiry on a perfSONAR listserv while troubleshooting an identical issue: https://lists.internet2.edu/sympa/arc/perfsonar-user/2017-06/msg00065.html

 

I noted that the Cassandra service attempts to load the last known/written db on startup; if it is corrupt (which can occur if the server has cron scheduled reboots or any interruption in Cassandra writing to filesystem(s)), then Cassandra will fail to start and you’ll encounter the issue you’ve described in your inquiry.

 

I was able to resolve the issue by moving the last known Cassandra db file (as described in a tail of Cassandra.log) that was attempted (in my case, /var/lib/cassandra/data/system/local/system-local-jb-872-Data.db) to a temp location outside of the Cassandra db dir.

 

After init 6 (rather than restarting Cassandra and httpd), Cassandra turned up properly and the perfSONAR server was working without issue.

 

I didn’t wipe any of the other files within the directory as it was noted that Cassandra ‘cleans up’ when a new db file is generated. All system-local-jb-#-* files should have matching #’s after Cassandra starts; if not, then I’d clear the dir as a next step prior to restarting services or rebooting.

 

I did end up losing approximately 1 day of data using this method due to no noted methods for repairing Cassandra db files, however, it was worth it in this case as this particular system is a mobile, small form factor low priority autonomous node used to verify sla on provided links.

 

I haven’t tested this, however, perhaps adding the following (noted in your listserv msg) to shutdown or reboot script(s) might help resolve some of these scenarios?

 

 

 

pkill -9 -f java

 

systemctl stop cassandra

 

systemctl stop httpd

 

init 0 / init 6

 

 

 

Hope this helps, as perfSONAR is one of the best open source network testing platforms I’ve ever used.

 

Regards

 

- Kent

 

---

*** PROPRIETARY, CONFIDENTIAL OR PRIVILEGED COMMUNICATION ***
This message has been prepared on resources owned by Midwest Energy & Communications and may contain proprietary, privileged, confidential or Critical Energy Infrastructure information protected by law and is subject to the Electronic Communications Policy of Midwest Energy & Communications. It is solely intended for the use of the individual or entity to whom it is addressed. Any review, dissemination, distribution, forwarding, or copying of this communication by someone other than the intended recipient, or the employee responsible for delivering this communication to the intended recipient, is prohibited. If you have received this communication in error, please immediately notify the sender via email, then destroy the original message.
***Please consider the environment before printing this email.***