perfSONAR User Q&A and Other Discussion

[Dan Doyle <>]

Kathy,

No worries - happens to the best of us. :)

re: regulartesting, looks like it may be a similar issue to the cassandra issue? It can't write to to /var/log/perfsonar/regulartesting.log, which based on what you said may also have lived on that separated RAID1 partition and thus may have the same userid issues?

I *think* you should be able to just chown it to the user and it will be corrected. Can you try that on both of them?

Dan Doyle
GlobalNOC Software Developer
1-812-856-3892

On Nov 17, 2016, at 3:24 PM, Kathy Benninger <> wrote:

Hi Don,

Oops, yes, the example I cut/pasted was me as non-root.  Unfortunately, starting it as root also fails:

[root@ps perfsonar]# /etc/init.d/perfsonar-regulartesting start
/usr/lib/perfsonar/bin/regulartesting.pl --config=/etc/perfsonar/regulartesting.conf --pidfile=regulartesting.pid --piddir=/var/run --logger=/etc/perfsonar/regulartesting-logger.conf --user=perfsonar --group=perfsonar --daemonize
Cannot write to '/var/log/perfsonar/regulartesting.log': Permission denied at /usr/share/perl5/vendor_perl/Log/Dispatch/File.pm line 96.
/etc/init.d/perfsonar-regulartesting start: perfSONAR Regular Testing could not be started

1. Space is OK:

[root@ps perfsonar]# df
Filesystem            1K-blocks     Used  Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      241369536  3621756  225482616   2% /
tmpfs                  12298128        0   12298128   0% /dev/shm
/dev/sda1                487652   145674     316378  32% /boot
/dev/md0             1922598072 26270420 1798658616   2% /var

2. Does /var/run/cassandra exist and is writeable by cassandra user and group?

Thanks!  This question put me onto what happened and at least part of the problem. 

The original system build had /var mounted as a separate partition on a RAID1 hard disk pair, which provided redundancy for the collected measurement data (this was before central archive support).   / and /boot were on an SSD.

I recently did the system re-installation onto a new SSD and mounted the RAID set as /var (like it was originally), with the intention of keeping the historical data intact.

Unfortunately it looks like userid and group numbers in /etc/passwd and /etc/group are not quite the same between the initial installation and my recent rebuild.  The result is that the original cassandra account was user:group 490:491, but is now 497:497.  esmond is 491:491.  pulse is 490:492 and pulse-access is group 491 so I think that is causing problems with file ownership and permissions.

I'm not exactly sure how to go about fixing this problem, or if I can without a total rebuild.

Kathy

On 11/17/2016 12:43 PM, Dan Doyle wrote:

Hi Kathy,

Regarding regulartesting, it looks like you tried to start it as a non root user. Can you try again with "sudo /etc/init.d/perfsonar-regulartesting start" or run the command as the root user?

A few ideas on the cassandra issue:

1 - kind of a long shot given the re-install note, but is the partition cassandra is on full by chance?

2 - can you verify the permissions on the things it is complaining about? Does /var/run/cassandra exist and is writeable by cassandra user and group?

Dan Doyle
GlobalNOC Software Developer
1-812-856-3892

On Nov 17, 2016, at 11:58 AM, Kathy Benninger <> wrote:

I re-installed perfsonar 3.5.1 toolkit and am having trouble restarting regulartesting.  It doesn't start on boot, nor can I restart it manually.

Some of the messages and log file info:

[benninge@ps ~]$ /etc/init.d/perfsonar-regulartesting start
/usr/lib/perfsonar/bin/regulartesting.pl --config=/etc/perfsonar/regulartesting.conf --pidfile=regulartesting.pid --piddir=/var/run --logger=/etc/perfsonar/regulartesting-logger.conf --user=perfsonar --group=perfsonar --daemonize
Can't write pidfile: /var/run/regulartesting.pid
at /usr/lib/perfsonar/bin/regulartesting.pl line 220
       main::lockPIDFile('/var/run', 'regulartesting.pid') called at /usr/lib/perfsonar/bin/regulartesting.pl line 46
/etc/init.d/perfsonar-regulartesting start: perfSONAR Regular Testing could not be started

[root@ps httpd]# /etc/init.d/cassandra start
Starting Cassandra: OK
[root@ps httpd]# /etc/init.d/cassandra status
cassandra dead but subsys locked

================================================================
/var/log/cassandra/cassandra.log says:

/usr/sbin/cassandra: line 155: /var/run/cassandra/cassandra.pid: Permission denied
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/cassandra/system.log (Permission denied)
       at java.io.FileOutputStream.open(Native Method)
       at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
       at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
       at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
       at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
       at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
       at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
       at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
       at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
       at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
       at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
       at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
       at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
       at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395)
       at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:922)
       at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89)
       at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58)
       at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:914)
       at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:461)
       at org.apache.cassandra.service.CassandraDaemon.initLog4j(CassandraDaemon.java:140)
       at org.apache.cassandra.service.CassandraDaemon.<clinit>(CassandraDaemon.java:88)
INFO 11:29:46,369 Logging initialized
INFO 11:29:46,509 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO 11:29:46,884 Data files directories: [/var/lib/cassandra/data]
INFO 11:29:46,884 Commit log directory: /var/lib/cassandra/commitlog
INFO 11:29:46,884 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO 11:29:46,884 disk_failure_policy is stop
INFO 11:29:46,885 commit_failure_policy is stop
INFO 11:29:46,893 Global memtable threshold is enabled at 1463MB
INFO 11:29:47,058 Not using multi-threaded compaction
INFO 11:29:47,397 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO 11:29:47,418 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO 11:29:47,438 JVM vendor/version: OpenJDK 64-Bit Server VM/1.7.0_121
WARN 11:29:47,438 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO 11:29:47,438 Heap size: 6138298368/6138298368
INFO 11:29:47,438 Code Cache Non-heap memory: init = 2555904(2496K) used = 646592(631K) committed = 2555904(2496K) max = 50331648(49152K)
INFO 11:29:47,439 Par Eden Space Heap memory: init = 1259208704(1229696K) used = 151105552(147564K) committed = 1259208704(1229696K) max = 1259208704(1229696K)
INFO 11:29:47,439 Par Survivor Space Heap memory: init = 157351936(153664K) used = 0(0K) committed = 157351936(153664K) max = 157351936(153664K)
INFO 11:29:47,439 CMS Old Gen Heap memory: init = 4721737728(4611072K) used = 0(0K) committed = 4721737728(4611072K) max = 4721737728(4611072K)
INFO 11:29:47,439 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 14081912(13751K) committed = 21757952(21248K) max = 174063616(169984K)
INFO 11:29:47,439 Classpath: /etc/cassandra/conf:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.14.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.14.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.14.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
INFO 11:29:47,440 JNA not found. Native methods will be disabled.
WARN 11:29:47,441 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
ERROR 11:29:47,626 Doesn't have write permissions for /var/lib/cassandra/data directory
================================================================

Nothing's been logged in regulartesting.log since 4-Nov.

Please let me know how I might proceed with fixing the problem.

Kathy Benninger
Pittsburgh Supercomputing Center

Attachment: smime.p7s
Description: S/MIME cryptographic signature