perfSONAR User Q&A and Other Discussion

["Garnizov, Ivan (RRZE)" <>]

Hi Andy,

Out of the blue sky with the GEANT deployment pS 3.4.2, I started receiving 
error events on in the regular_testing logs.
reg_test.txt
At the bottom of the attached file you will find excerpt from the esmond.log 
which based on the of the occurrence do not appear to be relevant.

I have restarted both cassandra , postgresql, regular_testing, but the 
warnings continued. I suspect the errors are about to reappear.
I would like also to assure you that Apache is up and running and I am able 
to see measurement results stored locally and on Central MA. There might have 
been some incidental overloads of the interface, but I would not expect to 
see this frequency of the errors. 
In the course of usage of the MP for the last 66 days access to the interface 
was always available on 5min checks.


The questions:
How do I find which performance results are missing?
Is there relation between the WARN and ERROR states?
Am I safe to ignore the error code without missing other substantial 
information?




Best regards,
Ivan
2016/01/12 16:36:03 (52537) ERROR> MeasurementArchiveChild.pm:125 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - 
Problem handling test results: Problem storing results: Error writing 
metadata: 500 INTERNAL SERVER ERROR at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm
 line 122.
2016/01/12 16:36:12 (52564) ERROR> EsmondBase.pm:65 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - 
Error writing data (500) 500 Timeout
2016/01/12 16:36:12 (52564) ERROR> MeasurementArchiveChild.pm:209 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results 
- Problem storing results: Error writing data: 500 Timeout
2016/01/12 16:36:12 (52564) ERROR> MeasurementArchiveChild.pm:125 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - 
Problem handling test results: Problem storing results: Error writing data: 
500 Timeout at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm
 line 122.
2016/01/12 16:36:15 (52592) ERROR> EsmondBase.pm:65 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - 
Error writing data (500) 500 Timeout
2016/01/12 16:36:15 (52592) ERROR> MeasurementArchiveChild.pm:209 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results 
- Problem storing results: Error writing data: 500 Timeout
2016/01/12 16:36:15 (52592) ERROR> MeasurementArchiveChild.pm:125 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - 
Problem handling test results: Problem storing results: Error writing data: 
500 Timeout at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm
 line 122.
2016/01/12 16:36:18 (52606) ERROR> EsmondBase.pm:65 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - 
Error writing data (500) 500 Timeout
2016/01/12 16:36:18 (52606) ERROR> MeasurementArchiveChild.pm:209 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results 
- Problem storing results: Error writing data: 500 Timeout
2016/01/12 16:36:18 (52606) ERROR> MeasurementArchiveChild.pm:125 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - 
Problem handling test results: Problem storing results: Error writing data: 
500 Timeout at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm
 line 122.
2016/01/12 16:36:18 (52607) ERROR> EsmondBase.pm:65 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - 
Error writing data (500) 500 Timeout
2016/01/12 16:36:18 (52607) ERROR> MeasurementArchiveChild.pm:209 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results 
- Problem storing results: Error writing data: 500 Timeout
2016/01/12 16:36:18 (52607) ERROR> MeasurementArchiveChild.pm:125 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - 
Problem handling test results: Problem storing results: Error writing data: 
500 Timeout at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm
 line 122.
2016/01/12 16:36:19 (52626) ERROR> EsmondBase.pm:65 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - 
Error writing data (500) 500 Timeout
2016/01/12 16:36:19 (52626) ERROR> MeasurementArchiveChild.pm:209 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results 
- Problem storing results: Error writing data: 500 Timeout
2016/01/12 16:36:19 (52626) ERROR> MeasurementArchiveChild.pm:125 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - 
Problem handling test results: Problem storing results: Error writing data: 
500 Timeout at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm
 line 122.
2016/01/12 16:36:28 (52672) ERROR> EsmondBase.pm:65 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - 
Error writing data (500) 500 Timeout
2016/01/12 16:36:28 (52672) ERROR> MeasurementArchiveChild.pm:209 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results 
- Problem storing results: Error writing data: 500 Timeout
2016/01/12 16:36:28 (52672) ERROR> MeasurementArchiveChild.pm:125 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - 
Problem handling test results: Problem storing results: Error writing data: 
500 Timeout at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm
 line 122.
2016/01/12 16:36:52 (52771) ERROR> EsmondBase.pm:65 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - 
Error writing data (500) 500 Timeout
2016/01/12 16:36:52 (52771) ERROR> MeasurementArchiveChild.pm:209 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results 
- Problem storing results: Error writing data: 500 Timeout
2016/01/12 16:36:52 (52771) ERROR> MeasurementArchiveChild.pm:125 
perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - 
Problem handling test results: Problem storing results: Error writing data: 
500 Timeout at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm
 line 122.
2016/01/12 16:40:58 (54122) WARN> EsmondBase.pm:207 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::add_data - 
Error posting data to MA: Time series value already exists with event type 
throughput at time 1452616536
2016/01/12 16:41:09 (54621) WARN> EsmondBase.pm:207 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::add_data - 
Error posting data to MA: Time series value already exists with event type 
packet-count-sent at time 1452616439
2016/01/12 16:41:10 (54650) WARN> EsmondBase.pm:207 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::add_data - 
Error posting data to MA: Time series value already exists with event type 
packet-count-sent at time 1452616435
2016/01/12 16:41:11 (54675) WARN> EsmondBase.pm:207 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::add_data - 
Error posting data to MA: Time series value already exists with event type 
packet-count-sent at time 1452616439
2016/01/12 16:41:12 (54684) WARN> EsmondBase.pm:207 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::add_data - 
Error posting data to MA: Time series value already exists with event type 
packet-count-sent at time 1452616438
2016/01/12 16:41:13 (54694) WARN> EsmondBase.pm:207 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::add_data - 
Error posting data to MA: Time series value already exists with event type 
packet-count-sent at time 1452616451
2016/01/12 16:42:00 (56299) WARN> daemon:103 main::__ANON__ - Warned: Use of 
uninitialized value in pattern match (m//) at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/MeasurementArchives/EsmondBase.pm
 line 334.
2016/01/12 16:42:00 (56303) WARN> daemon:103 main::__ANON__ - Warned: Use of 
uninitialized value in pattern match (m//) at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/MeasurementArchives/EsmondBase.pm
 line 334.
2016/01/12 16:51:08 (54123) WARN> daemon:103 main::__ANON__ - Warned: 
IPC::DirQueue: killed stale lockfile: 
/var/lib/perfsonar/regular_testing/esmond_latency_prod-psma-gn-01-buc-ro.geant.net/active/active/50.20160112164045160891.EMjkzNQ
 at /usr/share/perl5/IPC/DirQueue.pm line 519.
2016/01/12 16:51:09 (59312) WARN> EsmondBase.pm:207 
perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::add_data - 
Error posting data to MA: Time series value already exists with event type 
packet-count-sent at time 1452616714
2016/01/12 16:51:33 (59549) WARN> daemon:103 main::__ANON__ - Warned: Use of 
uninitialized value in pattern match (m//) at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/MeasurementArchives/EsmondBase.pm
 line 334.
2016/01/12 16:51:33 (59550) WARN> daemon:103 main::__ANON__ - Warned: Use of 
uninitialized value in pattern match (m//) at 
/opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/MeasurementArchives/EsmondBase.pm
 line 334.





esmond.log
2016-01-12 02:30:03,796 [INFO] /opt/esmond/esmond/cassandra.py: 
Checking/creating column families
2016-01-12 02:30:03,798 [INFO] /opt/esmond/esmond/cassandra.py: Schema check 
done
2016-01-12 02:30:03,805 [INFO] /opt/esmond/esmond/cassandra.py: Connected to 
['localhost:9160']
2016-01-12 02:30:10,901 [INFO] 
/opt/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: 
Connection 30487248 (localhost:9160) in pool 30460432 failed: 
TimedOutException(acknowledged_by=None, acknowledged_by_batchlog=None)
2016-01-12 02:30:15,923 [INFO] 
/opt/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: 
Connection 30487248 (localhost:9160) in pool 30460432 failed: 
TimedOutException(acknowledged_by=None, acknowledged_by_batchlog=None)
2016-01-12 02:30:20,965 [INFO] 
/opt/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: 
Connection 30487248 (localhost:9160) in pool 30460432 failed: 
TimedOutException(acknowledged_by=None, acknowledged_by_batchlog=None)
2016-01-12 02:30:26,047 [INFO] 
/opt/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: 
Connection 30487248 (localhost:9160) in pool 30460432 failed: 
TimedOutException(acknowledged_by=None, acknowledged_by_batchlog=None)
2016-01-12 02:30:31,210 [INFO] 
/opt/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: 
Connection 30487248 (localhost:9160) in pool 30460432 failed: 
TimedOutException(acknowledged_by=None, acknowledged_by_batchlog=None)
2016-01-12 02:30:36,533 [INFO] 
/opt/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: 
Connection 30487248 (localhost:9160) in pool 30460432 failed: 
TimedOutException(acknowledged_by=None, acknowledged_by_batchlog=None)
2016-01-12 02:30:42,176 [INFO] 
/opt/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: 
Connection 30487248 (localhost:9160) in pool 30460432 failed: 
TimedOutException(acknowledged_by=None, acknowledged_by_batchlog=None)
2016-01-12 02:30:48,460 [INFO] 
/opt/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: 
Connection 30487248 (localhost:9160) in pool 30460432 failed: 
TimedOutException(acknowledged_by=None, acknowledged_by_batchlog=None)
2016-01-12 02:30:56,025 [INFO] 
/opt/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: 
Connection 30487248 (localhost:9160) in pool 30460432 failed: 
TimedOutException(acknowledged_by=None, acknowledged_by_batchlog=None)
2016-01-12 02:31:06,152 [INFO] 
/opt/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: 
Connection 30487248 (localhost:9160) in pool 30460432 failed: 
TimedOutException(acknowledged_by=None, acknowledged_by_batchlog=None)
2016-01-12 02:31:21,401 [INFO] 
/opt/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: 
Connection 30487248 (localhost:9160) in pool 30460432 failed: 
TimedOutException(acknowledged_by=None, acknowledged_by_batchlog=None)