perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

Group,

     I've now shut down the central MA and more than doubled the size of the Disk (root volume).  So it's no longer a issue of raw space.  But still the ps_remove_data.py won't run to completion.  I get somewhere from 10 to 30 minutes in (it seems to vary) and it ends with something like the following:

Deleted 1 rows for metadata_key=189be3bd15fb482a91f2bfb9524c4473, event_type=histogram-rtt, summary_type=aggregation, summary_window=3600

Sending request to delete 1 rows for metadata_key=189be3bd15fb482a91f2bfb9524c4473, event_type=histogram-rtt, summary_type=statistics, summary_window=3600

Deleted 1 rows for metadata_key=189be3bd15fb482a91f2bfb9524c4473, event_type=histogram-rtt, summary_type=statistics, summary_window=3600

Sending request to delete 11 rows for metadata_key=189be3bd15fb482a91f2bfb9524c4473, event_type=histogram-ttl-reverse, summary_type=base, summary_window=0

Deleted 11 rows for metadata_key=189be3bd15fb482a91f2bfb9524c4473, event_type=histogram-ttl-reverse, summary_type=base, summary_window=0

Sending request to delete 11 rows for metadata_key=189be3bd15fb482a91f2bfb9524c4473, event_type=packet-count-lost-bidir, summary_type=base, summary_window=0

Error: Retried 1 times. Last failure was timeout: timed out

None of these:

/var/log/esmond/esmond.log

/var/log/cassandra/cassandra.log

/var/log/httpd/access_log

/var/log/https/error_log

appear to contain anything interesting when the problem occurs.  I've tried running a manual compaction with nodetool in case there were simply so many tombstones hanging out there that it was causing cassandra or Java/Http a problem in processing, didn't make any difference.  Anyone have any thoughts on anything else I should try (adjusting cassandra config file settings, etc.) before I just delete this database and start it fresh?

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047

On Wed, Jul 26, 2017 at 10:15 AM, Casey Russell <> wrote:

Group,

     I dug myself a hole and I only see a couple of ways out now.  I wasn't watching the database size on my central MA and my disk utilization is now over 90%.  I've tried using the ps_remove_data.py script several times with several different variations on the config script, but it will invariably end some minutes or hours later with a timeout like this:

Sending request to delete 24 rows for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=histogram-rtt, summary_type=aggregation, summary_window=3600

Deleted 24 rows for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=histogram-rtt, summary_type=aggregation, summary_window=3600

Query error for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=histogram-rtt, summary_type=base, summary_window=0, begin_time=0, expire_time=1485529380

Query error for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=histogram-rtt, summary_type=statistics, summary_window=0, begin_time=0, expire_time=1485529380

Sending request to delete 24 rows for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=histogram-rtt, summary_type=statistics, summary_window=3600

Deleted 24 rows for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=histogram-rtt, summary_type=statistics, summary_window=3600

Query error for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=histogram-ttl-reverse, summary_type=base, summary_window=0, begin_time=0, expire_time=1469545381

Query error for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=packet-count-lost-bidir, summary_type=base, summary_window=0, begin_time=0, expire_time=1469545381

Query error for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=packet-duplicates-bidir, summary_type=base, summary_window=0, begin_time=0, expire_time=1469545503

Sending request to delete 24 rows for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=packet-loss-rate-bidir, summary_type=aggregation, summary_window=3600

Deleted 24 rows for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=packet-loss-rate-bidir, summary_type=aggregation, summary_window=3600

Query error for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=packet-loss-rate-bidir, summary_type=base, summary_window=0, begin_time=0, expire_time=1469545505

Query error for metadata_key=1bdb8f32fe9d4194828d134f37fb37b0, event_type=packet-reorders-bidir, summary_type=base, summary_window=0, begin_time=0, expire_time=1469545509

Sending request to delete 6 rows for metadata_key=1be4b626486c46be88776b3530819ce8, event_type=packet-count-lost, summary_type=base, summary_window=0

Deleted 6 rows for metadata_key=1be4b626486c46be88776b3530819ce8, event_type=packet-count-lost, summary_type=base, summary_window=0

Sending request to delete 6 rows for metadata_key=1be4b626486c46be88776b3530819ce8, event_type=packet-count-lost, summary_type=base, summary_window=0

Error: Retried 1 times. Last failure was timeout: timed out

[root@ps-dashboard esmond]# du -h /var/lib/cassandra/data/esmond/

47G     /var/lib/cassandra/data/esmond/raw_data

4.0K    /var/lib/cassandra/data/esmond/stat_aggregations

9.9G    /var/lib/cassandra/data/esmond/rate_aggregations

13G     /var/lib/cassandra/data/esmond/base_rates

69G     /var/lib/cassandra/data/esmond/

[root@ps-dashboard esmond]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup-lv_root

                       95G   82G  8.1G  91% /

tmpfs                 3.9G  4.0K  3.9G   1% /dev/shm

/dev/sda1             477M   99M  353M  22% /boot

At the time of the "timeout" as I watch, the disk reaches 100% utilization.  It appears to me that during the deletion of rows, Cassandra/Esmond uses chunks of disk space to store temporary data, and flushes that data.  During the process the disk utilization varies up and down from 91% to 100% until it finally reaches full and the timeout error occurs.

At the end of the failed attempt, even if I restart cassandra, the disk space utilization is approximately what it was before the failed run.  

So, without enough disk space to finish the ps_remove_data.py script, it would appear to me, I have two options.  Delete all my data and start over with a clean database, or shut the machine down and allocate more space to it (it's a VM, but I can't add the space "hot").

Before I take one of those approaches, does anyone else have other ideas or thoughts?

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047