perfSONAR User Q&A and Other Discussion

[Hyojoon Kim <>]

Hello,

We have a Cubox-i4Pro with perfSONAR installed (aka, perfCube), using the pre-made image (perfCube-3.4.0.img) with a 16G microSD card. 

We run OWAMP measurements, and we are having some spikes of packet loss time to time (sometimes upto 25% packet loss). Whenever there is a spike, it seems to happen together with a high IO activity and long write wait time. For example, we had a 23% packet loss between 8:10pm-8:20pm, and below is the result of command "iostat -x -t 120” around that time. As you can see, there is a huge spike of w_await, up to 9 seconds. 

This seems to happen with all three perfCube boxes we have. It varies a bit between boxes, but it seems there are 1— 3 spikes like these every 12 hours. 

Before I really track what the problematic process is, etc, I want to just check: Is this a known problem, or was there anything like this before in perfCube boxes? 

Thanks,

Joon 

09/02/2015 08:13:11 PM

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           1.77    0.00    1.54    0.74    0.00   95.95

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

mmcblk0           0.00     6.49    0.00    7.09     0.00    57.97    16.35     3.26  459.76    0.00  459.76   7.07   5.01

09/02/2015 08:15:11 PM

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           1.50    0.00    1.57   16.22    0.00   80.71

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

mmcblk0           0.00     5.53    0.00    6.64     0.00    51.23    15.43    33.57 3268.47    0.00 3268.47  74.21  49.29

09/02/2015 08:17:11 PM

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           2.24    0.00    1.58   25.48    0.00   70.70

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

mmcblk0           0.00     5.57    0.00    6.48     0.00    52.23    16.11    49.66 9489.77    0.00 9489.77 120.91  78.39

09/02/2015 08:19:11 PM

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           1.74    0.00    1.43   18.21    0.00   78.62

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

mmcblk0           0.00     5.93    0.00    5.77     0.00    50.30    17.45    50.01 8292.04    0.00 8292.04 124.84  71.99

09/02/2015 08:21:11 PM

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           1.99    0.00    1.66   11.45    0.00   84.91

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

mmcblk0           0.00     5.69    0.00    7.40     0.00    57.07    15.42    50.61 7134.64    0.00 7134.64  69.05  51.09

09/02/2015 08:23:11 PM

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           1.76    0.00    1.49    0.62    0.00   96.13

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

mmcblk0           0.00     5.87    0.00    6.75     0.00    54.43    16.13     1.44  213.38    0.00  213.38   5.67   3.83

-Joon