perfSONAR User Q&A and Other Discussion

[Muhammad Tayyab <>]

Hi Valentin,

I'm not able to connect with  Postgres database, would you please assist me 
how I can grab the output from the PS node.

I have attached disk utilization logs. 

Regards,
Tayyab


-----Original Message-----
From:  
[] On Behalf Of Valentin Vidic
Sent: Sunday, May 12, 2019 1:58 PM
To: 
Subject: Re: [perfsonar-user] perfSONAR nodes are down due to Memory 
utilization issue

On Sun, May 12, 2019 at 10:35:37AM +0000, Muhammad Tayyab wrote:
> Thank you for your quick response, Please find attached outputs from 
> two different PS nodes. Thanks

Looks like postgresql and pscheduler are using most of the memory.

You can send the postgresql query status for:

  SELECT * FROM pg_stat_activity

and also disk utilization with:

  iostat -xz 1 10

--
Valentin Vidic
Computer Systems Engineer - Expert
Department of Computer Infrastructure and Services Croatian Academic and 
Research Network - CARNET Josipa Marohnica 5, HR-10000 Zagreb, Croatia
tel: +385 1 6661 714, fax. +385 1 6661 635
gsm: +385 91 2480 919
www.CARNet.hr
 14:51:09(801):[root@ps-pul-h4309b ~]#   iostat -xz 1 10
 14:51:09(854):Linux 3.10.0-957.10.1.el7.x86_64 (ps-pul-h4309b.kaust.edu.sa)  
 05/12/2019      _x86_64_        (4 CPU)
 14:51:09(855):
 14:51:09(879):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:09(883):           8.01    0.00    8.17   55.25    0.00   28.56
 14:51:09(883):
 14:51:09(892):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:09(903):sda              21.34    22.94  270.59   92.13 19850.13  
1067.44   115.34     0.09    0.24   10.26    9.33   2.75  99.61
 14:51:09(913):dm-0              0.00     0.00  275.72   94.95 19782.99  
1010.01   112.19     2.94    7.94   11.64   34.61   2.69  99.60
 14:51:09(922):dm-1              0.00     0.00   16.78   14.35    67.12    
57.41     8.00     1.23   39.54  121.24  191.46   6.41  19.97
 14:51:09(932):dm-2              0.00     0.00    0.00    0.00     0.00     
0.00    68.36     0.00   10.75   11.12    2.50  10.41   0.00
 14:51:09(932):
 14:51:10(842):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:10(850):          31.41    0.00   18.59   37.19    0.00   12.81
 14:51:10(850):
 14:51:10(873):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:10(897):sda               7.00     0.00  191.00   14.00  2324.00   
168.00    24.31     2.78   13.13   13.12   13.36   4.88 100.00
 14:51:10(914):dm-0              0.00     0.00  190.00   10.00  2292.00   
168.00    24.60     2.42   11.68   11.31   18.70   5.00 100.00
 14:51:10(930):dm-1              0.00     0.00    8.00    0.00    32.00     
0.00     8.00     2.85  356.00  356.00    0.00  44.50  35.60
 14:51:10(930):
 14:51:13(088):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:13(092):          29.11    0.00   19.75   37.97    0.00   13.16
 14:51:13(092):
 14:51:13(101):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:13(112):sda               0.00    67.00  135.00  251.00  1640.00  
3669.00    27.51    67.81   71.18   19.44   99.00   2.59 100.00
 14:51:13(123):dm-0              0.00     0.00  135.00  461.00  1640.00  
4878.50    21.87    76.80   57.83   19.43   69.08   1.68 100.00
 14:51:13(123):
 14:51:13(126):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:13(128):          30.38    0.00   18.73   43.54    0.00    7.34
 14:51:13(129):
 14:51:13(137):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:13(146):sda               0.00    56.00   14.00  330.00   108.00  
4218.50    25.15   143.22  465.41  192.43  476.99   2.91 100.00
 14:51:13(155):dm-0              0.00     0.00   14.00  307.00   108.00  
4157.00    26.57   169.30  572.25  192.43  589.57   3.12 100.00
 14:51:13(156):
 14:51:13(852):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:13(865):          30.10    0.00   18.62   33.42    0.00   17.86
 14:51:13(865):
 14:51:13(886):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:13(904):sda               0.00     2.00   71.00  118.00   984.00  
1512.00    26.41    25.53  259.59   37.82  393.03   5.29 100.00
 14:51:13(916):dm-0              0.00     0.00   72.00   35.00   988.00   
364.00    25.27    30.82  549.91   37.31 1604.40   9.35 100.00
 14:51:13(916):
 14:51:14(843):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:14(849):          31.23    0.00   19.90   34.01    0.00   14.86
 14:51:14(849):
 14:51:14(865):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:14(877):sda               0.00     0.00  221.00    9.00  2472.00    
99.50    22.36     2.42   10.57   10.65    8.56   4.35 100.00
 14:51:14(889):dm-0              0.00     0.00  220.00    6.00  2468.00    
99.50    22.72     2.42   10.75   10.70   12.67   4.42 100.00
 14:51:14(889):
 14:51:15(845):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:15(855):          32.24    0.00   18.64   33.00    0.00   16.12
 14:51:15(855):
 14:51:15(877):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:15(893):sda               0.00     0.00  204.00   28.00  2740.00   
293.00    26.15     2.54   10.95   10.64   13.21   4.31 100.00
 14:51:15(909):dm-0              0.00     0.00  204.00   18.00  2740.00   
293.00    27.32     2.54   11.45   10.64   20.56   4.50 100.00
 14:51:15(911):
 14:51:16(853):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:16(866):          31.06    0.00   18.18   33.84    0.00   16.92
 14:51:16(866):
 14:51:16(886):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:16(903):sda               0.00     0.00  206.00   12.00  2288.00    
98.00    21.89     2.28   10.44   10.42   10.67   4.59 100.00
 14:51:16(916):dm-0              0.00     0.00  207.00    8.00  2304.00    
98.00    22.34     2.28   10.58   10.37   16.00   4.65 100.00
 14:51:16(916):
 14:51:17(846):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:17(858):          30.63    0.00   18.99   32.15    0.00   18.23
 14:51:17(858):
 14:51:17(877):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:17(895):sda               0.00     0.00  173.00   25.00  2452.00   
257.00    27.36     2.53   12.77   12.93   11.64   5.05 100.00
 14:51:17(907):dm-0              0.00     0.00  171.00   18.00  2428.00   
257.00    28.41     2.53   13.38   13.08   16.17   5.29 100.00
 14:51:17(907):
 14:51:18(847):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:18(855):          30.65    0.00   19.35   31.16    0.00   18.84
 14:51:18(856):
 14:51:18(873):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:18(888):sda               0.00     0.00  224.00   22.00  3368.00   
184.00    28.88     2.73   11.04   11.00   11.45   4.07 100.00
 14:51:18(900):dm-0              0.00     0.00  225.00   16.00  3376.00   
203.50    29.71     2.73   11.27   10.96   15.69   4.15 100.00
 14:51:18(901):
 14:51:20(985):Display all 119 possibilities? (y or n)
 14:51:23(345):[root@ps-clc2-g3057c ~]# 
 14:51:23(845):[root@ps-clc2-g3057c ~]# 
 14:51:23(878):[root@ps-clc2-g3057c ~]# 
 14:51:23(912):[root@ps-clc2-g3057c ~]# 
 14:51:23(944):[root@ps-clc2-g3057c ~]# 
 14:51:23(978):[root@ps-clc2-g3057c ~]# 
 14:51:24(010):[root@ps-clc2-g3057c ~]# 
 14:51:24(043):[root@ps-clc2-g3057c ~]# 
 14:51:24(077):[root@ps-clc2-g3057c ~]# 
 14:51:24(108):[root@ps-clc2-g3057c ~]# 
 14:51:24(142):[root@ps-clc2-g3057c ~]# 
 14:51:24(175):[root@ps-clc2-g3057c ~]# 
 14:51:24(209):[root@ps-clc2-g3057c ~]# 
 14:51:25(353):[root@ps-clc2-g3057c ~]#  iostat -xz 1 10
 14:51:25(618):Linux 3.10.0-957.10.1.el7.x86_64 (ps-clc2-g3057c.kaust.edu.sa) 
 05/12/2019      _x86_64_        (4 CPU)
 14:51:25(625):
 14:51:25(631):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:25(636):          14.65    0.00    4.36   54.14    0.00   26.84
 14:51:25(636):
 14:51:25(647):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:25(659):sda              16.69    24.01  327.92   88.30 26321.83  
1045.38   131.50     1.18    2.82    3.33    0.95   2.40  99.82
 14:51:25(668):dm-0              0.00     0.00  333.67   94.55 26275.96   
997.28   127.38     1.59    3.71    4.66    0.34   2.33  99.82
 14:51:25(678):dm-1              0.00     0.00   11.46   12.02    45.83    
48.07     8.00     0.63   26.98  126.57  137.03   6.23  14.63
 14:51:25(687):dm-2              0.00     0.00    0.00    0.00     0.00     
0.00    67.58     0.00   18.48   19.13    2.50  18.39   0.00
 14:51:25(687):
 14:51:26(583):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:26(594):          26.01    0.00    3.79   40.91    0.00   29.29
 14:51:26(594):
 14:51:26(617):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:26(631):sda               6.00     0.00  747.00   13.00 94356.00   
140.50   248.68     3.31    4.46    4.31   12.62   1.28  97.60
 14:51:26(643):dm-0              0.00     0.00  751.00    9.00 94344.00   
162.00   248.70     3.41    4.59    4.42   18.22   1.28  97.60
 14:51:26(644):
 14:51:27(583):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:27(590):          26.01    0.00    2.53   43.69    0.00   27.78
 14:51:27(590):
 14:51:27(620):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:27(633):sda               2.00     0.00  792.00    6.00 100912.00    
54.50   253.05     4.24    2.65    2.63    5.67   1.25 100.00
 14:51:27(644):dm-0              0.00     0.00  800.00    3.00 100948.00    
33.00   251.51     4.28    2.71    2.68   11.33   1.25 100.00
 14:51:27(645):
 14:51:28(585):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:28(598):          25.94    0.00    5.04   46.10    0.00   22.92
 14:51:28(599):
 14:51:28(616):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:28(633):sda               4.00     0.00  719.00    7.00 88984.00    
57.50   245.29     4.41    7.69    7.55   21.71   1.38 100.00
 14:51:28(642):dm-0              0.00     0.00  721.00    5.00 88972.00    
57.50   245.26     4.54    7.87    7.71   30.40   1.38 100.00
 14:51:28(642):
 14:51:29(584):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:29(597):          26.08    0.00    2.78   44.05    0.00   27.09
 14:51:29(597):
 14:51:29(626):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:29(646):sda               0.00     0.00  823.00    0.00 104104.00     
0.00   252.99     4.07    4.42    4.42    0.00   1.22 100.00
 14:51:29(660):dm-0              0.00     0.00  821.00    0.00 103992.00     
0.00   253.33     4.07    4.43    4.43    0.00   1.22 100.00
 14:51:29(660):
 14:51:30(575):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:30(583):          26.01    0.00    3.28   33.59    0.00   37.12
 14:51:30(583):
 14:51:30(614):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:30(634):sda              17.00     3.00  467.00   43.00 55196.00   
393.00   218.00     3.72    9.81    9.55   12.60   1.96 100.00
 14:51:30(648):dm-0              0.00     0.00  486.00   33.00 55260.00   
393.00   214.46     4.18   10.44   10.03   16.42   1.93 100.00
 14:51:30(649):
 14:51:31(584):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:31(598):          26.13    0.00    3.27   38.19    0.00   32.41
 14:51:31(598):
 14:51:31(631):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:31(651):sda               5.00     4.00  611.00   14.00 73632.00    
78.50   235.87     3.40    5.22    5.10   10.64   1.60 100.00
 14:51:31(667):dm-0              0.00     0.00  614.00   15.00 73672.00    
78.50   234.50     3.51    5.45    5.34    9.93   1.59 100.00
 14:51:31(671):
 14:51:32(571):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:32(574):          26.01    0.00    2.53   48.23    0.00   23.23
 14:51:32(574):
 14:51:32(583):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:32(592):sda               0.00     0.00  657.00    2.00 81140.00     
8.00   246.28     5.04    7.32    7.35    0.00   1.52 100.00
 14:51:32(602):dm-0              0.00     0.00  658.00    2.00 81148.00     
8.00   245.93     5.04    7.31    7.33    0.00   1.52 100.00
 14:51:32(602):
 14:51:33(579):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:33(589):          25.75    0.00    2.75   41.25    0.00   30.25
 14:51:33(589):
 14:51:33(614):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:33(631):sda               7.00     0.00  543.00   16.00 65696.00   
214.00   235.81     5.10    9.43    8.73   32.88   1.79 100.00
 14:51:33(646):dm-0              0.00     0.00  552.00    9.00 65816.00   
214.00   235.40     5.38    9.90    9.11   58.44   1.78 100.00
 14:51:33(646):
 14:51:34(579):avg-cpu:  %user   %nice %system %iowait  %steal   %idle
 14:51:34(592):          25.88    0.00    2.26   38.94    0.00   32.91
 14:51:34(592):
 14:51:34(613):Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    
wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
 14:51:34(631):sda              12.00    25.00  402.00   51.00 44612.00   
348.00   198.50    12.52   11.86   12.28    8.53   2.21 100.00
 14:51:34(648):dm-0              0.00     0.00  409.00  204.00 44468.00  
1740.00   150.76    14.03    9.40   12.97    2.25   1.63 100.00
 14:51:34(650):