perfSONAR User Q&A and Other Discussion

[Marian Babik <>]

Hi,
following 4.0.2 upgrade, a latency node (SLC6) with uptime of just 22 hours, 
so recently rebooted, which is configured to run 128 one-directional latency 
tests gives me:
[root@perfsonar-lt ~]# ps waxxxu | grep "powstream/run" | wc -l
496
[root@perfsonar-lt ~]# ps waxxxu | grep powstream | wc -l
1236

I think I should expect 128 powstream/run processes and about 4x the number 
of powstream, right ? So this looks way off.

The box has 16G RAM and there is quite some memory pressure in effect: 
[root@perfsonar-lt ~]#  ps -A --no-headers  -o pcpu,rss,command | grep 
"powstream"  | awk '{cpu += $1; rss += $2} END {print rss}'
9276304
[root@perfsonar-lt ~]# ps -A --no-headers  -o pcpu,rss,command | grep 
"cassandra"  | awk '{cpu += $1; rss += $2} END {print rss}'
4685812
[root@perfsonar-lt ~]# ps -A --no-headers  -o pcpu,rss,command | grep httpd  
| awk '{cpu += $1; rss += $2} END {print rss}'
1258452
[root@perfsonar-lt ~]# ps -A --no-headers  -o pcpu,rss,command | grep httpd24 
 | awk '{cpu += $1; rss += $2} END {print rss}’
1057600

Trying to understand what’s going on gives me (we have separate meshes for 
IPv4 and IPv6, so the reported ratio is about right):
[root@perfsonar-lt ~]# ps -ef | grep "/usr/bin/powstream" | grep " -6 " | wc 
-l
144
[root@perfsonar-lt ~]# ps -ef | grep "/usr/bin/powstream" | grep " -4 " | wc 
-l
597

I should have just one source as one-direction only is tested:
[root@perfsonar-lt ~]# ps -ef | grep "/usr/bin/powstream" | grep " -6 " | awk 
'{print $23}' | sort | uniq -c
    142 perfsonar-lt
[root@perfsonar-lt ~]# ps -ef | grep "/usr/bin/powstream" | grep " -4 " | awk 
'{print $23}' | sort | uniq -c
      3 <IP of perfsonar-lt>
    591 perfsonar-lt
      3 [perfs-wig-lt]:861
so this looks good (perfs-wig-lt is configured manually), for destinations I 
get:

[root@perfsonar-lt ~]# ps -ef | grep "/usr/bin/powstream" | grep " -4 " | awk 
'{print $24}' | sort | uniq | wc -l
100
[root@perfsonar-lt ~]# ps -ef | grep "/usr/bin/powstream" | grep " -6 " | awk 
'{print $24}' | sort | uniq | wc -l
28
so 128 is correct.

However, looking for counts per destination:
# ps -ef | grep "/usr/bin/powstream" | grep " -4 " | awk '{print $24}' | sort 
| uniq -c | sort -k 1 --numeric | tail
gives me something like this: 
     16 [host1]:861
     16 [host2]:861
     16 [host3]:861
     16 [host4]:861
...
     19 [host5]:861
     29 [host6]:861
     30 [host7 (btw. @aglt2)]:861

# ps -ef | grep <host7>  then gives me 30 powstream processes 
(/usr/bin/powstream) that all have stime within 3-4 minutes.

Any hints what’s going on ?

Thanks,
Marian

Attachment: smime.p7s
Description: S/MIME cryptographic signature