perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi Johann,

Thanks for reporting this. I am going to keep this message short, and provide more details in a separate thread in case people aren’t paying attention to this one. We have indeed confirmed the behavior with the docker container and have uploaded new images for the latest, systemd, v4.4.4 and v4.4.4-systemd tags. The new images move memcached to localhost as intended. Thanks again for sharing this.

Andy

On September 1, 2022 at 6:36:49 AM, Johann Hugo () wrote:

Hi all

A warning to people running a testpoint docker image.

Memcached is exposed to the Internet and the container may be the victim of an amplification attack. The image does not have any firewall configured and it does not include the perfsonar-toolkit-security package. When you use macvlan networking, the only ip stack is the one in the container, so the firewall rules on the host are never called. You have to configure the iptables rules inside the container itself.

Regards
Johann

On Wed, Aug 24, 2022 at 5:51 PM Johann Hugo <> wrote:

No, that did not solve it. The problem is still there.  CPU utilization for memcached goes up and just stays there.

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  67641 995       20   0   16.3g  12208   2792 S 400.0   0.0 233:11.58 memcached

On Wed, Aug 24, 2022 at 4:37 PM Johann Hugo <> wrote:

Hi

Looks like the CPU utilization for memcached goes up and just stays there (400%). If I down/up memcached, then the CPU utilization goes down to about 6%

Host: Ubuntu 20.04.4 LTS

Docker image: perfsonar/testpoint:latest. (Started with docker-compose)

memcached: VERSION 1.4.15

Top

%Cpu(s):  0.1 us,  6.2 sy,  0.0 ni, 93.4 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
MiB Mem : 128743.4 total, 117899.7 free,   8877.0 used,   1966.7 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used. 118600.7 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 3292 995       20   0  363996  24208   2832 S 400.3   0.0   2736:33 memcached

I can trigger it by running some iperf2 tests manually.

Adding this option to memcached seems to fix it.

vi /etc/sysconfig/memcached

  OPTIONS="-o hashpower=31"

I think it's related to this issue:

https://github.com/memcached/memcached/issues/855

Regards

Johann

--

SANReN Engineer

South African National Research Network (SANReN)

National Integrated Cyber Infrastructure System (NICIS)

CSIR NextGen Enterprises and Institutions Cluster

Office: 012 841 2066Email: , Website: www.sanren.ac.za / www.csir.co.za

--

SANReN Engineer

South African National Research Network (SANReN)

National Integrated Cyber Infrastructure System (NICIS)

CSIR NextGen Enterprises and Institutions Cluster

Office: 012 841 2066Email: , Website: www.sanren.ac.za / www.csir.co.za

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user