perfSONAR User Q&A and Other Discussion

[Luke Whitworth <>]

Hi all,

 

We run approx. 15 testpoints on site on Raspberry Pi 4’s with 2GB RAM on them.  Biggest issue we’ve had historically was SD card burnout or, once we got past that by using overlayfs, RAM exhaustion.  With the release of ver 5 we hit the ram constraint even harder, crashing the Pi’s a lot, and have been forced to reconsider how we do this with the Pi’s.

 

To that end I started to look at whether we could containerise the testpoint for the Pi’s, allowing us to keep OS level stuff writing leisurely to the SD card but to have testpoint related things going to ramdisk.  I’ve managed to achieve a small amount of success so felt that I’d share for others who might find it useful.

 

Warning:  This is no doubt a huge mess, there’s probably much better ways to do this and running the container throws errors repeatedly such as:

 

--- Logging error ---

Traceback (most recent call last):

  File "/usr/lib/python3.8/logging/handlers.py", line 934, in emit

    self.socket.send(msg)

OSError: [Errno 9] Bad file descriptor

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File "/usr/lib/python3.8/logging/handlers.py", line 855, in _connect_unixsocket

    self.socket.connect(address)

FileNotFoundError: [Errno 2] No such file or directory

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File "/usr/lib/python3.8/logging/handlers.py", line 937, in emit

    self._connect_unixsocket(self.address)

  File "/usr/lib/python3.8/logging/handlers.py", line 866, in _connect_unixsocket

    self.socket.connect(address)

FileNotFoundError: [Errno 2] No such file or directory

Call stack:

  File "/usr/lib/pscheduler/daemons/archiver", line 1134, in <module>

    while main_program():

 File "/usr/lib/pscheduler/daemons/archiver", line 1049, in main_program

    log.info("Warm boot; restarting.")

  File "/usr/lib/python3/dist-packages/pscheduler/log.py", line 357, in info

    self.__log(self.logger, INFO, format, *args)

  File "/usr/lib/python3/dist-packages/pscheduler/log.py", line 343, in __log

    logger.log(level, chunk)

Message: 'Warm boot; restarting.'

Arguments: ()

 

That said, the tests are running and logging off to our opensearch instance, and the host has available ram still:

 

The docker build file can be found at:  https://github.com/lukewhitworth/perfsonar-tp-docker, I’ve built this and published it to our internal repo for docker images for the Pi’s to pull it down.

 

Then what I’ve done is I’ve edited the docker daemon file on the Pi (/etc/docker/daemon.json) to:

 

{

  "data-root": "/docker"

}

 

And I’ve got a tmpfs at /docker/volumes, e.g. something like (gotta persist this to fstab still):

  sudo mount -t tmpfs -o rw,size=1280M tmpfs /docker/volumes

 

Then it’s just:

docker run -d --name testpoint --network host our-internal-docker-repo/perfsonar-testpoint

 

Like I say, very early days on this, at the point where I’m going to leave it running for a few days and hope that the host stays up and the tests continue….we’ll see!

 

Cheers,

 

Luke