perfSONAR User Q&A and Other Discussion

["Whitworth, Luke" <>]

Thanks to both Mark and Peter for their responses, lots of good food for thought here, and yes I completely understand that we’re making these issues by trying to do it as cheaply as possible 😉

 

I will give the options a go and let you know how I get on.  Certainly “If you’ve got RAM to spare and don’t mind the system losing state across reboots, use AuFS to mount a RAM disk atop the PostgreSQL data” might be the best of both worlds without having to rely hugely on the network for keeping the network monitoring running!

 

Much appreciate the replies (and the project itself),

 

Luke

 

From: Peter Lambrechtsen <>
Sent: 09 September 2020 22:42
To: Mark Feit <>
Cc: Whitworth, Luke <>;
Subject: Re: [perfsonar-user] SD card lifespan

 

Also worth investigating alternative SBC hardware and eMMC storage. 

I’ve had success with Odroids and eMMC storage running in probes now for 3+ years. 

 

I was involved in a pilot of 12 Odroid C2s for broadband testing and apart from bricking one due to filling the storage and it tried to automatically patch it they have been running for over 3 years now without issue.

 

The C2s came with 2gb of ram and are able to saturate a 1GB link with iPerf with 50+ threads on TCP and UDP, and that was impressive imho.

 

They now make a N2 which is even faster not that I have tested it as our C2s worked fine.

 

The main thing we did was to get eMMC storage rather than SD.

Also I use ramfs for most things in /var/tmp and /var/log that need to change including capturing pcaps as long as they are small enough then copy to local storage. 

I was planning to move to overlayfs or aufs but scripting copies of files from ramfs seemed to work well enough for us.

 

Also the clearfog SBCs looked good but we were still happy on the C2.

 

YMMV but I love them :)

 

On Thu, 10 Sep 2020 at 08:56, Mark Feit <> wrote:

 

 

 

 

Whitworth, Luke writes:

 

 

 

 

 

Seems though that postgres, specifically the pscheduler db, are still generating continual writes so are therefore our current suspect.  I’ve been looking at having just a simple central postgres server to point

the Pi’s to, but am hitting issues in so much as if I specify a different DB name during the initial install it still seems to create a pscheduler database that’s got the schema in it, and an empty db with my chosen name. 

 

… 

 

Just wondering if what I’m trying to achieve is even possible, or if I just need to look at a way to tune the write level of postgres maybe…….

 

 

 

As you discovered in a subsequent post, there are external programs the database needs to execute on the local system.  There are architectural reasons for it that will be a fact of life until there’s a compelling reason to change it.

 

 

 

perfSONAR has a split constituency with opposing needs: network operators that field robust hardware to do high-speed testing and campuses wanting to do mass deployments at low cost, which means less-capable systems.  Our choice of architecture

tried to split the difference but, ultimately, favors the network operators who provide most of the labor.  Fortunately, small systems have become a lot more capable than they were a few years ago, and that helps close the gap.  We don’t anticipate any architectural

changes

 

 

 

Potentially but of course that would incur additional cost, I’m currently looking at the network booting options too.  There’s a few ways to achieve this I would guess but the ideal for me is to have the SD card

handling the OS etc. with all the heavy stuff off on remote / alternate storage.

 

 

 

I can offer a few suggestions:

 

 

 

Evaluate the flash you’re using to make sure it’s up to the job.  Inexpensive flash isn’t built with components that can endure a lot of write cycles, and the controllers in the vast majority of cards don’t do anything to maximize them. 

An application like this one will effectively blow a hole in the card by writing the same blocks repeatedly.  There are cards that do wear leveling, but you have to go find them and verify with the manufacturer that they actually do it.  (Look for cards designed

for surveillance, industrial or embedded applications.)   Some brands list theirs as high-endurance but really mean they have more environmental tolerance.  On this side of the pond, the premium for a card that does wear leveling is a few dollars.  There also

cards available with SLC flash that can take many more write cycles, but at their price point, it’s cheaper to strap an external spinning USB drive to the Pi and be done with it.

 

 

 

If you’ve got RAM to spare and don’t mind the system losing state across reboots, use AuFS to mount a RAM disk atop the PostgreSQL data.  That would keep all of the database activity in memory.   A less-extreme version of this would be

to set a high commit time on the filesystem that holds /var since the contents of that are largely expendable.

 

 

 

Remote storage is also a possibility; an iSCSI or NFS volume for /var would probably solve the problem, or you could just remote boot the entire thing.   The hazard here is that without a separate interface dedicated to storage, the perfSONAR

node will have a dependency on the network it measures, which isn’t a good thing.  Also, depending on what’s being measured, the extra traffic may color some of the measurements.  I’d do this as a last resort.

 

 

 

HTH.

 

 

 

--Mark

 

 

 

 

--

To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user