perfSONAR User Q&A and Other Discussion

["Whitworth, Luke" <>]

One final bit on this, everything seems quite happy with ASD running, only thing is the tables run and archiving in the pscheduler DB seem to grow and grow:

 

    table_schema    |       table_name        | row_estimate |   total    |   index    |   toast    |   table   

--------------------+-------------------------+--------------+------------+------------+------------+------------

public             | run                     |        13214 | 59 MB      | 40 MB      | 5576 kB    | 13 MB

public             | archiving               |        13140 | 17 MB      | 7784 kB    | 8192 bytes | 9296 kB

 

Is there a safe way to truncate these periodically……or is there a flag to bypass them completely?

 

Cheers,

 

Luke

 

From: <> On Behalf Of Whitworth, Luke
Sent: 11 September 2020 12:17
To: Mark Feit <>;
Subject: RE: [perfsonar-user] SD card lifespan

 

Just to say thanks for the help with this.  Spent the morning looking into overlayfs and had a nice enough solution, then stumbled across:

 

https://wiki.archlinux.org/index.php/Anything-sync-daemon

 

Compiled it, told it to look after /var and seemingly life is now very happy.

 

Much appreciate the help and comments from all.  Have great weekends!

 

Luke

 

From: Mark Feit <>
Sent: 09 September 2020 21:56
To: Whitworth, Luke <>;
Subject: Re: [perfsonar-user] SD card lifespan

 

Whitworth, Luke writes:

 

Seems though that postgres, specifically the pscheduler db, are still generating continual writes so are therefore our current suspect.  I’ve been looking at having just a simple central postgres server to point the Pi’s to, but am hitting issues in so much as if I specify a different DB name during the initial install it still seems to create a pscheduler database that’s got the schema in it, and an empty db with my chosen name. 

… 

Just wondering if what I’m trying to achieve is even possible, or if I just need to look at a way to tune the write level of postgres maybe…….

 

As you discovered in a subsequent post, there are external programs the database needs to execute on the local system.  There are architectural reasons for it that will be a fact of life until there’s a compelling reason to change it.

 

perfSONAR has a split constituency with opposing needs: network operators that field robust hardware to do high-speed testing and campuses wanting to do mass deployments at low cost, which means less-capable systems.  Our choice of architecture tried to split the difference but, ultimately, favors the network operators who provide most of the labor.  Fortunately, small systems have become a lot more capable than they were a few years ago, and that helps close the gap.  We don’t anticipate any architectural changes

 

Potentially but of course that would incur additional cost, I’m currently looking at the network booting options too.  There’s a few ways to achieve this I would guess but the ideal for me is to have the SD card handling the OS etc. with all the heavy stuff off on remote / alternate storage.

 

I can offer a few suggestions:

 

Evaluate the flash you’re using to make sure it’s up to the job.  Inexpensive flash isn’t built with components that can endure a lot of write cycles, and the controllers in the vast majority of cards don’t do anything to maximize them.  An application like this one will effectively blow a hole in the card by writing the same blocks repeatedly.  There are cards that do wear leveling, but you have to go find them and verify with the manufacturer that they actually do it.  (Look for cards designed for surveillance, industrial or embedded applications.)   Some brands list theirs as high-endurance but really mean they have more environmental tolerance.  On this side of the pond, the premium for a card that does wear leveling is a few dollars.  There also cards available with SLC flash that can take many more write cycles, but at their price point, it’s cheaper to strap an external spinning USB drive to the Pi and be done with it.

 

If you’ve got RAM to spare and don’t mind the system losing state across reboots, use AuFS to mount a RAM disk atop the PostgreSQL data.  That would keep all of the database activity in memory.   A less-extreme version of this would be to set a high commit time on the filesystem that holds /var since the contents of that are largely expendable.

 

Remote storage is also a possibility; an iSCSI or NFS volume for /var would probably solve the problem, or you could just remote boot the entire thing.   The hazard here is that without a separate interface dedicated to storage, the perfSONAR node will have a dependency on the network it measures, which isn’t a good thing.  Also, depending on what’s being measured, the extra traffic may color some of the measurements.  I’d do this as a last resort.

 

HTH.

 

--Mark