perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Alan Whinery writes:

I will investigate whether pscheduler can be optimized to work satisfactorily a small platform. Of Beaglebone Black rev C., I have 50  (on which bwctl-testpoint worked perfectly).  There are questions to be answered still, when resources are non-plentiful, as to the facility of Apache HTTPD server versus something lighter, as to Postgres versus sqlite, or other reductions or optimizations.

Here's the insight I can give you on the subject:

Memory.  You should be able to make pScheduler work on a small-ish platform but not a tiny one.  I’ve done all of my development and testing on dual-core, 2GB VMs since day one and have yet to bump the ceiling.  Those systems do have swap, but I never see much of it being used and have turned it off with no ill effects.  In an idle state, memory usage is about 750 MB.  1 GB is tight but workable, 2 GB leaves lots of breathing room.  If the workload is periodic throughput/latency/RTT, the system isn’t going to sink.  Large quantities of streaming latency are problematic because the way powstream works isn’t particularly resource-efficient.  (We’re in the very early stages of looking at better ways to do it.)

HTTP Server.  The pScheduler API is WSGI-based and doesn’t require any special Apache features, so nginx and uWSGI would be a low-effort change.  I’d have no problem making the switch, but if it’s going to be a wholesale replacement, we’d have to see how that impacts anything else that use Apache.

Database.  pScheduler makes extensive use of PostgreSQL’s features to make the innards work; it isn’t a simple table store.  It could probably be ported to any of the other full-featured RDBMSes, but they’re just as heavy and there wouldn’t be much benefit in doing it.  Using PgSQL this way saved a ton of development time and produced a more-robust, more-reliable product than rolling the code to do the same things by hand.  SQLite’s great but is, as its author says, a competitor for fopen().  We’re using PgSQL in its default configuration, which is fairly conservative, but it can be tuned down a bit more and will probably run pScheduler just fine.

Weight.  BWCTL was very light, no question, but it wasn’t going to support the kind of features we were being asked to provide.  pScheduler was designed to satisfy that demand, and it was a result of give and take among a lot of variables, weight being just one of them.  Going heavier wasn’t taken lightly, but I think the change we made should hold us for a very long time.  BWCTL is still a viable program for what it does and can still be fielded in tightly-constrained systems; we’re just no longer maintaining it.

--Mark