perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Marian Babik and John Hill write:

    I noticed that the toolkit web page shows only 35 entries, but the 
    auto-URL for the host shows a lot more hosts to be tested as the node 
    was added to the ATLAS and LHCb meshes - unsure when exactly this 
    happened...

This would be a good thing to run down since a change like that will put 
significantly more load on a system.  ATLAS and LHC are running the largest 
meshes I know of and are often where we find the hairy edges of what 
perfSONAR can do.
    
    > On Jan 10, 2019, at 4:26 PM, John Hill 
<>
 wrote:
    > 
    >   The problem showed up about 24 hours after the host was updated to 
4.1.5 - is this new version more resource hungry?

It shouldn't be.    The last release to introduce anything new was 4.1, and 
as pScheduler goes, that was almost a non-event.  Everything since has been 
bugfixes and very minor improvements.

    >  I see quite a few errors in /var/log/pscheduler/pscheduler.log of the 
type
    > 
    > Failed to post run for result: Database connection pool exhausted. 
Unable to get connection after 60 attempts.

Some under-the-hood insight:  One of the internal parts of pScheduler is 
called the runner, which is responsible for overseeing the execution of 
measurements and storing the results.  It's multithreaded and maintains a 
pool of connections to the PostgreSQL database that the threads can acquire, 
use and return when needed.  The size of that pool depends on the maximum 
number of connections available on the database server.  Currently, we have 
that set at 500 and the runner takes half to leave some for other programs 
that use it.   The pool will spend up to a minute waiting for a connection to 
be available, making it more resilient in situations where demand has spiked 
enough to exhaust it.

A large-enough workload can make exhaustion of the pool a regular event.  
Most measurements don't contribute too much because they're either relatively 
infrequent (RTT, trace) or self-regulating (throughput, which doesn't run 
more than one at a time).  Because it produces a continuous stream of 
results, each streaming latency task causes its corresponding thread in the 
runner to be almost always in possession of a connection from the pool.  As 
Marian pointed out, the process of getting measurements stored is I/O-bound, 
so it's entirely possible that the database isn't going fast enough  to 
prevent pool exhaustion.  Raising the connection limit might help, but the 
other possibility is that a system problem is reducing I/O throughput enough 
that threads hold onto connections longer than they usually would, exhausting 
the pool.

The lowest-hanging fruit would be to check the system's general health, 
especially if it's an old machine.  Make sure the kernel isn't complaining 
about I/O retires or memory problems.  Check that the swap space isn't in 
heavy use.  If the disk is RAIDed, make sure the controller isn't spending a 
lot of time reconstructing data because of a failed disk that wasn't replaced.

 Hope that helps.

--Mark