perfSONAR User Q&A and Other Discussion

[Kathy Benninger <>]

Hello!

Follow-up to my prior inquiry:

A second run of ps-migrate-restore.sh restored the archived data, which is now being displayed. Yay!

Unfortunately, some of the pscheduler services seem to need a manual restart after reboot to run cleanly:
    systemctl -l status pscheduler-scheduler
    systemctl -l status pscheduler-archiver
    systemctl -l status pscheduler-ticker
all show errors (copied below) after a reboot. pscheduler-runner starts and runs cleanly without intervention.

Since pscheduler-scheduler doesn't start, scheduled testing doesn't start. After a manual:
    systemctl restart pscheduler-scheduler
testing runs, old and new data are graphed, things look OK. Yay!

systemctl -l status pscheduler-archiver
systemctl -l status pscheduler-ticker
also show errors until they are restarted.

Thanks for any ideas on what to do. I haven't seen this problem on any of the four machines that I've previously upgraded so I'm a bit stumped. (I manage the machine remotely and, while I have a great contact there to help with hands-on, I'm trying to debug as much as I can before I need to pull him back into the fray.)

Kathy


[root@ps benninge]# systemctl -l status pscheduler-scheduler
â pscheduler-scheduler.service - pScheduler server - scheduler
   Loaded: loaded (/usr/lib/systemd/system/pscheduler-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-06-25 17:06:52 EDT; 2min 9s ago
 Main PID: 2258 (scheduler)
   CGroup: /system.slice/pscheduler-scheduler.service
           ââ2258 /usr/bin/python /usr/libexec/pscheduler/daemons/scheduler --daemon --pid-file /var/run/pscheduler-scheduler.pid --dsn @/etc/pscheduler/database/database-dsn

Jun 25 17:06:52 ps.nics.xsede.org systemd[1]: Starting pScheduler server - scheduler...
Jun 25 17:06:52 ps.nics.xsede.org systemd[1]: Started pScheduler server - scheduler.
Jun 25 17:06:52 ps.nics.xsede.org scheduler[2258]: scheduler INFO     Started
Jun 25 17:07:12 ps.nics.xsede.org scheduler[2258]: safe_run/scheduler ERROR    Program threw an exception after 0:00:19.816066
Jun 25 17:07:12 ps.nics.xsede.org scheduler[2258]: safe_run/scheduler ERROR    Exception: DatabaseError: server closed the connection unexpectedly
                                                           This probably means the server terminated abnormally
                                                           before or while processing the request.

                                                   Traceback (most recent call last):
                                                     File "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 72, in safe_run
                                                       function()
                                                     File "/usr/libexec/pscheduler/daemons/scheduler", line 809, in <lambda>
                                                       pscheduler.safe_run(lambda: main_program())
                                                     File "/usr/libexec/pscheduler/daemons/scheduler", line 764, in main_program
                                                       cursor.execute(query, args)
                                                   DatabaseError: server closed the connection unexpectedly
                                                           This probably means the server terminated abnormally
                                                           before or while processing the request.
Jun 25 17:07:12 ps.nics.xsede.org scheduler[2258]: safe_run/scheduler ERROR    Waiting 0.25 seconds before restarting
Jun 25 17:07:12 ps.nics.xsede.org scheduler[2258]: safe_run/scheduler ERROR    Restarting: ['/usr/libexec/pscheduler/daemons/scheduler', '--daemon', '--pid-file', '/var/run/pscheduler-scheduler.pid', '--dsn', '@/etc/pscheduler/database/database-dsn']
Jun 25 17:07:12 ps.nics.xsede.org scheduler[2258]: scheduler INFO     Started

[root@ps benninge]# systemctl -l status pscheduler-archiver
â pscheduler-archiver.service - pScheduler server - archiver
   Loaded: loaded (/usr/lib/systemd/system/pscheduler-archiver.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-06-25 17:06:52 EDT; 2min 19s ago
 Main PID: 2250 (archiver)
   CGroup: /system.slice/pscheduler-archiver.service
           ââ2250 /usr/bin/python /usr/libexec/pscheduler/daemons/archiver --daemon --pid-file /var/run/pscheduler-archiver.pid --dsn @/etc/pscheduler/database/database-dsn
           ââ5649 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/archiver/esmond/archive
           ââ5731 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/archiver/esmond/archive
           ââ5796 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/archiver/esmond/archive
           ââ5803 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/archiver/esmond/archive
           ââ5810 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/archiver/esmond/archive
           ââ5816 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/archiver/esmond/archive
           ââ5826 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/archiver/esmond/archive
           ââ5846 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/archiver/esmond/archive
           ââ5854 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/archiver/esmond/archive
           ââ5864 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/archiver/esmond/archive

Jun 25 17:06:52 ps.nics.xsede.org systemd[1]: Starting pScheduler server - archiver...
Jun 25 17:06:52 ps.nics.xsede.org systemd[1]: Started pScheduler server - archiver.
Jun 25 17:06:52 ps.nics.xsede.org archiver[2250]: archiver INFO     Started
Jun 25 17:07:12 ps.nics.xsede.org archiver[2250]: safe_run/archiver ERROR    Program threw an exception after 0:00:19.816605
Jun 25 17:07:12 ps.nics.xsede.org archiver[2250]: safe_run/archiver ERROR    Exception: DatabaseError: server closed the connection unexpectedly
                                                          This probably means the server terminated abnormally
                                                          before or while processing the request.

                                                  Traceback (most recent call last):
                                                    File "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 72, in safe_run
                                                      function()
                                                    File "/usr/libexec/pscheduler/daemons/archiver", line 855, in <lambda>
                                                      pscheduler.safe_run(lambda: main_program())
                                                    File "/usr/libexec/pscheduler/daemons/archiver", line 811, in main_program
                                                      [options.max_parallel])
                                                    File "/usr/lib/python2.7/site-packages/pscheduler/db.py", line 188, in query
                                                      cursor.execute(query, args)
                                                  DatabaseError: server closed the connection unexpectedly
                                                          This probably means the server terminated abnormally
                                                          before or while processing the request.
Jun 25 17:07:12 ps.nics.xsede.org archiver[2250]: safe_run/archiver ERROR    Waiting 0.25 seconds before restarting
Jun 25 17:07:12 ps.nics.xsede.org archiver[2250]: safe_run/archiver ERROR    Restarting: ['/usr/libexec/pscheduler/daemons/archiver', '--daemon', '--pid-file', '/var/run/pscheduler-archiver.pid', '--dsn', '@/etc/pscheduler/database/database-dsn']
Jun 25 17:07:12 ps.nics.xsede.org archiver[2250]: archiver INFO     Started

[root@ps benninge]# systemctl -l status pscheduler-ticker
â pscheduler-ticker.service - pScheduler server - ticker
   Loaded: loaded (/usr/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-06-25 17:06:52 EDT; 2min 24s ago
 Main PID: 2255 (ticker)
   CGroup: /system.slice/pscheduler-ticker.service
           ââ2255 /usr/bin/python /usr/libexec/pscheduler/daemons/ticker --daemon --pid-file /var/run/pscheduler-ticker.pid --dsn @/etc/pscheduler/database/database-dsn

Jun 25 17:06:52 ps.nics.xsede.org systemd[1]: Starting pScheduler server - ticker...
Jun 25 17:06:52 ps.nics.xsede.org systemd[1]: Started pScheduler server - ticker.
Jun 25 17:07:12 ps.nics.xsede.org ticker[2255]: ticker WARNING  Queue maintainer got exception server closed the connection unexpectedly
                                                        This probably means the server terminated abnormally
                                                        before or while processing the request.
Jun 25 17:07:16 ps.nics.xsede.org ticker[2255]: safe_run/ticker ERROR    Program threw an exception after 0:00:23.667070
Jun 25 17:07:16 ps.nics.xsede.org ticker[2255]: safe_run/ticker ERROR    Exception: OperationalError: terminating connection due to administrator command
                                                server closed the connection unexpectedly
                                                        This probably means the server terminated abnormally
                                                        before or while processing the request.

                                                Traceback (most recent call last):
                                                  File "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 72, in safe_run
                                                    function()
                                                  File "/usr/libexec/pscheduler/daemons/ticker", line 157, in <lambda>
                                                    pscheduler.safe_run(lambda: main_program())
                                                  File "/usr/libexec/pscheduler/daemons/ticker", line 137, in main_program
                                                    cursor.execute("SELECT ticker()")
                                                OperationalError: terminating connection due to administrator command
                                                server closed the connection unexpectedly
                                                        This probably means the server terminated abnormally
                                                        before or while processing the request.
Jun 25 17:07:16 ps.nics.xsede.org ticker[2255]: safe_run/ticker ERROR    Waiting 0.25 seconds before restarting
Jun 25 17:07:16 ps.nics.xsede.org ticker[2255]: safe_run/ticker ERROR    Restarting: ['/usr/libexec/pscheduler/daemons/ticker', '--daemon', '--pid-file', '/var/run/pscheduler-ticker.pid', '--dsn', '@/etc/pscheduler/database/database-dsn']

On 6/20/2018 2:26 PM, Kathy Benninger wrote:

Hi Andy,

I ran:

[root@ps log]# /usr/lib/perfsonar/scripts/system_environment/configure_esmond
Restarting cassandra (via systemctl):                      [  OK  ]
[root@ps log]#

about 45 minutes ago. pscheduler monitor still looks like everything is fine. Unfortunately there's still "No data available in table".

Can you think of anything else I should try?

If not, is there a way to clean out the databases and try a restore again? Or should we do a complete rebuild and try again from scratch?

Thanks for taking a look at this for us!
Kathy

On 6/20/2018 12:22 PM, Andrew Lake wrote:

Hi Kathy,

Sorry for the slow reply, this is certainly an interesting one. Try running /usr/lib/perfsonar/scripts/toolkit/system_environment/configure_esmond to see if that helps.

You provided lots of good information and looking through your pscheduler tasks, it looks like the most recent ones seem to think they are succeeding. What has me most confused is that esmond seems to be showing only “partial” results where it has the test parameters but no “event-types” filled in which means no pointers to test results (see https://ps.nics.xsede.org/esmond/perfsonar/archive/ and the empty array in each “event-types field). I am not quite sure how that is even possible, my best guess is that somehow that database tables got setup incorrectly. Since that field is not filled-in it is also difficult to tell what is old data vs new data. I find it most strange that it is in this state but seems to think it is storing results. I am hoping the script above will repair the tables since they should re-run the database setup scripts.

Thanks,

Andy

On June 19, 2018 at 2:43:52 PM, Kathy Benninger () wrote:

After upgrading the XSEDE perfSONAR at NICS (ps.nics.xsede.org) from CentOS6 to CentOS7-based Toolkit, the homepage stubbornly says "No data available in table". The upgrade was done as a net install last Thursday, 14-June.

I ran migrate-backup and migrate-restore with the --data option so it should have both old and new accumulated data to display.

It had the "401: Invalid token" errors in /var/log/perfsonar/pscheduler.log caused by a double set of three "<measurement_archive>" blocks in /etc/perfsonar/meshconfig-agent-tasks.conf so I fixed that. Sadly, graphing still didn't start working.

ps.nics gets its test specs from a meshconfig file and all looks as expected on the Test Configuration web page. pscheduler monitor --host shows lots of tests coming and going. I can manually initiate tests with pscheduler.

I don't see any obvious (at least to me) problem in /var/log/messages.

Tried:
    yum reinstall pscheduler-server
    pscheduler internal db-update

The following is systemctl and log output that might help the experts identify the problem/fix. Please let me know if other evidence would be helpful.

Thanks!
Kathy Benninger
Pittsburgh Supercomputing Center


[root@ps benninge]# systemctl status pscheduler-scheduler -l
â pscheduler-scheduler.service - pScheduler server - scheduler
   Loaded: loaded (/usr/lib/systemd/system/pscheduler-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-06-19 11:35:29 EDT; 2h 1min ago
 Main PID: 2312 (scheduler)
   CGroup: /system.slice/pscheduler-scheduler.service
           ââ2312 /usr/bin/python /usr/libexec/pscheduler/daemons/scheduler --daemon --pid-file /var/run/pscheduler-scheduler.pid --dsn @/etc/pscheduler/database/database-dsn

Jun 19 11:35:50 ps.nics.xsede.org scheduler[2312]: safe_run/scheduler ERROR    Program threw an exception after 0:00:20.392415
Jun 19 11:35:50 ps.nics.xsede.org scheduler[2312]: safe_run/scheduler ERROR    Exception: DatabaseError: server closed the connection unexpectedly
                                                           This probably means the server terminated abnormally
                                                           before or while processing the request.

                                                   Traceback (most recent call last):
                                                     File "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 72, in safe_run
                                                       function()
                                                     File "/usr/libexec/pscheduler/daemons/scheduler", line 809, in <lambda>
                                                       pscheduler.safe_run(lambda: main_program())
                                                     File "/usr/libexec/pscheduler/daemons/scheduler", line 764, in main_program
                                                       cursor.execute(query, args)
                                                   DatabaseError: server closed the connection unexpectedly
                                                           This probably means the server terminated abnormally
                                                           before or while processing the request.
Jun 19 11:35:50 ps.nics.xsede.org scheduler[2312]: safe_run/scheduler ERROR    Waiting 0.25 seconds before restarting
Jun 19 11:35:50 ps.nics.xsede.org scheduler[2312]: safe_run/scheduler ERROR    Restarting: ['/usr/libexec/pscheduler/daemons/scheduler', '--daemon', '--pid-file', '/var/run/pscheduler-scheduler.pid', '--dsn', '@/etc/pscheduler/database/database-dsn']
Jun 19 11:35:50 ps.nics.xsede.org scheduler[2312]: scheduler INFO     Started
Jun 19 13:22:56 ps.nics.xsede.org scheduler[2312]: safe_run/scheduler ERROR    Program threw an exception after 1:47:05.897896
Jun 19 13:22:56 ps.nics.xsede.org scheduler[2312]: safe_run/scheduler ERROR    Exception: DatabaseError: server closed the connection unexpectedly
                                                           This probably means the server terminated abnormally
                                                           before or while processing the request.

                                                   Traceback (most recent call last):
                                                     File "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 72, in safe_run
                                                       function()
                                                     File "/usr/libexec/pscheduler/daemons/scheduler", line 809, in <lambda>
                                                       pscheduler.safe_run(lambda: main_program())
                                                     File "/usr/libexec/pscheduler/daemons/scheduler", line 764, in main_program
                                                       cursor.execute(query, args)
                                                   DatabaseError: server closed the connection unexpectedly
                                                           This probably means the server terminated abnormally
                                                           before or while processing the request.
Jun 19 13:22:56 ps.nics.xsede.org scheduler[2312]: safe_run/scheduler ERROR    Waiting 0.5 seconds before restarting
Jun 19 13:22:57 ps.nics.xsede.org scheduler[2312]: safe_run/scheduler ERROR    Restarting: ['/usr/libexec/pscheduler/daemons/scheduler', '--daemon', '--pid-file', '/var/run/pscheduler-scheduler.pid', '--dsn', '@/etc/pscheduler/database/database-dsn']
Jun 19 13:22:57 ps.nics.xsede.org scheduler[2312]: scheduler INFO     Started

+++++++++++++++++++++++++++++++++++++++++++++

[root@ps benninge]# systemctl status pscheduler-ticker -l
â pscheduler-ticker.service - pScheduler server - ticker
   Loaded: loaded (/usr/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-06-19 11:35:29 EDT; 2h 3min ago
 Main PID: 2317 (ticker)
   CGroup: /system.slice/pscheduler-ticker.service
           ââ2317 /usr/bin/python /usr/libexec/pscheduler/daemons/ticker --daemon --pid-file /var/run/pscheduler-ticker.pid --dsn @/etc/pscheduler/database/database-dsn

Jun 19 11:35:29 ps.nics.xsede.org systemd[1]: Starting pScheduler server - ticker...
Jun 19 11:35:29 ps.nics.xsede.org systemd[1]: Started pScheduler server - ticker.
Jun 19 11:35:50 ps.nics.xsede.org ticker[2317]: ticker WARNING  Queue maintainer got exception server closed the connection unexpectedly
                                                        This probably means the server terminated abnormally
                                                        before or while processing the request....
Jun 19 11:36:01 ps.nics.xsede.org ticker[2317]: safe_run/ticker ERROR    Program threw an exception after 0:00:31.352763
Jun 19 11:36:01 ps.nics.xsede.org ticker[2317]: safe_run/ticker ERROR    Exception: OperationalError: terminating connection due to administrator command
                                                server closed the connection unexpectedly
                                                        This probably means the server terminated abnormally...
Jun 19 11:36:01 ps.nics.xsede.org ticker[2317]: safe_run/ticker ERROR    Waiting 0.25 seconds before restarting
Jun 19 11:36:01 ps.nics.xsede.org ticker[2317]: safe_run/ticker ERROR    Restarting: ['/usr/libexec/pscheduler/daemons/ticker', '--daemon', '--pid-file', '/var/run/pscheduler-t...tabase-dsn']
Hint: Some lines were ellipsized, use -l to show in full.
[root@ps benninge]# systemctl status pscheduler-ticker -l
â pscheduler-ticker.service - pScheduler server - ticker
   Loaded: loaded (/usr/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-06-19 11:35:29 EDT; 2h 3min ago
 Main PID: 2317 (ticker)
   CGroup: /system.slice/pscheduler-ticker.service
           ââ2317 /usr/bin/python /usr/libexec/pscheduler/daemons/ticker --daemon --pid-file /var/run/pscheduler-ticker.pid --dsn @/etc/pscheduler/database/database-dsn

Jun 19 11:35:29 ps.nics.xsede.org systemd[1]: Starting pScheduler server - ticker...
Jun 19 11:35:29 ps.nics.xsede.org systemd[1]: Started pScheduler server - ticker.
Jun 19 11:35:50 ps.nics.xsede.org ticker[2317]: ticker WARNING  Queue maintainer got exception server closed the connection unexpectedly
                                                        This probably means the server terminated abnormally
                                                        before or while processing the request.
Jun 19 11:36:01 ps.nics.xsede.org ticker[2317]: safe_run/ticker ERROR    Program threw an exception after 0:00:31.352763
Jun 19 11:36:01 ps.nics.xsede.org ticker[2317]: safe_run/ticker ERROR    Exception: OperationalError: terminating connection due to administrator command
                                                server closed the connection unexpectedly
                                                        This probably means the server terminated abnormally
                                                        before or while processing the request.

                                                Traceback (most recent call last):
                                                  File "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 72, in safe_run
                                                    function()
                                                  File "/usr/libexec/pscheduler/daemons/ticker", line 157, in <lambda>
                                                    pscheduler.safe_run(lambda: main_program())
                                                  File "/usr/libexec/pscheduler/daemons/ticker", line 137, in main_program
                                                    cursor.execute("SELECT ticker()")
                                                OperationalError: terminating connection due to administrator command
                                                server closed the connection unexpectedly
                                                        This probably means the server terminated abnormally
                                                        before or while processing the request.
Jun 19 11:36:01 ps.nics.xsede.org ticker[2317]: safe_run/ticker ERROR    Waiting 0.25 seconds before restarting
Jun 19 11:36:01 ps.nics.xsede.org ticker[2317]: safe_run/ticker ERROR    Restarting: ['/usr/libexec/pscheduler/daemons/ticker', '--daemon', '--pid-file', '/var/run/pscheduler-ticker.pid', '--dsn', '@/etc/pscheduler/database/database-dsn']


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Snippet from /var/log/esmond/django.log :

AllServersUnavailable: An attempt was made to connect to each of the servers twice, but none of the attempts succeeded. The last failure was TTransportException: Could not connect to localhost:9160
2018-06-18 15:05:53,057 [ERROR] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py: Internal Server Error: /esmond/perfsonar/archive/f47607965bb84e52a0fc715ab1ecd996/
Traceback (most recent call last):
  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = get_response(request)
  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/viewsets.py", line 90, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 932, in update
    obj.save()
  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 407, in save
    existing = PSTimeSeriesObject.query_database(self.metadata_key, self.event_type, 'base', None, int(self.time), int(self.time), 1)
  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 473, in query_database
    cf='average', ts_min=begin_millis, ts_max=end_millis, column_count=max_results)
  File "/usr/lib/esmond/esmond/cassandra.py", line 650, in query_aggregation_timerange
    column_count=cols)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/columnfamily.py", line 772, in multiget
    packed_keys[offset:offset + buffer_size], cp, sp, consistency)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 577, in execute
    return getattr(conn, f)(*args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 125, in new_f
    self._pool._replace_wrapper() # puts a new wrapper in the queue
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 458, in _replace_wrapper
    conn = self._create_connection()
  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 431, in _create_connection
    (exc.__class__.__name__, exc))
AllServersUnavailable: An attempt was made to connect to each of the servers twice, but none of the attempts succeeded. The last failure was TTransportException: Could not connect to localhost:9160
2018-06-19 06:26:09,824 [WARNING] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py: Not Found: /esmond/perfsonar/archive/?event-type=histogram-owdelay&limit=1/
2018-06-19 06:29:37,671 [WARNING] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py: Not Found: /esmond/perfsonar/archive/?event-type=throughput&limit=1/
2018-06-19 09:59:12,691 [WARNING] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py: Not Found: /esmond/perfsonar/archive/?event-type=histogram-owdelay&limit=1/
2018-06-19 10:01:52,122 [WARNING] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py: Not Found: /esmond/perfsonar/archive/?event-type=throughput&limit=1/
2018-06-19 10:45:02,285 [WARNING] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py: Not Found: /esmond/perfsonar/archive/?event-type=histogram-owdelay&limit=1/
2018-06-19 10:48:02,119 [WARNING] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py: Not Found: /esmond/perfsonar/archive/?event-type=throughput&limit=1/
(END)

++++++++++++++++++++++++++++++++++++++++++++++++

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user