perfSONAR User Q&A and Other Discussion

[Daniel Manzo <>]

Here’s the last day’s entries in /var/log/httpd/error_log (I modified the IPs to ‘xx’):

 

[Sun Oct 16 01:00:41 2016] [error] ImportError: No module named os

[Mon Oct 17 09:27:18 2016] [warn] [client 10.83.xx.xx] incomplete redirection target of '/toolkit/' for URI '/' modified to 'http://xx.xx.xx.xx/toolkit/'

[Mon Oct 17 09:27:19 2016] [error] [client 127.0.0.1] mod_wsgi (pid=24588): Target WSGI script '/usr/lib/esmond/esmond/wsgi.py' cannot be loaded as Python module.

[Mon Oct 17 09:27:19 2016] [error] [client 127.0.0.1] mod_wsgi (pid=24587): Target WSGI script '/usr/lib/esmond/esmond/wsgi.py' cannot be loaded as Python module.

[Mon Oct 17 09:27:19 2016] [error] [client 127.0.0.1] mod_wsgi (pid=24588): Exception occurred processing WSGI script '/usr/lib/esmond/esmond/wsgi.py'.

[Mon Oct 17 09:27:19 2016] [error] [client 127.0.0.1] mod_wsgi (pid=24587): Exception occurred processing WSGI script '/usr/lib/esmond/esmond/wsgi.py'.

[Mon Oct 17 09:27:19 2016] [error] Traceback (most recent call last):

[Mon Oct 17 09:27:19 2016] [error]   File "/usr/lib/esmond/esmond/wsgi.py", line 4, in <module>

[Mon Oct 17 09:27:19 2016] [error]     import os

[Mon Oct 17 09:27:19 2016] [error] Traceback (most recent call last):

[Mon Oct 17 09:27:19 2016] [error]   File "/usr/lib/esmond/esmond/wsgi.py", line 4, in <module>

[Mon Oct 17 09:27:19 2016] [error] ImportError: No module named os

[Mon Oct 17 09:27:19 2016] [error]     import os

[Mon Oct 17 09:27:19 2016] [error] ImportError: No module named os

 

And here’s the last error from /var/log/esmond/django.log:

 

2016-10-13 11:09:06,686 [ERROR] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py: Internal Server Error: /esmond/perfsonar/archive/

Traceback (most recent call last):

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response

    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view

    return view_func(*args, **kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/viewsets.py", line 85, in view

return self.dispatch(request, *args, **kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 456, in dispatch

    response = self.handle_exception(exc)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 444, in dispatch

    self.initial(request, *args, **kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 358, in initial

    self.perform_authentication(request)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 296, in perform_authentication

    request.user

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/request.py", line 489, in __getattribute__

    return super(Request, self).__getattribute__(attr)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/request.py", line 264, in user

    self._authenticate()

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/request.py", line 452, in _authenticate

    user_auth_tuple = authenticator.authenticate(self)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/authentication.py", line 177, in authenticate_credentials

    token = self.model.objects.select_related('user').get(key=key)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get

    num = len(clone)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__

    self._fetch_all()

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all

    self._result_cache = list(self.iterator())

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator

    results = compiler.execute_sql()

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 838, in execute_sql

    cursor = self.connection.cursor()

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/base/base.py", line 164, in cursor

    cursor = self.make_cursor(self._cursor())

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/base/base.py", line 135, in _cursor

    self.ensure_connection()

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/base/base.py", line 130, in ensure_connection

    self.connect()

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__

    six.reraise(dj_exc_type, dj_exc_value, traceback)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/base/base.py", line 130, in ensure_connection

    self.connect()

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/base/base.py", line 119, in connect

    self.connection = self.get_new_connection(conn_params)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection

    connection = Database.connect(**conn_params)

  File "/usr/lib/esmond/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect

    conn = _connect(dsn, connection_factory=connection_factory, async=async)

OperationalError: could not connect to server: No such file or directory

        Is the server running locally and accepting

        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

 

From: Andrew Lake [mailto:]
Sent: Monday, October 17, 2016 9:49 AM
To: perfsonar-user; Daniel Manzo
Subject: Re: [perfsonar-user] Error writing metadata (500) Internal server error

 

Hi,

 

Those errors look completely independent of MaDDash. It looks like an issue on your measurement archive. What is in /var/log/httpd/error_log and /var/log/esmond/django.log on the host with the problematic archive?

 

Thanks,

Andy

 

 

On October 17, 2016 at 9:40:39 AM, Daniel Manzo () wrote:

Hi all,

 

My current perfSONAR setup is an HP server with 8 GB of RAM and an OS of RHEL 6.8. I installed using the CentOS bundle method with perfsonar-toolkit. After installing MaDDash 2.0 from the staging repo, I realized that perfsonar-graphs updated to 4.0, which is not compatible with the version of perfsonar-toolkit I have (3.5.1). I used yum to remove MaDDash and the staging repo, but I have not been able to see the graphs on the web interface. It just shows a red bar with “Error loading test listing: Internal Server Error”. I have been getting the same three errors in regulartesting.log for a while and have not been able to fix them:

 

2016/10/16 01:00:41 (8232) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (500) 500 Internal Server Error

2016/10/16 01:00:41 (8232) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 500 Internal Server Error

2016/10/16 01:00:41 (8232) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 500 Internal Server Error at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122.

2016/10/16 01:00:41 (8233) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (500) 500 Internal Server Error

2016/10/16 01:00:41 (8233) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 500 Internal Server Error

2016/10/16 01:00:41 (8233) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 500 Internal Server Error at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122.

 

I have tried the following steps to fix the problem:

rm –rf /var/lib/pgsql/data/*

/usr/lib/perfsonar/scripts/system_environment/configure_esmond

/usr/lib/perfsonar/scripts/system_environment/configure_regulartesting

/etc/init.d/httpd restart

/etc/init.d/perfsonar-regulartesting restart

cd /usr/lib/esmond

source /opt/rh/python27/enable

/opt/rh/python27/root/usr/bin/virtualenv --prompt="(esmond)" .

. bin/activate

python esmond/manage.py migrate

 

Maybe there is something else that I need to remove that was leftover from MaDDash 2.0 that is affecting the database/test results? Any advice or tips would be very helpful.

 

Thanks,

 

Dan Manzo

Scientific Computing Competence Ctr

_______________________

 

Bayer: Science For A Better Life

 

Bayer Business and Technology Services LLC

BBTS-IAO-COMM TECH NAIT College Program

Building B16

Pittsburgh, PA 15205-9741, USA

Tel:        412-777-2171

Mobile:   412-525-8332

E-mail:   

Web:      http://www.bayer.com