perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi,

Ok, I’m actually able to recreate these exact exceptions if my host boots with only an IPv6 address. Even after I assign it an IPv4 address things don’t work until I restart some of the daemons. Do your hosts have just an IPv6 address? I see the hostname Shkelzen shared is dual-stack so in theory it should have both and be fine, but maybe there is some race condition with when addresses get assigned vs when the daemons come up? Try the following if you host has an IPv4 address as well, since the commands below seemed to fix things for me after assigning an IPv4 address:

systemctl restart cassandra

systemctl restart httpd

systemctl restart pscheduler-scheduler

systemctl restart pscheduler-archiver

systemctl restart pscheduler-ticker

systemctl restart pscheduler-runner

If you want a quick command to add something to your archive run:

pscheduler task --archive @/usr/share/pscheduler/psc-archiver-esmond.json idle —duration PT5S

curl -k https://127.0.0.1/esmond/perfsonar/archive/

The second command should return a non-empty list if things are back to working.

If you host is indeed IPv6 only then the above is likely not enough. Cassandra not working properly on a host with IPv6 only by default is a known issue and the fix for that is documented in an FAQ: http://docs.perfsonar.net/FAQ.html#q-how-can-i-get-cassandra-to-run-on-a-host-that-only-has-an-ipv6-address. Unfortunately that one is mostly out of our control so best we can do is provide the FAQ with the fix. The Django error is new, so we can dig closer at that. 

Thanks,

Andy

On December 12, 2017 at 3:36:50 AM, Shkelzen Rugovac () wrote:

Hi all,

Indeed, after applying "yum reinstall esmond" the "internal Server Error" disappeared but not the error in the django.log. 

Not knowing what possible service to restart, I just rebooted the whole server and now django logs the following error:

2017-12-12 09:16:43,508 [ERROR] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py: Internal Server Error: /esmond/perfsonar/archive

Traceback (most recent call last):

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py", line 42, in inner

    response = get_response(request)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 244, in _legacy_get_response

    response = middleware_method(request)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/middleware/common.py", line 62, in process_request

    if self.should_redirect_with_slash(request):

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/middleware/common.py", line 80, in should_redirect_with_slash

    not is_valid_path(request.path_info, urlconf) and

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/urls/base.py", line 157, in is_valid_path

    resolve(path, urlconf)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/urls/base.py", line 27, in resolve

    return get_resolver(urlconf).resolve(path)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/urls/resolvers.py", line 270, in resolve

    for pattern in self.url_patterns:

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__

    res = instance.__dict__[self.name] = self.func(instance)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/urls/resolvers.py", line 313, in url_patterns

    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__

    res = instance.__dict__[self.name] = self.func(instance)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/urls/resolvers.py", line 306, in urlconf_module

    return import_module(self.urlconf_name)

  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module

    __import__(name)

  File "/usr/lib/esmond/esmond/urls.py", line 10, in <module>

    from esmond.api.api_v2 import (

  File "/usr/lib/esmond/esmond/api/api_v2.py", line 52, in <module>

    raise ConnectionException(str(e))

ConnectionException: '"System Manager can\'t connect to Cassandra at localhost:9160 - Could not connect to localhost:9160"'

And http://ps02.iihe.ac.be/esmond/perfsonar/archive/ shows an empty list.

Any hint?

Cheers,

Shkelzen

2017-12-11 17:45 GMT+01:00 Doug Wussler <>:

Andrew –

 

That was me.  The “yum reinstall esmond” cleared up the “Internal Server Error” I was getting in the “Test Results” section when I visited the toolkit page.  But it did not clear up the issue Shkelzen is reporting, I have the same error even after reinstalling Esmond and rebooting.  We both did a full install of the 4.02 ISO and our /var/log/esmond/django.log is showing the same failure.  The reinstall has not fixed this problem.

 

Doug Wussler

Florida State University 

 

From: <> on behalf of Andrew Lake <>
Date: Monday, December 11, 2017 at 9:07 AM
To: "" <>, Shkelzen Rugovac <>
Subject: Re: [perfsonar-user] Django error after a fresh install

 

Hi,

 

Someone else reported a similar issue last week and they were able to correct it by doing a “yum reinstall esmond”. The dev team is investigating closer as it doesn't appear to happen in our test builds, so must be some condition or inconsistent package ordering that is triggering it.

 

Thanks,

Andy

 

 

 

 

 

On December 11, 2017 at 3:27:21 AM, Shkelzen Rugovac () wrote:

Hi,

 

After a fresh installation via the FULL 4.0.2 ISO + yum update [1],[2] and [3], I have the following error in  /var/log/esmond/django.log :

 

2017-12-11 09:18:31,761 [ERROR] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py: Internal Server Error: /esmond/perfsonar/archive/

Traceback (most recent call last):

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py", line 42, in inner

    response = get_response(request)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response

    response = self._get_response(request)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response

    response = self.process_exception_by_middleware(e, request)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response

    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view

    return view_func(*args, **kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/viewsets.py", line 90, in view

    return self.dispatch(request, *args, **kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 489, in dispatch

    response = self.handle_exception(exc)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 449, in handle_exception

    self.raise_uncaught_exception(exc)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 486, in dispatch

    response = handler(request, *args, **kwargs)

  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 838, in list

    return super(ArchiveViewset, self).list(request)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/mixins.py", line 42, in list

    page = self.paginate_queryset(queryset)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/generics.py", line 173, in paginate_queryset

    return self.paginator.paginate_queryset(queryset, self.request, view=self)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/pagination.py", line 335, in paginate_queryset

    self.count = _get_count(queryset)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/pagination.py", line 53, in _get_count

    return queryset.count()

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/query.py", line 369, in count

    return self.query.get_count(using=self.db)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/sql/query.py", line 476, in get_count

    number = obj.get_aggregation(using, ['__count'])['__count']

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/sql/query.py", line 457, in get_aggregation

    result = compiler.execute_sql(SINGLE)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql

    cursor.execute(sql, params)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute

    return self.cursor.execute(sql, params)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__

    six.reraise(dj_exc_type, dj_exc_value, traceback)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute

    return self.cursor.execute(sql, params)

ProgrammingError: relation "ps_metadata" does not exist

LINE 1: ...e" AS Col3, "ps_metadata"."checksum" AS Col4 FROM "ps_metada...

 

 

Why ps_metadata not (yet) present?

 

[1] http://docs.perfsonar.net/install_centos_fullinstall.html

[2] http://docs.perfsonar.net/install_config_first_time.html

[3] http://docs.perfsonar.net/manage_regular_tests.html

 

Cheers,

Shkelzen

 

 

--

Shkelzen RUGOVAC       

Université Libre de Bruxelles
Bd du Triomphe, CP230
1050 Bruxelles - Belgium

Office at VUB:
Building G - Level 0 - Room 147

Tel: (+32) 2 629 33 26

--

Shkelzen RUGOVAC       

Université Libre de Bruxelles
Bd du Triomphe, CP230
1050 Bruxelles - Belgium

Office at VUB:
Building G - Level 0 - Room 147

Tel: (+32) 2 629 33 26