perfSONAR User Q&A and Other Discussion

[Pedro Henrique Diniz Da Silva <>]

Andy,

Actions are described bellow.

Regards,

Pedro Diniz

---

From: Andrew Lake []
Sent: Wednesday, October 25, 2017 11:39 AM
To: Marian Babik; Pedro Henrique Diniz Da Silva
Cc: ;
Subject: RE: [perfsonar-user] PerfSONAR Toolkit Problem

Hi Pedro,

I am guessing there is something wrong with your postgresql instance. Both esmond and pscheduler use postgresql and pscheduler is also throwing an error: https://152.84.102.250/pscheduler/tasks.

You can try a couple things:

1. Try restarting postgresql-9.5: /etc/init.d/postgresql-9.5 restart

        Nothing different. https://152.84.102.250/pscheduler/tasks

2. Look in /var/log/esmond/django.log for more detailed errors. 

        What should I look for? I could find 2 types of errors that can be seem below. It is attached.

2017-04-09 01:05:03,618 [ERROR] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py: Internal Server Error: /esmond/perfsonar/archive/1132c4034e624f98b5141f6203c80d00/histogram-owdelay/statistics/0

Traceback (most recent call last):

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response

    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view

    return view_func(*args, **kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/viewsets.py", line 85, in view

    return self.dispatch(request, *args, **kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 456, in dispatch

    response = self.handle_exception(exc)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 453, in dispatch

    response = handler(request, *args, **kwargs)

  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 1079, in retrieve

    results = PSTimeSeriesObject.query_database(metadata_key, event_type, summary_type, freq, begin_time, end_time, max_results)

  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 455, in query_database

    ts_min=begin_millis, ts_max=end_millis, column_count=max_results)

  File "/usr/lib/esmond/esmond/cassandra.py", line 721, in query_raw_data

    column_count=cols)

  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/columnfamily.py", line 772, in multiget

    packed_keys[offset:offset + buffer_size], cp, sp, consistency)

  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 577, in execute

    return getattr(conn, f)(*args, **kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 127, in new_f

    result = f(self, *args, **kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/cassandra/Cassandra.py", line 669, in multiget_slice

    return self.recv_multiget_slice()

  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/cassandra/Cassandra.py", line 690, in recv_multiget_slice

    result.read(self._iprot)

  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/cassandra/Cassandra.py", line 3816, in read

    fastbinary.decode_binary(self, iprot.trans, (self.__class__, self.thrift_spec))

OverflowError: list size out of the sanity limit (10000 items max)

2017-04-09 01:05:33,025 [ERROR] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py: Internal Server Error: /esmond/perfsonar/archive/734680650ca54a5e931b823a40ff488a/

Traceback (most recent call last):

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response

    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view

    return view_func(*args, **kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/viewsets.py", line 85, in view

    return self.dispatch(request, *args, **kwargs)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 456, in dispatch

    response = self.handle_exception(exc)

  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 453, in dispatch

    response = handler(request, *args, **kwargs)

  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 905, in update

    db.flush()

  File "/usr/lib/esmond/esmond/cassandra.py", line 267, in flush

    self.rates.send()

  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/batch.py", line 146, in send

    allow_retries=self.allow_retries)

  File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", line 148, in new_f

    (self._retry_count, exc.__class__.__name__, exc))

MaximumRetryException: Retried 1 times. Last failure was timeout: timed out

3. Make sure the file /var/lib/pgsql/9.5/data/pg_hba.conf is non-empty. If it is empty let me know and I can help you fix (this is a bug we’ve ran into before)

        It is non-empty (file attached). It seems it has default configs. Is it all right?

Thanks,

Andy

On October 25, 2017 at 8:28:37 AM, Pedro Henrique Diniz Da Silva () wrote:

Marian,

I did what you suggested (rebooting the node and restarting httppd servers), but same result as attached (Connection refused: proxy: HTTP: attempt to connect to [::1]:11413).

Other suggestions?

Regards,
Pedro Diniz
________________________________________
From: Marian Babik
Sent: Wednesday, October 25, 2017 6:55 AM
To: Pedro Henrique Diniz Da Silva
Cc: ;
Subject: Re: [perfsonar-user] PerfSONAR Toolkit Problem

Hi Pedro,
I suggest to reboot the node to start with, assuming this is SL6/centos6 installation and since there are some proxy errors I suggest to restart both httpd servers as well:
service httpd restart
service httpd24-httpd restart

Once done, please re-check the httpd error log (both /var/log/httpd/error_log and /var/log/httpd24/error_log) if there are still errors afterwards. Let me know if that helped.

Thanks,
Marian


> On Oct 24, 2017, at 11:56 PM, Pedro Henrique Diniz Da Silva <> wrote:
>
> Attaching...
> From: [] on behalf of Pedro Henrique Diniz Da Silva []
> Sent: Tuesday, October 24, 2017 7:51 PM
> To:
> Cc:
> Subject: [perfsonar-user] PerfSONAR Toolkit Problem
>
> Good Afternoon,
>
> My name is Pedro Diniz, and I am a Network Engineer at the Brazilian Center for Research in Physics.
>
> I have updated our PS yesterday to 4.0.1 and we are ruunning into an issue. We cannot seem to view test results. There is a red box below the “Test Results” Section on the dashboard that reads. “Error loading test listing: Internal Server Error.”
>
> PS is located at: https://152.84.102.250/toolkit/
>
> There are a bunch of errors in a bunch of places so I'm unable to find where the problems are.
>
> For example, regulartesting.log:
>
> Error writing metadata: 500 Can't connect to localhost:80 (connect: Conexão recusada) at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122.
> 2017/10/23 15:05:06 (3441) WARN> EsmondBase.pm:207 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::add_data - Error posting data to MA: {"detail":"Time series value already exists with event type path-mtu at time 1508775201"}
> 2017/10/23 16:18:44 (6762) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (500) 500 Internal Server Error
> 2017/10/23 16:18:44 (6762) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 500 Internal Server Error
> 2017/10/23 16:18:44 (6762) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 500 Internal Server Error at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122.
> 2017/10/23 16:19:28 (7345) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (500) 500 Internal Server Error
> 2017/10/23 16:19:28 (7345) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 500 Internal Server Error
> 2017/10/23 16:19:28 (7345) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 500 Internal Server Error at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122.
>
>
> I attached multiple logs that I think that can help. They are the content of /var/log/esmond/, the full file /var/log/httpd/error_log and the result of "yum info python27-mod_wsgi".
>
> In http://152.84.102.250/esmond/perfsonar/archive we get "Service Temporarily Unavailable" error (print screen attached)
>
> If you can point me in the correct direction, that would be greatly appreciated. Thank you very much.
>
> Best Regards,
> Pedro Diniz
>
>
>
>
> <screen_esmond_perfsonar_archive.jpg><httpd_error_log><regulartesting.log><esmond_django.log><esmond_esmond.log><esmond_install.log>

Attachment: pg_hba.conf
Description: pg_hba.conf

Attachment: django.log
Description: django.log