perfSONAR User Q&A and Other Discussion

[Thomas Angle <>]

ssl_error_log show this message repeating over and over

[Mon Mar 08 15:24:07.362994 2021] [wsgi:error] [pid 2130] [remote 192.33.116.75:39346] esmond [INFO] Schema check done
[Mon Mar 08 15:24:07.363092 2021] [wsgi:error] [pid 2130] [remote 192.33.116.75:39346] esmond [DEBUG] Opening ConnectionPool
[Mon Mar 08 15:24:07.378523 2021] [wsgi:error] [pid 2130] [remote 192.33.116.75:39346] esmond [INFO] Connected to ['localhost:9160']

The esmond.log show this repeating

2021-03-08 15:10:13,409 [INFO] /usr/lib/esmond/esmond/cassandra.py: Schema check done
2021-03-08 15:10:13,424 [INFO] /usr/lib/esmond/esmond/cassandra.py: Connected to ['localhost:9160']
2021-03-08 15:24:07,358 [INFO] /usr/lib/esmond/esmond/cassandra.py: Checking/creating column families

The django.log show the below. The error states "psycopg2.OperationalError: FATAL:  role "esmond" does not exist"

2021-03-08 15:28:47,969 [ERROR] /usr/lib/esmond/lib/python3.6/site-packages/django/core/handlers/exception.py: Internal Server Error: /esmond/perfsonar/archive/
Traceback (most recent call last):
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/backends/base/base.py", line 189, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/lib64/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  role "esmond" does not exist


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/viewsets.py", line 116, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 892, in list
    return super(ArchiveViewset, self).list(request)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/mixins.py", line 42, in list
    page = self.paginate_queryset(queryset)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/generics.py", line 173, in paginate_queryset
    return self.paginator.paginate_queryset(queryset, self.request, view=self)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/pagination.py", line 325, in paginate_queryset
    self.count = self.get_count(queryset)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/pagination.py", line 466, in get_count
    return queryset.count()
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/models/query.py", line 364, in count
    return self.query.get_count(using=self.db)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/models/sql/query.py", line 499, in get_count
    number = obj.get_aggregation(using, ['__count'])['__count']
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/models/sql/query.py", line 480, in get_aggregation
    result = compiler.execute_sql(SINGLE)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 887, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/backends/base/base.py", line 254, in cursor
    return self._cursor()
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/backends/base/base.py", line 229, in _cursor
    self.ensure_connection()
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/backends/base/base.py", line 189, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/lib64/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL:  role "esmond" does not exist

On 3/8/21 10:09 AM, Garnizov, Ivan wrote:

Hello Thomas,

 

There must be a reason for the 500 Error on attempts to connect to your Esmond archiver. Please try to see what stands behind

 It. Also please note that the Esmond communication is SSL protected, meaning you should also check the Apache ssl_error_log

 

The JMX warnings are irrelevant. The 7199 port / connection is not relevant for pS and is therefore not exposed.

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

 

 

From: Thomas Angle []
Sent: Monday, March 8, 2021 2:48 PM
To: Garnizov, Ivan (RRZE) ; Edward Colone
Cc:
Subject: Re: [perfsonar-user] Error - Couldn't save configuration file

 

Thanks for giving me a place too look.

It is running.

systemctl status cassandra
● cassandra.service - SYSV: Starts and stops Cassandra
   Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
   Active: active (exited) since Fri 2021-03-05 09:47:48 EST; 2 days ago
     Docs: man:systemd-sysv-generator(8)

Mar 05 09:47:43 perfsonar.gb.nrao.edu systemd[1]: Starting SYSV: Starts and stops Cassandra...
Mar 05 09:47:43 perfsonar.gb.nrao.edu su[1461]: (to cassandra) root on none
Mar 05 09:47:48 perfsonar.gb.nrao.edu cassandra[1418]: Starting Cassandra: OK
Mar 05 09:47:48 perfsonar.gb.nrao.edu systemd[1]: Started SYSV: Starts and stops Cassandra.

I found this in /var/log/cassandra/system.log.

 WARN [main] 2021-03-05 09:47:52,661 CassandraDaemon.java (line 153) JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.

If is enabled in /etc/cassandra/default.conf/cassandra-env.sh

JMX_PORT="7199"
LOCAL_JMX=yes

The port is open

tcp        0      0 127.0.0.1:7199          0.0.0.0:*               LISTEN      2666/java          

I did notice a port not open in firewalld

firewall-cmd --list-ports

firewall-cmd --list-services
bwctl dhcpv6-client http https ntp owamp-control ssh traceroute twamp-contr

8760-9960/udp 8760-9960/tcp 18760-19960/udp 18760-19960/tcp 5201/tcp 5201/udp 5001/tcp 5001/udp 5000/tcp 5101/tcp 5000/udp 5101/udp 5890-5900/tcp

This is a fresh install with some tests added. Did I miss something when installing? Thanks

Tom

On 3/8/21 8:09 AM, Garnizov, Ivan wrote:

Hi Thomas,

 

The link you provided shows a failure for the Esmond archiver.

This apparently leads to the issues you report with the pS GUI.

 

So what do you have in the Cassandra logs?

/var/log/cassandra/system.log

 

Are you sure Cassandra service is up and running?

 

Could that be a resource exhaustion problem?

From your messages we can only guess, but do you run the pS Toolkit? How did you install it / what procedure did you follow?

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

From: [mailto:] On Behalf Of Thomas Angle
Sent: Friday, March 5, 2021 4:39 PM
To: Edward Colone <>
Cc:
Subject: Re: [perfsonar-user] Error - Couldn't save configuration file

 

No, I think it was the migration script (from CentOS . I did reinstall and could manually add the tests and it saved fine.

I have an error in the Test Results  page.

Error loading test listing; measurement archive unreachable:

https://perfsonar.gb.nrao.edu/esmond/perfsonar/archive/

I did find this in the /var/log/httpd/error_log

[Fri Mar 05 09:58:06.750040 2021] [wsgi:error] [pid 2218] path= ['/usr/lib/esmond', '/usr/lib/esmond/esmond', '/usr/lib/esmond/lib/python3.6', '/usr/lib/esmond/lib/python3.6/site-packages', '/usr/lib/esmond', '/usr/lib/esmond/esmond_client', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/usr/lib64/python3.6/site-packages', '/usr/lib/python3.6/site-packages']

[Fri Mar 05 09:40:57.607594 2021] [wsgi:error] [pid 2116] [remote 10.16.96.111:56660] esmond [INFO] Schema check done
[Fri Mar 05 09:40:57.607682 2021] [wsgi:error] [pid 2116] [remote 10.16.96.111:56660] esmond [DEBUG] Opening ConnectionPool
[Fri Mar 05 09:40:57.623181 2021] [wsgi:error] [pid 2116] [remote 10.16.96.111:56660] esmond [INFO] Connected to ['localhost:9160']

On 3/5/21 10:12 AM, Edward Colone wrote:

Was this a result of a pSconfig command, or some other operation?

 

Thanks,

 

-Ed

 

 

On Fri, Mar 5, 2021 at 10:10 AM Thomas Angle <> wrote:

After a clean install to CentOS 7 and a migration of the configuration,
I get the following error when trying to save a config change.

Error - Couldn't save configuration file

Any help would be appreciated. Thanks

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user