perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi,

A couple things:

- You don't actually need perfsonar-psb_to_esmond since that script only exists to upgrade hosts from 3.5. Since you’re a new install, even if it worked it wouldn't be doing anything anyways, you might want to just try disabling it. 

- You might want to checkout https://hub.docker.com/r/perfsonar/testpoint/ if you haven't already. We don’t fully support it yet, but that’s a version of the perfsonar-testpoint bundle put together as a docker image by the perfSONAR dev team. The testpoint has all the tools needed to run regular tests (pscheduler, mesh-config, iperf3, owamp, etc) but does not include esmond, the toolkit gui, or any of the system environment stuff that tunes the kernel, sets the firewall, etc. You may be able to just use it depending on what you are trying to do. It will allow you to run measurements and push the results to an esmond instance running somewhere else. If it’s not what you need, it might be useful to at least look at the dockerfile. 

- If you are trying to do a full-blown toolkit, its possible you’ll run into other issues since it touches a lot of pieces of the system that may not make sense in a container. It’s why we started with the tool and testpoint bundles in dockerhub. Esmond you should be able to get running in a container if you really need it, though the trick will be getting postgresql setup correctly. The testpoint bundle already does some of that with pscheduler, so may be things you can draw-on from there. It’s on the list of things for us to tackle in the near future, but we haven’t quite gotten to it yet, so sorry I don't have more guidance. If you do go down this road though, we’d certainly be interested to hear what you find. 

Hope that helps,

Andy

On August 16, 2017 at 11:15:51 PM, John Tai (johntai) () wrote:

Hi PerfSonar Experts!

 

I’m trying to build a docker container based on centos7 and perfsonar-toolkit and have a perfsonar-psb_to_esmond.service start failure related to the /etc/init.d/perfsonar-psb_to_esmond service startup script call to

MA_KEY=`python esmond/manage.py add_api_key_user perfsonar | grep "Key:" | cut -f2 -d " "`

 

Which fails with:

perfsonar-psb_to_esmond: django.db.utils.OperationalError: FATAL:  password authentication failed for user "esmond"

 

Detail messages below. Also included is the Dockerfile being used.

 

“docker run -it -p7443:443 -v /var/run  --privileged=true test/perfsonar”  is used to run the container.

 

 

We have a KVM equivalent based on the provided pS-Toolkit-4.0 CentOS7-FullInstall-x86_64 iso.

 

I’m stumped on what is missing in the docker version which is causing the authentication failure.

I did find a previously created docker perfsonar but it’s a bit out-of-date and it has a different startup failure as well (https://github.com/MigielDV/perfsonar)

 

Any suggestions are appreciated!

 

// Error listed in “messages”

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: System check identified some issues:

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: WARNINGS:

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: api.IfRef.ipAddr: (fields.W900) IPAddressField has been deprecated. Support for it (except in historical migrations) will be removed in Django 1.9.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: HINT: Use GenericIPAddressField instead.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: api.LSPOpStatus.dstAddr: (fields.W900) IPAddressField has been deprecated. Support for it (except in historical migrations) will be removed in Django 1.9.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: HINT: Use GenericIPAddressField instead.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: api.LSPOpStatus.srcAddr: (fields.W900) IPAddressField has been deprecated. Support for it (except in historical migrations) will be removed in Django 1.9.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: HINT: Use GenericIPAddressField instead.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: Traceback (most recent call last):

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "esmond/manage.py", line 14, in <module>

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: execute_from_command_line(sys.argv)

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/management/__init__.py", line 351, in execute_from_command_line

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: utility.execute()

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/management/__init__.py", line 343, in execute

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: self.fetch_command(subcommand).run_from_argv(self.argv)

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: self.execute(*args, **cmd_options)

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: output = self.handle(*args, **options)

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/esmond/api/management/commands/add_api_key_user.py", line 36, in handle

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: u = User.objects.get(username=user)

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: return getattr(self.get_queryset(), name)(*args, **kwargs)

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: num = len(clone)

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: self._fetch_all()

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: self._result_cache = list(self.iterator())

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: results = compiler.execute_sql()

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 838, in execute_sql

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: cursor = self.connection.cursor()

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/base/base.py", line 164, in cursor

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: cursor = self.make_cursor(self._cursor())

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/base/base.py", line 135, in _cursor

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: self.ensure_connection()

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/base/base.py", line 130, in ensure_connection

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: self.connect()

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: six.reraise(dj_exc_type, dj_exc_value, traceback)

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/base/base.py", line 130, in ensure_connection

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: self.connect()

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/base/base.py", line 119, in connect

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: self.connection = self.get_new_connection(conn_params)

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: connection = Database.connect(**conn_params)

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: File "/usr/lib/esmond/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: conn = _connect(dsn, connection_factory=connection_factory, async=async)

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: django.db.utils.OperationalError: FATAL:  password authentication failed for user "esmond"

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: /usr/lib/perfsonar/scripts/psb_to_esmond.pl -d --pidfile=/var/run/psb_to_esmond.pid --logfile=/var/log/perfsonar/psb_to_esmond.log --user=perfsonar --group=perfsonar --statefile /var/lib/perfsonar/db_backups/psb_to_esmond.state --owmesh-dir /opt/perfsonar_ps/perfsonarbuoy_ma/etc --mapassword

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: Option mapassword requires an argument

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: clean_pSB_db.pl <options>

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: -h,--help                            displays this message.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: --maxdays days                       maximum age (in days) of data to keep in database. Not valid for bwctl databases. Defaults to 90.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: --maxmonths months                   maximum number of months to keep in database. Must be used for bwctl databases. Defaults to 3.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: --dbtype type                        Indicates type of data in database. Valid value are 'owamp', 'bwctl', and 'traceroute'. Defaults to 'owamp'.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: --dbuser user                        name of database user. Defaults to root.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: --dbpassword password                password to access database. Defaults to empty string.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: --dbhost host                        database host to access. Defaults to localhost.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: --owmesh-dir dir                     location of owmesh.conf file with database username and password.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: overrides dbuser, dbpassword, and dbhost.

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: --verbose                            increase amount of output from program

Aug 16 21:32:58 d5528c65188c perfsonar-psb_to_esmond: /etc/rc.d/init.d/perfsonar-psb_to_esmond start: perfSONAR MA Data Conversion could not be started

Aug 16 21:32:58 d5528c65188c systemd: perfsonar-psb_to_esmond.service: control process exited, code=exited status=3

Aug 16 21:32:58 d5528c65188c systemd: Failed to start SYSV: perfSONAR MA data conversion tool.

Aug 16 21:32:58 d5528c65188c systemd: Unit perfsonar-psb_to_esmond.service entered failed state.

Aug 16 21:32:58 d5528c65188c systemd: perfsonar-psb_to_esmond.service failed.

 

 

 

// Dockerfile

====

FROM centos7-systemd

 

# Base Perfsonar installs

RUN yum install -y epel-release

RUN yum install -y http://software.internet2.edu/rpms/el7/x86_64/main/RPMS/Internet2-repo-0.7-1.noarch.rpm

RUN yum clean all

RUN yum install -y perfsonar-toolkit

 

# Missing CentOs Utilities

RUN yum -y install traceroute mc net-tools ntp python-setuptools

RUN yum install -y ethtool

RUN yum clean all

 

RUN /usr/lib/perfsonar/scripts/configure_firewall install

RUN /usr/lib/perfsonar/scripts/service_watcher

 

CMD ["/usr/sbin/init"]

=====

 

Regards,

John

 

============

John Tai, Jr.

408-853-9540