Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] issues with setting up a maddash mesh

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] issues with setting up a maddash mesh


Chronological Thread 
  • From: Michael Johnson <>
  • To: "Robinson, John-Paul" <>
  • Cc: "" <>
  • Subject: Re: [perfsonar-user] issues with setting up a maddash mesh
  • Date: Fri, 21 Sep 2018 17:30:51 -0400
  • Ironport-phdr: 9a23:vjpFcRRCW5+LAiTXjhDB3Sb/LNpsv+yvbD5Q0YIujvd0So/mwa6yYxCN2/xhgRfzUJnB7Loc0qyK6/+mATRIyK3CmUhKSIZLWR4BhJdetC0bK+nBN3fGKuX3ZTcxBsVIWQwt1Xi6NU9IBJS2PAWK8TW94jEIBxrwKxd+KPjrFY7OlcS30P2594HObwlSizexfbF/IA+qoQnNq8IbnZZsJqEtxxXTv3BGYf5WxWRmJVKSmxbz+MK994N9/ipTpvws6ddOXb31cKokQ7NYCi8mM30u683wqRbDVwqP6WACXWgQjxFFHhLK7BD+Xpf2ryv6qu9w0zSUMMHqUbw5Xymp4qF2QxHqlSgHLSY0/m/XhMJukaxVoxyhqBNjzIHJYo6YL/V+cr/HcN4AWWZNQsRcWipcCY28dYsPCO8BMP5foYn8u1QOrQGxCheoBOjyzjFIgX/23ash0+QmFwHNwQstH9cMsHnXstn4L6gSUeSyzKbT1zrDa/VW2TDh54TScxAhvOuAUqxtfsrM0EQiER7OgFuXqYzgJTyV1+INvnCa7+pmSeKglXQnqwdvrTiz2MsgkJfGiZ8Iylzc7iV5x4k0Kce+SEFlfd6pFoZbuSKCN4ZuX88vTGVltDwnxrAHuZO3ZjUGxZooyhLFafGKc46F6Q/5WumLOzd3nndldaq/hxms9UigzfXxVsyu31ZLqipJiMXDtmwR2BDJ5ciHUP19/kC91jqV1gDT9/tILl4pmqrGM5Ihw7gwmYQPsUnbAyP7l0H7gLWLekgl+uWk8fnrb7vpq5OGNYJ5ihnyMqE0lcy+BeQ4PBIOX2+e+emk0b3j/070TKtRg/05lqnZt4vVKt4fpq6/BA9azpwv6xeiDzu8ztsXgGQHIEpfeB2bl4jpJ03OIPfgAPennVusijlrx+vBPr3nGJrNNGHPkKr6fblj8U5R0w4zzdFE55JIEbENPuj/Wk73tNzEEBA5KQq0zPj7CNljzI8RR3+AArLKeJ7UqkKCs+IzP/GXNshSvDfmN+Nj5vjygGU/lENHO6SlwNwMeXWgF7N9IkqfZnTixc8MC2kRuQw3Vqn3k1CYAgJUMjyIWKg94Co8DsbuL4DDQMWViaGFlm/vHIVMTn1PABaBHWq+JKueXPJZTSOZIsZslnQmUrinUIgnnUWrtl+g47VjKeDd/isfvtTiyZ546/CFxkJ6ziB9E8nIizLFdGpzhG5dHzI=

Yes, and having the cassandra username and password empty in the django
config is expected.

When you run your script, does it give you some output? It is fine to use
whatever username you want, including a generic one ... you can assign
multiple IP addresses/CIDR blocks to one username if you want.

When you add the IP, it should say either that it created a record for the
IP/user, or that it already exists so it won't create it (or some error
message if it fails). Do you see any output from the script? What happens if
you manually run the same command on one of the IPs in question?

I'm a little confused by your syntax here:
sudo /usr/sbin/esmond_manage add_user_ip_address example_user $ip
143.215.191.0/26

The $ip part makes sense, but what is the purpose of 143.215.191.0/26 in your
script?

Thanks,
Michael

On Fri, Sep 21, 2018 at 06:38:39PM +0000, Robinson, John-Paul wrote:
More evidence that esmond is happy with cassandra:

workshop@soxws14:/usr/lib/esmond$ sudo /usr/sbin/esmond_manage check
cassandra_db [INFO] Checking/creating column families
cassandra_db [INFO] Schema check done
cassandra_db [DEBUG] Opening ConnectionPool
cassandra_db [INFO] Connected to ['localhost:9160']
cassandra_db [INFO] Checking/creating column families
cassandra_db [INFO] Checking/creating column families
cassandra_db [INFO] Schema check done
cassandra_db [INFO] Schema check done
cassandra_db [DEBUG] Opening ConnectionPool
cassandra_db [DEBUG] Opening ConnectionPool
cassandra_db [INFO] Connected to ['localhost:9160']
cassandra_db [INFO] Connected to ['localhost:9160']
System check identified no issues (0 silenced).

I'm focusing on the esmond permissions from clients now.

sudo /usr/sbin/esmond_manage help add_user_ip_addr

On 09/21/2018 02:29 PM, Robinson, John-Paul wrote:
Thanks for the hints.  I agree these are the likely problem places.

I stopped cassandra and that cleaned up the processes.   Started it up
but it has the same status.  While it was down, django did log
complaints about not getting to cassandra.  It doesn't say anything
about getting to cassandra, so it either is or is not reaching cassandra. :)

The systemctl status may be due to systemctl loosing track of the
process it starts.  I'll look there.

Considering the danjgo config, does it need a user/pass to connect
cassandra?  In the config that go created we have:

cassandra_servers = localhost:9160
cassandra_user =
cassandra_pass =

Given the complaints when cassandra went down, I'm favoring the IP authz
rules issue.  Any hints on what needs to be set here?  Is using a
generic user name valid or does that need to be explicitly configured?

I do see clients connecting to the maddash server via netstat (https and
owamp) so I assume they are succeeding on the TCP level at least.

John-Paul


On 09/21/2018 02:10 PM, Michael Johnson wrote:
Hi,

Comments below:

On Fri, Sep 21, 2018 at 05:12:37PM +0000, Robinson, John-Paul wrote:

We suspect something is up with cassandra the status shows somewhat
confusing output "active (exited)":

workshop@soxws14:~/projects/mesh$ sudo systemctl status cassandra
● cassandra.service - SYSV: Starts and stops Cassandra
   Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset:
disabled)
   Active: active (exited) since Fri 2018-09-21 16:43:46 UTC; 24min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 5622 ExecStop=/etc/rc.d/init.d/cassandra stop (code=exited,
status=1/FAILURE)
  Process: 5934 ExecStart=/etc/rc.d/init.d/cassandra start
(code=exited, status=0/SUCCESS)

Sep 21 16:43:46 soxws14.sox.net systemd[1]: Starting SYSV: Starts and
stops Cassandra...
Sep 21 16:43:46 soxws14.sox.net su[5943]: (to cassandra) root on none
Sep 21 16:43:46 soxws14.sox.net cassandra[5934]: Starting Cassandra: OK
Sep 21 16:43:46 soxws14.sox.net systemd[1]: Started SYSV: Starts and
stops Cassandra.
To clarify a couple points, "active (exited)" means the service is
*enabled* (this is what "active" means in this context), but it is not
running (exited).

I would suggest stopping cassandra using
$ sudo systemctl stop cassandra

and then checking to make sure there are no more cassandra processes
running:

$ ps aux | grep cassandra

If there are any remaining processes, kill them. Then start cassandra
back up.

Your statement that local tests are being archived leads me to believe
that cassandra is running and your other hosts are unable to
authenticate (in this case, your ip auth may not be working).

On the other hand, those Django errors usually mean Cassandra is
down/unreachable from the point of view of esmond, regardless of what
cassandra itself tells you.
These two ideas do seem to conflict, but I'd try these steps to see
where they gets you.

Thanks,
Michael



--
To unsubscribe from this list:
https://lists.internet2.edu/sympa/signoff/perfsonar-user


--
Michael Johnson
GlobalNOC DevOps Engineer

Attachment: smime.p7s
Description: S/MIME cryptographic signature




Archive powered by MHonArc 2.6.19.

Top of Page