perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

On Mar 25, 2015, at 7:31 AM, Pedro Henrique Diniz Da Silva <> wrote:

Hi Andy,

Two problems that I think that can be correlated: 

#1 - The traceroute data for those tests is being collected by an old mesh config that is not being held anymore, it is called "Traceroute Test Between WLCG Bandwidth Hosts". As these traceroutes tests are provided by a mesh config I don't know how to disable it, is there a way to do that?

If you no longer want to participate in the mesh, you can remove the <mesh></mesh> from /opt/perfsonar_ps/mesh_config/etc/agent_configuration.conf. That being said, I don't think this has anything to do with the problem you are trying to debug in #2 (i.e. missin throughout data) other than its clogging up your interface and making it harder to tell what throughput tests are actually working.

#2 - The main problem is that I have other bw tests that have been held (e.g. perfsonar-bw.cern.ch, but it's not the only one) but is not being graphed. That's why I thought that cleaning the MA could solve the problem. 

* For example, based on "/var/log/perfsonar/owamp_bwctl.log" the connections seem to be working properly, right?

Those log messages could be just from the bwtraceroute command, not necessarily iperf. They also look more like other hosts connecting to you rather than vice versa. Maybe you could send me /opt/perfsonar_ps/regular_testing/etc/regular_testing.conf and /var/log/perfsonar/regular_testing.log? That should hopefully give a better idea of what your host is configured to run and what types of results it gets when it tries.

[gridlafex@ps02 ~]$ cat /var/log/perfsonar/owamp_bwctl.log | grep "Mar 25" | grep "cern"

Mar 25 00:14:57 ps02 bwctld[22298]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-omd-dev.cern.ch:56275)

Mar 25 00:29:41 ps02 bwctld[24769]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:53194)

Mar 25 00:29:42 ps02 bwctld[24769]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:53194)

Mar 25 01:14:57 ps02 bwctld[483]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-omd-dev.cern.ch:46295)

Mar 25 01:32:12 ps02 bwctld[4133]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:56909)

Mar 25 01:32:12 ps02 bwctld[4133]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:56909)

Mar 25 01:34:35 ps02 bwctld[4582]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:57052)

Mar 25 01:34:36 ps02 bwctld[4582]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:57052)

Mar 25 01:35:57 ps02 bwctld[4844]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:6183) from (perfsonar-bw.cern.ch:49859)

Mar 25 01:35:57 ps02 bwctld[4844]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:6183) accepted from userid(nil):([perfsonar-bw.cern.ch]:49859)

Mar 25 02:14:57 ps02 bwctld[11916]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-omd-dev.cern.ch:36320)

Mar 25 02:27:24 ps02 bwctld[14098]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:59846)

Mar 25 02:27:24 ps02 bwctld[14098]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:59846)

Mar 25 02:29:46 ps02 bwctld[14511]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:60246)

Mar 25 02:29:46 ps02 bwctld[14511]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:60246)

Mar 25 03:14:57 ps02 bwctld[22734]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-omd-dev.cern.ch:54659)

Mar 25 03:27:40 ps02 bwctld[24944]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:35327)

Mar 25 03:27:41 ps02 bwctld[24944]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:35327)

Mar 25 03:31:55 ps02 bwctld[25690]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:36115)

Mar 25 03:31:56 ps02 bwctld[25690]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:36115)

Mar 25 03:35:34 ps02 bwctld[26319]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:6176) from (perfsonar-bw.cern.ch:43720)

Mar 25 03:35:34 ps02 bwctld[26319]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:6176) accepted from userid(nil):([perfsonar-bw.cern.ch]:43720)

Mar 25 03:41:34 ps02 bwctld[27535]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:6154) from (perfsonar-bw.cern.ch:49137)

Mar 25 03:41:35 ps02 bwctld[27535]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:6154) accepted from userid(nil):([perfsonar-bw.cern.ch]:49137)

Mar 25 04:14:57 ps02 bwctld[1208]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-omd-dev.cern.ch:44801)

Mar 25 04:29:17 ps02 bwctld[4262]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:39159)

Mar 25 04:29:18 ps02 bwctld[4262]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:39159)

Mar 25 04:30:43 ps02 bwctld[4611]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:6172) from (perfsonar-bw.cern.ch:48749)

Mar 25 04:30:43 ps02 bwctld[4611]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:6172) accepted from userid(nil):([perfsonar-bw.cern.ch]:48749)

Mar 25 04:31:07 ps02 bwctld[4690]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:39623)

Mar 25 04:31:08 ps02 bwctld[4690]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:39623)

Mar 25 05:14:57 ps02 bwctld[12735]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-omd-dev.cern.ch:38553)

Mar 25 05:27:06 ps02 bwctld[14844]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:42857)

Mar 25 05:27:06 ps02 bwctld[14844]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:42857)

Mar 25 05:36:06 ps02 bwctld[16444]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:43875)

Mar 25 05:36:06 ps02 bwctld[16444]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:43875)

Mar 25 06:14:57 ps02 bwctld[23454]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-omd-dev.cern.ch:56949)

Mar 25 06:27:30 ps02 bwctld[25552]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:46630)

Mar 25 06:27:31 ps02 bwctld[25552]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:46630)

Mar 25 06:38:10 ps02 bwctld[27431]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:6132) from (perfsonar-bw.cern.ch:35187)

Mar 25 06:38:10 ps02 bwctld[27431]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:6132) accepted from userid(nil):([perfsonar-bw.cern.ch]:35187)

Mar 25 06:40:25 ps02 bwctld[27785]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:47871)

Mar 25 06:40:25 ps02 bwctld[27785]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:47871)

Mar 25 07:14:57 ps02 bwctld[1738]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-omd-dev.cern.ch:46971)

Mar 25 07:32:51 ps02 bwctld[3893]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:50859)

Mar 25 07:32:51 ps02 bwctld[3893]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:50859)

Mar 25 07:43:01 ps02 bwctld[9198]: FILE=sapi.c, LINE=353, Connection to (ps02.cat.cbpf.br:4823) from (perfsonar-bw.cern.ch:51742)

Mar 25 07:43:01 ps02 bwctld[9198]: FILE=sapi.c, LINE=510, ControlSession([ps02.cat.cbpf.br]:4823) accepted from userid(nil):([perfsonar-bw.cern.ch]:51742)

* Based on "/var/log/esmond/esmond.log" it doesn't seem that ESMOND has problems.

[gridlafex@ps02 ~]$ cat /var/log/esmond/esmond.log | grep "2015-03-25"

2015-03-25 02:30:01,396 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 02:30:01,398 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 02:30:01,410 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 02:30:01,423 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 02:30:01,424 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 02:30:01,434 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:23:46,446 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:23:46,448 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:23:46,459 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:24:12,584 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:24:12,586 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:24:12,596 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:24:13,785 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:24:13,787 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:24:13,796 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:24:14,762 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:24:14,764 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:24:14,774 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:24:33,834 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:24:33,836 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:24:33,845 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:24:34,571 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:24:34,573 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:24:34,582 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:24:57,833 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:24:57,835 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:24:57,844 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:30:13,079 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:30:13,081 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:30:13,092 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:30:13,844 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:30:13,846 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:30:13,857 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:30:31,111 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:30:31,113 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:30:31,122 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:40:48,656 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:40:48,712 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:40:48,761 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:40:49,886 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:40:49,890 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:40:49,905 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

2015-03-25 07:46:30,561 [INFO] /opt/esmond/esmond/cassandra.py: Checking/creating column families

2015-03-25 07:46:30,564 [INFO] /opt/esmond/esmond/cassandra.py: Schema check done

2015-03-25 07:46:30,578 [INFO] /opt/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

* Is there any other place that we could look for errors?

Thank you very much,

Pedro Diniz

---

From: Andrew Lake []
Sent: Tuesday, March 24, 2015 10:04 AM
To: Pedro Henrique Diniz Da Silva
Cc: 
Subject: Re: [perfsonar-user] How to clean esmond/perfsonar/archive ????

Hi Pedro,

It looks like the script worked correctly, but you are still collecting traceroute data for those tests. Using the example given there is no BWCTL or OWAMP data, but traceroute is still being updated. The graphs do provide a link to the traceroute data if available, so I think this is a subtlety of how that initial page determines what to display. You need to disable the traceroute tests and then you can either re-run the script in a few days or they will finally disappear on their own.

Thanks,

Andy

On Mar 23, 2015, at 7:24 PM, Pedro Henrique Diniz Da Silva <> wrote:

Hi Andy,

Yes, there was an extra comma :|

I tried all the steps you told and it seemed that the data could be cleaned.

But even after doing all the steps, the error seems to be the same.

* Nodes that we aren't doing more tests continue to appear on http://ps02.cat.cbpf.br/serviceTest/psGraph.cgi

* On the graph pages above, the error is still the same as following: http://ps02.cat.cbpf.br/serviceTest/graphWidget.cgi?source=81.180.86.41&dest=152.84.101.142&url="http%3A%2F%2Flocalhost%2Fesmond%2Fperfsonar%2Farchive%2F

 

Any idea?

Cheers,

Pedro Diniz

---

From: Andrew Lake []
Sent: Monday, March 23, 2015 11:06 AM
To: Pedro Henrique Diniz Da Silva
Cc: 
Subject: Re: [perfsonar-user] How to clean esmond/perfsonar/archive ????

Hi Pedro,

That's my fault, there was an extra comma in the config file I sent you. See attached for a valid file.

Thanks,

Andy