perfSONAR User Q&A and Other Discussion

["Uhl, George D. (GSFC-423.0)[SGT INC]" <>]

Hi,

The following bwtraceroute command is generated from my regulartesting.conf file. it generates incorrect syntax in the exec_line traceroute command and throws the error: "Extra arg `40' (position 3, argc 8)”.  It also fails when I use the same syntax on the CLI.

# /usr/bin/bwtraceroute -s psmp-fra1-te.x-win.dfn.de -c enpl-pt2-10g.eos.nasa.gov -T tracepath,traceroute -B enpl-pt2-10g.eos.nasa.gov -4 -v -M 30 -l 40 -y a

Messages being sent to syslog(user,err)

bwtraceroute: Using psmp-fra1-te.x-win.dfn.de as the address for remote sender

bwtraceroute: Using enpl-pt2-10g.eos.nasa.gov as the address for remote receiver

bwtraceroute: Available in-common: traceroute 

bwtraceroute: Using tool: traceroute

bwtraceroute: Server 'enpl-pt2-10g.eos.nasa.gov' accepted test request at time 1481398927.265358

bwtraceroute: Client 'psmp-fra1-te.x-win.dfn.de' accepted test request at time 1481398927.265358

bwtraceroute: 27 seconds until test results available

SENDER START

bwctl: start_endpoint: 3690387711.064473

bwctl: run_endpoint: receiver: 169.154.197.28

bwctl: run_endpoint: sender: 195.37.209.102

bwctl: exec_line: traceroute -m 30 40 -n -s 195.37.209.102 169.154.197.28 40

bwctl: run_tool: tester: traceroute

bwctl: run_tool: receiver: 169.154.197.28

bwctl: run_tool: sender: 195.37.209.102

bwctl: start_tool: 3690387727.265482

Extra arg `40' (position 3, argc 8)

bwctl: local tool exited before expected with status=2

bwctl: stop_tool: 3690387727.266134

bwctl: stop_endpoint: 3690387730.524235

SENDER END

I tried running the same bwtraceroute command using IP addresses instead of FQDNs and got the same results.  It only works when I remove the “-l 40” parameter from the command line.

When I swap the sender and receiver, everything works fine:

# /usr/bin/bwtraceroute -s enpl-pt2-10g.eos.nasa.gov -c psmp-fra1-te.x-win.dfn.de -T tracepath,traceroute -B enpl-pt2-10g.eos.nasa.gov -4 -v -M 30 -l 40 -y a -E

Messages being sent to syslog(user,err)

bwtraceroute: Server side does not require an endpoint

bwtraceroute: Using enpl-pt2-10g.eos.nasa.gov as the address for remote sender

bwtraceroute: Using psmp-fra1-te.x-win.dfn.de as the address for remote receiver

bwtraceroute: Available in-common: traceroute 

bwtraceroute: Using tool: traceroute

bwtraceroute: Server 'psmp-fra1-te.x-win.dfn.de' accepted test request at time 1481398771.401221

bwtraceroute: Client 'enpl-pt2-10g.eos.nasa.gov' accepted test request at time 1481398771.401221

bwtraceroute: 27 seconds until test results available

SENDER START

bwctl: start_endpoint: 3690387555.158175

bwctl: run_endpoint: receiver: 195.37.209.102

bwctl: run_endpoint: sender: 169.154.197.28

bwctl: exec_line: traceroute -m 30 -n -s 169.154.197.28 195.37.209.102 40

bwctl: run_tool: tester: traceroute

bwctl: run_tool: receiver: 195.37.209.102

bwctl: run_tool: sender: 169.154.197.28

bwctl: start_tool: 3690387571.400429

traceroute to 195.37.209.102 (195.37.209.102), 30 hops max, 40 byte packets

 1  169.154.197.25  0.550 ms  0.599 ms  0.659 ms

 2  169.154.192.49  9.204 ms  9.193 ms  9.190 ms

 3  206.196.178.49  0.506 ms  0.502 ms  0.552 ms

 4  206.196.178.80  1.490 ms  1.587 ms  1.593 ms

 5  206.196.177.79  2.107 ms  2.145 ms  2.156 ms

 6  62.40.124.44  76.786 ms  76.781 ms  76.857 ms

 7  62.40.98.81  84.351 ms  84.346 ms  84.126 ms

 8  62.40.98.61  105.408 ms  105.507 ms  105.451 ms

 9  62.40.112.146  112.654 ms  112.455 ms  112.498 ms

10  188.1.146.209  100.048 ms  99.773 ms  100.214 ms

11  188.1.144.222  112.984 ms  112.569 ms  112.708 ms

12  195.37.209.102  99.601 ms  99.587 ms  99.492 ms

bwctl: stop_tool: 3690387571.592511

bwctl: stop_endpoint: 3690387571.592511

SENDER END

Thanks,

George

From: <> on behalf of Andrew Lake <>
Date: Friday, December 9, 2016 at 8:18 AM
To: Purna Mididuddi <>, "Garnizov, Ivan (RRZE)" <>, "" <>
Subject: RE: [perfsonar-user] Lookup service fails

Hi,

It depends on your setup. For one, if you’re behind a NAT, that limits some of the testing that can be done. You’ll be able to do throughput tests in both directions, but in order to make incoming tests work you’ll need to use the —flip option with BWCTL if testing from the command-line and set the local_firewall option in /etc/perfsonar/regulartesting.conf under your throughput test specification (http://docs.perfsonar.net/config_regular_testing.html#local-firewall-directive). For OWAMP tests you’ll just be able to do outgoing. Of course, this also depends on you have the correct firewall rules, which it already appears your network is blocking outgoing ports in some cases, so you might have an uphill battle there. See the firewall section of this page for more info on what is required of the firewall: http://www.perfsonar.net/deploy/security-considerations/

Thanks,

Andy

On December 8, 2016 at 5:01:12 PM, Purna Mididuddi () wrote:

Hi Andy,

Yes the curl on :8090 isn’t working either.

We just began testing out perfsonar and will eventually move to hosting the software (with a public IP) so others can reach. But for now, I’m looking to start off by doing the tests to the servers that are already hosted.

If not hosting the service, will I be able to run the basic tests like bwctl, owamp (the “perfsonar-notes”) manually from the private network?

 

Thanks,

Purna

 

From: Andrew Lake [mailto:]
Sent: Thursday, December 08, 2016 12:53 PM
To: Purna Mididuddi <>; Garnizov, Ivan (RRZE) <>;
Subject: RE: [perfsonar-user] Lookup service fails

 

Hi,

 

Looks like it might not like connections to http://ps-west.es.net:8090/lookup/records. You can verify that as well with:

 

 curl "http://ps-west.es.net:8090/lookup/records?type=host”

 

Unfortunately don’t have a good way around that. On the other hand, its nice to register from the perspective of knowing who is running the software, but if your host is running a private IP address, no one can reach it anyways to test so its probably not critical it gets registered.

 

Thanks,

Andy

 

 

 

On December 8, 2016 at 2:24:21 PM, Purna Mididuddi () wrote:

Thanks a lot Andrew and Ivan for the responses.

 

I’m trying this from a server in a corporate network. Most likely I’m behind a firewall (tried activehosts.json from home and that worked).

Added the ls_instance per Andrew’s suggestion. Also enabled the private IPs config (allow_internal_addresses     1).

 

Now I’m seeing the below issues. Is it because I have a private IP? Since I’m running from a corporate network, I cannot assign a public IP for this host. Is there a way to get around this. Thanks for the support.

 

2016/12/08 10:19:28 (8988) WARN> MA.pm:92 perfSONAR_PS::LSRegistrationDaemon::Services::MA::init_dependencies - Unable to get MA tests 404 Not Found

2016/12/08 10:19:28 (8988) INFO> Base.pm:257 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record is up, registering (description=em1)

2016/12/08 10:19:29 (8988) ERROR> Base.pm:312 perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem registering service. Will retry full registration next time: 500 Status read failed: Connection reset by peer(key=NONE, description=em1)

2016/12/08 10:19:29 (8988) INFO> Base.pm:257 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record is up, registering (description=10.24.11.23)

2016/12/08 10:19:29 (8988) ERROR> Base.pm:312 perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem registering service. Will retry full registration next time: 500 Status read failed: Connection reset by peer(key=NONE, description=10.24.11.23)

2016/12/08 10:19:29 (8988) INFO> Base.pm:257 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record is up, registering (description=Purna Co. Ping Responder)

2016/12/08 10:19:29 (8988) ERROR> Base.pm:312 perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem registering service. Will retry full registration next time: 500 Status read failed: Connection reset by peer(key=NONE, description=Purna Co. Ping Responder)

2016/12/08 10:19:29 (8988) INFO> Base.pm:257 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record is up, registering (description=Purna Co. Traceroute Responder)

2016/12/08 10:19:29 (8988) ERROR> Base.pm:312 perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem registering service. Will retry full registration next time: 500 Status read failed: Connection reset by peer(key=NONE, description=Purna Co. Traceroute Responder)

2016/12/08 10:19:29 (8988) INFO> Base.pm:257 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record is up, registering (description=Purna Co. OWAMP Server)

2016/12/08 10:19:29 (8988) ERROR> Base.pm:312 perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem registering service. Will retry full registration next time: 500 Status read failed: Connection reset by peer(key=NONE, description=Purna Co. OWAMP Server)

2016/12/08 10:19:29 (8988) INFO> Base.pm:257 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record is up, registering (description=Purna Co. BWCTL Server)

2016/12/08 10:19:29 (8988) ERROR> Base.pm:312 perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem registering service. Will retry full registration next time: 500 Status read failed: Connection reset by peer(key=NONE, description=Purna Co. BWCTL Server)

2016/12/08 10:19:29 (8988) INFO> Base.pm:273 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record is down (key=NONE, description=Purna Co. NDT Server)

2016/12/08 10:19:29 (8988) INFO> Base.pm:273 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record is down (key=NONE, description=Purna Co. NPAD Server)

2016/12/08 10:19:29 (8988) INFO> Base.pm:257 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record is up, registering (description=Purna Co. Measurement Archive)

2016/12/08 10:19:29 (8988) ERROR> Base.pm:312 perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem registering service. Will retry full registration next time: 500 Status read failed: Connection reset by peer(key=NONE, description=Purna Co. Measurement Archive)

2016/12/08 10:19:29 (8988) INFO> Base.pm:257 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record is up, registering (description=Purna Co. BWCTL Measurement Point)

2016/12/08 10:19:29 (8988) ERROR> Base.pm:312 perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem registering service. Will retry full registration next time: 500 Status read failed: Connection reset by peer(key=NONE, description=Purna Co. BWCTL Measurement Point)

2016/12/08 10:19:29 (8988) INFO> Base.pm:257 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record is up, registering (description=Purna Co. OWAMP Measurement Point)

2016/12/08 10:19:29 (8988) ERROR> Base.pm:312 perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem registering service. Will retry full registration next time: 500 Status read failed: Connection reset by peer(key=NONE, description=Purna Co. OWAMP Measurement Point)

 

Thanks,

Purna

 

From: Andrew Lake [mailto:]
Sent: Thursday, December 08, 2016 3:48 AM
To: Purna Mididuddi <>; Garnizov, Ivan (RRZE) <>;
Subject: RE: [perfsonar-user] Lookup service fails

 

Hi,

 

The lsregistrationdaemon by default bootstraps (i.e. gets the list of available lookup services) by downloading the a file at http://ps1.es.net:8096/lookup/activehosts.json. It appears to be available. If you have a proxy or firewall blocking access to that outgoing port, it may be the cause of your issue. If that is indeed the case, a workaround is to add “ls_instance http://ps-west.es.net:8090/lookup/records” to the top of your /etc/perfsonar/lsregistrationdaemon.conf file.

 

Hope that helps,

Andy

 

 

 

On December 8, 2016 at 4:58:02 AM, Garnizov, Ivan (RRZE) () wrote:

I just wanted to add this:

curl http://ps-west.es.net/lookup/activehosts.json
{"hosts":[{"priority":100,"locator":"http://ps-west.es.net:8090/lookup/records","status":"alive"},{"priority":100,"locator":"http://ps-east.es.net:8090/lookup/records","status":"alive"},{"priority":100,"locator":"http://monipe-ls.rnp.br:8090/lookup/records","status":"alive"},{"priority":100,"locator":"http://ps-sls.sanren.ac.za:8090/lookup/records","status":"alive"},{"priority":100,"locator":"http://nsw-brwy-sls1.aarnet.net.au:8090/lookup/records/","status":"alive"}]}

Regards,
Ivan Garnizov

GEANT SA1T2: pS deployments GN Operations
GEANT SA2T3: pS development team
GEANT SA3T5: eduPERT team

-----Original Message-----
From: [mailto:] On Behalf Of Garnizov, Ivan (RRZE)
Sent: Donnerstag, 8. Dezember 2016 10:49
To: ;
Subject: RE: [perfsonar-user] Lookup service fails

Hi Purna,

Please share your lsregistrationdaemon.conf, since I would not expect to have it registering with port 8096, but with 8090.

Regards,
Ivan Garnizov

GEANT SA1T2: pS deployments GN Operations
GEANT SA2T3: pS development team
GEANT SA3T5: eduPERT team


-----Original Message-----
From: [mailto:] On Behalf Of
Sent: Donnerstag, 8. Dezember 2016 09:43
To:
Subject: [perfsonar-user] Lookup service fails

Hi,
I'm new to perfSONAR. Would appreciate any help.
I've installed the ps-testpoint on Ubuntu 14. Followed all the instructions in http://docs.perfsonar.net/install_debian. Made basic changes to /etc/perfsonar/lsregistrationdaemon.conf.
After starting the etc/init.d/perfsonar-lsregistrationdaemon service, seeing the below logs in /var/log/perfsonar/lsregistrationdaemon.log.

2016/12/08 00:27:41 (6209) ERROR> LookupService.pm:113 perfSONAR_PS::Utils::LookupService::discover_lookup_services - Problem retrieving http://ps1.es.net:8096/lookup/activehosts.json: 500 Status read
failed: Connection reset by peer
2016/12/08 00:27:41 (6209) WARN> lsregistrationdaemon.pl:144 main::__ANON__ -
Warned: Use of uninitialized value $current_ls_instance in concatenation (.) or string at /usr/lib/perfsonar/bin/lsregistrationdaemon.pl line 257.
2016/12/08 00:27:41 (6209) INFO> lsregistrationdaemon.pl:257 main:: - Initial LS URL set to
2016/12/08 00:27:41 (6209) ERROR> lsregistrationdaemon.pl:301 main:: - Unable to determine ls_instance so not performing any operations


A curl request on http://ps1.es.net succeeds but fails with connection reset by peer when curl'ing on http://ps1.es.net:8096

Please help.

Thanks,
Purna