perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi,

Installing perfSONAR in AWS is not officially supported. You touched upon the “greatest hits” of why that’s the case :) I don't know of anyone that’s tried a full toolkit and got it working (though maybe someone on the user list will correct me). The NAT combined with differences in the underlying distro make this a challenge. I know of some cases where perfsonar-testpoint got successfully installed which is just the pieces you need to get regular testing going (no GUI or esmond). You still run into the NAT though which makes it difficult to run tests, particularly those incoming to the AWS instance. If you run pscheduler throughput tests by hand you can do a —flip option with reverse tests which essentially make the iperf transfer a download initiated by the AWS machine instead of an upload initiated by the remote machine. Packets flow in the same direction, but who initiates the TCP connection changes which makes the NAT happy. 

This has long been a line-item to at least understand better and we did spend some time last release cycle looking at things to plug holes in NATs without much success unfortunately.  I don't know that the fundamental NAT issues are going away anytime soon. I will say that if anyone in the community comes-up with good strategies or better recipes for getting the code on AWS we would very much welcome it.

Thanks,

Andy

On August 23, 2017 at 1:53:11 PM, Jim Nauer CWRU () wrote:

We have attempted to set up a simple PS node at Amazon in order to do some latency testing with history/trend graphing, but it is not going well.

First of all, the documentation does not appear to match reality when installing from the "bundle" rpm's instead of the ISO.  

The installation consisted of deploying the official CentOS 7 Server AMI,  installing EPEL & Perfsonar repos, then:

yum install perfsonar-toolkit

yum install perfsonar-toolkit-systemenv-testpoint

yum install perfsonar-toolkit-security

I then confirmed NTP config,  ran the sysctl & firewall configs, and confirmed that SELinux is Disabled.

Finally, I ran nptoolkit-configure.py to configure the first user account.  That script successfully created a web user account, but completely skipped the prompt to enable SSH login or sudo access.

Next, I tried the 'root' login thing - which can't be done on a default AWS / CentOS 7 image, so I just did 'sudo su - ' from the 'centos' account.  That prompted me to set up a sudo user (skipped the "Website Administrator", since one had already been created) and disable root login (which was already done by the base CentOS image).

After all of that, I was able to log in to the web GUI, but it has never quite worked right. On most screens, a "Loading..." spinner will appear but never go away until I click on the page - then it just disappears, without actually loading e.g. the "services" list on the home page (both "public" and "logged in" versions).

I had been able to define a couple of tests - ping and latency to an on-campus location, but none of them appear to run correctly, and I can't get the 'tests' page to load fully anymore (that "loading..." spinner thing again).

The "latency" test does collect & show traceroute data and a correct latency value from the ping/traceroute data, but also reports "100% loss ... (owamp) [powstream]".

Digging into the log files, I found that 'owampd' is trying to bind to the _outside_ Amazon Elastic IP [1:1 NAT] for the system, not its own "inside" (RFC-1918) IP address.  I found a note on this list from last year indicating that 'bwctl' had (has?) the same issue, and that it might be fixed "in the next version" - but that was a year ago.  Any progress on this front, or are OWAMP/BWCTL just not usable in a NAT/AWS environment?

OR...did I configure the system wrong?  After I found the bind-to-the-wrong-IP log message, I tried fixing things by setting up valid forward & reverse DNS for the "inside" IP (172.21.*), plus a DNS view that returns the outside IP (AWS Elastic IP) only to hosts querying from outside the AWS subnets. Then restarted PerfSONAR -- but that has made things worse. Now the web GUI won't show any of the "service" status information at all (where it did before).  I also cannot find any config files (in /etc/{perfsonar,owamp-server,pscheduler}/*)with IP address or the local host name in them, so I'm not sure where to look to get things back to the less-broken state (service status is displayed and tests can be configured).

Any helpful hints?  Or should I just nuke & re-install with the current DNS config, and live with a PS node that can record & graph 'ping' results but not much else?

--

James A. Nauer                   | "I shall not yield one whit of  maturity,
Engineer III, Enterprise Systems | not grace, not respectability, to  the
University Technology, [U]Tech   | passing of time. I declare that I  shall
Case Western Reserve University  | forever be, if not a child,  certainly
(216) 368-MACS  (368-6227)       | childish"  --Kennet Shardik
USPA D-25604