perfSONAR User Q&A and Other Discussion

[Zhi-Wei Lu <>]

Hi Mark,

 

Thank you very much for pointing me to the right direction!  The problem was not the limits.conf file or perfsonar update, it was the “hidden” security devices that were blocking certain traffic between certain hosts. 

 

Between p0 and p51, ssh, http, and other traffic works just fine, but the security device blocked the https traffic between those two hosts (and a few other personar boxes), pscheduler thus failed to work, and no perfsonar tests were scheduled and run.

 

After looking at the data, perfsonar tests between many hosts stopped working right after the security devices kicked inline (not all our traffics are sent to those security devices), it took us nearly two weeks to get management approval to remove perfsonar traffic from those security devices, and the security group responsible for the devices still claimed that no traffic were blocked for those perfsonar hosts!!!

 

Those security devices probably are causing problems for other systems with legitimate traffic, people may not notice right away.

 

Thanks again for helping us debug this problem.

 

Zhi-Wei Lu

IET-CR-Network Operations Center

University of California, Davis

(530) 752-0155

 

From: Mark Feit <>
Sent: Wednesday, January 09, 2019 9:21 AM
To: Zhi-Wei Lu <>;
Subject: Re: [perfsonar-user] RE: limits.conf file

 

Zhi-Wei Lu writes:

 

curl gets nothing from pb -> p0, I was able to get output from other servers from pb. For example  between pb -> p5

"This is the pScheduler API server on p5.noc.ucdavis.edu (mammoth-v4.noc.ucdavis.edu)."

During the curl run, there are packet exchanges between pb<->p0, but there was no output on the command line, stuck there.

 

This problem started on Nov. 29, 2018, there were perfsonar updates on Nove 27 on all our servers (automatic update).

Currently,  between some of our servers, your "curl" commands work, but others fail.  

 

I have no problem pinning this on a problem with perfSONAR if that’s what it is, but right now I’m having a hard time making a case for it:

 

• The last release that made any changes to the toolkit’s firewall was 4.1.2 (September 13).  Nothing since has touched it.

 

• There was a two-day gap between the upgrade and the problem occurring, which could just be that the problem wasn’t noticed until then.

 

• Everything I can observe from where I sit (AS701, non-R&E) says that p0, p5 and pb are functioning correctly.  All three are reachable from here and their APIs answer requests as expected.  I’m able to run tasks on p0 and pb.  p5 won’t let me run tasks because of the same limit configuration problem we discussed earlier, but is otherwise functional.  (I have no reason to believe that contributes to the problem you’re seeing.)

 

• I also had a look at things from the perspective of melange-owamp.v4, which is listed in the lookup service.  It happily fields requests from p0 and pb (I can’t run anything from p5 because the limit configuration forbids it).  It also runs tests successfully from itself to p5 and pb but is unable to complete an HTTPS request to p0.

 

Based on your input and what I’ve been able to do from here, the recurring theme seems to be that hosts at UCD aren’t able to complete HTTPS requests to p0.  I think at this point, the network between them needs to be conclusively ruled out the cause.  The change happened over a holiday weekend, which isn’t an unusual time for changes to be deployed.  Having full access to p0 from off campus and the on-campus hosts having none leads me to wonder if an ACL in a network device near p0 has a mistake in it where a deny was done instead of an allow.

 

Hope that helps.

 

--Mark