perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Zhi-Wei Lu writes:

 

curl gets nothing from pb -> p0, I was able to get output from other servers from pb. For example  between pb -> p5

"This is the pScheduler API server on p5.noc.ucdavis.edu (mammoth-v4.noc.ucdavis.edu)."

During the curl run, there are packet exchanges between pb<->p0, but there was no output on the command line, stuck there.

 

This problem started on Nov. 29, 2018, there were perfsonar updates on Nove 27 on all our servers (automatic update).

Currently,  between some of our servers, your "curl" commands work, but others fail.  

 

I have no problem pinning this on a problem with perfSONAR if that’s what it is, but right now I’m having a hard time making a case for it:

 

• The last release that made any changes to the toolkit’s firewall was 4.1.2 (September 13).  Nothing since has touched it.

 

• There was a two-day gap between the upgrade and the problem occurring, which could just be that the problem wasn’t noticed until then.

 

• Everything I can observe from where I sit (AS701, non-R&E) says that p0, p5 and pb are functioning correctly.  All three are reachable from here and their APIs answer requests as expected.  I’m able to run tasks on p0 and pb.  p5 won’t let me run tasks because of the same limit configuration problem we discussed earlier, but is otherwise functional.  (I have no reason to believe that contributes to the problem you’re seeing.)

 

• I also had a look at things from the perspective of melange-owamp.v4, which is listed in the lookup service.  It happily fields requests from p0 and pb (I can’t run anything from p5 because the limit configuration forbids it).  It also runs tests successfully from itself to p5 and pb but is unable to complete an HTTPS request to p0.

 

Based on your input and what I’ve been able to do from here, the recurring theme seems to be that hosts at UCD aren’t able to complete HTTPS requests to p0.  I think at this point, the network between them needs to be conclusively ruled out the cause.  The change happened over a holiday weekend, which isn’t an unusual time for changes to be deployed.  Having full access to p0 from off campus and the on-campus hosts having none leads me to wonder if an ACL in a network device near p0 has a mistake in it where a deny was done instead of an allow.

 

Hope that helps.

 

--Mark