perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi George,

A couple more ideas:

1. What does /opt/perfsonar_ps/regular_testing/etc/regular_testing.conf look 
like. Maybe it's trying to send to the wrong place?

2. What happens when you run "curl 
http://archive.eos.nasa.gov/esmond/perfsonar/archive/?limit=1"; from the 
problematic host? 

Thanks,
Andy


On Mar 27, 2015, at 9:47 AM, "Uhl, George D. (GSFC-423.0)[SGT INC]" 
<>
 wrote:

> Andy,
> 
> I grabbed a snapshot of the esmond database records from the agent host
> for tests between it and the no agent host using the following URL
> 
> https://198.118.198.5/serviceTest/graphData.cgi?action=data&url=http://loca
> lhost/esmond/perfsonar/archive/&src=198.118.198.5&dest=152.61.6.5&start=142
> 0121813&end=1427379413&window=86400
> 
> The output is attached.  I’m hoping it can shed some light on why this
> data isn’t making it to the central MA.
> 
> Thanks,
> George
> 
> On 3/26/15, 4:16 PM, "Uhl, George D. (GSFC-423.0)[SGT INC]"
> <>
>  wrote:
> 
>> Andy,
>> 
>> It’s not a FW issue. I can reach the central MA from the problematic host
>> over ports 80 and 443. Both are running iptables that I control and there
>> is no other firewall in the path.
>> 
>> Thanks,
>> George
>> 
>> On 3/26/15, 2:10 PM, "Andrew Lake" 
>> <>
>>  wrote:
>> 
>>> Hi George,
>>> 
>>> That could definitely be it. If this is the only host experiencing that
>>> problem and since I believe you use a central MA, could it just be a
>>> firewall causing the timeout?
>>> 
>>> Thanks,
>>> Andy
>>> 
>>> 
>>> On Mar 26, 2015, at 1:51 PM, "Uhl, George D. (GSFC-423.0)[SGT INC]"
>>> <>
>>>  wrote:
>>> 
>>>> Andy,
>>>> 
>>>> In the regular_testing log file on the agent host that’s unable to
>>>> transmit data to the central MA, I have the following repeating error
>>>> message sequence:
>>>> 
>>>> 2015/03/26 13:25:13 (10072) ERROR> EsmondBase.pm:53
>>>> perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__
>>>> -
>>>> Error writing metadata (500) 500 Timeout
>>>> 2015/03/26 13:25:13 (10072) ERROR> MeasurementArchiveChild.pm:209
>>>> 
>>>> perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_re
>>>> s
>>>> ul
>>>> ts - Problem storing results: Error writing metadata: 500 Timeout
>>>> 2015/03/26 13:25:13 (10072) ERROR> MeasurementArchiveChild.pm:125
>>>> perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__
>>>> -
>>>> Problem handling test results: Problem storing results: Error writing
>>>> metadata: 500 Timeout at
>>>> 
>>>> /opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting
>>>> /
>>>> Ma
>>>> ster/MeasurementArchiveChild.pm line 122.
>>>> 
>>>> 
>>>> Could this be an indication of the problem?
>>>> 
>>>> Thanks,
>>>> George
>>>> 
>>>> On 3/25/15, 3:05 PM, "Uhl, George D. (GSFC-423.0)[SGT INC]"
>>>> <>
>>>>  wrote:
>>>> 
>>>>> Hi Andy,
>>>>> 
>>>>> In the statistics page that displays after I click on the on the
>>>>> orange
>>>>> cell, I get this:
>>>>> 
>>>>> graphUrl: 
>>>>> 
>>>>> http://archive.eos.nasa.gov/serviceTest/graphWidget.cgi?url=http://arch
>>>>> i
>>>>> ve
>>>>> .
>>>>> 
>>>>> eos.nasa.gov/esmond/perfsonar/archive/&dest=edclxw41.cr.usgs.gov&source
>>>>> =
>>>>> es
>>>>> d
>>>>> is-ps.eosdis.nasa.gov
>>>>> maUrl: http://archive.eos.nasa.gov/esmond/perfsonar/archive/
>>>>> 
>>>>> 
>>>>> When I click on a cell with current data, I get this:
>>>>> 
>>>>> Count: 112
>>>>> graphUrl: 
>>>>> 
>>>>> http://archive.eos.nasa.gov/serviceTest/graphWidget.cgi?url=http://arch
>>>>> i
>>>>> ve
>>>>> .
>>>>> 
>>>>> eos.nasa.gov/esmond/perfsonar/archive/&dest=nasatest2.asf.alaska.edu&so
>>>>> u
>>>>> rc
>>>>> e
>>>>> =esdis-ps.eosdis.nasa.gov
>>>>> Max: 4
>>>>> Standard_Deviation: 0.6628955437552
>>>>> Average: 0.330357142857143
>>>>> maUrl: http://archive.eos.nasa.gov/esmond/perfsonar/archive/
>>>>> Min: 0
>>>>> 
>>>>> 
>>>>> Thanks,
>>>>> George
>>>>> 
>>>>> On 3/23/15, 8:57 AM, "Andrew Lake" 
>>>>> <>
>>>>>  wrote:
>>>>> 
>>>>>> Hi George,
>>>>>> 
>>>>>> A no_agent host means that it cannot initiate tests or register any
>>>>>> data
>>>>>> to an MA. All it should do it make both the top and the bottom of the
>>>>>> box
>>>>>> use the MA set for the host that has the agent. If you click on the
>>>>>> orange box, on the page that loads the details of the check you
>>>>>> should
>>>>>> see a vertical tab labelled "Statistics". Click on that to expand it
>>>>>> and
>>>>>> verify it has the correct MA set for the maUrl displayed.
>>>>>> 
>>>>>> Thanks,
>>>>>> Andy
>>>>>> 
>>>>>> On Mar 19, 2015, at 1:52 PM, "Uhl, George D. (GSFC-423.0)[SGT INC]"
>>>>>> <>
>>>>>>  wrote:
>>>>>> 
>>>>>>> Andy,
>>>>>>> 
>>>>>>> I can run the tests by hand without a problem.  When I go to the
>>>>>>> toolkit page of esdis-ps.eosdis.nasa.gov and display the throughput
>>>>>>> and
>>>>>>> latency/loss graphs I have current data (see attached).  I verified
>>>>>>> connectivity to the standalone MA over port 80 & 443.  In a mesh,
>>>>>>> which
>>>>>>> hosts are reporting test results to the MA - the source, the
>>>>>>> destination
>>>>>>> or both?  What happens if one of them is a no_agent host?
>>>>>>> 
>>>>>>> In my case I have two agent hosts testing with one no_agent host.
>>>>>>> On
>>>>>>> both agent hosts, the throughput/latency/loss graphs shows the
>>>>>>> source is
>>>>>>> the no_agent host and the destination is the agent host.  Yet agent
>>>>>>> host
>>>>>>> is able to send data to the central MA,it¹s displaying on the
>>>>>>> dashboard,
>>>>>>> while the other is not.
>>>>>>> 
>>>>>>> I¹m not sure what log files will indicate the cause of the failure.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> George
>>>>>>> 
>>>>>>> From: Andrew Lake 
>>>>>>> <>
>>>>>>> Date: Wednesday, March 18, 2015 at 4:44 PM
>>>>>>> To: George Uhl 
>>>>>>> <>
>>>>>>> Cc: perfsonar-user 
>>>>>>> <>
>>>>>>> Subject: Re: [perfsonar-user] Dashboard unable to retrieve data
>>>>>>> 
>>>>>>> Hi george,
>>>>>>> 
>>>>>>> Looking at the data it appears the OWAMP tests stopped working
>>>>>>> around
>>>>>>> January 26th and the BWCTL tests ~January 23rd. You might want to
>>>>>>> double
>>>>>>> check the test is still configured. Could also be a firewall change
>>>>>>> maybe? If you run owping or bwctl by hand do the tests complete?
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Andy
>>>>>>> 
>>>>>>> 
>>>>>>> On Mar 18, 2015, at 9:42 AM, "Uhl, George D. (GSFC-423.0)[SGT INC]"
>>>>>>> <>
>>>>>>>  wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> I have a situation where I¹m unable to retrieve data from OWAMP or
>>>>>>>> BWCTL/iperf tests between two nodes on my dashboard.  The dashboard
>>>>>>>> is
>>>>>>>> run on a standalone centralized MA and I see test results data
>>>>>>>> between
>>>>>>>> these hosts in the esmond perfsonar archive.  One node,
>>>>>>>> esdis-ps.eosdis.nasa.gov is participating in a mesh.  The other
>>>>>>>> node,
>>>>>>>> edclxw41.cr.usgs.gov is a no_agent mesh member.  This same no_agent
>>>>>>>> member is running tests with another member, 207.151.223.222, and
>>>>>>>> those
>>>>>>>> results are being displayed in the dashboard.   I¹ve attached a
>>>>>>>> gzipped
>>>>>>>> copy of the esmond archive from the MA and a screen shot of the
>>>>>>>> dashboard.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> George
>>>>>>>> <MAesmondPSarchive.json.gz><Screen Shot 2015-03-18 at 9.37.00
>>>>>>>> AM.png>
>>>>>>> 
>>>>>>> <Screen Shot 2015-03-19 at 1.18.13 PM.png>
>>>>>> 
>>> 
>> 
> 
> <esmond_snapshot.txt>