perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Hi Zafar;

The 'pool' servers may be better than using the US based servers when it 
comes to the latency metric, but these are still not great in other 
respects.  The way the NTP pool works is to provide you with a 
potentially different server (though the generic DNS name) each time 
your server requests a time heartbeat.  This usually translates into 
high jitter, which is also not good for the measurement tools.  I think 
it would be best if you could try to find some stable stratum 1 or 
stratum 2 clocks in the region that you could directly poll instead of 
using the pool.

Let us know how the rest of the testing goes when you are able to SSH or 
otherwise access the machines.

Thanks;

-jason

On 11/27/10 1:14 AM, Zafar Gilani wrote:
> I've actually found and added some Asian NTP servers:
>
> pk.pool.ntp.org <http://pk.pool.ntp.org>
> 0.asia.pool.ntp.org <http://0.asia.pool.ntp.org>
> 1.asia.pool.ntp.org <http://1.asia.pool.ntp.org>
> 2.asia.pool.ntp.org <http://2.asia.pool.ntp.org>
> 3.asia.pool.ntp.org <http://3.asia.pool.ntp.org>
>
> On Sat, Nov 27, 2010 at 11:09 AM, Zafar Gilani 
> <
> <mailto:>>
>  wrote:
>
>     Hi Jason,
>
>     I've changed the NTP servers to closest ones but don't know of the
>     closest to Pakistan. I will try to get the list and add those as
>     replacement. We've blocked SSH on these machines so I won't be able
>     to run bwctl tests remotely. Will let you know on these. Thanks for
>     all the help! I really appreciate this.
>
>
>     On Fri, Nov 26, 2010 at 6:06 PM, Jason Zurawski
>     
> <
>  
> <mailto:>>
>  wrote:
>
>         Hi Zafar;
>
>
>         On 11/26/10 6:32 AM, Zafar Gilani wrote:
>
>             Hi Jason,
>
>             Checked server time, it seems to be fine.
>
>
>
>         One thought that springs to mind is that these hosts are
>         physically located in Pakistan?  The 'default' NTP servers that
>         come with the toolkit are located in the US on R&E networks.
>           The NTP peer status can be found in this manner:
>
>             
> [root@lab236
>  ~]# ntpq -p
>                  remote           refid      st t when poll reach
>             delay   offset  jitter
>             
> ==============================================================================
>             -otc2.psu.edu <http://otc2.psu.edu>    147.84.59.145    2 u
>               565 1024  377   23.018    0.010   0.110
>             *navobs1.oar.net <http://navobs1.oar.net> .USNO.           1
>             u  697 1024  377    7.271   -0.125   0.093
>             +2001:468:1:12:: 130.207.244.240  2 u  669 1024  377
>             27.101   -0.005   0.137
>             -2001:468:2:12:: 64.57.16.34      2 u  590 1024  377
>             30.220   -0.251   0.081
>             +2001:468:6:12:: .PPS.            1 u  611 1024  377
>             18.816   -0.165   0.076
>
>
>         In the case above, the 'delay' field shows how close the target
>         NTP servers are to my host.  Having these be small values is
>         good.  The latency to synchronize over the distance from the US
>         to Pakistan (more than 300ms when I did a ping) will have an
>         effect on the accuracy of the measurement tools.  This could
>         contribute to the error you report below.
>
>         If you know of NTP servers that are 'closer' it would be good to
>         add them to the NTP configuration file manually.  Edit
>         '/etc/ntp.conf' to add lines like this:
>
>           server NTP_SERVER_IN_PAKISTAN iburst
>
>         And either comment out or delete the other server lines.  Try to
>         find at least 3 (preferably 5) in that region of the world.
>         Restart NTP (sudo /etc/init.d/ntpd restart) after making any
>         changes.
>
>
>
>             The databases have data for
>             both owamp and bwctl tables for LA and BW nodes
>             respectively. However
>             the tests return the following:
>
>             For bwctl "remote peer cancelled test". However it does say
>             server
>             listening on xy port and binding to local address. For owamp
>             tests I see
>             first and last results but it also says 0 sent, 0 lost. This
>             is strange.
>
>             What do you think might be the problem? I do see data which
>             suggests the
>             firewall might not be the culprit.
>
>
>
>         If you see data, then firewalls are probably not to blame.  Try
>         to implement the suggestion to NTP above, but you can also try
>         this test immediately:
>
>           bwctl -c BW_HOST_IN_SCHEDULE -a 5
>           bwctl -s BW_HOST_IN_SCHEDULE -a 5
>
>         The 'a' option tells BWCTL to allow an extra bit of offset (+ or
>         - the specified value) when calculating the measurement
>         start/stop.  Its normally used in cases where the clock may be
>         off between hosts.  Raise the value to 10 or even 20 and see if
>         that allows the tests to complete.
>
>         I also think upgrading to 3.2 is a good idea, but lets solve
>         this problem first :)
>
>         Thanks;
>
>         -jason
>
>
>             On Thu, Nov 25, 2010 at 10:31 PM, Jason Zurawski
>             
> <
>  
> <mailto:>
>             
> <mailto:
>             
> <mailto:>>>
>  wrote:
>
>                 Hi Zafar;
>
>
>                 On 11/25/10 12:26 PM, Zafar Gilani wrote:
>
>                     Hi Jason,
>
>                     Thanks for the reply. I'm not able to access the
>             machines from
>                     my home,
>                     I'll try tomorrow morning and let you know.
>             Possibility of a
>                     firewall
>                     also comes to my mind. I also checked system clock
>             and it seemed
>                     to be
>                     correct. Addresses:
>
>                     BW: 115.186.131.105
>                     LA: 115.186.131.107
>
>
>
>                 I am able to reach both, one thing I am noticing is that
>             these are
>                 both 3.1.2.  I would suggest upgrading to 3.2 if at all
>             possible.
>
>                 In any event, report back if you are seeing data when
>             you get a
>                 chance - also if you suspect a firewall may be an issue,
>             try some
>                 simple tests to some of the hosts that are in the
>             regular testing
>                 groupings, e.g.:
>
>                   owping HOST_IN_LATENCY_TESTS
>                   bwctl -c HOST_IN_BW_TESTS
>                   bwctl -s HOST_IN_BW_TESTS
>
>                 If these fail for any of the hosts, firewalls may be an
>             issue.
>
>                 Thanks;
>
>                 -jason
>
>
>                     These machines are deployed at NUST, Islamabad,
>             Pakistan.
>
>                     On Thu, Nov 25, 2010 at 5:48 PM, Jason Zurawski
>             
> <
>  
> <mailto:>
>             
> <mailto:
>  
> <mailto:>>
>             
> <mailto:
>             
> <mailto:>
>             
> <mailto:
>             
> <mailto:>>>>
>
>                     wrote:
>
>                         Hi Zafar;
>
>                         I am CCing the performance-node-users list on
>             this response.
>                           Comments below:
>
>
>                         On 11/25/10 5:57 AM, Zafar Gilani wrote:
>
>                             Hi,
>
>                             I've deployed new BW and LA PS nodes but
>             can't seem to
>                     view any
>                             graphs.
>                             Whenever I go to Service Graphs ->
>             Throughput or Service
>                     Graphs ->
>                             One-Way Latency, I get the following:
>
>                             Problem Handling Request
>                             MA
>             */http://localhost:8085/perfSONAR_PS/services/pSB/*
>                     did not
>                             return
>                             the expected response, be sure it is
>             configured and
>                     populated
>                             with data.
>
>                             For OWAMP I try to view graph and receive
>             following:
>
>
>                                 Internal Error - Service returned data,
>             but it is not
>                             plotable for
>                                 this measurement pair.
>
>                             Anyone having any ideas? I've already
>             checked the
>                     scheduled tests.
>                             Everything seems to be correct. I also checked
>             http://psps.perfsonar.net/toolkit/FAQs.html#Q19 but I don't
>                             think this
>                             is the case as it has been more than 24
>             hours since the
>                     nodes
>                             were deployed.
>
>
>                         What is the address of this performance node, is
>             it world
>                     accessible
>                         or does it have a local address?
>
>                         I would suggest looking in the database by hand
>             to see if
>                     there is
>                         recent data:
>
>                            mysql -u root
>
>                         Once at the mysql prompt try these commands:
>
>                            use owamp;
>                            show tables;
>
>                         Look for a table that has data for 20101125_* or
>             similar.
>                       Try the
>                         same for bwctl:
>
>                            use bwctl;
>                            show tables;
>                            exit;
>
>                         There should be data for 201011_*.  If you don't
>             see recent data
>                         tables, you are not getting data from the
>             regular tests
>                     (even though
>                         you did set them up).  This could be because
>             your system
>                     clock is
>                         not correct/not running, firewalls are
>             preventing the
>                     BWCTL/OWAMP
>                         tests, or perhaps some other reason.
>
>                         Let us know what you see and we can debug
>             further, thanks;
>
>                         -jason