perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Hi Lixin;

Thanks for the clarification, I understand your specific question now.  There 
are a couple of delays, let me try to explain them:

 - Delay between when the LS daemon runs on the local server, and talks to 
the remote server 
 - Delay between when the remote server talks to a caching service (an 
optimization we added - used to generate the GUI information for 'Global 
Performance Services') and creates a new file for everyone to download
 - Delay between when the local server downloads the latest copy of the 
cached content to populate the GUI

In general the answer is that it takes a couple of hours for this all to 
happen, as little as 2-3 if the timing is right, as many as 6-8 if it is 
wrong.  Since we know that your host is in the directory via that REST query 
that was shown below, there is nothing to worry about.  It will just take 
some time to get into the GUI displays, and there isn't much that can be done 
to speed that up.

The SElinux issue shouldn't be the root cause, but I believe we either set 
this to permissive or disabled.  

Thanks;

-jason

On Sep 27, 2013, at 1:40 PM, "Lixin Liu" 
<>
 wrote:

> Hi Jason,
> 
> Sorry I should mention that the latency host is down right now.
> Someone will take a look at the machine itself. Will let you
> know when it comes up.
> 
> So looks like the bandwidth host is registered, but how long I
> need to wait to see its services listed in "Global Performance
> Services"?
> 
> I noticed these two sites has one thing in common: SELinux is
> enabled. I disabled it, but not sure if that is root cause of
> our service registration issue.
> 
> Thanks,
> 
> Lixin.
> 
> 
>> -----Original Message-----
>> From: Jason Zurawski 
>> [mailto:]
>> Sent: September-27-13 10:26 AM
>> To: Lixin Liu
>> Cc: 
>> ;
>>  Performance Node Users
>> Subject: Re: [perf-node-users] Re: [perfsonar-user] LSRegistrationDaemon
>> duplicate checksum lookup/host/ error
>> 
>> Hi Lixin;
>> 
>> Looking in one of the global servers. I do see your bdw host:
>> 
>> http://ndb1.internet2.edu:8090/lookup/records?host-name=bdw-
>> umanitoba.westgrid.ca
>> 
>> Unfortunately you are correct in that the latency host has not registered, 
>> I
>> don't see that anywhere.  Can you send the latest log message for lat-
>> umanitoba.westgrid.ca again?  Everything after the point where you did the
>> removal of the db file and the restart.
>> 
>> Thanks;
>> 
>> -jason
>> 
>> On Sep 27, 2013, at 1:02 PM, "Lixin Liu" 
>> <>
>>  wrote:
>> 
>>> Hi Jason,
>>> 
>>> Thanks for your information.
>>> 
>>> I rebooted two hosts (in University of Saskatoon) and they are now
>>> showing up in the list of Global Services.
>>> 
>>> I still have issue with two other hosts (in University of Manitoba).
>>> I followed your suggestion. It has been more than two hours, but I
>>> still do not see this host in the list. However, there was a network
>>> (BGP) problem early today that may affect the registration.
>>> 
>>> Here is the log file (hostname bdw-umanitoba.westgrid.ca).
>>> 
>>> The NTP server on the host you mentioned was changed by the local admin
>>> to use local NTP servers, but there are only two servers in the config.
>>> I added two more and should be fine now.
>>> 
>>> Thanks,
>>> 
>>> Lixin.
>>> 
>>>> -----Original Message-----
>>>> From: Jason Zurawski 
>>>> [mailto:]
>>>> Sent: September-27-13 6:19 AM
>>>> To: Lixin Liu
>>>> Cc: 
>>>> ;
>>>>  Performance Node Users
>>>> Subject: Re: [perf-node-users] Re: [perfsonar-user] 
>>>> LSRegistrationDaemon
>>>> duplicate checksum lookup/host/ error
>>>> 
>>>> And naturally I meant "rm -f
>>>> /var/lib/perfsonar/ls_registration_daemon/lsKey.db" for the cache 
>>>> file …
>>>> 
>>>> Thanks;
>>>> 
>>>> -jason
>>>> 
>>>> On Sep 27, 2013, at 9:17 AM, Jason Zurawski 
>>>> <>
>>>>  wrote:
>>>> 
>>>>> Hi Lixin;
>>>>> 
>>>>> In looking at the logs, we don't see anything out of the ordinary, the
>>>> 'duplicate' message you see is actually just unfortunate wording on our
>>>> part
>>>> - it means 'renew' the registration instead of making a new one.  Since
>>>> you
>>>> noted you changed DNS/IP, it may be an issue of a stake cache.  Try the
>>>> following steps:
>>>>> 
>>>>>> sudo /etc/init.d/ls_registration_daemon stop
>>>>>> /var/lib/perfsonar/ls_registration_daemon/lsKey.db
>>>>>> sudo /etc/init.d/ls_registration_daemon start
>>>>> 
>>>>> 
>>>>> This will stop the service, delete the local db, and then start it all
>>>> over.  After a couple of hours the information should show up, or at a
>>>> minimum the logs will let us know if there is something else locally 
>>>> bad
>>>> that is going on.
>>>>> 
>>>>> As an aside, it appears that your NTP is not configured, some of the
>>>>> tools
>>>> may not work until that is fixed.  You may want to run the NTP
>>>> configuration
>>>> script again.
>>>>> 
>>>>> Thanks;
>>>>> 
>>>>> -jason
>>>>> 
>>>>> On Sep 26, 2013, at 3:43 PM, "Lixin Liu" 
>>>>> <>
>>>>>  wrote:
>>>>> 
>>>>>> Sorry I didn’t realize that the link is password protected and I
>>>>>> already logged as admin.
>>>>>> 
>>>>>> Here is the log.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Lixin.
>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: 
>>>>>>> 
>>>>>>>  [
>>>>>>> ]
>>>>>>>  On Behalf Of Lixin Liu
>>>>>>> Sent: September-26-13 12:36 PM
>>>>>>> To: 
>>>>>>> 
>>>>>>> Subject: [perfsonar-user] LSRegistrationDaemon duplicate checksum
>>>>>>> lookup/host/ error
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> On a number of our perfSONAR hosts, we get
>>>>>>> 
>>>>>>> perfSONAR_PS::LSRegistrationDaemon::Base::find_duplicate - Found
>>>> duplicate
>>>>>>> checksum lookup/host
>>>>>>> 
>>>>>>> error and hosts do not show up in "Global Service and Data View".
>>>>>>> These cases may be related to the IP and DNS changes. But I think
>>>>>>> hosts are configured correctly.
>>>>>>> 
>>>>>>> How do I correct the problem?
>>>>>>> 
>>>>>>> You can see the log from this link:
>>>>>>> 
>>>>>>> https://lat-
>>>>>>> umanitoba.westgrid.ca/toolkit/admin/logs/ls_registration_daemon.log
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Lixin.
>>>>>>> 
>>>>>>> 
>>>>>>> =======================
>>>>>>> Lixin Liu
>>>>>>> IT Services
>>>>>>> Simon Fraser University
>>>>>> 
>>>>>> <ls_registration_daemon.log>
>>> <ls_registration_daemon.log>