perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Lixin;

There are several directory servers that manage registrations, not just the 
one.  I am able to see some of the hosts you mention below, note that 
sometimes a host registers with an IP address if there was a lack of DNS name 
to start:

> bdw-usask.westgrid.ca
        http://ndb1.internet2.edu:8090/lookup/records?host-name=206.12.26.19

> bdw-sfu.westgrid.ca
        http://antg.es.net:8090/lookup/records?host-name=bdw-sfu.westgrid.ca

> bdw-ucalgary.westgrid.ca
        *cannot find*

> bdw-ubc.westgrid.ca
        http://antg.es.net:8090/lookup/records?host-name=206.12.24.189

> bdw-uvic.westgrid.ca
        *cannot find*

> lat-sfu.westgrid.ca
        I see this: 
http://ps4.es.net:9095/lookup/records?host-name=ps-latency.sfu.westgrid.ca - 
did the host's name change recently?

> lat-ucalgary.westgrid.ca
        *cannot find*

> lat-ubc.westgrid.ca
        *cannot find*

> lat-uvic.westgrid.ca

        http://antg.es.net:8090/lookup/records?host-name=lat-uvic.westgrid.ca

You can try to remove the database on the hosts as you did before if the 
hosts had a recent DNS or IP address change.  You can also check to be sure 
there are no firewalls blocking registration.  

Thanks;

-jason

On Sep 30, 2013, at 12:07 PM, Lixin Liu 
<>
 wrote:

> Hi Jason,
> 
> Thank you very much. Indeed these two are on the list now.
> 
> However, there are a number of hosts previous in the list
> are disappeared from the list over the weekend. They are:
> 
> bdw-usask.westgrid.ca
> bdw-sfu.westgrid.ca
> bdw-ucalgary.westgrid.ca
> bdw-ubc.westgrid.ca
> bdw-uvic.westgrid.ca
> lat-sfu.westgrid.ca
> lat-ucalgary.westgrid.ca
> lat-ubc.westgrid.ca
> lat-uvic.westgrid.ca
> 
> Your URL:
> 
> http://ndb1.internet2.edu:8090/lookup/records?host-name=<hostname>
> 
> display only [].
> 
> Should I try to restart ls_registration_daemon on them? Do I need
> to remove the database?
> 
> Thanks again.
> 
> Lixin.
> 
> 
>> -----Original Message-----
>> From: Jason Zurawski 
>> [mailto:]
>> Sent: September-30-13 6:35 AM
>> To: Lixin Liu
>> Cc: 
>> ;
>>  Performance Node Users
>> Subject: Re: [perf-node-users] [perfsonar-user] LSRegistrationDaemon
>> duplicate checksum lookup/host/ error
>> 
>> Hi Lixin;
>> 
>> We were able to track down an operational issue on our end with the 
>> caching
>> service - as I previously noted your two servers are showing up, so this 
>> is
>> not a problem on your end:
>> 
>>  http://ndb1.internet2.edu:8090/lookup/records?host-name=bdw-
>> umanitoba.westgrid.ca
>>  http://ndb1.internet2.edu:8090/lookup/records?host-name=lat-
>> umanitoba.westgrid.ca
>> 
>> They were not being placed into the cache though because, the Internet2 LS
>> had a typo in the URL.  If you do the following:
>> 
>>  sudo /etc/init.d/ls_cache_daemon restart
>> 
>> It will re-download the cache file and your hosts should appear in the
>> Global Listing.
>> 
>> Thanks;
>> 
>> -jason
>> 
>> On Sep 29, 2013, at 12:07 PM, Lixin Liu 
>> <>
>>  wrote:
>> 
>>> Hi Jason,
>>> 
>>> I still do not see lat-umanitoba.westgrid.ca and 
>>> bdw-umanitoba.westgrid.ca
>>> in
>>> the Global Performance Services, and I can't see them in the 
>>> ComputeCanada
>>> community. I even do you suggested (stop registration, remove db and 
>>> start
>>> registration) on bdw-umanitoba yesterday and still unable to see.
>>> 
>>> Is there anything else I can check?
>>> 
>>> Thanks,
>>> 
>>> Lixin.
>>> 
>>> On 2013-09-27 10:58 AM, "Jason Zurawski" 
>>> <>
>>>  wrote:
>>> 
>>>> Hi Lixin;
>>>> 
>>>> Thanks for the clarification, I understand your specific question now.
>>>> There are a couple of delays, let me try to explain them:
>>>> 
>>>> - Delay between when the LS daemon runs on the local server, and talks
>>>> to the remote server
>>>> - Delay between when the remote server talks to a caching service (an
>>>> optimization we added - used to generate the GUI information for 
>>>> 'Global
>>>> Performance Services') and creates a new file for everyone to download
>>>> - Delay between when the local server downloads the latest copy of the
>>>> cached content to populate the GUI
>>>> 
>>>> In general the answer is that it takes a couple of hours for this all 
>>>> to
>>>> happen, as little as 2-3 if the timing is right, as many as 6-8 if it 
>>>> is
>>>> wrong.  Since we know that your host is in the directory via that REST
>>>> query that was shown below, there is nothing to worry about.  It will
>>>> just take some time to get into the GUI displays, and there isn't much
>>>> that can be done to speed that up.
>>>> 
>>>> The SElinux issue shouldn't be the root cause, but I believe we either
>>>> set this to permissive or disabled.
>>>> 
>>>> Thanks;
>>>> 
>>>> -jason
>>>> 
>>>> On Sep 27, 2013, at 1:40 PM, "Lixin Liu" 
>>>> <>
>>>>  wrote:
>>>> 
>>>>> Hi Jason,
>>>>> 
>>>>> Sorry I should mention that the latency host is down right now.
>>>>> Someone will take a look at the machine itself. Will let you
>>>>> know when it comes up.
>>>>> 
>>>>> So looks like the bandwidth host is registered, but how long I
>>>>> need to wait to see its services listed in "Global Performance
>>>>> Services"?
>>>>> 
>>>>> I noticed these two sites has one thing in common: SELinux is
>>>>> enabled. I disabled it, but not sure if that is root cause of
>>>>> our service registration issue.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Lixin.
>>>>> 
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Jason Zurawski 
>>>>>> [mailto:]
>>>>>> Sent: September-27-13 10:26 AM
>>>>>> To: Lixin Liu
>>>>>> Cc: 
>>>>>> ;
>>>>>>  Performance Node Users
>>>>>> Subject: Re: [perf-node-users] Re: [perfsonar-user]
>>>>>> LSRegistrationDaemon
>>>>>> duplicate checksum lookup/host/ error
>>>>>> 
>>>>>> Hi Lixin;
>>>>>> 
>>>>>> Looking in one of the global servers. I do see your bdw host:
>>>>>> 
>>>>>> http://ndb1.internet2.edu:8090/lookup/records?host-name=bdw-
>>>>>> umanitoba.westgrid.ca
>>>>>> 
>>>>>> Unfortunately you are correct in that the latency host has not
>>>>>> registered,
>>>>>> I
>>>>>> don't see that anywhere.  Can you send the latest log message for 
>>>>>> lat-
>>>>>> umanitoba.westgrid.ca again?  Everything after the point where you 
>>>>>> did
>>>>>> the
>>>>>> removal of the db file and the restart.
>>>>>> 
>>>>>> Thanks;
>>>>>> 
>>>>>> -jason
>>>>>> 
>>>>>> On Sep 27, 2013, at 1:02 PM, "Lixin Liu" 
>>>>>> <>
>>>>>>  wrote:
>>>>>> 
>>>>>>> Hi Jason,
>>>>>>> 
>>>>>>> Thanks for your information.
>>>>>>> 
>>>>>>> I rebooted two hosts (in University of Saskatoon) and they are now
>>>>>>> showing up in the list of Global Services.
>>>>>>> 
>>>>>>> I still have issue with two other hosts (in University of Manitoba).
>>>>>>> I followed your suggestion. It has been more than two hours, but I
>>>>>>> still do not see this host in the list. However, there was a network
>>>>>>> (BGP) problem early today that may affect the registration.
>>>>>>> 
>>>>>>> Here is the log file (hostname bdw-umanitoba.westgrid.ca).
>>>>>>> 
>>>>>>> The NTP server on the host you mentioned was changed by the local
>>>>>>> admin
>>>>>>> to use local NTP servers, but there are only two servers in the
>>>>>>> config.
>>>>>>> I added two more and should be fine now.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Lixin.
>>>>>>> 
>>>>>>>> -----Original Message-----
>>>>>>>> From: Jason Zurawski 
>>>>>>>> [mailto:]
>>>>>>>> Sent: September-27-13 6:19 AM
>>>>>>>> To: Lixin Liu
>>>>>>>> Cc: 
>>>>>>>> ;
>>>>>>>>  Performance Node Users
>>>>>>>> Subject: Re: [perf-node-users] Re: [perfsonar-user]
>>>>>>>> LSRegistrationDaemon
>>>>>>>> duplicate checksum lookup/host/ error
>>>>>>>> 
>>>>>>>> And naturally I meant "rm -f
>>>>>>>> /var/lib/perfsonar/ls_registration_daemon/lsKey.db" for the cache
>>>>>>>> file Š
>>>>>>>> 
>>>>>>>> Thanks;
>>>>>>>> 
>>>>>>>> -jason
>>>>>>>> 
>>>>>>>> On Sep 27, 2013, at 9:17 AM, Jason Zurawski 
>>>>>>>> <>
>>>>>>>>  
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Lixin;
>>>>>>>>> 
>>>>>>>>> In looking at the logs, we don't see anything out of the ordinary,
>>>>>>>>> the
>>>>>>>> 'duplicate' message you see is actually just unfortunate wording on
>>>>>>>> our
>>>>>>>> part
>>>>>>>> - it means 'renew' the registration instead of making a new one.
>>>>>>>> Since
>>>>>>>> you
>>>>>>>> noted you changed DNS/IP, it may be an issue of a stake cache.  Try
>>>>>>>> the
>>>>>>>> following steps:
>>>>>>>>> 
>>>>>>>>>> sudo /etc/init.d/ls_registration_daemon stop
>>>>>>>>>> /var/lib/perfsonar/ls_registration_daemon/lsKey.db
>>>>>>>>>> sudo /etc/init.d/ls_registration_daemon start
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> This will stop the service, delete the local db, and then start it
>>>>>>>>> all
>>>>>>>> over.  After a couple of hours the information should show up, or 
>>>>>>>> at
>>>>>>>> a
>>>>>>>> minimum the logs will let us know if there is something else 
>>>>>>>> locally
>>>>>>>> bad
>>>>>>>> that is going on.
>>>>>>>>> 
>>>>>>>>> As an aside, it appears that your NTP is not configured, some of 
>>>>>>>>> the
>>>>>>>>> tools
>>>>>>>> may not work until that is fixed.  You may want to run the NTP
>>>>>>>> configuration
>>>>>>>> script again.
>>>>>>>>> 
>>>>>>>>> Thanks;
>>>>>>>>> 
>>>>>>>>> -jason
>>>>>>>>> 
>>>>>>>>> On Sep 26, 2013, at 3:43 PM, "Lixin Liu" 
>>>>>>>>> <>
>>>>>>>>>  wrote:
>>>>>>>>> 
>>>>>>>>>> Sorry I didn¹t realize that the link is password protected and I
>>>>>>>>>> already logged as admin.
>>>>>>>>>> 
>>>>>>>>>> Here is the log.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> 
>>>>>>>>>> Lixin.
>>>>>>>>>> 
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: 
>>>>>>>>>>> 
>>>>>>>>>>>  
>>>>>>>>>>> [
>>>>>>>>>>> ]
>>>>>>>>>>>  On Behalf Of Lixin Liu
>>>>>>>>>>> Sent: September-26-13 12:36 PM
>>>>>>>>>>> To: 
>>>>>>>>>>> 
>>>>>>>>>>> Subject: [perfsonar-user] LSRegistrationDaemon duplicate 
>>>>>>>>>>> checksum
>>>>>>>>>>> lookup/host/ error
>>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> On a number of our perfSONAR hosts, we get
>>>>>>>>>>> 
>>>>>>>>>>> perfSONAR_PS::LSRegistrationDaemon::Base::find_duplicate - Found
>>>>>>>> duplicate
>>>>>>>>>>> checksum lookup/host
>>>>>>>>>>> 
>>>>>>>>>>> error and hosts do not show up in "Global Service and Data 
>>>>>>>>>>> View".
>>>>>>>>>>> These cases may be related to the IP and DNS changes. But I 
>>>>>>>>>>> think
>>>>>>>>>>> hosts are configured correctly.
>>>>>>>>>>> 
>>>>>>>>>>> How do I correct the problem?
>>>>>>>>>>> 
>>>>>>>>>>> You can see the log from this link:
>>>>>>>>>>> 
>>>>>>>>>>> https://lat-
>>>>>>>>>>> 
>>>>>>>>>>> 
>> umanitoba.westgrid.ca/toolkit/admin/logs/ls_registration_daemon.log
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Lixin.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> =======================
>>>>>>>>>>> Lixin Liu
>>>>>>>>>>> IT Services
>>>>>>>>>>> Simon Fraser University
>>>>>>>>>> 
>>>>>>>>>> <ls_registration_daemon.log>
>>>>>>> <ls_registration_daemon.log>
>>> 
>>> <ls_registration_daemon.log>