perfSONAR User Q&A and Other Discussion

["Lixin Liu" <>]

Hi Jason,

It took very much time, more than 2 days, but all of these
nodes are showing up in the global services list. So I guess
there is nothing wrong, may just take long time to show up.

Thank you very much.

Lixin.

> -----Original Message-----
> From: Jason Zurawski 
> [mailto:]
> Sent: October-01-13 6:00 AM
> To: Lixin Liu
> Cc: 
> ;
>  Performance Node Users
> Subject: Re: [perf-node-users] [perfsonar-user] LSRegistrationDaemon
> duplicate checksum lookup/host/ error
>
> Hi Lixin;
>
> In speaking with the maintainers, they note that what you are seeing is
> natural and consistent with the behavior of removing the local database, 
> and
> registration lapses on the server.  It can take several hours for things 
> to
> recover, and that is the case for your hosts as well.  In looking at the
> list again, several are now showing up:
>
> > bdw-usask.westgrid.ca
> >     http://ndb1.internet2.edu:8090/lookup/records?host-name=206.12.26.19
> >
> > bdw-sfu.westgrid.ca
> >     http://antg.es.net:8090/lookup/records?host-name=bdw-sfu.westgrid.ca
> >
> > bdw-ubc.westgrid.ca
> >     http://antg.es.net:8090/lookup/records?host-name=206.12.24.189
> >
> > lat-uvic.westgrid.ca
> >     http://antg.es.net:8090/lookup/records?host-name=lat-uvic.westgrid.ca
> >
> > bdw-ucalgary.westgrid.ca
> >     http://ndb1.internet2.edu:8090/lookup/records?host-name=bdw-
> ucalgary.westgrid.ca
> >
> > lat-ucalgary.westgrid.ca
> >     http://ndb1.internet2.edu:8090/lookup/records?host-name=lat-
> ucalgary.westgrid.ca
> >
> > bdw-uvic.westgrid.ca
> >     *not found*
> >
> > lat-sfu.westgrid.ca
> >     *not found*
> >
> > lat-ubc.westgrid.ca
> >     *not found*
>
>
> At this stage its a waiting exercise, so please be patient and the hosts
> will show up when the old registrations expire, and the new data is able 
> to
> synchronize.
>
> Thanks;
>
> -jason
>
> On Sep 30, 2013, at 5:11 PM, Lixin Liu 
> <>
>  wrote:
>
> > Hi Jason,
> >
> > Yes, all of them are 3.3.1. This is from bdw-ucalgary.westgrid.ca:
> >
> > # ls -l /opt/SimpleLS/bootstrap/etc/service_url
> > -rw-r----- 1 perfsonar perfsonar 472 Sep 30 14:57
> > /opt/SimpleLS/bootstrap/etc/service_url
> >
> > # cat /opt/SimpleLS/bootstrap/etc/service_url
> > ---
> > hosts:
> >  -
> >    latency: 34.27
> >    locator: http://antg.es.net:8090/lookup/records
> >    priority: 100
> >    status: alive
> >  -
> >    latency: 100.18
> >    locator: http://ps4.es.net:9095/lookup/records
> >    priority: 99
> >    status: alive
> >  -
> >    latency: 92.82
> >    locator: http://ndb1.internet2.edu:8090/lookup/records
> >    priority: 100
> >    status: alive
> >  -
> >    latency: 147.44
> >    locator: http://ps1.jp.apan.net:8090/lookup/records
> >    priority: 100
> >    status: alive
> >
> > This host is in Calgary, Mountain time zone.
> >
> > Thanks,
> >
> > Lixin.
> >
> >> -----Original Message-----
> >> From: Jason Zurawski 
> >> [mailto:]
> >> Sent: September-30-13 2:04 PM
> >> To: Lixin Liu
> >> Cc: 
> >> ;
> >>  Performance Node Users
> >> Subject: Re: [perf-node-users] [perfsonar-user] LSRegistrationDaemon
> >> duplicate checksum lookup/host/ error
> >>
> >> Hi Lixin;
> >>
> >> 2 Questions:
> >>
> >> 1) Are these all 3.3 or 3.3.1 hosts?
> >>
> >> 2) What is the contents of this file for each of the hosts:
> >> /opt/SimpleLS/bootstrap/etc/service_url
> >>
> >> Thanks;
> >>
> >> -jason
> >>
> >> On Sep 30, 2013, at 5:00 PM, "Lixin Liu" 
> >> <>
> >>  wrote:
> >>
> >>> Hi Jason,
> >>>
> >>> Looking at the ls_registration_daemon.log on number of our hosts,
> >>> it appears we have some difficulties to connect to your directory
> >>> servers. I see all these hosts report "400 Bad Request" errors
> >>> every hour.
> >>>
> >>> Thanks,
> >>>
> >>> Lixin.
> >>>
> >>> 2013/09/30 14:06:00 (15279) INFO> Interface.pm:127
> >>> perfSONAR_PS::LSRegistrationDaemon::Interface::build_checksum - 
> >>> Checksum
> >> is
> >>> ca4JrKF9a0bW+9fMVcgcSQ
> >>> 2013/09/30 14:06:00 (15279) INFO> Base.pm:178
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'p2p1' is 
> >>> up,
> >>> registering
> >>> 2013/09/30 14:06:00 (15279) ERROR> Base.pm:226
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem 
> >>> registering
> >>> service. Will retry full registration next time: 403 Forbidden
> >>> 2013/09/30 14:06:00 (15279) WARN> daemon.pl:251 main::__ANON__ - 
> >>> Warned:
> >> Use
> >>> of uninitialized value in sort at
> >>>
> >>
> /opt/perfsonar_ps/ls_registration_daemon/bin/../lib/perfSONAR_PS/LSRegistrat
> >> ionDaemon/Host.pm
> >>> line 334.
> >>> 2013/09/30 14:06:00 (15279) WARN> daemon.pl:251 main::__ANON__ - 
> >>> Warned:
> >> Use
> >>> of uninitialized value in join or string at
> >>>
> >>
> /opt/perfsonar_ps/ls_registration_daemon/bin/../lib/perfSONAR_PS/LSRegistrat
> >> ionDaemon/Host.pm
> >>> line 334.
> >>> 2013/09/30 14:06:00 (15279) INFO> Host.pm:309
> >>> perfSONAR_PS::LSRegistrationDaemon::Host::build_checksum - Checksum is
> >>> 0TtxBBUueOQpqJhpT/fdBA
> >>> 2013/09/30 14:06:00 (15279) INFO> Base.pm:178
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record
> >>> 'bdw-ucalgary.westgrid.ca' is up, registering
> >>> 2013/09/30 14:06:00 (15279) ERROR> Base.pm:226
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem 
> >>> registering
> >>> service. Will retry full registration next time: 400 Bad Request
> >>> 2013/09/30 14:06:00 (15279) INFO> Service.pm:223
> >>> perfSONAR_PS::LSRegistrationDaemon::Service::build_checksum - Checksum
> >>> is
> >>> d7gIByEaY/6Ic3BkjzOyJg
> >>> 2013/09/30 14:06:00 (15279) INFO> Base.pm:178
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'University
> >>> of
> >>> Calgary Ping Responder' is up, registering
> >>> 2013/09/30 14:06:01 (15279) ERROR> Base.pm:226
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem 
> >>> registering
> >>> service. Will retry full registration next time: 400 Bad Request
> >>> 2013/09/30 14:06:01 (15279) INFO> Service.pm:223
> >>> perfSONAR_PS::LSRegistrationDaemon::Service::build_checksum - Checksum
> >>> is
> >>> vu6Wo2l/GUk7CbRvFFovow
> >>> 2013/09/30 14:06:01 (15279) INFO> Base.pm:178
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'University
> >>> of
> >>> Calgary Traceroute Responder' is up, registering
> >>> 2013/09/30 14:06:01 (15279) ERROR> Base.pm:226
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem 
> >>> registering
> >>> service. Will retry full registration next time: 400 Bad Request
> >>> 2013/09/30 14:06:01 (15279) INFO> Base.pm:194
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'University
> >>> of
> >>> Calgary OWAMP Server' is down
> >>> 2013/09/30 14:06:01 (15279) INFO> Service.pm:223
> >>> perfSONAR_PS::LSRegistrationDaemon::Service::build_checksum - Checksum
> >>> is
> >>> TBtP9qqBGm8nKPruWIySXQ
> >>> 2013/09/30 14:06:01 (15279) INFO> Base.pm:178
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'University
> >>> of
> >>> Calgary BWCTL Server' is up, registering
> >>> 2013/09/30 14:06:01 (15279) ERROR> Base.pm:226
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem 
> >>> registering
> >>> service. Will retry full registration next time: 400 Bad Request
> >>> 2013/09/30 14:06:01 (15279) INFO> Service.pm:223
> >>> perfSONAR_PS::LSRegistrationDaemon::Service::build_checksum - Checksum
> >>> is
> >>> JZMxKg3PkoEbBRTKtu8uog
> >>> 2013/09/30 14:06:01 (15279) INFO> Base.pm:178
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'University
> >>> of
> >>> Calgary NDT Server' is up, registering
> >>> 2013/09/30 14:06:01 (15279) ERROR> Base.pm:226
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem 
> >>> registering
> >>> service. Will retry full registration next time: 400 Bad Request
> >>> 2013/09/30 14:06:01 (15279) INFO> Service.pm:223
> >>> perfSONAR_PS::LSRegistrationDaemon::Service::build_checksum - Checksum
> >>> is
> >>> XxvLx+/jXY4OUM7up8PJLA
> >>> 2013/09/30 14:06:01 (15279) INFO> Base.pm:178
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'University
> >>> of
> >>> Calgary NPAD Server' is up, registering
> >>> 2013/09/30 14:06:01 (15279) ERROR> Base.pm:226
> >>> perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem 
> >>> registering
> >>> service. Will retry full registration next time: 400 Bad Request
> >>>
> >>>> -----Original Message-----
> >>>> From: Jason Zurawski 
> >>>> [mailto:]
> >>>> Sent: September-30-13 9:33 AM
> >>>> To: Lixin Liu
> >>>> Cc: 
> >>>> ;
> >>>>  Performance Node Users
> >>>> Subject: Re: [perf-node-users] [perfsonar-user] LSRegistrationDaemon
> >>>> duplicate checksum lookup/host/ error
> >>>>
> >>>> Lixin;
> >>>>
> >>>> There are several directory servers that manage registrations, not 
> >>>> just
> >>>> the
> >>>> one.  I am able to see some of the hosts you mention below, note that
> >>>> sometimes a host registers with an IP address if there was a lack of
> >>>> DNS
> >>>> name to start:
> >>>>
> >>>>> bdw-usask.westgrid.ca
> >>>>  http://ndb1.internet2.edu:8090/lookup/records?host-name=206.12.26.19
> >>>>
> >>>>> bdw-sfu.westgrid.ca
> >>>>  http://antg.es.net:8090/lookup/records?host-name=bdw-sfu.westgrid.ca
> >>>>
> >>>>> bdw-ucalgary.westgrid.ca
> >>>>  *cannot find*
> >>>>
> >>>>> bdw-ubc.westgrid.ca
> >>>>  http://antg.es.net:8090/lookup/records?host-name=206.12.24.189
> >>>>
> >>>>> bdw-uvic.westgrid.ca
> >>>>  *cannot find*
> >>>>
> >>>>> lat-sfu.westgrid.ca
> >>>>  I see this: http://ps4.es.net:9095/lookup/records?host-name=ps-
> >>>> latency.sfu.westgrid.ca - did the host's name change recently?
> >>>>
> >>>>> lat-ucalgary.westgrid.ca
> >>>>  *cannot find*
> >>>>
> >>>>> lat-ubc.westgrid.ca
> >>>>  *cannot find*
> >>>>
> >>>>> lat-uvic.westgrid.ca
> >>>>
> >>>> 
> >>>> http://antg.es.net:8090/lookup/records?host-name=lat-uvic.westgrid.ca
> >>>>
> >>>> You can try to remove the database on the hosts as you did before if
> >>>> the
> >>>> hosts had a recent DNS or IP address change.  You can also check to 
> >>>> be
> >>>> sure
> >>>> there are no firewalls blocking registration.
> >>>>
> >>>> Thanks;
> >>>>
> >>>> -jason
> >>>>
> >>>> On Sep 30, 2013, at 12:07 PM, Lixin Liu 
> >>>> <>
> >>>>  wrote:
> >>>>
> >>>>> Hi Jason,
> >>>>>
> >>>>> Thank you very much. Indeed these two are on the list now.
> >>>>>
> >>>>> However, there are a number of hosts previous in the list
> >>>>> are disappeared from the list over the weekend. They are:
> >>>>>
> >>>>> bdw-usask.westgrid.ca
> >>>>> bdw-sfu.westgrid.ca
> >>>>> bdw-ucalgary.westgrid.ca
> >>>>> bdw-ubc.westgrid.ca
> >>>>> bdw-uvic.westgrid.ca
> >>>>> lat-sfu.westgrid.ca
> >>>>> lat-ucalgary.westgrid.ca
> >>>>> lat-ubc.westgrid.ca
> >>>>> lat-uvic.westgrid.ca
> >>>>>
> >>>>> Your URL:
> >>>>>
> >>>>> http://ndb1.internet2.edu:8090/lookup/records?host-name=<hostname>
> >>>>>
> >>>>> display only [].
> >>>>>
> >>>>> Should I try to restart ls_registration_daemon on them? Do I need
> >>>>> to remove the database?
> >>>>>
> >>>>> Thanks again.
> >>>>>
> >>>>> Lixin.
> >>>>>
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Jason Zurawski 
> >>>>>> [mailto:]
> >>>>>> Sent: September-30-13 6:35 AM
> >>>>>> To: Lixin Liu
> >>>>>> Cc: 
> >>>>>> ;
> >>>>>>  Performance Node Users
> >>>>>> Subject: Re: [perf-node-users] [perfsonar-user] 
> >>>>>> LSRegistrationDaemon
> >>>>>> duplicate checksum lookup/host/ error
> >>>>>>
> >>>>>> Hi Lixin;
> >>>>>>
> >>>>>> We were able to track down an operational issue on our end with the
> >>>>>> caching
> >>>>>> service - as I previously noted your two servers are showing up, so
> >>>>>> this
> >>>>>> is
> >>>>>> not a problem on your end:
> >>>>>>
> >>>>>> http://ndb1.internet2.edu:8090/lookup/records?host-name=bdw-
> >>>>>> umanitoba.westgrid.ca
> >>>>>> http://ndb1.internet2.edu:8090/lookup/records?host-name=lat-
> >>>>>> umanitoba.westgrid.ca
> >>>>>>
> >>>>>> They were not being placed into the cache though because, the
> >>>>>> Internet2
> >>>> LS
> >>>>>> had a typo in the URL.  If you do the following:
> >>>>>>
> >>>>>> sudo /etc/init.d/ls_cache_daemon restart
> >>>>>>
> >>>>>> It will re-download the cache file and your hosts should appear in
> >>>>>> the
> >>>>>> Global Listing.
> >>>>>>
> >>>>>> Thanks;
> >>>>>>
> >>>>>> -jason
> >>>>>>
> >>>>>> On Sep 29, 2013, at 12:07 PM, Lixin Liu 
> >>>>>> <>
> >>>>>>  wrote:
> >>>>>>
> >>>>>>> Hi Jason,
> >>>>>>>
> >>>>>>> I still do not see lat-umanitoba.westgrid.ca and
> >>>>>>> bdw-umanitoba.westgrid.ca
> >>>>>>> in
> >>>>>>> the Global Performance Services, and I can't see them in the
> >>>>>>> ComputeCanada
> >>>>>>> community. I even do you suggested (stop registration, remove db 
> >>>>>>> and
> >>>>>>> start
> >>>>>>> registration) on bdw-umanitoba yesterday and still unable to see.
> >>>>>>>
> >>>>>>> Is there anything else I can check?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Lixin.
> >>>>>>>
> >>>>>>> On 2013-09-27 10:58 AM, "Jason Zurawski" 
> >>>>>>> <>
> >>>>>>>  wrote:
> >>>>>>>
> >>>>>>>> Hi Lixin;
> >>>>>>>>
> >>>>>>>> Thanks for the clarification, I understand your specific question
> >>>>>>>> now.
> >>>>>>>> There are a couple of delays, let me try to explain them:
> >>>>>>>>
> >>>>>>>> - Delay between when the LS daemon runs on the local server, and
> >>>>>>>> talks
> >>>>>>>> to the remote server
> >>>>>>>> - Delay between when the remote server talks to a caching service
> >>>>>>>> (an
> >>>>>>>> optimization we added - used to generate the GUI information for
> >>>>>>>> 'Global
> >>>>>>>> Performance Services') and creates a new file for everyone to
> >>>>>>>> download
> >>>>>>>> - Delay between when the local server downloads the latest copy 
> >>>>>>>> of
> >>>>>>>> the
> >>>>>>>> cached content to populate the GUI
> >>>>>>>>
> >>>>>>>> In general the answer is that it takes a couple of hours for this
> >>>>>>>> all
> >>>>>>>> to
> >>>>>>>> happen, as little as 2-3 if the timing is right, as many as 6-8 
> >>>>>>>> if
> >>>>>>>> it
> >>>>>>>> is
> >>>>>>>> wrong.  Since we know that your host is in the directory via that
> >>>>>>>> REST
> >>>>>>>> query that was shown below, there is nothing to worry about.  It
> >>>>>>>> will
> >>>>>>>> just take some time to get into the GUI displays, and there isn't
> >>>>>>>> much
> >>>>>>>> that can be done to speed that up.
> >>>>>>>>
> >>>>>>>> The SElinux issue shouldn't be the root cause, but I believe we
> >>>>>>>> either
> >>>>>>>> set this to permissive or disabled.
> >>>>>>>>
> >>>>>>>> Thanks;
> >>>>>>>>
> >>>>>>>> -jason
> >>>>>>>>
> >>>>>>>> On Sep 27, 2013, at 1:40 PM, "Lixin Liu" 
> >>>>>>>> <>
> >>>>>>>>  wrote:
> >>>>>>>>
> >>>>>>>>> Hi Jason,
> >>>>>>>>>
> >>>>>>>>> Sorry I should mention that the latency host is down right now.
> >>>>>>>>> Someone will take a look at the machine itself. Will let you
> >>>>>>>>> know when it comes up.
> >>>>>>>>>
> >>>>>>>>> So looks like the bandwidth host is registered, but how long I
> >>>>>>>>> need to wait to see its services listed in "Global Performance
> >>>>>>>>> Services"?
> >>>>>>>>>
> >>>>>>>>> I noticed these two sites has one thing in common: SELinux is
> >>>>>>>>> enabled. I disabled it, but not sure if that is root cause of
> >>>>>>>>> our service registration issue.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>>
> >>>>>>>>> Lixin.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Jason Zurawski 
> >>>>>>>>>> [mailto:]
> >>>>>>>>>> Sent: September-27-13 10:26 AM
> >>>>>>>>>> To: Lixin Liu
> >>>>>>>>>> Cc: 
> >>>>>>>>>> ;
> >>>>>>>>>>  Performance Node Users
> >>>>>>>>>> Subject: Re: [perf-node-users] Re: [perfsonar-user]
> >>>>>>>>>> LSRegistrationDaemon
> >>>>>>>>>> duplicate checksum lookup/host/ error
> >>>>>>>>>>
> >>>>>>>>>> Hi Lixin;
> >>>>>>>>>>
> >>>>>>>>>> Looking in one of the global servers. I do see your bdw host:
> >>>>>>>>>>
> >>>>>>>>>> http://ndb1.internet2.edu:8090/lookup/records?host-name=bdw-
> >>>>>>>>>> umanitoba.westgrid.ca
> >>>>>>>>>>
> >>>>>>>>>> Unfortunately you are correct in that the latency host has not
> >>>>>>>>>> registered,
> >>>>>>>>>> I
> >>>>>>>>>> don't see that anywhere.  Can you send the latest log message 
> >>>>>>>>>> for
> >>>>>>>>>> lat-
> >>>>>>>>>> umanitoba.westgrid.ca again?  Everything after the point where
> >>>>>>>>>> you
> >>>>>>>>>> did
> >>>>>>>>>> the
> >>>>>>>>>> removal of the db file and the restart.
> >>>>>>>>>>
> >>>>>>>>>> Thanks;
> >>>>>>>>>>
> >>>>>>>>>> -jason
> >>>>>>>>>>
> >>>>>>>>>> On Sep 27, 2013, at 1:02 PM, "Lixin Liu" 
> >>>>>>>>>> <>
> >>>>>>>>>>  wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Jason,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for your information.
> >>>>>>>>>>>
> >>>>>>>>>>> I rebooted two hosts (in University of Saskatoon) and they are
> >>>>>>>>>>> now
> >>>>>>>>>>> showing up in the list of Global Services.
> >>>>>>>>>>>
> >>>>>>>>>>> I still have issue with two other hosts (in University of
> >>>>>>>>>>> Manitoba).
> >>>>>>>>>>> I followed your suggestion. It has been more than two hours, 
> >>>>>>>>>>> but
> >>>>>>>>>>> I
> >>>>>>>>>>> still do not see this host in the list. However, there was a
> >>>>>>>>>>> network
> >>>>>>>>>>> (BGP) problem early today that may affect the registration.
> >>>>>>>>>>>
> >>>>>>>>>>> Here is the log file (hostname bdw-umanitoba.westgrid.ca).
> >>>>>>>>>>>
> >>>>>>>>>>> The NTP server on the host you mentioned was changed by the
> >>>>>>>>>>> local
> >>>>>>>>>>> admin
> >>>>>>>>>>> to use local NTP servers, but there are only two servers in 
> >>>>>>>>>>> the
> >>>>>>>>>>> config.
> >>>>>>>>>>> I added two more and should be fine now.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> Lixin.
> >>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Jason Zurawski 
> >>>>>>>>>>>> [mailto:]
> >>>>>>>>>>>> Sent: September-27-13 6:19 AM
> >>>>>>>>>>>> To: Lixin Liu
> >>>>>>>>>>>> Cc: 
> >>>>>>>>>>>> ;
> >>>>>>>>>>>>  Performance Node Users
> >>>>>>>>>>>> Subject: Re: [perf-node-users] Re: [perfsonar-user]
> >>>>>>>>>>>> LSRegistrationDaemon
> >>>>>>>>>>>> duplicate checksum lookup/host/ error
> >>>>>>>>>>>>
> >>>>>>>>>>>> And naturally I meant "rm -f
> >>>>>>>>>>>> /var/lib/perfsonar/ls_registration_daemon/lsKey.db" for the
> >>>>>>>>>>>> cache
> >>>>>>>>>>>> file Š
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks;
> >>>>>>>>>>>>
> >>>>>>>>>>>> -jason
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sep 27, 2013, at 9:17 AM, Jason Zurawski 
> >>>>>>>>>>>> <>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Lixin;
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In looking at the logs, we don't see anything out of the
> >>>>>>>>>>>>> ordinary,
> >>>>>>>>>>>>> the
> >>>>>>>>>>>> 'duplicate' message you see is actually just unfortunate
> >>>>>>>>>>>> wording
> >>>>>>>>>>>> on
> >>>>>>>>>>>> our
> >>>>>>>>>>>> part
> >>>>>>>>>>>> - it means 'renew' the registration instead of making a new
> >>>>>>>>>>>> one.
> >>>>>>>>>>>> Since
> >>>>>>>>>>>> you
> >>>>>>>>>>>> noted you changed DNS/IP, it may be an issue of a stake 
> >>>>>>>>>>>> cache.
> >>>>>>>>>>>> Try
> >>>>>>>>>>>> the
> >>>>>>>>>>>> following steps:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> sudo /etc/init.d/ls_registration_daemon stop
> >>>>>>>>>>>>>> /var/lib/perfsonar/ls_registration_daemon/lsKey.db
> >>>>>>>>>>>>>> sudo /etc/init.d/ls_registration_daemon start
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This will stop the service, delete the local db, and then
> >>>>>>>>>>>>> start
> >>>>>>>>>>>>> it
> >>>>>>>>>>>>> all
> >>>>>>>>>>>> over.  After a couple of hours the information should show 
> >>>>>>>>>>>> up,
> >>>>>>>>>>>> or
> >>>>>>>>>>>> at
> >>>>>>>>>>>> a
> >>>>>>>>>>>> minimum the logs will let us know if there is something else
> >>>>>>>>>>>> locally
> >>>>>>>>>>>> bad
> >>>>>>>>>>>> that is going on.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> As an aside, it appears that your NTP is not configured, 
> >>>>>>>>>>>>> some
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>> tools
> >>>>>>>>>>>> may not work until that is fixed.  You may want to run the 
> >>>>>>>>>>>> NTP
> >>>>>>>>>>>> configuration
> >>>>>>>>>>>> script again.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks;
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -jason
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Sep 26, 2013, at 3:43 PM, "Lixin Liu" 
> >>>>>>>>>>>>> <>
> >>>>>>>>>>>>>  wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sorry I didn¹t realize that the link is password protected
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>> I
> >>>>>>>>>>>>>> already logged as admin.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Here is the log.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Lixin.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>>> From: 
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> [
> >>>>>>>>>>>>>>> ]
> >>>>>>>>>>>>>>>  On Behalf Of Lixin Liu
> >>>>>>>>>>>>>>> Sent: September-26-13 12:36 PM
> >>>>>>>>>>>>>>> To: 
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Subject: [perfsonar-user] LSRegistrationDaemon duplicate
> >>>>>>>>>>>>>>> checksum
> >>>>>>>>>>>>>>> lookup/host/ error
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On a number of our perfSONAR hosts, we get
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> perfSONAR_PS::LSRegistrationDaemon::Base::find_duplicate -
> >>>>>>>>>>>>>>> Found
> >>>>>>>>>>>> duplicate
> >>>>>>>>>>>>>>> checksum lookup/host
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> error and hosts do not show up in "Global Service and Data
> >>>>>>>>>>>>>>> View".
> >>>>>>>>>>>>>>> These cases may be related to the IP and DNS changes. But 
> >>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>> hosts are configured correctly.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> How do I correct the problem?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> You can see the log from this link:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://lat-
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>> umanitoba.westgrid.ca/toolkit/admin/logs/ls_registration_daemon.log
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Lixin.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> =======================
> >>>>>>>>>>>>>>> Lixin Liu
> >>>>>>>>>>>>>>> IT Services
> >>>>>>>>>>>>>>> Simon Fraser University
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> <ls_registration_daemon.log>
> >>>>>>>>>>> <ls_registration_daemon.log>
> >>>>>>>
> >>>>>>> <ls_registration_daemon.log>