perfSONAR User Q&A and Other Discussion

["Garnizov, Ivan (RRZE)" <>]

Hi Roland,

 

Yes there was. There wasn’t one in the IT-554 ticket, but the system takes it from the dns, where the error was. It was solved yesterday.

 

Best regards,

Ivan

 

From: Karch, Roland (RRZE)
Sent: Thursday, November 13, 2014 9:22 AM
To: Aaron Brown; Garnizov, Ivan (RRZE)
Cc: Andrew Lake;
Subject: RE: [perfsonar-user] LSRegistrationDaemon::Base::refresh issue

 

Hi,

 

for the danger of stating something obvious / trivial, but isn’t there a typo in there?

2014/11/12 17:16:03 (31671) DEBUG> TCP_Service.pm:75 perfSONAR_PS::LSRegistrationDaemon::Services::TCP_Service::is_up - Connecting to BWCTL Measurement Point server: bwclt2.fra.de.geant.net:80

 

For reference:

$ nslookup bwclt2.fra.de.geant.net

Server:         213.133.100.100

Address:        213.133.100.100#53

 

** server can't find bwclt2.fra.de.geant.net: NXDOMAIN

 

With best wishes,

Roland

 

From: [] On Behalf Of Aaron Brown
Sent: Wednesday, November 12, 2014 5:30 PM
To: Garnizov, Ivan (RRZE)
Cc: Andrew Lake;
Subject: Re: [perfsonar-user] LSRegistrationDaemon::Base::refresh issue

 

Hey Ivan,

 

Looking at the log file, I’m seeing things like:

 

2014/11/12 17:16:03 (31671) DEBUG> TCP_Service.pm:75 perfSONAR_PS::LSRegistrationDaemon::Services::TCP_Service::is_up - Connecting to BWCTL Measurement Point server: bwclt2.fra.de.geant.net:80

 

For some reason, the ls registration daemon can’t connect to port 80 on that host. Are you able to manually connect to port 80 on that host from the same machine that the ls registration daemon is running on?

 

Cheers,

Aaron

 

On Nov 12, 2014, at 11:23 AM, Garnizov, Ivan (RRZE) <> wrote:

 

Hi Aaron,

 

Yes that brought more details, but I still can’t figure it out.

PERFSONAR="${BINDIR}/daemon.pl --verbose --config=${CONFFILE} --pidfile=${PIDFILE} --logger=${LOGGERFILE} --user=${USER} --group=${GROUP}"

 

 

There is a new problem that came with this setting. The service tries to start and stalls ….I made several attempts but it just reaches a specific point and stops responding. I can break the operation, but that does not help for starting the service.

 

I am applying the log:

 

Thanks,

Ivan

 

 

 

 

 

 

From: Aaron Brown [] 
Sent: Wednesday, November 12, 2014 3:42 PM
To: Garnizov, Ivan (RRZE)
Cc: Andrew Lake;
Subject: Re: [perfsonar-user] LSRegistrationDaemon::Base::refresh issue

 

Hey Ivan,

 

Try adding the “—verbose” flag to the “PERFSONAR=“ line in /etc/init.d/ls_registration_daemon, and then restart the daemon. That should hopefully actually enable debugging.

 

Cheers,

Aaron

 

On Nov 12, 2014, at 10:31 AM, Garnizov, Ivan (RRZE) <> wrote:

Hi Andrew,

 

Thanks for your prompt response. Generally I am pretty sure the settings are correct, since the moment we brought those servers up we ran tests with some remote PS stations (both OWMAP and BWCTL)

 

Could you please point me to a server on your side? And/or run a test towards this server.

 

Here are FRA details:

62.40.106.83 = bwctl2.fra.de.geant.net

62.40.106.81 = owamp2.fra.de.geant.net

 

Naming scheme is self-explanatory for the services. I am not sure who/what is the thing that tests these ports.

 

I am applying a snapshot of the graph on the collected measurement from that test.

I also made another dump including the 4823 port and then restarted the ls_registration_daemon and I did not see any attempts for external verification.

 

Best regards,

Ivan

 

 

 

 

-----Original Message-----
From: Andrew Lake [] 
Sent: Wednesday, November 12, 2014 3:06 PM
To: Garnizov, Ivan (RRZE)
Cc: Aaron Brown; 
Subject: Re: [perfsonar-user] LSRegistrationDaemon::Base::refresh issue

 

Hi Ivan,

 

That means it can't open a TCP connection to port 4823 of the address you specified for the BWCTL service in you ls_registartion_daemon.conf file. Are you listening on a different port? and are you sure BWCTL is listening on the address in the ls_registartion_daemon.conf file?

 

Thanks,

Andy

 

 

 

 

 

On Nov 12, 2014, at 10:00 AM, "Garnizov, Ivan (RRZE)" <> wrote:

 

> Hi Aaron, Sowmya,

> 

> The debug setting did not improve things by much (if not at all).

> There is no firewall issue and the net-dump file demonstrates that. Connectivity to all LS servers and communication with the GEANT LS.

> The error is the same.

> Can someone please explain what is the meaning behind?

> .....- Record 'Geant BWCTL Server' is down

> 

> Probably my conclusions below are wrong and there is no secondary discovery process, but then how do we maintain the LS databases and records?

> 

> Best regards,

> Ivan

> 

> 

> 

> 

> 

> 

> -----Original Message-----

> From: Aaron Brown []

> Sent: Wednesday, November 12, 2014 12:44 PM

> To: Garnizov, Ivan (RRZE)

> Cc: 

> Subject: Re: [perfsonar-user] LSRegistrationDaemon::Base::refresh issue

> 

> Hi Ivan,

> 

> Try enabling debug on the ls registration daemon by editing /opt/perfsonar_ps/ls_registration_daemon/etc/ls_registration_daemon-logger.conf, and changing the INFO line to DEBUG, and then restarting the daemon. My guess would be firewall issues, or similar, because the LS reg daemon tries to connect to the service to see if it's up. The responder stuff just checks if it can be ping'd whereas the rest of the services are checked with a TCP connect or similar (modulo the MPs, I've no clue what they're doing).

> 

> Cheers,

> Aaron

> 

> On Nov 12, 2014, at 4:41 AM, Garnizov, Ivan (RRZE) <> wrote:

> 

>> Hi group,

>> 

>> I have been struggling to understand what is the reason for not having some services on a toolkit deployment missing. Unluckily I have not resolved the issue, but also I see there is discrepancy in service status discovery. It appears there are 2 means of service status checks. Please refer to the applied image and excerpt in the logs below:

>> 

>> 2014/11/12 09:46:37 (64611) INFO> Base.pm:255 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'Geant Traceroute Responder' is up, refreshing registration

>> 2014/11/12 09:46:38 (64611) INFO> Base.pm:267 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'Geant OWAMP Server' is down

>> 2014/11/12 09:46:38 (64611) INFO> Base.pm:267 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'Geant BWCTL Server' is down

>> 2014/11/12 09:46:38 (64611) INFO> Base.pm:267 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'Geant NDT Server' is down

>> 2014/11/12 09:46:38 (64611) INFO> Base.pm:267 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'Geant NPAD Server' is down

>> 2014/11/12 09:46:38 (64611) INFO> Base.pm:267 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'Geant Measurement Archive' is down

>> 2014/11/12 09:46:38 (64611) INFO> Base.pm:267 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'Geant BWCTL Measurement Point' is down

>> 2014/11/12 09:46:38 (64611) INFO> Base.pm:267 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'Geant OWAMP Measurement Point' is down

>> 

>> Please help me find the source of discrepancy and how would I be able to fulfil the requirements of the ls_registration daemon!

>> 

>> As a feature request (should I put that as a ticket?):

>> It would be of great benefit to have both status states available on the local services page!

>> 

>> Best regards,

>> Ivan

>> 

>> 

>> 

>> 

>> 

>> 

>> 

>> 

>> From: Sowmya Balasubramanian []

>> Sent: Monday, November 10, 2014 6:57 PM

>> To: Garnizov, Ivan (RRZE)

>> Cc: 

>> Subject: Re: [perfsonar-user] "Globally Registered" issue

>> 

>> Hi Ivan,

>> 

>> Let me know if the problem is resolved after you bring the services up.

>> 

>> If not, we can delete the entries manually from the LS and then re-register them.

>> 

>> Thanks,

>> Sowmya

>> 

>> 

>> 

>> On Mon, Nov 10, 2014 at 9:51 AM, Garnizov, Ivan (RRZE) <> wrote:

>> Hi group,

>> 

>> 

>> 

>> In fact I see the problem in the logs. The LS registration service on its discovery process sees my services as down!

>> 

>> 

>> 

>> 2014/11/10 09:26:56 (43975) INFO> Base.pm:267 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'Geant BWCTL Server' is down

>> 

>> 

>> 

>> I have no idea as of how this can be and intend on making another check tomorrow, but I am sure the services of OWAMP and BWCTL are up and running, since I have checked them several times on other occasions.

>> 

>> 

>> 

>> Will continue tomorrow...

>> 

>> 

>> 

>> Best regards,

>> 

>> Ivan

>> 

>> 

>> 

>> -----Original Message-----

>> From:  [] On Behalf Of Garnizov, Ivan (RRZE)

>> 

>> Sent: Monday, November 10, 2014 11:46 AM

>> To: 

>> Cc: 

>> Subject: RE: [perfsonar-user] "Globally Registered" issue

>> 

>> 

>> Hi Sowmya, group,

>> 

>> 

>> 

>> Please consult me on an issue with LS registrations similar to the one in applied in the email. As we have replaced 2 of our servers in GEANT with the pS 3.4, we came across an issue with LS registration. One of our boxes is unable to register correctly. The problem continues for almost a week and it seems the old registration still breaks/prevents the new installation to appear properly in the LS (check the applied excerpt). The other server did not have former registration and I suspect that is the reason for being successful now.

>> 

>> 

>> 

>> Please consult me on how to expire/remove old registration and is the current case with conflicting registrations a problem.

>> 

>> Would it be possible to have a procedure to more quickly update/clean-up the LS service.

>> 

>> 

>> 

>> There is another problem I see there. After the installation process, we reviewed the network config and increased the MTU sizes. This update though again has not been indicated on the LS for the failing server. The other did receive an update. Probably this issue is an implication from the general registration issue we experience and the registrations for IP are just matching our previous config (as expected)?

>> 

>> 

>> 

>> I have applied both log and excerpt from LS registration.

>> 

>> 

>> 

>> Best regards,

>> 

>> Ivan

>> 

>> 

>> 

>> 

>> 

>> -----Original Message-----

>> 

>> From:  [] On Behalf Of Garnizov, Ivan (RRZE)

>> 

>> Sent: Tuesday, November 04, 2014 9:01 AM

>> 

>> To: 

>> 

>> Subject: RE: [perfsonar-user] "Globally Registered" issue

>> 

>> 

>> 

>> Hi group,

>> 

>> 

>> 

>> I can only add here that the DNS registration appears consistent:

>> 

>> 

>> 

>> [root@sophia-bg ~]# host perfsonar-m1.twaren.net perfsonar-m1.twaren.net has address 211.79.61.148 [root@sophia-bg ~]# host 211.79.61.148 148.61.79.211.in-addr.arpa domain name pointer perfsonar-m1.twaren.net.

>> 

>> [root@sophia-bg ~]#

>> 

>> 

>> 

>> I believe the problem hides here:

>> 

>> 

>> 

>> 2014/11/04 10:07:03 (2931) INFO> Base.pm:251 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'eth0' is up, registering

>> 

>> 2014/11/04 10:07:03 (2931) ERROR> Base.pm:304 perfSONAR_PS::LSRegistrationDaemon::Base::register - Problem registering service. Will retry full registration next time: 403 Forbidden

>> 

>> 2014/11/04 10:07:03 (2931) INFO> Base.pm:496 perfSONAR_PS::LSRegistrationDaemon::Base::build_duplicate_checksum - Duplicate checksum prior to md5 is person::Che-nan Yang

>> 

>> 2014/11/04 10:07:03 (2931) INFO> Base.pm:500 perfSONAR_PS::LSRegistrationDaemon::Base::build_duplicate_checksum - Duplicate checksum is f9tp79sxYhfwbyOYRv2yMQ

>> 

>> 2014/11/04 10:07:03 (2931) INFO> Base.pm:479 perfSONAR_PS::LSRegistrationDaemon::Base::build_checksum - Checksum prior to md5 is host::perfsonar-m1.twaren.net995 MB2992.834 MHz24CentOS6.5 (Final)Linux 2.6.32-431.29.2.el6.aufs.web100.x86$

>> 

>> 2014/11/04 10:07:03 (2931) INFO> Base.pm:483 perfSONAR_PS::LSRegistrationDaemon::Base::build_checksum - Checksum is tH0Tat7J6QLpqglqHErj9Q

>> 

>> 2014/11/04 10:07:03 (2931) INFO> Base.pm:251 perfSONAR_PS::LSRegistrationDaemon::Base::refresh - Record 'perfsonar-m1.twaren.net' is up, registering

>> 

>> 2014/11/04 10:07:03 (2931) INFO> Base.pm:496 perfSONAR_PS::LSRegistrationDaemon::Base::build_duplicate_checksum - Duplicate checksum prior to md5 is person::Che-nan Yang

>> 

>> 2014/11/04 10:07:03 (2931) INFO> Base.pm:500 perfSONAR_PS::LSRegistrationDaemon::Base::build_duplicate_checksum - Duplicate checksum is f9tp79sxYhfwbyOYRv2yMQ

>> 

>> 2014/11/04 10:07:03 (2931) INFO> Base.pm:301 perfSONAR_PS::LSRegistrationDaemon::Base::register - Next Refresh: 1415070423

>> 

>> 

>> 

>> Best regards,

>> 

>> Ivan

>> 

>> 

>> 

>> -----Original Message-----

>> 

>> From:  [] On Behalf Of Che-nan Yang

>> 

>> Sent: Tuesday, November 04, 2014 3:24 AM

>> 

>> To: 

>> 

>> Subject: [perfsonar-user] "Globally Registered" issue

>> 

>> 

>> 

>> Hi,

>> 

>> 

>> 

>> I'm having a problem with a 3.4 install of PerfSonar too.

>> 

>> I downloaded  the Full Install ISO and installed it.

>> 

>> The field "Globally Registered" with "No".

>> 

>> https://211.79.61.148/toolkit/gui/services/

>> 

>> 

>> 

>> Here are the contents of /var/log/httpd/error_log and /var/log/perfsonar/ls_registration_daemon.log files attached below.

>> 

>> 

>> 

>> Chenan

>> 

>> TWAREN

>> 

>> 

>> 

>> 

>> <fra-services.jpg>

> 

> <net dump.txt><ls_registration_daemon.log>

 

<performance test.jpg>

 

<ls_registration_daemon.log>