Skip to Content.
Sympa Menu

perfsonar-dev - Re: [pS-dev] Add geant2-java-rrdma to psUI

Subject: perfsonar development work

List archive

Re: [pS-dev] Add geant2-java-rrdma to psUI


Chronological Thread 
  • From: Jason Zurawski <>
  • To: Nina Jeliazkova <>
  • Cc: "Jeff W. Boote" <>, Szymon Trocha <>, "" <>,
  • Subject: Re: [pS-dev] Add geant2-java-rrdma to psUI
  • Date: Fri, 06 Mar 2009 08:57:59 -0500
  • Organization: Internet2
Nina;

Thank you for the detailed reply, comments inline:

Jeff,

Jeff W. Boote написа:
On Feb 25, 2009, at 9:00 AM, Nina Jeliazkova wrote:
The performance problems I was refering was observed when trying to
query gLS infrastructure for specific IP adresses, required by the
traceroute, not retrieving everything.
Hi Nina,

If you have any data on this performance problem, we would very much
like to see it ASAP.

We are gearing up for a new release - hopefully by the end of the
month, and have another week before our code-freeze. So, if you can
give us information on this, we may be able to address the issue
before our next release.

Thanks,
jeff

Here is a concrete use case, which illustrates some performance issues
(org.perfsonar.perfsonarui.test.ls.TraceRouteTest#testNewLookup ).

1) A request to gLS through gLS API (org.perfsonar.perfsonarui.ls.Lookup class) for the following four IP
addresses:

150.254.185.146 (PIONIER)
192.5.40.187
212.191.224.74
194.141.0.10 (BREN)

Please note 4 separate requests are sent in parallel, rather than one
with the four IPs; otherwise, the responses can not be distinguished.

2) PerfsonarUI receives an answer after 60250ms(!!! - note this is much
better than December 2008 tests where same request took up to 5 minutes)


60 seconds after querying the gLS and several hLS instances seems reasonable - particularly when searching for data sets that contain multiple results as yours does above. We will be profiling performance of the hLS and gLS servers after our next release to identify bottlenecks, but my guess is that distributed searching and pulling the information from remote locations could be to blame.


that these IP addresses are served by the following MAs: 150.254.185.146
Indiana Gigapop SNMP
MA,http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/IUsnmpMA,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
>> Internet2 SNMP
MA,http://packrat.internet2.edu:2008/perfSONAR_PS/services/snmp/SC08,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
>> Internet2 Network SNMP
MA,http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/snmpMA,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
192.5.40.187
>> Indiana Gigapop SNMP
MA,http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/IUsnmpMA,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
>> Internet2 SNMP
MA,http://packrat.internet2.edu:2008/perfSONAR_PS/services/snmp/SC08,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
>> Internet2 Network SNMP
MA,http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/snmpMA,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0

212.191.224.74
>> Internet2 SNMP
MA,http://packrat.internet2.edu:2008/perfSONAR_PS/services/snmp/SC08,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0

194.141.0.10
>> Indiana Gigapop SNMP
MA,http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/IUsnmpMA,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
>> Internet2 SNMP
MA,http://packrat.internet2.edu:2008/perfSONAR_PS/services/snmp/SC08,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0

In addition to the long response time (e.g. no user would be ever
pleased to wait for 60 seconds to get an answer to a simple DNS query),


This isn't exactly a simple DNS query, we are not looking up a straightforward key/value pair - there is a little extra work involved:

- parse the request, perform operations in a native database
- formulate and transmit sometimes large responses
- involves many services

While I sympathize that 'no user would be ever pleased to wait for 60 seconds', the fact that this is non-trivial will always be there. We can attempt to improve performance in target areas, but never to the level of a dig or nslookup operation. This is compounded when asking for arbitrary and complex results. I believe this is the first area where managing expectations is required; first for developers and second for users.


there is also an issue with 194.141.0.10, which belongs to the Bulgarian
NREN and is served by a MA, which is not mentioned in gLS’ answer.
Instead of this, completely unrelated MAs, which don’t actually have any
info on 194.141.0.10 are returned. We have also received a hint that
this is a side effect of the gLS summarization algorithm and we should
check with separate queries to each MA contained in the initial gLS
response, whether it has any info on the particular IP address indeed.


This was what we talked about in December. I can give an explanation again though. Currently the summarization (see here: http://anonsvn.internet2.edu/svn/nmwg/trunk/nmwg/doc/dLS/gLS/phase_1_color.html#implementation_perl_summarization) is based on a design that errors on the side of caution, and makes the summarized data sets larger than they may need to be.

The basic idea is best seen by taking a look at the tree diagram; I only make a rough guess of what the 'tightest' boundaries are for a given set of inputs. In general we will overestimate what is available e.g. claiming we have the entire class A subnet for 128 when in fact we are not even close to having that amount. We implemented the discovery this way to simplify the task of trying to distribute the summarized datasets: the discovery answer may return multiple hLS instances that could be the source of the data. The burden of searching will fall on client tools to ask each subsequent hLS to find specific instances. This assumption is spelled out on the design document.

A student has already been selected to alter this initial algorithm to give more accurate results. For now it is unfortunate that a specific hLS may claim to be the home of all 128.0.0.0/8 data, but the subsequent searching a small number of hLS instances with an additional message that should be run in parallel is a fair tradeoff.


So basically, the end result is that we have to wait almost a minute to
get (possibly) wrong info from gLS, which has to be checked with
subsequent queries to other services later on, which will take
additional time to complete. This doesn’t look as a very optimal design,
without even considering implementation’s performance.


I take issue with your statement that 'wrong' data is being returned, while it may not be as exact as expected the returned results are used for guidance, e.g. 'what you want may be in these locations'. We were very clear in the design documents that this is a multi-step process that will require a series of queries to find the exact source of information. Searching multiple hLS instances will never be eliminated, even when the summarization is altered to be more complete.

If you still feel that we have sacrificed optimality in our design to shortcut other issues, then we would like to hear some of your ideas on re-designing portions of this architecture. As a designer of the primary GUI for perfSONAR we would be very appreciative of your thoughts when it comes to user expectations and efficiency related to client applications since we did not get much feedback when we started this project over a year ago.


I’ll fill a bug report in bugzilla as well.


Please use the perfSONAR-PS issue tracker for hLS/gLS issues:

http://code.google.com/p/perfsonar-ps/issues/list

Bugzilla can be used for the MDM Lookup Service or the MDM gLS API

I would also appreciate if you could submit the exact XML messages that are being generated from pS-UI/gLS API to better understand what is being sent at each stage, and why it is being sent.

Thanks;

-jason


Hope this helps,
Nina

The entire log:
- Creating LSClient.
STATUS_MESSAGE=.
STATUS_MESSAGE=Discovering ...
STATUS_MESSAGE=.
STATUS_MESSAGE=Discovering ...
STATUS_MESSAGE=.
STATUS_MESSAGE=Discovering ...
STATUS_MESSAGE=.
STATUS_MESSAGE=Discovering ...
- Demoting Lookup
service('http://ls.perfsonar.pionier.net.pl:9990/perfSONAR_PS/services/gLS')
org.perfsonar.lsclient.LsQueryException: no success (error.ls.xmldb)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.checkEventType(JaxbTools.java:159)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.checkEventType(JaxbTools.java:124)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.readMetadatas(JaxbTools.java:97)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.unMarshallPerfsonarMessage(JaxbTools.java:65)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl.firstStep(DiscoveryProcessImpl.java:92)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl$1.next(DiscoveryProcessImpl.java:60)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl$1.next(DiscoveryProcessImpl.java:65)
at
org.perfsonar.lsclient.helpers.AbstractLevel1.discover(AbstractLevel1.java:40)
at
org.perfsonar.lsclient.helpers.AbstractLsClient.discover(AbstractLsClient.java:56)
at
org.perfsonar.perfsonarui.ls.LookupProcess$LookupTask.run(LookupProcess.java:74)
at java.lang.Thread.run(Thread.java:595)
- Demoting Lookup
service('http://ls.perfsonar.pionier.net.pl:9990/perfSONAR_PS/services/gLS')
org.perfsonar.lsclient.LsQueryException: no success (error.ls.xmldb)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.checkEventType(JaxbTools.java:159)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.checkEventType(JaxbTools.java:124)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.readMetadatas(JaxbTools.java:97)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.unMarshallPerfsonarMessage(JaxbTools.java:65)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl.firstStep(DiscoveryProcessImpl.java:92)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl$1.next(DiscoveryProcessImpl.java:60)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl$1.next(DiscoveryProcessImpl.java:65)
at
org.perfsonar.lsclient.helpers.AbstractLevel1.discover(AbstractLevel1.java:40)
at
org.perfsonar.lsclient.helpers.AbstractLsClient.discover(AbstractLsClient.java:56)
at
org.perfsonar.perfsonarui.ls.LookupProcess$LookupTask.run(LookupProcess.java:74)
at java.lang.Thread.run(Thread.java:595)
- Demoting Lookup
service('http://ls.perfsonar.pionier.net.pl:9990/perfSONAR_PS/services/gLS')
org.perfsonar.lsclient.LsQueryException: no success (error.ls.xmldb)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.checkEventType(JaxbTools.java:159)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.checkEventType(JaxbTools.java:124)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.readMetadatas(JaxbTools.java:97)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.unMarshallPerfsonarMessage(JaxbTools.java:65)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl.firstStep(DiscoveryProcessImpl.java:92)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl$1.next(DiscoveryProcessImpl.java:60)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl$1.next(DiscoveryProcessImpl.java:65)
at
org.perfsonar.lsclient.helpers.AbstractLevel1.discover(AbstractLevel1.java:40)
at
org.perfsonar.lsclient.helpers.AbstractLsClient.discover(AbstractLsClient.java:56)
at
org.perfsonar.perfsonarui.ls.LookupProcess$LookupTask.run(LookupProcess.java:74)
at java.lang.Thread.run(Thread.java:595)
- Demoting Lookup
service('http://ls.perfsonar.pionier.net.pl:9990/perfSONAR_PS/services/gLS')
org.perfsonar.lsclient.LsQueryException: no success (error.ls.xmldb)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.checkEventType(JaxbTools.java:159)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.checkEventType(JaxbTools.java:124)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.readMetadatas(JaxbTools.java:97)
at
org.perfsonar.ri.lsclient.level1.jaxb.JaxbTools.unMarshallPerfsonarMessage(JaxbTools.java:65)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl.firstStep(DiscoveryProcessImpl.java:92)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl$1.next(DiscoveryProcessImpl.java:60)
at
org.perfsonar.ri.lsclient.level1.DiscoveryProcessImpl$1.next(DiscoveryProcessImpl.java:65)
at
org.perfsonar.lsclient.helpers.AbstractLevel1.discover(AbstractLevel1.java:40)
at
org.perfsonar.lsclient.helpers.AbstractLsClient.discover(AbstractLsClient.java:56)
at
org.perfsonar.perfsonarui.ls.LookupProcess$LookupTask.run(LookupProcess.java:74)
at java.lang.Thread.run(Thread.java:595)
STATUS_MESSAGE=Found services for [Indiana Gigapop SNMP MA]
194.141.0.10 N/A N/A - Found services for [Indiana Gigapop SNMP MA] 194.141.0.10 N/A N/A , Lookup service http://ndb1.internet2.edu:9996/perfSONAR_PS/services/hLS
STATUS_MESSAGE=Found services for [Indiana Gigapop SNMP MA,
Internet2 SNMP MA] 194.141.0.10 N/A N/A ,,
- Found services for [Indiana Gigapop SNMP MA, Internet2 SNMP MA]
194.141.0.10 N/A N/A ,, Lookup service http://ndb1.internet2.edu:9996/perfSONAR_PS/services/hLS
STATUS_PERCENT=100.0
onFinish
STATUS_MESSAGE=Completed requests 3true
STATUS_MESSAGE=Found services for [Internet2 SNMP MA] 212.191.224.74
N/A N/A ( From Thu Jan 01 02:00:00 EET 1970 To Thu Jan 01 02:00:00 EET 1970 ),
- Found services for [Internet2 SNMP MA] 212.191.224.74 N/A N/A ( From Thu Jan 01 02:00:00 EET 1970 To Thu Jan 01 02:00:00 EET 1970
), Lookup service http://ndb1.internet2.edu:9996/perfSONAR_PS/services/hLS
STATUS_PERCENT=100.0
onFinish
STATUS_MESSAGE=Completed requests 2true
STATUS_MESSAGE=Found services for [Indiana Gigapop SNMP MA]
192.5.40.187 N/A N/A ,
- Found services for [Indiana Gigapop SNMP MA] 192.5.40.187 N/A N/A Lookup service http://ndb1.internet2.edu:9996/perfSONAR_PS/services/hLS
STATUS_MESSAGE=Found services for [Indiana Gigapop SNMP MA,
Internet2 SNMP MA] 192.5.40.187 N/A N/A - Found services for [Indiana Gigapop SNMP MA, Internet2 SNMP MA]
192.5.40.187 N/A N/A ,, Lookup service http://ndb1.internet2.edu:9996/perfSONAR_PS/services/hLS
STATUS_MESSAGE=Found services for [Indiana Gigapop SNMP MA,
Internet2 SNMP MA, Internet2 Network SNMP MA] 192.5.40.187 N/A N/A - Found services for [Indiana Gigapop SNMP MA, Internet2 SNMP MA,
Internet2 Network SNMP MA] 192.5.40.187 N/A N/A ,,, Lookup service http://ndb1.internet2.edu:9996/perfSONAR_PS/services/hLS
STATUS_PERCENT=100.0
onFinish
STATUS_MESSAGE=Completed requests 1true
STATUS_MESSAGE=Found services for [Indiana Gigapop SNMP MA]
150.254.185.146 N/A N/A - Found services for [Indiana Gigapop SNMP MA] 150.254.185.146 N/A N/A , Lookup service http://ndb1.internet2.edu:9996/perfSONAR_PS/services/hLS
STATUS_MESSAGE=Found services for [Indiana Gigapop SNMP MA,
Internet2 SNMP MA] 150.254.185.146 N/A N/A - Found services for [Indiana Gigapop SNMP MA, Internet2 SNMP MA]
150.254.185.146 N/A N/A ( From Thu Jan 01 02:00:00 EET 1970 To Thu Jan 01 02:00:00 EET 1970
),, Lookup service http://ndb1.internet2.edu:9996/perfSONAR_PS/services/hLS
STATUS_MESSAGE=Found services for [Indiana Gigapop SNMP MA,
Internet2 SNMP MA, Internet2 Network SNMP MA] 150.254.185.146 N/A N/A - Found services for [Indiana Gigapop SNMP MA, Internet2 SNMP MA,
Internet2 Network SNMP MA] 150.254.185.146 N/A N/A ,,, Lookup service http://ndb1.internet2.edu:9996/perfSONAR_PS/services/hLS
STATUS_PERCENT=100.0
onFinish
STATUS_MESSAGEElapsed 60250ms
=Completed requests 0true
[Indiana Gigapop SNMP MA, Internet2 SNMP MA, Internet2 Network
SNMP MA] 150.254.185.146 N/A N/A >> Indiana Gigapop SNMP
MA,http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/IUsnmpMA,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
>> Internet2 SNMP
MA,http://packrat.internet2.edu:2008/perfSONAR_PS/services/snmp/SC08,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
>> Internet2 Network SNMP
MA,http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/snmpMA,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
[Indiana Gigapop SNMP MA, Internet2 SNMP MA, Internet2 Network
SNMP MA] 192.5.40.187 N/A N/A >> Indiana Gigapop SNMP
MA,http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/IUsnmpMA,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
>> Internet2 SNMP
MA,http://packrat.internet2.edu:2008/perfSONAR_PS/services/snmp/SC08,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
>> Internet2 Network SNMP
MA,http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/snmpMA,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
[Internet2 SNMP MA] 212.191.224.74 N/A N/A
>> Internet2 SNMP
MA,http://packrat.internet2.edu:2008/perfSONAR_PS/services/snmp/SC08,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
[Indiana Gigapop SNMP MA, Internet2 SNMP MA] 194.141.0.10 N/A N/A
>> Indiana Gigapop SNMP
MA,http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/IUsnmpMA,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0
>> Internet2 SNMP
MA,http://packrat.internet2.edu:2008/perfSONAR_PS/services/snmp/SC08,http://schemas.perfsonar.net/2.0,,http://ggf.org/ns/nmwg/characteristic/utilization/2.0

Archive powered by MHonArc 2.6.16.

Top of Page