perfsonar development work

[Maciej Glowiak <>]

Hi Jeff, Joe,

Thanks for input for benchmarking.

Some of my concerns are:

1. Network performance between Service and LS and Client and LS does 
matter. We should test two cases: ideal one (all of them in the local 
network) and real one (for instance LS in PSNC and Client/Service in I2)

2. Wall-clock time is ok :) We may use "time" command for scripts or 
System.currentTimeInMillis() in java

3. We use XQuery language. I don't see any possibility to not allow the 
client to do his query, except parsing and "understanding" XQuery.
Another service may be a proxy that keeps some predefined queries -- we 
talked about it some time ago, but I'd prefer not to put such 
functionality into LS. Let's keep our services simple.

4. Defining queries. I agree the developers of client applications 
should give us a hint what type of query would be useful. But Vedrin is 
on holiday, so...
Basically Jeff's a,b,c are ok, but we should define what is the output 
of them. Asking for "services of specific type" we my get their IP 
addresses (accessPoints) or full lookup information. So that's very 
important to define output.

5. We have a couple of things to prepare:

   a - Prepare Lookup Service (if we want it to measure time it'd need
       some changes)

   b - Set up 2 LSes (b[1]-one in PSNC, it's almost ready,
       I must just check the version and turn the machine on;
       b[2]-the second one in I2/ESNet)

   c - Prepare data sets of 1, 10, 50, 100, 500, 1000 and 5000 lookup
       info/interfaces (I am not sure which option we'll be testing: one
       lookup info means full information about a service - for instance
       1 short metadata of MA  + 100 data with interfaces details). The
       data should be real (should contain for instance varoious IP
       addresses)

   d - discuss queries - what we want to ask about, and what should
       results look like (what output)

        >     a: All services of type X
        >     b: All interfaces with 'utilization' data available
        >     c: same as b, but make it more specific somehow

        These are good but "output" need to be specified

    e - prepare testing scripts (using existing client application)

    f - perform tests

           - local registration  (PSNC->PSNC or I2->I2)
                 - using Registration component for MA
                 - using SOAP Client and request XML file
                (It'd be valuable to compare these two approaches)

           - remote registration (PSNC->I2 or I2->PSNC)

           - local query (PSNC->PSNC or I2->I2)

           - remote query (PSNC->I2 or I2->PSNC)

    g - publish results (wiki or mailing list)


I can get a -if necessary, b1, d - prepare queries after discussion, e, 
and part of f/g (I don't know when exactly I will have my holidays, but 
I think if we prepare everything (a-e) tests will go easy.

Maciej




Jeff W. Boote wrote:
> Joe Metzger wrote:
> > Hi,
> > Here is a strawman to start the discussion on benchmarking the LS.
>
> Nice job Joe.
>
> My additions follow:
>
> > We need to benchmark the lookup service in order to make reasonable 
> > decisions about how to move forward. IE, how many networks can
> > register with a signal LS, what should the registration interval
> > be, and how will they perform under client query loads.
> >
> >
> >
> > I think the benchmarking should collect the data neccessary to
> > determine the function terms that define the following
> > performance criteria:
> >
> > The time to register an MA in a lookup service should be a function 
> > with 3 terms:
> >     1. A constant time to process a message.
> >     2. A variable time relative to the size of the message
> >     3. A variable time relative to the size of data already in the LS.
> >
> > The time to make a query is also a function with 3 terms:
> >     1. A constant time to process a message
> >     2. A time relative to the size of the database in the LS
> >     3. A time relative to the size of the query results.
>     4. A variable time relative to the complexity of the query.
>
> (i.e. We need to benchmark a few different types of queries.)
>
>  > I think the range of the parameter space tested should be large enough
> > so that we understand how the performance scales in reasonable scenarios:
> >     Registration Messages with interface counts:
> >         1, 10, 50, 100, 500, 1000 and 5000
> >     Queries that retrieve interface counts:
> >         1, 10, 50, 100, 500, 1000 and 5000
> >     LS Database sizes of 500 - 10,000 interfaces.
>
> Adding in different queries gives us several dimensions of tests to run. 
> I think this is more than adequate for the first go around.
>
> > At this point I think we can constrain our analysis to wall-clock time
> > and completely ignore additional components of performance such as the
> > network performance between the MA and the LS.  Eventually we may want
> > to characterize those impacts, as well as processor vs wall clock time
> > so we can understand how overlapping requests/queries will perform.
>
> Agreed. For the purposes of this test I think we should benchmark on 
> systems that are otherwise unloaded. We can deal with the more complex 
> issues later.
>
> > I am not familiar enough with the LS architecture to know if there will
> > be additional data specific aspects that significantly affect scaling. 
> > IE, will one query that retrieves 100 interfaces perform the same as
> > one that retrieves a different 100 interfaces?  Do we expect significant
> > variance based on the data?
>
> I think we need/will catch *some* of this by using different types of 
> queries.*
>
>
> So, does the following benchmarking pseudo-code capture the tests we 
> want to run?
>
> for dbsize (500 1000 5000 10000){
>     for regsize (1 10 50 100 500 1000 5000){
>         clear_db
>         load_db_fake_data(size=>dbsize)
>         time_register_data(size=>regsize)
>     }
>     clear_db
>     load_db_fake_data(size=>dbsize)
>     for qtype (a b c d){
>         for qresultsize (1 10 50 100 500 1000 5000){
>             time_query(type=>qtype,size=>qresultsize)
>         }
>     }
> }
>
> Not sure what qtype's a-d are yet. I envision things like:
>     a: All services of type X
>     b: All interfaces with 'utilization' data available
>     c: same as b, but make it more specific somehow
>
> Suggestions from GUI developers about what types of queries they intend 
> to make (what information they need) would be very helpful. I just have 
> not been thinking along these lines lately.
>
> jeff
>
>
> * One thing this brings to mind for me is to make sure clients are not 
> doing 'global' queries for all the information available. (zone 
> transfers in DNS speak) We may need to enhance the service with a 
> 'request denied' message that indicates the query is to broad in this 
> case. We need to figure out a way how to tell the client what they can 
> do to make the query more specific perhaps. I would prefer NOT to have 
> to deal with the 'batching' issue, but we might have to eventually.


--

 --------------------------------------------------------------------
| Maciej Glowiak        Network Research and Development            ||
| 

     Poznan Supercomputing and Networking Center ||
| (+48 61) 858 2024     http://monstera.man.poznan.pl/              ||
 ====================================================================