perfsonar development work

["Jeff W. Boote" <>]

Joe Metzger wrote:
> Hi,
> Here is a strawman to start the discussion on benchmarking the LS.

Nice job Joe.

My additions follow:

> We need to benchmark the lookup service in order to make reasonable 
> decisions about how to move forward. IE, how many networks can
> register with a signal LS, what should the registration interval
> be, and how will they perform under client query loads.
>
>
>
> I think the benchmarking should collect the data neccessary to
> determine the function terms that define the following
> performance criteria:
>
> The time to register an MA in a lookup service should be a function with 
> 3 terms:
>     1. A constant time to process a message.
>     2. A variable time relative to the size of the message
>     3. A variable time relative to the size of data already in the LS.
>
> The time to make a query is also a function with 3 terms:
>     1. A constant time to process a message
>     2. A time relative to the size of the database in the LS
>     3. A time relative to the size of the query results.
        4. A variable time relative to the complexity of the query.

(i.e. We need to benchmark a few different types of queries.)

 > I think the range of the parameter space tested should be large enough
> so that we understand how the performance scales in reasonable scenarios:
>     Registration Messages with interface counts:
>         1, 10, 50, 100, 500, 1000 and 5000
>     Queries that retrieve interface counts:
>         1, 10, 50, 100, 500, 1000 and 5000
>     LS Database sizes of 500 - 10,000 interfaces.

Adding in different queries gives us several dimensions of tests to run. I think 
this is more than adequate for the first go around.

> At this point I think we can constrain our analysis to wall-clock time
> and completely ignore additional components of performance such as the
> network performance between the MA and the LS.  Eventually we may want
> to characterize those impacts, as well as processor vs wall clock time
> so we can understand how overlapping requests/queries will perform.

Agreed. For the purposes of this test I think we should benchmark on systems 
that are otherwise unloaded. We can deal with the more complex issues later.

> I am not familiar enough with the LS architecture to know if there will
> be additional data specific aspects that significantly affect scaling. 
> IE, will one query that retrieves 100 interfaces perform the same as
> one that retrieves a different 100 interfaces?  Do we expect significant
> variance based on the data?

I think we need/will catch *some* of this by using different types of 
queries.*


So, does the following benchmarking pseudo-code capture the tests we want to 
run?

for dbsize (500 1000 5000 10000){
        for regsize (1 10 50 100 500 1000 5000){
                clear_db
                load_db_fake_data(size=>dbsize)
                time_register_data(size=>regsize)
        }
        clear_db
        load_db_fake_data(size=>dbsize)
        for qtype (a b c d){
                for qresultsize (1 10 50 100 500 1000 5000){
                        time_query(type=>qtype,size=>qresultsize)
                }
        }
}

Not sure what qtype's a-d are yet. I envision things like:
        a: All services of type X
        b: All interfaces with 'utilization' data available
        c: same as b, but make it more specific somehow

Suggestions from GUI developers about what types of queries they intend to make 
(what information they need) would be very helpful. I just have not been 
thinking along these lines lately.

jeff


* One thing this brings to mind for me is to make sure clients are not doing 
'global' queries for all the information available. (zone transfers in DNS 
speak) We may need to enhance the service with a 'request denied' message that 
indicates the query is to broad in this case. We need to figure out a way how to 
tell the client what they can do to make the query more specific perhaps. I 
would prefer NOT to have to deal with the 'batching' issue, but we might have to 
eventually.