perfsonar development work

[Joe Metzger <>]

Nina,
How much historical data does perfSONARUI fetch for each interface,  
and have you
thought about scaling this according to the number of interfaces  
found in the
metadata?

IE, if you find 1000 interfaces in the metadata, can you scale way  
back the
number of datapoints you are fetching for each one by going to longer  
time
frame averages. (ie fetch two 1-hour data points instead of 2 hours  
worth of
5 minute data points.) This will still allow you to have some info about
each interface, it should respond a whole lot quicker, and it won't blow
the heap and crash everytime somebody tries to use it with the ESnet MA.

--Joe

On Dec 4, 2007, at 9:46 AM, Nicolas Simar wrote:

> Chris Welti wrote:
> > Hi Nicolas,
> > Reducing the SWITCH data set from over 300 interfaces down to 38  
> > like in GEANT,
>
> That's not really an option...
> If you were to need different hardware, we can still provide it to  
> you (from the Pilot Budget).
>
> > I get similar performance, around 1-2 seconds for the list of  
> > interfaces and about 5 seconds to get all the data from all  
> > interfaces. (btw, this also depends a little bit on the kind of  
> > time period that you select).
> > It seems that the MA does not scale well for many interfaces. It  
> > should be tested how many interfaces are still feasible.
> > Chris
>
> Time is an issue (Ilias working on a benchmarking scripts for the  
> RRD MA and Roman is investigating means of improving the  
> performance of the RRD MA). I am investigating with them when  
> they'll have the benchmark tests ready (I'll get back to you).
>
> But I am wondering if retrieving all the data from all the  
> interface, is it the way pSUi will be used by network operators/NOC  
> people (it is for sure the case for tools as CNM).
>
> I'd like to concentrate on few scenarios (for the January workshop)  
> that we would implement on pSUI that would help NOC people: e.g.  
> retrieving show commands from multiple router, utilisation/error/ 
> drops from a set of interfaces, etc. I'll get back to you next week  
> on this.
>
> Cheers,
> Nicolas
>
> > Nicolas Simar wrote:
> > > Chris Welti wrote:
> > >
> > > > Personally I currently don't see the "retrieve all" to make any  
> > > > sense,
> > > > as it does not seem to work properly anyway.
> > > > If it takes more than a minute to get the results, what's the  
> > > > point?
> > > > I even tried to use the perfsonarUI to do an retrieve all from the
> > > > GEANT MA, which does not have that many interfaces, and it just  
> > > > locks
> > > > up after a while, without any results.
> > > > Does that function work for anyone that is running a production MA?
> > >
> > > I can do retrieve all from
> > >
> > > 1) GARR, 114 interfaces, 58 secs to get the data from all the  
> > > interfaces
> > >
> > > 2) GARR, 114 interfaces, 9 sec to get the list of interfaces,  
> > > stopped
> > > after 2 minutes, didn't get a single result
> > >
> > > 3) GEANT2, 38 interfaces, 2 sec to get the list of interfaces, 6  
> > > secs to
> > > get the data from all the interfaces
> > >
> > > 4) Hungarnet, 141 interfaces  , 4 sec to get the list of  
> > > interfaces, 141
> > > interfaces, pSUI indicated completed but no data available.
> > >
> > > 5) GARR, 114 interfaces, 9 sec to get the list of interfaces,1:31  
> > > to get
> > > the data from the first 10 interfaces, 1:42 to get all of them,  
> > > Error
> > > code     error.rrdma.rrdjtool; clear data
> > >
> > > 6) GEANT2, 38 interfaces, 2 sec to get the list of interfaces, 4  
> > > secs to
> > > get the first data, 3 second to get all the data completed. Error  
> > > code
> > > error.ma.query; clear data
> > >
> > > 7) Hungarnet, 141 interfaces , 8 sec to get the list of  
> > > interfaces, no
> > > data received. Error code     error.ma.query
> > >
> > > Nicolas
> > >
> > >
> > > > Regards,
> > > > Chris
> > > >
> > > >
> > > > Jason Zurawski wrote:
> > > >
> > > >
> > > >
> > > > > > > I think the "retrieve all" is a good thing to have, but  
> > > > > > > particularly
> > > > > > > when testing large deployments (ESnet has 1000s of interfaces  
> > > > > > > for
> > > > > > > example) it would be good to have functionality to do the  
> > > > > > > metadata
> > > > > > > request to populate pS-UI, then selectively retrieve  
> > > > > > > interface info
> > > > > > > for
> > > > > > > data requests instead of automatically getting everythihng.  
> > > > > > > Working on
> > > > > > > a Mac mini at SC, it was impossible to run pS-UI due to the  
> > > > > > > volume of
> > > > > > > data generated by certain services (also caused repeated Heap
> > > > > > > Overflows).
> > > > > >
> > > > > > I agree for testing purposes it will be convenient to send obnly
> > > > > > metadata request.  I am thinking of two options
> > > > > > 1) add a new menu item (in addition to retrieve all) with
> > > > > > functionality to send only metadata request , therefore the  
> > > > > > table will
> > > > > > be filled in only with interface metadata and no utilization
> > > > > > statistics, and no data will be displayed at barchart /radar  
> > > > > > chart. In
> > > > > > this case, could you suggest an appropriate title for the new  
> > > > > > menu
> > > > > > item?
> > > > > > or
> > > > > > 2)add an option somewhere in order to switch the behaviour of
> > > > > > "Retrieve All" between current behaviour and sending only  
> > > > > > metadata
> > > > > > request.
> > > > > >
> > > > > > Which will be the preferred way?
> > > > >
> > > > > Option 1 makes the most sense to me, I would call this choice  
> > > > > "Populate"
> > > > > or something similar.  After this step there would need to be two
> > > > > options for data retrieval: the current "retrieve all", then  
> > > > > the ability
> > > > > to select interfaces from the table, and simply "retrieve  
> > > > > selected".
> > > > > Anyone else feel strongly one way or another about this?
> > > > >
> > > > > -jason
>
> --
> Nicolas
> ______________________________________________________________________
>
> Nicolas Simar
> Network Engineer
>
> DANTE - www.dante.net
>
> Tel - BE: +32 (0) 4 366 93 49
> Tel - UK: +44 (0)1223 371 300
> Mobile: +44 (0) 7740 176 883
>
> City House, 126-130 Hills Road
> Cambridge CB2 1PQ
> UK
> _____________________________________________________________________
>
>
>         
>