perfsonar development work

["Jeff W. Boote" <>]

On Feb 25, 2009, at 9:00 AM, Nina Jeliazkova wrote:

> Jeff,
>
> PsUI doesn't try to retrieve all available servers from gLS (the query
> tab was just a demo put for illustration purposes by Michael).

Ok, that is the functionality that I believe is problematic to attempt  
to provide to every user. I'm glad to hear that is not the expected  
interaction model.

> The performance problems I was refering was observed when trying to
> query gLS infrastructure for specific IP adresses, required by the
> traceroute, not retrieving everything.

This sounds like a real problem that we need to get more information  
on so we can attempt to address it.

> Besides, relying on gls.hints URL  is a single point of failure, not
> much different that MA.conf.

There are significant differences. First, the gls.hints file is  
intended to be distributed from an anycast address or using something  
akin to an akamai distributed host setup. And, even in the current  
temporary setup the www.perfsonar.net webserver has a failover system,  
so it is not a single point of failure.

And, more fundamentally the difference between the gls.hints file and  
the MA.conf file can be described in two different dimensions:  
timeliness, and scale.

For timeliness, new MA's have been coming on line nearly daily.  
However, new gLS hosts are now only expected to change when we do  
hardware/software upgrades. The gls.hints file changed maybe 3-4 times  
over the past year, and I would expect it to become less frequent as  
the infrastructure becomes more mature. And for scale, we never expect  
to have more than 6-10 gLS hosts listed in that file. That is far less  
than the number of MAs that are out there.

As I have said in previous messages to the list. I believe the MA.conf  
file has a good use - I would think of it as a 'bookmarks' like  
functionality. The problem is simply that it is currently manually  
updated and therefore hard to keep current. But, imagine how useful it  
would be if you had a central host periodically searching for services  
with a specific key-word to find services related to a specific  
project (say LHC or MDM) and then have it update a project-specific  
MA.conf file with all the services it finds.

My only point in the previous message was that these kinds of bulk  
queries are not likely to be quick for any distributed information  
architecture, so we need to implement caching if that is a desired  
feature (and it seems to be).

jeff

> Regards,
> Nina
>
>
>
> Jeff W.Boote написа:
> > I think there is a fundamental problem with the way pS-UI is trying  
> > to
> > use to use the gLS infrastructure.
> >
> > When you bring up a web-browser, you don't say: tell me all web
> > servers in existence so I can pick from them. That is not a scalable
> > user interface, and it is not a scalable information architecture.
> >
> > The gLS infrastructure should not be expected to be able to give you
> > all current bwctl-mp servers in a blink of an eye, just like you  
> > don't
> > expect your web browser to be able to collect all news services that
> > fast.  For the web-browser, you make use of an intermediary that
> > caches the information, and I would suggest that is the most likely
> > best case scenario here. I recommend having a daemon run on a
> > centralized host that is continuously being updated based on what it
> > finds in the gLS to generate an 'current' MA.conf file. (This could  
> > be
> > implemented in a similar way to
> > http://www.perfsonar.net/activeServices.html.)
> >
> > If every client tries to do this functionality, it is not scalable.
> > gLS/hLS servers will quickly see things that look like DOS attacks  
> > and
> > will need to add hooks to protect themselves from excessive queries.
> > (Note - DNS servers do not allow zone transfers in general for the
> > same reason.) The distributed gLS/hLS database is designed to answer
> > more specific queries in the same model as DNS.
> >
> > jeff
> >
> > On Feb 25, 2009, at 7:36 AM, Szymon Trocha wrote:
> >
> > > Dear all,
> > >
> > > You raised an important point for discussion. The list of services  
> > > in
> > > psUI config file is already long and contains services which do not
> > > work or responding. It is also becoming more difficult to maintain.
> > > While at the same time a lot of work was put in creating LS
> > > capabilities.
> > >
> > > Michael Bischoff pisze:
> > > > "why is the MA.conf file still necessary?"
> > > [...]
> > > > Stripping MA.conf method from psui in one go would be to abrupt on
> > > > our users. While psui is
> > > > gLS capable, we would want to allow some period where the
> > > > functionally can be matured as well
> > > > as deployments increase.
> > > > MA.conf still serves use, albeit in a somewhat evolved way, it  
> > > > would
> > > > be useful to allow
> > > > users/(small)usergroups to have access to their personal list of
> > > > favourite/frequently used
> > > > services. This could even evolve in to a locally hosted cached list
> > > > of gls entries.
> > >
> > > Yes, and this is already a case where LHCOPN has its own dedicated
> > > instance of psUI, with a dedicated list of services, while not using
> > > LS capability yet. So this is at least one reason to leave it.
> > >
> > > > Dynamically obtained service addresses should for the most part
> > > > replace more 'statically'
> > >
> > > I fully support Jason's view here to make use of LS infrastructure
> > > and finally (possibly) replace the config file. Although I may still
> > > consider some period of time where configuration file exists but is
> > > static and e.g. restricted for MDM participants (just as en example
> > > of user group). I'm strongly against including test, non-productive
> > > or external services in the list.
> > >
> > > > MA.conf. The adoption rate here should be a direct indication as to
> > > > the quality of the
> > > > provided functionality. Additional stimulation above the added ease
> > > > of use that dynamically
> > > > obtaining service addresses brings, would only and frustrate the
> > > > user. Increasing awareness
> > > > is more the direction I would opt for. So don't say 'use this', but
> > > > rather 'did you know..'
> > > > and gather feedback - improve the functionally - which should
> > > > improve adoption rate.
> > >
> > > Currently within GN2 the adoption rate is not fast although the  
> > > tools
> > > are already there (don't know the reason why yet). I'm not even
> > > talking about LS but also MA instances. MDM is still small in size
> > > and not up-to-date. The only wide deployment is taking place for
> > > LHCOPN, our first real user. Increasing the adoption rate in terms  
> > > of
> > > service coverage and then LS cloud deployment is necessary to make
> > > the whole system useful and functional. This is one of my top
> > > priorities for GN3 especially that perfSONAR will be a service
> > > activity under GN3. This should also bring us feedback as Michael  
> > > wrote.
> > >
> > > And if there are any issues or doubts regarding hLS/gLS we should
> > > discuss them and solve. There was so much good work done until this
> > > time to leave it unused.
> > >
> > > Regards,
> > > --
> > > Szymon Trocha
> > >
> > > Poznan Supercomputing & Netw. Center ::: NETWORK OPERATION CENTER
> > > Tel. +48 618582022 ::: http://noc.man.poznan.pl
>
> --
> ---------------------------------
> Dr. Nina Nikolova-Jeliazkova
> Institute for Parallel Processing
> Bulgarian Academy of Sciences
> Acad. G. Bonchev St 25-A
> 1113 Sofia, Bulgaria
> Tel: +359 886 802011
> ICQ: 10705013
> www: http://ambit.acad.bg/nina
> ---------------------------------
> PGP Public Key
> http://cert.acad.bg/pgp-keys/keys/nina-nikolova-0xEEABA669.asc
>         8E99 8BAD D804 1A43 27B7  7F87 CF04 C7D1 EEAB A669
> ---------------------------------------------------------------