perfsonar development work

[Martin Swany <>]

Hi all,

I think that the overhead of the best practices format is justified by
the gains that Brian mentioned.  Also note that the log format is
analogous to the NM-WG format in that it has a hierarchical
event names (eventType).  I think there are some obvious wins
that come from having the same namespaces used in the log
messages and in the result codes.

best,
martin

On Apr 23, 2008, at 12:45 PM, Fausto Vetter wrote:

> Hi,
>
> Thats a good point. It needs to have a grammar otherwise it becomes  
> hard to parse and get some useful information from it. Putting in a  
> database is a step forward to go in some time, thats why we are  
> gonna setup a syslog server.
>
> Cheers,
> Fausto
>
> Brian Tierney wrote:
> > Luchesar:
> >
> > One of the main advantages of a 'best practice'-like format is that  
> > if you think there might be a time you'd want to put these logs  
> > into a database, then the logs must have some grammar, and not be  
> > totally freeform.
> >
> > Im not sure there is an immediate use case for putting perfSONAR  
> > logs into a database, but it might be good to make sure that the  
> > logs all follow the same grammar in case such a use case arises in  
> > the future.
> >
> >
> > One possible use case I can think of is the following.
> >
> > Imagine that every perfSONAR service generated a syslog message  
> > logging each client connection. If the syslog messages where  
> > collected and stored in a database, it would be very each to make  
> > queries such as 'tell me the 10 top clients of perfSONAR services  
> > last week'.
> >
> > Clearly a log message that looked like something like this:
> >
> > event=org.perfSONAR.MA.connect client=A.B.C.D request=RetrieveData
> >
> > would be much easier to parse and insert into a database that one  
> > that might look like this:
> >
> > 'received request for data from client x.y.w.z'
> >
> > Something to consider.
> >
> > Cheers.
> >
> >
> >
> >
> > Luchesar V. ILIEV wrote:
> > > Hi Brian, Jochen, all,
> > >
> > > First of all, Brian, thank you very much for providing a pointer  
> > > to this very interesting document. It'll certainly be helpful in  
> > > our efforts within perfSONAR.
> > >
> > > At the same time, I have to agree with Jochen on most of his  
> > > points. Indeed, one of the reasons to use syslog is exactly the  
> > > fact that it already does provide standard approach to some of the  
> > > issues.
> > >
> > > The "key/value" pairing also does not appeal very much to me  
> > > (although I'm sure it is chosen for very good reasons in the  
> > > context of the Grid Troubleshooting project.) On the other hand,  
> > > the fixed positioning of the data is something that will depend  
> > > very much on what information is there going to be.
> > >
> > > * * *
> > >
> > > For that latter reason, I'd like to kindly ask once more the  
> > > developers of the pS services to provide me with at least a few  
> > > representative examples of messages that could be generated during  
> > > the operation of their service. Otherwise I'll have to base my  
> > > proposal on purely theoretical considerations -- perhaps not the  
> > > best way to go.
> > >
> > > * * *
> > >
> > > Brian, Jochen, once again many thanks for your feedback.
> > >
> > > Cheers,
> > > Luchesar
> > >
> > >
> > > Jochen Reinwand wrote:
> > > > Hi Brian, hi all,
> > > >
> > > > I think this Best Practice is perhaps too complex for us. Things  
> > > > like timestamp and level are already part of the syslog  
> > > > functionality anyway.
> > > > For perfSONAR we already have error codes, that can be used more  
> > > > or less directly also for syslogging. I would really recommend  
> > > > this! This way the syslog receives the same messages and codes as  
> > > > the user does via the perfSONAR response.
> > > > There are some message that do not necessarily issue a perfSONAR  
> > > > error response. I think these are all typical UNIX system error  
> > > > like "File not found", "Execution failed" and so on. These should  
> > > > be send to syslog in a very simple form to distinguish them from  
> > > > the perfSONAR errors.
> > > > I'm not sure whether a key value pair style string is really  
> > > > handy. Most tools don't use such a format for syslogging. A  
> > > > string with fixed positions with fixed meanings is normally  
> > > > easier to handle and makes it easier to enforces a common  
> > > > discipline.
> > > >
> > > > regards,
> > > > Jochen
> > > >
> > > > On Sunday 20 April 2008 17:40, Brian Tierney wrote:
> > > > > Hi all:
> > > > >
> > > > > I am working on a Grid Troubleshooting project where for logging  
> > > > > we
> > > > > agreed on a syslog format where everything is 'name=value' pairs.
> > > > > We felt that this was the best compromise of easy to parse and  
> > > > > easy to
> > > > > read.
> > > > >
> > > > > We also said that each log line should have a required
> > > > > 'event=eventname', where eventname should be similar to a java  
> > > > > class name.
> > > > >
> > > > > I recommend that perfSONAR logs use this format as well.
> > > > >
> > > > > For more information.
> > > > >
> > > > > http://www.cedps.net/index.php/LoggingBestPractices
> > > > >
> > > > > This format is the same as the EGEE "Middleware Security Audit  
> > > > > Logging
> > > > > Guidelines": http://edms.cern.ch/document/793208
> > > > >
> > > > > Cheers.
> > > > >
> > > > > Luchesar V. ILIEV wrote:
> > > > > > Hi guys,
> > > > > >
> > > > > > This mail is addressed to all pS service developers. As you  
> > > > > > know, last
> > > > > > week in Zagreb we discussed the issues of monitoring the  
> > > > > > deployed pS
> > > > > > services to ensure they are operating as intended, and really  
> > > > > > serving
> > > > > > the end-users' requests up to the users' satisfaction.
> > > > > >
> > > > > > As a next step in this process (having used so far ICMP, HTTP  
> > > > > > GET, and
> > > > > > perfSONAR EchoRequest/EchoResponse messages,) in order to  
> > > > > > achieve more
> > > > > > in-depth monitoring, which could, in turn, provide a more true- 
> > > > > > to-life
> > > > > > view on the functioning of the services, we've decided in  
> > > > > > Zagreb to
> > > > > > utilize the well-known and widely-used mechanism of sysloging.
> > > > > >
> > > > > > I've the task to prepare a proposal for a common format of the  
> > > > > > syslog
> > > > > > messages, or (depending on your feedback) at least some  
> > > > > > guidelines,
> > > > > > which would ensure having consistency amongst the different  
> > > > > > service's
> > > > > > messages' general structure.
> > > > > >
> > > > > > Therefore, I'd like to ask you to provide me either with a full  
> > > > > > list of
> > > > > > the possible messages that the service you're working on could
> > > > > > eventually write to the log, or provide me with information on  
> > > > > > the
> > > > > > structure of the (different types of these) messages.
> > > > > >
> > > > > > It could be helpful to have also some indication about the  
> > > > > > severity of
> > > > > > each message (from your point of view, of course): e.g., debug,
> > > > > > information, warning, error, critical failure, etc. As decided in
> > > > > > Zagreb, you have to synchronize this with the Service Desk --  
> > > > > > if you
> > > > > > could do this beforehand, it would be great. If not, I would be  
> > > > > > OK with
> > > > > > your own (from service developer's point of view) classification.
> > > > > >
> > > > > > I'll appreciate if you could send me your feedback before the  
> > > > > > end of
> > > > > > next business week: that is, 25th of April inclusive.
> > > > > >
> > > > > > If there's something unclear in my e-mail and specifically  
> > > > > > about my
> > > > > > request, please do contact me.
> > > > > >
> > > > > > Many thanks in advance,
> > > > > >
> > > > > > Cheers,
> > > > > > Luchesar