perfsonar development work

[Jochen Reinwand <>]

Hi all,

I don't think that the EGEE guidelines are "wrong". They are not even 
unusable 
for us! But I have the feeling that these key/value pairs tend to grow to a 
standardisation horror. Parsing is easy! I'm quite sure Fausto is thinking 
about analysing the syslog via Perl. Parsing key/value pairs in Perl is like 
writting "Hello world!" in C. Eh, OK it's even easier...
But I fear that we will get into a lot of different key/value pairs that will 
be very hard to interpret (the necessary second step after parsing). I'm 
forseeing log lines like these:

my-service-error=12345 xyz-stack=pop \
  time=3-quarters-to-4-hours-after-local-midnight

Using a fixed format will perhaps prevent such a misuse. I see something like 
a decision tree:

system;Cannot create pid file

perfSONAR;success.ls.remove;Lookup info registered with key \
  http://stout.pc.cis.udel.edu:8080/services/RRDMA has been removed!

hades;database;No data found

Here "system" means related to perfSONAR independent errors comming from 
typical system interaction (e.g. daemon creating a pid file). With 
"perfSONAR" prefix typical perfSONAR/NM-WG status messages are logged. The 
idea here: Write to syslog what the user sees in the response. The last 
example is a "background system" related error. Most perfSONAR services use 
some tools or databases in the background. It is most likely usefull to log 
errors in this subsystems this way.

Once again: This can also be done using key/value pairs, but I fear to much 
freedom here...

greetings,
Jochen

On Wednesday 23 April 2008 20:26, Martin Swany wrote:
> Hi all,
>
> I think that the overhead of the best practices format is justified by
> the gains that Brian mentioned.  Also note that the log format is
> analogous to the NM-WG format in that it has a hierarchical
> event names (eventType).  I think there are some obvious wins
> that come from having the same namespaces used in the log
> messages and in the result codes.
>
> best,
> martin
>
> On Apr 23, 2008, at 12:45 PM, Fausto Vetter wrote:
> > Hi,
> >
> > Thats a good point. It needs to have a grammar otherwise it becomes
> > hard to parse and get some useful information from it. Putting in a
> > database is a step forward to go in some time, thats why we are
> > gonna setup a syslog server.
> >
> > Cheers,
> > Fausto
> >
> > Brian Tierney wrote:
> >> Luchesar:
> >>
> >> One of the main advantages of a 'best practice'-like format is that
> >> if you think there might be a time you'd want to put these logs
> >> into a database, then the logs must have some grammar, and not be
> >> totally freeform.
> >>
> >> Im not sure there is an immediate use case for putting perfSONAR
> >> logs into a database, but it might be good to make sure that the
> >> logs all follow the same grammar in case such a use case arises in
> >> the future.
> >>
> >>
> >> One possible use case I can think of is the following.
> >>
> >> Imagine that every perfSONAR service generated a syslog message
> >> logging each client connection. If the syslog messages where
> >> collected and stored in a database, it would be very each to make
> >> queries such as 'tell me the 10 top clients of perfSONAR services
> >> last week'.
> >>
> >> Clearly a log message that looked like something like this:
> >>
> >> event=org.perfSONAR.MA.connect client=A.B.C.D request=RetrieveData
> >>
> >> would be much easier to parse and insert into a database that one
> >> that might look like this:
> >>
> >> 'received request for data from client x.y.w.z'
> >>
> >> Something to consider.
> >>
> >> Cheers.
> >>
> >> Luchesar V. ILIEV wrote:
> >>> Hi Brian, Jochen, all,
> >>>
> >>> First of all, Brian, thank you very much for providing a pointer
> >>> to this very interesting document. It'll certainly be helpful in
> >>> our efforts within perfSONAR.
> >>>
> >>> At the same time, I have to agree with Jochen on most of his
> >>> points. Indeed, one of the reasons to use syslog is exactly the
> >>> fact that it already does provide standard approach to some of the
> >>> issues.
> >>>
> >>> The "key/value" pairing also does not appeal very much to me
> >>> (although I'm sure it is chosen for very good reasons in the
> >>> context of the Grid Troubleshooting project.) On the other hand,
> >>> the fixed positioning of the data is something that will depend
> >>> very much on what information is there going to be.
> >>>
> >>> * * *
> >>>
> >>> For that latter reason, I'd like to kindly ask once more the
> >>> developers of the pS services to provide me with at least a few
> >>> representative examples of messages that could be generated during
> >>> the operation of their service. Otherwise I'll have to base my
> >>> proposal on purely theoretical considerations -- perhaps not the
> >>> best way to go.
> >>>
> >>> * * *
> >>>
> >>> Brian, Jochen, once again many thanks for your feedback.
> >>>
> >>> Cheers,
> >>> Luchesar
> >>>
> >>> Jochen Reinwand wrote:
> >>>> Hi Brian, hi all,
> >>>>
> >>>> I think this Best Practice is perhaps too complex for us. Things
> >>>> like timestamp and level are already part of the syslog
> >>>> functionality anyway.
> >>>> For perfSONAR we already have error codes, that can be used more
> >>>> or less directly also for syslogging. I would really recommend
> >>>> this! This way the syslog receives the same messages and codes as
> >>>> the user does via the perfSONAR response.
> >>>> There are some message that do not necessarily issue a perfSONAR
> >>>> error response. I think these are all typical UNIX system error
> >>>> like "File not found", "Execution failed" and so on. These should
> >>>> be send to syslog in a very simple form to distinguish them from
> >>>> the perfSONAR errors.
> >>>> I'm not sure whether a key value pair style string is really
> >>>> handy. Most tools don't use such a format for syslogging. A
> >>>> string with fixed positions with fixed meanings is normally
> >>>> easier to handle and makes it easier to enforces a common
> >>>> discipline.
> >>>>
> >>>> regards,
> >>>> Jochen
> >>>>
> >>>> On Sunday 20 April 2008 17:40, Brian Tierney wrote:
> >>>>> Hi all:
> >>>>>
> >>>>> I am working on a Grid Troubleshooting project where for logging
> >>>>> we
> >>>>> agreed on a syslog format where everything is 'name=value' pairs.
> >>>>> We felt that this was the best compromise of easy to parse and
> >>>>> easy to
> >>>>> read.
> >>>>>
> >>>>> We also said that each log line should have a required
> >>>>> 'event=eventname', where eventname should be similar to a java
> >>>>> class name.
> >>>>>
> >>>>> I recommend that perfSONAR logs use this format as well.
> >>>>>
> >>>>> For more information.
> >>>>>
> >>>>> http://www.cedps.net/index.php/LoggingBestPractices
> >>>>>
> >>>>> This format is the same as the EGEE "Middleware Security Audit
> >>>>> Logging
> >>>>> Guidelines": http://edms.cern.ch/document/793208
> >>>>>
> >>>>> Cheers.
> >>>>>
> >>>>> Luchesar V. ILIEV wrote:
> >>>>>> Hi guys,
> >>>>>>
> >>>>>> This mail is addressed to all pS service developers. As you
> >>>>>> know, last
> >>>>>> week in Zagreb we discussed the issues of monitoring the
> >>>>>> deployed pS
> >>>>>> services to ensure they are operating as intended, and really
> >>>>>> serving
> >>>>>> the end-users' requests up to the users' satisfaction.
> >>>>>>
> >>>>>> As a next step in this process (having used so far ICMP, HTTP
> >>>>>> GET, and
> >>>>>> perfSONAR EchoRequest/EchoResponse messages,) in order to
> >>>>>> achieve more
> >>>>>> in-depth monitoring, which could, in turn, provide a more true-
> >>>>>> to-life
> >>>>>> view on the functioning of the services, we've decided in
> >>>>>> Zagreb to
> >>>>>> utilize the well-known and widely-used mechanism of sysloging.
> >>>>>>
> >>>>>> I've the task to prepare a proposal for a common format of the
> >>>>>> syslog
> >>>>>> messages, or (depending on your feedback) at least some
> >>>>>> guidelines,
> >>>>>> which would ensure having consistency amongst the different
> >>>>>> service's
> >>>>>> messages' general structure.
> >>>>>>
> >>>>>> Therefore, I'd like to ask you to provide me either with a full
> >>>>>> list of
> >>>>>> the possible messages that the service you're working on could
> >>>>>> eventually write to the log, or provide me with information on
> >>>>>> the
> >>>>>> structure of the (different types of these) messages.
> >>>>>>
> >>>>>> It could be helpful to have also some indication about the
> >>>>>> severity of
> >>>>>> each message (from your point of view, of course): e.g., debug,
> >>>>>> information, warning, error, critical failure, etc. As decided in
> >>>>>> Zagreb, you have to synchronize this with the Service Desk --
> >>>>>> if you
> >>>>>> could do this beforehand, it would be great. If not, I would be
> >>>>>> OK with
> >>>>>> your own (from service developer's point of view) classification.
> >>>>>>
> >>>>>> I'll appreciate if you could send me your feedback before the
> >>>>>> end of
> >>>>>> next business week: that is, 25th of April inclusive.
> >>>>>>
> >>>>>> If there's something unclear in my e-mail and specifically
> >>>>>> about my
> >>>>>> request, please do contact me.
> >>>>>>
> >>>>>> Many thanks in advance,
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Luchesar

-- 
Jochen Reinwand, DFN-Labor
Friedrich-Alexander-Universität Erlangen-Nürnberg
Regionales RechenZentrum Erlangen (RRZE)
Martensstraße 1, 91058 Erlangen, Germany
Tel. +49 9131 85-28689, -28800, Fax +49 9131 302941

www.win-labor.dfn.de