perfsonar development work

[Jochen Reinwand <>]

Hi,

OK for me! I do not really like CSV also ;)
It's really an advantage to have readable, "normal" syslog entries. Not 
everybody wants to set up a centralised parsing after installing only a 
simple perfSONAR service.

Jochen

On Thursday 24 April 2008 10:49, Fausto Vetter wrote:
> Hi,
>
> I like the idea of having a format, but I don't like the idea of just
> CSV (comma, semi-comma, etc) format. It becomes easy to parse but not
> easy to read. It would be better for example:
>
> [pS-XML-LS] success.ls.remove 'Lookup info registered with key
> http://stout.pc.cis.udel.edu:8080/services/RRDMA has been removed!'
>
> where:
>
> [SERVICE_TYPE] RESULT.ACTION 'DESCRIPTION'
>
> Other fields like date & time and source are inherent to syslog format.
>
> What do you think?
>
> Fausto
>
> Jochen Reinwand wrote:
> > Hi all,
> >
> > I don't think that the EGEE guidelines are "wrong". They are not even
> > unusable for us! But I have the feeling that these key/value pairs tend
> > to grow to a standardisation horror. Parsing is easy! I'm quite sure
> > Fausto is thinking about analysing the syslog via Perl. Parsing key/value
> > pairs in Perl is like writting "Hello world!" in C. Eh, OK it's even
> > easier...
> > But I fear that we will get into a lot of different key/value pairs that
> > will be very hard to interpret (the necessary second step after parsing).
> > I'm forseeing log lines like these:
> >
> > my-service-error=12345 xyz-stack=pop \
> >   time=3-quarters-to-4-hours-after-local-midnight
> >
> > Using a fixed format will perhaps prevent such a misuse. I see something
> > like a decision tree:
> >
> > system;Cannot create pid file
> >
> > perfSONAR;success.ls.remove;Lookup info registered with key \
> >   http://stout.pc.cis.udel.edu:8080/services/RRDMA has been removed!
> >
> > hades;database;No data found
> >
> > Here "system" means related to perfSONAR independent errors comming from
> > typical system interaction (e.g. daemon creating a pid file). With
> > "perfSONAR" prefix typical perfSONAR/NM-WG status messages are logged.
> > The idea here: Write to syslog what the user sees in the response. The
> > last example is a "background system" related error. Most perfSONAR
> > services use some tools or databases in the background. It is most likely
> > usefull to log errors in this subsystems this way.
> >
> > Once again: This can also be done using key/value pairs, but I fear to
> > much freedom here...
> >
> > greetings,
> > Jochen
> >
> > On Wednesday 23 April 2008 20:26, Martin Swany wrote:
> >> Hi all,
> >>
> >> I think that the overhead of the best practices format is justified by
> >> the gains that Brian mentioned.  Also note that the log format is
> >> analogous to the NM-WG format in that it has a hierarchical
> >> event names (eventType).  I think there are some obvious wins
> >> that come from having the same namespaces used in the log
> >> messages and in the result codes.
> >>
> >> best,
> >> martin
> >>
> >> On Apr 23, 2008, at 12:45 PM, Fausto Vetter wrote:
> >>> Hi,
> >>>
> >>> Thats a good point. It needs to have a grammar otherwise it becomes
> >>> hard to parse and get some useful information from it. Putting in a
> >>> database is a step forward to go in some time, thats why we are
> >>> gonna setup a syslog server.
> >>>
> >>> Cheers,
> >>> Fausto
> >>>
> >>> Brian Tierney wrote:
> >>>> Luchesar:
> >>>>
> >>>> One of the main advantages of a 'best practice'-like format is that
> >>>> if you think there might be a time you'd want to put these logs
> >>>> into a database, then the logs must have some grammar, and not be
> >>>> totally freeform.
> >>>>
> >>>> Im not sure there is an immediate use case for putting perfSONAR
> >>>> logs into a database, but it might be good to make sure that the
> >>>> logs all follow the same grammar in case such a use case arises in
> >>>> the future.
> >>>>
> >>>>
> >>>> One possible use case I can think of is the following.
> >>>>
> >>>> Imagine that every perfSONAR service generated a syslog message
> >>>> logging each client connection. If the syslog messages where
> >>>> collected and stored in a database, it would be very each to make
> >>>> queries such as 'tell me the 10 top clients of perfSONAR services
> >>>> last week'.
> >>>>
> >>>> Clearly a log message that looked like something like this:
> >>>>
> >>>> event=org.perfSONAR.MA.connect client=A.B.C.D request=RetrieveData
> >>>>
> >>>> would be much easier to parse and insert into a database that one
> >>>> that might look like this:
> >>>>
> >>>> 'received request for data from client x.y.w.z'
> >>>>
> >>>> Something to consider.
> >>>>
> >>>> Cheers.
> >>>>
> >>>> Luchesar V. ILIEV wrote:
> >>>>> Hi Brian, Jochen, all,
> >>>>>
> >>>>> First of all, Brian, thank you very much for providing a pointer
> >>>>> to this very interesting document. It'll certainly be helpful in
> >>>>> our efforts within perfSONAR.
> >>>>>
> >>>>> At the same time, I have to agree with Jochen on most of his
> >>>>> points. Indeed, one of the reasons to use syslog is exactly the
> >>>>> fact that it already does provide standard approach to some of the
> >>>>> issues.
> >>>>>
> >>>>> The "key/value" pairing also does not appeal very much to me
> >>>>> (although I'm sure it is chosen for very good reasons in the
> >>>>> context of the Grid Troubleshooting project.) On the other hand,
> >>>>> the fixed positioning of the data is something that will depend
> >>>>> very much on what information is there going to be.
> >>>>>
> >>>>> * * *
> >>>>>
> >>>>> For that latter reason, I'd like to kindly ask once more the
> >>>>> developers of the pS services to provide me with at least a few
> >>>>> representative examples of messages that could be generated during
> >>>>> the operation of their service. Otherwise I'll have to base my
> >>>>> proposal on purely theoretical considerations -- perhaps not the
> >>>>> best way to go.
> >>>>>
> >>>>> * * *
> >>>>>
> >>>>> Brian, Jochen, once again many thanks for your feedback.
> >>>>>
> >>>>> Cheers,
> >>>>> Luchesar
> >>>>>
> >>>>> Jochen Reinwand wrote:
> >>>>>> Hi Brian, hi all,
> >>>>>>
> >>>>>> I think this Best Practice is perhaps too complex for us. Things
> >>>>>> like timestamp and level are already part of the syslog
> >>>>>> functionality anyway.
> >>>>>> For perfSONAR we already have error codes, that can be used more
> >>>>>> or less directly also for syslogging. I would really recommend
> >>>>>> this! This way the syslog receives the same messages and codes as
> >>>>>> the user does via the perfSONAR response.
> >>>>>> There are some message that do not necessarily issue a perfSONAR
> >>>>>> error response. I think these are all typical UNIX system error
> >>>>>> like "File not found", "Execution failed" and so on. These should
> >>>>>> be send to syslog in a very simple form to distinguish them from
> >>>>>> the perfSONAR errors.
> >>>>>> I'm not sure whether a key value pair style string is really
> >>>>>> handy. Most tools don't use such a format for syslogging. A
> >>>>>> string with fixed positions with fixed meanings is normally
> >>>>>> easier to handle and makes it easier to enforces a common
> >>>>>> discipline.
> >>>>>>
> >>>>>> regards,
> >>>>>> Jochen
> >>>>>>
> >>>>>> On Sunday 20 April 2008 17:40, Brian Tierney wrote:
> >>>>>>> Hi all:
> >>>>>>>
> >>>>>>> I am working on a Grid Troubleshooting project where for logging
> >>>>>>> we
> >>>>>>> agreed on a syslog format where everything is 'name=value' pairs.
> >>>>>>> We felt that this was the best compromise of easy to parse and
> >>>>>>> easy to
> >>>>>>> read.
> >>>>>>>
> >>>>>>> We also said that each log line should have a required
> >>>>>>> 'event=eventname', where eventname should be similar to a java
> >>>>>>> class name.
> >>>>>>>
> >>>>>>> I recommend that perfSONAR logs use this format as well.
> >>>>>>>
> >>>>>>> For more information.
> >>>>>>>
> >>>>>>> http://www.cedps.net/index.php/LoggingBestPractices
> >>>>>>>
> >>>>>>> This format is the same as the EGEE "Middleware Security Audit
> >>>>>>> Logging
> >>>>>>> Guidelines": http://edms.cern.ch/document/793208
> >>>>>>>
> >>>>>>> Cheers.
> >>>>>>>
> >>>>>>> Luchesar V. ILIEV wrote:
> >>>>>>>> Hi guys,
> >>>>>>>>
> >>>>>>>> This mail is addressed to all pS service developers. As you
> >>>>>>>> know, last
> >>>>>>>> week in Zagreb we discussed the issues of monitoring the
> >>>>>>>> deployed pS
> >>>>>>>> services to ensure they are operating as intended, and really
> >>>>>>>> serving
> >>>>>>>> the end-users' requests up to the users' satisfaction.
> >>>>>>>>
> >>>>>>>> As a next step in this process (having used so far ICMP, HTTP
> >>>>>>>> GET, and
> >>>>>>>> perfSONAR EchoRequest/EchoResponse messages,) in order to
> >>>>>>>> achieve more
> >>>>>>>> in-depth monitoring, which could, in turn, provide a more true-
> >>>>>>>> to-life
> >>>>>>>> view on the functioning of the services, we've decided in
> >>>>>>>> Zagreb to
> >>>>>>>> utilize the well-known and widely-used mechanism of sysloging.
> >>>>>>>>
> >>>>>>>> I've the task to prepare a proposal for a common format of the
> >>>>>>>> syslog
> >>>>>>>> messages, or (depending on your feedback) at least some
> >>>>>>>> guidelines,
> >>>>>>>> which would ensure having consistency amongst the different
> >>>>>>>> service's
> >>>>>>>> messages' general structure.
> >>>>>>>>
> >>>>>>>> Therefore, I'd like to ask you to provide me either with a full
> >>>>>>>> list of
> >>>>>>>> the possible messages that the service you're working on could
> >>>>>>>> eventually write to the log, or provide me with information on
> >>>>>>>> the
> >>>>>>>> structure of the (different types of these) messages.
> >>>>>>>>
> >>>>>>>> It could be helpful to have also some indication about the
> >>>>>>>> severity of
> >>>>>>>> each message (from your point of view, of course): e.g., debug,
> >>>>>>>> information, warning, error, critical failure, etc. As decided in
> >>>>>>>> Zagreb, you have to synchronize this with the Service Desk --
> >>>>>>>> if you
> >>>>>>>> could do this beforehand, it would be great. If not, I would be
> >>>>>>>> OK with
> >>>>>>>> your own (from service developer's point of view) classification.
> >>>>>>>>
> >>>>>>>> I'll appreciate if you could send me your feedback before the
> >>>>>>>> end of
> >>>>>>>> next business week: that is, 25th of April inclusive.
> >>>>>>>>
> >>>>>>>> If there's something unclear in my e-mail and specifically
> >>>>>>>> about my
> >>>>>>>> request, please do contact me.
> >>>>>>>>
> >>>>>>>> Many thanks in advance,
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Luchesar

-- 
Jochen Reinwand, DFN-Labor
Friedrich-Alexander-Universität Erlangen-Nürnberg
Regionales RechenZentrum Erlangen (RRZE)
Martensstraße 1, 91058 Erlangen, Germany
Tel. +49 9131 85-28689, -28800, Fax +49 9131 302941

www.win-labor.dfn.de