perfsonar development work

[Ilias Tsompanidis <>]

I looked into the tools Mario proposed
http://egee-jra1-testing.web.cern.ch/egee-jra1-testing/tools.html

Apart from the web-site / gui applications testing tools, the open 
source web service testing tools don't have much to offer, compared to 
soapUI (which is already used). The commercial ones also have similar 
functionality with soapUI. What we could use are the distributed testing 
tools, which could help us spot bottlenecks at services on production 
and also perform tests remotely.

Apart from these, i have made some notes on some other kind of software 
(as the "japitools" API checker, "Valgrind" system profiler and "Dart" 
nightly build checker), which were on the website Mario proposed, and 
looked interesting.

You can find my notes for each one on the attached file.

Greets,
Ilias

Test Manager - Freeware - closed source
        Java based tool, extendable using home-made java classes. Too generic 
for web service tests. 
        Easy to handle file equality checks. Could be used to create a 
general report on the results/log files of other testing tools

TETWare - 3.8 version is commercial , 3.7 open source
        This is actually an API for JAVA, C/C++ and perl. Still much coding 
is needed. Doesn't offer any real advantages over the 
        gui tools, or the written scripts.

QMTest - GNU - Open Source
        Testing framework, wraps easily around the "home-made" scripts, 
integrated web server for presenting the results. 
        Checks exit codes and produced output. Will execute any program / 
script. Works on python (available for UNIX/ Windows)

Canoo WebTest - java - open source
        Web site testing tool. Groovy can be used and in cooperation with 
ant, ant targets can be invoked. Specifically tailored
        for web sites (and not so much for web services)

DejaGNU - GNU - Tcl
        Testing framework, with no additional ready to use features. Depends 
heavily on "Expect" to run the external tests 
        External test can be batch oriented or interactive programs. Not 
really noteworthy for our situation.

QAWizard/QADirector - Commercial licences.
        Used for web sites and windows application testing. Can execute 
external scripts, and can used specially-tailored scripts
        (Based on a VB-like language).

STAF/STAX
        Automation / testing framework
        Staf presents an initial setup overhead, since it is more of a 
distributed system. Maybe consider it for monitoring
        and testing Multi-LS

Dart    - Freeware 
        Regression Testing at build time. Supports automatic "nightly build" 
testing. Quite irrelevant to functional testing, but
        could be used to ensure that trunk is always "compile-able". I'll 
look into it some more when time allows.
        Based on a client/server architecture, which could allow testing 
local changes, and storing the results centrally.


Unit Testing - Various
        There are some extensions to junit and also junit equivalents for C, 
C++ , Perl and python. 


Japitools - Java - GNU
        Could be very valuable if we produce a perfSONAR API. Runs 
compatibility tests for APIs. Would be used for backwards
        compatibility checks and also common API elements checks. (An API for 
the same version of sql-ma and rrd-ma should be 
        identical for the outside world)

Netlogger - c,java,python API - LBNL
        A compilation of libraries / tools used for performance analysis on 
distributed systems and applications
        Used when you want to:
                do performance/bottleneck analysis on distributed applications
                determine which hardware components to upgrade to alleviate 
bottlenecks
                do real-time or post-mortem analysis of applications
                correlate application performance with system information 
(ie: TCP retransmission's)
                works best with applications where you can follow a specific 
item (data block, message, object) through the system

Clover - java - Commercial
        Junit addon. Checks the percentage of the code tested by junit tests 
and percentage of the code actually used (helps bring 
        out dead code). Commercial with evaluation period licences.



Rational Performance Tester - Java / Web Browser - Commercial - IBM
        Can perfon functional and most importantly multi-user performance 
testing on web interfaces.
        Tests are created by monitoring the user's steps on his browser. Then 
they are scheduled  and repeated via a provided tool.
        Java code can be written to enrich the tests.
        Single user licence (12 months) : 1572 Euros
        Floating user licence (12 months) : 3124 Euros
        30 - day evaluation period for the trial, full-featured

Insure ++ - C++ - Commercial
        C++ code optimization tool. Useless for us.

Valgrind - C++ - GNU
        Valgrind is a suite of profiling tools. Consists of:
                Memory space monitoring (detects leaks and heap block 
overruns)
                Cache profiler
                Heap profiler
                Thread debbuger
        It is supposed to be able to profile programs run on linux, written 
in any language. It's very low level. Maybe the developers
        would be interested in using this for optimising their code. The 
memory usage profiling is the most common functionality used.


SOAtest - java(most probably) - Commercial
        SOAP Tester / extremely similar to soapUI. Reads wsdl, supports xpath 
queries. Even look and feel is similar to soapUI.
        A demonstration can be found at: 
http://www.parasoft.com/jsp/products/support/presentation/flash/soap/demo/4.5/STD.html

e-TEST suite. - .NET - Commercial
        Web site testing. Records the behaviour of the user and acts like 
one. Tests can be tweaked and optimized to resemble different
        user choices. The suite contains the test creator, the functional and 
regression testing component and the load and performance
        testing component. 

TestMaker - java - Open Source
        Another soapUI-equivalent. It can automate web-site testing also, but 
it is muh more difficult to use, compairing to soapUI.

LoadRunner - java - Commercial
        It is a load testing tool , hence the name, under the brand of HP. 
Didn't have the opportunity to do a hands-on test.
        Seems inadequate for our needs, especially when other tools offer 
functional-and load testing the same time.

Rational Functional Tester - IBM - java/.NET - Commercial
        Yet another functional and regression testing tool, used for web-form 
based applications. There's no need to use something
        like this for most of the services (not web-form , web-page based 
ones anyway)

QARun - java / .NET - Commercial
        A bit outdated. Point-and-click test creation for web-form based 
applications. A little tricky to get more information.
        Not such a good choice.

WebInject - perl / XML - OpenSource
        Strict on the type of tests. Home made scripts/classes and soapUI 
offer so much more! It can invoke web-forms and create SOAP
        envelopes for already created XML files. It  has an easy way to 
convert a test to MRTG external script or to a Nagios plugin.

Avignon - java / XML - opensource
        Uses XML to build test scripts for web-form based applications. 
Memorable only because the tests created are executed via
        junit, as testcases. Poor Documentation and even less satisfactory 
features. Doesn't support SOAP and web services. Used
        for acceptance testing.


GUI testing tools

Pounder - java / junit - opensource
        No longer developed. Poor documentation. Can record user behaviour to 
create tests and uses junit to execute them. The poor 
        documentation makes it quite difficult to use. Can be used for GUI 
layout tests, platform independently.

QF-Test (qftestJUI) - Java / Jython - Commercial , evaluation licences for 
opensource projects
        Can be used for funstional testing of GUI tools. Easy to use 
recorder, quite powerful, communicates with the given application
        at java-level (uses RPCs). External scripts can be created using 
jython. Licence is needed in order to save and replay
        new tests.

Xrunner - no longer available



Static analysis and metrics tools

C++ Test (code wizard) - C++ - commercial
        toolkit for code analysis, code review, automated unit and component 
testing, coverage analysis, and regression testing.
        Exclusively used for c++ programs. Licences are required.

Logiscope - C++ / Java - commercial
        Logiscope checks code against a set of project-defined programming 
rules. It helps maintain a style, and good-programming
        practices across a project. Works under Windows / linux (redhat) / 
solaris , and can be used against c,c++,ADA and java
        code. Licences are required.

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature