perfsonar development work

[Michael Michalis <>]

Hi Jochen,

You have done a very thorough investigating work, and the information you
have provided us is very useful. I haven't really tested soapUI but the
ability to create requests based on a given WSWDL description is great and
could save a lot of time. Also the schema validating ability is great,
although I thing it can also be done by using a DTD specification in JDOM(I
didn't have the time to check it out yet :) ). I have some remarks though.
Firstly when you are functional testing a service you may need to provide
requests that are not well formatted or not NMWG schema compliant or just
missing some attributes or elements that are not important and the service
can live without them. I'm not sure how these tools can provide the former
kinds of requests. Secondly and most importantly creating, sending and
validating the requests to the service is not considered to be the hardest
job. The most important and tricky job of functional testing is to check the
data and the result code of the responses, against the actual data stored
inside the service database. From what you have written these tools don't
provide such functionality and thus scripts or classes doing this kind of
work still need to be created. Last but not least functional testing may
need to be automated, this means that functional testing should be carried
out without the presence of a human being, maybe running at late hours with
the results being stored in a text file. I'm pretty sure that anteater can
be automated but you mentioned that soapUI uses a GUI, meaning perhaps that
it can not run without a human surveillance. Concluding my long remarks;), I
have to say that these tools can really simplify the creation of requests
and the schema validation of the responses but IMHO they can not do all that
is needed for functional testing. Now because we seem to be on a tight
schedule and functional testing is an extra work load for you, maybe using
these tools in conjunction with some scripts or classes or even only these
tools if time is not enough, may be the best solution. Loukik may have more
to say about that.

Merry Christmas and a happy new year to you too Jochen and everyone else
reading this mail.

Best Regards

Michalis Michael                    

-----Original Message-----
From: Jochen Reinwand 
[mailto:]
 
Sent: Friday, December 22, 2006 3:38 PM
To: 

Subject: Re: [pS-dev] Functional Testing

Hello everybody,

Because Loukik thought it might be a good idea, I took a closer look at 
anteater and found also another very interesting tool called soapUI. Since
we 
haven't seen any code from Ilias that could help us in our functional
testing 
ambitions and Michalis code is in Java, that we are not so familiar with, we

really need some other framework for our functional testing. To start with 
the conclusion: Both anteater and soapUI seem to be sufficient for
functional 
testing!

The concepts of both tools are a bit different, but the overall abilities in

functional testing are very similar. Further on I will describe for both 
tools the typical work that needs to be done to get started with functional 
testing.



Anteater

Let's start with Anteater (http://aft.sourceforge.net/). Anteater is more or

less an enhancement of Ant, that, as you all know, is already used in 
perfSONAR as deployment tool. But Ant can very easily be expanded to do a
lot 
of other things. Anteater uses this features to enhance Ant with the 
capabilities for testing web services.

Here is the typical work cycle: You should have an example request for your 
service and an rnc schema for the corresponding response. Both things should

be available for perfSONAR services, because they are expected to be found
in 
the handover documentation ;-)
Now you just have to write an Ant XML file that uses the special targets of 
Anteater. You tell it to send the example XML as an request and to validate 
the result against the schema. The more accurate the schema, the better the 
testing. But you can do a lot more things:
- Analyse the header of the HTTP request.
- Use XPath syntax to extract parts of the document.
- Use regexps to validate the extracted values or the whole response.

There are some minor problems, when using Anteater. For example it does not 
generate nor parse the SOAP envelope and simply does an HTTP Get. So you
have 
to take care about some SOAP specific things, which is not really bad and 
also very helpful in some cases, e.g. you can do functional testing on the 
SOAP part of the service itself.

I attached a tarball containing a simple testing framework for our BWCTL MP.

It only sends a simple SetupDataRequest, validates the SetupDataResponse and

extracts a value using XPath.

You will find the following files:

anteater-test/bwctl-mp.xml
This is the Anteater enhanced Ant file. After installing Anteater, which is 
very simple, you just have to enter the anteater-test directory and use 
"anteater -f bwctl-mp.xml" to start the test.

anteater-test/resources/requests/bwctl-req.xml
anteater-test/resources/schema/bwctl-res.rnc
These are the example request and the rnc schema for the response directly 
copied from our release documentation.

anteater-test/resources/soap-footer.xml
anteater-test/resources/soap-header.xml
Our requests need a SOAP envelope, that is not generated by Anteater 
automatically. By 'cat'ing together soap-header.xml, bwctl-req.xml and 
soap-footer.xml we get our full request

anteater-test/resources/schema/soap-bwctl-res.rnc
anteater-test/resources/schema/soap.rnc
Since the response contains the SOAP envelope, these rnc schemas help us 
validating the response against bwctl-req.xml by taking care of the SOAP 
part.

Another very interesting feature of Anteater is the built in web server. You

can set up a simple SOAP server and test clients against it. You can even 
start a server and run test requests using the same Anteater process. Very 
funny.

But now the great drawback of Anteater: Seems like development has
completely 
stopped in the year 2003! Since the Anteater package contains nearly all 
needed third party software, you won't realise it at once. But the bundled 
software is rather old and you sooner or later find e.g. missing Ant 
features, that can only be found in newer versions of Ant.

If we decide to use Anteater it might be useful/necessary to take over 
development. It seems like Ant is really powerful and working on updating
and 
improving Anteater should be very simple. But I'm not an Ant or Java expert,

so I don't want to say to much about this topic (yet). Perhaps someone else 
wants to take a look at it...



soapUI

Let me simply quote the web page (http://www.soapui.org/) as starting point 
for my writing about soapUI:
"soapUI is a desktop application for inspecting, invoking, developing and 
functional/load/compliance testing of web services over HTTP. It is mainly 
aimed at developers/testers providing and/or consuming web services 
(java, .net, etc). Functional and Load-Testing can be done both
interactively 
in soapUI or within an automated build/integration process using the soapUI 
command-line tools. IDE-plugins are available for eclipse, IntelliJ IDEA, 
NetBeans and a specialized eclipse-plugin for JBossWS. soapUI requires java 
1.5 and is licensed under the LGPL license."

This said I should add that this tool seems to be the standard OpenSource
tool 
for functional testing of web services. Perhaps Anteater was abandoned 
because of this tool...
Because soapUI is a graphical user interface it's quite easy to "click" 
together functional tests. The typical workcycle: You import your service 
specification in WSDL format. Based on this soapUI knows how to use your 
service. You can now generate template XML requests, fill them with data and

send them to the service. By cloning you can easily create a lot of similar 
messages. You can even import data from text files and automatically fill
the 
request with them. Based on these requests you can build test cases. soapUI 
will send the request and then do some tests on the response. Typical tests 
are quite similar to Anteater:
- Validate against schema (WSDL)
- Extract and test values via XPath (more features than Anteater!)
All these tests are easy to create and you can test them at once against the

former received response. That's really comfortable!

There are some problems with soapUI that are more or less perfSONAR/NMWG 
specific. NMWG's request/response cannot be described using WSDL, like it 
should normally be done when using web services. Please note: My knowledge 
about SOAP and WSDL is far from being perfect! If you know how to describe a

perfSONAR/NMWG service using WSDL or if you already have a WSDL file, please

inform me about it! Having one will make functional testing for the
described 
service possible in a few hours!

On the other hand it's no great problem to work around the WSDL problem.
WSDL 
cannot (yet) use RNC/RNG for schema description, but both can be converted
to 
XML Schema (XSD) using e.g. trang. Since XML Schema is not as powerful as 
Relax-NG I have some problems at the moment. I cannot present to you a ready

and nice looking functional testing example, like I did for Anteater. Would 
be easy to hack something useful, but I first want to work on fixing some 
more problems. I will send a good example asap.

BTW: One drawback compared to Anteater could be the missing web server 
functionality for testing clients. soapUI is only a client...



Congratulations! You've managed to read through this "small" email!
As some sort of reward, let me wish you a Merry Christmas and a Happy New 
Year!
Jochen


On Tuesday 12 December 2006 14:26, Loukik Kudarimoti wrote:
> Jochen,
>
> I am not aware of any of us having tested this tool. From the
> description that you have given me, it looks very powerful and probably
> what we want. If you are going to do some testing with it, we would be
> very interested in hearing your experiences and outcomes from this
testing.
>
> Thanks,
> Loukik.
>
> Jochen Reinwand wrote:
> > Hi everybody,
> >
> > I just came across the Open Source tool Anteater, which describes itself
> > as "Ant-based functional testing framework": http://aft.sourceforge.net/
> >
> > It seems to be very powerful in doing functional testing of web services
> > (including SOAP). For testing you have to write ant scripts extended by
> > special tags for the testing targets. I'm not very familiar with ant,
but
> > the anteater scripts look very simple yet powerful. You select an xml
> > file to be sent to the service and can give conditions that the reply
> > should fullfill. You can e.g give a Relax-ng schema the reply must
> > validate against and you can do further tests on the content, like
> > pattern matching on parts of the reply denoted using XPath.
> > It's also possible to use anteater as "testing service" to validate
> > clients.
> >
> > Has anybody taken a closer look at this tool? I'm thinking about doing
> > some testing with it. If somebody knows, that anteater is not really
> > useful, please inform me to prevent me from doing stupid things ;-)
> >
> > regards,
> > Jochen

-- 
Jochen Reinwand                                 Tel: +49 9131 852-8689
DFN Labor

Regionales Rechenzentrum Erlangen
Martensstrasse 1
91058 Erlangen
 
email: