Shibboleth Developers

[Walter Hoehn <>]

On Nov 17, 2005, at 6:35 AM, Chad La Joie wrote:

> So, if I can try to summarize Will's and Howard's response:
>
> What we need to test:
>  - Individual classes, or small groups of highly inter-related classes
>  - Subsystems (e.g. the AA, AAP, Trust engine)

I'd group these tests together in the same category and call them  
"unit tests".  We have many tests written for the IdP at this level,  
and they all use junit without requiring mockrunner.  When tests  
exercise the functionality of a subsystem, it seems desirable to  
isolate that subsystem as much as possible within the test.  When a  
test fails, it should be relatively easy to determine the source of  
the problem.

Until now, we've been referring to the level of testing above this as  
"black box testing".  The point of these tests is to exercise a  
system,  ie: the IdP.  Will wrote a suite of these tests for the IdP  
some months ago and they were committed to the main cvs a month or  
two back.  It is true that these sorts of tests are "integration  
tests", in the sense that they indirectly test the interdependencies  
of the various subsystems.  However, the main motivation for writing  
these tests was simply that a lot of the code I considered to be  
brittle was not in the subsystems, but rather in the core (previously  
in the servlets and now in the handlers).  Testing these classes is  
similar to testing the subsystems, the main difference being that the  
interface is the servlet interface instead of java interfaces that  
are defined by us.  Enter mockrunner...

>  - Round trip functional test (e.g. making an attribute query and  
> getting a valid response)

I'm not one to argue against more testing, but I do think you loose  
testing value as you become less specific about what you are  
testing.  It becomes hard to understand exactly what you are testing  
and hard to trace problems to their root.  Having said that, if it is  
easier to do system testing on the SP by using output of the IdP, so  
be it.  I'm 100% not sure I buy this, though, since this limits you  
to testing exactly what the IdP can emit at the time of the test.

-Walter