Shibboleth Developers

[Walter Hoehn <>]

On Nov 17, 2005, at 9:28 AM, Chad La Joie wrote:

> General comment.  I was specifically avoiding the user of terms  
> like "unit test" and "integration test", I didn't even want to use  
> "functional test" but couldn't think of anything better.  I avoided  
> them because people have different meanings for each of these terms  
> and I wanted to avoid any semantic war.

I think we needs these sorts of terms.  It doesn't matter so much how  
we define them, as long as we all agree on their meaning.

> Walter Hoehn wrote:
> > On Nov 17, 2005, at 6:35 AM, Chad La Joie wrote:
> > > So, if I can try to summarize Will's and Howard's response:
> > >
> > > What we need to test:
> > >  - Individual classes, or small groups of highly inter-related  
> > > classes
> > >  - Subsystems (e.g. the AA, AAP, Trust engine)
> > I'd group these tests together in the same category and call them   
> > "unit tests".  We have many tests written for the IdP at this  
> > level,  and they all use junit without requiring mockrunner.  When  
> > tests  exercise the functionality of a subsystem, it seems  
> > desirable to  isolate that subsystem as much as possible within  
> > the test.  When a  test fails, it should be relatively easy to  
> > determine the source of  the problem.
>
> I totally agree.  I was just trying to delimit testing a single  
> class and testing a subsystem.  You might do the testing in the  
> same way, with the same tools.
>
> > Until now, we've been referring to the level of testing above this  
> > as  "black box testing".  The point of these tests is to exercise  
> > a  system,  ie: the IdP.  Will wrote a suite of these tests for  
> > the IdP  some months ago and they were committed to the main cvs a  
> > month or  two back.  It is true that these sorts of tests are  
> > "integration  tests", in the sense that they indirectly test the  
> > interdependencies  of the various subsystems.  However, the main  
> > motivation for writing  these tests was simply that a lot of the  
> > code I considered to be  brittle was not in the subsystems, but  
> > rather in the core (previously  in the servlets and now in the  
> > handlers).  Testing these classes is  similar to testing the  
> > subsystems, the main difference being that the  interface is the  
> > servlet interface instead of java interfaces that  are defined by  
> > us.  Enter mockrunner...
> > >  - Round trip functional test (e.g. making an attribute query  
> > > and  getting a valid response)
> > I'm not one to argue against more testing, but I do think you  
> > loose  testing value as you become less specific about what you  
> > are  testing.  It becomes hard to understand exactly what you are  
> > testing  and hard to trace problems to their root.  Having said  
> > that, if it is  easier to do system testing on the SP by using  
> > output of the IdP, so  be it.  I'm 100% not sure I buy this,  
> > though, since this limits you  to testing exactly what the IdP can  
> > emit at the time of the test.
>
> I don't necessarily agree here.  I think if you try to use these  
> higher level tests as way to provide test coverage for the lower  
> level (individual class or subsystem) tests then yes, I think you  
> loose value.  If you use these to tests that all the components are  
> working together (i.e. that the sum of the parts isn't introducing  
> problems) then you gain value.

Maybe so, but there is almost nothing but overlap here, unless the  
highest level tests are very minimal.

-Walter