Shibboleth Developers

[Chad La Joie <>]

General comment.  I was specifically avoiding the user of terms like 
"unit test" and "integration test", I didn't even want to use 
"functional test" but couldn't think of anything better.  I avoided them 
because people have different meanings for each of these terms and I 
wanted to avoid any semantic war.

Walter Hoehn wrote:
> On Nov 17, 2005, at 6:35 AM, Chad La Joie wrote:
>
> > So, if I can try to summarize Will's and Howard's response:
> >
> > What we need to test:
> >  - Individual classes, or small groups of highly inter-related classes
> >  - Subsystems (e.g. the AA, AAP, Trust engine)
>
>
> I'd group these tests together in the same category and call them  "unit 
> tests".  We have many tests written for the IdP at this level,  and they 
> all use junit without requiring mockrunner.  When tests  exercise the 
> functionality of a subsystem, it seems desirable to  isolate that 
> subsystem as much as possible within the test.  When a  test fails, it 
> should be relatively easy to determine the source of  the problem.

I totally agree.  I was just trying to delimit testing a single class 
and testing a subsystem.  You might do the testing in the same way, with 
the same tools.

> Until now, we've been referring to the level of testing above this as  
> "black box testing".  The point of these tests is to exercise a  
> system,  ie: the IdP.  Will wrote a suite of these tests for the IdP  
> some months ago and they were committed to the main cvs a month or  two 
> back.  It is true that these sorts of tests are "integration  tests", in 
> the sense that they indirectly test the interdependencies  of the 
> various subsystems.  However, the main motivation for writing  these 
> tests was simply that a lot of the code I considered to be  brittle was 
> not in the subsystems, but rather in the core (previously  in the 
> servlets and now in the handlers).  Testing these classes is  similar to 
> testing the subsystems, the main difference being that the  interface is 
> the servlet interface instead of java interfaces that  are defined by 
> us.  Enter mockrunner...
>
> >  - Round trip functional test (e.g. making an attribute query and  
> > getting a valid response)
>
>
> I'm not one to argue against more testing, but I do think you loose  
> testing value as you become less specific about what you are  testing.  
> It becomes hard to understand exactly what you are testing  and hard to 
> trace problems to their root.  Having said that, if it is  easier to do 
> system testing on the SP by using output of the IdP, so  be it.  I'm 
> 100% not sure I buy this, though, since this limits you  to testing 
> exactly what the IdP can emit at the time of the test.

I don't necessarily agree here.  I think if you try to use these higher 
level tests as way to provide test coverage for the lower level 
(individual class or subsystem) tests then yes, I think you loose value. 
 If you use these to tests that all the components are working together 
(i.e. that the sum of the parts isn't introducing problems) then you 
gain value.
--
Chad La Joie             315Q St. Mary's Hall
OIS-Middleware           202.687.0124