[squeak-dev] Why is TestFailure non-resumable?

On 10-Sep-09, at 1:52 PM, Eliot Miranda wrote:

Hi All,

   if I want to run through a complex test that tests a number of combinations, some of which have failures, its convenient to wrap the test in an exception handler for TestFailure and run the tests resuming on each failure.  But I can't because TestFaiure isn't resumable.  Any good reason why?

e.g. I'm currently testing the generation of a divide/remainder routine for the x86 where I'm enumerating over many combinations of register quads, dividend, divisor, quotient, remainder, and I'd like to do the following to count how many failures I'm getting with each tweak of the register juggling code:

Well, SUnit wasn't designed to be used that way. Each test is supposed to have a binary result. Either it passes, or it doesn't. Sometimes the UI distinguishes between an assertion failing and an error, but that's sort of incidental. So TestFailure isn't resumable because we already have a result for the test: it failed. There's no need to continue executing the test, because further failures don't give any new information. In fact, further assertions are meaningless - we already know that *something* isn't as it should be. If further assertions fail, does that indicate additional problems, or the same one manifesting its self in different ways?

The "proper" way to do this in SUnit would be to have a whole bunch of different tests that run the same code with different inputs. That's probably not practical.  A better way would be to write your own test harness. SUnit is great, but it's not the last word on testing, right?

Colin

On 10-Sep-09, at 1:52 PM, Eliot Miranda wrote:

Hi All,

   if I want to run through a complex test that tests a number of combinations, some of which have failures, its convenient to wrap the test in an exception handler for TestFailure and run the tests resuming on each failure.  But I can't because TestFaiure isn't resumable.  Any good reason why?

e.g. I'm currently testing the generation of a divide/remainder routine for the x86 where I'm enumerating over many combinations of register quads, dividend, divisor, quotient, remainder, and I'd like to do the following to count how many failures I'm getting with each tweak of the register juggling code:

Well, SUnit wasn't designed to be used that way. Each test is supposed to have a binary result. Either it passes, or it doesn't. Sometimes the UI distinguishes between an assertion failing and an error, but that's sort of incidental. So TestFailure isn't resumable because we already have a result for the test: it failed. There's no need to continue executing the test, because further failures don't give any new information. In fact, further assertions are meaningless - we already know that *something* isn't as it should be. If further assertions fail, does that indicate additional problems, or the same one manifesting its self in different ways?

The "proper" way to do this in SUnit would be to have a whole bunch of different tests that run the same code with different inputs. That's probably not practical.  A better way would be to write your own test harness. SUnit is great, but it's not the last word on testing, right?

Colin