Smalltalk › Squeak › Squeak - Dev

[squeak-dev] ENH: SUnit - shuffling the order tests are executed in

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

10 messages Options

Brent Pinkney-2

[squeak-dev] ENH: SUnit - shuffling the order tests are executed in

Hi,

I would like to propose an enhancement to SUnit:

Currently the order in which TestCase #test... methods are executed by TestSuite is fixed.
This can cause dependencies to creep in if
a) the TestCase class uses a TestResource and
b) one #test.. method alters an aspect of the TestResource that another #test... assumes is invariant.

By adding #shuffle to TestSuite >> run:, it is more likely that this corruption is exposed.

viz.
TestSuite >> run: aResult
self tests shuffled do:
[ :each |
self changed: each.
each run: aResult ]
!

Please comment and I will submit a fix.
PS. Who maintains SUnit ?

Brent

--
Brent

david54

Re: [squeak-dev] ENH: SUnit - shuffling the order tests are executed in

I like the ability to shuffle the test order, but I think this should be a preference and that you should be able to re-run the same exact test sequence.
I can't give you a crisp reason, but my gut tells me that you will want a way to re-run the same test sequence when things go strange.

-david

On Thu, Aug 14, 2008 at 11:12 AM, Brent Pinkney <[hidden email]> wrote:

Hi,

I would like to propose an enhancement to SUnit:

Currently the order in which TestCase #test... methods are executed by TestSuite is fixed.
This can cause dependencies to creep in if
a) the TestCase class uses a TestResource and
b) one #test.. method alters an aspect of the TestResource that another #test... assumes is invariant.

By adding #shuffle to TestSuite >> run:, it is more likely that this corruption is exposed.

viz.
TestSuite >> run: aResult
self tests shuffled do:
[ :each |
self changed: each.
each run: aResult ]
!

Please comment and I will submit a fix.
PS. Who maintains SUnit ?

Brent

--
Brent

Zulq Alam-2

[squeak-dev] Re: ENH: SUnit - shuffling the order tests are executed in

Is your gut thinking something like this?

- A test fails
- You do something you believe fixes it
- The test passes when rerun in different sequence
- You move on without actually fixing the bug

I think that simply rerunning the failed test after you believe you've
fixed the problem may not be sufficient because other tests may have
modified the TestResource since the failure.

For the shuffle to be useful you need to be lucky. Lucky enough that the
sequence that was causing a test to pass incorrectly is broken. Then you
need to be just as lucky to repeat the failing sequence.

Z.

David Pennell wrote:

> I like the ability to shuffle the test order, but I think this should be
> a preference and that you should be able to re-run the same exact test
> sequence.
> I can't give you a crisp reason, but my gut tells me that you will want
> a way to re-run the same test sequence when things go strange.
>
> -david
>
> On Thu, Aug 14, 2008 at 11:12 AM, Brent Pinkney <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> Hi,
>
> I would like to propose an enhancement to SUnit:
>
> Currently the order in which TestCase #test... methods are executed
> by TestSuite is fixed.
> This can cause dependencies to creep in if
> a) the TestCase class uses a TestResource and
> b) one #test.. method alters an aspect of the TestResource
> that another #test... assumes is invariant.
>
> By adding #shuffle to TestSuite >> run:, it is more likely that this
> corruption is exposed.
>
> viz.
> TestSuite >> run: aResult
> self tests shuffled do:
> [ :each |
> self changed: each.
> each run: aResult ]
> !
>
> Please comment and I will submit a fix.
> PS. Who maintains SUnit ?
>
> Brent
>
>
> --
> Brent
>
>
>
> ------------------------------------------------------------------------
>
>

david54

Re: [squeak-dev] Re: ENH: SUnit - shuffling the order tests are executed in

Yep, I think that's what it was telling me. Either that or I'm hungry :)

On Thu, Aug 14, 2008 at 7:04 PM, Zulq Alam <[hidden email]> wrote:

Is your gut thinking something like this?

- A test fails
- You do something you believe fixes it
- The test passes when rerun in different sequence
- You move on without actually fixing the bug

I think that simply rerunning the failed test after you believe you've fixed the problem may not be sufficient because other tests may have modified the TestResource since the failure.

For the shuffle to be useful you need to be lucky. Lucky enough that the sequence that was causing a test to pass incorrectly is broken. Then you need to be just as lucky to repeat the failing sequence.

Z.

David Pennell wrote:

I like the ability to shuffle the test order, but I think this should be a preference and that you should be able to re-run the same exact test sequence.
I can't give you a crisp reason, but my gut tells me that you will want a way to re-run the same test sequence when things go strange.
-david

On Thu, Aug 14, 2008 at 11:12 AM, Brent Pinkney <[hidden email] <mailto:[hidden email]>> wrote:

Hi,

I would like to propose an enhancement to SUnit:

Currently the order in which TestCase #test... methods are executed
by TestSuite is fixed.
This can cause dependencies to creep in if
a) the TestCase class uses a TestResource and
b) one #test.. method alters an aspect of the TestResource
that another #test... assumes is invariant.

By adding #shuffle to TestSuite >> run:, it is more likely that this
corruption is exposed.

viz.
TestSuite >> run: aResult
self tests shuffled do:
[ :each |
self changed: each.
each run: aResult ]
!

Please comment and I will submit a fix.
PS. Who maintains SUnit ?

Brent

--
Brent

------------------------------------------------------------------------

Andres Valloud-3

Re: [squeak-dev] ENH: SUnit - shuffling the order tests are executed in

In reply to this post by david54

In my opinion, this is an indirect fix for improperly written tests.
Detecting dependencies that should not be there is a good thing, but I
think it's even better to write tests that are truly standalone. My
concern is that, if I were writing poor tests, as soon as I find out
that my contributed tests make this technique useful, I will eventually
find a way to prevent my tests being detected, and then my
accomplishment will be to find yet another way to write tests poorly.
In other words, I see little incentive to write my tests as well as I
write my code because my goal is to avoid "punishment". For this
reason, I'd rather invest the time in training myself to write tests
better in the first place. Then the tests are better, and I do not need
to spend time implementing ways to detect bad code.

But... this is only what works for me. Your mileage may vary.

Andres.

David Pennell wrote:

> I like the ability to shuffle the test order, but I think this should be a
> preference and that you should be able to re-run the same exact test
> sequence.
> I can't give you a crisp reason, but my gut tells me that you will want a
> way to re-run the same test sequence when things go strange.
>
> -david
>
> On Thu, Aug 14, 2008 at 11:12 AM, Brent Pinkney <[hidden email]> wrote:
>
>
>> Hi,
>>
>> I would like to propose an enhancement to SUnit:
>>
>> Currently the order in which TestCase #test... methods are executed by
>> TestSuite is fixed.
>> This can cause dependencies to creep in if
>> a) the TestCase class uses a TestResource and
>> b) one #test.. method alters an aspect of the TestResource that
>> another #test... assumes is invariant.
>>
>> By adding #shuffle to TestSuite >> run:, it is more likely that this
>> corruption is exposed.
>>
>> viz.
>> TestSuite >> run: aResult
>> self tests shuffled do:
>> [ :each |
>> self changed: each.
>> each run: aResult ]
>> !
>>
>> Please comment and I will submit a fix.
>> PS. Who maintains SUnit ?
>>
>> Brent
>>
>>
>> --
>> Brent
>>
>>
>>
>
>
> ------------------------------------------------------------------------
>
>
>

Colin Putney

Re: [squeak-dev] Re: ENH: SUnit - shuffling the order tests are executed in

In reply to this post by Zulq Alam-2

On 14-Aug-08, at 5:04 PM, Zulq Alam wrote:

> Is your gut thinking something like this?
>
> - A test fails
> - You do something you believe fixes it
> - The test passes when rerun in different sequence
> - You move on without actually fixing the bug

It doesn't need to be anything that exotic. If some dependency between
tests *does* creep in, the shuffling will cause the failure to be
intermittent. Test B fails only when it comes after Test A, but before
TestC or something like that. If the suite is shuffled on every run
it'll be hard to reproduce the failure. A better strategy would be to
shuffle when the suite is created, then always run in the same order.
That way you can reproduce the sequence that failed, and figure out
how to fix it.

Colin

Andreas.Raab

[squeak-dev] Re: ENH: SUnit - shuffling the order tests are executed in

Colin Putney wrote:
> It doesn't need to be anything that exotic. If some dependency between
> tests *does* creep in, the shuffling will cause the failure to be
> intermittent. Test B fails only when it comes after Test A, but before
> TestC or something like that. If the suite is shuffled on every run
> it'll be hard to reproduce the failure. A better strategy would be to
> shuffle when the suite is created, then always run in the same order.
> That way you can reproduce the sequence that failed, and figure out how
> to fix it.

Alternatively, only shuffle if the last run succeeded. If it fails,
re-run the tests in the same order until it's been fixed.

Cheers,
- Andreas

Colin Putney

Re: [squeak-dev] ENH: SUnit - shuffling the order tests are executed in

In reply to this post by Andres Valloud-3

On 14-Aug-08, at 5:30 PM, Andres Valloud wrote:

> In my opinion, this is an indirect fix for improperly written
> tests. Detecting dependencies that should not be there is a good
> thing, but I think it's even better to write tests that are truly
> standalone. My concern is that, if I were writing poor tests, as
> soon as I find out that my contributed tests make this technique
> useful, I will eventually find a way to prevent my tests being
> detected, and then my accomplishment will be to find yet another way
> to write tests poorly. In other words, I see little incentive to
> write my tests as well as I write my code because my goal is to
> avoid "punishment". For this reason, I'd rather invest the time in
> training myself to write tests better in the first place. Then the
> tests are better, and I do not need to spend time implementing ways
> to detect bad code.
>
> But... this is only what works for me. Your mileage may vary.

Yeah, the tests are improperly written. But just saying "don't do
that" doesn't help.

I want to write the best tests I can. I *do* write the best tests I
can. But sometimes, when the test suite is large, and when I haven't
personally written all the tests, dependencies can creep in, despite
my best efforts. In that case, I want the best tools possible to help
me find and fix the problem. Shuffling the tests makes it more likely
that I'll notice the problem. Being able to reliably repeat a
problematic test run makes it easier to diagnose the problem.

Brent's proposal isn't about "punishing" developers who write
"improper" tests. It's about making the problems more visible so they
can be found and fixed more easily. Sounds good to me.

Colin

Colin Putney

Re: [squeak-dev] Re: ENH: SUnit - shuffling the order tests are executed in

In reply to this post by Andreas.Raab

On 14-Aug-08, at 5:50 PM, Andreas Raab wrote:

> Colin Putney wrote:
>> It doesn't need to be anything that exotic. If some dependency
>> between tests *does* creep in, the shuffling will cause the failure
>> to be intermittent. Test B fails only when it comes after Test A,
>> but before TestC or something like that. If the suite is shuffled
>> on every run it'll be hard to reproduce the failure. A better
>> strategy would be to shuffle when the suite is created, then always
>> run in the same order. That way you can reproduce the sequence that
>> failed, and figure out how to fix it.
>
> Alternatively, only shuffle if the last run succeeded. If it fails,
> re-run the tests in the same order until it's been fixed.

Even better.

Colin

Bergel, Alexandre

Re: [squeak-dev] Re: ENH: SUnit - shuffling the order tests are executed in

>>
>> Alternatively, only shuffle if the last run succeeded. If it fails,
>> re-run the tests in the same order until it's been fixed.
>
> Even better.

+1

Alexandre
--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.