[squeak-dev] Mutation Testing

Although I didn't get to see the presentation on Mutation Testing at ESUG, I did have a chance to
see it the following week after my XtC talk in London. At the time I had promised to try it out.

http://www.slideshare.net/esug/mutation-testing

My Fractions Calculator has 105 Sunits to confirm the keyboard state machine works, the calculator grammar via Smacc is sane,
and the math results are as defined by either Wolfram Alpha, or Microsoft Excel rules.  Actually the number of tests are 210 because they are
subclassed for Excel, but running just the Wolfram Alpha based ones were sufficient to test the code base.

I had two problems, one where the RBParser choked on
#(#foo:. #bar:.)
Oops the code really should have been #(#foo: #bar:) or {#foo:. #bar:.}, still it shouldn't have given a walkback.

The second was that I had to increase the memory ceiling of the macintosh squeak carbon VM from 512MB to 1.5GB via the info.plist
so that a recursive error situation would not make the VM run out of memory before the Mutation testing runtime monitor would kill the
runaway task.

As a result over the 105 SUnits it reported:

It made 874 mutants, 873 got killed, 1 was terminated.

Which means I don't have any head scratching this afternoon.

--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================

Hi John
 great! so you covered all the cases!!.... it means you are a great programmer or mutation testing is not good :-) haha.
 Seriously, it is great you could try it and thank you for the report.
 We will check to see if we can do something to avoid changing the memory ceiling.

 Bye,
 Hernan.

On Tue, Sep 22, 2009 at 6:28 PM, John M McIntosh <[hidden email]> wrote:

Although I didn't get to see the presentation on Mutation Testing at ESUG, I did have a chance to
see it the following week after my XtC talk in London. At the time I had promised to try it out.

http://www.slideshare.net/esug/mutation-testing

My Fractions Calculator has 105 Sunits to confirm the keyboard state machine works, the calculator grammar via Smacc is sane,
and the math results are as defined by either Wolfram Alpha, or Microsoft Excel rules.  Actually the number of tests are 210 because they are
subclassed for Excel, but running just the Wolfram Alpha based ones were sufficient to test the code base.

I had two problems, one where the RBParser choked on
#(#foo:. #bar:.)
Oops the code really should have been #(#foo: #bar:) or {#foo:. #bar:.}, still it shouldn't have given a walkback.

The second was that I had to increase the memory ceiling of the macintosh squeak carbon VM from 512MB to 1.5GB via the info.plist
so that a recursive error situation would not make the VM run out of memory before the Mutation testing runtime monitor would kill the
runaway task.

As a result over the 105 SUnits it reported:

It made 874 mutants, 873 got killed, 1 was terminated.

Which means I don't have any head scratching this afternoon.

--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================