Re: Asking newcomers to make SUnit tests

[BUG] Complex equality problem
stéphane ducasse ducasse at iam.unibe.ch
Wed Feb 8 16:58:32 CET 2006  wrote:


>I suggest that you write SUnit tests to document your
intention and  
>that we can follow what you are doing.

context:
http://lists.squeakfoundation.org/pipermail/squeak-dev/2006-February/100484.html

Hi stef,
I want to take issue with this suggestion to a new
comer. I know you are trying to improve the quality of
squeak. And I wish that you would think about the
amount of effort you are calling for and who you are
asking to shoulder that effort.

This is like asking someone who comes across a cow
wandering in the wild to find a test that the pastures
gate is closed in the future. Its rather an imposition
on someone who has just done you a favor.

One, using SUnit tests requires learning how. This
learning curve requires time and personal resources to
master.  It is not neccessarily the purpose of the one
who is trying to solve the problem created and left by
another who did not write a SUnit test. Obviously they
did not have time to master Sunit tests as well.

Secondly, stomping on a bug and writing exhaustive
tests to prove the bug remains stomped are two
different efforts.  The second is much more exacting
and to a certain extent made surperfluous by the
eradication of the bug now being tested for.

Thirdly, all you really need is a good interactive “am
I in trouble test.”  Not exhaustive, but a way of
quickly torturing the problem to see if fails.
Usually the original symptoms of the bug just need to
be recreated and tested for.  This is often an
interactive test and not a SUnit.

Fourthly, looking at the results of SUnit tests is so
boring that some have started writing code to allow
“ignoring” tests that are known to “fail.  That
suggests timely bug fixing is more important.

The proper time to write SUnits is either at the
design stage so you have a way to prove the code works
once you’ve written it. Then the test becomes part of
the documation of the code.

The other proper way to get SUnit tests is get a team
who are trained in the ins and outs of SUnit testing.
Have them work with the bug finders.  If a bug is
found then there will general be a “am I in trouble
test” for it. Have the team take that info and produce
the SUnit.

I realize that producing a team would take
considerable effort on the part of Squeak maintainers.
 Is that effort worth making? If not, putting the
burden on the causal bug finder is not justifiable and
to my sensebilities somewhat obnoxious.

At the same time any confusions that come up in
turning the finders test into SUnits can be used to
add training documentation to teach SUnit building to
others.

You once warned me that email is a poor way to
communicate. I realize some of my passions have got
into this message.  Please make allowance for that.  I
appreciate that you have spent lots of time and effort
to help squeak grow and that your passion shows in
your suggest for tests.  I hope this discussion gets
us furthur torwards what we both want a better more
capable squeak and a commmunity that has fun using and
developing it.

Yours in service, -- Jerome Peace






__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

On 08.02.2006, at 23:02, Peace Jerome wrote:

If you look at the mails of nicolas (that stef was answering to),
than it seems quite clear to me that learning how to write
an SUnit test for testing that bug will not be impossible for him.

> One, using SUnit tests requires learning how. This
> learning curve requires time and personal resources to
> master.  It is not neccessarily the purpose of the one
> who is trying to solve the problem created and left by
> another who did not write a SUnit test. Obviously they
> did not have time to master Sunit tests as well.
>

It is hard to write really good tests, yes. But it is not
hard to write the kind of tests that we are talking about
here: the Complex equality bug.

Tests have multiple purposes, and maybe we should just
name these kinds of tests different: "Bug-showing test"
It was not asked for an exaustive unit-test suite for Complex.

> Secondly, stomping on a bug and writing exhaustive
> tests to prove the bug remains stomped are two
> different efforts.

But isn't having that one test better than having no test at all?

>
> Thirdly, all you really need is a good interactive “am
> I in trouble test.”  Not exhaustive, but a way of
> quickly torturing the problem to see if fails.
> Usually the original symptoms of the bug just need to
> be recreated and tested for.  This is often an
> interactive test and not a SUnit.
>

But not in the case of the Complex zero that we are talking about
gere.

> Fourthly, looking at the results of SUnit tests is so
> boring that some have started writing code to allow
> “ignoring” tests that are known to “fail.  That
> suggests timely bug fixing is more important.
>

Why that? There have been multiple cases in the past
that stuff was fixed only because there was a failing test.

>
> I realize that producing a team would take
> considerable effort on the part of Squeak maintainers.
>  Is that effort worth making? If not, putting the
> burden on the causal bug finder is not justifiable and
> to my sensebilities somewhat obnoxious.
>

The causual bug finder is not asked to write a complete
unit testing collection, just a test showing his bug so that
it's easy to reproduce.
>
> You once warned me that email is a poor way to
> communicate. I realize some of my passions have got
> into this message.

And I really don't understand how you can think that
test as a means for reproducing bugs is a bad idea.

      marcus

Le Mercredi 08 Février 2006 23:02, Peace Jerome a écrit :

Hi Jerome,

I do not totally agree. SUnit is easy to use and usefull to anyone writing
some code. So why not learning SUnit ?

> Secondly, stomping on a bug and writing exhaustive
> tests to prove the bug remains stomped are two
> different efforts. ..

A bug finder won't produce exhaustive test, just a little brick usefull for
non-regression purpose.
SUnit needs not being well structured, uniform and clever. Just a patchwork of
small independent or overlaping tests will do. So a brick is usefull.

> Thirdly, all you really need is a good interactive “am
> I in trouble test.”  Not exhaustive, but a way of
> quickly torturing the problem to see if fails.
> Usually the original symptoms of the bug just need to
> be recreated and tested for.  This is often an
> interactive test and not a SUnit.

you maybe right, but this does not really apply on the small Complex bug found
here.
This bug was infered using bottom up analysis of code, so writing a small
SUnit in this case is quite obvious.

> Fourthly, looking at the results of SUnit tests is so
> boring that some have started writing code to allow
> “ignoring” tests that are known to “fail.  That
> suggests timely bug fixing is more important.

It seems to me that it is just a good way of organizing things.
You should have a way to categorize tests into
 - non regression tests so far working
 - non yet fixed tests

At least you should replay the non regression test painlessly !
That suggests the SUnit tool need some addition to help categorizing, but if
guys are writing code, we will all profit by i hope.

> The other proper way to get SUnit tests is get a team
> who are trained in the ins and outs of SUnit testing.
> Have them work with the bug finders.  If a bug is
> found then there will general be a "am I in trouble
> test" for it. Have the team take that info and produce
> the SUnit.

Note that simple coverage tools would not help on the Complex equality bug.
You would have to be more exhaustive at class level and maybe at value level
(for each variable...). Something impossible even for well trained i guess.

Some tests at design step, some a posteriori, some small patchwork bricks and
some independent analysis of code and refactoring, you have to mix all of
that, i do not think there are only two proper ways.

>
> ... I hope this discussion gets
> us furthur torwards what we both want a better more
> capable squeak and a commmunity that has fun using and
> developing it.
>
> Yours in service, -- Jerome Peace
>
Thanks for your great work.

Peace Jerome writes:
 > Fourthly, looking at the results of SUnit tests is so
 > boring that some have started writing code to allow
 > �ignoring� tests that are known to �fail.  That
 > suggests timely bug fixing is more important.

That is a problem but not writing tests doesn't seem like
the right solution. One answer would be to refuse to merge
code unless all tests pass.

Bryce

On 8-Feb-06, at 2:02 PM, Peace Jerome wrote:

I guess I'd agree with you statement *partially*. Certainly for a  
complete newcomer it might be very daunting although if someone could  
take a few moments (an hour or so?) to write a swiki page on creating  
a simple SUnit it could be very helpful as an introduction to a very  
good way of helping to explain a problem and the solution and  
providing an ongoing test. It would also be a quite broad learning  
experience since it involves using a browser, editing code,  
iteratively trying things and probably getting to know the debugger a  
little. How much better an introduction could we invent?

In this case I think the questioner has a fair bit of Smalltalk  
experience and an SUnit is a reasonable thing to suggest.

Think of it this way:-
you write a little code to express your starting situation -  
answer := (4 +4i) * (Pi + e i)
you write a comment along the lines of "I think this should result in  
whatever"
you write a line to express your expected answer - self should:
[ answer = (1.693243300522992 + 23.43949792819535 i)]
If the SUnit fails you have a nice compact way of explaining the  
problem to the rest of us! Good result all round.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Great leaders inspire by example. When that's not an option, brute  
intimidation works pretty well.

Hi Nicholas, Marcus, Tim and all who replied to my
original message.

http://lists.squeakfoundation.org/pipermail/squeak-dev/2006-February/100497.html

Well, if Nicholas thinks its no problem then I
acknowledge I've picked the wrong example to complain
about.

And asking politely for something is always
appropriate. As long as the askee knows he can
decline.

Its a good way to get needed contributions.

Still I'm glad I wrote that all down. Some of the
discussion looks helpful.

If its possible to easily write an "Am-I-in-trouble."
Sunit test, Maybe someone could put up a page of
documentation and a template somewhere. To show us who
have to learn everything from scratch and usually the
hard way.

Consider this a polite request with a Please and Thank
you attached to it.

Tests are a very good idea.  Heaping work on a
newcomer without thinking of the effort being
requested is of concern to me.

I like Nicolas's quest for precision and order. I hope
his efforts are supported and his time contribution
appreciated.

I think some of my bug hunting frustrations got into
that last post.

Thanks for you attention.

Yours in service, -- Jerome Peace

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

No problem :)
Nicolas told me that he read the complete compiler of Smalltalk/V
so ...

But seriously having tests is really excellent to talk about a problem.
No english interpretation just facts. :)

Tim I wrote a tutorial to explain how to write tests and rick fixed  
my english :)

http://www.iam.unibe.ch/~ducasse/Programmez/OnTheWeb/SUnitEnglish2.pdf
I also have videos (for zapping generation)
http://www.iam.unibe.ch/~ducasse/Videos/SqueakOriginalMov/
Stef



On 9 févr. 06, at 03:20, Peace Jerome wrote:

"tim Rowledge" <[hidden email]> asks

> Certainly for a  
> complete newcomer it might be very daunting although if someone could  
> take a few moments (an hour or so?) to write a swiki page on creating  
> a simple SUnit it could be very helpful as an introduction to a very  
> good way of helping to explain a problem and the solution and  
> providing an ongoing test. It would also be a quite broad learning  
> experience since it involves using a browser, editing code,  
> iteratively trying things and probably getting to know the debugger a  
> little. How much better an introduction could we invent?

Stef's already mentioned his book. There's also
  http://www.chrisburkert.de/index.php?node_id=70
to which
  http://minnow.cc.gatech.edu/squeak/SUnit links.

frank