Weird SUnit/TestRunner behavior

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Weird SUnit/TestRunner behavior

Attila Magyar
Hi,

This was originally reported as a BabyMock bug but it looks like it effects
normal test cases as well.

Let's say I have a test like this:

MyTest>>initialize
    super initialize.
    s := OrderedCollection new. "note that this is initialized here not in
the setUp as we normally do"

MyTest>>testWhatever
    s add: 42.
    1 / 0.

Open the Test Runner tool, run the test, it'll fail as expected. Now click
on the failed test (in the test runner window) and the debugger will popup
saying that #add: was sent to nil. Instance variables are cleared out after
a test run but initialize is not called before running it again. Is this
expected? The same doesn't cause any problems when I run it from the System
Browser.




--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Reply | Threaded
Open this post in threaded view
|

Re: Weird SUnit/TestRunner behavior

ducasse
Thanks for the report.
We are working on SUnit improvements.

Stef

> On 18 Mar 2019, at 13:35, Attila Magyar <[hidden email]> wrote:
>
> Hi,
>
> This was originally reported as a BabyMock bug but it looks like it effects
> normal test cases as well.
>
> Let's say I have a test like this:
>
> MyTest>>initialize
>    super initialize.
>    s := OrderedCollection new. "note that this is initialized here not in
> the setUp as we normally do"
>
> MyTest>>testWhatever
>    s add: 42.
>    1 / 0.
>
> Open the Test Runner tool, run the test, it'll fail as expected. Now click
> on the failed test (in the test runner window) and the debugger will popup
> saying that #add: was sent to nil. Instance variables are cleared out after
> a test run but initialize is not called before running it again. Is this
> expected? The same doesn't cause any problems when I run it from the System
> Browser.
>
>
>
>
> --
> Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>



Reply | Threaded
Open this post in threaded view
|

Re: Weird SUnit/TestRunner behavior

ducasse
In reply to this post by Attila Magyar
You can add a ticket in DrTests in JulienDelplanque repo on github.

Stef

On 18 Mar 2019, at 13:35, Attila Magyar <[hidden email]> wrote:

Hi,

This was originally reported as a BabyMock bug but it looks like it effects
normal test cases as well.

Let's say I have a test like this:

MyTest>>initialize
   super initialize.
   s := OrderedCollection new. "note that this is initialized here not in
the setUp as we normally do"

MyTest>>testWhatever
   s add: 42.
   1 / 0.

Open the Test Runner tool, run the test, it'll fail as expected. Now click
on the failed test (in the test runner window) and the debugger will popup
saying that #add: was sent to nil. Instance variables are cleared out after
a test run but initialize is not called before running it again. Is this
expected? The same doesn't cause any problems when I run it from the System
Browser.




--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html


Reply | Threaded
Open this post in threaded view
|

Re: Weird SUnit/TestRunner behavior

Attila Magyar
Thanks, I added the ticket.



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Reply | Threaded
Open this post in threaded view
|

Re: Weird SUnit/TestRunner behavior

Max Leske
To answer your question: yes that has been the case for a long time. The
same is true for #runCase BTW. Only #setUp is sent after #tearDown (see
#prepareToRunAgain).

Max

On 18 Mar 2019, at 14:05, Attila Magyar wrote:

> Thanks, I added the ticket.
>
>
>
> --
> Sent from:
> http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Reply | Threaded
Open this post in threaded view
|

Re: Weird SUnit/TestRunner behavior

Attila Magyar
Max Leske wrote
> To answer your question: yes that has been the case for a long time. The
> same is true for #runCase BTW. Only #setUp is sent after #tearDown (see
> #prepareToRunAgain).
>
> Max

That's weird. RunCase sends a setUp, a tearDown and
cleanUpInstanceVariables. So the test will be executed with uninitialized
instance variables at the second time. Why do we need to clean up the
instance variables? Or why don't we reinitialize them before running the
test?



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Reply | Threaded
Open this post in threaded view
|

Re: Weird SUnit/TestRunner behavior

gcotelli
If you don't clean up the instance variables, when you have a lot of tests and a test runner is still open on the results all this garbage is still referenced (because the test results keeps a reference to the original test case instance) and the image keeps growing like crazy. I can tell you for my experience that in our case it gets easily to GB-sized images, and the GC is not very happy with this.

On Tue, Mar 19, 2019 at 12:04 PM Attila Magyar <[hidden email]> wrote:
Max Leske wrote
> To answer your question: yes that has been the case for a long time. The
> same is true for #runCase BTW. Only #setUp is sent after #tearDown (see
> #prepareToRunAgain).
>
> Max

That's weird. RunCase sends a setUp, a tearDown and
cleanUpInstanceVariables. So the test will be executed with uninitialized
instance variables at the second time. Why do we need to clean up the
instance variables? Or why don't we reinitialize them before running the
test?



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Reply | Threaded
Open this post in threaded view
|

Re: Weird SUnit/TestRunner behavior

Max Leske
On 19 Mar 2019, at 16:16, Gabriel Cotelli wrote:

> If you don't clean up the instance variables, when you have a lot of
> tests
> and a test runner is still open on the results all this garbage is
> still
> referenced (because the test results keeps a reference to the original
> test
> case instance) and the image keeps growing like crazy. I can tell you
> for
> my experience that in our case it gets easily to GB-sized images, and
> the
> GC is not very happy with this.

Indeed.

>
> On Tue, Mar 19, 2019 at 12:04 PM Attila Magyar <[hidden email]>
> wrote:
>
>> Max Leske wrote
>>> To answer your question: yes that has been the case for a long time.
>>> The
>>> same is true for #runCase BTW. Only #setUp is sent after #tearDown
>>> (see
>>> #prepareToRunAgain).
>>>
>>> Max
>>
>> That's weird. RunCase sends a setUp, a tearDown and
>> cleanUpInstanceVariables. So the test will be executed with
>> uninitialized
>> instance variables at the second time. Why do we need to clean up the
>> instance variables? Or why don't we reinitialize them before running
>> the
>> test?

I understand your grief. My solution: never use #initalize in tests, use
#setUp to initialize instance variables.

Max

>>
>>
>>
>> --
>> Sent from:
>> http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>>
>>



Reply | Threaded
Open this post in threaded view
|

Re: Weird SUnit/TestRunner behavior

ducasse
Indeed I do not even know what means initialize in a testCase. 

Stef


On 19 Mar 2019, at 17:28, Max Leske <[hidden email]> wrote:

On 19 Mar 2019, at 16:16, Gabriel Cotelli wrote:

If you don't clean up the instance variables, when you have a lot of tests
and a test runner is still open on the results all this garbage is still
referenced (because the test results keeps a reference to the original test
case instance) and the image keeps growing like crazy. I can tell you for
my experience that in our case it gets easily to GB-sized images, and the
GC is not very happy with this.

Indeed.


On Tue, Mar 19, 2019 at 12:04 PM Attila Magyar <[hidden email]> wrote:

Max Leske wrote
To answer your question: yes that has been the case for a long time. The
same is true for #runCase BTW. Only #setUp is sent after #tearDown (see
#prepareToRunAgain).

Max

That's weird. RunCase sends a setUp, a tearDown and
cleanUpInstanceVariables. So the test will be executed with uninitialized
instance variables at the second time. Why do we need to clean up the
instance variables? Or why don't we reinitialize them before running the
test?

I understand your grief. My solution: never use #initalize in tests, use #setUp to initialize instance variables.

Max




--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Reply | Threaded
Open this post in threaded view
|

Re: Weird SUnit/TestRunner behavior

Torsten Bergmann
In reply to this post by Max Leske
Attila wrote:
>why don't we reinitialize them before running the test?

SUnit does - but requires it to be in #setUp. One just has to follow the SUnit framework rules.

So if one implements

   MyTest>>initialize
       super initialize.
       s := OrderedCollection new. "note that this is initialized here not in the setUp as we normally do"

then this is not the problem or werd behavior of BabyMock, SUnit or Pharo.

It is a problem of the Test class (and of the one who implemented a test like this) as the developer did then not follow Sunit
framework rules and best practice patterns.


If you look into Simple Smalltalk Testing: With Patterns from Kent Beck (see http://swing.fit.cvut.cz/projects/stx/doc/online/english/tools/misc/testfram.htm)
it clearly says: "Override setUp to initialize the variables"

<quote>
    Design a test fixture.
     - Subclass TestCase
     - Add an instance variable for each known object in the fixture
     - Override setUp to initialize the variables
</quote>

One just have to stick to the rules - if not then for sure you might run into side effects or weird behavior.


Max wrote:
> I understand your grief. My solution: never use #initalize in tests, use
> #setUp to initialize instance variables.

Yes - I also wonder why you use #initialize instead of #setUp in a test? Also the expectation for someone to read or check
a test would be to look in #setUp, not in #initialize because of the SUnit standards.


So it is the way SUnit was defined, works and always worked (even unit testing in non Smalltalk solutions):

 - #setUp
     1. setup all nececessary thing BEFORE running or rerunning a test
     2. as test's could inherit from other (base) tests the #setUp method should always start with a "super setUp" call
 - #tearDown
     3. cleanup after running each test case
     4. as test's could inherit from each other and resources might need to be freed in reverse order (compared to allocation in #setup)
        the #tearDown should always end with a "super tearDown" call

I invested lot of time last year to clean up also tests for Pharo 7:
 - to follow SUnit standard
 - to introduce two new ruls (ShouldSendSuperSetUpAsFirstMessage and ShouldSendSuperTearDownAsLastMessage) so developers can see when they violate 2. or 3.
 - introduced a release test to keep this quality level (see "ProperlyImplementedSUnitClassesTest" checking for alignment with 2. and 3. for
   all tests in our base image)

All our tests follow this standard and use #setUp and if you now check using

    TestCase allSubclasses select: [:each |
        each methodDictionary keys includes: #initialize
    ]

the only test who uses #initialize is TimeMeasuringTest but for the sole reason to provide the possibility to measure time when debugging.
Which is fine and no violation of the definition and rules.

Bye
T.

Reply | Threaded
Open this post in threaded view
|

Re: Weird SUnit/TestRunner behavior

Attila Magyar
Thanks for the explanation Torsten. I do use setUp for initialization but one
exception is BabyMock2TesCase which serves as a base class for other tests.
The reason why its variables are created in the init is that the users would
likely forget sending a setUp to the super class in their own setUp. I'm
glad there are rules defined for this now. However I still have mixed
feelings about this. The usage of cleanUpInstanceVariables feels like a hack
and runCase feels broken from symmetrical point of view. If a convention
breaks something, it's not just a convention anymore and should be enforced
more strictly.



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html