SUnit

Niko and me are thinking about doing some work on SUnit.

Background is that we are working on Phexample, which extends SUnit with test
 that expand on one another [1]. We got test dependencies working with the
 current SUnit, but had to do some quick ugly hacks. So we thought about
 cleaning the internals SUnit up, and then include test dependencies in the
 core.

Things we would possible change (we will certainly identify more as we clean
 up SUnit...)

- store test resources in a dynamic variable rather than a static one
- refactor the assert: protocol to a trait, such that they can be used in
 setup and teardown of test resources as well (actually I already did that,
 you can find a version on my squeaksource account)
- apply the fixes to test resources in Niall's slide deck that Lukas showed us
- add expectations matchers as used in behavior-driven testing
- break testcase into two classes, and to run the test and one to store the
 result (no more #cleanUpInstanceVariables hack to avoid memory leaks)
- add test dependencies to test results, such that one test can extend on the
 return value of another test
- fix expected failures such that they are a first-class result (no more
 #shouldPass with O(n^2) or worse behavior)

So you see, the idea is twofold: first to clean up what is there, and second
 to add test dependencies as a new feature. For those that dont know test
 dependencies. They had first been proposed by Markus Gälli based on an
 analysis of all Squeak tests years ago. He found that most tests cover a
 superset of another test's coverage set. A prototype in Java as been
 implemented later by Lea Hänsenberger and me. We published a case study at
 XP 2008, which lead to PHP folks to include test dependencies in the latest
 PHPUnit. The idea of explicit test dependencies is to shift the burden of
 isolating test cases from the developers to the framework. You avoid
 duplicate test code and you get less red test cases per failure. Please
 refer to Niko's blog post for more details [2].

We got some more ideas (such as API baseline testing, search in test results,
 etc...) but they will not likely make it into a first clean up.

One thing we do not know is how many folks are out there that depend on
 internal representation of SUnit. I talked to some folks and identified two
 requirements, first that legacy tests should keep running and second that
 contributions to other sunit forks should keep being mergeable. The first
 seems feasible, the second sounds like it boils down to not cleaning
 anything :)

What do you think?

cheers,
AA

[1]: http://www.squeaksource.com/phexample.html
[2]: http://smalltalkthoughts.blogspot.com/2009/11/phexample-because-examples-expand-on.html


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi adrian

I like the idea of cleaning. Now here is what I would like
        - get the same API (I think that sharing code amongst smalltalk is a too large constraint but
        having the same API is important)
        - integrate key features of extensions so that SUNIT and SUNITExtended is cooler.


> - store test resources in a dynamic variable rather than a static one

for this one I would not rely on dynamic variables. Keep it for later.
And you will have to really argue. Static scoping is nice.

> - refactor the assert: protocol to a trait, such that they can be used in
> setup and teardown of test resources as well (actually I already did that,
> you can find a version on my squeaksource account)

:) Easy I'm biaised :)

> - apply the fixes to test resources in Niall's slide deck that Lukas showed us

Pleaseeeeeeeee.
I would really like to also
        - check what keith did on his Sunit extensions
        - have expected failures
        - have the possibility to know whether a test was red or green when it was saved and
        now when I run it, is it my fault or what is just before like that

> - add expectations matchers as used in behavior-driven testing
Please not keep this separated or in another package.
I do not like this
        "bla be something" kind of violation of law of demeter

> - break testcase into two classes, and to run the test and one to store the
> result (no more #cleanUpInstanceVariables hack to avoid memory leaks)

what would be the impact on existing subclasses?

> - add test dependencies to test results, such that one test can extend on the
> return value of another test
I originally did not like the idea but I can be convinced easily :)
I think that if it can help other without getting in our way then this is cool.


Yes clean up first!


> One thing we do not know is how many folks are out there that depend on
> internal representation of SUnit. I talked to some folks and identified two
> requirements, first that legacy tests should keep running and second that
> contributions to other sunit forks should keep being mergeable. The first
> seems feasible, the second sounds like it boils down to not cleaning
> anything :)

Merging does not mean that this is done automatically.
I think that having a good testing framework is important.
Now good does not mean that it has a lot of not that useful features.
I suggest the following
        - clean and get a strong core
        - make sure that all the tests still run
        - get a set of extensions that people can load

> What do you think?

I'm interested :) but people will have to discuss your proposal :)

Stef

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Mon, Dec 21, 2009 at 8:06 AM, Adrian Kuhn <[hidden email]> wrote:
> One thing we do not know is how many folks are out there that depend on
>  internal representation of SUnit. I talked to some folks and identified two
>  requirements, first that legacy tests should keep running and second that
>  contributions to other sunit forks should keep being mergeable. The first
>  seems feasible, the second sounds like it boils down to not cleaning
>  anything :)

Well, the second may require working *with* SUnit maintainers on other
platforms. :) I guess I don't know exactly what you define as
"internal representation" but from Seaside's point of view it is
absolutely essential that unit tests be runnable the same way on all
platforms - it's the cornerstone of portability. The fact that we
can't write lint rules that work on all platforms is already sad
enough.

So if you're going to do anything that breaks compatibility between
platforms, please rename it and make sure both are loadable in the
same system (and we, at least, won't be able to use it).

It would be really nice to have *one* SUnit that worked everywhere
with only minor platform-dependent pieces (like Seaside) where
necessary. This would be much more sustainable than the forked
situation we have currently, but that's a whole different battle.
Although now I've said it, why not just start from scratch? There's
not a lot of code there... implement something that works the way you
want, build it using Grease/Slime so we can port it easily to
everywhere else, and that's something Seaside could use (assuming it
was compellingly better). And suddenly we'd have a test framework that
*did* run everywhere with a common code base.

my 2c,

Julian

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Thanks julian. I think that this is nice way to look at it.

Defining another one would be good - the only important point is that we could load it and
it can transparently replace SUnit.

BTW I do not know what is the status of the recent SUnit changes made by Nial
I know that at ESUG they published a squeak version so it would be good to start there.
Stef



_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi,
_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Mon, Dec 21, 2009 at 9:36 AM, Stéphane Ducasse
<[hidden email]> wrote:
> Thanks julian. I think that this is nice way to look at it.
>
> Defining another one would be good - the only important point is that we could load it and
> it can transparently replace SUnit.

Yes, it would be nice if it still supported the same basic protocol
(#assert:, #deny:, etc) so you could simply change the superclass of
your test cases and things would (at least more or less) work.

Julian

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

> BTW I do not know what is the status of the recent SUnit changes made by Nial
> I know that at ESUG they published a squeak version so it would be good to start there.

I tries to merge that code, but more than half of the code-base is a
conflict. The SUnit framework in Squeak/Pharo has seen an uncountable
number of tiny changes that prevents a strait forward merge with the
official framework.

Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

laurent laffont <laurent.laffont@...> writes:

> Hi,
>
> Isn't it similar to Thoughbot shoulda's nested contexts ?
> http://github.com/thoughtbot/shoulda

Yes it is, and no.

Nested contexts from a tree, while test dependencies may possibly form a graph.
 At least when I last time looked at Shoulda and friends, the context were
 limited to a tree. A context can only be nested in at most one enclosing
 context. While with Phexample and friends, examples are nodes in a graph and
 may depend on as many other examples as required.

cheers,
AA


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Julian Fitzell <jfitzell@...> writes:

> Well, the second may require working *with* SUnit maintainers on other
> platforms. :)

Where can I find their names?

> I guess I don't know exactly what you define as
> "internal representation" but from Seaside's point of view it is
> absolutely essential that unit tests be runnable the same way on all
> platforms - it's the cornerstone of portability.

Good point. Added as a requirement: current test should remain runnable, and
 it should be possible to limit oneself to a set features that runs on all
 platforms.

One problem I see with legacy tests though is that TestCase serves currently
 two purposes: a the superclass of all classes that contain tests, and as the
 implementation of the internal logic that runs the tests. I don't how common
 it is to override internal methods of test case in subclass (as eg #runCase
 etc). In Seaside, eg, are you doing this?

> Although now I've said it, why not just start from scratch? There's
> not a lot of code there... implement something that works the way you
> want, build it using Grease/Slime so we can port it easily to
> everywhere else, and that's something Seaside could use (assuming it
> was compellingly better). And suddenly we'd have a test framework that
> *did* run everywhere with a common code base.

Compelling, but can this be done without being a cross-platform expert? I
 recall the pain of getting Fame to run in both VW and Squeak, eventually I
 gave up. Cross-platform integration server, anyone?

What is Grease/Slime?

--AA



_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Stéphane Ducasse <stephane.ducasse@...> writes:

>  BTW I do not know what is the status of the recent SUnit changes made by
> Nial. I know that at ESUG they published a squeak version so it would be
> good to start there.

Where can I find these changes?

--AA




_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Stéphane Ducasse <stephane.ducasse@...> writes:

> - get the same API (I think that sharing code amongst smalltalk is a too
> large constraint but having the same API is important)

Yes, legacy tests should keep running.

On the other hand, what is an API in Smalltalk? We just got methods, and they
 are *all* public. So when cleaning SUnit we have to draw a line. No doubt,
 #assert: and friends are API, but what about eg #runCase?

> - integrate key features of extensions so that SUNIT and SUNITExtended is
> cooler.

Is "SUNITExtended" a project for Pharo?

> - check what keith did on his Sunit extensions

Where can I find those? Which keith, email?

> - have expected failures
> - have the possibility to know whether a test was red or green when it was
> saved and now when I run it, is it my fault or what is just before like that

+1 ... no, +100 :)

> > - add expectations matchers as used in behavior-driven testing
> Please not keep this separated or in another package.

Users should be free to choose between assertions and expectations. Assertions
 will stay.

> > - break testcase into two classes, and to run the test and one to store the
> > result (no more #cleanUpInstanceVariables hack to avoid memory leaks)
>
> what would be the impact on existing subclasses?

There should be none.

Tools however might have to be changed, since a test result will contain test
 descriptions rather than zombified test cases.

> I suggest the following
> - clean and get a strong core
> - make sure that all the tests still run
> - get a set of extensions that people can load

To me, the effort of cleaning up Sunit is worthwhile if we get a strong core
 with support for test dependencies in the core. Phexample as an extension that
 people can load is what we have now. What I imagine is a simple core with
 clearly defined API, that allows other to write their own runners for their
 test classes. For example, I would like to separate the role of running tests
 and of containing tests (see my comment on class TestCase in the reply to
 Julien.) This should allow better extensibility for everyone.

--AA


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Mon, Dec 21, 2009 at 4:15 PM, Adrian Kuhn <[hidden email]> wrote:
> Julian Fitzell <jfitzell@...> writes:
>
>> Well, the second may require working *with* SUnit maintainers on other
>> platforms. :)
>
> Where can I find their names?

Dunno... guess this is a start: http://sunit.sourceforge.net/people.htm

It seems we do not, though we do override #performTest in two cases.
Both are cases where we need to wrap the execution in an exception
handler.

I don't know - we've become pseudo-experts for Seaside but it doesn't
seem to be that tough in our case - just a matter of avoiding lots of
things that aren't compatible.

Grease is the light-weight compatibility layer we have ended up
developing out of the Seaside project. I also used it to port Magritte
and Pier to VA Smalltalk and Colin and I are now beginning to use it
for MC2.

Slime is a set of lint rules - currently some of them are
Seaside-specific but others are essentially a counterpart to Grease -
detecting things that you cannot do if you want your code to be
portable.

The hope is that others can leverage the experience we've developed
and our porters' efforts getting Grease running on their platforms. I
still need to finish a write-up on what Grease is and post it
somewhere - on my todo list for the holidays.

Julian

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

But this is why I was talking about API spec sharing vs code sharing.

Stef

On Dec 21, 2009, at 10:09 PM, Lukas Renggli wrote:


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

> - get the same API (I think that sharing code amongst smalltalk is a too
>> large constraint but having the same API is important)
>
> Yes, legacy tests should keep running.
>
> On the other hand, what is an API in Smalltalk? We just got methods, and they
> are *all* public. So when cleaning SUnit we have to draw a line. No doubt,
> #assert: and friends are API, but what about eg #runCase?

you cut some line.
>
>> - integrate key features of extensions so that SUNIT and SUNITExtended is
>> cooler.
>
> Is "SUNITExtended" a project for Pharo?

yes for example.

>> - check what keith did on his Sunit extensions
>
> Where can I find those? Which keith, email?

keith hodges

For me expectations are already implemented in SSpec and keith merged them in his
SUnit extensions and nobody used them and after he was not happy.

So keep the core small concentrate on what is the key aspect
we do not want a bloated SUnit. So tests dependencies seem important
while expectations someextra features.

We do not want NGUnit where you can also do the washing.


ok

>
> --AA
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Dec 22, 2009, at 1:16 AM, Adrian Kuhn wrote:

> Stéphane Ducasse <stephane.ducasse@...> writes:
>
>> BTW I do not know what is the status of the recent SUnit changes made by
>> Nial. I know that at ESUG they published a squeak version so it would be
>> good to start there.
>
> Where can I find these changes?
>
Niall Ross <[hidden email]>
they published them on squeaksource but I do not know where

> --AA
>
>
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Dec 22, 2009, at 4:34 AM, Julian Fitzell wrote:

> On Mon, Dec 21, 2009 at 4:15 PM, Adrian Kuhn <[hidden email]> wrote:
>> Julian Fitzell <jfitzell@...> writes:
>>
>>> Well, the second may require working *with* SUnit maintainers on other
>>> platforms. :)
>>
>> Where can I find their names?
>
> Dunno... guess this is a start: http://sunit.sourceforge.net/people.htm

I would not go down this road :)

Just define a new one that is compatible.



_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

>> On the other hand, what is an API in Smalltalk? We just got methods, and they
>> are *all* public. So when cleaning SUnit we have to draw a line. No doubt,
>> #assert: and friends are API, but what about eg #runCase?
>
> you cut some line.

I know a lot of SUnit tests override #runCase. Basically all tests
that need to setup a dynamic environment (e.g. to simulate a session).

Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Lukas Renggli <renggli@...> writes:

> I know a lot of SUnit tests override #runCase. Basically all tests
> that need to setup a dynamic environment (e.g. to simulate a session).

For cases such as `Whatever while: [ super runCase ]` that cannot be split into
 setUp and tearDown method. Good point, added to requirements list.

--AA


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Mariano Martinez Peck <marianopeck@...> writes:

> I think it is this:   http://www.squeaksource.com/Testing

53 tests, hmm...

> http://wiki.squeak.org/squeak/SUnit%20for%203.10

28 tests, I am ... underwhelmed.

> And there is also some extensions made by Hernán Wilkinson.

Looks they are already in Pharo, some under other names.

Thanks.

--AA


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project