Smalltalk › Squeak › Squeak - Dev

Re: [squeak-dev] Continuous integration (was: All tests running green)

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

8 messages Options

laurent laffont

Re: [squeak-dev] Continuous integration (was: All tests running green)

Hi,

I've started to work on a ContinuousIntegration package to periodically generate test reports for images provided by Damien Cassou (and more ?). You can find more info here: http://www.squeaksource.com/ContIntegration.html
To run all tests and generate a xml report per test case, type

CITestRunner runAllTests.

I've also made a bash script (squeakci) to run all the tests. It could later be run in a cron job for example: http://adhocmusic.org/squeak/squeakci

Here is a (basic) first report:
http://adhocmusic.org/squeak/squeak_test_report.html

It is not complete, as I have encountered some problems:

1. some code run by the tests have syntax errors. (for example: ChildrenToSiblingsTest>>#testModelChildrenToSibling). On a syntax error squeak open the debugger and the test run stop. To run all the tests, I prefer that errors like SyntaxError generates only a test error (only for continuous integration). So I've written an ugly hacky workaround in SyntaxError>>open

open: aSyntaxError
    "Answer a standard system view whose model is an instance of me."
    | topView |
    <primitive: 19> "Simulation guard"
    shouldOpenDebuggerOnError ifFalse: [ ^ false ].
    Smalltalk isMorphic ifTrue:
....

so before running CITestRunner, I write "SyntaxErrror openDebuggerOnError: false" to disable the openning of the debugger.
I think there must be a better way to do this, but as I'm a totally SmallTalk newbie ... idea ?

2. Some tests require user interaction with popups, ... I want these tests to generate an error when trying to open such dialogs, as unit tests should never require user interaction. How can I do this ?

3. PushUpMethodTest freezes squeak

Laurent Laffont

Rob Rothwell

Re: [squeak-dev] Continuous integration (was: All tests running green)

Nice idea...!

If you saved and categorized the errors each time you did this, you could do some long term data collection to allow developers to do some analysis around what is problematic.

At which point, your report could be more...graphical?!

I am currently trying to automate the output of daily Data Warehouse Integrations in a similar fashion, so I [think] I know where you are coming from...

Rob

On Sat, Jul 26, 2008 at 12:30 PM, laurent laffont <[hidden email]> wrote:

Hi,

I've started to work on a ContinuousIntegration package to periodically generate test reports for images provided by Damien Cassou (and more ?). You can find more info here: http://www.squeaksource.com/ContIntegration.html
To run all tests and generate a xml report per test case, type

CITestRunner runAllTests.

I've also made a bash script (squeakci) to run all the tests. It could later be run in a cron job for example: http://adhocmusic.org/squeak/squeakci

Here is a (basic) first report:
http://adhocmusic.org/squeak/squeak_test_report.html

It is not complete, as I have encountered some problems:

1. some code run by the tests have syntax errors. (for example: ChildrenToSiblingsTest>>#testModelChildrenToSibling). On a syntax error squeak open the debugger and the test run stop. To run all the tests, I prefer that errors like SyntaxError generates only a test error (only for continuous integration). So I've written an ugly hacky workaround in SyntaxError>>open

open: aSyntaxError
    "Answer a standard system view whose model is an instance of me."
    | topView |
    <primitive: 19> "Simulation guard"
    shouldOpenDebuggerOnError ifFalse: [ ^ false ].
    Smalltalk isMorphic ifTrue:
....

so before running CITestRunner, I write "SyntaxErrror openDebuggerOnError: false" to disable the openning of the debugger.
I think there must be a better way to do this, but as I'm a totally SmallTalk newbie ... idea ?

2. Some tests require user interaction with popups, ... I want these tests to generate an error when trying to open such dialogs, as unit tests should never require user interaction. How can I do this ?

3. PushUpMethodTest freezes squeak

Laurent Laffont

laurent laffont

Re: [squeak-dev] Continuous integration (was: All tests running green)

At which point, your report could be more...graphical?!

Yes, do you have some ideas ?

CITestReporter generate xml reports like this:

$ cat TEST-LocaleTest.xml

<testsuite errors="2" failures="0" passed="2">
<passed>
<pass>LocaleTest>>#testEncodingName {testing}</pass>
<pass>LocaleTest>>#testFontFullName {testing}</pass></passed>
<failures></failures>
<errors>
<error>LocaleTest>>#testLocaleChanged {testing}</error>
<error>LocaleTest>>#testIsFontAvailable {testing}</error></errors></testsuite>

so we just have to write a parser for a fancy xhtml output with css, ...

Laurent Laffont

keith1y

Re: [squeak-dev] Continuous integration (was: All tests running green)

In reply to this post by laurent laffont

Dear Laurent,

just to let you know there are currently several continuous integration
projects available and in the wings.

The CI tools is a specific goal for 3.11 which I have volunteered to
lead. However our CI strategy covers all squeak forks, so can be used by
anyone and everyone so it will not be limited to 3.11.

I have a tool in development called Bob. Bob builds images using
Installer scripts in combination with Sake/Packages and tests them using
TestReporter. The SUnit-improved package includes a TestReporter which
generates text files for the results. Damien has tested generating his
dev-images using Sake/Packages.

Your xml generating test runner would be the third or fourth time this
type of thing has been written for squeak. Ralph did one, and I did Bob
version 1.

If you are around in irc #squeak you could join myself and matthew and
could combine our goals/efforts

best regards

Keith

Tapple Gao

Re: [squeak-dev] Continuous integration (was: All tests running green)

In reply to this post by laurent laffont

On Sat, Jul 26, 2008 at 06:30:46PM +0200, laurent laffont wrote:
> 1. some code run by the tests have syntax errors. (for example:
> ChildrenToSiblingsTest>>#testModelChildrenToSibling).

catch the error and do nothing? Here is the code:

[
"some code"
] on: SyntaxErrorNotification do: [
"nothing"
]

> 2. Some tests require user interaction with popups, ... I want these
> tests to generate an error when trying to open such dialogs, as unit tests
> should never require user interaction. How can I do this ?

[] on: ProvideAnswerNotification do: [self error]

To see where those notifications are used, highlight
"ProvideAnswerNotification" and press Alt-Shift-N (find
references)

--
Matthew Fulmer -- http://mtfulmer.wordpress.com/

cedreek

Re: [squeak-dev] Continuous integration (was: All tests running green)

In reply to this post by keith1y

Hi,

just as a side note, Laurent chose this project to learn squeak. I think the goal was for Dev images, so as to have a report for tests. I think he based his work on Ralph one...

My 2 cents ;)

Cédrick

Dear Laurent,

just to let you know there are currently several continuous integration projects available and in the wings.

The CI tools is a specific goal for 3.11 which I have volunteered to lead. However our CI strategy covers all squeak forks, so can be used by anyone and everyone so it will not be limited to 3.11.

I have a tool in development called Bob. Bob builds images using Installer scripts in combination with Sake/Packages and tests them using TestReporter. The SUnit-improved package includes a TestReporter which generates text files for the results. Damien has tested generating his dev-images using Sake/Packages.

Your xml generating test runner would be the third or fourth time this type of thing has been written for squeak. Ralph did one, and I did Bob version 1.

If you are around in irc #squeak you could join myself and matthew and could combine our goals/efforts

best regards

Keith

laurent laffont

Re: [squeak-dev] Continuous integration (was: All tests running green)

In reply to this post by Tapple Gao

catch the error and do nothing? Here is the code:

[
"some code"
] on: SyntaxErrorNotification do: [
"nothing"

]

> 2. Some tests require user interaction with popups, ... I want these
> tests to generate an error when trying to open such dialogs, as unit tests
> should never require user interaction. How can I do this ?

[] on: ProvideAnswerNotification do: [self error]

To see where those notifications are used, highlight
"ProvideAnswerNotification" and press Alt-Shift-N (find
references)

--
Matthew Fulmer -- http://mtfulmer.wordpress.com/

Thank you !

Laurent Laffont

laurent laffont

Re: [squeak-dev] Continuous integration (was: All tests running green)

In reply to this post by keith1y

The CI tools is a specific goal for 3.11 which I have volunteered to lead. However our CI strategy covers all squeak forks, so can be used by anyone and everyone so it will not be limited to 3.11.

Is it possible to use it in 3.10 as Damien needs reports for 3.10 ?

Your xml generating test runner would be the third or fourth time this type of thing has been written for squeak. Ralph did one, and I did Bob version 1.

OK, I will take a look at Bob , thanks.

Laurent Laffont.