Smalltalk › Squeak › Squeak VM

OSProcess fork issue with Debian built VM

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

44 messages Options

123

Luke Gorrie

Re: OSProcess fork issue with Debian built VM

Hi Eliot,

Sorry about the slow reply -

On 11 July 2017 at 21:44, Eliot Miranda <[hidden email]> wrote:

Tests that stress the VM:
- Running the complete test suite
- running update to Squeak trunk tip from Squeak 5.0
- recompiling the entire system
- building a Moose image

Having these run automatically by a CI server would be very nice.

What is the success criteria for the tests? Or is it just that the VM doesn't crash?

Are there scripts anywhere for running these tests and checking the results that could be used for reference?

Once upon a time I included the built-in test suite in my build but it didn't have 100% success rate and I was not sure whether that was a problem and how to make a suitable pass/fail decision.

Eliot Miranda-2

Re: OSProcess fork issue with Debian built VM

Hi Luke.

On Jul 24, 2017, at 10:04 AM, Luke Gorrie <[hidden email]> wrote:

Hi Eliot,

Sorry about the slow reply -

On 11 July 2017 at 21:44, Eliot Miranda <[hidden email]> wrote:
Tests that stress the VM:
- Running the complete test suite
- running update to Squeak trunk tip from Squeak 5.0
- recompiling the entire system
- building a Moose image

Having these run automatically by a CI server would be very nice.

What is the success criteria for the tests?

The success criterion is that they all succeed. By extension this implies that the vm doesn't crash.

Or is it just that the VM doesn't crash?

Are there scripts anywhere for running these tests and checking the results that could be used for reference?

Ask on the list of folks like Fabio Niephaus.

Once upon a time I included the built-in test suite in my build but it didn't have 100% success rate and I was not sure whether that was a problem and how to make a suitable pass/fail decision.

Luke Gorrie

Re: OSProcess fork issue with Debian built VM

On 29 July 2017 at 18:31, Eliot Miranda <[hidden email]> wrote:

The success criterion is that they all succeed. By extension this implies that the vm doesn't crash.

Does this mean that the Test Runner should report 100% pass rate? I have never seen that even with the stock image+vm from pharo.org. So I have a bootstrapping problem if I don't have a baseline that passes the CI test.

Just now I ran - randomly - a clean Pharo 5.0 image that I have on my Mac and the result I see is: 9098 run, 7408 passes, 8 skipped, 81 expected failures, 22 failures, 587 errors, 0 unexpected passes. If resolving those 22 failures and 587 errors is a blocker for running a CI then that is quite an obstacle from my perspective.

are other people really seeing 100% pass rates?

alistairgrant

Re: OSProcess fork issue with Debian built VM

On Thu, Aug 03, 2017 at 09:47:49AM +0200, Luke Gorrie wrote:

>
> On 29 July 2017 at 18:31, Eliot Miranda <[hidden email]> wrote:
>
> The success criterion is that they all succeed. By extension this implies
> that the vm doesn't crash.
>
>
> Does this mean that the Test Runner should report 100% pass rate? I have never
> seen that even with the stock image+vm from pharo.org. So I have a
> bootstrapping problem if I don't have a baseline that passes the CI test.
>
> Just now I ran - randomly - a clean Pharo 5.0 image that I have on my Mac and
> the result I see is: 9098 run, 7408 passes, 8 skipped, 81 expected failures, 22
> failures, 587 errors, 0 unexpected passes. If resolving those 22 failures and
> 587 errors is a blocker for running a CI then that is quite an obstacle from my
> perspective.
>
> are other people really seeing 100% pass rates?

Nope, I don't remember ever seeing a 100% pass rate for the entire test
suite. (I don't know how this is handled in CI environments)

Cheers,
Alistair

123