SUnit improvements need review and feedback

Denis

can we also have

assert: aNumber withPrecision: precision equals: otherNumber

	self 
	assert: (aNumber round: precision) 
	equals: otherNumber

assert: aNumber closeTo: otherNumber

	assert: aNumber withPrecision: self defaultPrecision equals: otherNumber

defaultPrecision
	^ 2

Hi.

I am working on SUnit improvements. I open issue 19015. Slice is inbox which waits your review and feedback.

I was trying to address three problems:

1) Tests should never hang. They should be always executed within time limit.

 

I give them 10 minutes for now to not change existing behaviour of tests. At next step it should be  really reduced to ~100 milliseconds (?). 

Any TestCase could redefine time limit by method #defaultTimeLimit.

Or it could be specified directly in test code by 

self timeLimit: 10 seconds

(could be changed at any time of test execution).

To implement this logic special watch dog process is running for given test suite to control execution time of tests. It is single process for full test suite.

2) When test completes all forked processes should be terminated. 

If your tested code produced zombie processes SUnit will take care about destroying all such garbage. 

(it not means that you don't need to clean table in #tearDown but any code could be broken and running tests should not produce dirty system).

3) Any failures inside forked processes should not spawn debugger while running tests.

Only when we debug tests we need debugger on forked failed processes.

During normal run SUnit should prevent such "background debuggers" and mark such tests as failed.

To implement this behaviour SUnit will handle errors from forked processes by suspending them and collecting them in special collection. 

I introduce TestFailedByForkedProcess error to signal these kind of problems at the end of tests. This error is resumable and resume will opens debuggers of suspended failures (in fact it will resume suspended processes). 

So to debug background failures you will need extra Proceed action on debugger when TestFailedByForkedProcess is signalled.

But in normal run such tests will be just failed by TestFailedByForkedProcess error.

Now details on how it is done:

I introduce special process specific variable CurrentExecutionEnvironment. It is not installed by default and as default value it returns DefaultExecutionEnvironment instance.

This variable is inheritable. If your install concrete environment into process it will be installed to any child process. 

So value of variable is instance of ExecutionEnvironment subclasses and you can install it by:

anYourExecutionEnvironment beActiveDuring: aBlock

When block completes previous environment is restored.

For default environment there is class side method:

DefaultExecutionEnvironment beActiveDuring: aBlock

And to reset current environment to default:

DefaultExecutionEnvironment beActive.

SUnit introduces TestExecutionEnvironment which implements all described behaviour for time limits and forked processes.

To activate environment there is new method #runCaseManaged. And submitted slice uses it instead of simple runCase.

TestCase>>runCaseManaged

CurrentExecutionEnvironment runTestCase: self

DefaultExecutionEnvironment will install new TestExecutionEnvironment and delegate processing to it. And if TestExecutionEnvironment is already active it will just process new test case. 

Now monkey has problem in checking slice (annoying timeout for loading). 

So I can't see real system impact. But it should not stop you from review :)) 

I think it is very important features for all our tests. And environment idea will lead to very interesting future.

Best regards,

Denis

Denis Kudriashov wrote

Super!!! It's integrated (In 60201). Thank's integrators.

Now we need correct default time limit and mark long tests as long. I
opened new issue 19035
<https://pharo.fogbugz.com/f/cases/19035/Default-time-limit-for-tests-should-be-small>
.
What you think about default value? I would use 200 milliseconds.

2016-08-31 14:10 GMT+02:00 Denis Kudriashov <[hidden email]>:

> Hi.
>
> I am working on SUnit improvements. I open issue 19015
> <https://pharo.fogbugz.com/f/cases/19015/Tests-should-never-hang-and-leave-forked-processes-live>.
> Slice is inbox which waits your review and feedback.
> I was trying to address three problems:
>
> *1) Tests should never hang. They should be always executed within time
> limit.*
>
> I give them 10 minutes for now to not change existing behaviour of tests.
> At next step it should be  really reduced to ~100 milliseconds (?).
> Any TestCase could redefine time limit by method #defaultTimeLimit.
> Or it could be specified directly in test code by
> self timeLimit: 10 seconds
> (could be changed at any time of test execution).
>
> To implement this logic special watch dog process is running for given
> test suite to control execution time of tests. It is single process for
> full test suite.
>
> *2) When test completes all forked processes should be terminated.*
>
> If your tested code produced zombie processes SUnit will take care about
> destroying all such garbage.
> (it not means that you don't need to clean table in #tearDown but any code
> could be broken and running tests should not produce dirty system).
>
> *3) Any failures inside forked processes should not spawn debugger while
> running tests.*
> Only when we debug tests we need debugger on forked failed processes.
> During normal run SUnit should prevent such "background debuggers" and
> mark such tests as failed.
>
> To implement this behaviour SUnit will handle errors from forked processes
> by suspending them and collecting them in special collection.
> I introduce TestFailedByForkedProcess error to signal these kind of
> problems at the end of tests. This error is resumable and resume will opens
> debuggers of suspended failures (in fact it will resume suspended
> processes).
> So to debug background failures you will need extra Proceed action on
> debugger when TestFailedByForkedProcess is signalled.
> But in normal run such tests will be just failed by
> TestFailedByForkedProcess error.
>
> *Now details on how it is done:*
>
> I introduce special process specific variable CurrentExecutionEnvironment.
> It is not installed by default and as default value it returns
> DefaultExecutionEnvironment instance.
> This variable is inheritable. If your install concrete environment into
> process it will be installed to any child process.
>
> So value of variable is instance of ExecutionEnvironment subclasses and
> you can install it by:
>
> anYourExecutionEnvironment beActiveDuring: aBlock
>
>
> When block completes previous environment is restored.
> For default environment there is class side method:
>
> DefaultExecutionEnvironment beActiveDuring: aBlock
>
>
> And to reset current environment to default:
>
> DefaultExecutionEnvironment beActive.
>
>
> SUnit introduces TestExecutionEnvironment which implements all described
> behaviour for time limits and forked processes.
> To activate environment there is new method #runCaseManaged. And submitted
> slice uses it instead of simple runCase.
>
> TestCase>>runCaseManaged
> CurrentExecutionEnvironment runTestCase: self
>
>
> DefaultExecutionEnvironment will install new TestExecutionEnvironment and
> delegate processing to it. And if TestExecutionEnvironment is already
> active it will just process new test case.
>
> Now monkey has problem in checking slice (annoying timeout for loading).
> So I can't see real system impact. But it should not stop you from review
> :))
> I think it is very important features for all our tests. And environment
> idea will lead to very interesting future.
>
> Best regards,
> Denis
>

Paul DeBruicker wrote

How do you propose raising the limit for a specific test would go?  

Pragma?

Denis Kudriashov wrote

Super!!! It's integrated (In 60201). Thank's integrators.

Now we need correct default time limit and mark long tests as long. I
opened new issue 19035
<https://pharo.fogbugz.com/f/cases/19035/Default-time-limit-for-tests-should-be-small>
.
What you think about default value? I would use 200 milliseconds.

2016-08-31 14:10 GMT+02:00 Denis Kudriashov <[hidden email]>:

> Hi.
>
> I am working on SUnit improvements. I open issue 19015
> <https://pharo.fogbugz.com/f/cases/19015/Tests-should-never-hang-and-leave-forked-processes-live>.
> Slice is inbox which waits your review and feedback.
> I was trying to address three problems:
>
> *1) Tests should never hang. They should be always executed within time
> limit.*
>
> I give them 10 minutes for now to not change existing behaviour of tests.
> At next step it should be  really reduced to ~100 milliseconds (?).
> Any TestCase could redefine time limit by method #defaultTimeLimit.
> Or it could be specified directly in test code by
> self timeLimit: 10 seconds
> (could be changed at any time of test execution).
>
> To implement this logic special watch dog process is running for given
> test suite to control execution time of tests. It is single process for
> full test suite.
>
> *2) When test completes all forked processes should be terminated.*
>
> If your tested code produced zombie processes SUnit will take care about
> destroying all such garbage.
> (it not means that you don't need to clean table in #tearDown but any code
> could be broken and running tests should not produce dirty system).
>
> *3) Any failures inside forked processes should not spawn debugger while
> running tests.*
> Only when we debug tests we need debugger on forked failed processes.
> During normal run SUnit should prevent such "background debuggers" and
> mark such tests as failed.
>
> To implement this behaviour SUnit will handle errors from forked processes
> by suspending them and collecting them in special collection.
> I introduce TestFailedByForkedProcess error to signal these kind of
> problems at the end of tests. This error is resumable and resume will opens
> debuggers of suspended failures (in fact it will resume suspended
> processes).
> So to debug background failures you will need extra Proceed action on
> debugger when TestFailedByForkedProcess is signalled.
> But in normal run such tests will be just failed by
> TestFailedByForkedProcess error.
>
> *Now details on how it is done:*
>
> I introduce special process specific variable CurrentExecutionEnvironment.
> It is not installed by default and as default value it returns
> DefaultExecutionEnvironment instance.
> This variable is inheritable. If your install concrete environment into
> process it will be installed to any child process.
>
> So value of variable is instance of ExecutionEnvironment subclasses and
> you can install it by:
>
> anYourExecutionEnvironment beActiveDuring: aBlock
>
>
> When block completes previous environment is restored.
> For default environment there is class side method:
>
> DefaultExecutionEnvironment beActiveDuring: aBlock
>
>
> And to reset current environment to default:
>
> DefaultExecutionEnvironment beActive.
>
>
> SUnit introduces TestExecutionEnvironment which implements all described
> behaviour for time limits and forked processes.
> To activate environment there is new method #runCaseManaged. And submitted
> slice uses it instead of simple runCase.
>
> TestCase>>runCaseManaged
> CurrentExecutionEnvironment runTestCase: self
>
>
> DefaultExecutionEnvironment will install new TestExecutionEnvironment and
> delegate processing to it. And if TestExecutionEnvironment is already
> active it will just process new test case.
>
> Now monkey has problem in checking slice (annoying timeout for loading).
> So I can't see real system impact. But it should not stop you from review
> :))
> I think it is very important features for all our tests. And environment
> idea will lead to very interesting future.
>
> Best regards,
> Denis
>

Nicolas I like your ideas can you propose (code) something?

Stef

For floating points it would be good to also have something related to unit of least precision like I think it exists in google tests:

assert: aFloat isWithin: anInteger ulpFrom: anotherFloat

    ^(aFloat - anotherFloat) abs <= (anotherFloat ulp max: aFloat ulp)

It's testing if the result are the same with a tolerance on least significant bits.
(anotherFloat ulp max: aFloat ulp) is not really necessary, it's only for making the comparison symmetric.

indeed, 2 and 2 predecessor do not have the same exponent, so do not have the same ulp.

If we don't have symmetry, then

    self assert: 2.0 predecessor predecessor isWithin: 1 ulpFrom: 2.0.
might differ from:
    self assert: 2.0 isWithin: 1 ulpFrom: 2.0 predecessor predecessor .

Note that the this message cannot be used to test for zero result (because 0.0 ulp is the smallest denormal).

For example:

    self assert: 2.0 - 2.0 predecessor isWithin: 4 ulpFrom: 0.0.

would anwer false, but it makes no sense!

One should write

    self assert: 2.0 isWithin: 4 ulpFrom: 2.0 predecessor.

2016-09-03 9:22 GMT+02:00 stepharo <[hidden email]>:

Denis

can we also have

assert: aNumber withPrecision: precision equals: otherNumber

	self 
	assert: (aNumber round: precision) 
	equals: otherNumber

assert: aNumber closeTo: otherNumber

	assert: aNumber withPrecision: self defaultPrecision equals: otherNumber

defaultPrecision
	^ 2

Le 31/8/16 à 14:10, Denis Kudriashov a écrit :

Hi.

I am working on SUnit improvements. I open issue 19015. Slice is inbox which waits your review and feedback.

I was trying to address three problems:

1) Tests should never hang. They should be always executed within time limit.

 

I give them 10 minutes for now to not change existing behaviour of tests. At next step it should be  really reduced to ~100 milliseconds (?). 

Any TestCase could redefine time limit by method #defaultTimeLimit.

Or it could be specified directly in test code by 

self timeLimit: 10 seconds

(could be changed at any time of test execution).

would be good

To implement this logic special watch dog process is running for given test suite to control execution time of tests. It is single process for full test suite.

2) When test completes all forked processes should be terminated. 

If your tested code produced zombie processes SUnit will take care about destroying all such garbage. 

(it not means that you don't need to clean table in #tearDown but any code could be broken and running tests should not produce dirty system).

Yes!!!

3) Any failures inside forked processes should not spawn debugger while running tests.

Only when we debug tests we need debugger on forked failed processes.

During normal run SUnit should prevent such "background debuggers" and mark such tests as failed.

Definitevely. Now when we execute tests from the testRunner and nautilus we should get the debugger, right?

To implement this behaviour SUnit will handle errors from forked processes by suspending them and collecting them in special collection. 

I introduce TestFailedByForkedProcess error to signal these kind of problems at the end of tests. This error is resumable and resume will opens debuggers of suspended failures (in fact it will resume suspended processes). 

So to debug background failures you will need extra Proceed action on debugger when TestFailedByForkedProcess is signalled.

But in normal run such tests will be just failed by TestFailedByForkedProcess error.

We should really write a little doc so that I can add it to the Sunit Chapter.

Now details on how it is done:

I introduce special process specific variable CurrentExecutionEnvironment. It is not installed by default and as default value it returns DefaultExecutionEnvironment instance.

This variable is inheritable. If your install concrete environment into process it will be installed to any child process. 

So value of variable is instance of ExecutionEnvironment subclasses and you can install it by:

anYourExecutionEnvironment beActiveDuring: aBlock

When block completes previous environment is restored.

For default environment there is class side method:

DefaultExecutionEnvironment beActiveDuring: aBlock

And to reset current environment to default:

DefaultExecutionEnvironment beActive.

SUnit introduces TestExecutionEnvironment which implements all described behaviour for time limits and forked processes.

To activate environment there is new method #runCaseManaged. And submitted slice uses it instead of simple runCase.

TestCase>>runCaseManaged

CurrentExecutionEnvironment runTestCase: self

DefaultExecutionEnvironment will install new TestExecutionEnvironment and delegate processing to it. And if TestExecutionEnvironment is already active it will just process new test case. 

Now monkey has problem in checking slice (annoying timeout for loading). 

So I can't see real system impact. But it should not stop you from review :)) 

I think it is very important features for all our tests. And environment idea will lead to very interesting future.

Indeed!

Best regards,

Denis

2016-09-03 9:22 GMT+02:00 stepharo <[hidden email]>:

Denis

can we also have

assert: aNumber withPrecision: precision equals: otherNumber

	self 
	assert: (aNumber round: precision) 
	equals: otherNumber

assert: aNumber closeTo: otherNumber

	assert: aNumber withPrecision: self defaultPrecision equals: otherNumber

defaultPrecision
	^ 2

We could add it but as different issue.

Also I really want move away from assertions to should expressions. It is much nicer to write things like this (which available in StateSpecs):

10.1234 should equal: 10.12 within: 0.01  "success"

10.1234 should equal: 10.12 within: 0.001 "failure"

Also I has idea that float assertions should always use default precision (like 0.00000000001) without special message.

(80.85 * 100) should equal: 8085      

It will fail now but I want it always succeed in such cases. Because in most cases users don't care about this. And when it has real value users will specify concrete precision

argh!!!!!!

May be we should revert the change.

And make the changes optional.

ARgh, since this got integrated I keep crashing images...

I'm just executing some OSSubprocess like this:

process := OSSUnixSubprocess new

command: 'ls';

redirectStdout;

runAndWaitOnExitDo: [ :command :outString | ^ outString ].

and my image crashes with a:

(scheduler could not find a runnable process)

So this is messing up badly the scheduler. We have to fix this :/

On Sun, Sep 4, 2016 at 9:58 AM, stepharo <[hidden email]> wrote:

And it is worth breaking all the book chapters and videos.
I think that assert: closeTo: kind of solution is
    - simple
    - regular with the other assert:
    - incremental

So maximum quality.

Stef

Guille, 

Just for the record, there has been some crashes in Pharo using OSSubprocess, but I (we) were never able to reproduce nor found the cause, and even less a fix :(

So....any help in this direction is really appreciated as I want OSSubprocess to be robust. 

Cheers, 

On Mon, Sep 5, 2016 at 2:53 PM, Guillermo Polito <[hidden email]> wrote:

I mean crash.

Somehow my image's processes were killed and I only had the UI process: no delay process, no finalization, no idle process. Then, when the VM tries to suspend the current process and change to a second one, it finds none and it just quits.

So it's not a VM problem I think but mostly something in the image.

And in any case, now that I'm with a colder head (because I was mad after losing a couple of hours of work this afternoon :)) ):

 - first, I could not load OSSubprocess in latest pharo. I saw there were two test watcher processes and *killed* them.

 - then, I could load OSSubprocess, but it started crashing the image when running the tests from nautilus.

 - I tested with complex and simple OSSubprocess calls and I got always the same result

 - I even removed the child watcher process to see if that was the problem, but still crashing.

In any case, I'm not sure that this is because of this addition, it's just a hunch for the moment.

On Mon, Sep 5, 2016 at 7:47 PM, stepharo <[hidden email]> wrote:

ARgh, since this got integrated I keep crashing images...

do you mean real crash or DNU.
Because indeed I got some crashes too.

--

Mariano
http://marianopeck.wordpress.com