Squeak's AssertionFailure vs. SUnit's TestFailure

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Squeak's AssertionFailure vs. SUnit's TestFailure

marcel.taeumel
Hi, there.

The Kernel package provides "AssertionFailure" through "self assert: ...".
The SUnit package provides "TestFailure" through "self assert: ..." in test cases.

AssertionFailure is a subclass of Halt.
TestFailure is a subclass of Exception.

Now, the test runner catches all errors (red) and all test failures (yellow). However, the rest runner stops working if an AssertionFailure is raised. The Decompiler can do that, for example. In that case, the user cannot easily get an overview of all non-passing tests.

Here are my questions:

- Should AssertionFailure be an Error instead of a Halt?
- Should SUnit treat "real" AssertionFailures also as errors (red)?

Find attached two changesets with two different strategies.

See TestResult >> #runCase: to see which exceptions are caught during test execution.

Best,
Marcel



assert-in-sunit.1.cs (363 bytes) Download Attachment
assert-in-sunit-alternative.1.cs (260 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Squeak's AssertionFailure vs. SUnit's TestFailure

Nicolas Cellier
Hi Marcel,
Halt is a breakpoint explicitly set in the source code.
In some way, Assertion could be viewed as conditional halt explicitly set in the source code...
However, I don't like this POV. IMO AssertionFailure should not be subclass of Halt.
Assertion failures are failure to fullfil some pre/post condition, and as such programming Error.
I understand that Halt is just a way to invite users to debug programming Error, but an Error works equally well.
Why using Halt?

Le lun. 11 nov. 2019 à 16:03, Marcel Taeumel <[hidden email]> a écrit :
Hi, there.

The Kernel package provides "AssertionFailure" through "self assert: ...".
The SUnit package provides "TestFailure" through "self assert: ..." in test cases.

AssertionFailure is a subclass of Halt.
TestFailure is a subclass of Exception.

Now, the test runner catches all errors (red) and all test failures (yellow). However, the rest runner stops working if an AssertionFailure is raised. The Decompiler can do that, for example. In that case, the user cannot easily get an overview of all non-passing tests.

Here are my questions:

- Should AssertionFailure be an Error instead of a Halt?
- Should SUnit treat "real" AssertionFailures also as errors (red)?

Find attached two changesets with two different strategies.

See TestResult >> #runCase: to see which exceptions are caught during test execution.

Best,
Marcel



Reply | Threaded
Open this post in threaded view
|

Re: Squeak's AssertionFailure vs. SUnit's TestFailure

Christoph Thiede

I understand that Halt is just a way to invite users to debug programming Error, but an Error works equally well.


Related question: Why was Halt designed to behave exactly like an Error? I understand Halt as a note for a programmer to investigate something, but imho it does not indicate that something went wrong, so I would not expect it to abort the process (neglecting the fact that there is usually a debugger).

Imagine something like a ProductionToolSet (similar to the approach in https://wiki.squeak.org/squeak/6633), I would assume it should ignore Halts and should just resume. So wouldn't it be better if Halt raised an UnhandledHalt, which could be ignored by certain ToolSets? (I would even propose this behavior for the CommandLineToolSet.)

Please see the attached changeset.


---


Concerning AssertionFailures, I think outside of tests, this code


1 to: aCollection size do: [:index |

    aCollection remove: (aCollection at: index)].

self assert: [aCollection isEmpty].


should be broadly equivalent to


1 to: aCollection size do: [:index |

    aCollection remove: (aCollection at: index)].

aCollection isEmpty ifFalse: [

    self error: 'Internal error occurred! Collection is not empty'

    "Unless there is a programmer debugging, execution won't be continued"].


So I would agree with Marcel's proposal to derive AssertionFailure from Error.
Just a few days ago, I realized that an AssertionFaillure raised in a drawing method would damage the image, even if we catch "Error, Warning, Halt".
So it seems the right approach to me to refine the definition of exception classes instead of enumerating them at multiple different places.


Best,

Christoph



Von: Squeak-dev <[hidden email]> im Auftrag von Nicolas Cellier <[hidden email]>
Gesendet: Montag, 11. November 2019 21:24 Uhr
An: The general-purpose Squeak developers list
Betreff: Re: [squeak-dev] Squeak's AssertionFailure vs. SUnit's TestFailure
 
Hi Marcel,
Halt is a breakpoint explicitly set in the source code.
In some way, Assertion could be viewed as conditional halt explicitly set in the source code...
However, I don't like this POV. IMO AssertionFailure should not be subclass of Halt.
Assertion failures are failure to fullfil some pre/post condition, and as such programming Error.
I understand that Halt is just a way to invite users to debug programming Error, but an Error works equally well.
Why using Halt?

Le lun. 11 nov. 2019 à 16:03, Marcel Taeumel <[hidden email]> a écrit :
Hi, there.

The Kernel package provides "AssertionFailure" through "self assert: ...".
The SUnit package provides "TestFailure" through "self assert: ..." in test cases.

AssertionFailure is a subclass of Halt.
TestFailure is a subclass of Exception.

Now, the test runner catches all errors (red) and all test failures (yellow). However, the rest runner stops working if an AssertionFailure is raised. The Decompiler can do that, for example. In that case, the user cannot easily get an overview of all non-passing tests.

Here are my questions:

- Should AssertionFailure be an Error instead of a Halt?
- Should SUnit treat "real" AssertionFailures also as errors (red)?

Find attached two changesets with two different strategies.

See TestResult >> #runCase: to see which exceptions are caught during test execution.

Best,
Marcel




Halt is not an Error.1.cs (2K) Download Attachment
Carpe Squeak!
Reply | Threaded
Open this post in threaded view
|

Re: Squeak's AssertionFailure vs. SUnit's TestFailure

marcel.taeumel
Hi Christoph.

Why was Halt designed to behave exactly like an Error?

Well, it is not. Halt can be proceeded (in the debugger) (or resumed) without caution. :-) In Squeak's exception-handling terms, Halt seems to be a mix of Error and Notification.

> ... so I would not expect it to abort the process ...

It does not. A Halt just suspends the process.

So wouldn't it be better if Halt raised an UnhandledHalt, which could be ignored by certain ToolSets?

Hmm... it is not that difficult to check for "Halt" in a toolset's #handleError:. I think that such halt's should not stay in code anyway. So, it is just a quick way to check something in the control flow. 

Best,
Marcel

Am 11.11.2019 22:29:22 schrieb Thiede, Christoph <[hidden email]>:

I understand that Halt is just a way to invite users to debug programming Error, but an Error works equally well.


Related question: Why was Halt designed to behave exactly like an Error? I understand Halt as a note for a programmer to investigate something, but imho it does not indicate that something went wrong, so I would not expect it to abort the process (neglecting the fact that there is usually a debugger).

Imagine something like a ProductionToolSet (similar to the approach in https://wiki.squeak.org/squeak/6633), I would assume it should ignore Halts and should just resume. So wouldn't it be better if Halt raised an UnhandledHalt, which could be ignored by certain ToolSets? (I would even propose this behavior for the CommandLineToolSet.)

Please see the attached changeset.


---


Concerning AssertionFailures, I think outside of tests, this code


1 to: aCollection size do: [:index |

    aCollection remove: (aCollection at: index)].

self assert: [aCollection isEmpty].


should be broadly equivalent to


1 to: aCollection size do: [:index |

    aCollection remove: (aCollection at: index)].

aCollection isEmpty ifFalse: [

    self error: 'Internal error occurred! Collection is not empty'

    "Unless there is a programmer debugging, execution won't be continued"].


So I would agree with Marcel's proposal to derive AssertionFailure from Error.
Just a few days ago, I realized that an AssertionFaillure raised in a drawing method would damage the image, even if we catch "Error, Warning, Halt".
So it seems the right approach to me to refine the definition of exception classes instead of enumerating them at multiple different places.


Best,

Christoph



Von: Squeak-dev <[hidden email]> im Auftrag von Nicolas Cellier <[hidden email]>
Gesendet: Montag, 11. November 2019 21:24 Uhr
An: The general-purpose Squeak developers list
Betreff: Re: [squeak-dev] Squeak's AssertionFailure vs. SUnit's TestFailure
 
Hi Marcel,
Halt is a breakpoint explicitly set in the source code.
In some way, Assertion could be viewed as conditional halt explicitly set in the source code...
However, I don't like this POV. IMO AssertionFailure should not be subclass of Halt.
Assertion failures are failure to fullfil some pre/post condition, and as such programming Error.
I understand that Halt is just a way to invite users to debug programming Error, but an Error works equally well.
Why using Halt?

Le lun. 11 nov. 2019 à 16:03, Marcel Taeumel <[hidden email]> a écrit :
Hi, there.

The Kernel package provides "AssertionFailure" through "self assert: ...".
The SUnit package provides "TestFailure" through "self assert: ..." in test cases.

AssertionFailure is a subclass of Halt.
TestFailure is a subclass of Exception.

Now, the test runner catches all errors (red) and all test failures (yellow). However, the rest runner stops working if an AssertionFailure is raised. The Decompiler can do that, for example. In that case, the user cannot easily get an overview of all non-passing tests.

Here are my questions:

- Should AssertionFailure be an Error instead of a Halt?
- Should SUnit treat "real" AssertionFailures also as errors (red)?

Find attached two changesets with two different strategies.

See TestResult >> #runCase: to see which exceptions are caught during test execution.

Best,
Marcel



Reply | Threaded
Open this post in threaded view
|

Re: Squeak's AssertionFailure vs. SUnit's TestFailure

marcel.taeumel
Hi Christoph.

Please see the attached changeset.

Ah, I see. In #defaultAction, Squeak treats a Halt similar to an Error. Then again, what are notifications that need user input to resume automatically? Some design challenges to tackle here ... :-)

Best,
Marcel

Am 12.11.2019 10:15:05 schrieb Marcel Taeumel <[hidden email]>:

Hi Christoph.

Why was Halt designed to behave exactly like an Error?

Well, it is not. Halt can be proceeded (in the debugger) (or resumed) without caution. :-) In Squeak's exception-handling terms, Halt seems to be a mix of Error and Notification.

> ... so I would not expect it to abort the process ...

It does not. A Halt just suspends the process.

So wouldn't it be better if Halt raised an UnhandledHalt, which could be ignored by certain ToolSets?

Hmm... it is not that difficult to check for "Halt" in a toolset's #handleError:. I think that such halt's should not stay in code anyway. So, it is just a quick way to check something in the control flow. 

Best,
Marcel

Am 11.11.2019 22:29:22 schrieb Thiede, Christoph <[hidden email]>:

I understand that Halt is just a way to invite users to debug programming Error, but an Error works equally well.


Related question: Why was Halt designed to behave exactly like an Error? I understand Halt as a note for a programmer to investigate something, but imho it does not indicate that something went wrong, so I would not expect it to abort the process (neglecting the fact that there is usually a debugger).

Imagine something like a ProductionToolSet (similar to the approach in https://wiki.squeak.org/squeak/6633), I would assume it should ignore Halts and should just resume. So wouldn't it be better if Halt raised an UnhandledHalt, which could be ignored by certain ToolSets? (I would even propose this behavior for the CommandLineToolSet.)

Please see the attached changeset.


---


Concerning AssertionFailures, I think outside of tests, this code


1 to: aCollection size do: [:index |

    aCollection remove: (aCollection at: index)].

self assert: [aCollection isEmpty].


should be broadly equivalent to


1 to: aCollection size do: [:index |

    aCollection remove: (aCollection at: index)].

aCollection isEmpty ifFalse: [

    self error: 'Internal error occurred! Collection is not empty'

    "Unless there is a programmer debugging, execution won't be continued"].


So I would agree with Marcel's proposal to derive AssertionFailure from Error.
Just a few days ago, I realized that an AssertionFaillure raised in a drawing method would damage the image, even if we catch "Error, Warning, Halt".
So it seems the right approach to me to refine the definition of exception classes instead of enumerating them at multiple different places.


Best,

Christoph



Von: Squeak-dev <[hidden email]> im Auftrag von Nicolas Cellier <[hidden email]>
Gesendet: Montag, 11. November 2019 21:24 Uhr
An: The general-purpose Squeak developers list
Betreff: Re: [squeak-dev] Squeak's AssertionFailure vs. SUnit's TestFailure
 
Hi Marcel,
Halt is a breakpoint explicitly set in the source code.
In some way, Assertion could be viewed as conditional halt explicitly set in the source code...
However, I don't like this POV. IMO AssertionFailure should not be subclass of Halt.
Assertion failures are failure to fullfil some pre/post condition, and as such programming Error.
I understand that Halt is just a way to invite users to debug programming Error, but an Error works equally well.
Why using Halt?

Le lun. 11 nov. 2019 à 16:03, Marcel Taeumel <[hidden email]> a écrit :
Hi, there.

The Kernel package provides "AssertionFailure" through "self assert: ...".
The SUnit package provides "TestFailure" through "self assert: ..." in test cases.

AssertionFailure is a subclass of Halt.
TestFailure is a subclass of Exception.

Now, the test runner catches all errors (red) and all test failures (yellow). However, the rest runner stops working if an AssertionFailure is raised. The Decompiler can do that, for example. In that case, the user cannot easily get an overview of all non-passing tests.

Here are my questions:

- Should AssertionFailure be an Error instead of a Halt?
- Should SUnit treat "real" AssertionFailures also as errors (red)?

Find attached two changesets with two different strategies.

See TestResult >> #runCase: to see which exceptions are caught during test execution.

Best,
Marcel



Reply | Threaded
Open this post in threaded view
|

Re: Squeak's AssertionFailure vs. SUnit's TestFailure

Christoph Thiede

Then again, what are notifications that need user input to resume automatically?


Well, a halt, nothing else so far, I guess :-)


Hmm... it is not that difficult to check for "Halt" in a toolset's #handleError:


I think I see your point of holding the ToolSet interface clean, but making the halt check in a specific toolset implementation would stop us from providing different variation points to each toolset implementation, wouldn't it?


Best,

Christoph


Von: Squeak-dev <[hidden email]> im Auftrag von Taeumel, Marcel
Gesendet: Dienstag, 12. November 2019 10:17:57
An: John Pfersich via Squeak-dev
Betreff: Re: [squeak-dev] Squeak's AssertionFailure vs. SUnit's TestFailure
 
Hi Christoph.

Please see the attached changeset.

Ah, I see. In #defaultAction, Squeak treats a Halt similar to an Error. Then again, what are notifications that need user input to resume automatically? Some design challenges to tackle here ... :-)

Best,
Marcel

Am 12.11.2019 10:15:05 schrieb Marcel Taeumel <[hidden email]>:

Hi Christoph.

Why was Halt designed to behave exactly like an Error?

Well, it is not. Halt can be proceeded (in the debugger) (or resumed) without caution. :-) In Squeak's exception-handling terms, Halt seems to be a mix of Error and Notification.

> ... so I would not expect it to abort the process ...

It does not. A Halt just suspends the process.

So wouldn't it be better if Halt raised an UnhandledHalt, which could be ignored by certain ToolSets?

Hmm... it is not that difficult to check for "Halt" in a toolset's #handleError:. I think that such halt's should not stay in code anyway. So, it is just a quick way to check something in the control flow. 

Best,
Marcel

Am 11.11.2019 22:29:22 schrieb Thiede, Christoph <[hidden email]>:

I understand that Halt is just a way to invite users to debug programming Error, but an Error works equally well.


Related question: Why was Halt designed to behave exactly like an Error? I understand Halt as a note for a programmer to investigate something, but imho it does not indicate that something went wrong, so I would not expect it to abort the process (neglecting the fact that there is usually a debugger).

Imagine something like a ProductionToolSet (similar to the approach in https://wiki.squeak.org/squeak/6633), I would assume it should ignore Halts and should just resume. So wouldn't it be better if Halt raised an UnhandledHalt, which could be ignored by certain ToolSets? (I would even propose this behavior for the CommandLineToolSet.)

Please see the attached changeset.


---


Concerning AssertionFailures, I think outside of tests, this code


1 to: aCollection size do: [:index |

    aCollection remove: (aCollection at: index)].

self assert: [aCollection isEmpty].


should be broadly equivalent to


1 to: aCollection size do: [:index |

    aCollection remove: (aCollection at: index)].

aCollection isEmpty ifFalse: [

    self error: 'Internal error occurred! Collection is not empty'

    "Unless there is a programmer debugging, execution won't be continued"].


So I would agree with Marcel's proposal to derive AssertionFailure from Error.
Just a few days ago, I realized that an AssertionFaillure raised in a drawing method would damage the image, even if we catch "Error, Warning, Halt".
So it seems the right approach to me to refine the definition of exception classes instead of enumerating them at multiple different places.


Best,

Christoph



Von: Squeak-dev <[hidden email]> im Auftrag von Nicolas Cellier <[hidden email]>
Gesendet: Montag, 11. November 2019 21:24 Uhr
An: The general-purpose Squeak developers list
Betreff: Re: [squeak-dev] Squeak's AssertionFailure vs. SUnit's TestFailure
 
Hi Marcel,
Halt is a breakpoint explicitly set in the source code.
In some way, Assertion could be viewed as conditional halt explicitly set in the source code...
However, I don't like this POV. IMO AssertionFailure should not be subclass of Halt.
Assertion failures are failure to fullfil some pre/post condition, and as such programming Error.
I understand that Halt is just a way to invite users to debug programming Error, but an Error works equally well.
Why using Halt?

Le lun. 11 nov. 2019 à 16:03, Marcel Taeumel <[hidden email]> a écrit :
Hi, there.

The Kernel package provides "AssertionFailure" through "self assert: ...".
The SUnit package provides "TestFailure" through "self assert: ..." in test cases.

AssertionFailure is a subclass of Halt.
TestFailure is a subclass of Exception.

Now, the test runner catches all errors (red) and all test failures (yellow). However, the rest runner stops working if an AssertionFailure is raised. The Decompiler can do that, for example. In that case, the user cannot easily get an overview of all non-passing tests.

Here are my questions:

- Should AssertionFailure be an Error instead of a Halt?
- Should SUnit treat "real" AssertionFailures also as errors (red)?

Find attached two changesets with two different strategies.

See TestResult >> #runCase: to see which exceptions are caught during test execution.

Best,
Marcel



Carpe Squeak!
Reply | Threaded
Open this post in threaded view
|

Re: Squeak's AssertionFailure vs. SUnit's TestFailure

marcel.taeumel
Well, a halt, nothing else so far, I guess :-)

I don't mean that kind of user input. ;-) I mean dialog windows being spawn in a notification's #defaultAction. Well, maybe those are related ...

Best,
Marcel

Am 14.11.2019 15:42:00 schrieb Thiede, Christoph <[hidden email]>:

Then again, what are notifications that need user input to resume automatically?


Well, a halt, nothing else so far, I guess :-)


Hmm... it is not that difficult to check for "Halt" in a toolset's #handleError:


I think I see your point of holding the ToolSet interface clean, but making the halt check in a specific toolset implementation would stop us from providing different variation points to each toolset implementation, wouldn't it?


Best,

Christoph


Von: Squeak-dev <[hidden email]> im Auftrag von Taeumel, Marcel
Gesendet: Dienstag, 12. November 2019 10:17:57
An: John Pfersich via Squeak-dev
Betreff: Re: [squeak-dev] Squeak's AssertionFailure vs. SUnit's TestFailure
 
Hi Christoph.

Please see the attached changeset.

Ah, I see. In #defaultAction, Squeak treats a Halt similar to an Error. Then again, what are notifications that need user input to resume automatically? Some design challenges to tackle here ... :-)

Best,
Marcel

Am 12.11.2019 10:15:05 schrieb Marcel Taeumel <[hidden email]>:

Hi Christoph.

Why was Halt designed to behave exactly like an Error?

Well, it is not. Halt can be proceeded (in the debugger) (or resumed) without caution. :-) In Squeak's exception-handling terms, Halt seems to be a mix of Error and Notification.

> ... so I would not expect it to abort the process ...

It does not. A Halt just suspends the process.

So wouldn't it be better if Halt raised an UnhandledHalt, which could be ignored by certain ToolSets?

Hmm... it is not that difficult to check for "Halt" in a toolset's #handleError:. I think that such halt's should not stay in code anyway. So, it is just a quick way to check something in the control flow. 

Best,
Marcel

Am 11.11.2019 22:29:22 schrieb Thiede, Christoph <[hidden email]>:

I understand that Halt is just a way to invite users to debug programming Error, but an Error works equally well.


Related question: Why was Halt designed to behave exactly like an Error? I understand Halt as a note for a programmer to investigate something, but imho it does not indicate that something went wrong, so I would not expect it to abort the process (neglecting the fact that there is usually a debugger).

Imagine something like a ProductionToolSet (similar to the approach in https://wiki.squeak.org/squeak/6633), I would assume it should ignore Halts and should just resume. So wouldn't it be better if Halt raised an UnhandledHalt, which could be ignored by certain ToolSets? (I would even propose this behavior for the CommandLineToolSet.)

Please see the attached changeset.


---


Concerning AssertionFailures, I think outside of tests, this code


1 to: aCollection size do: [:index |

    aCollection remove: (aCollection at: index)].

self assert: [aCollection isEmpty].


should be broadly equivalent to


1 to: aCollection size do: [:index |

    aCollection remove: (aCollection at: index)].

aCollection isEmpty ifFalse: [

    self error: 'Internal error occurred! Collection is not empty'

    "Unless there is a programmer debugging, execution won't be continued"].


So I would agree with Marcel's proposal to derive AssertionFailure from Error.
Just a few days ago, I realized that an AssertionFaillure raised in a drawing method would damage the image, even if we catch "Error, Warning, Halt".
So it seems the right approach to me to refine the definition of exception classes instead of enumerating them at multiple different places.


Best,

Christoph



Von: Squeak-dev <[hidden email]> im Auftrag von Nicolas Cellier <[hidden email]>
Gesendet: Montag, 11. November 2019 21:24 Uhr
An: The general-purpose Squeak developers list
Betreff: Re: [squeak-dev] Squeak's AssertionFailure vs. SUnit's TestFailure
 
Hi Marcel,
Halt is a breakpoint explicitly set in the source code.
In some way, Assertion could be viewed as conditional halt explicitly set in the source code...
However, I don't like this POV. IMO AssertionFailure should not be subclass of Halt.
Assertion failures are failure to fullfil some pre/post condition, and as such programming Error.
I understand that Halt is just a way to invite users to debug programming Error, but an Error works equally well.
Why using Halt?

Le lun. 11 nov. 2019 à 16:03, Marcel Taeumel <[hidden email]> a écrit :
Hi, there.

The Kernel package provides "AssertionFailure" through "self assert: ...".
The SUnit package provides "TestFailure" through "self assert: ..." in test cases.

AssertionFailure is a subclass of Halt.
TestFailure is a subclass of Exception.

Now, the test runner catches all errors (red) and all test failures (yellow). However, the rest runner stops working if an AssertionFailure is raised. The Decompiler can do that, for example. In that case, the user cannot easily get an overview of all non-passing tests.

Here are my questions:

- Should AssertionFailure be an Error instead of a Halt?
- Should SUnit treat "real" AssertionFailures also as errors (red)?

Find attached two changesets with two different strategies.

See TestResult >> #runCase: to see which exceptions are caught during test execution.

Best,
Marcel