I'd like to better understand the semantics of "expected failures" in
TestRunner. It seems to me that if you want to ensure that a certain operation fails, in a test you'd wrap it as follows... shouldFailed=false. [ self operationThatShouldFail ] on: Error do: [ shouldFailed := true ]. self assert: shouldFailed. So is tagging methods with pragma <expectedFailure> or in method #expectedFailures a temporary measure used to bypass a failing test when the judgment is that it is not critical to fix immediately? What is the process for tacking and resolving expected failures. It would give a warm fuzzy feeling if no expected failures are reported in TestRunner. Otherwise it leaves some residual uncertainty that something is wrong, even though a failure is "expected". cheers -ben |
ben
camillo once wrote a nice mail about shouldnot: Error. If I remember correctly. Stef On 25 Feb 2014, at 17:04, [hidden email] wrote: > I'd like to better understand the semantics of "expected failures" in TestRunner. It seems to me that if you want to ensure that a certain operation fails, in a test you'd wrap it as follows... > > shouldFailed=false. > [ self operationThatShouldFail ] on: Error do: [ shouldFailed := true ]. > self assert: shouldFailed. > > So is tagging methods with pragma <expectedFailure> or in method #expectedFailures a temporary measure used to bypass a failing test when the judgment is that it is not critical to fix immediately? What is the process for tacking and resolving expected failures. It would give a warm fuzzy feeling if no expected failures are reported in TestRunner. Otherwise it leaves some residual uncertainty that something is wrong, even though a failure is "expected". > > cheers -ben > |
Hi,
I tend to wrap then like this: [ self operationThatShouldFail. self fail: 'why!'. ] on: Error do: [ "check that is the right error" ] > -----Mensaje original----- > De: Pharo-dev [mailto:[hidden email]] En nombre de > Pharo4Stef > Enviado el: Martes, 25 de Febrero de 2014 16:51 > Para: Pharo Development List > Asunto: Re: [Pharo-dev] expected failures > > ben > > camillo once wrote a nice mail about shouldnot: Error. If I remember > correctly. > > Stef > On 25 Feb 2014, at 17:04, [hidden email] wrote: > > > I'd like to better understand the semantics of "expected failures" in > TestRunner. It seems to me that if you want to ensure that a certain > operation fails, in a test you'd wrap it as follows... > > > > shouldFailed=false. > > [ self operationThatShouldFail ] on: Error do: [ shouldFailed := > true ]. > > self assert: shouldFailed. > > > > So is tagging methods with pragma <expectedFailure> or in method > #expectedFailures a temporary measure used to bypass a failing test when > the judgment is that it is not critical to fix immediately? What is the > process for tacking and resolving expected failures. It would give a > warm fuzzy feeling if no expected failures are reported in TestRunner. > Otherwise it leaves some residual uncertainty that something is wrong, > even though a failure is "expected". > > > > cheers -ben > > |
In reply to this post by Ben Coman
On 2014-02-25, at 17:04, [hidden email] wrote: > I'd like to better understand the semantics of "expected failures" in TestRunner. It seems to me that if you want to ensure that a certain operation fails, in a test you'd wrap it as follows... > > shouldFailed=false. > [ self operationThatShouldFail ] on: Error do: [ shouldFailed := true ]. > self assert: shouldFailed. I prefer: self should: [ self operationThatShouldFail ] raise: ASpecificError Otherwise you can just run the code itself: self operationThatShouldFail the test framework will take care of it and mark the test correctly as a failure. signature.asc (457 bytes) Download Attachment |
Camillo Bruni wrote:
Camillo, I don't quite follow your second case. Do you refer to using <expectedFailure> ? Doing that might be community convention, but "expected failures" for this makes me uncomfortable. It is not specific about what the failure should be, so the occurrence of non-expected failures is masked. I really like #should:raise since it is specific. It would be nice if as many <expectedFailure>s as possible were converted to #should:raise, with <epxectedFailure> being used only for fubared tests being prioritised to deal with later on, with the ultimate goal of having no <expected Failure>s in the image.On 2014-02-25, at 17:04, [hidden email] wrote:I'd like to better understand the semantics of "expected failures" in TestRunner. It seems to me that if you want to ensure that a certain operation fails, in a test you'd wrap it as follows... shouldFailed=false. [ self operationThatShouldFail ] on: Error do: [ shouldFailed := true ]. self assert: shouldFailed.I prefer: self should: [ self operationThatShouldFail ] raise: ASpecificError Otherwise you can just run the code itself: self operationThatShouldFail the test framework will take care of it and mark the test correctly as a failure. cheers -ben |
On 2014-02-26, at 19:11, [hidden email] wrote: > Camillo Bruni wrote: >> On 2014-02-25, at 17:04, [hidden email] >> wrote: >> >> >> >>> I'd like to better understand the semantics of "expected failures" in TestRunner. It seems to me that if you want to ensure that a certain operation fails, in a test you'd wrap it as follows... >>> >>> shouldFailed=false. >>> [ self operationThatShouldFail ] on: Error do: [ shouldFailed := true ]. >>> self assert: shouldFailed. >>> >>> >> >> I prefer: >> >> self should: [ self operationThatShouldFail ] raise: ASpecificError >> >> Otherwise you can just run the code itself: >> >> self operationThatShouldFail >> >> the test framework will take care of it and mark the test correctly as a failure. >> >> > Camillo, I don't quite follow your second case. Do you refer to using <expectedFailure> ? Doing that might be community convention, but "expected failures" for this makes me uncomfortable. It is not specific about what the failure should be, so the occurrence of non-expected failures is masked. I really like #should:raise since it is specific. It would be nice if as many <expectedFailure>s as possible were converted to #should:raise, with <epxectedFailure> being used only for fubared tests being prioritised to deal with later on, with the ultimate goal of having no <expected Failure>s in the image. I don't use <expectedFailure> since I don't understand what it means :P I prefer #should:raise: too. And in the case to mark a test failing (aka skip) we can do the following: self skip: 'Because it does not work' signature.asc (457 bytes) Download Attachment |
Free forum by Nabble | Edit this page |