expected failures

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

expected failures

Ben Coman
I'd like to better understand the semantics of "expected failures" in
TestRunner.  It seems to me that if you want to ensure that a certain
operation fails, in a test you'd wrap it as follows...

    shouldFailed=false.
    [ self operationThatShouldFail ] on: Error do: [ shouldFailed := true ].
    self assert: shouldFailed.

So is tagging methods with pragma <expectedFailure> or in method
#expectedFailures a temporary measure used to bypass a failing test when
the judgment is that it is not critical to fix immediately?  What is the
process for tacking and resolving expected failures.  It would give a
warm fuzzy feeling if no expected failures are reported in TestRunner.  
Otherwise it leaves some residual uncertainty that something is wrong,
even though a failure is "expected".

cheers -ben

Reply | Threaded
Open this post in threaded view
|

Re: expected failures

pharo4Stef@free.fr
ben

camillo once wrote a nice mail about shouldnot: Error. If I remember correctly.

Stef
On 25 Feb 2014, at 17:04, [hidden email] wrote:

> I'd like to better understand the semantics of "expected failures" in TestRunner.  It seems to me that if you want to ensure that a certain operation fails, in a test you'd wrap it as follows...
>
>   shouldFailed=false.
>   [ self operationThatShouldFail ] on: Error do: [ shouldFailed := true ].
>   self assert: shouldFailed.
>
> So is tagging methods with pragma <expectedFailure> or in method #expectedFailures a temporary measure used to bypass a failing test when the judgment is that it is not critical to fix immediately?  What is the process for tacking and resolving expected failures.  It would give a warm fuzzy feeling if no expected failures are reported in TestRunner.  Otherwise it leaves some residual uncertainty that something is wrong, even though a failure is "expected".
>
> cheers -ben
>


Reply | Threaded
Open this post in threaded view
|

Re: expected failures

Emilio Oca-3
Hi,

I tend to wrap then like this:

[ self operationThatShouldFail.
  self fail: 'why!'.
] on: Error do: [
  "check that is the right error"
]

> -----Mensaje original-----
> De: Pharo-dev [mailto:[hidden email]] En nombre de
> Pharo4Stef
> Enviado el: Martes, 25 de Febrero de 2014 16:51
> Para: Pharo Development List
> Asunto: Re: [Pharo-dev] expected failures
>
> ben
>
> camillo once wrote a nice mail about shouldnot: Error. If I remember
> correctly.
>
> Stef
> On 25 Feb 2014, at 17:04, [hidden email] wrote:
>
> > I'd like to better understand the semantics of "expected failures" in
> TestRunner.  It seems to me that if you want to ensure that a certain
> operation fails, in a test you'd wrap it as follows...
> >
> >   shouldFailed=false.
> >   [ self operationThatShouldFail ] on: Error do: [ shouldFailed :=
> true ].
> >   self assert: shouldFailed.
> >
> > So is tagging methods with pragma <expectedFailure> or in method
> #expectedFailures a temporary measure used to bypass a failing test when
> the judgment is that it is not critical to fix immediately?  What is the
> process for tacking and resolving expected failures.  It would give a
> warm fuzzy feeling if no expected failures are reported in TestRunner.
> Otherwise it leaves some residual uncertainty that something is wrong,
> even though a failure is "expected".
> >
> > cheers -ben
> >



Reply | Threaded
Open this post in threaded view
|

Re: expected failures

Camillo Bruni-3
In reply to this post by Ben Coman

On 2014-02-25, at 17:04, [hidden email] wrote:

> I'd like to better understand the semantics of "expected failures" in TestRunner.  It seems to me that if you want to ensure that a certain operation fails, in a test you'd wrap it as follows...
>
>   shouldFailed=false.
>   [ self operationThatShouldFail ] on: Error do: [ shouldFailed := true ].
>   self assert: shouldFailed.

I prefer:

        self should: [ self operationThatShouldFail ] raise: ASpecificError

Otherwise you can just run the code itself:

        self operationThatShouldFail

the test framework will take care of it and mark the test correctly as a failure.

signature.asc (457 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: expected failures

Ben Coman
Camillo Bruni wrote:
On 2014-02-25, at 17:04, [hidden email] wrote:

  
I'd like to better understand the semantics of "expected failures" in TestRunner.  It seems to me that if you want to ensure that a certain operation fails, in a test you'd wrap it as follows...

  shouldFailed=false.
  [ self operationThatShouldFail ] on: Error do: [ shouldFailed := true ].
  self assert: shouldFailed.
    

I prefer:

	self should: [ self operationThatShouldFail ] raise: ASpecificError

Otherwise you can just run the code itself:

	self operationThatShouldFail

the test framework will take care of it and mark the test correctly as a failure.
  
Camillo, I don't quite follow your second case.  Do you refer to using <expectedFailure> ?  Doing that might be community convention, but "expected failures" for this makes me uncomfortable.  It is not specific about what the failure should be, so the occurrence of non-expected failures is masked.  I really like #should:raise since it is specific.  It would be nice if as many <expectedFailure>s as possible were converted to #should:raise, with <epxectedFailure> being used only for fubared tests being prioritised to deal with later on, with the ultimate goal of having no <expected Failure>s in the image.

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: expected failures

Camillo Bruni-3

On 2014-02-26, at 19:11, [hidden email] wrote:

> Camillo Bruni wrote:
>> On 2014-02-25, at 17:04, [hidden email]
>>  wrote:
>>
>>  
>>
>>> I'd like to better understand the semantics of "expected failures" in TestRunner.  It seems to me that if you want to ensure that a certain operation fails, in a test you'd wrap it as follows...
>>>
>>>   shouldFailed=false.
>>>   [ self operationThatShouldFail ] on: Error do: [ shouldFailed := true ].
>>>   self assert: shouldFailed.
>>>    
>>>
>>
>> I prefer:
>>
>> self should: [ self operationThatShouldFail ] raise: ASpecificError
>>
>> Otherwise you can just run the code itself:
>>
>> self operationThatShouldFail
>>
>> the test framework will take care of it and mark the test correctly as a failure.
>>  
>>
> Camillo, I don't quite follow your second case.  Do you refer to using <expectedFailure> ?  Doing that might be community convention, but "expected failures" for this makes me uncomfortable.  It is not specific about what the failure should be, so the occurrence of non-expected failures is masked.  I really like #should:raise since it is specific.  It would be nice if as many <expectedFailure>s as possible were converted to #should:raise, with <epxectedFailure> being used only for fubared tests being prioritised to deal with later on, with the ultimate goal of having no <expected Failure>s in the image.
ah sorry, you are right, I didn't answer correctly.
I don't use <expectedFailure> since I don't understand what it means :P

I prefer #should:raise: too. And in the case to mark a test failing (aka skip)
we can do the following:

        self skip: 'Because it does not work'


signature.asc (457 bytes) Download Attachment