Strategy for investigating test failures

> On 20 Oct 2019, at 16:37, PBKResearch <[hidden email]> wrote:
>
> Hello all
>  
> I have done some experiments, and basically all the questions in my previous mail can be ignored. I think it is all a question of timing, which I sort of understand. Perhaps someone can clarify for me.
>  
> All my previous tests involved running the whole test suite, to ensure the set-up and tear-down methods were run. As an experiment, I selected the one red method and clicked ‘Run Tests’. After about 10 seconds, a halt appeared with the message ‘Test took too long’. When I clicked ‘Proceed’, the green appeared almost immediately. I tried again several times with similar results, although the halt occurred at different places in the code. I went back to my old P6 version and tried running tests on the one method; each time, it ran to green in about 11-12 seconds.
>  
> So back now to the P7 version; wondered if there is now a setting for permitted test duration. Eventually found it under ‘System’, default 10 secs. Changed to 15 secs, re-run red test, now comes up green. Re-run whole test suite, all green.
>  
> So it all comes down to my ignorance of test settings. But Pharo is less than helpful; if you run the whole test suite, there is no way to distinguish a test which failed due to a wrong assertion from one which was terminated for taking too long – or did I miss something?
>  
> Peter Kenny

> On 20 Oct 2019, at 16:37, PBKResearch <[hidden email]> wrote:
>
> Hello all
>  
> I have done some experiments, and basically all the questions in my previous mail can be ignored. I think it is all a question of timing, which I sort of understand. Perhaps someone can clarify for me.
>  
> All my previous tests involved running the whole test suite, to ensure the set-up and tear-down methods were run. As an experiment, I selected the one red method and clicked ‘Run Tests’. After about 10 seconds, a halt appeared with the message ‘Test took too long’. When I clicked ‘Proceed’, the green appeared almost immediately. I tried again several times with similar results, although the halt occurred at different places in the code. I went back to my old P6 version and tried running tests on the one method; each time, it ran to green in about 11-12 seconds.
>  
> So back now to the P7 version; wondered if there is now a setting for permitted test duration. Eventually found it under ‘System’, default 10 secs. Changed to 15 secs, re-run red test, now comes up green. Re-run whole test suite, all green.
>  
> So it all comes down to my ignorance of test settings. But Pharo is less than helpful; if you run the whole test suite, there is no way to distinguish a test which failed due to a wrong assertion from one which was terminated for taking too long – or did I miss something?
>  
> Peter Kenny

> On 20 Oct 2019, at 20:11, PBKResearch <[hidden email]> wrote:
>
> Sven
>
> Thanks - the real problem is my unfamiliarity with P7, so hints like this are helpful. I am happier with familiar systems, but I will switch to the latest Moose suite soon.
>
> Peter
>
> -----Original Message-----
> From: Pharo-users <[hidden email]> On Behalf Of Sven Van Caekenberghe
> Sent: 20 October 2019 17:31
> To: Any question about pharo is welcome <[hidden email]>
> Subject: Re: [Pharo-users] Strategy for investigating test failures
>
> Peter,
>
> You can override #defaultTimeLimit on the class side of the offending test.
>
> Yes, this can be confusing.
>
> There is a new test runner under development, DrTest, in Pharo 8.
>
> Sven
>
>> On 20 Oct 2019, at 16:37, PBKResearch <[hidden email]> wrote:
>>
>> Hello all
>>
>> I have done some experiments, and basically all the questions in my previous mail can be ignored. I think it is all a question of timing, which I sort of understand. Perhaps someone can clarify for me.
>>
>> All my previous tests involved running the whole test suite, to ensure the set-up and tear-down methods were run. As an experiment, I selected the one red method and clicked ‘Run Tests’. After about 10 seconds, a halt appeared with the message ‘Test took too long’. When I clicked ‘Proceed’, the green appeared almost immediately. I tried again several times with similar results, although the halt occurred at different places in the code. I went back to my old P6 version and tried running tests on the one method; each time, it ran to green in about 11-12 seconds.
>>
>> So back now to the P7 version; wondered if there is now a setting for permitted test duration. Eventually found it under ‘System’, default 10 secs. Changed to 15 secs, re-run red test, now comes up green. Re-run whole test suite, all green.
>>
>> So it all comes down to my ignorance of test settings. But Pharo is less than helpful; if you run the whole test suite, there is no way to distinguish a test which failed due to a wrong assertion from one which was terminated for taking too long – or did I miss something?
>>
>> Peter Kenny
>
>
>

> On 20 Oct 2019, at 20:11, PBKResearch <[hidden email]> wrote:
>
> Sven
>
> Thanks - the real problem is my unfamiliarity with P7, so hints like this are helpful. I am happier with familiar systems, but I will switch to the latest Moose suite soon.
>
> Peter
>
> -----Original Message-----
> From: Pharo-users <[hidden email]> On Behalf Of Sven Van Caekenberghe
> Sent: 20 October 2019 17:31
> To: Any question about pharo is welcome <[hidden email]>
> Subject: Re: [Pharo-users] Strategy for investigating test failures
>
> Peter,
>
> You can override #defaultTimeLimit on the class side of the offending test.
>
> Yes, this can be confusing.
>
> There is a new test runner under development, DrTest, in Pharo 8.
>
> Sven
>
>> On 20 Oct 2019, at 16:37, PBKResearch <[hidden email]> wrote:
>>
>> Hello all
>>
>> I have done some experiments, and basically all the questions in my previous mail can be ignored. I think it is all a question of timing, which I sort of understand. Perhaps someone can clarify for me.
>>
>> All my previous tests involved running the whole test suite, to ensure the set-up and tear-down methods were run. As an experiment, I selected the one red method and clicked ‘Run Tests’. After about 10 seconds, a halt appeared with the message ‘Test took too long’. When I clicked ‘Proceed’, the green appeared almost immediately. I tried again several times with similar results, although the halt occurred at different places in the code. I went back to my old P6 version and tried running tests on the one method; each time, it ran to green in about 11-12 seconds.
>>
>> So back now to the P7 version; wondered if there is now a setting for permitted test duration. Eventually found it under ‘System’, default 10 secs. Changed to 15 secs, re-run red test, now comes up green. Re-run whole test suite, all green.
>>
>> So it all comes down to my ignorance of test settings. But Pharo is less than helpful; if you run the whole test suite, there is no way to distinguish a test which failed due to a wrong assertion from one which was terminated for taking too long – or did I miss something?
>>
>> Peter Kenny
>
>
>

> On 20 Oct 2019, at 20:11, PBKResearch <[hidden email]> wrote:
>
> Sven
>
> Thanks - the real problem is my unfamiliarity with P7, so hints like this are helpful. I am happier with familiar systems, but I will switch to the latest Moose suite soon.
>
> Peter
>
> -----Original Message-----
> From: Pharo-users <[hidden email]> On Behalf Of Sven Van Caekenberghe
> Sent: 20 October 2019 17:31
> To: Any question about pharo is welcome <[hidden email]>
> Subject: Re: [Pharo-users] Strategy for investigating test failures
>
> Peter,
>
> You can override #defaultTimeLimit on the class side of the offending test.
>
> Yes, this can be confusing.
>
> There is a new test runner under development, DrTest, in Pharo 8.
>
> Sven
>
>> On 20 Oct 2019, at 16:37, PBKResearch <[hidden email]> wrote:
>>
>> Hello all
>>
>> I have done some experiments, and basically all the questions in my previous mail can be ignored. I think it is all a question of timing, which I sort of understand. Perhaps someone can clarify for me.
>>
>> All my previous tests involved running the whole test suite, to ensure the set-up and tear-down methods were run. As an experiment, I selected the one red method and clicked ‘Run Tests’. After about 10 seconds, a halt appeared with the message ‘Test took too long’. When I clicked ‘Proceed’, the green appeared almost immediately. I tried again several times with similar results, although the halt occurred at different places in the code. I went back to my old P6 version and tried running tests on the one method; each time, it ran to green in about 11-12 seconds.
>>
>> So back now to the P7 version; wondered if there is now a setting for permitted test duration. Eventually found it under ‘System’, default 10 secs. Changed to 15 secs, re-run red test, now comes up green. Re-run whole test suite, all green.
>>
>> So it all comes down to my ignorance of test settings. But Pharo is less than helpful; if you run the whole test suite, there is no way to distinguish a test which failed due to a wrong assertion from one which was terminated for taking too long – or did I miss something?
>>
>> Peter Kenny
>
>
>