[Question/Bug] SUnit CodeCoverage uses #valueUnpreemptively

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Question/Bug] SUnit CodeCoverage uses #valueUnpreemptively

Christoph Thiede

Hi all,


I have a question regarding to the implementation of TestRunner >> #collectCoverageFor:. Why does it use #valueUnpreemptively during running the coverage tests?


I found two major disadvantages of using #valueUnpreemptively in the TestRunner:


  1. You cannot interrupt the system via cmd+dot if you lose your patience to wait for all tests to complete.

  2. In addition, I found a quite interesting bug that makes your image unusable if you do the following:

TestCase subclass: #MyTestCase

instanceVariableNames: ''

classVariableNames: ''

poolDictionaries: ''

category: 'UserObjects'.

MyTestCase compile: 'testTimeout

[] repeat'.

TestRunner openForSuite: MyTestCase suite.

"Now press 'Run coverage' abd choose any non-empty package."


The coverage test will never end!


#valueUnpreemptively's comment says that you should use it very carefully, what makes absolutely sense to me, and it has only two senders in the Trunk at all (the second one is ClassBuilder >> #update:to:).

Why should it be necessary to use it in the TestRunner? Because we replace CompiledMethods? That's nothing WrappedBreakpoint wouldn't do, too. I replaced #valueUnpreemptively in this method by a normal #value in my image and the world did not end - thanks to dynamic forwarding, I could even keep browsing the patched methods. However, I admit it would not be a good idea to recompile any method during the coverage tests are running. (We could also use some property like Class >> #beLocked to avoid this, hypothetically).


I am looking forward to your advice.


Best,

Christoph



Reply | Threaded
Open this post in threaded view
|

Re: [Question/Bug] SUnit CodeCoverage uses #valueUnpreemptively

marcel.taeumel
Hi Christoph,

I would only start collecting coverage for tests that run fine in the first place. And if I would collect coverage, I want to get reliable numbers -- if that's even possible when speaking of "coverage" at all. :-D ... Anyway, higher-level processes could largely interfere with the results without you knowing it. Results could change just by re-running coverage, which would be bad.

Maybe one could bump the "user interrupt watcher" from 60 to 81 while running those coverage tests? To make CMD+Dot work ...

Hmm... you can also not run coverage on tests that rely on other processes to finish unless those processes have a priority relative to the current one and not hard-coded via "Processor userBackgroundPriority" etc. Those are fixed.

Just my thoughts on this topic.

Best,
Marcel

Am 09.04.2020 15:33:23 schrieb Thiede, Christoph <[hidden email]>:

Hi all,


I have a question regarding to the implementation of TestRunner >> #collectCoverageFor:. Why does it use #valueUnpreemptively during running the coverage tests?


I found two major disadvantages of using #valueUnpreemptively in the TestRunner:


  1. You cannot interrupt the system via cmd+dot if you lose your patience to wait for all tests to complete.

  2. In addition, I found a quite interesting bug that makes your image unusable if you do the following:

TestCase subclass: #MyTestCase

instanceVariableNames: ''

classVariableNames: ''

poolDictionaries: ''

category: 'UserObjects'.

MyTestCase compile: 'testTimeout

[] repeat'.

TestRunner openForSuite: MyTestCase suite.

"Now press 'Run coverage' abd choose any non-empty package."


The coverage test will never end!


#valueUnpreemptively's comment says that you should use it very carefully, what makes absolutely sense to me, and it has only two senders in the Trunk at all (the second one is ClassBuilder >> #update:to:).

Why should it be necessary to use it in the TestRunner? Because we replace CompiledMethods? That's nothing WrappedBreakpoint wouldn't do, too. I replaced #valueUnpreemptively in this method by a normal #value in my image and the world did not end - thanks to dynamic forwarding, I could even keep browsing the patched methods. However, I admit it would not be a good idea to recompile any method during the coverage tests are running. (We could also use some property like Class >> #beLocked to avoid this, hypothetically).


I am looking forward to your advice.


Best,

Christoph



Reply | Threaded
Open this post in threaded view
|

Re: [Question/Bug] SUnit CodeCoverage uses #valueUnpreemptively

Christoph Thiede

Hi Marcel, thanks for your points! :-) I think the coverage detection should work with hard-coded priorities, too. There are just too many users that don't respect Processor activePriority etc.


Hm ... Speaking hypothetically, couldn't we check Processor activeProcess in every single TestCoverage, i.e.:

- Register the testing process when creating a coverage wrapper

- Only #mark the method if Processor activeProcess equals the testing process or a child of it?


Then we could get completely rid of global state and you should be able to run coverage testing completely asynchronously (for example in AutoTDD)! :-)

However, this could slow down coverage testing significantly, I did not yet try it out ...


Best,

Christoph


Von: Squeak-dev <[hidden email]> im Auftrag von Taeumel, Marcel
Gesendet: Montag, 20. April 2020 19:01:15
An: gettimothy via Squeak-dev
Betreff: Re: [squeak-dev] [Question/Bug] SUnit CodeCoverage uses #valueUnpreemptively
 
Hi Christoph,

I would only start collecting coverage for tests that run fine in the first place. And if I would collect coverage, I want to get reliable numbers -- if that's even possible when speaking of "coverage" at all. :-D ... Anyway, higher-level processes could largely interfere with the results without you knowing it. Results could change just by re-running coverage, which would be bad.

Maybe one could bump the "user interrupt watcher" from 60 to 81 while running those coverage tests? To make CMD+Dot work ...

Hmm... you can also not run coverage on tests that rely on other processes to finish unless those processes have a priority relative to the current one and not hard-coded via "Processor userBackgroundPriority" etc. Those are fixed.

Just my thoughts on this topic.

Best,
Marcel

Am 09.04.2020 15:33:23 schrieb Thiede, Christoph <[hidden email]>:

Hi all,


I have a question regarding to the implementation of TestRunner >> #collectCoverageFor:. Why does it use #valueUnpreemptively during running the coverage tests?


I found two major disadvantages of using #valueUnpreemptively in the TestRunner:


  1. You cannot interrupt the system via cmd+dot if you lose your patience to wait for all tests to complete.

  2. In addition, I found a quite interesting bug that makes your image unusable if you do the following:

TestCase subclass: #MyTestCase

instanceVariableNames: ''

classVariableNames: ''

poolDictionaries: ''

category: 'UserObjects'.

MyTestCase compile: 'testTimeout

[] repeat'.

TestRunner openForSuite: MyTestCase suite.

"Now press 'Run coverage' abd choose any non-empty package."


The coverage test will never end!


#valueUnpreemptively's comment says that you should use it very carefully, what makes absolutely sense to me, and it has only two senders in the Trunk at all (the second one is ClassBuilder >> #update:to:).

Why should it be necessary to use it in the TestRunner? Because we replace CompiledMethods? That's nothing WrappedBreakpoint wouldn't do, too. I replaced #valueUnpreemptively in this method by a normal #value in my image and the world did not end - thanks to dynamic forwarding, I could even keep browsing the patched methods. However, I admit it would not be a good idea to recompile any method during the coverage tests are running. (We could also use some property like Class >> #beLocked to avoid this, hypothetically).


I am looking forward to your advice.


Best,

Christoph