Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
Locked
Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
 
|
On Mon, Apr 8, 2013 at 9:32 AM, Alexandre Bergel <[hidden email]> wrote:
I just implemented something to this effect for our project at Cadence. It works by
- an assembler/disassebler that can convert a method into a sequence of messages, one for each of its bytecodes and back - using the disassembler to obtain labels which occur at the start of each basic block
- replacing the first bytecode in every basic block with an illegal bytecode, remembering the pc and its correct bytecode in the method properties - implementing a method on MethodContext which receives the unknowBytecode message sent by the VM when it encounters an illegal bytecode
- the method replaces the illegal bytecode with the correct bytecode, removing the entry from properties, and continues The JIT refuses to compile methods that contain illegal bytecodes so this runs at interpreter speed until a method no longer contains illegal bytecodes (has been completely covered). But the illegal bytecode is executed at most once for each basic block, since it is replaced immediately it is executed. The information remaining in properties is that of the basic blocks that have not been executed.
In the Squeak/Pharo bytecode set there are three unknown bytecodes, 126, 127 & 139. I have a Monticello package and tests if you're interested.
best, Eliot
|
Re: [Vm-dev] Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
|
|
On 9 April 2013 02:02, Eliot Miranda <[hidden email]> wrote:
> > > > On Mon, Apr 8, 2013 at 9:32 AM, Alexandre Bergel <[hidden email]> wrote: >> >> What I had in mind, is something like: >> -=-=-=-=-=-=-=-=-=-= >> myMethodWithBranch >> self foo. >> v >> ifTrue: [ ... ] >> ifFalse: [ ... ] >> -=-=-=-=-=-=-=-=-=-= >> >> -=-=-=-=-=-=-=-=-=-= >> myMethodWithBranch >> self foo. >> v >> ifTrue: [ self checkPoint: 1. ... ] >> ifFalse: [ self checkPoint: 2. ... ]. >> self checkPoint: 3. >> -=-=-=-=-=-=-=-=-=-= >> >> All these checkpoint will be inserted by manipulating the bytecode. You can then optimize a bit by removing checkpoints that have been reached. >> The method #checkPoint: could then use some attributes to the CompiledMethod. >> >> This is a common technique when (i) the VM does not give you what you need and (ii) when you do not want to change the VM. > > > I just implemented something to this effect for our project at Cadence. It works by > - an assembler/disassebler that can convert a method into a sequence of messages, one for each of its bytecodes and back > - using the disassembler to obtain labels which occur at the start of each basic block > - replacing the first bytecode in every basic block with an illegal bytecode, remembering the pc and its correct bytecode in the method properties > - implementing a method on MethodContext which receives the unknowBytecode message sent by the VM when it encounters an illegal bytecode > - the method replaces the illegal bytecode with the correct bytecode, removing the entry from properties, and continues > > The JIT refuses to compile methods that contain illegal bytecodes so this runs at interpreter speed until a method no longer contains illegal bytecodes (has been completely covered). But the illegal bytecode is executed at most once for each basic block, since it is replaced immediately it is executed. The information remaining in properties is that of the basic blocks that have not been executed. > > In the Squeak/Pharo bytecode set there are three unknown bytecodes, 126, 127 & 139. I have a Monticello package and tests if you're interested. I most definitely am! frank >> This is something I wanted to do for year in Hapao... >> >> Alexandre >> >> >> >> On Apr 7, 2013, at 3:21 AM, Clément Bera <[hidden email]> wrote: >> >> I don't know. The main problem is that the AST-interpreter is not fast enough. The best solution for speed would be to improve it as they do in this paper : http://dl.acm.org/citation.cfm?id=2384587 (their ast-interpreter is faster than a byte code interpreter). >> >> Another solution would be to execute some method with the vm and other one with the ast-interpreter, but it is quite some work. >> >> >> 2013/4/7 Alexandre Bergel <[hidden email]> >> Would it be possible to annotate the method with some checkpoints within the method? This should be fast enough I guess. >> >> Alexandre >> >> >> On Apr 6, 2013, at 10:35 AM, Clément Bera <[hidden email]> wrote: >> >> > This is exactly that. The tree coverage tool is a subclass of AST-interpreter, which adding to the interpretation marks the nodes. And then I use this marking to color the source code. And I use as you said RB ast. >> > >> > This is why I implemented it in a few hours, I had already an AST-interpreter, so I've just created a quick subclass and a spec UI. >> > >> > >> > 2013/4/6 Frank Shearar <[hidden email]> >> > So if you don't mind me being very loose with terminology, you >> > interpret (RB) ASTs, and the act of interpreting marks the nodes in >> > these ASTs, which you can then use to colour source? >> > >> > frank >> > >> > On 6 April 2013 11:00, Clément Bera <[hidden email]> wrote: >> > > Actually, it is quite different than Jejak. >> > > >> > > Implementation wise, Jejak used Opal so I guess it relies on bytecode >> > > modification to do its analysis where my tool relies on ast annotations. >> > > >> > > This implies some difference in features. It seems that Jejak record all >> > > calls, all values and all assignments in a method (which is byte code level >> > > feature), whereas I record for each ast node if it was interpreted or not >> > > (which is ast level feature). I guess there would be a little difference in >> > > the use cases. >> > > >> > > >> > > 2013/4/6 Frank Shearar <[hidden email]> >> > >> >> > >> On 6 April 2013 07:47, Stéphane Ducasse <[hidden email]> wrote: >> > >>> >> > >>> Clement built in a couple of hours the following nice branch >> > >>> analysecoverage tools :) >> > >>> It is dead slow but cool. >> > >> >> > >> >> > >> By "branch coverage tool" do you mean Jejak? >> > >> >> > >> frank >> > >> >> > >> >> > > >> > > >> > > >> > > -- >> > > Clément Béra >> > > Mate Virtual Machine Engineer >> > > Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq >> > >> > >> > >> > >> > -- >> > Clément Béra >> > Mate Virtual Machine Engineer >> > Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq >> >> -- >> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: >> Alexandre Bergel http://www.bergel.eu >> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. >> >> >> >> >> >> >> >> -- >> Clément Béra >> Mate Virtual Machine Engineer >> Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq >> >> >> -- >> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: >> Alexandre Bergel http://www.bergel.eu >> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. >> >> >> > > > > -- > best, > Eliot > |
Re: [Vm-dev] Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
|
Me, too! :)
Best, Marcel |
Re: [Vm-dev] Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
|
|
In reply to this post by Frank Shearar-3
On Mon, Apr 8, 2013 at 6:02 PM, Eliot Miranda <[hidden email]> wrote:
On Mon, Apr 8, 2013 at 11:06 PM, Frank Shearar <[hidden email]> wrote:
... -- I most definitely am! Me, too! :) On Tue, Apr 9, 2013 at 12:32 PM, Alexandre Bergel <[hidden email]> wrote: ... Yes please. I am interested! OK, in http://ss3.gemstone.com/ss/MethodMassage are the two packages needed to make this work in Squeak trunk. I added Frank and Alexandre as developers. Marcel, I couldn't find you. If you want to be added you'll have to register. BytecodeCoverageTests is the place to start looking.
I'd love to see someone use this to do fast shape change on class redefinition. cheers, Eliot
|
Re: [Vm-dev] Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
|
|
In reply to this post by Frank Shearar-3
I built a test coverage tool in VisualWorks using the probes. The tool
inserted a probe into the beginning and end of every block so it records all entrances and exits, except for exception exits. The tool would also displayed statistics of the code coverage. It is interesting to let it run for a while and look at the results. Because it shows branch coverage, you can find branches that don't seem to be executed. > -----Original Message----- > From: [hidden email] [mailto:squeak- > [hidden email]] On Behalf Of Frank Shearar > Sent: Tuesday, April 09, 2013 2:07 AM > To: Squeak Virtual Machine Development Discussion > Cc: The general-purpose Squeak developers list; Pharo Development > Subject: [squeak-dev] Re: [Vm-dev] Re: [Pharo-project] [Preview] Branch > test coverage analysis tool for Pharo 3.0 :) > > On 9 April 2013 02:02, Eliot Miranda <[hidden email]> wrote: > > > > > > > > On Mon, Apr 8, 2013 at 9:32 AM, Alexandre Bergel > <[hidden email]> wrote: > >> > >> What I had in mind, is something like: > >> -=-=-=-=-=-=-=-=-=-= > >> myMethodWithBranch > >> self foo. > >> v > >> ifTrue: [ ... ] > >> ifFalse: [ ... ] > >> -=-=-=-=-=-=-=-=-=-= > >> > >> -=-=-=-=-=-=-=-=-=-= > >> myMethodWithBranch > >> self foo. > >> v > >> ifTrue: [ self checkPoint: 1. ... ] > >> ifFalse: [ self checkPoint: 2. ... ]. > >> self checkPoint: 3. > >> -=-=-=-=-=-=-=-=-=-= > >> > >> All these checkpoint will be inserted by manipulating the bytecode. You > can then optimize a bit by removing checkpoints that have been reached. > >> The method #checkPoint: could then use some attributes to the > CompiledMethod. > >> > >> This is a common technique when (i) the VM does not give you what you > need and (ii) when you do not want to change the VM. > > > > > > I just implemented something to this effect for our project at > > Cadence. It works by > > - an assembler/disassebler that can convert a method into a sequence > > of messages, one for each of its bytecodes and back > > - using the disassembler to obtain labels which occur at the start of > > each basic block > > - replacing the first bytecode in every basic block with an illegal > > bytecode, remembering the pc and its correct bytecode in the method > > properties > > - implementing a method on MethodContext which receives the > > unknowBytecode message sent by the VM when it encounters an illegal > > bytecode > > - the method replaces the illegal bytecode with the correct bytecode, > > removing the entry from properties, and continues > > > > The JIT refuses to compile methods that contain illegal bytecodes so > runs at interpreter speed until a method no longer contains illegal bytecodes > (has been completely covered). But the illegal bytecode is executed at most > once for each basic block, since it is replaced immediately it is executed. The > information remaining in properties is that of the basic blocks that have not > been executed. > > > > In the Squeak/Pharo bytecode set there are three unknown bytecodes, > 126, 127 & 139. I have a Monticello package and tests if you're interested. > > I most definitely am! > > frank > > >> This is something I wanted to do for year in Hapao... > >> > >> Alexandre > >> > >> > >> > >> On Apr 7, 2013, at 3:21 AM, Clément Bera <[hidden email]> > wrote: > >> > >> I don't know. The main problem is that the AST-interpreter is not fast > enough. The best solution for speed would be to improve it as they do in > paper : http://dl.acm.org/citation.cfm?id=2384587 (their ast-interpreter is > faster than a byte code interpreter). > >> > >> Another solution would be to execute some method with the vm and > other one with the ast-interpreter, but it is quite some work. > >> > >> > >> 2013/4/7 Alexandre Bergel <[hidden email]> Would it be > >> possible to annotate the method with some checkpoints within the > method? This should be fast enough I guess. > >> > >> Alexandre > >> > >> > >> On Apr 6, 2013, at 10:35 AM, Clément Bera <[hidden email]> > wrote: > >> > >> > This is exactly that. The tree coverage tool is a subclass of AST- > interpreter, which adding to the interpretation marks the nodes. And then > use this marking to color the source code. And I use as you said RB ast. > >> > > >> > This is why I implemented it in a few hours, I had already an AST- > interpreter, so I've just created a quick subclass and a spec UI. > >> > > >> > > >> > 2013/4/6 Frank Shearar <[hidden email]> So if you don't > >> > mind me being very loose with terminology, you interpret (RB) ASTs, > >> > and the act of interpreting marks the nodes in these ASTs, which > >> > you can then use to colour source? > >> > > >> > frank > >> > > >> > On 6 April 2013 11:00, Clément Bera <[hidden email]> wrote: > >> > > Actually, it is quite different than Jejak. > >> > > > >> > > Implementation wise, Jejak used Opal so I guess it relies on > >> > > bytecode modification to do its analysis where my tool relies on > annotations. > >> > > > >> > > This implies some difference in features. It seems that Jejak > >> > > record all calls, all values and all assignments in a method > >> > > (which is byte code level feature), whereas I record for each ast > >> > > node if it was interpreted or not (which is ast level feature). I > >> > > guess there would be a little difference in the use cases. > >> > > > >> > > > >> > > 2013/4/6 Frank Shearar <[hidden email]> > >> > >> > >> > >> On 6 April 2013 07:47, Stéphane Ducasse > <[hidden email]> wrote: > >> > >>> > >> > >>> Clement built in a couple of hours the following nice branch > >> > >>> analysecoverage tools :) It is dead slow but cool. > >> > >> > >> > >> > >> > >> By "branch coverage tool" do you mean Jejak? > >> > >> > >> > >> frank > >> > >> > >> > >> > >> > > > >> > > > >> > > > >> > > -- > >> > > Clément Béra > >> > > Mate Virtual Machine Engineer > >> > > Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq > >> > > >> > > >> > > >> > > >> > -- > >> > Clément Béra > >> > Mate Virtual Machine Engineer > >> > Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq > >> > >> -- > >> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: > >> Alexandre Bergel http://www.bergel.eu > >> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. > >> > >> > >> > >> > >> > >> > >> > >> -- > >> Clément Béra > >> Mate Virtual Machine Engineer > >> Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq > >> > >> > >> -- > >> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: > >> Alexandre Bergel http://www.bergel.eu > >> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. > >> > >> > >> > > > > > > > > -- > > best, > > Eliot > > |
Re: [Vm-dev] Re: [Pharo-project] [Preview] Branch test coverage analysis tool for Pharo 3.0 :)
|
|
Might the vw code be available somewhere? Sounds really interesting.
Ken G. Brown, from my iPhone On 2013-04-12, at 12:32, Terry Raymond <[hidden email]> wrote: > I built a test coverage tool in VisualWorks using the probes. The tool > inserted > a probe into the beginning and end of every block so it records all > entrances and exits, > except for exception exits. > > The tool would also displayed statistics of the code coverage. It is > interesting to > let it run for a while and look at the results. Because it shows branch > coverage, you > can find branches that don't seem to be executed. > >> -----Original Message----- >> From: [hidden email] [mailto:squeak- >> [hidden email]] On Behalf Of Frank Shearar >> Sent: Tuesday, April 09, 2013 2:07 AM >> To: Squeak Virtual Machine Development Discussion >> Cc: The general-purpose Squeak developers list; Pharo Development >> Subject: [squeak-dev] Re: [Vm-dev] Re: [Pharo-project] [Preview] Branch >> test coverage analysis tool for Pharo 3.0 :) >> >> On 9 April 2013 02:02, Eliot Miranda <[hidden email]> wrote: >>> >>> >>> >>> On Mon, Apr 8, 2013 at 9:32 AM, Alexandre Bergel >> <[hidden email]> wrote: >>>> >>>> What I had in mind, is something like: >>>> -=-=-=-=-=-=-=-=-=-= >>>> myMethodWithBranch >>>> self foo. >>>> v >>>> ifTrue: [ ... ] >>>> ifFalse: [ ... ] >>>> -=-=-=-=-=-=-=-=-=-= >>>> >>>> -=-=-=-=-=-=-=-=-=-= >>>> myMethodWithBranch >>>> self foo. >>>> v >>>> ifTrue: [ self checkPoint: 1. ... ] >>>> ifFalse: [ self checkPoint: 2. ... ]. >>>> self checkPoint: 3. >>>> -=-=-=-=-=-=-=-=-=-= >>>> >>>> All these checkpoint will be inserted by manipulating the bytecode. You >> can then optimize a bit by removing checkpoints that have been reached. >>>> The method #checkPoint: could then use some attributes to the >> CompiledMethod. >>>> >>>> This is a common technique when (i) the VM does not give you what you >> need and (ii) when you do not want to change the VM. >>> >>> >>> I just implemented something to this effect for our project at >>> Cadence. It works by >>> - an assembler/disassebler that can convert a method into a sequence >>> of messages, one for each of its bytecodes and back >>> - using the disassembler to obtain labels which occur at the start of >>> each basic block >>> - replacing the first bytecode in every basic block with an illegal >>> bytecode, remembering the pc and its correct bytecode in the method >>> properties >>> - implementing a method on MethodContext which receives the >>> unknowBytecode message sent by the VM when it encounters an illegal >>> bytecode >>> - the method replaces the illegal bytecode with the correct bytecode, >>> removing the entry from properties, and continues >>> >>> The JIT refuses to compile methods that contain illegal bytecodes so > this >> runs at interpreter speed until a method no longer contains illegal > bytecodes >> (has been completely covered). But the illegal bytecode is executed at > most >> once for each basic block, since it is replaced immediately it is > executed. The >> information remaining in properties is that of the basic blocks that have > not >> been executed. >>> >>> In the Squeak/Pharo bytecode set there are three unknown bytecodes, >> 126, 127 & 139. I have a Monticello package and tests if you're > interested. >> >> I most definitely am! >> >> frank >> >>>> This is something I wanted to do for year in Hapao... >>>> >>>> Alexandre >>>> >>>> >>>> >>>> On Apr 7, 2013, at 3:21 AM, Clément Bera <[hidden email]> >> wrote: >>>> >>>> I don't know. The main problem is that the AST-interpreter is not fast >> enough. The best solution for speed would be to improve it as they do in > this >> paper : http://dl.acm.org/citation.cfm?id=2384587 (their ast-interpreter > is >> faster than a byte code interpreter). >>>> >>>> Another solution would be to execute some method with the vm and >> other one with the ast-interpreter, but it is quite some work. >>>> >>>> >>>> 2013/4/7 Alexandre Bergel <[hidden email]> Would it be >>>> possible to annotate the method with some checkpoints within the >> method? This should be fast enough I guess. >>>> >>>> Alexandre >>>> >>>> >>>> On Apr 6, 2013, at 10:35 AM, Clément Bera <[hidden email]> >> wrote: >>>> >>>>> This is exactly that. The tree coverage tool is a subclass of AST- >> interpreter, which adding to the interpretation marks the nodes. And then > I >> use this marking to color the source code. And I use as you said RB ast. >>>>> >>>>> This is why I implemented it in a few hours, I had already an AST- >> interpreter, so I've just created a quick subclass and a spec UI. >>>>> >>>>> >>>>> 2013/4/6 Frank Shearar <[hidden email]> So if you don't >>>>> mind me being very loose with terminology, you interpret (RB) ASTs, >>>>> and the act of interpreting marks the nodes in these ASTs, which >>>>> you can then use to colour source? >>>>> >>>>> frank >>>>> >>>>> On 6 April 2013 11:00, Clément Bera <[hidden email]> wrote: >>>>>> Actually, it is quite different than Jejak. >>>>>> >>>>>> Implementation wise, Jejak used Opal so I guess it relies on >>>>>> bytecode modification to do its analysis where my tool relies on > ast >> annotations. >>>>>> >>>>>> This implies some difference in features. It seems that Jejak >>>>>> record all calls, all values and all assignments in a method >>>>>> (which is byte code level feature), whereas I record for each ast >>>>>> node if it was interpreted or not (which is ast level feature). I >>>>>> guess there would be a little difference in the use cases. >>>>>> >>>>>> >>>>>> 2013/4/6 Frank Shearar <[hidden email]> >>>>>>> >>>>>>> On 6 April 2013 07:47, Stéphane Ducasse >> <[hidden email]> wrote: >>>>>>>> >>>>>>>> Clement built in a couple of hours the following nice branch >>>>>>>> analysecoverage tools :) It is dead slow but cool. >>>>>>> >>>>>>> >>>>>>> By "branch coverage tool" do you mean Jejak? >>>>>>> >>>>>>> frank >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Clément Béra >>>>>> Mate Virtual Machine Engineer >>>>>> Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Clément Béra >>>>> Mate Virtual Machine Engineer >>>>> Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq >>>> >>>> -- >>>> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: >>>> Alexandre Bergel http://www.bergel.eu >>>> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Clément Béra >>>> Mate Virtual Machine Engineer >>>> Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq >>>> >>>> >>>> -- >>>> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: >>>> Alexandre Bergel http://www.bergel.eu >>>> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. >>> >>> >>> >>> -- >>> best, >>> Eliot > > > > |
«
Return to Squeak - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |