Performance Issues

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance Issues

tesonep@gmail.com
Hi, this will be a long mail. I have organized it in different sections.

Resume:
======

- There are performance issues in Pharo 7.
- I have made benchmarks.
- With Guille we made an improvement in saving source code in methods.
- More improvements to come.
- Automation and CI in the future.


Background
=========

Since the release of Pharo 7 (and before it), it was notorious that
there exists a performance regression in the normal execution of the
image. By checking the usual operations, we have seen ( and many have
also detected) that there was an issue with the loading, compilation,
and unloading of code. Also with the creation of classes, traits and
the use of slots.

Benchmarks
=========

Although we were sure that there is a performance regression, we have
to show it in a way we can test it and measure it. If we cannot
measure it or repeat its execution it is worthless.
For doing so, I have created an initial set of micro-benchmarks to
cover normal operations of the image.

The set of benchmarks is available here:
https://github.com/tesonep/pharo-benchmarks

These benchmarks are developed on top of SMark, only adding a command
line tool and the ability to generate a CSV file.

The idea is to run the benchmarks in different versions of Pharo an
assert that we are not breaking anything.

The first results were basically a nightmare, some operations take
almost 20 times more in Pharo 7. Especially, the ones that are
compiling methods.

In the attached document, there is the detail of all the benchmarks,
the different results and the analysis of the improvements and
regressions (Positive percentages are regressions (more time),
negative are improvements (less time)).

I have checked the results in OSX with 64 bits images. But as the
problem is in pure Smalltalk implementations the problems are (and the
solutions) cross platforms.

First Improvement
==============

Having the benchmarks, it was easy to start looking for the problems.
Thanks to the help of Guille we have detected some problems in the
implementation of SourceFile. Objects of this class have the
responsibility to handle the save of the source code when a method is
modified.

Improving this implementation we have gotten to results similar to
Pharo 6 in the compilation of methods.

Comparing a stock Pharo8 image with the one with the fix, we have the
following improvements:

AddingAnInstanceVariableToAClassWithASubclass -3.96%
AddingAnInstanceVariableToAClassWithoutASubclassAndInstances -15.91%
AddingAnInstanceVariableToAClassWithoutSubclasses -28.50%
ClassAndSubclassCreation -25.67%
ClassUsingATrait -90.05%
SimpleClassCreation -26.55%
TraitCreation -92.95%
TraitCreationUsingOtherTrait -91.68%
CompileAMethodAndModifyIt -32.92%
CompileAMethodInAClassUsingATrait -83.28%
CompileAMethodInATraitNotUsed -85.12%
CompileAMethodInATraitUsed -88.16%
CompileANewMethod -40.73%
CompileANewMethodUsingInstanceVariable -33.93%
MethodAccessingAGlobal -47.57%

Again there are more details in the attached file.

Also, we have ported this fix to Pharo 7.

Next Steps
========

- Making it a part of the CI infrastructure: making it run in each PR
and build to detect regressions in the changes.

- Adding more micro and macro benchmarks. I have a list of things to
test, but I am open to more:

# Micro

- Slot Implementation
- Process handling
- Refactorings
- Files (open / write / read)

# Macro

- Loading:  Moose  / Seaside
- Recompile All
- Condense Sources
- Startup
- Shutdown

We also know that there are platform related issues (especially
Windows), so the idea it will be the same, build a benchmark, measure
it, improve it.

The idea is to have a more robust way of detecting and handling the
performance of Pharo.

Of course, I am open to all your comments.

Cheers,
Pablo

Report.xls (23K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Performance Issues

ducasse
Hi Pablo

Super super cool.
I knew that you were working on it but it is over my expectation.
You forgot to mention that this effort is supported by the consortium and by Lifeware and Schmidt support.
We will announce but now we can say it.

Stef

> On 2 Apr 2019, at 16:22, [hidden email] wrote:
>
> Hi, this will be a long mail. I have organized it in different sections.
>
> Resume:
> ======
>
> - There are performance issues in Pharo 7.
> - I have made benchmarks.
> - With Guille we made an improvement in saving source code in methods.
> - More improvements to come.
> - Automation and CI in the future.
>
>
> Background
> =========
>
> Since the release of Pharo 7 (and before it), it was notorious that
> there exists a performance regression in the normal execution of the
> image. By checking the usual operations, we have seen ( and many have
> also detected) that there was an issue with the loading, compilation,
> and unloading of code. Also with the creation of classes, traits and
> the use of slots.
>
> Benchmarks
> =========
>
> Although we were sure that there is a performance regression, we have
> to show it in a way we can test it and measure it. If we cannot
> measure it or repeat its execution it is worthless.
> For doing so, I have created an initial set of micro-benchmarks to
> cover normal operations of the image.
>
> The set of benchmarks is available here:
> https://github.com/tesonep/pharo-benchmarks
>
> These benchmarks are developed on top of SMark, only adding a command
> line tool and the ability to generate a CSV file.
>
> The idea is to run the benchmarks in different versions of Pharo an
> assert that we are not breaking anything.
>
> The first results were basically a nightmare, some operations take
> almost 20 times more in Pharo 7. Especially, the ones that are
> compiling methods.
>
> In the attached document, there is the detail of all the benchmarks,
> the different results and the analysis of the improvements and
> regressions (Positive percentages are regressions (more time),
> negative are improvements (less time)).
>
> I have checked the results in OSX with 64 bits images. But as the
> problem is in pure Smalltalk implementations the problems are (and the
> solutions) cross platforms.
>
> First Improvement
> ==============
>
> Having the benchmarks, it was easy to start looking for the problems.
> Thanks to the help of Guille we have detected some problems in the
> implementation of SourceFile. Objects of this class have the
> responsibility to handle the save of the source code when a method is
> modified.
>
> Improving this implementation we have gotten to results similar to
> Pharo 6 in the compilation of methods.
>
> Comparing a stock Pharo8 image with the one with the fix, we have the
> following improvements:
>
> AddingAnInstanceVariableToAClassWithASubclass -3.96%
> AddingAnInstanceVariableToAClassWithoutASubclassAndInstances -15.91%
> AddingAnInstanceVariableToAClassWithoutSubclasses -28.50%
> ClassAndSubclassCreation -25.67%
> ClassUsingATrait -90.05%
> SimpleClassCreation -26.55%
> TraitCreation -92.95%
> TraitCreationUsingOtherTrait -91.68%
> CompileAMethodAndModifyIt -32.92%
> CompileAMethodInAClassUsingATrait -83.28%
> CompileAMethodInATraitNotUsed -85.12%
> CompileAMethodInATraitUsed -88.16%
> CompileANewMethod -40.73%
> CompileANewMethodUsingInstanceVariable -33.93%
> MethodAccessingAGlobal -47.57%
>
> Again there are more details in the attached file.
>
> Also, we have ported this fix to Pharo 7.
>
> Next Steps
> ========
>
> - Making it a part of the CI infrastructure: making it run in each PR
> and build to detect regressions in the changes.
>
> - Adding more micro and macro benchmarks. I have a list of things to
> test, but I am open to more:
>
> # Micro
>
> - Slot Implementation
> - Process handling
> - Refactorings
> - Files (open / write / read)
>
> # Macro
>
> - Loading:  Moose  / Seaside
> - Recompile All
> - Condense Sources
> - Startup
> - Shutdown
>
> We also know that there are platform related issues (especially
> Windows), so the idea it will be the same, build a benchmark, measure
> it, improve it.
>
> The idea is to have a more robust way of detecting and handling the
> performance of Pharo.
>
> Of course, I am open to all your comments.
>
> Cheers,
> Pablo
> <Report.xls>



Reply | Threaded
Open this post in threaded view
|

Re: Performance Issues

Sven Van Caekenberghe-2
In reply to this post by tesonep@gmail.com
This is simply fantastic work !

> On 2 Apr 2019, at 16:22, [hidden email] wrote:
>
> Hi, this will be a long mail. I have organized it in different sections.
>
> Resume:
> ======
>
> - There are performance issues in Pharo 7.
> - I have made benchmarks.
> - With Guille we made an improvement in saving source code in methods.
> - More improvements to come.
> - Automation and CI in the future.
>
>
> Background
> =========
>
> Since the release of Pharo 7 (and before it), it was notorious that
> there exists a performance regression in the normal execution of the
> image. By checking the usual operations, we have seen ( and many have
> also detected) that there was an issue with the loading, compilation,
> and unloading of code. Also with the creation of classes, traits and
> the use of slots.
>
> Benchmarks
> =========
>
> Although we were sure that there is a performance regression, we have
> to show it in a way we can test it and measure it. If we cannot
> measure it or repeat its execution it is worthless.
> For doing so, I have created an initial set of micro-benchmarks to
> cover normal operations of the image.
>
> The set of benchmarks is available here:
> https://github.com/tesonep/pharo-benchmarks
>
> These benchmarks are developed on top of SMark, only adding a command
> line tool and the ability to generate a CSV file.
>
> The idea is to run the benchmarks in different versions of Pharo an
> assert that we are not breaking anything.
>
> The first results were basically a nightmare, some operations take
> almost 20 times more in Pharo 7. Especially, the ones that are
> compiling methods.
>
> In the attached document, there is the detail of all the benchmarks,
> the different results and the analysis of the improvements and
> regressions (Positive percentages are regressions (more time),
> negative are improvements (less time)).
>
> I have checked the results in OSX with 64 bits images. But as the
> problem is in pure Smalltalk implementations the problems are (and the
> solutions) cross platforms.
>
> First Improvement
> ==============
>
> Having the benchmarks, it was easy to start looking for the problems.
> Thanks to the help of Guille we have detected some problems in the
> implementation of SourceFile. Objects of this class have the
> responsibility to handle the save of the source code when a method is
> modified.
>
> Improving this implementation we have gotten to results similar to
> Pharo 6 in the compilation of methods.
>
> Comparing a stock Pharo8 image with the one with the fix, we have the
> following improvements:
>
> AddingAnInstanceVariableToAClassWithASubclass -3.96%
> AddingAnInstanceVariableToAClassWithoutASubclassAndInstances -15.91%
> AddingAnInstanceVariableToAClassWithoutSubclasses -28.50%
> ClassAndSubclassCreation -25.67%
> ClassUsingATrait -90.05%
> SimpleClassCreation -26.55%
> TraitCreation -92.95%
> TraitCreationUsingOtherTrait -91.68%
> CompileAMethodAndModifyIt -32.92%
> CompileAMethodInAClassUsingATrait -83.28%
> CompileAMethodInATraitNotUsed -85.12%
> CompileAMethodInATraitUsed -88.16%
> CompileANewMethod -40.73%
> CompileANewMethodUsingInstanceVariable -33.93%
> MethodAccessingAGlobal -47.57%
>
> Again there are more details in the attached file.
>
> Also, we have ported this fix to Pharo 7.
>
> Next Steps
> ========
>
> - Making it a part of the CI infrastructure: making it run in each PR
> and build to detect regressions in the changes.
>
> - Adding more micro and macro benchmarks. I have a list of things to
> test, but I am open to more:
>
> # Micro
>
> - Slot Implementation
> - Process handling
> - Refactorings
> - Files (open / write / read)
>
> # Macro
>
> - Loading:  Moose  / Seaside
> - Recompile All
> - Condense Sources
> - Startup
> - Shutdown
>
> We also know that there are platform related issues (especially
> Windows), so the idea it will be the same, build a benchmark, measure
> it, improve it.
>
> The idea is to have a more robust way of detecting and handling the
> performance of Pharo.
>
> Of course, I am open to all your comments.
>
> Cheers,
> Pablo
> <Report.xls>


Reply | Threaded
Open this post in threaded view
|

Re: Performance Issues

Tudor Girba-2
In reply to this post by tesonep@gmail.com
Great work!

Thanks,
Doru

> On Apr 2, 2019, at 4:22 PM, [hidden email] wrote:
>
> Hi, this will be a long mail. I have organized it in different sections.
>
> Resume:
> ======
>
> - There are performance issues in Pharo 7.
> - I have made benchmarks.
> - With Guille we made an improvement in saving source code in methods.
> - More improvements to come.
> - Automation and CI in the future.
>
>
> Background
> =========
>
> Since the release of Pharo 7 (and before it), it was notorious that
> there exists a performance regression in the normal execution of the
> image. By checking the usual operations, we have seen ( and many have
> also detected) that there was an issue with the loading, compilation,
> and unloading of code. Also with the creation of classes, traits and
> the use of slots.
>
> Benchmarks
> =========
>
> Although we were sure that there is a performance regression, we have
> to show it in a way we can test it and measure it. If we cannot
> measure it or repeat its execution it is worthless.
> For doing so, I have created an initial set of micro-benchmarks to
> cover normal operations of the image.
>
> The set of benchmarks is available here:
> https://github.com/tesonep/pharo-benchmarks
>
> These benchmarks are developed on top of SMark, only adding a command
> line tool and the ability to generate a CSV file.
>
> The idea is to run the benchmarks in different versions of Pharo an
> assert that we are not breaking anything.
>
> The first results were basically a nightmare, some operations take
> almost 20 times more in Pharo 7. Especially, the ones that are
> compiling methods.
>
> In the attached document, there is the detail of all the benchmarks,
> the different results and the analysis of the improvements and
> regressions (Positive percentages are regressions (more time),
> negative are improvements (less time)).
>
> I have checked the results in OSX with 64 bits images. But as the
> problem is in pure Smalltalk implementations the problems are (and the
> solutions) cross platforms.
>
> First Improvement
> ==============
>
> Having the benchmarks, it was easy to start looking for the problems.
> Thanks to the help of Guille we have detected some problems in the
> implementation of SourceFile. Objects of this class have the
> responsibility to handle the save of the source code when a method is
> modified.
>
> Improving this implementation we have gotten to results similar to
> Pharo 6 in the compilation of methods.
>
> Comparing a stock Pharo8 image with the one with the fix, we have the
> following improvements:
>
> AddingAnInstanceVariableToAClassWithASubclass -3.96%
> AddingAnInstanceVariableToAClassWithoutASubclassAndInstances -15.91%
> AddingAnInstanceVariableToAClassWithoutSubclasses -28.50%
> ClassAndSubclassCreation -25.67%
> ClassUsingATrait -90.05%
> SimpleClassCreation -26.55%
> TraitCreation -92.95%
> TraitCreationUsingOtherTrait -91.68%
> CompileAMethodAndModifyIt -32.92%
> CompileAMethodInAClassUsingATrait -83.28%
> CompileAMethodInATraitNotUsed -85.12%
> CompileAMethodInATraitUsed -88.16%
> CompileANewMethod -40.73%
> CompileANewMethodUsingInstanceVariable -33.93%
> MethodAccessingAGlobal -47.57%
>
> Again there are more details in the attached file.
>
> Also, we have ported this fix to Pharo 7.
>
> Next Steps
> ========
>
> - Making it a part of the CI infrastructure: making it run in each PR
> and build to detect regressions in the changes.
>
> - Adding more micro and macro benchmarks. I have a list of things to
> test, but I am open to more:
>
> # Micro
>
> - Slot Implementation
> - Process handling
> - Refactorings
> - Files (open / write / read)
>
> # Macro
>
> - Loading:  Moose  / Seaside
> - Recompile All
> - Condense Sources
> - Startup
> - Shutdown
>
> We also know that there are platform related issues (especially
> Windows), so the idea it will be the same, build a benchmark, measure
> it, improve it.
>
> The idea is to have a more robust way of detecting and handling the
> performance of Pharo.
>
> Of course, I am open to all your comments.
>
> Cheers,
> Pablo
> <Report.xls>

--
www.feenk.com

"If you can't say why something is relevant,
it probably isn't."


Reply | Threaded
Open this post in threaded view
|

Re: Performance Issues

tinchodias
Great!
SMark is a very useful package for benchmarking.

On Tue, Apr 2, 2019 at 12:06 PM Tudor Girba <[hidden email]> wrote:

>
> Great work!
>
> Thanks,
> Doru
>
> > On Apr 2, 2019, at 4:22 PM, [hidden email] wrote:
> >
> > Hi, this will be a long mail. I have organized it in different sections.
> >
> > Resume:
> > ======
> >
> > - There are performance issues in Pharo 7.
> > - I have made benchmarks.
> > - With Guille we made an improvement in saving source code in methods.
> > - More improvements to come.
> > - Automation and CI in the future.
> >
> >
> > Background
> > =========
> >
> > Since the release of Pharo 7 (and before it), it was notorious that
> > there exists a performance regression in the normal execution of the
> > image. By checking the usual operations, we have seen ( and many have
> > also detected) that there was an issue with the loading, compilation,
> > and unloading of code. Also with the creation of classes, traits and
> > the use of slots.
> >
> > Benchmarks
> > =========
> >
> > Although we were sure that there is a performance regression, we have
> > to show it in a way we can test it and measure it. If we cannot
> > measure it or repeat its execution it is worthless.
> > For doing so, I have created an initial set of micro-benchmarks to
> > cover normal operations of the image.
> >
> > The set of benchmarks is available here:
> > https://github.com/tesonep/pharo-benchmarks
> >
> > These benchmarks are developed on top of SMark, only adding a command
> > line tool and the ability to generate a CSV file.
> >
> > The idea is to run the benchmarks in different versions of Pharo an
> > assert that we are not breaking anything.
> >
> > The first results were basically a nightmare, some operations take
> > almost 20 times more in Pharo 7. Especially, the ones that are
> > compiling methods.
> >
> > In the attached document, there is the detail of all the benchmarks,
> > the different results and the analysis of the improvements and
> > regressions (Positive percentages are regressions (more time),
> > negative are improvements (less time)).
> >
> > I have checked the results in OSX with 64 bits images. But as the
> > problem is in pure Smalltalk implementations the problems are (and the
> > solutions) cross platforms.
> >
> > First Improvement
> > ==============
> >
> > Having the benchmarks, it was easy to start looking for the problems.
> > Thanks to the help of Guille we have detected some problems in the
> > implementation of SourceFile. Objects of this class have the
> > responsibility to handle the save of the source code when a method is
> > modified.
> >
> > Improving this implementation we have gotten to results similar to
> > Pharo 6 in the compilation of methods.
> >
> > Comparing a stock Pharo8 image with the one with the fix, we have the
> > following improvements:
> >
> > AddingAnInstanceVariableToAClassWithASubclass -3.96%
> > AddingAnInstanceVariableToAClassWithoutASubclassAndInstances -15.91%
> > AddingAnInstanceVariableToAClassWithoutSubclasses -28.50%
> > ClassAndSubclassCreation -25.67%
> > ClassUsingATrait -90.05%
> > SimpleClassCreation -26.55%
> > TraitCreation -92.95%
> > TraitCreationUsingOtherTrait -91.68%
> > CompileAMethodAndModifyIt -32.92%
> > CompileAMethodInAClassUsingATrait -83.28%
> > CompileAMethodInATraitNotUsed -85.12%
> > CompileAMethodInATraitUsed -88.16%
> > CompileANewMethod -40.73%
> > CompileANewMethodUsingInstanceVariable -33.93%
> > MethodAccessingAGlobal -47.57%
> >
> > Again there are more details in the attached file.
> >
> > Also, we have ported this fix to Pharo 7.
> >
> > Next Steps
> > ========
> >
> > - Making it a part of the CI infrastructure: making it run in each PR
> > and build to detect regressions in the changes.
> >
> > - Adding more micro and macro benchmarks. I have a list of things to
> > test, but I am open to more:
> >
> > # Micro
> >
> > - Slot Implementation
> > - Process handling
> > - Refactorings
> > - Files (open / write / read)
> >
> > # Macro
> >
> > - Loading:  Moose  / Seaside
> > - Recompile All
> > - Condense Sources
> > - Startup
> > - Shutdown
> >
> > We also know that there are platform related issues (especially
> > Windows), so the idea it will be the same, build a benchmark, measure
> > it, improve it.
> >
> > The idea is to have a more robust way of detecting and handling the
> > performance of Pharo.
> >
> > Of course, I am open to all your comments.
> >
> > Cheers,
> > Pablo
> > <Report.xls>
>
> --
> www.feenk.com
>
> "If you can't say why something is relevant,
> it probably isn't."
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Performance Issues

ducasse
Yes we want to use it to build a regression database :)

Stef


> On 2 Apr 2019, at 17:19, Martin Dias <[hidden email]> wrote:
>
> Great!
> SMark is a very useful package for benchmarking.
>
> On Tue, Apr 2, 2019 at 12:06 PM Tudor Girba <[hidden email]> wrote:
>>
>> Great work!
>>
>> Thanks,
>> Doru
>>
>>> On Apr 2, 2019, at 4:22 PM, [hidden email] wrote:
>>>
>>> Hi, this will be a long mail. I have organized it in different sections.
>>>
>>> Resume:
>>> ======
>>>
>>> - There are performance issues in Pharo 7.
>>> - I have made benchmarks.
>>> - With Guille we made an improvement in saving source code in methods.
>>> - More improvements to come.
>>> - Automation and CI in the future.
>>>
>>>
>>> Background
>>> =========
>>>
>>> Since the release of Pharo 7 (and before it), it was notorious that
>>> there exists a performance regression in the normal execution of the
>>> image. By checking the usual operations, we have seen ( and many have
>>> also detected) that there was an issue with the loading, compilation,
>>> and unloading of code. Also with the creation of classes, traits and
>>> the use of slots.
>>>
>>> Benchmarks
>>> =========
>>>
>>> Although we were sure that there is a performance regression, we have
>>> to show it in a way we can test it and measure it. If we cannot
>>> measure it or repeat its execution it is worthless.
>>> For doing so, I have created an initial set of micro-benchmarks to
>>> cover normal operations of the image.
>>>
>>> The set of benchmarks is available here:
>>> https://github.com/tesonep/pharo-benchmarks
>>>
>>> These benchmarks are developed on top of SMark, only adding a command
>>> line tool and the ability to generate a CSV file.
>>>
>>> The idea is to run the benchmarks in different versions of Pharo an
>>> assert that we are not breaking anything.
>>>
>>> The first results were basically a nightmare, some operations take
>>> almost 20 times more in Pharo 7. Especially, the ones that are
>>> compiling methods.
>>>
>>> In the attached document, there is the detail of all the benchmarks,
>>> the different results and the analysis of the improvements and
>>> regressions (Positive percentages are regressions (more time),
>>> negative are improvements (less time)).
>>>
>>> I have checked the results in OSX with 64 bits images. But as the
>>> problem is in pure Smalltalk implementations the problems are (and the
>>> solutions) cross platforms.
>>>
>>> First Improvement
>>> ==============
>>>
>>> Having the benchmarks, it was easy to start looking for the problems.
>>> Thanks to the help of Guille we have detected some problems in the
>>> implementation of SourceFile. Objects of this class have the
>>> responsibility to handle the save of the source code when a method is
>>> modified.
>>>
>>> Improving this implementation we have gotten to results similar to
>>> Pharo 6 in the compilation of methods.
>>>
>>> Comparing a stock Pharo8 image with the one with the fix, we have the
>>> following improvements:
>>>
>>> AddingAnInstanceVariableToAClassWithASubclass -3.96%
>>> AddingAnInstanceVariableToAClassWithoutASubclassAndInstances -15.91%
>>> AddingAnInstanceVariableToAClassWithoutSubclasses -28.50%
>>> ClassAndSubclassCreation -25.67%
>>> ClassUsingATrait -90.05%
>>> SimpleClassCreation -26.55%
>>> TraitCreation -92.95%
>>> TraitCreationUsingOtherTrait -91.68%
>>> CompileAMethodAndModifyIt -32.92%
>>> CompileAMethodInAClassUsingATrait -83.28%
>>> CompileAMethodInATraitNotUsed -85.12%
>>> CompileAMethodInATraitUsed -88.16%
>>> CompileANewMethod -40.73%
>>> CompileANewMethodUsingInstanceVariable -33.93%
>>> MethodAccessingAGlobal -47.57%
>>>
>>> Again there are more details in the attached file.
>>>
>>> Also, we have ported this fix to Pharo 7.
>>>
>>> Next Steps
>>> ========
>>>
>>> - Making it a part of the CI infrastructure: making it run in each PR
>>> and build to detect regressions in the changes.
>>>
>>> - Adding more micro and macro benchmarks. I have a list of things to
>>> test, but I am open to more:
>>>
>>> # Micro
>>>
>>> - Slot Implementation
>>> - Process handling
>>> - Refactorings
>>> - Files (open / write / read)
>>>
>>> # Macro
>>>
>>> - Loading:  Moose  / Seaside
>>> - Recompile All
>>> - Condense Sources
>>> - Startup
>>> - Shutdown
>>>
>>> We also know that there are platform related issues (especially
>>> Windows), so the idea it will be the same, build a benchmark, measure
>>> it, improve it.
>>>
>>> The idea is to have a more robust way of detecting and handling the
>>> performance of Pharo.
>>>
>>> Of course, I am open to all your comments.
>>>
>>> Cheers,
>>> Pablo
>>> <Report.xls>
>>
>> --
>> www.feenk.com
>>
>> "If you can't say why something is relevant,
>> it probably isn't."
>>
>>
>



Reply | Threaded
Open this post in threaded view
|

Re: Performance Issues

EstebanLM
In reply to this post by tesonep@gmail.com
Clap, clap, clap!
(De pie)

> On 2 Apr 2019, at 16:22, [hidden email] wrote:
>
> Hi, this will be a long mail. I have organized it in different sections.
>
> Resume:
> ======
>
> - There are performance issues in Pharo 7.
> - I have made benchmarks.
> - With Guille we made an improvement in saving source code in methods.
> - More improvements to come.
> - Automation and CI in the future.
>
>
> Background
> =========
>
> Since the release of Pharo 7 (and before it), it was notorious that
> there exists a performance regression in the normal execution of the
> image. By checking the usual operations, we have seen ( and many have
> also detected) that there was an issue with the loading, compilation,
> and unloading of code. Also with the creation of classes, traits and
> the use of slots.
>
> Benchmarks
> =========
>
> Although we were sure that there is a performance regression, we have
> to show it in a way we can test it and measure it. If we cannot
> measure it or repeat its execution it is worthless.
> For doing so, I have created an initial set of micro-benchmarks to
> cover normal operations of the image.
>
> The set of benchmarks is available here:
> https://github.com/tesonep/pharo-benchmarks
>
> These benchmarks are developed on top of SMark, only adding a command
> line tool and the ability to generate a CSV file.
>
> The idea is to run the benchmarks in different versions of Pharo an
> assert that we are not breaking anything.
>
> The first results were basically a nightmare, some operations take
> almost 20 times more in Pharo 7. Especially, the ones that are
> compiling methods.
>
> In the attached document, there is the detail of all the benchmarks,
> the different results and the analysis of the improvements and
> regressions (Positive percentages are regressions (more time),
> negative are improvements (less time)).
>
> I have checked the results in OSX with 64 bits images. But as the
> problem is in pure Smalltalk implementations the problems are (and the
> solutions) cross platforms.
>
> First Improvement
> ==============
>
> Having the benchmarks, it was easy to start looking for the problems.
> Thanks to the help of Guille we have detected some problems in the
> implementation of SourceFile. Objects of this class have the
> responsibility to handle the save of the source code when a method is
> modified.
>
> Improving this implementation we have gotten to results similar to
> Pharo 6 in the compilation of methods.
>
> Comparing a stock Pharo8 image with the one with the fix, we have the
> following improvements:
>
> AddingAnInstanceVariableToAClassWithASubclass -3.96%
> AddingAnInstanceVariableToAClassWithoutASubclassAndInstances -15.91%
> AddingAnInstanceVariableToAClassWithoutSubclasses -28.50%
> ClassAndSubclassCreation -25.67%
> ClassUsingATrait -90.05%
> SimpleClassCreation -26.55%
> TraitCreation -92.95%
> TraitCreationUsingOtherTrait -91.68%
> CompileAMethodAndModifyIt -32.92%
> CompileAMethodInAClassUsingATrait -83.28%
> CompileAMethodInATraitNotUsed -85.12%
> CompileAMethodInATraitUsed -88.16%
> CompileANewMethod -40.73%
> CompileANewMethodUsingInstanceVariable -33.93%
> MethodAccessingAGlobal -47.57%
>
> Again there are more details in the attached file.
>
> Also, we have ported this fix to Pharo 7.
>
> Next Steps
> ========
>
> - Making it a part of the CI infrastructure: making it run in each PR
> and build to detect regressions in the changes.
>
> - Adding more micro and macro benchmarks. I have a list of things to
> test, but I am open to more:
>
> # Micro
>
> - Slot Implementation
> - Process handling
> - Refactorings
> - Files (open / write / read)
>
> # Macro
>
> - Loading:  Moose  / Seaside
> - Recompile All
> - Condense Sources
> - Startup
> - Shutdown
>
> We also know that there are platform related issues (especially
> Windows), so the idea it will be the same, build a benchmark, measure
> it, improve it.
>
> The idea is to have a more robust way of detecting and handling the
> performance of Pharo.
>
> Of course, I am open to all your comments.
>
> Cheers,
> Pablo
> <Report.xls>


Reply | Threaded
Open this post in threaded view
|

Re: Performance Issues

Ben Coman
In reply to this post by tesonep@gmail.com
Thanks for this great initiative Pablo.
cheers -ben

On Tue, 2 Apr 2019 at 22:23, [hidden email] <[hidden email]> wrote:
Hi, this will be a long mail. I have organized it in different sections.

Resume:
======

- There are performance issues in Pharo 7.
- I have made benchmarks.
- With Guille we made an improvement in saving source code in methods.
- More improvements to come.
- Automation and CI in the future.

Reply | Threaded
Open this post in threaded view
|

Re: Performance Issues

Francisco Ortiz Peñaloza
This is great news! 

Thanks,
Francisco



On Wed, 3 Apr 2019 at 09:47 Ben Coman <[hidden email]> wrote:
Thanks for this great initiative Pablo.
cheers -ben

On Tue, 2 Apr 2019 at 22:23, [hidden email] <[hidden email]> wrote:
Hi, this will be a long mail. I have organized it in different sections.

Resume:
======

- There are performance issues in Pharo 7.
- I have made benchmarks.
- With Guille we made an improvement in saving source code in methods.
- More improvements to come.
- Automation and CI in the future.

--
Sent from the past
Reply | Threaded
Open this post in threaded view
|

Re: Performance Issues

Mariano Martinez Peck
In reply to this post by tesonep@gmail.com
Hola Pablito :)

That sounds amazing. Great job!!
About "Automation and CI in the future.", do you know if Smalltalk CI supports something for this? 

Thanks!

On Tue, Apr 2, 2019 at 11:22 AM [hidden email] <[hidden email]> wrote:
Hi, this will be a long mail. I have organized it in different sections.

Resume:
======

- There are performance issues in Pharo 7.
- I have made benchmarks.
- With Guille we made an improvement in saving source code in methods.
- More improvements to come.
- Automation and CI in the future.


Background
=========

Since the release of Pharo 7 (and before it), it was notorious that
there exists a performance regression in the normal execution of the
image. By checking the usual operations, we have seen ( and many have
also detected) that there was an issue with the loading, compilation,
and unloading of code. Also with the creation of classes, traits and
the use of slots.

Benchmarks
=========

Although we were sure that there is a performance regression, we have
to show it in a way we can test it and measure it. If we cannot
measure it or repeat its execution it is worthless.
For doing so, I have created an initial set of micro-benchmarks to
cover normal operations of the image.

The set of benchmarks is available here:
https://github.com/tesonep/pharo-benchmarks

These benchmarks are developed on top of SMark, only adding a command
line tool and the ability to generate a CSV file.

The idea is to run the benchmarks in different versions of Pharo an
assert that we are not breaking anything.

The first results were basically a nightmare, some operations take
almost 20 times more in Pharo 7. Especially, the ones that are
compiling methods.

In the attached document, there is the detail of all the benchmarks,
the different results and the analysis of the improvements and
regressions (Positive percentages are regressions (more time),
negative are improvements (less time)).

I have checked the results in OSX with 64 bits images. But as the
problem is in pure Smalltalk implementations the problems are (and the
solutions) cross platforms.

First Improvement
==============

Having the benchmarks, it was easy to start looking for the problems.
Thanks to the help of Guille we have detected some problems in the
implementation of SourceFile. Objects of this class have the
responsibility to handle the save of the source code when a method is
modified.

Improving this implementation we have gotten to results similar to
Pharo 6 in the compilation of methods.

Comparing a stock Pharo8 image with the one with the fix, we have the
following improvements:

AddingAnInstanceVariableToAClassWithASubclass -3.96%
AddingAnInstanceVariableToAClassWithoutASubclassAndInstances -15.91%
AddingAnInstanceVariableToAClassWithoutSubclasses -28.50%
ClassAndSubclassCreation -25.67%
ClassUsingATrait -90.05%
SimpleClassCreation -26.55%
TraitCreation -92.95%
TraitCreationUsingOtherTrait -91.68%
CompileAMethodAndModifyIt -32.92%
CompileAMethodInAClassUsingATrait -83.28%
CompileAMethodInATraitNotUsed -85.12%
CompileAMethodInATraitUsed -88.16%
CompileANewMethod -40.73%
CompileANewMethodUsingInstanceVariable -33.93%
MethodAccessingAGlobal -47.57%

Again there are more details in the attached file.

Also, we have ported this fix to Pharo 7.

Next Steps
========

- Making it a part of the CI infrastructure: making it run in each PR
and build to detect regressions in the changes.

- Adding more micro and macro benchmarks. I have a list of things to
test, but I am open to more:

# Micro

- Slot Implementation
- Process handling
- Refactorings
- Files (open / write / read)

# Macro

- Loading:  Moose  / Seaside
- Recompile All
- Condense Sources
- Startup
- Shutdown

We also know that there are platform related issues (especially
Windows), so the idea it will be the same, build a benchmark, measure
it, improve it.

The idea is to have a more robust way of detecting and handling the
performance of Pharo.

Of course, I am open to all your comments.

Cheers,
Pablo


--
Mariano Martinez Peck