Re: Jeff Gonis is now a build.squeak.org user

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Jeff Gonis is now a build.squeak.org user

Jeff Gonis-2
Hi Everyone,

As Frank mentioned, I am indeed interested in starting to get quantified performance data up and being logged for Squeak on a regular basis.  The motivation for this comes from visiting the site arewefastyet.com where Mozilla logs the Javascript performance of Firefox, and also from listening to a talk by Lars Bak, who worked on the Self VM and V8. He stated that one of the first things he does for any of his VM projects is to create a comprehensive set of performance benchmarks that is updated with every commit, because otherwise it is too easy for performance regressions to sneak in while development is occuring.

My hope is that by offering continuous feedback about the performance of Squeak, it will help provide a bit of a catalyst for anyone who is interested to go ahead and try to chip away at the performance of image or VM code. It might also help people understand Squeak's standing in the performance realm, relative to other languages.

I figure that I'll throw my goals out there, but I must caution that I have never worked with Jenkins before, so I will have some learning ramp up time before results start showing up:

- Get the performance plugin logging a simple dummy file written out by Squeak using the CI machinery.

- Use one of the tests that I have ported from the "Benchmarks Shoot-out game" to start getting a "real" number out and updating regularly, with a nice graph showing progress

- Get all of the benchmarks from the "Benchmarks Shoot-out game" ported and running in squeak to give an somewhat healthy spread of code.

- Get benchmarks for some other areas up and running.  My inital thoughts would be to have some drawing benchmarks, so that I can quantify the performance of VPRI's Gezira graphics library vs Balloon/FormCanvas.

Stretch goals:
- Get benchmarks for other critical parts of the system, maybe stream benchmarks so if we adopt Xtreams or something else we can quantify the impact of that, maybe some parsing benchmarks for XML, etc.
- Get some benchmarks from Octane and other suites ported over
- Have the server run the benchmarks for another language such as Python, or Ruby, or maybe Javascript so we can chart Squeak's progress against them
- Make some contributions to Squeak to speed some of these benchmarks up!

Anyway, if anyone has some input or suggestions I would be grateful to receive them. If you think that my plan is sound, I would love to hear that as well.

Thanks for your time,
Jeff!



On Wed, Jan 23, 2013 at 2:46 AM, Frank Shearar <[hidden email]> wrote:
Jeff's interested in answering "How fast is Squeak today?" letting us
view trends on performance, not just on build times.

He's going to be taking the Performance plugin for a spin. This plugin
renders JMeter format output.

frank



Reply | Threaded
Open this post in threaded view
|

Re: Jeff Gonis is now a build.squeak.org user

Nicolas Cellier
+1

For shootout please use and modify (it's global read/write)
http://ss3.gemstone.com/ss/Shootout.html/
http://ss3.gemstone.com/ss/Shootout.html/Wiki

I'm sure there have been other benchmarking efforts in Pharo (don't
know the status).

Nicolas

2013/1/23 Jeff Gonis <[hidden email]>:

> Hi Everyone,
>
> As Frank mentioned, I am indeed interested in starting to get quantified
> performance data up and being logged for Squeak on a regular basis.  The
> motivation for this comes from visiting the site arewefastyet.com where
> Mozilla logs the Javascript performance of Firefox, and also from listening
> to a talk by Lars Bak, who worked on the Self VM and V8. He stated that one
> of the first things he does for any of his VM projects is to create a
> comprehensive set of performance benchmarks that is updated with every
> commit, because otherwise it is too easy for performance regressions to
> sneak in while development is occuring.
>
> My hope is that by offering continuous feedback about the performance of
> Squeak, it will help provide a bit of a catalyst for anyone who is
> interested to go ahead and try to chip away at the performance of image or
> VM code. It might also help people understand Squeak's standing in the
> performance realm, relative to other languages.
>
> I figure that I'll throw my goals out there, but I must caution that I have
> never worked with Jenkins before, so I will have some learning ramp up time
> before results start showing up:
>
> - Get the performance plugin logging a simple dummy file written out by
> Squeak using the CI machinery.
>
> - Use one of the tests that I have ported from the "Benchmarks Shoot-out
> game" to start getting a "real" number out and updating regularly, with a
> nice graph showing progress
>
> - Get all of the benchmarks from the "Benchmarks Shoot-out game" ported and
> running in squeak to give an somewhat healthy spread of code.
>
> - Get benchmarks for some other areas up and running.  My inital thoughts
> would be to have some drawing benchmarks, so that I can quantify the
> performance of VPRI's Gezira graphics library vs Balloon/FormCanvas.
>
> Stretch goals:
> - Get benchmarks for other critical parts of the system, maybe stream
> benchmarks so if we adopt Xtreams or something else we can quantify the
> impact of that, maybe some parsing benchmarks for XML, etc.
> - Get some benchmarks from Octane and other suites ported over
> - Have the server run the benchmarks for another language such as Python, or
> Ruby, or maybe Javascript so we can chart Squeak's progress against them
> - Make some contributions to Squeak to speed some of these benchmarks up!
>
> Anyway, if anyone has some input or suggestions I would be grateful to
> receive them. If you think that my plan is sound, I would love to hear that
> as well.
>
> Thanks for your time,
> Jeff!
>
>
>
> On Wed, Jan 23, 2013 at 2:46 AM, Frank Shearar <[hidden email]>
> wrote:
>>
>> Jeff's interested in answering "How fast is Squeak today?" letting us
>> view trends on performance, not just on build times.
>>
>> He's going to be taking the Performance plugin for a spin. This plugin
>> renders JMeter format output.
>>
>> frank
>
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Jeff Gonis is now a build.squeak.org user

Colin Putney-3
In reply to this post by Jeff Gonis-2



On Wed, Jan 23, 2013 at 10:02 AM, Jeff Gonis <[hidden email]> wrote:
Hi Everyone,

As Frank mentioned, I am indeed interested in starting to get quantified performance data up and being logged for Squeak on a regular basis.  The motivation for this comes from visiting the site arewefastyet.com where Mozilla logs the Javascript performance of Firefox, and also from listening to a talk by Lars Bak, who worked on the Self VM and V8. He stated that one of the first things he does for any of his VM projects is to create a comprehensive set of performance benchmarks that is updated with every commit, because otherwise it is too easy for performance regressions to sneak in while development is occuring.

My hope is that by offering continuous feedback about the performance of Squeak, it will help provide a bit of a catalyst for anyone who is interested to go ahead and try to chip away at the performance of image or VM code. It might also help people understand Squeak's standing in the performance realm, relative to other languages.

Awesome! This is a great idea. 

Colin


Reply | Threaded
Open this post in threaded view
|

Re: Jeff Gonis is now a build.squeak.org user

Paul DeBruicker
In reply to this post by Jeff Gonis-2
Jeff Gonis-2 wrote
Hi Everyone,

As Frank mentioned, I am indeed interested in starting to get quantified
performance data up and being logged for Squeak on a regular basis.

One way you could get things going is to use Statsd (to log data: https://github.com/etsy/statsd)  to feed a Graphite instance (to make web accessible graphs: http://graphite.wikidot.com/)

The statsd package for Squeak is here: http://ss3.gemstone.com/ss/Statsd.html

Then for each benchmark do something like:

result:=1 tinyBenchmarks subStrings: ' '.
bytecodes := result at: 1.
sends:= result at: 3.
Statsd setGauge:'tinyBenchmarks-bytecodes' to: bytecodes.
Statsd setGauge:'tinyBenhmarks-sends' to: sends.


The Statsd client package sends a UDP packet to the Statsd server which aggregates the data and sends it to Graphite which makes the plots.  


The languages you mention have statsd clients here https://github.com/etsy/statsd/wiki
Reply | Threaded
Open this post in threaded view
|

Re: Jeff Gonis is now a build.squeak.org user

Jeff Gonis-2
Hi Paul,

From my readings, the performance plugin for jenkins will handle generating graphs of our performance progress as is.  All I need to do is generate an output file in JMeter format.  After reading up on this format, it seems like it will be pretty easy to at least get something barebones up and logging, with the possibility to add in more extensive logging later.

I will read up on the packages you mentioned, but for now I would like to try and keep everything contained within the squeak-ci infrastructure, and leverage off the work that Frank has been doing.

Thanks for your message,
Jeff


On Wed, Jan 23, 2013 at 2:04 PM, Paul DeBruicker <[hidden email]> wrote:
Jeff Gonis-2 wrote
> Hi Everyone,
>
> As Frank mentioned, I am indeed interested in starting to get quantified
> performance data up and being logged for Squeak on a regular basis.


One way you could get things going is to use Statsd (to log data:
https://github.com/etsy/statsd)  to feed a Graphite instance (to make web
accessible graphs: http://graphite.wikidot.com/)

The statsd package for Squeak is here:
http://ss3.gemstone.com/ss/Statsd.html

Then for each benchmark do something like:

result:=1 tinyBenchmarks subStrings: ' '.
bytecodes := result at: 1.
sends:= result at: 3.
Statsd setGauge:'tinyBenchmarks-bytecodes' to: bytecodes.
Statsd setGauge:'tinyBenhmarks-sends' to: sends.


The Statsd client package sends a UDP packet to the Statsd server which
aggregates the data and sends it to Graphite which makes the plots.


The languages you mention have statsd clients here
https://github.com/etsy/statsd/wiki



--
View this message in context: http://forum.world.st/Re-Jeff-Gonis-is-now-a-build-squeak-org-user-tp4664915p4664960.html
Sent from the Squeak - Dev mailing list archive at Nabble.com.