Speedcenter setup

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Speedcenter setup

timfelgentreff
 
Hi,

I finally got around to enabling automatic nightly benchmarking for
all branches for both opensmalltalk-vm and rsqueak. Before I had to
manually trigger benchmarks for opensmalltalk-vm. Now a nightly cron
job checks for new passed builds on all branches and then tries to run
benchmarks for the resulting binaries. Note, however, that we're only
uploading binaries for the master and Cog branch right now on
opensmalltalk-vm, so all other branches are schedules but then
skipped, because no binaries are available.

The benchmarking machine is still running at this point, a full set of
benchmarks takes about 4-5 hours per binary. So I expect the results
for yesterdays VMs to come in this afternoon GMT+1. Not all of the
benchmarks are well scaled right now, so that might need some
tweaking, but since there has been talk about various performance
optimizations recently, I figured we should be more vigilant in
tracking the results of that. For example, Clement mentioned recent
optimizations that should have results in binarytrees. It's nice if we
can see that in the timeline.

This page in particular may be interesting to check once in a while:

  http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on

If anyone wants to take a look at the code and benchmarks, the about
page links to all relevant repositories. I'm happy to give people
access so they can add or change or remove benchmarks.

cheers,
Tim
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

Ben Coman
 
On Wed, Jul 27, 2016 at 2:56 PM, Tim Felgentreff
<[hidden email]> wrote:

>
> Hi,
>
> I finally got around to enabling automatic nightly benchmarking for
> all branches for both opensmalltalk-vm and rsqueak. Before I had to
> manually trigger benchmarks for opensmalltalk-vm. Now a nightly cron
> job checks for new passed builds on all branches and then tries to run
> benchmarks for the resulting binaries. Note, however, that we're only
> uploading binaries for the master and Cog branch right now on
> opensmalltalk-vm, so all other branches are schedules but then
> skipped, because no binaries are available.
>
> The benchmarking machine is still running at this point, a full set of
> benchmarks takes about 4-5 hours per binary. So I expect the results
> for yesterdays VMs to come in this afternoon GMT+1. Not all of the
> benchmarks are well scaled right now, so that might need some
> tweaking, but since there has been talk about various performance
> optimizations recently, I figured we should be more vigilant in
> tracking the results of that. For example, Clement mentioned recent
> optimizations that should have results in binarytrees. It's nice if we
> can see that in the timeline.
>
> This page in particular may be interesting to check once in a while:
>
>   http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on
>
> If anyone wants to take a look at the code and benchmarks, the about
> page links to all relevant repositories. I'm happy to give people
> access so they can add or change or remove benchmarks.
>
> cheers,
> Tim

Are they all a "less is better" comparison ?

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

timfelgentreff
 
Yes, they are. The website isn't the best, but you can click your way
through to see the actual numbers and stdev, too.

Best,
Tim

On 27 July 2016 at 10:08, Ben Coman <[hidden email]> wrote:

>
> On Wed, Jul 27, 2016 at 2:56 PM, Tim Felgentreff
> <[hidden email]> wrote:
>>
>> Hi,
>>
>> I finally got around to enabling automatic nightly benchmarking for
>> all branches for both opensmalltalk-vm and rsqueak. Before I had to
>> manually trigger benchmarks for opensmalltalk-vm. Now a nightly cron
>> job checks for new passed builds on all branches and then tries to run
>> benchmarks for the resulting binaries. Note, however, that we're only
>> uploading binaries for the master and Cog branch right now on
>> opensmalltalk-vm, so all other branches are schedules but then
>> skipped, because no binaries are available.
>>
>> The benchmarking machine is still running at this point, a full set of
>> benchmarks takes about 4-5 hours per binary. So I expect the results
>> for yesterdays VMs to come in this afternoon GMT+1. Not all of the
>> benchmarks are well scaled right now, so that might need some
>> tweaking, but since there has been talk about various performance
>> optimizations recently, I figured we should be more vigilant in
>> tracking the results of that. For example, Clement mentioned recent
>> optimizations that should have results in binarytrees. It's nice if we
>> can see that in the timeline.
>>
>> This page in particular may be interesting to check once in a while:
>>
>>   http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on
>>
>> If anyone wants to take a look at the code and benchmarks, the about
>> page links to all relevant repositories. I'm happy to give people
>> access so they can add or change or remove benchmarks.
>>
>> cheers,
>> Tim
>
> Are they all a "less is better" comparison ?
>
> cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

Eliot Miranda-2
In reply to this post by timfelgentreff
 
Hi Tim (F),

    find attached a minor cleanup of SMark.  I'd like to have remission to write to the repository if that's ok with you; I don't have it right now.

On Tue, Jul 26, 2016 at 11:56 PM, Tim Felgentreff <[hidden email]> wrote:

Hi,

I finally got around to enabling automatic nightly benchmarking for
all branches for both opensmalltalk-vm and rsqueak. Before I had to
manually trigger benchmarks for opensmalltalk-vm. Now a nightly cron
job checks for new passed builds on all branches and then tries to run
benchmarks for the resulting binaries. Note, however, that we're only
uploading binaries for the master and Cog branch right now on
opensmalltalk-vm, so all other branches are schedules but then
skipped, because no binaries are available.

The benchmarking machine is still running at this point, a full set of
benchmarks takes about 4-5 hours per binary. So I expect the results
for yesterdays VMs to come in this afternoon GMT+1. Not all of the
benchmarks are well scaled right now, so that might need some
tweaking, but since there has been talk about various performance
optimizations recently, I figured we should be more vigilant in
tracking the results of that. For example, Clement mentioned recent
optimizations that should have results in binarytrees. It's nice if we
can see that in the timeline.

This page in particular may be interesting to check once in a while:

  http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on

If anyone wants to take a look at the code and benchmarks, the about
page links to all relevant repositories. I'm happy to give people
access so they can add or change or remove benchmarks.

cheers,
Tim



--
_,,,^..^,,,_
best, Eliot

SMark-eem.78.mcz (234K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

timfelgentreff
 
Hi Eliot,

sure, I can add you, do you have an account at http://www.hpi.uni-potsdam.de/hirschfeld/squeaksource?

Best,
Tim

On Fri, 7 Oct 2016 at 18:24 Eliot Miranda <[hidden email]> wrote:
Hi Tim (F),

    find attached a minor cleanup of SMark.  I'd like to have remission to write to the repository if that's ok with you; I don't have it right now.

On Tue, Jul 26, 2016 at 11:56 PM, Tim Felgentreff <[hidden email]> wrote:

Hi,

I finally got around to enabling automatic nightly benchmarking for
all branches for both opensmalltalk-vm and rsqueak. Before I had to
manually trigger benchmarks for opensmalltalk-vm. Now a nightly cron
job checks for new passed builds on all branches and then tries to run
benchmarks for the resulting binaries. Note, however, that we're only
uploading binaries for the master and Cog branch right now on
opensmalltalk-vm, so all other branches are schedules but then
skipped, because no binaries are available.

The benchmarking machine is still running at this point, a full set of
benchmarks takes about 4-5 hours per binary. So I expect the results
for yesterdays VMs to come in this afternoon GMT+1. Not all of the
benchmarks are well scaled right now, so that might need some
tweaking, but since there has been talk about various performance
optimizations recently, I figured we should be more vigilant in
tracking the results of that. For example, Clement mentioned recent
optimizations that should have results in binarytrees. It's nice if we
can see that in the timeline.

This page in particular may be interesting to check once in a while:

  http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on

If anyone wants to take a look at the code and benchmarks, the about
page links to all relevant repositories. I'm happy to give people
access so they can add or change or remove benchmarks.

cheers,
Tim



--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

Eliot Miranda-2
 
Hi Tim,

On Mon, Oct 10, 2016 at 6:58 AM, Tim Felgentreff <[hidden email]> wrote:
Hi Eliot,

sure, I can add you, do you have an account at http://www.hpi.uni-potsdam.de/hirschfeld/squeaksource?

 Yes, Eliot Miranda, username eem

 
Best,
Tim

On Fri, 7 Oct 2016 at 18:24 Eliot Miranda <[hidden email]> wrote:
Hi Tim (F),

    find attached a minor cleanup of SMark.  I'd like to have remission to write to the repository if that's ok with you; I don't have it right now.

On Tue, Jul 26, 2016 at 11:56 PM, Tim Felgentreff <[hidden email]> wrote:

Hi,

I finally got around to enabling automatic nightly benchmarking for
all branches for both opensmalltalk-vm and rsqueak. Before I had to
manually trigger benchmarks for opensmalltalk-vm. Now a nightly cron
job checks for new passed builds on all branches and then tries to run
benchmarks for the resulting binaries. Note, however, that we're only
uploading binaries for the master and Cog branch right now on
opensmalltalk-vm, so all other branches are schedules but then
skipped, because no binaries are available.

The benchmarking machine is still running at this point, a full set of
benchmarks takes about 4-5 hours per binary. So I expect the results
for yesterdays VMs to come in this afternoon GMT+1. Not all of the
benchmarks are well scaled right now, so that might need some
tweaking, but since there has been talk about various performance
optimizations recently, I figured we should be more vigilant in
tracking the results of that. For example, Clement mentioned recent
optimizations that should have results in binarytrees. It's nice if we
can see that in the timeline.

This page in particular may be interesting to check once in a while:

  http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on

If anyone wants to take a look at the code and benchmarks, the about
page links to all relevant repositories. I'm happy to give people
access so they can add or change or remove benchmarks.

cheers,
Tim



--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

timfelgentreff
 
Ok, added you and uploaded your update. 

The servers were offline for a while and I was out of the office so I didn't notice, but they're back and right now are running benchmarks for a build from Oct 4.

cheers,
Tim

On Mon, 10 Oct 2016 at 18:46 Eliot Miranda <[hidden email]> wrote:
Hi Tim,

On Mon, Oct 10, 2016 at 6:58 AM, Tim Felgentreff <[hidden email]> wrote:
Hi Eliot,

sure, I can add you, do you have an account at http://www.hpi.uni-potsdam.de/hirschfeld/squeaksource?

 Yes, Eliot Miranda, username eem

 
Best,
Tim

On Fri, 7 Oct 2016 at 18:24 Eliot Miranda <[hidden email]> wrote:
Hi Tim (F),

    find attached a minor cleanup of SMark.  I'd like to have remission to write to the repository if that's ok with you; I don't have it right now.

On Tue, Jul 26, 2016 at 11:56 PM, Tim Felgentreff <[hidden email]> wrote:

Hi,

I finally got around to enabling automatic nightly benchmarking for
all branches for both opensmalltalk-vm and rsqueak. Before I had to
manually trigger benchmarks for opensmalltalk-vm. Now a nightly cron
job checks for new passed builds on all branches and then tries to run
benchmarks for the resulting binaries. Note, however, that we're only
uploading binaries for the master and Cog branch right now on
opensmalltalk-vm, so all other branches are schedules but then
skipped, because no binaries are available.

The benchmarking machine is still running at this point, a full set of
benchmarks takes about 4-5 hours per binary. So I expect the results
for yesterdays VMs to come in this afternoon GMT+1. Not all of the
benchmarks are well scaled right now, so that might need some
tweaking, but since there has been talk about various performance
optimizations recently, I figured we should be more vigilant in
tracking the results of that. For example, Clement mentioned recent
optimizations that should have results in binarytrees. It's nice if we
can see that in the timeline.

This page in particular may be interesting to check once in a while:

  http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on

If anyone wants to take a look at the code and benchmarks, the about
page links to all relevant repositories. I'm happy to give people
access so they can add or change or remove benchmarks.

cheers,
Tim



--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

Eliot Miranda-2
 
Thanks!

On Mon, Oct 10, 2016 at 10:18 AM, Tim Felgentreff <[hidden email]> wrote:
Ok, added you and uploaded your update. 

The servers were offline for a while and I was out of the office so I didn't notice, but they're back and right now are running benchmarks for a build from Oct 4.

cheers,
Tim

On Mon, 10 Oct 2016 at 18:46 Eliot Miranda <[hidden email]> wrote:
Hi Tim,

On Mon, Oct 10, 2016 at 6:58 AM, Tim Felgentreff <[hidden email]> wrote:
Hi Eliot,

sure, I can add you, do you have an account at http://www.hpi.uni-potsdam.de/hirschfeld/squeaksource?

 Yes, Eliot Miranda, username eem

 
Best,
Tim

On Fri, 7 Oct 2016 at 18:24 Eliot Miranda <[hidden email]> wrote:
Hi Tim (F),

    find attached a minor cleanup of SMark.  I'd like to have remission to write to the repository if that's ok with you; I don't have it right now.

On Tue, Jul 26, 2016 at 11:56 PM, Tim Felgentreff <[hidden email]> wrote:

Hi,

I finally got around to enabling automatic nightly benchmarking for
all branches for both opensmalltalk-vm and rsqueak. Before I had to
manually trigger benchmarks for opensmalltalk-vm. Now a nightly cron
job checks for new passed builds on all branches and then tries to run
benchmarks for the resulting binaries. Note, however, that we're only
uploading binaries for the master and Cog branch right now on
opensmalltalk-vm, so all other branches are schedules but then
skipped, because no binaries are available.

The benchmarking machine is still running at this point, a full set of
benchmarks takes about 4-5 hours per binary. So I expect the results
for yesterdays VMs to come in this afternoon GMT+1. Not all of the
benchmarks are well scaled right now, so that might need some
tweaking, but since there has been talk about various performance
optimizations recently, I figured we should be more vigilant in
tracking the results of that. For example, Clement mentioned recent
optimizations that should have results in binarytrees. It's nice if we
can see that in the timeline.

This page in particular may be interesting to check once in a while:

  http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on

If anyone wants to take a look at the code and benchmarks, the about
page links to all relevant repositories. I'm happy to give people
access so they can add or change or remove benchmarks.

cheers,
Tim



--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

Eliot Miranda-2
In reply to this post by timfelgentreff
 
Hi Tim,

On Tue, Jul 26, 2016 at 11:56 PM, Tim Felgentreff <[hidden email]> wrote:

Hi,

I finally got around to enabling automatic nightly benchmarking for
all branches for both opensmalltalk-vm and rsqueak. Before I had to
manually trigger benchmarks for opensmalltalk-vm. Now a nightly cron
job checks for new passed builds on all branches and then tries to run
benchmarks for the resulting binaries. Note, however, that we're only
uploading binaries for the master and Cog branch right now on
opensmalltalk-vm, so all other branches are schedules but then
skipped, because no binaries are available.

The benchmarking machine is still running at this point, a full set of
benchmarks takes about 4-5 hours per binary. So I expect the results
for yesterdays VMs to come in this afternoon GMT+1. Not all of the
benchmarks are well scaled right now, so that might need some
tweaking, but since there has been talk about various performance
optimizations recently, I figured we should be more vigilant in
tracking the results of that. For example, Clement mentioned recent
optimizations that should have results in binarytrees. It's nice if we
can see that in the timeline.

This page in particular may be interesting to check once in a while:

  http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on

Would it be possible to add so me indication to the page of what is faster?  From the graphs I can't tell if higher is faster or longer.  There are no labels :-(

If anyone wants to take a look at the code and benchmarks, the about
page links to all relevant repositories. I'm happy to give people
access so they can add or change or remove benchmarks.

cheers,
Tim

_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

Levente Uzonyi
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

Bert Freudenberg
In reply to this post by Eliot Miranda-2
 
On Mon, Oct 10, 2016 at 7:40 PM, Eliot Miranda <[hidden email]> wrote:
 
On Tue, Jul 26, 2016 at 11:56 PM, Tim Felgentreff <[hidden email]> wrote:

  http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on

Would it be possible to add so me indication to the page of what is faster?  From the graphs I can't tell if higher is faster or longer.  There are no labels :-(

If you click on one of the tiny charts you get a full chart with labels. In all of the benchmarks, "up" means "more seconds" means "slower".

- Bert -
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

timfelgentreff
 
Indeed, "up" is "slower", but I will think how to add some indication of that the overview page nonetheless. I guess it isn't in there by default, because the speed center website is agnostic to what kinds of benchmarks you use, and writing it on each tiny chart may look messy. But I'll try it out and see how it looks :)

cheers,
Tim



On Mon, 10 Oct 2016 at 20:26 Bert Freudenberg <[hidden email]> wrote:
On Mon, Oct 10, 2016 at 7:40 PM, Eliot Miranda <[hidden email]> wrote:
On Tue, Jul 26, 2016 at 11:56 PM, Tim Felgentreff <[hidden email]> wrote:

  http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on

Would it be possible to add so me indication to the page of what is faster?  From the graphs I can't tell if higher is faster or longer.  There are no labels :-(

If you click on one of the tiny charts you get a full chart with labels. In all of the benchmarks, "up" means "more seconds" means "slower".

- Bert -
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

timfelgentreff
 
One issue we might want to discuss is the selection and sizing of the benchmarks.

All benchmarks are autosized to run at least 600ms right now, and then those 600ms runs are repeated up to 100 times to obtain measurements. But 600ms might be too short for some benchmarks, if the GC only kicks in rarely for them, I don't know.

About the selection of benchmarks, the ToolInteraction went up, but that is a very high level benchmark, and might also be influenced heavily by refactorings in Morphic. So maybe the benchmark isn't all that useful, or I should stop tracking trunk with the benchmark images, and instead stay on the release.

What do you think?

On Mon, 10 Oct 2016 at 21:48 Tim Felgentreff <[hidden email]> wrote:
Indeed, "up" is "slower", but I will think how to add some indication of that the overview page nonetheless. I guess it isn't in there by default, because the speed center website is agnostic to what kinds of benchmarks you use, and writing it on each tiny chart may look messy. But I'll try it out and see how it looks :)

cheers,
Tim



On Mon, 10 Oct 2016 at 20:26 Bert Freudenberg <[hidden email]> wrote:
On Mon, Oct 10, 2016 at 7:40 PM, Eliot Miranda <[hidden email]> wrote:
On Tue, Jul 26, 2016 at 11:56 PM, Tim Felgentreff <[hidden email]> wrote:

  http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on

Would it be possible to add so me indication to the page of what is faster?  From the graphs I can't tell if higher is faster or longer.  There are no labels :-(

If you click on one of the tiny charts you get a full chart with labels. In all of the benchmarks, "up" means "more seconds" means "slower".

- Bert -
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

Levente Uzonyi
 
If you want to benchmark the VM, then the image should be fix.
If you want to benchmark the system, then keep updating the image.
The best would be to have both.

Levente
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

Eliot Miranda-2
In reply to this post by timfelgentreff
 
Hi Tim,

On Mon, Oct 10, 2016 at 12:53 PM, Tim Felgentreff <[hidden email]> wrote:
 
One issue we might want to discuss is the selection and sizing of the benchmarks.

All benchmarks are autosized to run at least 600ms right now, and then those 600ms runs are repeated up to 100 times to obtain measurements. But 600ms might be too short for some benchmarks, if the GC only kicks in rarely for them, I don't know.

600ms is simply too short.  Back in the day when nfib was a popular activation benchmark used to compare different languages (and it's still useful for this today) the approach was to increase the argument to nfib until the result took 30 seconds or more to run.  To get activations per second one divided the result of nfib by the time taken.
 

About the selection of benchmarks, the ToolInteraction went up, but that is a very high level benchmark, and might also be influenced heavily by refactorings in Morphic. So maybe the benchmark isn't all that useful, or I should stop tracking trunk with the benchmark images, and instead stay on the release.

The original Smalltalk-80 benchmark suite unclouded senders and implementors whose performance depended not only on the implementation but on the size of the image.  I modified the VisualWorks version to scale by the number of methods used.  Necessary if one is to measure tool performance, and measuring tool performance is useful in producing a responsive system.  But one must be careful to measure something meaningful.

What do you think?

Categorising the benchmarks and describing them well is important.  Making sure they vary by quality of implementation and not extraneous causes such as code base size, is important.  Starting off all benchmarks for a consistent state, and not running a set of benchmarks in the same image (unless one "resets" by throwing away jutted code and GCing) is important.  Repeating each benchmark some number of times (e.g. three) and taking the average or the middle, is important.  Running benchmarks on an otherwise quiet machine is important.

And when we have Sista, running the benchmarks such that the system can warm up is important.  We will want to see baseline and warm performance compared.

On Mon, 10 Oct 2016 at 21:48 Tim Felgentreff <[hidden email]> wrote:
Indeed, "up" is "slower", but I will think how to add some indication of that the overview page nonetheless. I guess it isn't in there by default, because the speed center website is agnostic to what kinds of benchmarks you use, and writing it on each tiny chart may look messy. But I'll try it out and see how it looks :)

cheers,
Tim



On Mon, 10 Oct 2016 at 20:26 Bert Freudenberg <[hidden email]> wrote:
On Mon, Oct 10, 2016 at 7:40 PM, Eliot Miranda <[hidden email]> wrote:
On Tue, Jul 26, 2016 at 11:56 PM, Tim Felgentreff <[hidden email]> wrote:

  http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on

Would it be possible to add so me indication to the page of what is faster?  From the graphs I can't tell if higher is faster or longer.  There are no labels :-(

If you click on one of the tiny charts you get a full chart with labels. In all of the benchmarks, "up" means "more seconds" means "slower".

- Bert -




--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Speedcenter setup

timrowledge


> On 10-10-2016, at 2:33 PM, Eliot Miranda <[hidden email]> wrote:
> But one must be careful to measure something meaningful.

Exactly. The best one-liner I know about this is “When a measure becomes a benchmark it cease to be useful measure"

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Thesaurus: Ancient reptile with a truly extensive vocabulary