Squeak Full Test run - some stats

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
cbc
Reply | Threaded
Open this post in threaded view
|

Squeak Full Test run - some stats

cbc
Hi.

I'm trying to run the full test suite in Squeak, and it takes a while.  So, I've timed it and noted which parts are slow.

Full test suite in Test Runner (except the "Help System-*" tests - more on that later) took 26:21.798 - just under a 1/2 hour.

The top 6 test classes (those taking over a minute each):
  05:11.612  PureBehaviorTest
  03:15.047  DecompilerTests
  03:09.779  TraitTest
  03:01.287  TraitMethodDescriptionTest
  02:19.789  TraitCompositionTest
  01:12.013  TraitFileOutTest
So, about 2/3rds of the time on those tests.

Testing environment:
  Squeak5.2alpha
  latest update: #18145
  Image format 68021 (64 bit)
  Croquet Closure Stack VM [StackInterpreterPrimitives VMMaker.oscog-eem.2203]
  Win32 built on Apr 27 2017 03:08:19 GMT Compiler: 4.2.1 Compatible Clang 3.9.1 (tags/RELEASE_391/final)
  platform sources revision VM: 201704270225 
hardware:
  Processors: AMD FX(tm)-6300 Six-Core Processor             
Identifier: AMD64 Family 21 Model 2 Stepping 0
~MHZ: 3516
  Operating System: Windows 10 Home (Build 9200 )

Results:
  4479 run, 4426 passes, 29 expected failures, 23 failures, 0 errors, 1 unexpected passes
Issues:


So, what are the 'normal' failing tests that I should ignore, if any?

If you are interested in how the timing was done, I loaded this package (recently created for another purpose):
   MCHttpRepository
user: ''
password: ''
Load "Time Logging" (at least version cbc.2)
Load "Test logging sunit integration" (at least version cbc.1)

Then, any runs of tests in TestRunner should log the it takes.  The following commands show the results as noted above:
  (TimeLogging globalTimingLogs category: #testRunner) categoryTime. 
  (((TimeLogging globalTimingLogs category: #testRunner) slowestGroupTimes select: [:sgt| sgt second >= 1 minute]) collect: #printString) joinSeparatedBy: String cr.

* About the "Help-System-*" tests, they manage to block the TestRunner with a failure in showing one of the UI's, which after moving a window or two locked up the image. Not nice.  Will follow up in another message.

Also, why does Test Runner not keep a progress bar showing all of the time?  It is unnerving now knowing if it is running or not - several times it just seems locked while it is doing things in the background.

Thanks,
cbc


Reply | Threaded
Open this post in threaded view
|

Re: Squeak Full Test run - some stats

Levente Uzonyi
Is that a Stack VM? That could explain why it takes more than just a few
minutes to run the full test suite.

Levente

On Fri, 13 Jul 2018, Chris Cunningham wrote:

> Hi.
> I'm trying to run the full test suite in Squeak, and it takes a while.  So, I've timed it and noted which parts are slow.
>
> Full test suite in Test Runner (except the "Help System-*" tests - more on that later) took 26:21.798 - just under a 1/2 hour.
>
> The top 6 test classes (those taking over a minute each):
>   05:11.612  PureBehaviorTest
>   03:15.047  DecompilerTests
>   03:09.779  TraitTest
>   03:01.287  TraitMethodDescriptionTest
>   02:19.789  TraitCompositionTest
>   01:12.013  TraitFileOutTest
> So, about 2/3rds of the time on those tests.
>
> Testing environment:
>   Squeak5.2alpha
>   latest update: #18145
>   Image format 68021 (64 bit)
>   Croquet Closure Stack VM [StackInterpreterPrimitives VMMaker.oscog-eem.2203]
>   Win32 built on Apr 27 2017 03:08:19 GMT Compiler: 4.2.1 Compatible Clang 3.9.1 (tags/RELEASE_391/final)
>   platform sources revision VM: 201704270225 
> hardware:
>   Processors: AMD FX(tm)-6300 Six-Core Processor             
> Identifier: AMD64 Family 21 Model 2 Stepping 0
> ~MHZ: 3516
>   Operating System: Windows 10 Home (Build 9200 )
>
> Results:
>   4479 run, 4426 passes, 29 expected failures, 23 failures, 0 errors, 1 unexpected passes
> Issues:
> [IMAGE]
>
> So, what are the 'normal' failing tests that I should ignore, if any?
>
> If you are interested in how the timing was done, I loaded this package (recently created for another purpose):
>    MCHttpRepository
> location: 'http://www.squeaksource.com/CBCTimeLogging'
> user: ''
> password: ''
> Load "Time Logging" (at least version cbc.2)
> Load "Test logging sunit integration" (at least version cbc.1)
>
> Then, any runs of tests in TestRunner should log the it takes.  The following commands show the results as noted above:
>   (TimeLogging globalTimingLogs category: #testRunner) categoryTime. 
>   (((TimeLogging globalTimingLogs category: #testRunner) slowestGroupTimes select: [:sgt| sgt second >= 1 minute]) collect: #printString) joinSeparatedBy: String cr.
>
> * About the "Help-System-*" tests, they manage to block the TestRunner with a failure in showing one of the UI's, which after moving a window or two locked up the image. Not nice.  Will follow up in another
> message.
>
> Also, why does Test Runner not keep a progress bar showing all of the time?  It is unnerving now knowing if it is running or not - several times it just seems locked while it is doing things in the
> background.
>
> Thanks,
> cbc
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Squeak Full Test run - some stats

Edgar De Cleene
In reply to this post by cbc
Re: [squeak-dev] Squeak Full Test run - some stats Very thanks for this.


Edgar
@morplenauta





cbc
Reply | Threaded
Open this post in threaded view
|

Re: Squeak Full Test run - some stats

cbc
In reply to this post by Levente Uzonyi
Yes, stack.  It was the VM that came with the image I downloaded (months ago) for dev.

Using a more recent VM (cog/spur):
  Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2361]
  Win32 built on Apr  3 2018 17:24:23 GMT Compiler: 4.2.1 Compatible Clang 5.0.1 (tags/RELEASE_501/final)

Tests: 18:35.123
Slowest:
  05:24.681  PureBehaviorTest
  03:10.968  TraitTest
  02:32.973  TraitMethodDescriptionTest
  01:50.439  TraitCompositionTest
I find it interesting that these tests didn't really improve at all. On the other hand, Decompiler test went from 3 minutes to under a minute - it is sensitive to the VM, apparently (i.e., faster with newer VM's).

Oh, and the number of failures increased DRAMATICALLY:
4528 run, 4311 passes, 32 expected failures, 30 failures, 154 errors, 1 unexpected passes
(the stack VM had  "4479 run, 4426 passes, 29 expected failures, 23 failures, 0 errors, 1 unexpected passes".  It seems to have run 150+ new tests over the last run, and while I added the HelpSystem tests back in, those were only 13 total and all passed.  I would guess that the extra tests were previous filtered out for Windows, but that stable VM let them run, maybe).

Will run with latest VM in a bit, and see if I can find out more about the additional errors. That worries me.

-cbc

On Sat, Jul 14, 2018 at 2:58 AM, Levente Uzonyi <[hidden email]> wrote:
Is that a Stack VM? That could explain why it takes more than just a few minutes to run the full test suite.

Levente


On Fri, 13 Jul 2018, Chris Cunningham wrote:

Hi.
I'm trying to run the full test suite in Squeak, and it takes a while.  So, I've timed it and noted which parts are slow.

Full test suite in Test Runner (except the "Help System-*" tests - more on that later) took 26:21.798 - just under a 1/2 hour.

The top 6 test classes (those taking over a minute each):
  05:11.612  PureBehaviorTest
  03:15.047  DecompilerTests
  03:09.779  TraitTest
  03:01.287  TraitMethodDescriptionTest
  02:19.789  TraitCompositionTest
  01:12.013  TraitFileOutTest
So, about 2/3rds of the time on those tests.

Testing environment:
  Squeak5.2alpha
  latest update: #18145
  Image format 68021 (64 bit)
  Croquet Closure Stack VM [StackInterpreterPrimitives VMMaker.oscog-eem.2203]
  Win32 built on Apr 27 2017 03:08:19 GMT Compiler: 4.2.1 Compatible Clang 3.9.1 (tags/RELEASE_391/final)
  platform sources revision VM: 201704270225 
hardware:
  Processors: AMD FX(tm)-6300 Six-Core Processor             
Identifier: AMD64 Family 21 Model 2 Stepping 0
~MHZ: 3516
  Operating System: Windows 10 Home (Build 9200 )

Results:
  4479 run, 4426 passes, 29 expected failures, 23 failures, 0 errors, 1 unexpected passes
Issues:
[IMAGE]

So, what are the 'normal' failing tests that I should ignore, if any?

If you are interested in how the timing was done, I loaded this package (recently created for another purpose):
   MCHttpRepository
location: 'http://www.squeaksource.com/CBCTimeLogging'
user: ''
password: ''
Load "Time Logging" (at least version cbc.2)
Load "Test logging sunit integration" (at least version cbc.1)

Then, any runs of tests in TestRunner should log the it takes.  The following commands show the results as noted above:
  (TimeLogging globalTimingLogs category: #testRunner) categoryTime. 
  (((TimeLogging globalTimingLogs category: #testRunner) slowestGroupTimes select: [:sgt| sgt second >= 1 minute]) collect: #printString) joinSeparatedBy: String cr.

* About the "Help-System-*" tests, they manage to block the TestRunner with a failure in showing one of the UI's, which after moving a window or two locked up the image. Not nice.  Will follow up in another
message.

Also, why does Test Runner not keep a progress bar showing all of the time?  It is unnerving now knowing if it is running or not - several times it just seems locked while it is doing things in the
background.

Thanks,
cbc







cbc
Reply | Threaded
Open this post in threaded view
|

Re: Squeak Full Test run - some stats

cbc
Odd. With the latest cog/spur/win64x64

Test: 21:06.89
Slowest:
  05:24.970 PureBehaviorTest
  03:17.736 TraitTest
  03:02.790 TraitMethodDescriptionTest
  02:05.559 TraitCompositionTest
  01:07.887 TraitFileOutTest

So 3 minutes slower, although decompiler test are still significantly faster.

On the good side, the errors went away:
4494 run, 4442 passes, 29 expected failures, 22 failures, 0 errors, 1 unexpected passes
(probably because there were 134 less tests ran?)

Thanks,
cbc

On Sat, Jul 14, 2018 at 9:40 AM, Chris Cunningham <[hidden email]> wrote:
Yes, stack.  It was the VM that came with the image I downloaded (months ago) for dev.

Using a more recent VM (cog/spur):
  Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2361]
  Win32 built on Apr  3 2018 17:24:23 GMT Compiler: 4.2.1 Compatible Clang 5.0.1 (tags/RELEASE_501/final)

Tests: 18:35.123
Slowest:
  05:24.681  PureBehaviorTest
  03:10.968  TraitTest
  02:32.973  TraitMethodDescriptionTest
  01:50.439  TraitCompositionTest
I find it interesting that these tests didn't really improve at all. On the other hand, Decompiler test went from 3 minutes to under a minute - it is sensitive to the VM, apparently (i.e., faster with newer VM's).

Oh, and the number of failures increased DRAMATICALLY:
4528 run, 4311 passes, 32 expected failures, 30 failures, 154 errors, 1 unexpected passes
(the stack VM had  "4479 run, 4426 passes, 29 expected failures, 23 failures, 0 errors, 1 unexpected passes".  It seems to have run 150+ new tests over the last run, and while I added the HelpSystem tests back in, those were only 13 total and all passed.  I would guess that the extra tests were previous filtered out for Windows, but that stable VM let them run, maybe).

Will run with latest VM in a bit, and see if I can find out more about the additional errors. That worries me.

-cbc

On Sat, Jul 14, 2018 at 2:58 AM, Levente Uzonyi <[hidden email]> wrote:
Is that a Stack VM? That could explain why it takes more than just a few minutes to run the full test suite.

Levente


On Fri, 13 Jul 2018, Chris Cunningham wrote:

Hi.
I'm trying to run the full test suite in Squeak, and it takes a while.  So, I've timed it and noted which parts are slow.

Full test suite in Test Runner (except the "Help System-*" tests - more on that later) took 26:21.798 - just under a 1/2 hour.

The top 6 test classes (those taking over a minute each):
  05:11.612  PureBehaviorTest
  03:15.047  DecompilerTests
  03:09.779  TraitTest
  03:01.287  TraitMethodDescriptionTest
  02:19.789  TraitCompositionTest
  01:12.013  TraitFileOutTest
So, about 2/3rds of the time on those tests.

Testing environment:
  Squeak5.2alpha
  latest update: #18145
  Image format 68021 (64 bit)
  Croquet Closure Stack VM [StackInterpreterPrimitives VMMaker.oscog-eem.2203]
  Win32 built on Apr 27 2017 03:08:19 GMT Compiler: 4.2.1 Compatible Clang 3.9.1 (tags/RELEASE_391/final)
  platform sources revision VM: 201704270225 
hardware:
  Processors: AMD FX(tm)-6300 Six-Core Processor             
Identifier: AMD64 Family 21 Model 2 Stepping 0
~MHZ: 3516
  Operating System: Windows 10 Home (Build 9200 )

Results:
  4479 run, 4426 passes, 29 expected failures, 23 failures, 0 errors, 1 unexpected passes
Issues:
[IMAGE]

So, what are the 'normal' failing tests that I should ignore, if any?

If you are interested in how the timing was done, I loaded this package (recently created for another purpose):
   MCHttpRepository
location: 'http://www.squeaksource.com/CBCTimeLogging'
user: ''
password: ''
Load "Time Logging" (at least version cbc.2)
Load "Test logging sunit integration" (at least version cbc.1)

Then, any runs of tests in TestRunner should log the it takes.  The following commands show the results as noted above:
  (TimeLogging globalTimingLogs category: #testRunner) categoryTime. 
  (((TimeLogging globalTimingLogs category: #testRunner) slowestGroupTimes select: [:sgt| sgt second >= 1 minute]) collect: #printString) joinSeparatedBy: String cr.

* About the "Help-System-*" tests, they manage to block the TestRunner with a failure in showing one of the UI's, which after moving a window or two locked up the image. Not nice.  Will follow up in another
message.

Also, why does Test Runner not keep a progress bar showing all of the time?  It is unnerving now knowing if it is running or not - several times it just seems locked while it is doing things in the
background.

Thanks,
cbc