Squeak Full Test run - some stats

Yes, stack.  It was the VM that came with the image I downloaded (months ago) for dev.

Using a more recent VM (cog/spur):

  Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2361]

  Win32 built on Apr  3 2018 17:24:23 GMT Compiler: 4.2.1 Compatible Clang 5.0.1 (tags/RELEASE_501/final)

(this is the 'stable' VM from https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/latest):

Tests: 18:35.123

Slowest:

  05:24.681  PureBehaviorTest

  03:10.968  TraitTest

  02:32.973  TraitMethodDescriptionTest

  01:50.439  TraitCompositionTest

I find it interesting that these tests didn't really improve at all. On the other hand, Decompiler test went from 3 minutes to under a minute - it is sensitive to the VM, apparently (i.e., faster with newer VM's).

Oh, and the number of failures increased DRAMATICALLY:

4528 run, 4311 passes, 32 expected failures, 30 failures, 154 errors, 1 unexpected passes

(the stack VM had  "4479 run, 4426 passes, 29 expected failures, 23 failures, 0 errors, 1 unexpected passes".  It seems to have run 150+ new tests over the last run, and while I added the HelpSystem tests back in, those were only 13 total and all passed.  I would guess that the extra tests were previous filtered out for Windows, but that stable VM let them run, maybe).

Will run with latest VM in a bit, and see if I can find out more about the additional errors. That worries me.

-cbc

On Sat, Jul 14, 2018 at 2:58 AM, Levente Uzonyi <[hidden email]> wrote:

Is that a Stack VM? That could explain why it takes more than just a few minutes to run the full test suite.

Levente

On Fri, 13 Jul 2018, Chris Cunningham wrote:

Hi.
I'm trying to run the full test suite in Squeak, and it takes a while.  So, I've timed it and noted which parts are slow.

Full test suite in Test Runner (except the "Help System-*" tests - more on that later) took 26:21.798 - just under a 1/2 hour.

The top 6 test classes (those taking over a minute each):
  05:11.612  PureBehaviorTest
  03:15.047  DecompilerTests
  03:09.779  TraitTest
  03:01.287  TraitMethodDescriptionTest
  02:19.789  TraitCompositionTest
  01:12.013  TraitFileOutTest
So, about 2/3rds of the time on those tests.

Testing environment:
  Squeak5.2alpha
  latest update: #18145
  Image format 68021 (64 bit)
  Croquet Closure Stack VM [StackInterpreterPrimitives VMMaker.oscog-eem.2203]
  Win32 built on Apr 27 2017 03:08:19 GMT Compiler: 4.2.1 Compatible Clang 3.9.1 (tags/RELEASE_391/final)
  platform sources revision VM: 201704270225 
hardware:
  Processors: AMD FX(tm)-6300 Six-Core Processor             
Identifier: AMD64 Family 21 Model 2 Stepping 0
~MHZ: 3516
  Operating System: Windows 10 Home (Build 9200 )

Results:
  4479 run, 4426 passes, 29 expected failures, 23 failures, 0 errors, 1 unexpected passes
Issues:

[IMAGE]

So, what are the 'normal' failing tests that I should ignore, if any?

If you are interested in how the timing was done, I loaded this package (recently created for another purpose):
   MCHttpRepository
location: 'http://www.squeaksource.com/CBCTimeLogging'
user: ''
password: ''
Load "Time Logging" (at least version cbc.2)
Load "Test logging sunit integration" (at least version cbc.1)

Then, any runs of tests in TestRunner should log the it takes.  The following commands show the results as noted above:
  (TimeLogging globalTimingLogs category: #testRunner) categoryTime. 
  (((TimeLogging globalTimingLogs category: #testRunner) slowestGroupTimes select: [:sgt| sgt second >= 1 minute]) collect: #printString) joinSeparatedBy: String cr.

* About the "Help-System-*" tests, they manage to block the TestRunner with a failure in showing one of the UI's, which after moving a window or two locked up the image. Not nice.  Will follow up in another
message.

Also, why does Test Runner not keep a progress bar showing all of the time?  It is unnerving now knowing if it is running or not - several times it just seems locked while it is doing things in the
background.

Thanks,
cbc