Hi Nicolas.
> Do we really want to keep this kind of test?
Such benchmarks (and benchmark-like tests) should at least average over several runs and only fail as a test if something actually got slower on average. Or something like that. A single misbehaving run should not be the reason for such a test failure.
Maybe we can tweak #should:notTakeMoreThan: to evaluate the block several times? But then it cannot fail early on as it is doing now ... Hmmm...
Best,
Marcel
Am 05.01.2021 09:08:46 schrieb Nicolas Cellier <[hidden email]>:
Hi all,
sometimes, some build fail for just 1 test...
Here https://travis-ci.com/github/OpenSmalltalk/opensmalltalk-vm/jobs/468407844
a squeak.stack.v3
RenderBugz
✗ #testSetForward (7ms)
TestFailure: Block evaluation took more than the expected 0:00:00:00.004
RenderBugz(TestCase)>>assert:description:
RenderBugz(TestCase)>>should:notTakeMoreThan:
RenderBugz(TestCase)>>should:notTakeMoreThanMilliseconds:
RenderBugz>>shouldntTakeLong:
RenderBugz>>testSetForward ...shouldntTakeLong: [ t forwardDirection: 180.0 .
self assert: ( t forwardDirection = 0.0 ) ]
RenderBugz(TestCase)>>performTest
4ms, really? On C.I. infrastructure, anything can happen...
Do we really want to keep this kind of test?
We eventually could once startup performance is known (see
isLowerPerformance discussion on squeak-dev), but in the interim, I
suggest we neutralize this specific test in Smalltalk-CI.
Locked
