Aboud CI test retries

Hi all,

Initially the pharo build had tests that were failing randomly.

To cope with that, we introduced a retry of the tests.

Nowadays, this situation is actually very rare. Tests that fail, fail always, and randomly failing tests are not seen so often... This means however, that in the case that a test is persistently failing, we are (uselessly) retrying it, and making jobs take 10-15 minutes longer for nothing.

I propose that we remove the retries.

 - This will speed up the builds that are green and only penalize those that are not green.

 - Remove stress from our servers (that we use to have a higher ratio builds/hour :))

 - Randomly failing tests will just need to manually retry the build. But since green builds take now ~15-20 minutes, which is in the same order of magnitude of the retries, we only penalize the one that found the hiccup.

On 3 May 2018 at 23:35, Guillermo Polito <[hidden email]> wrote:

Hi all,

Initially the pharo build had tests that were failing randomly.

To cope with that, we introduced a retry of the tests.

Nowadays, this situation is actually very rare. Tests that fail, fail always, and randomly failing tests are not seen so often... This means however, that in the case that a test is persistently failing, we are (uselessly) retrying it, and making jobs take 10-15 minutes longer for nothing.

I'm curious...

are all tests retried, or only the failing one?

 

I propose that we remove the retries.

 - This will speed up the builds that are green and only penalize those that are not green.

If all tests pass first time there should be no retries and such a green build 

should take the same time regardless whether retries are enabled or not.... ?

 

 - Remove stress from our servers (that we use to have a higher ratio builds/hour :))

 - Randomly failing tests will just need to manually retry the build. But since green builds take now ~15-20 minutes, which is in the same order of magnitude of the retries, we only penalize the one that found the hiccup.

The "really annoying" random failures are just single failed test.

Perhaps assume if the count of failed tests is more than ten, then its not a "random" failure and immediately fail that job.

Rerunning max ten tests shouldn't add much to job time.  

But maybe the benefit isn't worth the added complexity to do it like that.

cheers -ben

On Fri, May 4, 2018 at 10:20 AM, Ben Coman <[hidden email]> wrote:

On 3 May 2018 at 23:35, Guillermo Polito <[hidden email]> wrote:

Hi all,

Initially the pharo build had tests that were failing randomly.

To cope with that, we introduced a retry of the tests.

Nowadays, this situation is actually very rare. Tests that fail, fail always, and randomly failing tests are not seen so often... This means however, that in the case that a test is persistently failing, we are (uselessly) retrying it, and making jobs take 10-15 minutes longer for nothing.

I'm curious...

are all tests retried, or only the failing one?

All of them. 

 

I propose that we remove the retries.

 - This will speed up the builds that are green and only penalize those that are not green.

If all tests pass first time there should be no retries and such a green build 

should take the same time regardless whether retries are enabled or not.... ?

True, I don't know what I wrote there ^^.

What I meant is that builds that are not green will fail sooner.

 

 - Remove stress from our servers (that we use to have a higher ratio builds/hour :))

 - Randomly failing tests will just need to manually retry the build. But since green builds take now ~15-20 minutes, which is in the same order of magnitude of the retries, we only penalize the one that found the hiccup.

The "really annoying" random failures are just single failed test.

Perhaps assume if the count of failed tests is more than ten, then its not a "random" failure and immediately fail that job.

But how can we distinguish between a real failing test and one that is random?

 

Rerunning max ten tests shouldn't add much to job time.  

But maybe the benefit isn't worth the added complexity to do it like that.

The thing is also that we have to implement something custom for that. And I'd like to put my effort on other things that add more value in the short term...