Smalltalk › Pharo › Pharo Smalltalk Developers

16666 CI validation Failure misreported as Success

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

5 messages Options

Ben Coman

16666 CI validation Failure misreported as Success

Of the eight issues currently "Resolved (Fix Reviewed By monkey)"
which passed validation successfully, three seem to actually be failures.
That is, Fogbugz shows SUCCESS when the validation report shows FAILURE.

* 16577 Random should be in its own package
* 16634 Rename layout to classLayout (step 2 use new accessors in the tests)
* 16645 RBParser should not accept # <whitespace> and # <number>

So...

Am I misinterpreting the validation reports?

Until this is resolved, should we require validation reports be
manually checked (with a comment on the Issue) before integration?

Could futher occurances be reported in a similar way to those already on...
https://pharo.fogbugz.com/default.asp?16666

* Scrape the Fogbugz comment to the top of a text file.
* Scrape the Validation Report into the text file.
* Extract significant text into a Fogbugz comment in 16666.
* Attach text file to comment.

cheers -ben

Nicolai Hess

Re: 16666 CI validation Failure misreported as Success

2015-09-30 3:16 GMT+02:00 Ben Coman <[hidden email]>:

Of the eight issues currently "Resolved (Fix Reviewed By monkey)"
which passed validation successfully, three seem to actually be failures.
That is, Fogbugz shows SUCCESS when the validation report shows FAILURE.

* 16577 Random should be in its own package
* 16634 Rename layout to classLayout (step 2 use new accessors in the tests)
* 16645 RBParser should not accept # <whitespace> and # <number>

So...

Am I misinterpreting the validation reports?

Until this is resolved, should we require validation reports be
manually checked (with a comment on the Issue) before integration?

Could futher occurances be reported in a similar way to those already on...
https://pharo.fogbugz.com/default.asp?16666

* Scrape the Fogbugz comment to the top of a text file.
* Scrape the Validation Report into the text file.
* Extract significant text into a Fogbugz comment in 16666.
* Attach text file to comment.

cheers -ben

Oh, that is bad.

Yes, we should check the validation report manually and comment.

We can reactivate the issue, resolve again and comment if the validation succeed for real.

And redo this after every automatic validation until the cause is found (or don't integrate anything?)

demarey

Re: 16666 CI validation Failure misreported as Success

In reply to this post by Ben Coman

Le 30 sept. 2015 à 03:16, Ben Coman a écrit :

> Of the eight issues currently "Resolved (Fix Reviewed By monkey)"
> which passed validation successfully, three seem to actually be failures.
> That is, Fogbugz shows SUCCESS when the validation report shows FAILURE.
>
> * 16577 Random should be in its own package

This one looks like a success both on fogbugz and on the validation report

> * 16634 Rename layout to classLayout (step 2 use new accessors in the tests)
> * 16645 RBParser should not accept # <whitespace> and # <number>

It looks like VM Crashes are not well handled

smime.p7s (5K) Download Attachment

Ben Coman

Re: 16666 CI validation Failure misreported as Success

On Wed, Sep 30, 2015 at 3:43 PM, Christophe Demarey
<[hidden email]> wrote:

>
> Le 30 sept. 2015 à 03:16, Ben Coman a écrit :
>
>> Of the eight issues currently "Resolved (Fix Reviewed By monkey)"
>> which passed validation successfully, three seem to actually be failures.
>> That is, Fogbugz shows SUCCESS when the validation report shows FAILURE.
>>
>> * 16577 Random should be in its own package
>
> This one looks like a success both on fogbugz and on the validation report
>
>> * 16634 Rename layout to classLayout (step 2 use new accessors in the tests)
>> * 16645 RBParser should not accept # <whitespace> and # <number>
>
>
> It looks like VM Crashes are not well handled

I've documented five cases now. One was a VM crash and the rest had
"startup errors"
All five failures have occurred when running on: pharo-linux.ci.inria.fr
Everything seems to always work when running on: pharo-linux64-2.ci.inria.fr

However I see a recent pharo-linux.ci.inria.fr has worked, so maybe
someone is working on it.
cheers -ben

demarey

Re: 16666 CI validation Failure misreported as Success

----- Mail original -----

> De: "Ben Coman" <[hidden email]>
> À: "Pharo Development List" <[hidden email]>
> Envoyé: Mercredi 30 Septembre 2015 14:14:06
> Objet: Re: [Pharo-dev] 16666 CI validation Failure misreported as Success
>
> I've documented five cases now. One was a VM crash and the rest had
> "startup errors"
> All five failures have occurred when running on: pharo-linux.ci.inria.fr
> Everything seems to always work when running on:
> pharo-linux64-2.ci.inria.fr

Yes, the issue is that the monkey cannot fork a worker image to check because of oof memory errors.

> However I see a recent pharo-linux.ci.inria.fr has worked, so maybe
> someone is working on it.

There were too many issues checked at the same time and they consume a lot of memory.
I restarted the slave to clean jobs.