Monkey validation report statistics

> Is there somewhere that statistics of failures from the CI validation
> report are gathered?
>
> We've all experienced transient CI-monkey validation failures
> unrelated to our submitted fix, where just resubmitting to the monkey
> without change works fine.  The problem is:
>
> * this may have been introduced several builds ago, but the transient
> failure passed the monkey at the time
>
> * it can be hard to identify which build introduced a transient error
>
> * doubly hard if the failure is due to some CI infrastructure change,
> or one particular build server
>
> * its hard to get an overview of how many issues get the same failure
> - is this a "once in six months" failure or is it happening 50% of the
> time in every issue.
>
> * it takes longer for an individual to observes a false-failure is
> common enough to worry about, than if reports were merged. Since
> different slices should produce different errors, failures common
> across all slices should stand out in statistics.
>
> If such statistics are not currently available, in cases where a
> simple resubmission passed the monkey, should we:
>
> * have an issue for each such false-failure, to get some early
> indication from manually track occurrences
>
> * where such an issue hasn't yet been created, first report comes to the list
>
> ------
> So, I just got the following false-failures in separate runs for:
> https://pharo.fogbugz.com/f/cases/16265
> * ProcessTerminateBug>>#testTerminationDuringUnwind
> * a CISmallIntRule: Timeout (after 0:00:40:00) occured while loading...
>
> Anyone else?
>
> cheers -ben
>
> P.S. I understand some errors are filtered. Can this list be output as
> part of the validation report (or is it already) ?
>
>

> HI ben
>
> you are right. We are all running like mad.
> We are focusing on the bootstrap because guille will leave to brussels end
> of the month.
> Esteban is fighting with FFI to get migrating to Spur.
> But yes this issue is a real one.
> Stef
>
> Le 23/8/15 04:08, Ben Coman a écrit :
>
>> Is there somewhere that statistics of failures from the CI validation
>> report are gathered?
>>
>> We've all experienced transient CI-monkey validation failures
>> unrelated to our submitted fix, where just resubmitting to the monkey
>> without change works fine.  The problem is:
>>
>> * this may have been introduced several builds ago, but the transient
>> failure passed the monkey at the time
>>
>> * it can be hard to identify which build introduced a transient error
>>
>> * doubly hard if the failure is due to some CI infrastructure change,
>> or one particular build server
>>
>> * its hard to get an overview of how many issues get the same failure
>> - is this a "once in six months" failure or is it happening 50% of the
>> time in every issue.
>>
>> * it takes longer for an individual to observes a false-failure is
>> common enough to worry about, than if reports were merged. Since
>> different slices should produce different errors, failures common
>> across all slices should stand out in statistics.
>>
>> If such statistics are not currently available, in cases where a
>> simple resubmission passed the monkey, should we:
>>
>> * have an issue for each such false-failure, to get some early
>> indication from manually track occurrences
>>
>> * where such an issue hasn't yet been created, first report comes to the
>> list
>>
>> ------
>> So, I just got the following false-failures in separate runs for:
>> https://pharo.fogbugz.com/f/cases/16265
>> * ProcessTerminateBug>>#testTerminationDuringUnwind
>> * a CISmallIntRule: Timeout (after 0:00:40:00) occured while loading...
>>
>> Anyone else?
>>
>> cheers -ben
>>
>> P.S. I understand some errors are filtered. Can this list be output as
>> part of the validation report (or is it already) ?
>>
>>
>
>