Smalltalk › Pharo › Pharo Smalltalk Developers

Re: [regression reporter]regression occurred

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

11 messages Options

Sven Van Caekenberghe-2

Re: [regression reporter]regression occurred

The following test seems to be failing a lot lately on the CI infrastructure, yet it always succeeds for me on my machine. Is there anybody who sees this fail on their machines ?

> On 24 Feb 2016, at 08:36, [hidden email] wrote:
>
> https://ci.inria.fr/pharo/job/Pharo-5.0-Update-Step-2.1-Validation-M-Z/label=mac/755/
>
> 1 regressions found.
> Zinc.Zodiac.ZnHTTPSTests.testAmazonAWS

Max Leske

Re: [regression reporter]regression occurred

OS X 10.11.3

./pharo Pharo.image test --junit-xml-output '^(?!Metacello)(?!NativeBoost)[M-Z].*'

4146 run, 4122 passes, 17 failures, 7 errors.

Output from the CI build:

4146 run, 4121 passes, 17 failures, 8 errors.

> On 24 Feb 2016, at 09:07, Sven Van Caekenberghe <[hidden email]> wrote:
>
> The following test seems to be failing a lot lately on the CI infrastructure, yet it always succeeds for me on my machine. Is there anybody who sees this fail on their machines ?
>
>> On 24 Feb 2016, at 08:36, [hidden email] wrote:
>>
>> https://ci.inria.fr/pharo/job/Pharo-5.0-Update-Step-2.1-Validation-M-Z/label=mac/755/
>>
>> 1 regressions found.
>> Zinc.Zodiac.ZnHTTPSTests.testAmazonAWS
>
>

Marcus Denker-4

Re: [regression reporter]regression occurred

In reply to this post by Sven Van Caekenberghe-2

one idea could be to add this to the filter of the CI runner.

It seems it fails due to network setup problems that are specific to the CI server...

Ben Coman

Re: [regression reporter]regression occurred

In reply to this post by Sven Van Caekenberghe-2

On Wed, Feb 24, 2016 at 4:07 PM, Sven Van Caekenberghe <[hidden email]> wrote:
> The following test seems to be failing a lot lately on the CI infrastructure,

On a related topic, its hard for an individual person to notice when
this sort of commonly failing test occurs. All they see is the one
issue they are working on. Having a daily email of the top 5 failing
tests would help quicker squash such cases. But probably just an idea
for the back burner.
cheers -ben

> yet it always succeeds for me on my machine. Is there anybody who sees this fail on their machines ?
>
>> On 24 Feb 2016, at 08:36, [hidden email] wrote:
>>
>> https://ci.inria.fr/pharo/job/Pharo-5.0-Update-Step-2.1-Validation-M-Z/label=mac/755/
>>
>> 1 regressions found.
>> Zinc.Zodiac.ZnHTTPSTests.testAmazonAWS
>
>

Ben Coman

Re: [regression reporter]regression occurred

In reply to this post by Marcus Denker-4

Maybe there can be a pre-test run at the shell level to flag that the
required network connectivity is available to run that test inside the
image. Pharo startup could read them in while starting.
cheers -ben

On Wed, Feb 24, 2016 at 5:38 PM, Marcus Denker <[hidden email]> wrote:

> one idea could be to add this to the filter of the CI runner.
>
> It seems it fails due to network setup problems that are specific to the CI server...
>
>> On 24 Feb 2016, at 09:07, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>> The following test seems to be failing a lot lately on the CI infrastructure, yet it always succeeds for me on my machine. Is there anybody who sees this fail on their machines ?
>>
>>> On 24 Feb 2016, at 08:36, [hidden email] wrote:
>>>
>>> https://ci.inria.fr/pharo/job/Pharo-5.0-Update-Step-2.1-Validation-M-Z/label=mac/755/
>>>
>>> 1 regressions found.
>>> Zinc.Zodiac.ZnHTTPSTests.testAmazonAWS
>>
>>
>
>

Sven Van Caekenberghe-2

Re: [regression reporter]regression occurred

> On 24 Feb 2016, at 10:50, Ben Coman <[hidden email]> wrote:
>
> Maybe there can be a pre-test run at the shell level to flag that the
> required network connectivity is available to run that test inside the
> image. Pharo startup could read them in while starting.

These networking problems are issues specific to the CI infrastructure. But these are real bugs: we don't know why they happen, we have no idea. It is not that there is no network, else almost everything would fail immediately. It should not be too much asked in the 21st century for a server to be able to access any website, right ?

> cheers -ben
>
> On Wed, Feb 24, 2016 at 5:38 PM, Marcus Denker <[hidden email]> wrote:
>> one idea could be to add this to the filter of the CI runner.
>>
>> It seems it fails due to network setup problems that are specific to the CI server...
>>
>>> On 24 Feb 2016, at 09:07, Sven Van Caekenberghe <[hidden email]> wrote:
>>>
>>> The following test seems to be failing a lot lately on the CI infrastructure, yet it always succeeds for me on my machine. Is there anybody who sees this fail on their machines ?
>>>
>>>> On 24 Feb 2016, at 08:36, [hidden email] wrote:
>>>>
>>>> https://ci.inria.fr/pharo/job/Pharo-5.0-Update-Step-2.1-Validation-M-Z/label=mac/755/
>>>>
>>>> 1 regressions found.
>>>> Zinc.Zodiac.ZnHTTPSTests.testAmazonAWS
>>>
>>>
>>
>>
>

Marcus Denker-4

Re: [regression reporter]regression occurred

In reply to this post by Ben Coman

The problem is that managing a CI server for a project like Pharo would be
one full time engineer in a company, we do not have the manpower.

So we need to find solutions that are cheap to do.

> On 24 Feb 2016, at 10:50, Ben Coman <[hidden email]> wrote:
>
> Maybe there can be a pre-test run at the shell level to flag that the
> required network connectivity is available to run that test inside the
> image. Pharo startup could read them in while starting.
> cheers -ben
>
> On Wed, Feb 24, 2016 at 5:38 PM, Marcus Denker <[hidden email]> wrote:
>> one idea could be to add this to the filter of the CI runner.
>>
>> It seems it fails due to network setup problems that are specific to the CI server...
>>
>>> On 24 Feb 2016, at 09:07, Sven Van Caekenberghe <[hidden email]> wrote:
>>>
>>> The following test seems to be failing a lot lately on the CI infrastructure, yet it always succeeds for me on my machine. Is there anybody who sees this fail on their machines ?
>>>
>>>> On 24 Feb 2016, at 08:36, [hidden email] wrote:
>>>>
>>>> https://ci.inria.fr/pharo/job/Pharo-5.0-Update-Step-2.1-Validation-M-Z/label=mac/755/
>>>>
>>>> 1 regressions found.
>>>> Zinc.Zodiac.ZnHTTPSTests.testAmazonAWS
>>>
>>>
>>
>>
>

Max Leske

Re: [regression reporter]regression occurred

Here’s an idea:

1. exclude Zinc tests from the validation tests
2. after the build, trigger a Travis build on Github via API (I just set that up for Fuel, so I can provide help there)
3. the Travis build only runs the Zinc tests
4. read build results from Travis

Very ugly, I know. But it’s done rather quickly and should solve all the network problems.

Max

> On 24 Feb 2016, at 11:01, Marcus Denker <[hidden email]> wrote:
>
> The problem is that managing a CI server for a project like Pharo would be
> one full time engineer in a company, we do not have the manpower.
>
> So we need to find solutions that are cheap to do.
>
>> On 24 Feb 2016, at 10:50, Ben Coman <[hidden email]> wrote:
>>
>> Maybe there can be a pre-test run at the shell level to flag that the
>> required network connectivity is available to run that test inside the
>> image. Pharo startup could read them in while starting.
>> cheers -ben
>>
>> On Wed, Feb 24, 2016 at 5:38 PM, Marcus Denker <[hidden email]> wrote:
>>> one idea could be to add this to the filter of the CI runner.
>>>
>>> It seems it fails due to network setup problems that are specific to the CI server...
>>>
>>>> On 24 Feb 2016, at 09:07, Sven Van Caekenberghe <[hidden email]> wrote:
>>>>
>>>> The following test seems to be failing a lot lately on the CI infrastructure, yet it always succeeds for me on my machine. Is there anybody who sees this fail on their machines ?
>>>>
>>>>> On 24 Feb 2016, at 08:36, [hidden email] wrote:
>>>>>
>>>>> https://ci.inria.fr/pharo/job/Pharo-5.0-Update-Step-2.1-Validation-M-Z/label=mac/755/
>>>>>
>>>>> 1 regressions found.
>>>>> Zinc.Zodiac.ZnHTTPSTests.testAmazonAWS
>>>>
>>>>
>>>
>>>
>>
>
>

Sven Van Caekenberghe-2

Re: [regression reporter]regression occurred

> On 24 Feb 2016, at 11:09, Max Leske <[hidden email]> wrote:
>
> Here’s an idea:
>
> 1. exclude Zinc tests from the validation tests
> 2. after the build, trigger a Travis build on Github via API (I just set that up for Fuel, so I can provide help there)
> 3. the Travis build only runs the Zinc tests
> 4. read build results from Travis
>
> Very ugly, I know. But it’s done rather quickly and should solve all the network problems.

I don't think the current problem is severe enough to put much work in it, right now.

But would it not be much better to run everything (all tests) on Travis ?

At first, maybe just as a limited experiment ? I would love to see that.

> Max
>
>> On 24 Feb 2016, at 11:01, Marcus Denker <[hidden email]> wrote:
>>
>> The problem is that managing a CI server for a project like Pharo would be
>> one full time engineer in a company, we do not have the manpower.
>>
>> So we need to find solutions that are cheap to do.
>>
>>> On 24 Feb 2016, at 10:50, Ben Coman <[hidden email]> wrote:
>>>
>>> Maybe there can be a pre-test run at the shell level to flag that the
>>> required network connectivity is available to run that test inside the
>>> image. Pharo startup could read them in while starting.
>>> cheers -ben
>>>
>>> On Wed, Feb 24, 2016 at 5:38 PM, Marcus Denker <[hidden email]> wrote:
>>>> one idea could be to add this to the filter of the CI runner.
>>>>
>>>> It seems it fails due to network setup problems that are specific to the CI server...
>>>>
>>>>> On 24 Feb 2016, at 09:07, Sven Van Caekenberghe <[hidden email]> wrote:
>>>>>
>>>>> The following test seems to be failing a lot lately on the CI infrastructure, yet it always succeeds for me on my machine. Is there anybody who sees this fail on their machines ?
>>>>>
>>>>>> On 24 Feb 2016, at 08:36, [hidden email] wrote:
>>>>>>
>>>>>> https://ci.inria.fr/pharo/job/Pharo-5.0-Update-Step-2.1-Validation-M-Z/label=mac/755/
>>>>>>
>>>>>> 1 regressions found.
>>>>>> Zinc.Zodiac.ZnHTTPSTests.testAmazonAWS
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
>

Max Leske

Re: [regression reporter]regression occurred

> On 24 Feb 2016, at 11:21, Sven Van Caekenberghe <[hidden email]> wrote:
>
>
>> On 24 Feb 2016, at 11:09, Max Leske <[hidden email]> wrote:
>>
>> Here’s an idea:
>>
>> 1. exclude Zinc tests from the validation tests
>> 2. after the build, trigger a Travis build on Github via API (I just set that up for Fuel, so I can provide help there)
>> 3. the Travis build only runs the Zinc tests
>> 4. read build results from Travis
>>
>> Very ugly, I know. But it’s done rather quickly and should solve all the network problems.
>
> I don't think the current problem is severe enough to put much work in it, right now.
>
> But would it not be much better to run everything (all tests) on Travis ?
>
> At first, maybe just as a limited experiment ? I would love to see that.

Sure. That wouldn’t be hard (although I think support for Windows is missing for the Smalltalk language, but I’m sure Fabio Niephaus would help us out there). We could use the Jenkins job as a trigger (or create a second job for experimenting first). In the long run, we probably would want to use the push / pull request hook to trigger the build but for quickly hacking things together I suggest using the API trigger.

I don’t have enough permissions on Github and Jenkins to pull this off alone but I’d be happy to help setting up the Travis stuff and providing the trigger script.

>
>> Max
>>
>>> On 24 Feb 2016, at 11:01, Marcus Denker <[hidden email]> wrote:
>>>
>>> The problem is that managing a CI server for a project like Pharo would be
>>> one full time engineer in a company, we do not have the manpower.
>>>
>>> So we need to find solutions that are cheap to do.
>>>
>>>> On 24 Feb 2016, at 10:50, Ben Coman <[hidden email]> wrote:
>>>>
>>>> Maybe there can be a pre-test run at the shell level to flag that the
>>>> required network connectivity is available to run that test inside the
>>>> image. Pharo startup could read them in while starting.
>>>> cheers -ben
>>>>
>>>> On Wed, Feb 24, 2016 at 5:38 PM, Marcus Denker <[hidden email]> wrote:
>>>>> one idea could be to add this to the filter of the CI runner.
>>>>>
>>>>> It seems it fails due to network setup problems that are specific to the CI server...
>>>>>
>>>>>> On 24 Feb 2016, at 09:07, Sven Van Caekenberghe <[hidden email]> wrote:
>>>>>>
>>>>>> The following test seems to be failing a lot lately on the CI infrastructure, yet it always succeeds for me on my machine. Is there anybody who sees this fail on their machines ?
>>>>>>
>>>>>>> On 24 Feb 2016, at 08:36, [hidden email] wrote:
>>>>>>>
>>>>>>> https://ci.inria.fr/pharo/job/Pharo-5.0-Update-Step-2.1-Validation-M-Z/label=mac/755/
>>>>>>>
>>>>>>> 1 regressions found.
>>>>>>> Zinc.Zodiac.ZnHTTPSTests.testAmazonAWS
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>
>

CyrilFerlicot

Re: [regression reporter]regression occurred

In reply to this post by Sven Van Caekenberghe-2

On 24/02/2016 09:07, Sven Van Caekenberghe wrote:

> The following test seems to be failing a lot lately on the CI infrastructure, yet it always succeeds for me on my machine. Is there anybody who sees this fail on their machines ?
>
>> On 24 Feb 2016, at 08:36, [hidden email] wrote:
>>
>> https://ci.inria.fr/pharo/job/Pharo-5.0-Update-Step-2.1-Validation-M-Z/label=mac/755/
>>
>> 1 regressions found.
>> Zinc.Zodiac.ZnHTTPSTests.testAmazonAWS
>
>

It does not fail all the time but on my Mac with a stable connection with:

20 timesRepeat: [
Transcript show: Time now asString, ': '.
Transcript show: (ZnHTTPSTests new setTestSelector: #testAmazonAWS; run);
cr
]

I got:

12:19:52.230023 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:19:53.028736 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:19:53.92617 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:19:55.23689 pm: 1 run, 0 passes, 0 skipped, 0 expected failures, 0
failures, 1 errors, 0 unexpected passes
12:20:35.851549 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:36.991826 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:37.955399 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:38.930401 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:39.81128 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:40.775596 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:42.524211 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:43.326577 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:44.291269 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:45.313826 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:46.949006 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:48.581509 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:49.546702 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:50.435629 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:51.403718 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes
12:20:53.225974 pm: 1 run, 1 passes, 0 skipped, 0 expected failures, 0
failures, 0 errors, 0 unexpected passes

So there was one test that failed.

--
Cyril Ferlicot

http://www.synectique.eu

165 Avenue Bretagne
Lille 59000 France

signature.asc (817 bytes) Download Attachment