Pharo CI Validation Timeout

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Pharo CI Validation Timeout

Peter Uhnak
Hi,

CI is failing on

a CIAndRule: Timeout (after 0:01:00:00) occured


Because the validation before it was ok and I just changed a protocol.
Is this my problem? Is this issue in CI?

Thanks,
Peter
Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

Marcus Denker-4

On 01 Sep 2015, at 17:22, Peter Uhnák <[hidden email]> wrote:

Hi,

CI is failing on

a CIAndRule: Timeout (after 0:01:00:00) occured


Because the validation before it was ok and I just changed a protocol.
Is this my problem? Is this issue in CI?


This is an issue with, I think, the virtual server below… it is strange: some times the run takes
very very very long. I have not yet found out why (there is an issue tracker entry for it).

Marcus

Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

Peter Uhnak
In reply to this post by Peter Uhnak

On Tue, Sep 1, 2015 at 5:22 PM, Peter Uhnák <[hidden email]> wrote:
Hi,

CI is failing on

a CIAndRule: Timeout (after 0:01:00:00) occured


Because the validation before it was ok and I just changed a protocol.
Is this my problem? Is this issue in CI?

Thanks,
Peter

Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

Peter Uhnak
In reply to this post by Marcus Denker-4
So do I wait, or do I keeping pushing the monkey until it succeeds?

On Tue, Sep 1, 2015 at 5:23 PM, Marcus Denker <[hidden email]> wrote:

On 01 Sep 2015, at 17:22, Peter Uhnák <[hidden email]> wrote:

Hi,

CI is failing on

a CIAndRule: Timeout (after 0:01:00:00) occured


Because the validation before it was ok and I just changed a protocol.
Is this my problem? Is this issue in CI?


This is an issue with, I think, the virtual server below… it is strange: some times the run takes
very very very long. I have not yet found out why (there is an issue tracker entry for it).

Marcus


Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

Marcus Denker-4
Yes, for now just restart.. I will try to find time to look into this.

On 01 Sep 2015, at 17:31, Peter Uhnák <[hidden email]> wrote:

So do I wait, or do I keeping pushing the monkey until it succeeds?

On Tue, Sep 1, 2015 at 5:23 PM, Marcus Denker <[hidden email]> wrote:

On 01 Sep 2015, at 17:22, Peter Uhnák <[hidden email]> wrote:

Hi,

CI is failing on

a CIAndRule: Timeout (after 0:01:00:00) occured


Because the validation before it was ok and I just changed a protocol.
Is this my problem? Is this issue in CI?


This is an issue with, I think, the virtual server below… it is strange: some times the run takes
very very very long. I have not yet found out why (there is an issue tracker entry for it).

Marcus



Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

Ben Coman
In reply to this post by Peter Uhnak
Do you want to try a post-load...
    Delay delaySchedulerClass: DelayExperimentalSemaphoreScheduler

which fixes the "super fast delay" on virtual machines reported here...
http://comments.gmane.org/gmane.comp.lang.smalltalk.pharo.devel/100873

Its named "experimental" only since it was integrated just before
Pharo 4 release and did not have wide spread community testing, but
its complete and working.  There is some further overall streamlining
of the DelayScheduler hierarchy I'd like to do later, but it would be
good to give this one a run as the default for a while to see how it
impacts the issue. (It would be good to get rough "CIAndRule: Timeout"
before and after statistics.)

btw, is that CI error indicating it times out after one hour?  Are
there any other timestamps present in the log that indicate if 1 hour
really passed?

cheers -ben

On Tue, Sep 1, 2015 at 11:23 PM, Peter Uhnák <[hidden email]> wrote:

> This is different issue, same problem
> https://ci.inria.fr/pharo/job/Pharo-5.0-Issue-Validator/12442//artifact/validationReport.html
>
> I'm blaming CI.
>
> On Tue, Sep 1, 2015 at 5:22 PM, Peter Uhnák <[hidden email]> wrote:
>>
>> Hi,
>>
>> CI is failing on
>>
>> a CIAndRule: Timeout (after 0:01:00:00) occured
>>
>>
>> https://ci.inria.fr/pharo/job/Pharo-5.0-Issue-Validator/12440//artifact/validationReport.html
>>
>> Because the validation before it was ok and I just changed a protocol.
>> Is this my problem? Is this issue in CI?
>>
>> Thanks,
>> Peter
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

stepharo
Ben can you open a bug entry because we should really try your change now.
And probably remove after the Experimental from the name.

Stef

Le 1/9/15 19:15, Ben Coman a écrit :

> Do you want to try a post-load...
>      Delay delaySchedulerClass: DelayExperimentalSemaphoreScheduler
>
> which fixes the "super fast delay" on virtual machines reported here...
> http://comments.gmane.org/gmane.comp.lang.smalltalk.pharo.devel/100873
>
> Its named "experimental" only since it was integrated just before
> Pharo 4 release and did not have wide spread community testing, but
> its complete and working.  There is some further overall streamlining
> of the DelayScheduler hierarchy I'd like to do later, but it would be
> good to give this one a run as the default for a while to see how it
> impacts the issue. (It would be good to get rough "CIAndRule: Timeout"
> before and after statistics.)
>
> btw, is that CI error indicating it times out after one hour?  Are
> there any other timestamps present in the log that indicate if 1 hour
> really passed?
>
> cheers -ben
>
> On Tue, Sep 1, 2015 at 11:23 PM, Peter Uhnák <[hidden email]> wrote:
>> This is different issue, same problem
>> https://ci.inria.fr/pharo/job/Pharo-5.0-Issue-Validator/12442//artifact/validationReport.html
>>
>> I'm blaming CI.
>>
>> On Tue, Sep 1, 2015 at 5:22 PM, Peter Uhnák <[hidden email]> wrote:
>>> Hi,
>>>
>>> CI is failing on
>>>
>>> a CIAndRule: Timeout (after 0:01:00:00) occured
>>>
>>>
>>> https://ci.inria.fr/pharo/job/Pharo-5.0-Issue-Validator/12440//artifact/validationReport.html
>>>
>>> Because the validation before it was ok and I just changed a protocol.
>>> Is this my problem? Is this issue in CI?
>>>
>>> Thanks,
>>> Peter
>>
>


Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

Marcus Denker-4
In reply to this post by Ben Coman

On 01 Sep 2015, at 19:15, Ben Coman <[hidden email]> wrote:

Do you want to try a post-load...
   Delay delaySchedulerClass: DelayExperimentalSemaphoreScheduler

which fixes the "super fast delay" on virtual machines reported here...
http://comments.gmane.org/gmane.comp.lang.smalltalk.pharo.devel/100873

Its named "experimental" only since it was integrated just before
Pharo 4 release and did not have wide spread community testing, but
its complete and working.  There is some further overall streamlining
of the DelayScheduler hierarchy I'd like to do later, but it would be
good to give this one a run as the default for a while to see how it
impacts the issue. (It would be good to get rough "CIAndRule: Timeout"
before and after statistics.)

btw, is that CI error indicating it times out after one hour?  Are
there any other timestamps present in the log that indicate if 1 hour
really passed?


Yes, I think so… 


Now looking at it (no idea why it is chinese), it seems that we have some
slowdown recently and all runs take 40-50 minutes, a slowdown to 60 can happen
just by having less compute power available (or running multiple things in parallel).

We should 

-> do a pass over slow tests
-> do a pass over slow rules

I will put the limit higher than 60 min for now.

Marcus


Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

Nicolai Hess
It is possible to see from the console output which rule or test takes a lot time?

2015-09-02 8:09 GMT+02:00 Marcus Denker <[hidden email]>:

On 01 Sep 2015, at 19:15, Ben Coman <[hidden email]> wrote:

Do you want to try a post-load...
   Delay delaySchedulerClass: DelayExperimentalSemaphoreScheduler

which fixes the "super fast delay" on virtual machines reported here...
http://comments.gmane.org/gmane.comp.lang.smalltalk.pharo.devel/100873

Its named "experimental" only since it was integrated just before
Pharo 4 release and did not have wide spread community testing, but
its complete and working.  There is some further overall streamlining
of the DelayScheduler hierarchy I'd like to do later, but it would be
good to give this one a run as the default for a while to see how it
impacts the issue. (It would be good to get rough "CIAndRule: Timeout"
before and after statistics.)

btw, is that CI error indicating it times out after one hour?  Are
there any other timestamps present in the log that indicate if 1 hour
really passed?


Yes, I think so… 


Now looking at it (no idea why it is chinese), it seems that we have some
slowdown recently and all runs take 40-50 minutes, a slowdown to 60 can happen
just by having less compute power available (or running multiple things in parallel).

We should 

-> do a pass over slow tests
-> do a pass over slow rules

I will put the limit higher than 60 min for now.

Marcus



Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

Marcus Denker-4
I don’t think so… the job used to run in 15 minutes. But we added more tests,
Rules got slower…

On 02 Sep 2015, at 08:34, Nicolai Hess <[hidden email]> wrote:

It is possible to see from the console output which rule or test takes a lot time?

2015-09-02 8:09 GMT+02:00 Marcus Denker <[hidden email]>:

On 01 Sep 2015, at 19:15, Ben Coman <[hidden email]> wrote:

Do you want to try a post-load...
   Delay delaySchedulerClass: DelayExperimentalSemaphoreScheduler

which fixes the "super fast delay" on virtual machines reported here...
http://comments.gmane.org/gmane.comp.lang.smalltalk.pharo.devel/100873

Its named "experimental" only since it was integrated just before
Pharo 4 release and did not have wide spread community testing, but
its complete and working.  There is some further overall streamlining
of the DelayScheduler hierarchy I'd like to do later, but it would be
good to give this one a run as the default for a while to see how it
impacts the issue. (It would be good to get rough "CIAndRule: Timeout"
before and after statistics.)

btw, is that CI error indicating it times out after one hour?  Are
there any other timestamps present in the log that indicate if 1 hour
really passed?


Yes, I think so… 


Now looking at it (no idea why it is chinese), it seems that we have some
slowdown recently and all runs take 40-50 minutes, a slowdown to 60 can happen
just by having less compute power available (or running multiple things in parallel).

We should 

-> do a pass over slow tests
-> do a pass over slow rules

I will put the limit higher than 60 min for now.

Marcus




Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

Guillermo Polito
I saw also that one of the linux slaves is much much slower than the others. Just check this:


In one of the slaves the build takes one hour. In the other it takes 6.

El mié., 2 de sept. de 2015 a la(s) 9:11 a. m., Marcus Denker <[hidden email]> escribió:
I don’t think so… the job used to run in 15 minutes. But we added more tests,
Rules got slower…

On 02 Sep 2015, at 08:34, Nicolai Hess <[hidden email]> wrote:

It is possible to see from the console output which rule or test takes a lot time?

2015-09-02 8:09 GMT+02:00 Marcus Denker <[hidden email]>:

On 01 Sep 2015, at 19:15, Ben Coman <[hidden email]> wrote:

Do you want to try a post-load...
   Delay delaySchedulerClass: DelayExperimentalSemaphoreScheduler

which fixes the "super fast delay" on virtual machines reported here...
http://comments.gmane.org/gmane.comp.lang.smalltalk.pharo.devel/100873

Its named "experimental" only since it was integrated just before
Pharo 4 release and did not have wide spread community testing, but
its complete and working.  There is some further overall streamlining
of the DelayScheduler hierarchy I'd like to do later, but it would be
good to give this one a run as the default for a while to see how it
impacts the issue. (It would be good to get rough "CIAndRule: Timeout"
before and after statistics.)

btw, is that CI error indicating it times out after one hour?  Are
there any other timestamps present in the log that indicate if 1 hour
really passed?


Yes, I think so… 


Now looking at it (no idea why it is chinese), it seems that we have some
slowdown recently and all runs take 40-50 minutes, a slowdown to 60 can happen
just by having less compute power available (or running multiple things in parallel).

We should 

-> do a pass over slow tests
-> do a pass over slow rules

I will put the limit higher than 60 min for now.

Marcus




Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

Marcus Denker-4
I have the feeling that is is an artefact of the virtualisation somehow. 

On 02 Sep 2015, at 10:24, Guillermo Polito <[hidden email]> wrote:

I saw also that one of the linux slaves is much much slower than the others. Just check this:


In one of the slaves the build takes one hour. In the other it takes 6.

El mié., 2 de sept. de 2015 a la(s) 9:11 a. m., Marcus Denker <[hidden email]> escribió:
I don’t think so… the job used to run in 15 minutes. But we added more tests,
Rules got slower…

On 02 Sep 2015, at 08:34, Nicolai Hess <[hidden email]> wrote:

It is possible to see from the console output which rule or test takes a lot time?

2015-09-02 8:09 GMT+02:00 Marcus Denker <[hidden email]>:

On 01 Sep 2015, at 19:15, Ben Coman <[hidden email]> wrote:

Do you want to try a post-load...
   Delay delaySchedulerClass: DelayExperimentalSemaphoreScheduler

which fixes the "super fast delay" on virtual machines reported here...
http://comments.gmane.org/gmane.comp.lang.smalltalk.pharo.devel/100873

Its named "experimental" only since it was integrated just before
Pharo 4 release and did not have wide spread community testing, but
its complete and working.  There is some further overall streamlining
of the DelayScheduler hierarchy I'd like to do later, but it would be
good to give this one a run as the default for a while to see how it
impacts the issue. (It would be good to get rough "CIAndRule: Timeout"
before and after statistics.)

btw, is that CI error indicating it times out after one hour?  Are
there any other timestamps present in the log that indicate if 1 hour
really passed?


Yes, I think so… 


Now looking at it (no idea why it is chinese), it seems that we have some
slowdown recently and all runs take 40-50 minutes, a slowdown to 60 can happen
just by having less compute power available (or running multiple things in parallel).

We should 

-> do a pass over slow tests
-> do a pass over slow rules

I will put the limit higher than 60 min for now.

Marcus





Reply | Threaded
Open this post in threaded view
|

Re: Pharo CI Validation Timeout

Nicolai Hess
Low memory ?

Running the rules fill up the ast cache with nearly one entry per compiled method instance.
(this are thousands instances of Array/OrderedCollections/RB and OC classes)
My pharo images runs from ~ 50 MB to ~ 350-400 MB
after running all rules for all classes!



2015-09-02 13:42 GMT+02:00 Marcus Denker <[hidden email]>:
I have the feeling that is is an artefact of the virtualisation somehow. 

On 02 Sep 2015, at 10:24, Guillermo Polito <[hidden email]> wrote:

I saw also that one of the linux slaves is much much slower than the others. Just check this:


In one of the slaves the build takes one hour. In the other it takes 6.

El mié., 2 de sept. de 2015 a la(s) 9:11 a. m., Marcus Denker <[hidden email]> escribió:
I don’t think so… the job used to run in 15 minutes. But we added more tests,
Rules got slower…

On 02 Sep 2015, at 08:34, Nicolai Hess <[hidden email]> wrote:

It is possible to see from the console output which rule or test takes a lot time?

2015-09-02 8:09 GMT+02:00 Marcus Denker <[hidden email]>:

On 01 Sep 2015, at 19:15, Ben Coman <[hidden email]> wrote:

Do you want to try a post-load...
   Delay delaySchedulerClass: DelayExperimentalSemaphoreScheduler

which fixes the "super fast delay" on virtual machines reported here...
http://comments.gmane.org/gmane.comp.lang.smalltalk.pharo.devel/100873

Its named "experimental" only since it was integrated just before
Pharo 4 release and did not have wide spread community testing, but
its complete and working.  There is some further overall streamlining
of the DelayScheduler hierarchy I'd like to do later, but it would be
good to give this one a run as the default for a while to see how it
impacts the issue. (It would be good to get rough "CIAndRule: Timeout"
before and after statistics.)

btw, is that CI error indicating it times out after one hour?  Are
there any other timestamps present in the log that indicate if 1 hour
really passed?


Yes, I think so… 


Now looking at it (no idea why it is chinese), it seems that we have some
slowdown recently and all runs take 40-50 minutes, a slowdown to 60 can happen
just by having less compute power available (or running multiple things in parallel).

We should 

-> do a pass over slow tests
-> do a pass over slow rules

I will put the limit higher than 60 min for now.

Marcus