CI Job artifacts

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

CI Job artifacts

Tobias Pape
Hi all

I may be mistaken, but when I download the
last build artifacts from http://build.squeak.org/job/SqueakTrunk/
I get a 4.4 that ist updated to 12337, and _not_ a current
Trunk Image.
Is that expected?

Best
        -Tobias






signature.asc (210 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CI Job artifacts

Tobias Pape
Hi all
On 16.10.2013, at 10:48, Tobias Pape <[hidden email]> wrote:

> Hi all
>
> I may be mistaken, but when I download the
> last build artifacts from http://build.squeak.org/job/SqueakTrunk/
> I get a 4.4 that ist updated to 12337, and _not_ a current
> Trunk Image.

More Info:
apparently the script took too long to execute. from the console of job 560:

2013-10-15T19:17:01.774+01:00: Updating http://source.squeak.org/trunk
2013-10-15T19:17:04.672+01:00: Checking http://source.squeak.org/trunk
!!! Killed command for exceeding allotted time: nice /home/jenkins/workspace/SqueakTrunk/target/Squeak-4.10.2.2614-src-32/bld/squeak.sh -vm-sound-null -vm-display-null "/home/jenkins/workspace/SqueakTrunk/target/TrunkImage.image" ../save-image.st.
!!! Killed command for exceeding allotted time: nice /home/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/squeak -vm-sound-null -vm-display-null "/home/jenkins/workspace/SqueakTrunk/target/TrunkImage.image" ../update-image.st.


Can the one with access to the script please include
`set -e` to abort the script when things like this happen?
Jenkins goes red then, and not yellow as it did now.

Best
        -Tobias



signature.asc (210 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Fwd: [squeak-dev] CI Job artifacts

Tobias Pape


Begin forwarded message:

> From: Tobias Pape <[hidden email]>
> Subject: Re: [squeak-dev] CI Job artifacts
> Date: 16. Oktober 2013 10:53:09 MESZ
> To: The general-purpose Squeak developers list <[hidden email]>
> Reply-To: The general-purpose Squeak developers list <[hidden email]>
> Delivered-To: [hidden email]
>
> Hi all
> On 16.10.2013, at 10:48, Tobias Pape <[hidden email]> wrote:
>
>> Hi all
>>
>> I may be mistaken, but when I download the
>> last build artifacts from http://build.squeak.org/job/SqueakTrunk/
>> I get a 4.4 that ist updated to 12337, and _not_ a current
>> Trunk Image.
>
> More Info:
> apparently the script took too long to execute. from the console of job 560:
> …
> 2013-10-15T19:17:01.774+01:00: Updating http://source.squeak.org/trunk
> 2013-10-15T19:17:04.672+01:00: Checking http://source.squeak.org/trunk
> !!! Killed command for exceeding allotted time: nice /home/jenkins/workspace/SqueakTrunk/target/Squeak-4.10.2.2614-src-32/bld/squeak.sh -vm-sound-null -vm-display-null "/home/jenkins/workspace/SqueakTrunk/target/TrunkImage.image" ../save-image.st.
> !!! Killed command for exceeding allotted time: nice /home/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/squeak -vm-sound-null -vm-display-null "/home/jenkins/workspace/SqueakTrunk/target/TrunkImage.image" ../update-image.st.
> …
Sorry, its job 562 with
2013-10-15T21:48:26.01+01:00: Resolved? true
!!! Killed command for exceeding allotted time: nice /home/jenkins/workspace/SqueakTrunk/target/Squeak-4.10.2.2614-src-32/bld/squeak.sh -vm-sound-null -vm-display-null "/home/jenkins/workspace/SqueakTrunk/target/TrunkImage.image" ../save-image.st.
!!! Killed command for exceeding allotted time: nice /home/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/squeak -vm-sound-null -vm-display-null "/home/jenkins/workspace/SqueakTrunk/target/TrunkImage.image" ../update-image.st.


Best
        -Tobias




signature.asc (210 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CI Job artifacts

Frank Shearar-3
On 16 October 2013 09:57, Tobias Pape <[hidden email]> wrote:

>
>
> Begin forwarded message:
>
>> From: Tobias Pape <[hidden email]>
>> Subject: Re: [squeak-dev] CI Job artifacts
>> Date: 16. Oktober 2013 10:53:09 MESZ
>> To: The general-purpose Squeak developers list <[hidden email]>
>> Reply-To: The general-purpose Squeak developers list <[hidden email]>
>> Delivered-To: [hidden email]
>>
>> Hi all
>> On 16.10.2013, at 10:48, Tobias Pape <[hidden email]> wrote:
>>
>>> Hi all
>>>
>>> I may be mistaken, but when I download the
>>> last build artifacts from http://build.squeak.org/job/SqueakTrunk/
>>> I get a 4.4 that ist updated to 12337, and _not_ a current
>>> Trunk Image.
>>
>> More Info:
>> apparently the script took too long to execute. from the console of job 560:
>> …
>> 2013-10-15T19:17:01.774+01:00: Updating http://source.squeak.org/trunk
>> 2013-10-15T19:17:04.672+01:00: Checking http://source.squeak.org/trunk
>> !!! Killed command for exceeding allotted time: nice /home/jenkins/workspace/SqueakTrunk/target/Squeak-4.10.2.2614-src-32/bld/squeak.sh -vm-sound-null -vm-display-null "/home/jenkins/workspace/SqueakTrunk/target/TrunkImage.image" ../save-image.st.
>> !!! Killed command for exceeding allotted time: nice /home/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/squeak -vm-sound-null -vm-display-null "/home/jenkins/workspace/SqueakTrunk/target/TrunkImage.image" ../update-image.st.
>> …
>
> Sorry, its job 562 with
> 2013-10-15T21:48:26.01+01:00: Resolved? true
> !!! Killed command for exceeding allotted time: nice /home/jenkins/workspace/SqueakTrunk/target/Squeak-4.10.2.2614-src-32/bld/squeak.sh -vm-sound-null -vm-display-null "/home/jenkins/workspace/SqueakTrunk/target/TrunkImage.image" ../save-image.st.
> !!! Killed command for exceeding allotted time: nice /home/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/squeak -vm-sound-null -vm-display-null "/home/jenkins/workspace/SqueakTrunk/target/TrunkImage.image" ../update-image.st.

Indeed. There are a bunch of problems with the SqueakTrunk job at the moment:
* the job is inherently unreliable because we can't make the scripts
actually fail when something goes wrong (see my pushing for this on
vm-dev yesterday) (*)
* updating is EXTREMELY SLOW. I've increased the timeout on the
updating to _25 minutes_ and it's still sometimes not long enough
* we have a serious problem in the form of a primitive crash. Or I do,
or Linux machines do. Something like that.

Once that last issue has been resolved, I'm going to have to hand-roll
a new base image, which will address the build time issue as well as
let us skip past the Parser problems Nicolas & I have recently
discussed.

frank

(*) This failure to bail the entire job means that the image the tests
run against is a didn't-actually-update image, so you end up with a
4.4-12327 image.

Reply | Threaded
Open this post in threaded view
|

Re: CI Job artifacts

Tobias Pape
Hi Frank

On 16.10.2013, at 11:12, Frank Shearar <[hidden email]> wrote:

> On 16 October 2013 09:57, Tobias Pape <[hidden email]> wrote:
>>
>>
>> Begin forwarded message:
>
[…]
> Indeed. There are a bunch of problems with the SqueakTrunk job at the moment:
> * the job is inherently unreliable because we can't make the scripts
> actually fail when something goes wrong (see my pushing for this on
> vm-dev yesterday) (*)

I have not seen the mail… which are you referring to?
Isn't it as simple as
set -e
in the bash script?

> * updating is EXTREMELY SLOW. I've increased the timeout on the
> updating to _25 minutes_ and it's still sometimes not long enough

So either we have to make a new release ;)
or do checkpoints…

> * we have a serious problem in the form of a primitive crash. Or I do,
> or Linux machines do. Something like that.

I tried updating my trunk image from july, Mac, current CogVM,
crashed, too.

>
> Once that last issue has been resolved, I'm going to have to hand-roll
> a new base image, which will address the build time issue as well as
> let us skip past the Parser problems Nicolas & I have recently
> discussed.

Ah well, checkpointing then ;)
If t works, I'm fine with it.


Best
        -Tobias


>
> frank
>
> (*) This failure to bail the entire job means that the image the tests
> run against is a didn't-actually-update image, so you end up with a
> 4.4-12327 image.


Apparently.




signature.asc (210 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CI Job artifacts

Frank Shearar-3
On 16 October 2013 10:23, Tobias Pape <[hidden email]> wrote:

> Hi Frank
>
> On 16.10.2013, at 11:12, Frank Shearar <[hidden email]> wrote:
>
>> On 16 October 2013 09:57, Tobias Pape <[hidden email]> wrote:
>>>
>>>
>>> Begin forwarded message:
>>
> […]
>> Indeed. There are a bunch of problems with the SqueakTrunk job at the moment:
>> * the job is inherently unreliable because we can't make the scripts
>> actually fail when something goes wrong (see my pushing for this on
>> vm-dev yesterday) (*)
>
> I have not seen the mail… which are you referring to?

http://lists.squeakfoundation.org/pipermail/vm-dev/2013-October/013852.html

> Isn't it as simple as
> set -e
> in the bash script?

That's what I want, but it's what I currently can't have, because
running the script will _always_ return success.

>> * updating is EXTREMELY SLOW. I've increased the timeout on the
>> updating to _25 minutes_ and it's still sometimes not long enough
>
> So either we have to make a new release ;)
> or do checkpoints…
>
>> * we have a serious problem in the form of a primitive crash. Or I do,
>> or Linux machines do. Something like that.
>
> I tried updating my trunk image from july, Mac, current CogVM,
> crashed, too.

Gah. OK, so it's a serious problem, and not just me.

>> Once that last issue has been resolved, I'm going to have to hand-roll
>> a new base image, which will address the build time issue as well as
>> let us skip past the Parser problems Nicolas & I have recently
>> discussed.
>
> Ah well, checkpointing then ;)
> If t works, I'm fine with it.
>
>
> Best
>         -Tobias
>
>
>>
>> frank
>>
>> (*) This failure to bail the entire job means that the image the tests
>> run against is a didn't-actually-update image, so you end up with a
>> 4.4-12327 image.
>
>
> Apparently.

Well, on the plus side we can at least understand why we get the
unexpected result!

frank

Reply | Threaded
Open this post in threaded view
|

Re: CI Job artifacts

Tobias Pape
Hi,

On 16.10.2013, at 12:36, Frank Shearar <[hidden email]> wrote:

> On 16 October 2013 10:23, Tobias Pape <[hidden email]> wrote:
>> […]
>> I have not seen the mail… which are you referring to?
>
> http://lists.squeakfoundation.org/pipermail/vm-dev/2013-October/013852.html

got it.

>
>> Isn't it as simple as
>> set -e
>> in the bash script?
>
> That's what I want, but it's what I currently can't have, because
> running the script will _always_ return success.

oO
I see.
https://github.com/squeak-smalltalk/squeak-ci/blob/master/lib/squeak-ci/build.rb#L259
At that point you want to make sure that the exit code of the
Rakefile is _never_ 0 when stuff like that happens…

>
>> […]
>> I tried updating my trunk image from july, Mac, current CogVM,
>> crashed, too.
>
> Gah. OK, so it's a serious problem, and not just me.
>>>
[…]
>>> (*) This failure to bail the entire job means that the image the tests
>>> run against is a didn't-actually-update image, so you end up with a
>>> 4.4-12327 image.
>>
>>
>> Apparently.
>
> Well, on the plus side we can at least understand why we get the
> unexpected result!


Yes. I think jenkins does the right thing™ when we exit with non-null
status, and a subsequent archiving step could be skipped…

Best
        -Tobias





signature.asc (210 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CI Job artifacts

Frank Shearar-3
On 16 October 2013 11:51, Tobias Pape <[hidden email]> wrote:

> Hi,
>
> On 16.10.2013, at 12:36, Frank Shearar <[hidden email]> wrote:
>
>> On 16 October 2013 10:23, Tobias Pape <[hidden email]> wrote:
>>> […]
>>> I have not seen the mail… which are you referring to?
>>
>> http://lists.squeakfoundation.org/pipermail/vm-dev/2013-October/013852.html
>
> got it.
>
>>
>>> Isn't it as simple as
>>> set -e
>>> in the bash script?
>>
>> That's what I want, but it's what I currently can't have, because
>> running the script will _always_ return success.
>
> oO
> I see.
> https://github.com/squeak-smalltalk/squeak-ci/blob/master/lib/squeak-ci/build.rb#L259
> At that point you want to make sure that the exit code of the
> Rakefile is _never_ 0 when stuff like that happens…

https://github.com/squeak-smalltalk/squeak-ci/issues/39 should satisfy
this criterion (when it's actually done, of course). It's not entirely
ideal, because we can't tell the difference between a successfully run
script and one that broke, but at least it would let us tell the
difference between a script that hung and was then killed, and one
that _completed_.

>>> […]
>>> I tried updating my trunk image from july, Mac, current CogVM,
>>> crashed, too.
>>
>> Gah. OK, so it's a serious problem, and not just me.
>>>>
> […]
>>>> (*) This failure to bail the entire job means that the image the tests
>>>> run against is a didn't-actually-update image, so you end up with a
>>>> 4.4-12327 image.
>>>
>>>
>>> Apparently.
>>
>> Well, on the plus side we can at least understand why we get the
>> unexpected result!
>
>
> Yes. I think jenkins does the right thing™ when we exit with non-null
> status, and a subsequent archiving step could be skipped…

Yes, I think so.

frank