How to restart an appveyor job?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

How to restart an appveyor job?

Nicolas Cellier
 
Hi all,
I wanted to solve the gcc compiler bug for pharo.sista.spur.

But of course, there were another failure due to fragile infrastructure...
https://ci.appveyor.com/project/OpenSmalltalk/vm/build/1.0.1207/job/dd3lnytbwb5jjlqa

Running Install scripts
Start-FileDownload "http://cygwin.com/setup-x86.exe" -FileName "setup-x86.exe"
Start-FileDownloadInternal : Error downloading remote file: One or more errors occurred.
Inner Exception: Unable to connect to the remote server
At C:\Program Files\AppVeyor\BuildAgent\Modules\build-worker-api\build-worker-api.psm1:242 char:2
+ Start-FileDownloadInternal -Url $Url -FileName $FileName -Timeout ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-FileDownloadInternal], Exception
+ FullyQualifiedErrorId : System.Exception,Appveyor.BuildAgent.Api.Utils.StartFileDownloadInternalCmdlet

We depend on volatile URL...
Do you know if there is a way to restart a job in appveyor?
Reply | Threaded
Open this post in threaded view
|

Re: How to restart an appveyor job?

Nicolas Cellier
 
Answering to myself: it's an opened issue
https://github.com/appveyor/ci/issues/58

I don't know if upvoting will change a thing, but I did.

2018-04-02 14:32 GMT+02:00 Nicolas Cellier <[hidden email]>:
Hi all,
I wanted to solve the gcc compiler bug for pharo.sista.spur.

But of course, there were another failure due to fragile infrastructure...
https://ci.appveyor.com/project/OpenSmalltalk/vm/build/1.0.1207/job/dd3lnytbwb5jjlqa

Running Install scripts
Start-FileDownload "http://cygwin.com/setup-x86.exe" -FileName "setup-x86.exe"
Start-FileDownloadInternal : Error downloading remote file: One or more errors occurred.
Inner Exception: Unable to connect to the remote server
At C:\Program Files\AppVeyor\BuildAgent\Modules\build-worker-api\build-worker-api.psm1:242 char:2
+ Start-FileDownloadInternal -Url $Url -FileName $FileName -Timeout ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-FileDownloadInternal], Exception
+ FullyQualifiedErrorId : System.Exception,Appveyor.BuildAgent.Api.Utils.StartFileDownloadInternalCmdlet

We depend on volatile URL...
Do you know if there is a way to restart a job in appveyor?

Reply | Threaded
Open this post in threaded view
|

Re: How to restart an appveyor job?

Ben Coman
In reply to this post by Nicolas Cellier
 


On 2 April 2018 at 20:32, Nicolas Cellier <[hidden email]> wrote:
 
Hi all,
I wanted to solve the gcc compiler bug for pharo.sista.spur.

But of course, there were another failure due to fragile infrastructure...
https://ci.appveyor.com/project/OpenSmalltalk/vm/build/1.0.1207/job/dd3lnytbwb5jjlqa

Running Install scripts
Start-FileDownload "http://cygwin.com/setup-x86.exe" -FileName "setup-x86.exe"
Start-FileDownloadInternal : Error downloading remote file: One or more errors occurred.
Inner Exception: Unable to connect to the remote server
At C:\Program Files\AppVeyor\BuildAgent\Modules\build-worker-api\build-worker-api.psm1:242 char:2
+ Start-FileDownloadInternal -Url $Url -FileName $FileName -Timeout ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-FileDownloadInternal], Exception
+ FullyQualifiedErrorId : System.Exception,Appveyor.BuildAgent.Api.Utils.StartFileDownloadInternalCmdlet

We depend on volatile URL...
Do you know if there is a way to restart a job in appveyor?


Two questions in ignorance...


Why do we download & install Cygwin when it seems already included?


so should I presume our builds use "Visual Studio 2015" image?


cheers -ben

btw, I followed your lead to add an upvote.
Reply | Threaded
Open this post in threaded view
|

Re: How to restart an appveyor job?

Nicolas Cellier
 


2018-04-02 16:09 GMT+02:00 Ben Coman <[hidden email]>:
 


On 2 April 2018 at 20:32, Nicolas Cellier <[hidden email]> wrote:
 
Hi all,
I wanted to solve the gcc compiler bug for pharo.sista.spur.

But of course, there were another failure due to fragile infrastructure...
https://ci.appveyor.com/project/OpenSmalltalk/vm/build/1.0.1207/job/dd3lnytbwb5jjlqa

Running Install scripts
Start-FileDownload "http://cygwin.com/setup-x86.exe" -FileName "setup-x86.exe"
Start-FileDownloadInternal : Error downloading remote file: One or more errors occurred.
Inner Exception: Unable to connect to the remote server
At C:\Program Files\AppVeyor\BuildAgent\Modules\build-worker-api\build-worker-api.psm1:242 char:2
+ Start-FileDownloadInternal -Url $Url -FileName $FileName -Timeout ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-FileDownloadInternal], Exception
+ FullyQualifiedErrorId : System.Exception,Appveyor.BuildAgent.Api.Utils.StartFileDownloadInternalCmdlet

We depend on volatile URL...
Do you know if there is a way to restart a job in appveyor?


Two questions in ignorance...


Why do we download & install Cygwin when it seems already included?


Hi Ben,
It's because we rely on specific pre-built packages which are not on the appveyor machine.

But in the next build that's Smalltalk-CI which failed to be downloaded, so we have multiple weakness.


so should I presume our builds use "Visual Studio 2015" image?


cheers -ben

btw, I followed your lead to add an upvote.


Maybe it's leading to nowhere, but thanks Ben ;)
Reply | Threaded
Open this post in threaded view
|

Re: How to restart an appveyor job?

Nicolas Cellier
 


2018-04-02 16:20 GMT+02:00 Nicolas Cellier <[hidden email]>:


2018-04-02 16:09 GMT+02:00 Ben Coman <[hidden email]>:
 


On 2 April 2018 at 20:32, Nicolas Cellier <[hidden email]> wrote:
 
Hi all,
I wanted to solve the gcc compiler bug for pharo.sista.spur.

But of course, there were another failure due to fragile infrastructure...
https://ci.appveyor.com/project/OpenSmalltalk/vm/build/1.0.1207/job/dd3lnytbwb5jjlqa

Running Install scripts
Start-FileDownload "http://cygwin.com/setup-x86.exe" -FileName "setup-x86.exe"
Start-FileDownloadInternal : Error downloading remote file: One or more errors occurred.
Inner Exception: Unable to connect to the remote server
At C:\Program Files\AppVeyor\BuildAgent\Modules\build-worker-api\build-worker-api.psm1:242 char:2
+ Start-FileDownloadInternal -Url $Url -FileName $FileName -Timeout ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-FileDownloadInternal], Exception
+ FullyQualifiedErrorId : System.Exception,Appveyor.BuildAgent.Api.Utils.StartFileDownloadInternalCmdlet

We depend on volatile URL...
Do you know if there is a way to restart a job in appveyor?


Two questions in ignorance...


Why do we download & install Cygwin when it seems already included?


Hi Ben,
It's because we rely on specific pre-built packages which are not on the appveyor machine.


and will happen again and again...

I am trying to replace
'Start-FileDownload "http://cygwin.com/setup-x86.exe" -FileName "setup-x86.exe"'
with curl, so as to at least manage the timeout and retry options:

 
But in the next build that's Smalltalk-CI which failed to be downloaded, so we have multiple weakness.


so should I presume our builds use "Visual Studio 2015" image?


cheers -ben

btw, I followed your lead to add an upvote.


Maybe it's leading to nowhere, but thanks Ben ;)

Reply | Threaded
Open this post in threaded view
|

Re: How to restart an appveyor job?

Ben Coman
In reply to this post by Nicolas Cellier
 


On 2 April 2018 at 20:32, Nicolas Cellier <[hidden email]> wrote:
 
Hi all,
I wanted to solve the gcc compiler bug for pharo.sista.spur.

But of course, there were another failure due to fragile infrastructure...
https://ci.appveyor.com/project/OpenSmalltalk/vm/build/1.0.1207/job/dd3lnytbwb5jjlqa

Running Install scripts
Start-FileDownload "http://cygwin.com/setup-x86.exe" -FileName "setup-x86.exe"
Start-FileDownloadInternal : Error downloading remote file: One or more errors occurred.
Inner Exception: Unable to connect to the remote server
At C:\Program Files\AppVeyor\BuildAgent\Modules\build-worker-api\build-worker-api.psm1:242 char:2
+ Start-FileDownloadInternal -Url $Url -FileName $FileName -Timeout ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-FileDownloadInternal], Exception
+ FullyQualifiedErrorId : System.Exception,Appveyor.BuildAgent.Api.Utils.StartFileDownloadInternalCmdlet

We depend on volatile URL...


Seems quite susceptible to the ground unknowingly moving under our feet.
Actually I found it quite difficult (i.e. failed) to determine a hard link to the version being installed.
To enhance reproduciblity, we should host our own versioned archive of the required packages somewhere fairly permanent.

This seems pertinent...
2.2.What about an automated Cygwin installation?
If you are deploying to multiple systems, the best way is to run through a full installation once, saving the entire downloaded package tree. Then, on target systems, run Cygwin Setup as a "Local Install" pointed at your downloaded package tree. You could do this non-interactively with the command line options -q -L -l x:\cygwin-local\, where your downloaded package tree is in x:\cygwin-local\ (see the next FAQ for an explanation of those options.)
(https://www.cygwin.com/faq.html)

And "pmcyg" looks interesting...  "intended to support construction of
a self-contained CDROM or DVD that can be used to install or upgrade Cygwin
on computers that do not have network access to a Cygwin mirror site."
That could be stored as a versioned archive which is added to the appveyor build cache 
so its only downloaded once,  saving per job build time and 
ensuring if it succeeds at the first job, all other jobs *should* succeed.

Options to host this versioned-cygwin-installer-archive (should be somewhere fairly permanent):
* bintray
* github releases
* git large file storage

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: How to restart an appveyor job?

fniephaus
 
On Mon, Apr 2, 2018 at 5:55 PM Ben Coman <[hidden email]> wrote:
On 2 April 2018 at 20:32, Nicolas Cellier <[hidden email]> wrote:
 
Hi all,
I wanted to solve the gcc compiler bug for pharo.sista.spur.

But of course, there were another failure due to fragile infrastructure...
https://ci.appveyor.com/project/OpenSmalltalk/vm/build/1.0.1207/job/dd3lnytbwb5jjlqa

Running Install scripts
Start-FileDownload "http://cygwin.com/setup-x86.exe" -FileName "setup-x86.exe"
Start-FileDownloadInternal : Error downloading remote file: One or more errors occurred.
Inner Exception: Unable to connect to the remote server
At C:\Program Files\AppVeyor\BuildAgent\Modules\build-worker-api\build-worker-api.psm1:242 char:2
+ Start-FileDownloadInternal -Url $Url -FileName $FileName -Timeout ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-FileDownloadInternal], Exception
+ FullyQualifiedErrorId : System.Exception,Appveyor.BuildAgent.Api.Utils.StartFileDownloadInternalCmdlet

We depend on volatile URL...


Seems quite susceptible to the ground unknowingly moving under our feet.
Actually I found it quite difficult (i.e. failed) to determine a hard link to the version being installed.
To enhance reproduciblity, we should host our own versioned archive of the required packages somewhere fairly permanent.

Instead of hosting our own Cygwin setup, maybe it's better to use the build cache [1]? We are already using that cache for third-party libs [2] and I also thought we'd cache Cygwin as well. @Nicolas: could you give this a go?

Fabio

 

This seems pertinent...
2.2.What about an automated Cygwin installation?
If you are deploying to multiple systems, the best way is to run through a full installation once, saving the entire downloaded package tree. Then, on target systems, run Cygwin Setup as a "Local Install" pointed at your downloaded package tree. You could do this non-interactively with the command line options -q -L -l x:\cygwin-local\, where your downloaded package tree is in x:\cygwin-local\ (see the next FAQ for an explanation of those options.)
(https://www.cygwin.com/faq.html)

And "pmcyg" looks interesting...  "intended to support construction of
a self-contained CDROM or DVD that can be used to install or upgrade Cygwin
on computers that do not have network access to a Cygwin mirror site."
That could be stored as a versioned archive which is added to the appveyor build cache 
so its only downloaded once,  saving per job build time and 
ensuring if it succeeds at the first job, all other jobs *should* succeed.

Options to host this versioned-cygwin-installer-archive (should be somewhere fairly permanent):
* bintray
* github releases
* git large file storage

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: How to restart an appveyor job?

Nicolas Cellier
 
Hi Fabio,
I've switched to curl for now and have a build in progress...
Anyway, cache or not cache, we are dependent on cygwin updates, and build can start failing anytime (for example because a compiler version changed).

2018-04-02 18:13 GMT+02:00 Fabio Niephaus <[hidden email]>:
 
On Mon, Apr 2, 2018 at 5:55 PM Ben Coman <[hidden email]> wrote:
On 2 April 2018 at 20:32, Nicolas Cellier <[hidden email]> wrote:
 
Hi all,
I wanted to solve the gcc compiler bug for pharo.sista.spur.

But of course, there were another failure due to fragile infrastructure...
https://ci.appveyor.com/project/OpenSmalltalk/vm/build/1.0.1207/job/dd3lnytbwb5jjlqa

Running Install scripts
Start-FileDownload "http://cygwin.com/setup-x86.exe" -FileName "setup-x86.exe"
Start-FileDownloadInternal : Error downloading remote file: One or more errors occurred.
Inner Exception: Unable to connect to the remote server
At C:\Program Files\AppVeyor\BuildAgent\Modules\build-worker-api\build-worker-api.psm1:242 char:2
+ Start-FileDownloadInternal -Url $Url -FileName $FileName -Timeout ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-FileDownloadInternal], Exception
+ FullyQualifiedErrorId : System.Exception,Appveyor.BuildAgent.Api.Utils.StartFileDownloadInternalCmdlet

We depend on volatile URL...


Seems quite susceptible to the ground unknowingly moving under our feet.
Actually I found it quite difficult (i.e. failed) to determine a hard link to the version being installed.
To enhance reproduciblity, we should host our own versioned archive of the required packages somewhere fairly permanent.

Instead of hosting our own Cygwin setup, maybe it's better to use the build cache [1]? We are already using that cache for third-party libs [2] and I also thought we'd cache Cygwin as well. @Nicolas: could you give this a go?

Fabio

 

This seems pertinent...
2.2.What about an automated Cygwin installation?
If you are deploying to multiple systems, the best way is to run through a full installation once, saving the entire downloaded package tree. Then, on target systems, run Cygwin Setup as a "Local Install" pointed at your downloaded package tree. You could do this non-interactively with the command line options -q -L -l x:\cygwin-local\, where your downloaded package tree is in x:\cygwin-local\ (see the next FAQ for an explanation of those options.)
(https://www.cygwin.com/faq.html)

And "pmcyg" looks interesting...  "intended to support construction of
a self-contained CDROM or DVD that can be used to install or upgrade Cygwin
on computers that do not have network access to a Cygwin mirror site."
That could be stored as a versioned archive which is added to the appveyor build cache 
so its only downloaded once,  saving per job build time and 
ensuring if it succeeds at the first job, all other jobs *should* succeed.

Options to host this versioned-cygwin-installer-archive (should be somewhere fairly permanent):
* bintray
* github releases
* git large file storage

cheers -ben


Reply | Threaded
Open this post in threaded view
|

Re: How to restart an appveyor job?

fniephaus
 
On Mon, Apr 2, 2018 at 6:23 PM Nicolas Cellier <[hidden email]> wrote:
 
Hi Fabio,
I've switched to curl for now and have a build in progress...

curl makes sense!
 
Anyway, cache or not cache, we are dependent on cygwin updates, and build can start failing anytime (for example because a compiler version changed).

That's right. Another idea for hosting our custom Cygwin: another Git repository, for example OpenSmalltalk/build-cygwin. This way, we could track changes as well.
 

2018-04-02 18:13 GMT+02:00 Fabio Niephaus <[hidden email]>:
 
On Mon, Apr 2, 2018 at 5:55 PM Ben Coman <[hidden email]> wrote:
On 2 April 2018 at 20:32, Nicolas Cellier <[hidden email]> wrote:
 
Hi all,
I wanted to solve the gcc compiler bug for pharo.sista.spur.

But of course, there were another failure due to fragile infrastructure...
https://ci.appveyor.com/project/OpenSmalltalk/vm/build/1.0.1207/job/dd3lnytbwb5jjlqa

Running Install scripts
Start-FileDownload "http://cygwin.com/setup-x86.exe" -FileName "setup-x86.exe"
Start-FileDownloadInternal : Error downloading remote file: One or more errors occurred.
Inner Exception: Unable to connect to the remote server
At C:\Program Files\AppVeyor\BuildAgent\Modules\build-worker-api\build-worker-api.psm1:242 char:2
+ Start-FileDownloadInternal -Url $Url -FileName $FileName -Timeout ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-FileDownloadInternal], Exception
+ FullyQualifiedErrorId : System.Exception,Appveyor.BuildAgent.Api.Utils.StartFileDownloadInternalCmdlet

We depend on volatile URL...


Seems quite susceptible to the ground unknowingly moving under our feet.
Actually I found it quite difficult (i.e. failed) to determine a hard link to the version being installed.
To enhance reproduciblity, we should host our own versioned archive of the required packages somewhere fairly permanent.

Instead of hosting our own Cygwin setup, maybe it's better to use the build cache [1]? We are already using that cache for third-party libs [2] and I also thought we'd cache Cygwin as well. @Nicolas: could you give this a go?

Fabio

 

This seems pertinent...
2.2.What about an automated Cygwin installation?
If you are deploying to multiple systems, the best way is to run through a full installation once, saving the entire downloaded package tree. Then, on target systems, run Cygwin Setup as a "Local Install" pointed at your downloaded package tree. You could do this non-interactively with the command line options -q -L -l x:\cygwin-local\, where your downloaded package tree is in x:\cygwin-local\ (see the next FAQ for an explanation of those options.)
(https://www.cygwin.com/faq.html)

And "pmcyg" looks interesting...  "intended to support construction of
a self-contained CDROM or DVD that can be used to install or upgrade Cygwin
on computers that do not have network access to a Cygwin mirror site."
That could be stored as a versioned archive which is added to the appveyor build cache 
so its only downloaded once,  saving per job build time and 
ensuring if it succeeds at the first job, all other jobs *should* succeed.

Options to host this versioned-cygwin-installer-archive (should be somewhere fairly permanent):
* bintray
* github releases
* git large file storage

cheers -ben


Reply | Threaded
Open this post in threaded view
|

Re: How to restart an appveyor job?

Ben Coman
 


On 3 April 2018 at 00:34, Fabio Niephaus <[hidden email]> wrote:
 
On Mon, Apr 2, 2018 at 6:23 PM Nicolas Cellier <[hidden email]> wrote:
 
Hi Fabio,
I've switched to curl for now and have a build in progress...

curl makes sense!
 
Anyway, cache or not cache, we are dependent on cygwin updates, and build can start failing anytime (for example because a compiler version changed).

That's right. Another idea for hosting our custom Cygwin: another Git repository, for example OpenSmalltalk/build-cygwin. This way, we could track changes as well.

Then later when its time to upgrade cygwin, someone would fork upstream https://github.com/cygwin/cygwin to their local machine, 
and after testing just push to  https://github.com/OpenSmalltalk/cygwin.

Or for the parnoid, clone locally direct from https://cygwin.com/git.html

Build instructions here
are not clear what is required to then install it. Would its setup.exe still need to be run anyway?

Somewhere would still be needed to host the binaries produced.
The cygwin build would only need to be done infrequently.

cheers -ben


 

2018-04-02 18:13 GMT+02:00 Fabio Niephaus <[hidden email]>:
 
On Mon, Apr 2, 2018 at 5:55 PM Ben Coman <[hidden email]> wrote:
On 2 April 2018 at 20:32, Nicolas Cellier <[hidden email]> wrote:
 
Hi all,
I wanted to solve the gcc compiler bug for pharo.sista.spur.

But of course, there were another failure due to fragile infrastructure...
https://ci.appveyor.com/project/OpenSmalltalk/vm/build/1.0.1207/job/dd3lnytbwb5jjlqa

Running Install scripts
Start-FileDownload "http://cygwin.com/setup-x86.exe" -FileName "setup-x86.exe"
Start-FileDownloadInternal : Error downloading remote file: One or more errors occurred.
Inner Exception: Unable to connect to the remote server
At C:\Program Files\AppVeyor\BuildAgent\Modules\build-worker-api\build-worker-api.psm1:242 char:2
+ Start-FileDownloadInternal -Url $Url -FileName $FileName -Timeout ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-FileDownloadInternal], Exception
+ FullyQualifiedErrorId : System.Exception,Appveyor.BuildAgent.Api.Utils.StartFileDownloadInternalCmdlet

We depend on volatile URL...


Seems quite susceptible to the ground unknowingly moving under our feet.
Actually I found it quite difficult (i.e. failed) to determine a hard link to the version being installed.
To enhance reproduciblity, we should host our own versioned archive of the required packages somewhere fairly permanent.

Instead of hosting our own Cygwin setup, maybe it's better to use the build cache [1]? We are already using that cache for third-party libs [2] and I also thought we'd cache Cygwin as well. @Nicolas: could you give this a go?

Fabio

 

This seems pertinent...
2.2.What about an automated Cygwin installation?
If you are deploying to multiple systems, the best way is to run through a full installation once, saving the entire downloaded package tree. Then, on target systems, run Cygwin Setup as a "Local Install" pointed at your downloaded package tree. You could do this non-interactively with the command line options -q -L -l x:\cygwin-local\, where your downloaded package tree is in x:\cygwin-local\ (see the next FAQ for an explanation of those options.)
(https://www.cygwin.com/faq.html)

And "pmcyg" looks interesting...  "intended to support construction of
a self-contained CDROM or DVD that can be used to install or upgrade Cygwin
on computers that do not have network access to a Cygwin mirror site."
That could be stored as a versioned archive which is added to the appveyor build cache 
so its only downloaded once,  saving per job build time and 
ensuring if it succeeds at the first job, all other jobs *should* succeed.

Options to host this versioned-cygwin-installer-archive (should be somewhere fairly permanent):
* bintray
* github releases
* git large file storage

cheers -ben