Metacello bootstrap fails due to download timeouts

Hi all,

My Travis CI builds for Squeak-Trunk 32 bits (and only these) started to
fail due to timeouts. When I looked at it in a Linux VM and it fails
to download Metacello-Base-dkh.109.mcz from SmalltalkHub. When I try
to access this version with a web browser, I get an error HTTP 500
VoMongoConnectionError
http://smalltalkhub.com/mc/dkh/metacello/main/Metacello-Base-dkh.109.mcz

A fresh 64 bit Trunk image from squeak.org when run on Windows instead
hangs when
downloading Metacello-Core-dkh.494.mcz when I do `Metacello new`.
Eventually it erred with "Error: retry with alternate repository
failed: 'ConnectionTimedOut: Data receive timed out.'" ...after about
half an hour or so. The error comes from
ConfigurationOfMetacello>>ensureMetacello:. Though it does look as if
it first tried with a repository at
http://seaside.gemtalksystems.com/ss/metacello and not SmalltalkHub. I
can download the version in a web browser without a problem:
http://seaside.gemtalksystems.com/ss/metacello/Metacello-Core-dkh.494.mcz

Yet a local 64 bit trunk smalltalkCI build succeeded and I don't know
why yet. It also needs to go through the MetacelloStub, doesn't it?

For a start, does anyone experience the same problem or is it my bad
luck with the connections? To test, print `Metacello new` in a fresh image
which does not have the Metacello class yet.

Now SmalltalkHub is deprecated as you can read on its
front page. http://smalltalkhub.com/

Tom, Tobias, Dale, you already started to discuss changing the
bootstrap of current Metacello in Squeak when the Tonel changes were
integrated, didn't you? Is there a path where we can live without any
unmaintained repositories like the ones on SmalltalkHub?

On a different level: shouldn't the bootstrapping of the
MetacelloStub, or rather the downloading of version signal an error
with a timeout sooner when it cannot retrieve a
version, instead of waiting or retrying for a long time?

Kind regards,
Jakob

Github was down today.
-t

Maybe, but it did not even get to the steps that involve GitHub as far
as I can tell. Also the 32 bit trunk builds already failed before
today.

Am Mo., 13. Juli 2020 um 19:23 Uhr schrieb Tobias Pape <[hidden email]>:

Hi Jakob,

smalltalkHub had also been offline for the past three days or so, see this thread on the Pharo mailing list: http://forum.world.st/smalltalkhub-down-td5119524.html

My vision for the Squeak/Metacello installation process is as follows (note that I haven't discussed this with anyone else yet, so there may be some false assumptions):
- The Metacello on Github gets a new CI task that, after a successful build for all Squeak platforms, produces a single MCZ that bundles the current Metacello from master
- MetacelloStub downloads the most recent MCZ for Metacello master and installs it
- A postInstall script configures the Github Metacello repository and registers it in the system, but does not download anything, since we already have the most up-to-date version of Metacello installed. This is just to provide the same impression as if we had gone out and downloaded it again.

This would have several advantages:
- no more bootstrapping in-image: the bootstrapping will essentially be moved to the CI to happen only once for everyone
- through this, we get a massively improved installation time (I assume): downloading an MCZ and installing it should be considerably faster than the various installation steps that we currently have to go through to get the most recent Metacello version
- only one source of truth: we no longer have different versions of Metacello on smalltalkHub/Gemsource, but only the one on Github. This would have prevented the short outage we had a couple weeks back where the bootstrap-Metacello version was not able to read some repositories as the format had changed since then


Concerning Squeak-trunk 32bit builds, I just tried to run one locally headful, and a failure occurs in primitiveSSLCreate, leading to a notification about ensuring to have internet and then a quiet image (see Installer>>ensureRecentMetacello, `self inform: ...`). I suppose we could consider changing this to an error instead, such that the CI build would fail rather than happily wait for ever after for you to click on the Okay button of the info window. Or, even better, create an Exception subclass for it that smalltalkCI can catch and handle appropriately, while by default the user sees the same info window.

Best,
Tom
________________________________________
From: Jakob Reschke <[hidden email]>
Sent: Monday, July 13, 2020 7:28:01 PM
To: Tobias Pape
Cc: The general-purpose Squeak developers list; Beckmann, Tom; Henrichs, Dale (Gemtalksystems); Niephaus, Fabio
Subject: Re: Metacello bootstrap fails due to download timeouts

Maybe, but it did not even get to the steps that involve GitHub as far
as I can tell. Also the 32 bit trunk builds already failed before
today.

Am Mo., 13. Juli 2020 um 19:23 Uhr schrieb Tobias Pape <[hidden email]>:

Ah sorry, I got confused with the various 32- vs 64-bit mentions.
Also, the SqueakSSL issue turned out to be just me forgetting that I did not have a well-configured 32bit environment.

I just tried running the Squot tests locally with squeak32-trunk on smalltalkCI, which turned out all green. For a couple hours now, smalltalkHub is also online again, so maybe the issues you were observing were all related to this part of the infrastructure failing and errors bubbling up to unexpected places? Or are the builds still failing on Travis?
________________________________________
From: Jakob Reschke <[hidden email]>
Sent: Monday, July 13, 2020 11:37:51 PM
To: Beckmann, Tom
Cc: Tobias Pape; The general-purpose Squeak developers list; Henrichs, Dale (Gemtalksystems); Niephaus, Fabio
Subject: Re: Metacello bootstrap fails due to download timeouts

Beckmann, Tom <[hidden email]<mailto:[hidden email]>> schrieb am Mo., 13. Juli 2020, 19:47:

Concerning Squeak-trunk 32bit builds, I just tried to run one locally headful, and a failure occurs in primitiveSSLCreate, leading to a notification about ensuring to have internet and then a quiet image (see Installer>>ensureRecentMetacello, `self inform: ...`). I suppose we could consider changing this to an error instead, such that the CI build would fail rather than happily wait for ever after for you to click on the Okay button of the info window. Or, even better, create an Exception subclass for it that smalltalkCI can catch and handle appropriately, while by default the user sees the same info window.


Since that is not what I observed, we seem to have multiple such points of non-failures...

All the previously errored builds are green now. So we can again stop
worrying about the present and return to thinking about the future.
;-)

I'm still somewhat confused that the download from the Seaside/Gemtalk
repository did not work and that SmalltalkHub was tried in the first
place. Are versions missing in the Gemtalk repository? Can look it up
myself at earliest on Saturday, more likely next week.


Am Di., 14. Juli 2020 um 08:28 Uhr schrieb Beckmann, Tom
<[hidden email]>: