Installer ensureRecentMetacello is now broken

After a deeply embarrassing attempt to demonstrate loading up an image with seaside/postgres/etc this morning I've been trying to track down the source of the error. Given that doing this just a couple of weeks ago worked perfectly, this is annoying.

The initial problem is that somebody has broken the stuff we get loaded by
` Installer ensureRecentMetacello.`

It now includes a bunch of code that relies upon Zinc, FileLocator and FileSystem. So far as I can tell the problems have been introduced by changed to -
BaselineOfMetaCello (tobe.98) {last working version is Esteban Lorenzo.97}
Metacello-MC (tobe.738) {last working version was Esteban Lorenzo.735}
Metacello-Platform (tobe.60) {last working version topa.45}

Clearly we have a problem here in that the changes were not actually tested adequately, probably the #ensureRecentMetacello method should actually use some form of certified set of packages, and anyone in reach of 'tobe' should give them a slap on the wrist and stop their chocolate privileges for a month.

I can't even see a way to fix this locally right now because the stupidity around git appears to prevent finding the last-working versions. Just why exactly is anyone enamoured of this?

Any solutions anyone can suggest here?

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful Latin Phrases:- Catapultam habeo. Nisi pecuniam omnem mihi dabis, ad caput tuum saxum immane mittam = I have a catapult. Give me all the money, or I will fling an enormous rock at your head.

'tobe' is a very nice and cautious person and has written just some 10 days ago:

>please excuse the issues. In our recent changes to Metacello for Squeak we missed issues with the bootstrapping process for Metacello. These issues should now be resolved.

> Through this, we also got helpful input from Dale on the bootstrapping process of Metacello in Squeak and will investigate making the process faster and more resilient in the future. See here for some more details: https://github.com/squeak-smalltalk/squeak-ston/pull/1

>Best,
>Tom

To solve this without fingerpointing, lets start with some infogathering:
- did you try on a 5.3 or a trunk?
- Can you give the transcript output during the load?


:D
-topa

> I can't even see a way to fix this locally right now because the stupidity around git appears to prevent finding the last-working versions. Just why exactly is anyone enamoured of this?
>
> Any solutions anyone can suggest here?
>
> tim

> On 2020-06-16, at 2:42 PM, Tobias Pape <[hidden email]> wrote:
>
> To solve this without fingerpointing, lets start with some infogathering:

'pointing fingers' is an important part of working out where things went wrong and discovering how to fix them. Collectively we should probably try to find a way to 'certify' a particular version of something foundational so that we avoid sudden and very publicly embarrassing problem like this. Relying on the head version isn't safe in general.  It isn't easy but we still need to try.

> - did you try on a 5.3 or a trunk?

A clean freshly loaded 5.3-19435-64bit image running on a Mac - also tried on a Windows machine during that failed demo. The successful install was a couple of weeks ago on a 5.3-19435-32 image on one of my Pi (which is why simply copying over that image is not a very useful solution - need to run a 64 bit image on the PC)

> - Can you give the transcript output during the load?

Sure -


Unknown cookie field: SameSiteUnknown cookie field: SameSiteUnknown cookie field: SameSite found ConfigurationOfMetacello-topa.803.mcz...

ConfigurationOfMetacello class>>unloadMetacello (Gofer is Undeclared) loaded

Loading 1.0-beta.31.1.8 of ConfigurationOfMetacello...
Fetched -> ConfigurationOfGofer-dkh.45 --- /Users/tim/Documents/Squeak/package-cache --- http://seaside.gemtalksystems.com/ss/metacello
Loaded -> ConfigurationOfGofer-dkh.45 --- /Users/tim/Documents/Squeak/package-cache --- http://seaside.gemtalksystems.com/ss/metacello
Project: Gofer Core 1.0.5.1
Fetched -> Gofer-Core.squeak-dkh.135 --- /Users/tim/Documents/Squeak/package-cache --- http://seaside.gemtalksystems.com/ss/metacello
Fetched -> Metacello-Core-dkh.494 --- /Users/tim/Documents/Squeak/package-cache --- http://seaside.gemtalksystems.com/ss/metacello
Fetched -> Metacello-Platform.squeak-dkh.17 --- /Users/tim/Documents/Squeak/package-cache --- http://seaside.gemtalksystems.com/ss/metacello
Loaded -> Gofer-Core.squeak-dkh.135 --- /Users/tim/Documents/Squeak/package-cache --- cache
Loaded -> Metacello-Core-dkh.494 --- /Users/tim/Documents/Squeak/package-cache --- cache
MetacelloSqueakPlatform>>downloadFile:to: (OSProcess is Undeclared)
MetacelloSqueakPlatform>>downloadFile:to: (OSProcess is Undeclared)
MetacelloSqueakPlatform>>extractRepositoryFrom:to: (OSProcess is Undeclared)
Loaded -> Metacello-Platform.squeak-dkh.17 --- /Users/tim/Documents/Squeak/package-cache --- cache
...finished 1.0-beta.31.1.8
Loading 1.0-beta.32.3 of ConfigurationOfMetacello...
Fetched -> ConfigurationOfFileTree-dkh.67 --- http://ss3.gemtalksystems.com/ss/FileTree --- /Users/tim/Documents/Squeak/package-cache
Loaded -> ConfigurationOfFileTree-dkh.67 --- http://ss3.gemtalksystems.com/ss/FileTree --- /Users/tim/Documents/Squeak/package-cache
Project: FileTree stable
Fetched -> MonticelloFileTree-Core.squeak43-dkh.169 --- http://ss3.gemtalksystems.com/ss/FileTree --- /Users/tim/Documents/Squeak/package-cache
Fetched -> MonticelloFileTree-FileDirectory-Utilities.squeak43-dkh.12 --- http://ss3.gemtalksystems.com/ss/FileTree --- /Users/tim/Documents/Squeak/package-cache
Project: Gofer stable
Fetched -> Metacello-Base-dkh.109 --- http://smalltalkhub.com/mc/dkh/metacello/main --- /Users/tim/Documents/Squeak/package-cache
Fetched -> Metacello-Core-dkh.695 --- http://seaside.gemtalksystems.com/ss/metacello --- /Users/tim/Documents/Squeak/package-cache
Fetched -> Metacello-MC-dkh.680 --- http://seaside.gemtalksystems.com/ss/metacello --- /Users/tim/Documents/Squeak/package-cache
Fetched -> Metacello-Platform.squeak-tfel.25 --- http://smalltalkhub.com/mc/dkh/metacello/main --- /Users/tim/Documents/Squeak/package-cache
Fetched -> Metacello-ToolBox-dkh.134 --- http://seaside.gemtalksystems.com/ss/metacello --- /Users/tim/Documents/Squeak/package-cache
Fetched -> Metacello-FileTree-dkh.29 --- http://smalltalkhub.com/mc/dkh/metacello/main --- /Users/tim/Documents/Squeak/package-cache
Fetched -> Metacello-GitHub-dkh.29 --- http://seaside.gemtalksystems.com/ss/metacello --- /Users/tim/Documents/Squeak/package-cache
MCFileTreeStCypressWriter class>>fileNameForSelector:(specials is shadowed)
Loaded -> MonticelloFileTree-Core.squeak43-dkh.169 --- http://ss3.gemtalksystems.com/ss/FileTree --- cache
Loaded -> MonticelloFileTree-FileDirectory-Utilities.squeak43-dkh.12 --- http://ss3.gemtalksystems.com/ss/FileTree --- cache
BaselineOf>>projectClass (MetacelloMCBaselineProject is Undeclared)
ConfigurationOf>>versionNumberClass (MetacelloSemanticVersionNumber is Undeclared)
Loaded -> Metacello-Base-dkh.109 --- http://smalltalkhub.com/mc/dkh/metacello/main --- cache
MetacelloProjectRegistration>>version (MetacelloMCBaselineProject is Undeclared)
MetacelloBaselineSpecGenerator>>projectSpecCreationBlock (MetacelloMCBaselineProject is Undeclared)
Loaded -> Metacello-Core-dkh.695 --- http://seaside.gemtalksystems.com/ss/metacello --- cache
Loaded -> Metacello-MC-dkh.680 --- http://seaside.gemtalksystems.com/ss/metacello --- cache
Loaded -> Metacello-Platform.squeak-tfel.25 --- http://smalltalkhub.com/mc/dkh/metacello/main --- cache
Loaded -> Metacello-ToolBox-dkh.134 --- http://seaside.gemtalksystems.com/ss/metacello --- cache
Loaded -> Metacello-FileTree-dkh.29 --- http://smalltalkhub.com/mc/dkh/metacello/main --- cache
Loaded -> Metacello-GitHub-dkh.29 --- http://seaside.gemtalksystems.com/ss/metacello --- cache
Using cache repository for ConfigurationOfGofer project package
Evaluated -> 1.0-beta.32.3 [ConfigurationOfMetacello] >> metacelloPrimeRegistry
...finished 1.0-beta.32.3
Fetched -> BaselineOfMetacello-tobe.98 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Loaded -> BaselineOfMetacello-tobe.98 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Loading baseline of BaselineOfMetacello...
Fetched -> BaselineOfFileTree-dkh.4 --- github://dalehenrich/filetree:squeak4.3/repository --- github://dalehenrich/filetree:squeak4.3/repository
Loaded -> BaselineOfFileTree-dkh.4 --- github://dalehenrich/filetree:squeak4.3/repository --- github://dalehenrich/filetree:squeak4.3/repository
Fetched -> ConfigurationOfHelpSystem-DaleHenrichs.13 --- http://www.squeaksource.com/MetacelloRepository --- http://www.squeaksource.com/MetacelloRepository
Loaded -> ConfigurationOfHelpSystem-DaleHenrichs.13 --- http://www.squeaksource.com/MetacelloRepository --- http://www.squeaksource.com/MetacelloRepository
Fetched -> BaselineOfSton-cypress.1 --- github://squeak-smalltalk/squeak-ston:squeak/repository --- github://squeak-smalltalk/squeak-ston:squeak/repository
Loaded -> BaselineOfSton-cypress.1 --- github://squeak-smalltalk/squeak-ston:squeak/repository --- github://squeak-smalltalk/squeak-ston:squeak/repository
Fetched -> Metacello-Base-topa.124 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Project: Ston baseline
Fetched -> STON-Core-cypress.1 --- github://squeak-smalltalk/squeak-ston:squeak/repository --- github://squeak-smalltalk/squeak-ston:squeak/repository
Fetched -> STON-Tests-cypress.1 --- github://squeak-smalltalk/squeak-ston:squeak/repository --- github://squeak-smalltalk/squeak-ston:squeak/repository
Fetched -> STON-Text support-cypress.1 --- github://squeak-smalltalk/squeak-ston:squeak/repository --- github://squeak-smalltalk/squeak-ston:squeak/repository
Project: Gofer stable
Fetched -> Metacello-Core-EstebanLorenzano.832 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Fetched -> Metacello-MC-tobe.738 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Fetched -> Metacello-Platform.squeak-tobe.60 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Fetched -> Metacello-ToolBox-EstebanLorenzano.149 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Fetched -> Metacello-Cypress-CyrilFerlicot.8 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Project: HelpSystem 1.1
Fetched -> Metacello-Help-dkh.12 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Project: FileTree baseline
Fetched -> MonticelloFileTree-Core-fn.185 --- github://dalehenrich/filetree:squeak4.3/repository --- github://dalehenrich/filetree:squeak4.3/repository
Fetched -> MonticelloFileTree-FileDirectory-Utilities-fn.14 --- github://dalehenrich/filetree:squeak4.3/repository --- github://dalehenrich/filetree:squeak4.3/repository
Fetched -> Metacello-FileTree-dkh.36 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Fetched -> Metacello-GitBasedRepository-dkh.21 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Fetched -> Metacello-GitHub-topa.46 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Fetched -> Metacello-Bitbucket-topa.6 --- github://Metacello/metacello:master/repository --- github://Metacello/metacello:master/repository
Loaded -> Metacello-Base-topa.124 --- github://Metacello/metacello:master/repository --- cache
Loaded -> STON-Core-cypress.1 --- github://squeak-smalltalk/squeak-ston:squeak/repository --- cache
STONReaderTest>>testFileReferenceBackwardsCompatibility (FileSystem is Undeclared)
STONReaderTest>>testFileReferenceBackwardsCompatibility (FileLocator is Undeclared)
STONReaderTest>>testMimeType (ZnMimeType is Undeclared)
STONReaderTest>>testMimeType (ZnMimeType is Undeclared)
STONTest class>>fastReadFromFileNamed: (ZnBufferedReadStream is Undeclared)
STONTest class>>fastWrite:toFileNamed: (ZnBufferedWriteStream is Undeclared)
STONWriteReadTest>>testDiskFiles (FileLocator is Undeclared)
STONWriteReadTest>>testDiskFiles (FileLocator is Undeclared)
STONWriteReadTest>>testDiskFiles (FileLocator is Undeclared)
STONWriteReadTest>>testFileSystemSupport (FileLocator is Undeclared)
STONWriteReadTest>>testFileSystemSupport (FileLocator is Undeclared)
STONWriteReadTest>>testFileSystemSupport (FileLocator is Undeclared)
STONWriteReadTest>>testFileSystemSupport (FileLocator is Undeclared)
STONWriteReadTest>>testFileSystemSupport (FileLocator is Undeclared)
STONWriteReadTest>>testFileSystemSupport (FileLocator is Undeclared)
STONWriteReadTest>>testMimeTypes (ZnMimeType is Undeclared)
STONWriteReadTest>>testMimeTypes (ZnMimeType is Undeclared)
STONWriterTest>>testMimeType (ZnMimeType is Undeclared)
STONWriterTest>>testMimeType (ZnMimeType is Undeclared)
Loaded -> STON-Tests-cypress.1 --- github://squeak-smalltalk/squeak-ston:squeak/repository --- cache
Loaded -> STON-Text support-cypress.1 --- github://squeak-smalltalk/squeak-ston:squeak/repository --- cache
Loaded -> Metacello-Core-EstebanLorenzano.832 --- github://Metacello/metacello:master/repository --- cache
Loaded -> Metacello-MC-tobe.738 --- github://Metacello/metacello:master/repository --- cache
MCFilesystemFetchOnlyRepository>>canUpgradeTo: (MCGitBasedNetworkRepository is Undeclared)
Loaded -> Metacello-Platform.squeak-tobe.60 --- github://Metacello/metacello:master/repository --- cache
Loaded -> Metacello-ToolBox-EstebanLorenzano.149 --- github://Metacello/metacello:master/repository --- cache
Loaded -> Metacello-Cypress-CyrilFerlicot.8 --- github://Metacello/metacello:master/repository --- cache
Loaded -> Metacello-Help-dkh.12 --- github://Metacello/metacello:master/repository --- cache
Loaded -> MonticelloFileTree-Core-fn.185 --- github://dalehenrich/filetree:squeak4.3/repository --- cache
Loaded -> MonticelloFileTree-FileDirectory-Utilities-fn.14 --- github://dalehenrich/filetree:squeak4.3/repository --- cache
Loaded -> Metacello-FileTree-dkh.36 --- github://Metacello/metacello:master/repository [:] --- cache
Loaded -> Metacello-GitBasedRepository-dkh.21 --- github://Metacello/metacello:master/repository [:] --- cache
Loaded -> Metacello-GitHub-topa.46 --- github://Metacello/metacello:master/repository [348ac29:master] --- cache
Loaded -> Metacello-Bitbucket-topa.6 --- github://Metacello/metacello:master/repository [348ac29:master] --- cache
priming registry from image. See https://github.com/dalehenrich/metacello-work/issues/197
Using cache repository for ConfigurationOfGofer project package
Evaluated -> baseline [BaselineOfMetacello] >> reprimeRegistryIssue197
...finished baseline

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Design: The activity of preparing for a design review.

Hi Tim

> On 17.06.2020, at 00:22, tim Rowledge <[hidden email]> wrote:
>
>
>
>> On 2020-06-16, at 2:42 PM, Tobias Pape <[hidden email]> wrote:
>>
>> To solve this without fingerpointing, lets start with some infogathering:
>
> 'pointing fingers' is an important part of working out where things went wrong and discovering how to fix them.

Yeah, i just happen to know that the person put a lot of effort and time into improving the Metacello bootstrap,
that was no just-like-that thing. I understand you're upset after your demo, seeing it having worked earlier and not anymore due to the workings of other people. I'm just getting a bit more sensitive to witty complaints, even if meant friendly.


> Collectively we should probably try to find a way to 'certify' a particular version of something foundational so that we avoid sudden and very publicly embarrassing problem like this. Relying on the head version isn't safe in general.  It isn't easy but we still need to try

That's why we here, to find out why the things at hand are not in the shape we thought.

>
>> - did you try on a 5.3 or a trunk?
>
> A clean freshly loaded 5.3-19435-64bit image running on a Mac - also tried on a Windows machine during that failed demo. The successful install was a couple of weeks ago on a 5.3-19435-32 image on one of my Pi (which is why simply copying over that image is not a very useful solution - need to run a 64 bit image on the PC)

Thanks!

>
>> - Can you give the transcript output during the load?

Thats helpful.
From what I see,  Zinc and FileSystem(incl. FileLocator) are Test-specific. These should surely be fixed but are most probably harmless for Metacello use.

That said, what was actually the faulty behavior you observed?
The load looks ok-ish from over here :)

Best regards
        -Tobias

> On 2020-06-16, at 4:02 PM, Tobias Pape <[hidden email]> wrote:
>
> From what I see,  Zinc and FileSystem(incl. FileLocator) are Test-specific. These should surely be fixed but are most probably harmless for Metacello use.

OK, that's interesting - and not at all obvious when you're desperately trying to rescue a demo!

>
> That said, what was actually the faulty behavior you observed?

The immediate fault was the Seaside error I've just sent a mail about (deprecation warning causing triggering of a deeper Seaside bug) and then one I still have no idea about (loading OSProcess with `Installer ss...` worked but following that with attempting to load Cryptography via `Installer ss...` lead to as-yet incomprehensible URL failures). You have to start debugging somewhere and seeing a load of errors in a foundational load system must inevitably make you suspicious of everything that follows.

OK,  now let's see what is actually happening with loading crypto...

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful random insult:- Not enough sense to come in out of the rain.

Hi Tim,

sorry to hear that you encountered troubles during your demo.

If I understand correctly we have already moved the blame from Metacello to a later step in the loading process, or am I mistaken? Could you send the do-its you were trying to execute so I can try to reproduce the problems here?

Best,
Tom
________________________________________
From: Squeak-dev <[hidden email]> on behalf of tim Rowledge <[hidden email]>
Sent: Wednesday, June 17, 2020 1:15:12 AM
To: The general-purpose Squeak developers list
Subject: Re: [squeak-dev] Installer ensureRecentMetacello is now broken

> On 2020-06-16, at 4:02 PM, Tobias Pape <[hidden email]> wrote:
>
> From what I see,  Zinc and FileSystem(incl. FileLocator) are Test-specific. These should surely be fixed but are most probably harmless for Metacello use.

OK, that's interesting - and not at all obvious when you're desperately trying to rescue a demo!

>
> That said, what was actually the faulty behavior you observed?

The immediate fault was the Seaside error I've just sent a mail about (deprecation warning causing triggering of a deeper Seaside bug) and then one I still have no idea about (loading OSProcess with `Installer ss...` worked but following that with attempting to load Cryptography via `Installer ss...` lead to as-yet incomprehensible URL failures). You have to start debugging somewhere and seeing a load of errors in a foundational load system must inevitably make you suspicious of everything that follows.

OK,  now let's see what is actually happening with loading crypto...

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful random insult:- Not enough sense to come in out of the rain.

Hi Tim,

Am Di., 16. Juni 2020 um 22:43 Uhr schrieb tim Rowledge <[hidden email]>:
>
> I can't even see a way to fix this locally right now because the stupidity around git appears to prevent finding the last-working versions. Just why exactly is anyone enamoured of this?
>

First let me second Tobias's vouch for Tom. After that, the above
remark triggered my sensitive side. ;-)

Can you tell us what exactly you tried to do to find the last-working versions?

In general, git is made for just these things and does not prevent
them, and works reliably. So I suspect there may be a problem with the
tools in Squeak around it (e. g. Metacello or the
download-repository-Monticello-adpaters involved). For example,
Metacallo sometimes needs some extra steps in a git-hosted project to
be able to update (or downgrade) it through Metacello. Or everything
worked as intended but it just did not solve your problem because it
was not related to a broken Metacello, as Tobias discussed with you,
we'll see.

Kind regards,
Jakob

> On 2020-06-16, at 11:37 PM, Beckmann, Tom <[hidden email]> wrote:
>
> sorry to hear that you encountered troubles during your demo.

And apologies for sounding a bit cross. I *was* a bit cross at the time of course but that isn't a reason to be rude to you.


>
> If I understand correctly we have already moved the blame from Metacello to a later step in the loading process, or am I mistaken? Could you send the do-its you were trying to execute so I can try to reproduce the problems here?

What happened is basically -
I had a simple script to load Seaside/Postgresql/osprocess/cryptography
Ran it as part of a meeting meant to show off how easy it is to use Squeak for development
So many things failed
 - loading metacello left complaints in the Transcript and things in Undeclared
 - loading Seaside seemed to go ok but see later
 - loading Postgresql looked ok
 - loading Cryptography just blew up
 - attempting to show off that at least Seaside worked... didn't

So inevitably the first reaction is that the apparent problems with installing metacello were likely to have cascaded into problems correctly loading the rest. Some digging showed the inappropriate usage of the Zinc net connection stuff and that implied issues big enough to be concerned about.

After Tobias suggested that the Zinc stuff might be a red herring I did more digging in the Seaside behaviour and noticed the failure to even handle the error walkback correctly; which eventually lead to discovering that the Seaside code is using some deprecated messages and that the walkback handling code has probably not worked properly in at least a decade!

After that, the puzzle was why load OSProcess worked immediately but Cryptography failed so spectacularly; which turned out to be an error in handling mcm files by the newly installed squeaksource upgrad.

All in all a really annoying storm of multiple small issues cascading to make a perfectly infuriating and very embarrassing demo. I'm hoping to rescue the situation today.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Long computations that yield zero are probably all for naught.