Recursively downloading Pharo packages / Building images with Nix

Howdy!

I want to build Pharo images using Nix. On the one hand this should look basically like the Jenkins scripts, but on the other hand the build must run inside a sandbox that is isolated from the network. So I need to download all of the code that I need ahead of time because I am not able to contact Github, SmalltalkHub, etc, once the build is running.

I am not sure how to approach this problem yet but I have a preliminary question.

Is there an easy way that I could "recursively" download all the Smalltalk code (etc) required for an application onto the local filesystem, and then load this into a Pharo image that does not have network access?

Concrete example - How could I write a Pharo script that downloads all the software required to install Moose in a standard Pharo 6.0 image? Then how could I load that into an image off the local filesystem? 

I feel like there should be a solution here, since Monticello seems to be able to cache packages locally on the filesystem, but I don't how enough experience to quite know how to approach it.

If this is possible then the "utopian" vision could be to recursively download every Pharo package that exists, in every relevant version, and make them available in the Nix universe. This is what Nix does with Python, R, Emacs, etc packages.

Or if this turns out to be difficult for some reason then I could perhaps build the Pharo image externally - e.g. on the Inria Jenkins - and then import that into Nix as an opaque binary. This is probably the simplest solution but it is not very appealing from my Nix-centric viewpoint because I won't have visibility into what has changed between one image build and the next.

Feedback would be much appreciated!

-Luke

P.S. Screenshot from the Pharo part of my application that is starting to come together, visualizing Lua JIT compiler IR code to make it more understandable than disassembly listings: https://github.com/raptorjit/raptorjit/pull/63#issuecomment-310138536.

Typically Monticello mcz files are hosted on services shared via HTTP that you can download with wget or curl, e.g....

http://smalltalkhub.com/mc/Pharo/Pharo60/main

I guess if you pre-seed the package-cache, Pharo won't try to go to the network to find the package.

Hi again Ben,

I have an idea for a hacky way to import all of the Pharo packages into my universe. Shooting holes in this idea would be welcome :).

Suppose that for each package "P" in the catalog I want to make a list of the recursive dependencies (mcz files) and their sha256 hashes. I would do this using a script that runs Pharo outside the sandbox and with access to the network:

- Start Pharo with an empty package cache.

- Download "P" and all of its dependencies automatically with Gofer.

- Inspect the package cache to see every mcz file that was downloaded and its sha256 hash.

Then I could use this process to bootstrap a package repository:

- Having a copy of all the .mcz files in a common ftp directory.

- Having a mapping of filename->sha256 to provide to nix.

- Having a mapping from package name ("P") to all .mcz dependencies.

This package repository could then be used for Nix such that I say which packages I want and it automatically downloads all of the required .mcz files, validates them with sha256 to make sure the build is reproducible, and then loads them all into an image and dumps the result ready for use.

So then on the Nix side I would write something like:

nix-build -E 'pharo6 { packages = [ roassal2 neojson ... ]; }'

and it would automatically create an image for me based on the package repository definition that it finds in Git.

Sane? Workable? Fatal flaws?

Hi again Ben,

I have an idea for a hacky way to import all of the Pharo packages into my universe. Shooting holes in this idea would be welcome :).

Suppose that for each package "P" in the catalog I want to make a list of the recursive dependencies (mcz files) and their sha256 hashes. I would do this using a script that runs Pharo outside the sandbox and with access to the network:

- Start Pharo with an empty package cache.

- Download "P" and all of its dependencies automatically with Gofer.

- Inspect the package cache to see every mcz file that was downloaded and its sha256 hash.

Then I could use this process to bootstrap a package repository:

- Having a copy of all the .mcz files in a common ftp directory.

- Having a mapping of filename->sha256 to provide to nix.

- Having a mapping from package name ("P") to all .mcz dependencies.

This package repository could then be used for Nix such that I say which packages I want and it automatically downloads all of the required .mcz files, validates them with sha256 to make sure the build is reproducible, and then loads them all into an image and dumps the result ready for use.

So then on the Nix side I would write something like:

nix-build -E 'pharo6 { packages = [ roassal2 neojson ... ]; }'

and it would automatically create an image for me based on the package repository definition that it finds in Git.

Sane? Workable? Fatal flaws?

Hi Luke,

On Fri, Jun 23, 2017 at 8:59 AM, Luke Gorrie <[hidden email]> wrote:

Hi again Ben,

I have an idea for a hacky way to import all of the Pharo packages into my universe. Shooting holes in this idea would be welcome :).

Suppose that for each package "P" in the catalog I want to make a list of the recursive dependencies (mcz files) and their sha256 hashes. I would do this using a script that runs Pharo outside the sandbox and with access to the network:

- Start Pharo with an empty package cache.

- Download "P" and all of its dependencies automatically with Gofer.

- Inspect the package cache to see every mcz file that was downloaded and its sha256 hash.

Then I could use this process to bootstrap a package repository:

- Having a copy of all the .mcz files in a common ftp directory.

- Having a mapping of filename->sha256 to provide to nix.

- Having a mapping from package name ("P") to all .mcz dependencies.

This package repository could then be used for Nix such that I say which packages I want and it automatically downloads all of the required .mcz files, validates them with sha256 to make sure the build is reproducible, and then loads them all into an image and dumps the result ready for use.

So then on the Nix side I would write something like:

nix-build -E 'pharo6 { packages = [ roassal2 neojson ... ]; }'

and it would automatically create an image for me based on the package repository definition that it finds in Git.

Sane? Workable? Fatal flaws?

Workable?  Yes.  Fatal flaws?  None.

Sane?  Why introduce your own ftp directory?  All you're doing is replicating all the repositories out there.  Basically you want to seed a package-cache with any and all versions of the relevant packages.  The ftp directory looks like a superfluous step.  What you need to construct is the repository crawler that locates as many versions as you can find.  The list of versions should be in each package; it's part of its history.  You then have to find out where they are.  As packages move from repository to repository (a sad fact of life) so tracking down these versions becomes difficult.

Sane?  Why introduce your own ftp directory?

For those projects described with Metacello baselines or configurations you
can issue a #fetch command to download all the Monticello packages needed to
load the projects into your image....

But btw note, IIUC the bootstrap hasn't got a far as programatically creating an empty file and writing nil,true,false to it yet (if ever?).  You will always be starting with a minimal.image rather than an empty.image.

Workable?  Yes.  Fatal flaws?  None.