Fwd: Nix package for Pharo flavor of opensmalltalk-vm

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fwd: Nix package for Pharo flavor of opensmalltalk-vm

Luke Gorrie
 
Hoi,

Just summarizing the "Pharo 6 snap install" thread from -users and -dev list,

I made a nix package for the latest (git master) Pharo VM with much help from the mailing list.

If you first install nix:


Then it's easy to install the package from my development branch:


and then you can run it with pharo-vm-x (or pharo-vm-nox for headless.)

Nix is a bit special. It will make a deep copy of all the dependencies including libcairo, glibc, etc. This means that it really should work the same on your computer as on mine, and it should also work the same if you run that command 5 years from now. (Like a cross between apt-get and docker.) The dependencies and be updated and debugged in a disciplined way (see example cute trick.)

Here is the source for the build script:

This is based on the mvm script for Pharo. On the one hand it's simpler because nix takes care of all the third-party dependencies automatically, but on the other hand it's complicated because nix builds have to be referentially transparent and can't use curl, whoami, date, git, etc. There are also potentially many versions of each program installed on the machine and so you can't use global names like /bin/rm but rather have to rely on $PATH to give you the right one.

So I made some straightforward adaptations to accommodate nix, and also e.g. to avoid building with gcc5 which crashes during compilation.

My script is based on work by Damien Cassou who packaged an earlier version of the VM and has since moved on to other projects. I picked up the maintainership from him and will send this package to nix upstream once I test it a little better.

Why bother packaging the Pharo VM with nix? There are a few reasons for me. I already run NixOS and the Pharo binaries don't work there. Building from source takes more work than 'configure && make install' so it's worth automating. I'm not comfortable with the zeroconf installers because they feel opaque and I am never quite sure what software I am running on each machine. And I want to build a large application including Pharo + many other components and I need a tool to keep me out of dependency hell.

Some open issues for me:

I'd like to be able to run the test suite, e.g. test runner for Pharo 6 image, and get a pass/fail result with no user interaction. I could have the nix upstream CI do this to check for breakages when dependencies are updated. Is there an easy way to do that? (How?)

I'd like to remove the warning asking for realtime scheduling at startup. I feel that cost/benefit is too high. The message is scary and it's the very first interaction with a new user. The instructions don't work on all distros e.g. not on NixOS. Playing with realtime scheduler priorities can have unintended consequences and has to be considered in context of the whole machine. I'd expect the default scheduler to provide decent latency with default settings e.g. ~10ms from event (?). So the benefits seem uncertain to me, but I don't want to do something that will degrade the user experience downstream and reflect badly on Pharo. I am open to words of caution. (I hate it when people package _my_ software and take artistic license.)

I have to think more about which VM variants to provide and how much magic to employ in choosing the right one when opening an image. Just now I am providing the 32-bit Spur VM by default and that seems to work with most images. I also have the 64-bit Spur VM as an option for people who specifically want to run 64-bit images. Do I need to provide more VMs? Which interesting Pharo images would I _not_ be able to run using the 32-bit Spur VM? Feedback would be welcome.

Have to think about whether to support MacOSX. Probably. I wrote support for it in the script but haven't tested it yet.

More on Nix: gentle introduction and deep dive thesis. This is the best new technology that I have stumbled upon in the past couple of years.

Cheers!
-Luke



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Nix package for Pharo flavor of opensmalltalk-vm

Ben Coman
 


On Tue, Apr 18, 2017 at 4:13 AM, Luke Gorrie <[hidden email]> wrote:
>
>  
> Hoi,
>
> Just summarizing the "Pharo 6 snap install" thread from -users and -dev list,
>
> I made a nix package for the latest (git master) Pharo VM with much help from the mailing list.

btw, just to clarify, the "master" branch of https://github.com/pharo-project/pharo-vm
is synchronised to the "Cog" branch of https://github.com/OpenSmalltalk/opensmalltalk-vm

>
> If you first install nix:
>     curl https://nixos.org/nix/install | sh
>
> Then it's easy to install the package from my development branch:
>     nix-env -f https://github.com/lukego/nixpkgs/archive/pharo6.tar.gz -iA pkgs.pharo-vm
>
> and then you can run it with pharo-vm-x (or pharo-vm-nox for headless.)

This worked fine for me on 32bit Debian 8 Jessie, resulting in...
  $ pharo-vm-x  -version
5.0-git.1c38b03fb043a2962f30f080db5b1292b5b7badb  Tue Apr 18 16:46:48 UTC 2017 gcc 6.3.0 [Production Spur VM]
CoInterpreter VMMaker.oscog-eem.2188 uuid: ff4ca601-cd05-4792-ab0d-dcdf19975239 Jan  1 1970
StackToRegisterMappingCogit VMMaker.oscog-eem.2188 uuid: ff4ca601-cd05-4792-ab0d-dcdf19975239 Jan  1 1970
VM: git.1c38b03fb043a2962f30f080db5b1292b5b7badb https://github.com/pharo-project/pharo-vm $ Date: Wed Apr 12 19:25:16 2017 +0200 $
Plugins:  
Linux dom0 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) i686 GNU/Linux
plugin path: /nix/store/w7hp77nli2khxvwp2dx3sbm6a3z4drlp-pharo-vm/lib/ [default: /nix/store/w7hp77nli2khxvwp2dx3sbm6a3z4drlp-pharo-vm/lib/]


>
> Nix is a bit special. It will make a deep copy of all the dependencies including libcairo, glibc, etc. This means that it really should work the same on your computer as on mine, and it should also work the same if you run that command 5 years from now. (Like a cross between apt-get and docker.) The dependencies and be updated and debugged in a disciplined way (see example cute trick.)
https://github.com/lukego/blog/issues/17

Thats quite amazing to be able to bisect the compiler tool chain dependencies across OS versions like that.  

 
>
>
> Here is the source for the build script:
> https://github.com/lukego/nixpkgs/blob/pharo6/pkgs/development/pharo/vm/build-vm.nix

How the parameters are provided is also interesting...


>
> This is based on the mvm script for Pharo. On the one hand it's simpler because nix takes care of all the third-party dependencies automatically, but on the other hand it's complicated because nix builds have to be referentially transparent and can't use curl, whoami, date, git, etc. There are also potentially many versions of each program installed on the machine and so you can't use global names like /bin/rm but rather have to rely on $PATH to give you the right one.
>
> So I made some straightforward adaptations to accommodate nix, and also e.g. to avoid building with gcc5 which crashes during compilation.
>
> My script is based on work by Damien Cassou who packaged an earlier version of the VM and has since moved on to other projects. I picked up the maintainership from him and will send this package to nix upstream once I test it a little better.


I had seen Damien mention nix before, but I didn't realise it covered reproducibility of the whole build environment so completely.
 
>
>
> Why bother packaging the Pharo VM with nix? There are a few reasons for me. I already run NixOS and the Pharo binaries don't work there. Building from source takes more work than 'configure && make install' so it's worth automating. I'm not comfortable with the zeroconf installers because they feel opaque and I am never quite sure what software I am running on each machine. And I want to build a large application including Pharo + many other components and I need a tool to keep me out of dependency hell.


> More on Nix: gentle introduction

This seems like useful tech for on-boarding new developers in a geographically dispersed team, "A Nix expression can serve as a declarative specification of all the dependencies that your project needs. Nix can then automatically build or download these dependencies in the exact versions required by your project, freeing developers from having to do this manually."  

Being able to try a compiler upgrade and then cleanly rollback is a great feature.  I gather that under the covers, so switching between two environments is just a quick link change? 
 
Having the package hash based on "the source code of the package and the build script, [and] also dependencies such as the C compiler and libraries against which [it] links (such as OpenSSL)" seems a really strong approach to configuration management and reproducibility of historical build environments.  As well as being great for production environments I could imagine a newcomer reporting "I'm having trouble building, here is my nix configuration" and an expert could use `nix-shell` to build within an identical environment to avoid the "works for me" phenomenon of hidden dependencies.  


> This is the best new technology that I have stumbled upon in the past couple of years.

The world turns.  Thanks for the heads up.  


Now would any of the other dialects like to add to this? 
At a minimum, perhaps just adding sources filesto...

cheers -ben


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Nix package for Pharo flavor of opensmalltalk-vm

Luke Gorrie
 
On 18 April 2017 at 18:59, Ben Coman <[hidden email]> wrote:
This worked fine for me on 32bit Debian 8 Jessie, resulting in...

Cool! Thanks for testing that.

The reason you see some "Jan 1 1970" timestamps is that nix resets file modification times back to the epoch to make the build more reproducible.

Being able to try a compiler upgrade and then cleanly rollback is a great feature.  I gather that under the covers, so switching between two environments is just a quick link change? 

Yep.

You can also skip the symlinks and construct the software environment you want with a custom $PATH. Here is a command to start a sub-shell with exactly the build environment used for compiling the vm:
This will start a shell with $PATH including exactly the expected versions of bash, gcc, autoconf, libtool, etc, that the build would use. Good for trouble shooting.

The binaries that you build will also remember exactly which shared objects they are supposed to link with (e.g. which build of glibc):

$ ldd $(which ls)
        linux-gate.so.1 (0xf7767000)
        librt.so.1 => /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/librt.so.1 (0xf7759000)
        libacl.so.1 => /nix/store/v480rw5maz01wh5ysrma8mhl6qy1kvjd-acl-2.2.52/lib/libacl.so.1 (0xf774f000)
        libattr.so.1 => /nix/store/h6k7bwddlfzv1sbwxbnl795h0317wrwp-attr-2.4.47/lib/libattr.so.1 (0xf7749000)
        libpthread.so.0 => /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/libpthread.so.0 (0xf772b000)
        libc.so.6 => /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/libc.so.6 (0xf7573000)

        /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/ld-linux.so.2 (0xf7769000)


Now would any of the other dialects like to add to this? 
At a minimum, perhaps just adding sources filesto...

That would be neat!


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Nix package for Pharo flavor of opensmalltalk-vm

Luke Gorrie
 
Hi Ben & all,

I revised the nix packaging. I'll do a little testing and cleaning up and then send it upstream. Quick testing or feedback would be really welcome!

This (long, single line) command installs the VMs (cog, spur, spur64) behind a simple wrapper:

nix-env --option tarball-ttl 0 -f https://github.com/lukego/nixpkgs/archive/pharo6.tar.gz -iA pharo.vm

Then you can start an image like this:

pharo-vm ...

You can also install a few off-the-shelf images with dedicated startup scripts. Here's how to install all of them:

nix-env --option tarball-ttl 0 -f https://github.com/lukego/nixpkgs/archive/pharo6.tar.gz -iA pharo

Then you can start one of those images with these shell commands:

pharo-6.0
pharo64-6.0
pharo-5.0
moose-6.1
pharo-launcher-2017.02.28

There is a little automatic testing: Each image is started under Xvfb to see if it crashes.

I am having some trouble doing "Save as..." to persist these images. The image seems to suggest saving them in the root directory, rather than the current directory, and I haven't managed to actually save an image. Weird. Have to look a bit harder. Let me know if you see the problem!

btw: The images are maintained manually. Could be automated in the future with a program to generate that file. Or maybe packaging images is the wrong idea and the launcher is better. Time will tell.


On 18 April 2017 at 23:00, Luke Gorrie <[hidden email]> wrote:
On 18 April 2017 at 18:59, Ben Coman <[hidden email]> wrote:
This worked fine for me on 32bit Debian 8 Jessie, resulting in...

Cool! Thanks for testing that.

The reason you see some "Jan 1 1970" timestamps is that nix resets file modification times back to the epoch to make the build more reproducible.

Being able to try a compiler upgrade and then cleanly rollback is a great feature.  I gather that under the covers, so switching between two environments is just a quick link change? 

Yep.

You can also skip the symlinks and construct the software environment you want with a custom $PATH. Here is a command to start a sub-shell with exactly the build environment used for compiling the vm:
This will start a shell with $PATH including exactly the expected versions of bash, gcc, autoconf, libtool, etc, that the build would use. Good for trouble shooting.

The binaries that you build will also remember exactly which shared objects they are supposed to link with (e.g. which build of glibc):

$ ldd $(which ls)
        linux-gate.so.1 (0xf7767000)
        librt.so.1 => /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/librt.so.1 (0xf7759000)
        libacl.so.1 => /nix/store/v480rw5maz01wh5ysrma8mhl6qy1kvjd-acl-2.2.52/lib/libacl.so.1 (0xf774f000)
        libattr.so.1 => /nix/store/h6k7bwddlfzv1sbwxbnl795h0317wrwp-attr-2.4.47/lib/libattr.so.1 (0xf7749000)
        libpthread.so.0 => /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/libpthread.so.0 (0xf772b000)
        libc.so.6 => /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/libc.so.6 (0xf7573000)

        /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/ld-linux.so.2 (0xf7769000)


Now would any of the other dialects like to add to this? 
At a minimum, perhaps just adding sources filesto...

That would be neat!



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Nix package for Pharo flavor of opensmalltalk-vm

Ben Coman
 


On Thu, Apr 20, 2017 at 12:21 AM, Luke Gorrie <[hidden email]> wrote:
 
Hi Ben & all,

I revised the nix packaging. I'll do a little testing and cleaning up and then send it upstream. Quick testing or feedback would be really welcome!

This (long, single line) command installs the VMs (cog, spur, spur64) behind a simple wrapper:

nix-env --option tarball-ttl 0 -f https://github.com/lukego/nixpkgs/archive/pharo6.tar.gz -iA pharo.vm


After I did that and it spent a few minutes pulling in and compiling dependencies and compiling the VM,
I ran it again, and it completed immediately taking minimal action.  Thats cool! 

Then you can start an image like this:

pharo-vm ...


 
You can also install a few off-the-shelf images with dedicated startup scripts. Here's how to install all of them:

nix-env --option tarball-ttl 0 -f https://github.com/lukego/nixpkgs/archive/pharo6.tar.gz -iA pharo


When I ran that, it kicked off a few more minutes compiling.  Is that expected?
Also I got an error "error: detected 64-bit image but 64-bit VM is not available"
perhaps since I'm on 32-bit Debian (Jessie-8).  Is there some way to guard against such error?
So the following did commands seem to have not been installed.

 
Then you can start one of those images with these shell commands:

pharo-6.0
pharo64-6.0

Since Pharo 6 is not yet released, what are the implications of naming these pharo-6.dev 
 
pharo-5.0
moose-6.1
pharo-launcher-2017.02.28

There is a little automatic testing: Each image is started under Xvfb to see if it crashes.

I am having some trouble doing "Save as..." to persist these images. The image seems to suggest saving them in the root directory, rather than the current directory, and I haven't managed to actually save an image. Weird. Have to look a bit harder. Let me know if you see the problem!

Is it directory permissions on the image file, or directory containing the image file? 
 

btw: The images are maintained manually. Could be automated in the future with a program to generate that file. Or maybe packaging images is the wrong idea and the launcher is better. Time will tell.

I think at least packaging the bleeding edge (i.e. atm Pharo 6) is wrong, since it changes fast.
If Squeak adopts this, it may make more sense for them.  Squeak trunk works more with an incremental growth of an Image, with a chain of updates scripts (someone might provide more detail), whereas Pharo has more of a bootstrap process where each build is not a child of the previous build, which is where PharoLauncher is useful. 

Packing the other images is maybe also wrong.  How would you deal with user state being stored in the image when a new build download in available?

cheers -ben
 


On 18 April 2017 at 23:00, Luke Gorrie <[hidden email]> wrote:
On 18 April 2017 at 18:59, Ben Coman <[hidden email]> wrote:
This worked fine for me on 32bit Debian 8 Jessie, resulting in...

Cool! Thanks for testing that.

The reason you see some "Jan 1 1970" timestamps is that nix resets file modification times back to the epoch to make the build more reproducible.

Being able to try a compiler upgrade and then cleanly rollback is a great feature.  I gather that under the covers, so switching between two environments is just a quick link change? 

Yep.

You can also skip the symlinks and construct the software environment you want with a custom $PATH. Here is a command to start a sub-shell with exactly the build environment used for compiling the vm:
This will start a shell with $PATH including exactly the expected versions of bash, gcc, autoconf, libtool, etc, that the build would use. Good for trouble shooting.

The binaries that you build will also remember exactly which shared objects they are supposed to link with (e.g. which build of glibc):

$ ldd $(which ls)
        linux-gate.so.1 (0xf7767000)
        librt.so.1 => /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/librt.so.1 (0xf7759000)
        libacl.so.1 => /nix/store/v480rw5maz01wh5ysrma8mhl6qy1kvjd-acl-2.2.52/lib/libacl.so.1 (0xf774f000)
        libattr.so.1 => /nix/store/h6k7bwddlfzv1sbwxbnl795h0317wrwp-attr-2.4.47/lib/libattr.so.1 (0xf7749000)
        libpthread.so.0 => /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/libpthread.so.0 (0xf772b000)
        libc.so.6 => /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/libc.so.6 (0xf7573000)

        /nix/store/djclawhih7363vlvsbrgfi1fr8l3glix-glibc-2.25/lib/ld-linux.so.2 (0xf7769000)


Now would any of the other dialects like to add to this? 
At a minimum, perhaps just adding sources filesto...

That would be neat!





Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Nix package for Pharo flavor of opensmalltalk-vm

Luke Gorrie
 
Thanks Ben for testing this, it's really helpful!

On 20 April 2017 at 06:03, Ben Coman <[hidden email]> wrote:
After I did that and it spent a few minutes pulling in and compiling dependencies and compiling the VM,
I ran it again, and it completed immediately taking minimal action.  Thats cool! 

This is the nice benefit of referential transparency on the builds. Since the result is always the same, because the whole build environment is held constant, you can safely reuse previous builds instead of making new ones. Your /nix/store directory contains all the builds you have made. (You can run 'nix-collect-garbage' to free up space from old builds.)

Once the packages go upstream nobody will need to compile it locally. Their 'nix-env' command will use the sha256 to get a binary from cache.nixos.org (populated by upstream CI builds.)

But yeah the first time it sure takes a while because you are basically downloading a whole Linux distro. I neglected to mention that you could write 'nix-env -j 10' to run many build steps (including downloads) in parallel. That can really speed things up at the expense of obfuscating the output.

When I ran that, it kicked off a few more minutes compiling.  Is that expected?

Installing the first image will also install Xvfb (+ dependencies) in order to run a unit test. (Could be that I should run headless pharo instead but figured that exercising the X11 display driver would be worth the extra installation time.)

Also I got an error "error: detected 64-bit image but 64-bit VM is not available"
perhaps since I'm on 32-bit Debian (Jessie-8).  Is there some way to guard against such error?

Good catch!

This error is from installing a 64-bit Pharo image. The unit test will check that the image actually runs, and this failed due to missing the required VM.

The solution is that I need to exclude the 64-bit images on 32-bit machines. (I did already exclude the spur64 VM but I didn't think to exclude the images too.)

Just as a workaround you could install images individually by changing '-iA pharo' to e.g. '-iA pharo5' or '-iA moose61'.

Since Pharo 6 is not yet released, what are the implications of naming these pharo-6.dev 

On reflection I will split up my packaging work into "vm" and "image" components. I will send the vm upstream ASAP. I will leave the images on a development branch, perhaps indefinitely, because I am not sure that I am on the right track there at all.

So, here are more thoughts on image packaging, but with the disclaimer that I will probably not actively pursue this topic further at the moment:

I assert that Pharo 5 and Pharo 6 are both released, and are both fast-moving targets.

On http://pharo.org/download there is a prominent link to Pharo 6 as the "development version" ready for download. Anecdotally the Pharo community that I interact with (mailing list, twitter, slack chat, etc) seem more focused on Pharo 6 than on Pharo 5. Occasionally people express surprise when I tell them I am "still" using Pharo 5.

On http://files.pharo.org/image/50/ there is a new release of Pharo 5 every few weeks. This means that each time I have downloaded "Pharo 5" in the past I have probably gotten a different image and a different VM. So from my end-user perspective I have no idea what software I am installing regardless of whether I choose Pharo 5 or Pharo 6.

So as a nix user it would make sense for me to be able to download both Pharo 5 and Pharo 6 and for the nix package definitions to determine exactly which software I am running (as opposed to getting an invisible update from Pharo upstream every few days/weeks potentially leading to "but it worked fine on my other machine..." sorts of problems that are exactly what can nix protect me from.)

Packing the other images is maybe also wrong.  How would you deal with user state being stored in the image when a new build download in available?

The ad-hoc idea that I ran with is this:

Running a script like pharo-5.0 (or moose-6.1, etc) will make a temporary copy of the image in the current directory. The changes file will be read-write (requirement) but the image file will be read-only. The model is "you saved it, you own it" i.e. if you want the image to survive after exit you need to "Save image as..." and once you do that then you are responsible for it.


I am not sure why "Save image as..." is misbehaving for me. I have tested every combination of read/write permissions on the working directory, the image, the changes file, etc, but it was always the same. I am sure that I can work this out via the Smalltalk debugger -- but at the moment it seems to make more sense to focus on the VM than on the images. (I want to wrap this up.)

The downside to skipping the images is that I don't maintain feature parity with Damien's original nix packaging. He packaged the launcher image and then that was the gateway to everything else. This solution doesn't feel very solid to me, since now in 2017 I don't think the launcher is accounting for all the different VM requirements properly, but I am not willing to pick up maintenance of that image so that is life.

Cheers,
-Luke


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Nix package for Pharo flavor of opensmalltalk-vm

Luke Gorrie
 
On 20 April 2017 at 09:59, Luke Gorrie <[hidden email]> wrote:
On reflection I will split up my packaging work into "vm" and "image" components. I will send the vm upstream ASAP. I will leave the images on a development branch, perhaps indefinitely, because I am not sure that I am on the right track there at all.

On further reflection it is possible that I am on the wrong track with the VM packaging too.

Pharo upstream are distributing releases as binary vm+image pairs with some third party libraries included. The most faithful way to package this for nix would be to take these binaries and make them work (using patchelf to fix up the shared library paths.) This is how people package other binary software for nix e.g. Skype.

The advantage of this approach would be to faithfully reproduce the software as it is released from the Pharo project. This way nix users would expect the software to work equally well as on Ubuntu or Fedora, say. This would also ease support because Pharo upstream would never be bothered by users having problems due to my packaging e.g. using the latest VM with an older image release if that is not supposed to be supported.

The disadvantage is that this does not suit my own use case. I want to develop a Pharo application, deploy it with nix, and support it myself. The binary releases are too opaque for my taste and I don't want to work with them. In this context I see a net benefit to building everything from sources and tracking all the dependencies explicitly on the nix level. So I would continue to use this source-based packaging myself, and I would not be interested in spending the time to package up the binaries for other people.

So - I need to rethink whether I am the right person to maintain the generic pharo packages for nix. (Could be that I should instead treat this as an opensmalltalk-vm packaging that is "use at own risk" with pharo images.)


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Nix package for Pharo flavor of opensmalltalk-vm

Ben Coman
In reply to this post by Luke Gorrie
 
Moved thread back to pharo-dev for image-side nix packaging discussion.

On Thu, Apr 20, 2017 at 3:59 PM, Luke Gorrie <[hidden email]> wrote:

>
> On 20 April 2017 at 06:03, Ben Coman <[hidden email]> wrote:
>
>> On Thu, Apr 20, 2017 at 12:21 AM, Luke Gorrie <[hidden email]> wrote:
>>>
>>> Then you can start one of those images with these shell commands:
>>>
>>> pharo-6.0
>>> pharo64-6.0
>>> pharo-5.0
>>> moose-6.1
>>> pharo-launcher-2017.02.28
>>
>> Since Pharo 6 is not yet released, what are the implications of naming those
>> pharo-6.dev & pharo64-6.dev [minor edit]
>
> On reflection I will split up my packaging work into "vm" and "image" components. I will send the vm upstream ASAP. I will leave the images on a development branch, perhaps indefinitely, because I am not sure that I am on the right track there at all.
That sounds like a good approach.

> So, here are more thoughts on image packaging, but with the disclaimer that I will probably not actively pursue this topic further at the moment:
>
> I assert that Pharo 5 and Pharo 6 are both released,

The "release" event will create the PharoV60.sources file,
null out the .changes file (it gets incorporated into PharoV60.sources)
and Pharo 7 development on will commence.
None of those things has happened yet.
Last I read, this was due in May, similar timeframe to Pharo 5 release
last year.

> and are both fast-moving targets.

The attached chart shows the dev-image build rate.
There is a clear change in Pharo 5 build rate after it was released on 12-May.
What happens is that during Pharo-6-development, a critical bug was
fixed and deemed worthy to backport to Pharo 5.  That doesn't put
released-Pharo-5 and in-development-Pharo-6 on equal footing.

Perhaps for packaging for distributions we need a modified process
for post-release-updates, such that the released image remains
constant and upon startup notifies that there are updates available to
install.

>
> On http://pharo.org/download there is a prominent link to Pharo 6 as the "development version" ready for download. Anecdotally the Pharo community that I interact with (mailing list, twitter, slack chat, etc) seem more focused on Pharo 6 than on Pharo 5.

The greater focus on Pharo 6 is because it is "in development"
and you are talking to pharo-developers.


> Occasionally people express surprise when I tell them I am "still" using Pharo 5.

On the one hand, that might be considered overzealous recommending
alpha/beta software for your primary platform.  On the other hand,
in practice many find the bleeding edge consistently very stable.
(opinion depends on the exact critical nature the application)

The thing is, if you hit a bug in the in-development version, it can
get fixed quite quickly.  More Pharo-developers work on the HEAD,
so its self interest to get things working smoothly. Policy is that *all*
fixes must *first* be done in the in-development-version.  Only critical
fixes get backported to the released-version.  So if you hit a bug in
a released-version, you might not see the fix before the next release.

So there is a lot of benefit following the tip of development.

>
> On http://files.pharo.org/image/50/ there is a new release of Pharo 5 every few weeks. This means that each time I have downloaded "Pharo 5" in the past I have probably gotten a different image and a different VM. So from my end-user perspective I have no idea what software I am installing regardless of whether I choose Pharo 5 or Pharo 6.

I consider this moving target for a released version (i.e. Pharo 5) a
problem. Personally I'd like to see post-release versionning like
5.0.x,
but other seem to disagree, or maybe its just resourcing.

Pharo 6 is "in-development" so different rules apply.
Personally I never use the "latest" download.
I manually choose the latest build using PharoLauncher (couldn't live
without it)


>
> So as a nix user it would make sense for me to be able to download both Pharo 5 and Pharo 6 and for the nix package definitions to determine exactly which software I am running (as opposed to getting an invisible update from Pharo upstream every few days/weeks potentially leading to "but it worked fine on my other machine..." sorts of problems that are exactly what can nix protect me from.)
>

I agree with you wrt to a released-Pharo-5, but not in-development-Pharo-6.


>> Packing the other images is maybe also wrong.  How would you deal with user state being stored in the image when a new build download in available?
>
>
> The ad-hoc idea that I ran with is this:
>
> Running a script like pharo-5.0 (or moose-6.1, etc) will make a temporary copy of the image in the current directory. The changes file will be read-write (requirement) but the image file will be read-only. The model is "you saved it, you own it" i.e. if you want the image to survive after exit you need to "Save image as..." and once you do that then you are responsible for it.

I hardly ever use "Save image as...", just "Save" and I think that might
confuse existing users if they *need* to do it.  Probably workable would be
having the "Save image as..." dialog appear the first that "Save" is chosen.

>
> Code: script to run an image[1] and script to start the right vm[2].
>
[1] https://github.com/lukego/nixpkgs/blob/pharo6/pkgs/development/pharo/images/build-image.nix#L27-L33
[2] https://github.com/lukego/nixpkgs/blob/pharo6/pkgs/development/pharo/wrapper/pharo-vm.sh


> I am not sure why "Save image as..." is misbehaving for me. I have tested every combination of read/write permissions on the working directory, the image, the changes file, etc, but it was always the same. I am sure that I can work this out via the Smalltalk debugger -- but at the moment it seems to make more sense to focus on the VM than on the images. (I want to wrap this up.)
>
> The downside to skipping the images is that I don't maintain feature parity with Damien's original nix packaging. He packaged the launcher image and then that was the gateway to everything else. This solution doesn't feel very solid to me, since now in 2017 I don't think the launcher is accounting for all the different VM requirements properly, but I am not willing to pick up maintenance of that image so that is life.

Skipping Images is different to skipping Launcher.  The Launcher
provides a *static* vm+image that is used to manage Images to solve
"movign target" problem you are hitting.  Launcher could perhaps do
with some further extension to distinguish 64-bit Images, since that
requirement has only arisen in the past year, and also perhaps be able
to manage multiple VMs, instead of just the pre-Spur & Spur VMs
configured in its settings.  But otherwise Launcher works fairly well.

If feasible, I'd suggest that you separately package the VMs, and then
have a package for PharoLauncher that depends on that package.

cheers -ben

>
> Cheers,
> -Luke
>
>
>

PharoBuildRate.png (59K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Nix package for Pharo flavor of opensmalltalk-vm

K K Subbu
In reply to this post by Luke Gorrie
 
On Thursday 20 April 2017 04:13 PM, Luke Gorrie wrote:
> Pharo upstream are distributing releases as binary vm+image pairs with
> some third party libraries included. The most faithful way to package
> this for nix would be to take these binaries and make them work (using
> patchelf to fix up the shared library paths.) This is how people package
> other binary software for nix e.g. Skype.

The vm+image is like chicken+egg ;-). Separated across time, they have a
recursive dependency between them. VM Maker app run on an existing image
(Pharo 5 or Squeak 5) to generate a new VM (e.g. Spur) and that in turn
is used to evolve the next image release (e.g. Squeak 6, Pharo 6). VM is
machine-specific code while Image an object graph on disk.

This is analogous to linux+rootfs except that linux can handle multiple
filesystems while our VM can only deal with one format at this time. In
Linux, filesystem modules translates objects on disk to object graph
(tree) in memory and the rest of the kernel refer to these objects
through a VFS namespace, so they are oblivious to the physical layout on
disk. This indirection is nicely brought out in :

https://www.dmst.aueb.gr/dds/pubs/inbook/beautiful_code/html/Spi07g.html

Linux itself is split into the kernel+initramfs, but there is no such
equivalent in our VM. We could introduce indirection by building a
bootstrap image into the VM itself (perhaps as a special plugin) and
have the vm load it initially. This image can then probe user-given
image files, match it with the right loader plugins and then switch to it.

Regards .. Subbu
"All problems in computer science can be solved by another level of
indirection" .. David Wheeler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Nix package for Pharo flavor of opensmalltalk-vm

Eliot Miranda-2
 
Hi Subbu,

> On Apr 23, 2017, at 6:25 AM, K K Subbu <[hidden email]> wrote:
>
>> On Thursday 20 April 2017 04:13 PM, Luke Gorrie wrote:
>> Pharo upstream are distributing releases as binary vm+image pairs with
>> some third party libraries included. The most faithful way to package
>> this for nix would be to take these binaries and make them work (using
>> patchelf to fix up the shared library paths.) This is how people package
>> other binary software for nix e.g. Skype.
>
> The vm+image is like chicken+egg ;-). Separated across time, they have a recursive dependency between them. VM Maker app run on an existing image (Pharo 5 or Squeak 5) to generate a new VM (e.g. Spur) and that in turn is used to evolve the next image release (e.g. Squeak 6, Pharo 6). VM is machine-specific code while Image an object graph on disk.
>
> This is analogous to linux+rootfs except that linux can handle multiple filesystems while our VM can only deal with one format at this time.

I think you're missing one, possibly two things more, and that is the object model, and the execution model.  Since 2008 I have been changing these to be able to increase performance and functionality.

The execution model concerns things like
- the representation of classes (their format word, their methodDictionary layout)
- the layout of a CompiledMethod header word (its first slot, objectAt: 1), which determines the max number of literals, argument count, temporary count, and range of available primitives
- how blocks work (whether they are non-reentrant as in the blue book, or fully reentrant closures
- the bytecode set, which determines things like the max number of instance variables in an instance, the max number of literals in a method, the max jump displacement
- how blocks and methods are related, whether block bytecode is embedded in the home method's bytecode or in an independent CompiledBlock object


The object model is the representation of objects in the heap, and concerns things like
- what are the layouts and different kinds of object headers, which determine things like the identityHash range, whether the system can support ephemerons, whether there are 8, 16, 32 and 64-bit non-pointer collections
- what the tag organization is, which determines whether there are immediate characters or immediate floats
- whether the heap can be grown by segments or not

The first major change I introduced was the block model to introduce fully reentrant closures.  Apart from being preferrable in and of themselves they also, when married with avoid representation for accessing closed over variables (see the bytecode for inject:into: and the creation of the indirection vector to hold nextValue, fully explained on my blog), enables the VM to efficiently and transparently map contexts to stack frames in the VM.  This is the StackInterpreter and Cog V3 VMs first released around 2010.

These systems share the object representation of the older Squeak 3.9 and earlier versions, but won't run older images because they depend on the older blue book non-reentrant block model.  

Note that it is a non-trivial change, affects my not just the VM, the bytecode set and the compiler, but the debugger, object (de)serialization, and some client code that depended on old block semantics.

Maintaining backward compatibility here would have hobbled the VM and severely limited the potential performance increase from context-to-stack mapping that gave us the Cog VM which is around 7x faster than the Squeak 3.9 interpreter.  So the break with backward compatibility was done consciously and with good reason.



The other change I introduced was Spur, which provides a very different object representation that
- simplifies the object header to one or two 64-but words
- increases the identityHash width
- introduces immediate characters and in 64-bits immediate floats for the most used 1/8 of the double precision range
- allows the heap to grow in segments
- provides a full 64-bit implementation (with 61-bit SmallIntegers) that shares the object header format between 32 and 64 bits
- allows the VM to implement much more Instantiations code in machine code because it is simpler

Maintaining backward compatibility here is simply not possible.  The bootstrap is ling and complex.  It had to be done as a new release.  Spur is around 2x the speed of the Cog V3 VM (some things are unchanged, some things, like widestring access are much faster).  

I took the opportunity to make an execution model change, moving the primitive field from the method header into the bytecode, to increase the number of literals to 32k. This also needs a bytecode change to enable 32k literaks and that evolution is taking place now with the Sista/Scorch development which will provide adaptive optimization via speculative inlining.

Note that while the conversion between 32-bit and 64-bit Spur images is automatic it is slow; minutes not seconds.  Not something one would want to inflict on people on startup, especially not as the first experience of using the system, hence separate 32-bit and 64-bit images.

So the break with backward compatibility, and the split between 32 and 64 bit images was done consciously and with good reason.



Now once the dust has settled the VM attempts to provide backward compatibility with any image that has the right execution model and object model.  One should always be able to use a Spur VM with any older Spur image, and any CogV3 CM with any older 4.x image.  Newer VMs may introduce new functionality that new images may depend on, but older images should continue to work.

To simplify this conceptually we boil this down to
- the newest vm for a major release is  backwards compatible with older images for that last major release

> In Linux, filesystem modules translates objects on disk to object graph (tree) in memory and the rest of the kernel refer to these objects through a VFS namespace, so they are oblivious to the physical layout on disk. This indirection is nicely brought out in :
>
> https://www.dmst.aueb.gr/dds/pubs/inbook/beautiful_code/html/Spi07g.html
>
> Linux itself is split into the kernel+initramfs, but there is no such equivalent in our VM. We could introduce indirection by building a bootstrap image into the VM itself (perhaps as a special plugin) and have the vm load it initially. This image can then probe user-given image files, match it with the right loader plugins and then switch to it.

But the information that differentiates between images is encoded in the image header version number and that can be accessed via od or hexdump et al and so can be done with a shell script.

I hope you can see that the differences between systems are more than between file system formats; it's not just a case of having the right driver.  Because the system is self defining there is much more of a circular dependency between vm and image.

I'm sorry that the complexity is inconvenient.  I do hope that things will settle down over the next few years after Sista/scorch is released.

>
> Regards .. Subbu
> "All problems in computer science can be solved by another level of indirection" .. David Wheeler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Nix package for Pharo flavor of opensmalltalk-vm

K K Subbu
 
On Sunday 23 April 2017 10:52 PM, Eliot Miranda wrote:

> Maintaining backward compatibility here would have hobbled the VM and
> severely limited the potential performance increase from
> context-to-stack mapping that gave us the Cog VM which is around 7x
> faster than the Squeak 3.9 interpreter.  So the break with backward
> compatibility was done consciously and with good reason.

I am very excited about the changes in stack/cog/sista and spur. My
point was not about backward compatibility but about introducing one
more level of indirection between vm and image to help package managers
deal with the emergence of multiple VMs. What we do today with launcher
scripts in terms of extracting paths and probing image files to pick the
right VM to run can be done by a primordial VM with a small built-in
image (say stkernel) to bootstrap a larger and modern VM. This stkernel
VM (like GRUB loader), will have the specific purpose of locating
configurations of larger VMs and then launching them. stkernel should be
easy to port, package and maintain on platforms since it has a specific
purpose. Applications can run on any of the second level VMs. The
two-level startup should be transparent to uses of squeak/cog/sista
users on 32-bit/64-bit hosts.

> I hope you can see that the differences between systems are more than
> between file system formats; it's not just a case of having the right
> driver.  Because the system is self defining there is much more of a
> circular dependency between vm and image.

I understand the model. VM creates Image and Image generates VM. If
proceed backwards, then we will arrive at single primordial VM+image
from which the split emerged. Today, we don't have to go that far. Just
have three - a stkernel with built-in image, then a collection of VMs
and Images on disk.

> I'm sorry that the complexity is inconvenient.  I do hope that things
> will settle down over the next few years after Sista/scorch is
> released.
No need to apologize. You have taken up arduous task of building next
generation VMs. I realize that complexity is part of this process. My
reference to Linux was only for concepts. Linux is a much larger project
with thousands of people working on it. We are a tiny team, so our
progress will be much drawn out. My suggestions were only within the
context of helping package managers.

Regards .. Subbu
Loading...