Packaging 64-bit and 32-bit linux VMs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Packaging 64-bit and 32-bit linux VMs

Eliot Miranda-2
 
Hi All,

    at today's board meeting we discussed the issue of packaging 32-bit and 64-bit linux VMs in the same directory hierarchy.  

Right now a hierarchy looks like

Sisyphus.image$ ls -R ../products/sqcogspurlinux
bin lib squeak

../products/sqcogspurlinux/bin:
squeak

../products/sqcogspurlinux/lib:
squeak

../products/sqcogspurlinux/lib/squeak:
5.0-201705031421

../products/sqcogspurlinux/lib/squeak/5.0-201705031421:
B3DAcceleratorPlugin UnicodePlugin vm-display-null
ImmX11Plugin UnixOSProcessPlugin vm-sound-ALSA
LocalePlugin XDisplayControlPlugin vm-sound-OSS
SqueakFFIPrims squeak vm-sound-null
SqueakSSL vm-display-X11 vm-sound-pulse
UUIDPlugin vm-display-fbdev

What we need is either the 32-bit VMs to use lib32 or the 64-bit VMs to use lib64, and then change the two squeak scripts to parse the command line to locate the image and then check the image format to launch the right VM.

I'd do it but I have other things on my plate right now. If anyone feels like volunteering I'm attaching some scripts.  sviver uses hexdump to print the image version number, and so something like

IMVER=`hexdump -n 4 -e '"%u\n"' "$IMAGE"`

can be used to set IMVER to 68021 for the 64-bit Spur image, 6251 for the 32-bit Spur image, etc.

spurcfvm launches the relevant VM depending on its name (spurcfvm is hard linked to spurcdvm spurcavm spur64cfvm et al to select a different VM).  It parses the command line to locate the image.

And I'm including svihdr that prints info from the image header on both 64-bit and 32-bit Spur images in case anyone finds it useful.

_,,,^..^,,,_
best, Eliot

spurcfvm (4K) Download Attachment
sviver (298 bytes) Download Attachment
svihdr (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Packaging 64-bit and 32-bit linux VMs

David T. Lewis
 
I am thinking in terms of a Unix (Linux) installation as I write this, but here
is how it should work:

- For a 64 bit Spur VM, the VM is invoked by /usr/bin/spur64 or /usr/local/bin/spur64.

- There must be no name conflicts between /usr/local/bin and /usr/bin, and "spur64"
above means "run the 64 bit Spur VM for a 64 bit image" regardless of whether it is
in /usr/bin or /usr/local/bin or ~/bin.

- For a 32 bit Spur VM, the VM is invoked by /usr/bin/spur or /usr/local/bin/spur.

- For a 32 bit Cog (V3) VM, the VM is invoked by /usr/bin/cog or /usr/local/bin/cog.

- For all classic interpreter 32 and 64 bit VMs (V3), the VM is invoked by /usr/bin/squeak
or /usr/local/bin/squeak.

- For any flavor of VM that cannot share external plugins with other flavors,
it is the responsibility of the installation procedure to ensure that the
binaries are installed in a distictly named directory under /usr/lib/squeak
or /usr/local/lib/squeak (or /usr/local/lib/opensmalltalkvm, it does not
matter as long as it is a distictly named location).

- In particular, it must be possible for the same version of OpenSmalltalkVM for
Spur64, Spur, and Cog to coexist in distinctly named folders on a single machine.
As an example, the library directories for the 64 and 32 bit V3 interpreters have
names like 4.16.3-3748 for the 32 bit image and 4.16.3-3748_64bit for the 64 bit
image. A similar naming convention is needed to distinguish Spur 64, Spur 32,
and Cog 32 for the same version of the VM on a single machine installation.

- A run script to handle all of the above should be called /usr/bin/run, such
that the Unix command to run an image is "run image". The choice of the name
"run" results in no naming conflicts on Unix systems .

I am attaching a copy of the /usr/[local/]bin/run that I use on my own PC as
an ad-hoc solution to all of the above. Its restrictions are that it requires
an interpreter VM to be installed (not an issue for me but not good in the
general case), and it requires that I manually install the various Spur/Cog
VMs, then rename their run scripts as /usr/bin/spur64, /usr/bin/spur, and
/usr/bin/cog as described above.

With the above naming conventions in place, any Squeak/Pharo/Cuis image named
"myimage.image" can be run with the command "run myimage".

Dave



On Wed, May 16, 2018 at 11:25:00AM -0700, Eliot Miranda wrote:

>  
> Hi All,
>
>     at today's board meeting we discussed the issue of packaging 32-bit and
> 64-bit linux VMs in the same directory hierarchy.
>
> Right now a hierarchy looks like
>
> Sisyphus.image$ ls -R ../products/sqcogspurlinux
> bin lib squeak
>
> ../products/sqcogspurlinux/bin:
> squeak
>
> ../products/sqcogspurlinux/lib:
> squeak
>
> ../products/sqcogspurlinux/lib/squeak:
> 5.0-201705031421
>
> ../products/sqcogspurlinux/lib/squeak/5.0-201705031421:
> B3DAcceleratorPlugin UnicodePlugin vm-display-null
> ImmX11Plugin UnixOSProcessPlugin vm-sound-ALSA
> LocalePlugin XDisplayControlPlugin vm-sound-OSS
> SqueakFFIPrims squeak vm-sound-null
> SqueakSSL vm-display-X11 vm-sound-pulse
> UUIDPlugin vm-display-fbdev
>
> What we need is either the 32-bit VMs to use lib32 or the 64-bit VMs to use
> lib64, and then change the two squeak scripts to parse the command line to
> locate the image and then check the image format to launch the right VM.
>
> I'd do it but I have other things on my plate right now. If anyone feels
> like volunteering I'm attaching some scripts.  sviver uses hexdump to print
> the image version number, and so something like
>
> IMVER=`hexdump -n 4 -e '"%u\n"' "$IMAGE"`
>
> can be used to set IMVER to 68021 for the 64-bit Spur image, 6251 for the
> 32-bit Spur image, etc.
>
> spurcfvm launches the relevant VM depending on its name (spurcfvm is hard
> linked to spurcdvm spurcavm spur64cfvm et al to select a different VM).  It
> parses the command line to locate the image.
>
> And I'm including svihdr that prints info from the image header on both
> 64-bit and 32-bit Spur images in case anyone finds it useful.
>
> _,,,^..^,,,_
> best, Eliot



Reply | Threaded
Open this post in threaded view
|

Re: Packaging 64-bit and 32-bit linux VMs

Chris Muller-3
 
> - For a 64 bit Spur VM, the VM is invoked by /usr/bin/spur64 or /usr/local/bin/spur64.
> ...
> - For a 32 bit Spur VM, the VM is invoked by /usr/bin/spur or /usr/local/bin/spur.

If 64-bit spur is the future, I hope you'll consider putting the bit
designation on the old one.  e.g., "spur" and "spur32".
Reply | Threaded
Open this post in threaded view
|

Re: Packaging 64-bit and 32-bit linux VMs

David T. Lewis
 
On Wed, May 16, 2018 at 09:48:01PM -0500, Chris Muller wrote:
>  
> > - For a 64 bit Spur VM, the VM is invoked by /usr/bin/spur64 or /usr/local/bin/spur64.
> > ...
> > - For a 32 bit Spur VM, the VM is invoked by /usr/bin/spur or /usr/local/bin/spur.
>
> If 64-bit spur is the future, I hope you'll consider putting the bit
> designation on the old one.  e.g., "spur" and "spur32".

Either way is fine, just as long as the names are distinct.

Dave
Reply | Threaded
Open this post in threaded view
|

Re: Packaging 64-bit and 32-bit linux VMs

K K Subbu
In reply to this post by David T. Lewis
 
On Thursday 17 May 2018 08:07 AM, David T. Lewis wrote:
>
> I am thinking in terms of a Unix (Linux) installation as I write this, but here
> is how it should work:
>
> - For a 64 bit Spur VM, the VM is invoked by /usr/bin/spur64 or /usr/local/bin/spur64.
>
> - There must be no name conflicts between /usr/local/bin and /usr/bin, and "spur64"
> above means "run the 64 bit Spur VM for a 64 bit image" regardless of whether it is
> in /usr/bin or /usr/local/bin or ~/bin.

I have a different take on this matter. bittiness is not just for the
main program but also on all its plugins, dependent libraries and data
files. Changing the name will also result in a cascade of changes in
launch wrappers and other user scripts.

Instead, what if we retain existing paths for 32-bit and just add
directories like bin64 or lib64 for the new 64-bit code? This only
involves changing $PATH and $LD_LIB_PATH.

Regards .. Subbu
Reply | Threaded
Open this post in threaded view
|

Re: Packaging 64-bit and 32-bit linux VMs

K K Subbu
In reply to this post by Eliot Miranda-2
 
On Wednesday 16 May 2018 11:55 PM, Eliot Miranda wrote:
> IMVER=`hexdump -n 4 -e '"%u\n"' "$IMAGE"`

od may be a better choice as it can handle endianness too. E.g.

$ od -A n -t u4 -N 4 --endian=little squeak.image
  6521
$ od -A n -t u4 -N 4 --endian=big squeak.image
  2031681536

The magic numbers may also be offset 512 bytes into the file:

$ od -j 512 .......

Regards .. Subbu
Reply | Threaded
Open this post in threaded view
|

Re: Packaging 64-bit and 32-bit linux VMs

Phil B
In reply to this post by Chris Muller-3
 
On Wed, May 16, 2018, 10:48 PM Chris Muller <[hidden email]> wrote:
 
> - For a 64 bit Spur VM, the VM is invoked by /usr/bin/spur64 or /usr/local/bin/spur64.
> ...
> - For a 32 bit Spur VM, the VM is invoked by /usr/bin/spur or /usr/local/bin/spur.

If 64-bit spur is the future, I hope you'll consider putting the bit
designation on the old one.  e.g., "spur" and "spur32".

+1... please be forward looking: if there must be a designation, put it on the version that is/will be fading out of common usage.
Reply | Threaded
Open this post in threaded view
|

Re: Packaging 64-bit and 32-bit linux VMs

Eliot Miranda-2
In reply to this post by K K Subbu
 
Hi Subbu,

On Thu, May 17, 2018 at 4:44 AM, K K Subbu <[hidden email]> wrote:

On Wednesday 16 May 2018 11:55 PM, Eliot Miranda wrote:
IMVER=`hexdump -n 4 -e '"%u\n"' "$IMAGE"`

od may be a better choice as it can handle endianness too. E.g.

$ od -A n -t u4 -N 4 --endian=little squeak.image
 6521
$ od -A n -t u4 -N 4 --endian=big squeak.image
 2031681536


Doesn't work on Mac OS X:

Sisyphus.oscogvm$ od -A n -t u4 -N 4 --endian=little ~/Cog/SpurWork64.image
od: illegal option -- -
usage: od [-aBbcDdeFfHhIiLlOosvXx] [-A base] [-j skip] [-N length] [-t type]
          [[+]offset[.][Bb]] [file ...]

And when I used od on Mac OS X I found there is no option to eliminate the byte offset/data address printing.  Whatever we use has to work across as many Unix dialects as possible.
 
The magic numbers may also be offset 512 bytes into the file:

$ od -j 512 .......

Regards .. Subbu



--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Packaging 64-bit and 32-bit linux VMs

Eliot Miranda-2
In reply to this post by K K Subbu
 


On Thu, May 17, 2018 at 4:07 AM, K K Subbu <[hidden email]> wrote:

On Thursday 17 May 2018 08:07 AM, David T. Lewis wrote:

I am thinking in terms of a Unix (Linux) installation as I write this, but here
is how it should work:

- For a 64 bit Spur VM, the VM is invoked by /usr/bin/spur64 or /usr/local/bin/spur64.

- There must be no name conflicts between /usr/local/bin and /usr/bin, and "spur64"
above means "run the 64 bit Spur VM for a 64 bit image" regardless of whether it is
in /usr/bin or /usr/local/bin or ~/bin.

I have a different take on this matter. bittiness is not just for the main program but also on all its plugins, dependent libraries and data files. Changing the name will also result in a cascade of changes in launch wrappers and other user scripts.

Instead, what if we retain existing paths for 32-bit and just add directories like bin64 or lib64 for the new 64-bit code? This only involves changing $PATH and $LD_LIB_PATH.

+1.  That's my instinct too.
 
Regards .. Subbu

_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Packaging 64-bit and 32-bit linux VMs

K K Subbu
In reply to this post by Eliot Miranda-2
 
On Friday 25 May 2018 01:49 AM, Eliot Miranda wrote:

> Hi Subbu,
>
> On Thu, May 17, 2018 at 4:44 AM, K K Subbu <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>     On Wednesday 16 May 2018 11:55 PM, Eliot Miranda wrote:
>
>         IMVER=`hexdump -n 4 -e '"%u\n"' "$IMAGE"`
>
>
>     od may be a better choice as it can handle endianness too. E.g.
>
>     $ od -A n -t u4 -N 4 --endian=little squeak.image
>       6521
>     $ od -A n -t u4 -N 4 --endian=big squeak.image
>       2031681536
>
> Doesn't work on Mac OS X
You're right! I didn't think of OS X since the subject was about Linux
VMs. But I won't be surprised if it breaks, because core CLI utils on OS
X have not kept pace with time.

The problem of a portable way of binding VMs and images at runtime is a
serious one. Now that we are faced with a sparse matrix of VMs and image
file combinations, the problem will only get worse with time.

I see two options (not exclusive) towards a long-term solution:

  #1 generate mimetypes for VM app and image files at build time and use
Info.plist/xdg-utils to complete the binding at install time.

  #2 Use a tiny bootstrap image and write all runtime scripts in st
itself. The VM will always start with this bootstrap which will then use
st scripts to probe the given image and then bind it to the right VM.

My own preference would be to create mimetypes at build time rather than
postpone the problem to run time.

The second option sounds more complicated with two images, but there has
always been a demand for a tiny headless REPL image and many people have
created their own kernel. Bundling one with the VM will gives us a
reference shell and the same can be used for builds and customizing
runtimes.

Regards .. Subbu