magic decode for image files

 
Hi,

I have attached a pattern match file that can be used by the file(1)
command to decode Squeak/Cog/Spur image files.

$ file -m magic Pharo-50770.image Pharo-40627.image
Pharo-50770.image: Spur image 32b little-endian
Pharo-40627.image: Cog image 32b little-endian,closure,nativefloat

$ file -m magic Squeak3.8.1-6747full.image Squeak64-3.8g-6548.image
Squeak3.8.1-6747full.image: Squeak image 32b little-endian
Squeak64-3.8g-6548.image: Spur image 64b big-endian

The file may be appended to /etc/magic (system-wide) or $HOME/.magic
(for one user). It will help package maintainers pick the right vm for
an Image.

Ideally, this file should be generated by VMMaker under
platforms/Cross/.., so patterns can be kept in sync with ImageFormat.

I am rejoining this list after a break that took a lot longer than I
expected :-(. Glad to be back, though it looks like I have a lot of
catchup to do.

Regards .. Subbu

 
Very nice :-)

See also the ckformat program, which is automatically generated as an executeable
from package ImageFormat in the VMMaker repository. It is used for selecting
the appropriate VM for a given image file in the traditional interpreter VM
in the /usr/local/bin/squeak script.

Dave

On Mon, Apr 10, 2017 at 05:52:49PM +0530, K K Subbu wrote:

 
On Monday 10 April 2017 06:52 PM, David T. Lewis wrote:
>
> Very nice :-)
>
> See also the ckformat program, which is automatically generated as an executeable
> from package ImageFormat in the VMMaker repository. It is used for selecting
> the appropriate VM for a given image file in the traditional interpreter VM
> in the /usr/local/bin/squeak script.

I saw the ckformat program and couldn't see why it was needed. Perhaps I
am missing something here.

If we generate magic and binfmt_misc registrations, then
package-managers can launch the right vm for any image file. It also
integrates well with other utilities like update-binfmts.

Regards .. Subbu

 
On Mon, Apr 10, 2017 at 07:31:47PM +0530, K K Subbu wrote: It answers the image format number on standard output, for use in shell scripts.

The attached "run" file is the script that I use on my own PC to select Spur 64,
Spur 32 bit, Cog 32 bit, interpreter VM 64 bit, or interpreter VM 32 bit.

The start script for interpreter VM uses this approach also:
http://squeakvm.org/cgi-bin/viewvc.cgi/squeak/trunk/platforms/unix/cmake/squeak.in?revision=3609&view=markup

>
> If we generate magic and binfmt_misc registrations, then
> package-managers can launch the right vm for any image file. It also
> integrates well with other utilities like update-binfmts.
>
> Regards .. Subbu

That approach would work also.

Your magic decode logic is nice, do you mind if I steal it and add it to ImageFormat?

Thanks,
Dave

 
On Tuesday 11 April 2017 05:03 AM, David T. Lewis wrote:
I use the commonly available od(1) to extract magic numbers:

$ alias sqmagic='od -An -N4 -t x1'
$ sqmagic SpurVMMaker.image
  79 19 00 00
$

The case statement duplicates binfmt_misc registrations. Each image
format requires a just a single line to register with binfmt_misc. E.g.

# :name:type:offset:pattern:mask:interpreter:flags

$ echo ':spur32le:M::\x79\x19\x00\x00::/opt/squeak/spur32:' | sudo tee
/proc/sys/fs/binfmt_misc/register

and removed with
$ echo -1 | sudo tee /proc/sys/fs/binfmt_misc/spur32le

The run script is fragile (e.g. it will break if $0 is symlink). If the
disambiguation can be done outside, then it can be simplified without
losing flexibility.

Other interpreters/emulators like python, lua, qemu work this way.

Please do. It is very nice of you to offer me to help merge the code.
Thanks. I am still in the catchup mode :-(. The magic file has to be
generated just like any other .c file. It can be compiled into a binary
file with "file -C ..." into magic.mgc bin file but that would be an
overkill for such a small file.

Regards .. Subbu

 
On Tue, Apr 11, 2017 at 12:47:57PM +0530, K K Subbu wrote:
Thanks, I added it to ImageFormat. One thing needed changing for 64-bit
formats on little-endian, where the format number is written to the second
four bytes (offset 4). The following seems to work now for all formats:

# Smalltalk image file formats
0 lelong 6502 Smalltalk image V3 32-bit no closure no native float (6502)
0 belong 6502 Smalltalk image V3 32-bit no closure no native float (6502)
0 lelong 6504 Smalltalk image V3 32-bit closure no native float (6504)
0 belong 6504 Smalltalk image V3 32-bit closure no native float (6504)
0 lelong 68000 Smalltalk image V3 64-bit no closure no native float (68000)
4 belong 68000 Smalltalk image V3 64-bit no closure no native float (68000)
0 lelong 68002 Smalltalk image V3 64-bit closure no native float (68002)
4 belong 68002 Smalltalk image V3 64-bit closure no native float (68002)
0 lelong 6505 Smalltalk image V3 32-bit closure native float (6505)
0 belong 6505 Smalltalk image V3 32-bit closure native float (6505)
0 lelong 68003 Smalltalk image V3 64-bit closure native float (68003)
4 belong 68003 Smalltalk image V3 64-bit closure native float (68003)
0 lelong 6521 Smalltalk image Spur 32-bit closure native float (6521)
0 belong 6521 Smalltalk image Spur 32-bit closure native float (6521)
0 lelong 68019 Smalltalk image Spur 64-bit closure native float (68019)
4 belong 68019 Smalltalk image Spur 64-bit closure native float (68019)
0 lelong 68021 Smalltalk image Spur 64-bit closure native float (68021)
4 belong 68021 Smalltalk image Spur 64-bit closure native float (68021)

Dave

 
On Tuesday 11 April 2017 05:33 PM, David T. Lewis wrote:
> Thanks, I added it to ImageFormat. One thing needed changing for 64-bit
> formats on little-endian, where the format number is written to the second
> four bytes (offset 4).

Is that in addition to the magic numbers in the first 4b? Then we need
to modify the match pattern.

Thank you very much. Each of these lines can be followed by:

!:mime application/squeak-image

or appropriate mime type for cog and spur. They can be used to generate
*.desktop files for launching the right VMs.

Which vm and base image do you use for your work?

I am on Ubuntu 16.04 (64-bit). Its squeak-vm package is very old
(4.10.2). I also have the Squeak 5 (5.0-3397) but that is also old I
guess. I tried buildspurtrunkvmmakerimage.sh from opensmalltalk-vm git
but it takes ages throws up errors in ClassBuilder while building
SpurVMMaker.image :-(. The image got built but I am not sure if I got
all the stuff in.

Regards .. Subbu

 
On Tue, Apr 11, 2017 at 07:34:41PM +0530, K K Subbu wrote:
>
> On Tuesday 11 April 2017 05:33 PM, David T. Lewis wrote:
> >Thanks, I added it to ImageFormat. One thing needed changing for 64-bit
> >formats on little-endian, where the format number is written to the second
> >four bytes (offset 4).
>
> Is that in addition to the magic numbers in the first 4b? Then we need
> to modify the match pattern.

Sorry, I said that backwards. It is the big endian versions that offset
the format number by four bytes. That was an artifact of the way that image
file headers were written in the original 64-bit image (circa 2005) and
is true of 64-bit V3 images since then. The original 64-bit distribution
was done on a big-endian machine, and those files are still on line at
http://squeakvm.org/squeak64/dist3/.

The 64-bit Spur image changes this, so the image format number is in the
first 4 bytes as you would expect. That would be true on a big-endian
machine also (although I do not know of any such implementation).

For Spur, the 64-bit magic decode should probably be this:

0 lelong 68021 Smalltalk image Spur 64-bit closure native float (68021)
0 belong 68021 Smalltalk image Spur 64-bit closure native float (68021)


>
> Thank you very much. Each of these lines can be followed by:
>
> !:mime application/squeak-image
>
> or appropriate mime type for cog and spur. They can be used to generate
> *.desktop files for launching the right VMs.
>
> Which vm and base image do you use for your work?

My working image in 64-bit Spur, Squeak trunk.

I use various Spur, Cog, and interpreter VMs, and various images. I regularly
use interpreter VMs in addition to Spur.

On my machine /usr/local/bin/squeak is always the standard interpreter VM
start script, which runs 32-bit and 64-bit V3 images. The Cog/Spur VMs are
/usr/local/bin/cog, /usr/local/bin/spur, and /usr/local/bin/spur64. The
/usr/local/bin/run script selects from these based on image format.

>
> I am on Ubuntu 16.04 (64-bit). Its squeak-vm package is very old
> (4.10.2).

I always compile the interprer VM locally, which produces a 64-bit VM
for the 32-bit V3 image (no 32-bit libraries). Google "squeak interpreter
vm recipe" for the instructions (http://wiki.squeak.org/squeak/6354).

> I also have the Squeak 5 (5.0-3397) but that is also old I
> guess. I tried buildspurtrunkvmmakerimage.sh from opensmalltalk-vm git
> but it takes ages throws up errors in ClassBuilder while building
> SpurVMMaker.image :-(. The image got built but I am not sure if I got
> all the stuff in.
>
> Regards .. Subbu

I sometimes compile Spur VMs locally, but it is a bit of a trick to get
all the libraries set up, and the old autotools build system is no fun.
I usually use VMs from squeak.org or the latest from the bintray site
instead.

Dave