Smalltalk › Squeak › Squeak VM

what about image reading?

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

16 messages Options

Tobias Pape

what about image reading?

Dear all,

I'm just getting grips on the squeak image format and
I am trying to make sense of the "512 bytes padding" possibility.
In a previous discussion, the method
Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset
was mentioned that can read the image format using the given offset.
However, neither in the Smalltalk code nor in the generated C code I was able
to find any uses of this.

SqueakVMMaker/src ✔ grep -iR readImageFromFileHeapSizeStartingAt *
vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset);
vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {

Is this by design? Did I miss something?

So Long,
-Tobias

Douglas Brebner

Re: what about image reading?

On 13/01/2011 16:26, Tobias Pape wrote:

>
> Dear all,
>
> I'm just getting grips on the squeak image format and
> I am trying to make sense of the "512 bytes padding" possibility.
> In a previous discussion, the method
> Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset
> was mentioned that can read the image format using the given offset.
> However, neither in the Smalltalk code nor in the generated C code I was able
> to find any uses of this.
>
> SqueakVMMaker/src ✔ grep -iR readImageFromFileHeapSizeStartingAt *
> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset);
> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
>
> Is this by design? Did I miss something?
>
> So Long,
> -Tobias

Isn't this so that on unix systems you can make the image executable and
put something like "#!/path/to/squeak/vm" at the start of the image so
you can run it directly?

David T. Lewis

Re: what about image reading?

In reply to this post by Tobias Pape

On Thu, Jan 13, 2011 at 05:26:34PM +0100, Tobias Pape wrote:

>
> Dear all,
>
> I'm just getting grips on the squeak image format and
> I am trying to make sense of the "512 bytes padding" possibility.
> In a previous discussion, the method
> Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset
> was mentioned that can read the image format using the given offset.
> However, neither in the Smalltalk code nor in the generated C code I was able
> to find any uses of this.
>
> SqueakVMMaker/src ??? grep -iR readImageFromFileHeapSizeStartingAt *
> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset);
> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
>
> Is this by design? Did I miss something?
>
> So Long,
> -Tobias

The purpose of this feature is to support "shebang" processing:
http://en.wikipedia.org/wiki/Shebang_(Unix)

The idea is to offset the start of the image by 512 bytes into
the image file, and put the hash-bang line at the beginning
of the file. The hash-bang line invokes the VM, and the VM
reads the file containing the image at offset 512. The VM
first tries to read the image starting at offset 0, which
fails because it sees the hash-bang instead of the image
format word. It then skips ahead 512 bytes, tries again, and
presto the image starts up.

The end result is that an image file can be turned into an
executable file that runs itself.

Dave

Tobias Pape

Re: what about image reading?

Am 2011-01-13 um 17:52 schrieb David T. Lewis:

>
> On Thu, Jan 13, 2011 at 05:26:34PM +0100, Tobias Pape wrote:
>> […]
>> In a previous discussion, the method
>> Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset
>> was mentioned that can read the image format using the given offset.
>> However, neither in the Smalltalk code nor in the generated C code I was able
>> to find any uses of this.
>>
>> SqueakVMMaker/src ??? grep -iR readImageFromFileHeapSizeStartingAt *
>> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset);
>> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
>>
>> Is this by design? Did I miss something?
>> […]
>
> The purpose of this feature is to support "shebang" processing:
> http://en.wikipedia.org/wiki/Shebang_(Unix)
>
> The idea is to offset the start of the image by 512 bytes into
> the image file, and put the hash-bang line at the beginning
> of the file. The hash-bang line invokes the VM, and the VM
> reads the file containing the image at offset 512. The VM
> first tries to read the image starting at offset 0, which
> fails because it sees the hash-bang instead of the image
> format word. It then skips ahead 512 bytes, tries again, and
> presto the image starts up.
>
> The end result is that an image file can be turned into an
> executable file that runs itself.

The _purpose_ of the 512byte skip was clear to me.
I was just puzzled that there is no reference to the indicated method whatsoever.
Can you give me a hint, where that is used?

So Long,
-Tobias

David T. Lewis

Re: what about image reading?

On Thu, Jan 13, 2011 at 06:20:28PM +0100, Tobias Pape wrote:

>
> Am 2011-01-13 um 17:52 schrieb David T. Lewis:
> >
> > On Thu, Jan 13, 2011 at 05:26:34PM +0100, Tobias Pape wrote:
> >> [?]
> >> In a previous discussion, the method
> >> Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset
> >> was mentioned that can read the image format using the given offset.
> >> However, neither in the Smalltalk code nor in the generated C code I was able
> >> to find any uses of this.
> >>
> >> SqueakVMMaker/src ??? grep -iR readImageFromFileHeapSizeStartingAt *
> >> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset);
> >> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
> >>
> >> Is this by design? Did I miss something?
> >> [?]
> >
> > The purpose of this feature is to support "shebang" processing:
> > http://en.wikipedia.org/wiki/Shebang_(Unix)
> >
> > The idea is to offset the start of the image by 512 bytes into
> > the image file, and put the hash-bang line at the beginning
> > of the file. The hash-bang line invokes the VM, and the VM
> > reads the file containing the image at offset 512. The VM
> > first tries to read the image starting at offset 0, which
> > fails because it sees the hash-bang instead of the image
> > format word. It then skips ahead 512 bytes, tries again, and
> > presto the image starts up.
> >
> > The end result is that an image file can be turned into an
> > executable file that runs itself.
>
>
> The _purpose_ of the 512byte skip was clear to me.
> I was just puzzled that there is no reference to the indicated method whatsoever.
> Can you give me a hint, where that is used?

I'm away from Squeak at the moment, but look in class Interpreter
in package VMMaker. I think the method is #readImageFromFile:HeapSize:StartingAt:
or similar. The colons get removed from the method/function name as
part of the translation to C.

Dave

Tobias Pape

Re: what about image reading?

Am 2011-01-13 um 19:46 schrieb David T. Lewis:

>
> On Thu, Jan 13, 2011 at 06:20:28PM +0100, Tobias Pape wrote:
>>
>> Am 2011-01-13 um 17:52 schrieb David T. Lewis:
>>>
>>> On Thu, Jan 13, 2011 at 05:26:34PM +0100, Tobias Pape wrote:
>>>> [?]
>>>> In a previous discussion, the method
>>>> Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset
>>>> was mentioned that can read the image format using the given offset.
>>>> However, neither in the Smalltalk code nor in the generated C code I was able
>>>> to find any uses of this.
>>>>
>>>> SqueakVMMaker/src ??? grep -iR readImageFromFileHeapSizeStartingAt *
>>>> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset);
>>>> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
>>>>

[…]
>>
>> The _purpose_ of the 512byte skip was clear to me.
>> I was just puzzled that there is no reference to the indicated method whatsoever.
>> Can you give me a hint, where that is used?
>
> I'm away from Squeak at the moment, but look in class Interpreter
> in package VMMaker. I think the method is #readImageFromFile:HeapSize:StartingAt:
> or similar. The colons get removed from the method/function name as
> part of the translation to C.

Yes, that is true. I've already found that, if you refer to my original post.
My point is, that
NEITHER the smalltalk method
Interpreter>>#readImageFromFile:HeapSize:StartingAt:
NOR the c function

is called at all:
No senders for the Smalltalk method, no reference when
searching all methods source code for readImageFrom, aside the actual implementation
and a mention in some ‘important methods’ collection.
For the C-function, refer to my grep inside the generated source tree. No call at all.

So, why are they existent, in the first place?

So Long
-Tobias

Michael Haupt-3

Re: what about image reading?

Hi,

Am 13.01.2011 um 20:57 schrieb Tobias Pape <[hidden email]>:
> For the C-function, refer to my grep inside the generated source tree. No call at all.

just guessing: is there perhaps a call outside generated code, i.e., in common code? Anywhere in the whole bunch of sources?

Best,

Michael

Eliot Miranda-2

Re: what about image reading?

In reply to this post by Tobias Pape

On Thu, Jan 13, 2011 at 11:57 AM, Tobias Pape <[hidden email]> wrote:

Am 2011-01-13 um 19:46 schrieb David T. Lewis:

>
> On Thu, Jan 13, 2011 at 06:20:28PM +0100, Tobias Pape wrote:
>>
>> Am 2011-01-13 um 17:52 schrieb David T. Lewis:
>>>
>>> On Thu, Jan 13, 2011 at 05:26:34PM +0100, Tobias Pape wrote:
>>>> [?]
>>>> In a previous discussion, the method
>>>> Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset
>>>> was mentioned that can read the image format using the given offset.
>>>> However, neither in the Smalltalk code nor in the generated C code I was able
>>>> to find any uses of this.
>>>>
>>>> SqueakVMMaker/src ??? grep -iR readImageFromFileHeapSizeStartingAt *
>>>> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset);
>>>> vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
>>>>

[…]

>>
>> The _purpose_ of the 512byte skip was clear to me.
>> I was just puzzled that there is no reference to the indicated method whatsoever.
>> Can you give me a hint, where that is used?
>
> I'm away from Squeak at the moment, but look in class Interpreter
> in package VMMaker. I think the method is #readImageFromFile:HeapSize:StartingAt:
> or similar. The colons get removed from the method/function name as
> part of the translation to C.

Yes, that is true. I've already found that, if you refer to my original post.
My point is, that
NEITHER the smalltalk method
Interpreter>>#readImageFromFile:HeapSize:StartingAt:
NOR the c function

is called at all:
No senders for the Smalltalk method, no reference when
searching all methods source code for readImageFrom, aside the actual implementation
and a mention in some ‘important methods’ collection.
For the C-function, refer to my grep inside the generated source tree. No call at all.

So, why are they existent, in the first place?

You're not searching all the C code ;) readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. See platforms/{Mac OS,iOS,win32,unix}/vm

McStalker.oscogvm$ grep readImageFromFile platforms/*/vm/*

platforms/Cross/vm/sq.h:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset);

platforms/Cross/vm/sq.h:#define readImageFromFileHeapSize(f, s) readImageFromFileHeapSizeStartingAt(f,s,0)

platforms/Mac OS/vm/sqMacMain.c: readImageFromFileHeapSizeStartingAt(f, sqGetAvailableMemory(), 0);

platforms/Mac OS/vm/sqMacNSPluginUILogic.c: readImageFromFileHeapSizeStartingAt(f, squeakHeapMBytes, 0);

platforms/unix/vm/sqUnixMain.c: readImageFromFileHeapSizeStartingAt(f, extraMemory, 0);

platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, 0);

platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, sqImageFilePosition(imageFile));

So Long
-Tobias

Michael Haupt-3

Re: what about image reading?

Hi,

Am 13.01.2011 um 22:45 schrieb Eliot Miranda <[hidden email]>:
> You're not searching all the C code ;) readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. ...

:-)

OK. Say, *is* there comprehensive VM documentation other than in the form of code and some human minds?

Michael

Eliot Miranda-2

Re: what about image reading?

On Thu, Jan 13, 2011 at 1:53 PM, Michael Haupt <[hidden email]> wrote:

Hi,

Am 13.01.2011 um 22:45 schrieb Eliot Miranda <[hidden email]>:
> You're not searching all the C code ;) readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. ...

:-)

OK. Say, *is* there comprehensive VM documentation other than in the form of code and some human minds?

No. The doc that we have form Tim, which is good, is out of date and doesn't describe Cog at all. I have yet to document the Cog design on my web site. There's a lot lacking in doc, but hopefully the minds are all present and correct :)

Michael

David T. Lewis

Re: what about image reading?

On Thu, Jan 13, 2011 at 02:04:11PM -0800, Eliot Miranda wrote:

>
> On Thu, Jan 13, 2011 at 1:53 PM, Michael Haupt <[hidden email]> wrote:
>
> >
> > Hi,
> >
> > Am 13.01.2011 um 22:45 schrieb Eliot Miranda <[hidden email]>:
> > > You're not searching all the C code ;)
> > readImageFromFile:HeapSize:StartingAt: is called from the C main routine,
> > and the C main routine is not generated. ...
> >
> > :-)
> >
> > OK. Say, *is* there comprehensive VM documentation other than in the form
> > of code and some human minds?
> >
>
> No. The doc that we have form Tim, which is good, is out of date and
> doesn't describe Cog at all. I have yet to document the Cog design on my
> web site. There's a lot lacking in doc, but hopefully the minds are all
> present and correct :)
>

See also John's platform API documentation here:
http://isqueak.org/PlatformVMAPI

Dave

Tobias Pape

Re: what about image reading?

In reply to this post by Eliot Miranda-2

Am 2011-01-13 um 22:45 schrieb Eliot Miranda:
[…]
>
> You're not searching all the C code ;) readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. See platforms/{Mac OS,iOS,win32,unix}/vm
>

Ok. I got that. Apologies, it was my fault, obviously.

> McStalker.oscogvm$ grep readImageFromFile platforms/*/vm/*
> platforms/Cross/vm/sq.h:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset);
> platforms/Cross/vm/sq.h:#define readImageFromFileHeapSize(f, s) readImageFromFileHeapSizeStartingAt(f,s,0)

ok. the definitions.

> platforms/Mac OS/vm/sqMacMain.c: readImageFromFileHeapSizeStartingAt(f, sqGetAvailableMemory(), 0);
> platforms/Mac OS/vm/sqMacNSPluginUILogic.c: readImageFromFileHeapSizeStartingAt(f, squeakHeapMBytes, 0);

So, the Carbon vm is _not_ dealing with the 512 offset?

> platforms/unix/vm/sqUnixMain.c: readImageFromFileHeapSizeStartingAt(f, extraMemory, 0);

and, most curiously, the unix-VM isn't either?

> platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, 0);
> platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, sqImageFilePosition(imageFile));
>

Ah, the Windows vm does.

I did one grep of my own and found:

vm has 512b-offset-support
Carbon no
Risc no
Cocoa no
unix no
Win32 yes

So, while only the win32-vm is _able_ to read the offset images, wouldn't it make sense
to drop this feature altogether?

So Long,
-Tobias

David T. Lewis

Re: what about image reading?

On Fri, Jan 14, 2011 at 12:10:23PM +0100, Tobias Pape wrote:

>
> Am 2011-01-13 um 22:45 schrieb Eliot Miranda:
> [?]
> >
> > You're not searching all the C code ;) readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. See platforms/{Mac OS,iOS,win32,unix}/vm
> >
>
> Ok. I got that. Apologies, it was my fault, obviously.
>
> > McStalker.oscogvm$ grep readImageFromFile platforms/*/vm/*
> > platforms/Cross/vm/sq.h:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset);
> > platforms/Cross/vm/sq.h:#define readImageFromFileHeapSize(f, s) readImageFromFileHeapSizeStartingAt(f,s,0)
>
> ok. the definitions.
>
> > platforms/Mac OS/vm/sqMacMain.c: readImageFromFileHeapSizeStartingAt(f, sqGetAvailableMemory(), 0);
> > platforms/Mac OS/vm/sqMacNSPluginUILogic.c: readImageFromFileHeapSizeStartingAt(f, squeakHeapMBytes, 0);
>
> So, the Carbon vm is _not_ dealing with the 512 offset?
>
> > platforms/unix/vm/sqUnixMain.c: readImageFromFileHeapSizeStartingAt(f, extraMemory, 0);
>
> and, most curiously, the unix-VM isn't either?
>
> > platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, 0);
> > platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, sqImageFilePosition(imageFile));
> >
>
> Ah, the Windows vm does.
>
> I did one grep of my own and found:
>
> vm has 512b-offset-support
> Carbon no
> Risc no
> Cocoa no
> unix no
> Win32 yes
>
> So, while only the win32-vm is _able_ to read the offset images, wouldn't it make sense
> to drop this feature altogether?

You're misreading the code. This feature is not platform specific, and is
implemented in the Smalltalk code. This is explained in the method comment
of #readImageFromFile:HeapSize:StartingAt: and you'll find the implementation
in #checkImageVersionFrom:startingAt: which is sent by
#readImageFromFile:HeapSize:StartingAt:

So main() calls #readImageFromFile:HeapSize:StartingAt: to load the
image file, and the 512 byte skip is implemented there.

You can also see how this works by loading an image file in the
InterpreterSimulator. See class comment for InterpreterSimulator,
and step through it in a debugger to see your image file being
loaded.

Dave

stephane ducasse-2

Re: what about image reading?

In reply to this post by Michael Haupt-3

We are planning to write some new chapters on vms so if people want to share effort they are welcomed.
Our work will be hosted on pbe on github.

Stef

On Jan 13, 2011, at 10:53 PM, Michael Haupt wrote:

>
> Hi,
>
> Am 13.01.2011 um 22:45 schrieb Eliot Miranda <[hidden email]>:
>> You're not searching all the C code ;) readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. ...
>
> :-)
>
> OK. Say, *is* there comprehensive VM documentation other than in the form of code and some human minds?
>
> Michael

Tobias Pape

Re: what about image reading?

In reply to this post by David T. Lewis

Am 2011-01-14 um 15:08 schrieb David T. Lewis:
> On Fri, Jan 14, 2011 at 12:10:23PM +0100, Tobias Pape wrote:
[…]

>>>
>>
>> Ah, the Windows vm does.
>>
>> I did one grep of my own and found:
>>
>> vm has 512b-offset-support
>> Carbon no
>> Risc no
>> Cocoa no
>> unix no
>> Win32 yes
>>
>> So, while only the win32-vm is _able_ to read the offset images, wouldn't it make sense
>> to drop this feature altogether?
>
> You're misreading the code. This feature is not platform specific, and is
> implemented in the Smalltalk code. This is explained in the method comment
> of #readImageFromFile:HeapSize:StartingAt: and you'll find the implementation
> in #checkImageVersionFrom:startingAt: which is sent by
> #readImageFromFile:HeapSize:StartingAt:
>
> So main() calls #readImageFromFile:HeapSize:StartingAt: to load the
> image file, and the 512 byte skip is implemented there.

Ah, I see.
Why is the windows vm doing it manually, then?

>
> You can also see how this works by loading an image file in the
> InterpreterSimulator. See class comment for InterpreterSimulator,
> and step through it in a debugger to see your image file being
> loaded.

Nice hint. Thank you.

I did that on a hand-crafted 512-byte offset file.
However, the InterpreterSimulator does not support 512byte offsetting:

InterpreterSimulator>>#openOn:extraMemory:
" .. snip .."

["begin ensure block..."
f := FileStream readOnlyFileNamed: fileName.
imageName := f fullName.
f binary.
" offset handling should be here
--> " version := self nextLongFrom: f. "current version: 16r1966 (=6502)"
versionToRun := version bitAnd: -2. "permit loading images with platform float ordering"
hasPlatformFloatOrdering := version ~= (version bitAnd: -2). "is low order bit set?"
(self readableFormat: versionToRun) "permit loading images with platform float ordering"
ifTrue: [swapBytes := false]
ifFalse: [(versionToRun := self byteSwapped: version) = self imageFormatVersion
ifTrue: [swapBytes := true]
ifFalse: [
" or at least here -->" self error: 'incompatible image format']].

I tested the same Image on the Carbon 4.2.5beta1U VM, but it does not load
either:
/Applications/Squeak\ 4.2.5beta1U.app/Contents/MacOS/Squeak\ VM\ Opt v39_le_32bit_off_clos_cog.SqueakLight.image
This interpreter (vers. 0) cannot read image file (vers. 1966022947).
Press CR to quit...

Note that the 1966022947 = 0x752F2123 => 0x23 0x21 0x2F 0x75
corresponds to the beginning of the file: '#!/u'

Is anyone interested in the offset-image to test it her/himself?

So Long,
-Tobias

Andreas.Raab

Re: what about image reading?

On 1/20/2011 6:37 PM, Tobias Pape wrote:
>> So main() calls #readImageFromFile:HeapSize:StartingAt: to load the
>> image file, and the 512 byte skip is implemented there.
>
> Ah, I see.
> Why is the windows vm doing it manually, then?

Probably because nobody told me that that wasn't necessary :-)
I'll happily remove it.

Cheers,
- Andreas