oops to zero

 
I am attempting to zero all ooP pointers before writng an image to disk.

I have added the following code to Interpreter-#writeImage:

writeImageFile: imageBytes

...
    self adjustAllOopsBy: (0 - memory).
    self writeImageFileIO: imageBytes.
    self adjustAllOopsBy: memory.
...

I managed to build a unix version of the vm with this change but it did
not work as hoped.
Starting an image saved with this vm crashes giving a bus error.

any ideas?

I am also unable to find any documentation on generating the .app
version of the unix vm for Mac OS X.

thanks in advance

Keith


_

To build this into squeak:

Loaded 3.9.8-12 stuff from http://www.squeakvm.org/unix/release/

I have the current trunk obtained by:

svn co http://squeakvm.org/svn/squeak/trunk squeak

I have

http://www.squeakvm.org/unix/release/unix-3.9-12.image.gz
http://www.squeakvm.org/unix/release/unix-3.9-12.changes.gz
http://www.squeakvm.org/unix/release/unix-3.9-12.vmm.config

I am on Mac OS X, using John 's OS X VM 3.8.18beta1U

The above config file is not visible by the VMaker Tool 'load' button,
when I save my own config, that one is.

Clicking the platform button:

UnixOSProcessPlugIn-#isResponsibleForThisPlatform
 
calls Smalltalk osVersion rather than SmalltalkImage current.

 

On Sep 29, 2007, at 21:56 , Keith Hodges wrote:

The base address is stored in the image, so before calling  
writeImageFileIO you would have to set memory to zero I guess.

- Bert -

 
Oops in

Interpreter>>writeImageFileIO:

you need to change the
self putLong: (self startOfMemory) toFile: f.

to I think
self putLong: 0 toFile: f.


On Sep 29, 2007, at 12:56 PM, Keith Hodges wrote:

> I managed to build a unix version of the vm with this change but it  
> did not work as hoped.
> Starting an image saved with this vm crashes giving a bus error.

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

 
Well if you're building a carbon os-x VM the build docs are in the  
readme in the mac os CVS tree

On Sep 29, 2007, at 12:56 PM, Keith Hodges wrote:

> I am also unable to find any documentation on generating the .app  
> version of the unix vm for Mac OS X.

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

 
I was so excited at the prospect of getting a certificate!
> Interpreter>>writeImageFileIO:
>
> you need to change the
> self putLong: (self startOfMemory) toFile: f.
>
> to I think
> self putLong: 0 toFile: f.
I had tried that:

but still no joy.

I have included my image writing method below

thanks for the support

Keith

----
 
writeImageFileIO: imageBytes

    | headerStart headerSize f bytesWritten sCWIfn okToWrite |
    self var: #f type: 'sqImageFile'.
    self var: #headerStart type: 'squeakFileOffsetType '.
    self var: #sCWIfn type: 'void *'.
       
    "kph mod - to support binary diffs"
   
    self adjustAllOopsBy: (0 - self startOfMemory).
   
    "If the security plugin can be loaded, use it to check for write
permission.
    If not, assume it's ok"
    sCWIfn := self ioLoadFunction: 'secCanWriteImage' From:
'SecurityPlugin'.
    sCWIfn ~= 0 ifTrue:[okToWrite := self cCode: '((sqInt
(*)(void))sCWIfn)()'.
        okToWrite ifFalse:[^self primitiveFail]].
   
    "local constants"
    headerStart := 0.
    headerSize := 64.  "header size in bytes; do not change!"

    f := self cCode: 'sqImageFileOpen(imageName, "wb")'.
    f = nil ifTrue: [
        "could not open the image file for writing"
        self success: false.
        ^ nil].

    headerStart := self cCode:
'sqImageFileStartLocation(f,imageName,headerSize+imageBytes)'.
    self cCode: '/* Note: on Unix systems one could put an exec command
here, padded to 512 bytes */'.
    "position file to start of header"
    self sqImageFile: f Seek: headerStart.

    self putLong: (self imageFormatVersion) toFile: f.
    self putLong: headerSize toFile: f.
    self putLong: imageBytes toFile: f.
    self putLong: 0 "(self startOfMemory) kph mod - for binary diffs"
toFile: f.
    self putLong: specialObjectsOop toFile: f.
    self putLong: lastHash toFile: f.
    self putLong: (self ioScreenSize) toFile: f.
    self putLong: fullScreenFlag toFile: f.
    self putLong: extraVMMemory toFile: f.
    1 to: 7 do: [:i | self putLong: 0 toFile: f].  "fill remaining
header words with zeros"
    successFlag ifFalse: [
        "file write or seek failure"
        self cCode: 'sqImageFileClose(f)'.
        ^ nil].

    "position file after the header"
    self sqImageFile: f Seek: headerStart + headerSize.

    "write the image data"
    bytesWritten := self cCode: 'sqImageFileWrite(pointerForOop(memory),
sizeof(unsigned char), imageBytes, f)'.
    self success: bytesWritten = imageBytes.
    self cCode: 'sqImageFileClose(f)'.

    self adjustAllOopsBy: (self startOfMemory).
   

 
perhaps my sources are not being regenerated properly.. ill check

Keith

 
Keith Hodges wrote:
>
> perhaps my sources are not being regenerated properly.. ill check
>
> Keith
That wasnt it.

The image carries on working after the save, indicating that the
swapping oop pointers is working fine.

It's loading in the saved image that causes the Bus Error.

Keith

 
Ok, I needed to adjust the saved value of specialObjectsOop.

it worked!

Now I need to ensure that the byte ordering is consistent

Keith

 

> Now I need to ensure that the byte ordering is consistent
>
> Keith
>
I am planning to build a minimal squeak vm for unix and windows which
can be added to the mercurial installation as an encoder for files of
type *.image (the unix vm build will be sufficient to do the job on Mac
OS X).

This is the best way that I can think of for supporting squeak
activities using vm's and images of every description. This way the
normalizing code doesn't need to be in every users vm.

Mercurial needs the encoder to be invokable from the commandline like
so, which should be doable.

squeak -normalize INFILE OUTFILE

In this case I dont even need the image to properly start up, so I dont
really need to unswizzle things having written the image.

I am not having much sucess with saving the image little-endian

any thoughts would be appreciated

Keith

---
appendix: The Story so far:

Interpreter-putLong: aWord toFile: aFile swap: swapFlag
    "swap byte order as requested"

    | objectsWritten w |
    self var: #aFile type: 'sqImageFile '.

    w := aWord.
    swapFlag ifTrue: [ w := self byteSwapped: w ].

    objectsWritten := self cCode: 'sqImageFileWrite(&w, sizeof(w), 1,
aFile)'.
    self success: objectsWritten = 1.


Interpreter-writeImageFileIO: imageBytes

    | headerStart headerSize f bytesWritten sCWIfn okToWrite swapToLittle |

    self var: #f type: 'sqImageFile'.
    self var: #headerStart type: 'squeakFileOffsetType '.
    self var: #sCWIfn type: 'void *'.
           
    "If the security plugin can be loaded, use it to check for write
permission.
    If not, assume it's ok"
    sCWIfn := self ioLoadFunction: 'secCanWriteImage' From:
'SecurityPlugin'.
    sCWIfn ~= 0 ifTrue:[okToWrite := self cCode: '((sqInt
(*)(void))sCWIfn)()'.
        okToWrite ifFalse:[^self primitiveFail]].

    "kph mod - to support binary diffs"  
    self adjustAllOopsBy: (0 - self startOfMemory).
    swapToLittle := self isBigEnder.
   
    "local constants"
    headerStart := 0.
    headerSize := 64.  "header size in bytes; do not change!"

    f := self cCode: 'sqImageFileOpen(imageName, "wb")'.
    f = nil ifTrue: [
        "could not open the image file for writing"
        self success: false.
        ^ nil].

    headerStart := self cCode:
'sqImageFileStartLocation(f,imageName,headerSize+imageBytes)'.
    self cCode: '/* Note: on Unix systems one could put an exec command
here, padded to 512 bytes */'.
    "position file to start of header"
    self sqImageFile: f Seek: headerStart.

    self putLong: (self imageFormatVersion) toFile: f swap: swapToLittle.
    self putLong: headerSize toFile: f swap: swapToLittle.
    self putLong: imageBytes toFile: f swap: swapToLittle.
    self putLong: 0 "(self startOfMemory) kph mod - for binary diffs"
toFile: f swap: swapToLittle.
    self putLong: (specialObjectsOop - self startOfMemory) toFile: f
swap: swapToLittle.
    self putLong: lastHash toFile: f swap: swapToLittle.
    self putLong: (self ioScreenSize) toFile: f swap: swapToLittle.
    self putLong: fullScreenFlag toFile: f swap: swapToLittle.
    self putLong: extraVMMemory toFile: f swap: swapToLittle.
    1 to: 7 do: [:i | self putLong: 0 toFile: f].  "fill remaining
header words with zeros"
    successFlag ifFalse: [
        "file write or seek failure"
        self cCode: 'sqImageFileClose(f)'.
        ^ nil].

    "position file after the header"
    self sqImageFile: f Seek: headerStart + headerSize.

    "write the image data"
    swapToLittle ifTrue: [ self reverseBytesInImage].
   
    bytesWritten := self cCode: 'sqImageFileWrite(pointerForOop(memory),
sizeof(unsigned char), imageBytes, f)'.
    self success: bytesWritten = imageBytes.
    self cCode: 'sqImageFileClose(f)'.

    swapToLittle ifTrue: [ self reverseBytesInImage].
    self adjustAllOopsBy: (self startOfMemory).
   

 


> Now I need to ensure that the byte ordering is consistent
>
> Keith
>
I am planning to build a minimal squeak vm for unix and windows which
can be added to the mercurial installation as an encoder for files of
type *.image (the unix vm build will be sufficient to do the job on Mac
OS X).

This is the best way that I can think of for supporting squeak
activities using vm's and images of every description. This way the
normalizing code doesn't need to be in every users vm.

Mercurial needs the encoder to be invokable from the commandline like
so, which should be doable.

squeak -normalize INFILE OUTFILE

In this case I dont even need the image to properly start up, so I dont
really need to unswizzle things having written the image.

I am not having much sucess with saving the image little-endian

any thoughts would be appreciated

Keith

appendix: The Story so far:

------------------------------------------

Interpreter-putLong: aWord toFile: aFile swap: swapFlag
        "sway byte order as requested"

        | objectsWritten w |
        self var: #aFile type: 'sqImageFile '.

        w := aWord.
        swapFlag ifTrue: [ w := self byteSwapped: aWord ].

        objectsWritten := self cCode: 'sqImageFileWrite(&w, sizeof(w), 1, aFile)'.
        self success: objectsWritten = 1.

------------------------------------------

Interpreter-writeImageFileIO: imageBytes

    | headerStart headerSize f bytesWritten sCWIfn okToWrite swapToLittle |

    self var: #f type: 'sqImageFile'.
    self var: #headerStart type: 'squeakFileOffsetType '.
    self var: #sCWIfn type: 'void *'.

    "If the security plugin can be loaded, use it to check for write
permission.
    If not, assume it's ok"
    sCWIfn := self ioLoadFunction: 'secCanWriteImage' From:
'SecurityPlugin'.
    sCWIfn ~= 0 ifTrue:[okToWrite := self cCode: '((sqInt
(*)(void))sCWIfn)()'.
        okToWrite ifFalse:[^self primitiveFail]].

    "kph mod - to support binary diffs"
    self adjustAllOopsBy: (0 - self startOfMemory).
    swapToLittle := self isBigEnder.

    "local constants"
    headerStart := 0.
    headerSize := 64.  "header size in bytes; do not change!"

    f := self cCode: 'sqImageFileOpen(imageName, "wb")'.
    f = nil ifTrue: [
        "could not open the image file for writing"
        self success: false.
        ^ nil].

    headerStart := self cCode:
'sqImageFileStartLocation(f,imageName,headerSize+imageBytes)'.
    self cCode: '/* Note: on Unix systems one could put an exec command
here, padded to 512 bytes */'.
    "position file to start of header"
    self sqImageFile: f Seek: headerStart.

    self putLong: (self imageFormatVersion) toFile: f swap: swapToLittle.
    self putLong: headerSize toFile: f swap: swapToLittle.
    self putLong: imageBytes toFile: f swap: swapToLittle.
    self putLong: 0 "(self startOfMemory) kph mod - for binary diffs"
toFile: f swap: swapToLittle.
    self putLong: (specialObjectsOop - self startOfMemory) toFile: f
swap: swapToLittle.
    self putLong: lastHash toFile: f swap: swapToLittle.
    self putLong: (self ioScreenSize) toFile: f swap: swapToLittle.
    self putLong: fullScreenFlag toFile: f swap: swapToLittle.
    self putLong: extraVMMemory toFile: f swap: swapToLittle.
    1 to: 7 do: [:i | self putLong: 0 toFile: f].  "fill remaining
header words with zeros"
    successFlag ifFalse: [
        "file write or seek failure"
        self cCode: 'sqImageFileClose(f)'.
        ^ nil].

    "position file after the header"
    self sqImageFile: f Seek: headerStart + headerSize.

    "write the image data"
    swapToLittle ifTrue: [ self reverseBytesInImage].

    bytesWritten := self cCode: 'sqImageFileWrite(pointerForOop(memory),
sizeof(unsigned char), imageBytes, f)'.
    self success: bytesWritten = imageBytes.
    self cCode: 'sqImageFileClose(f)'.

    swapToLittle ifTrue: [ self reverseBytesInImage].
    self adjustAllOopsBy: (self startOfMemory).

 
I am surprise to find that subclassing Interpreter to make my own
refinements does not work is this to be expected?

Keith

 
Having touched upon the hallowed ground of VMMaking, I have started to
desecrate the altar itself...

I have attempted to add limited but "useful-enough" inheritance to VMMaker.

Is there a repository to which I can upload any changes for
review/feedback/integration?

With my tweaks if you implement a subclass of Interpreter, and the class
method #inherits ^ true, then you can
implement an alternative interpreter by adding or overriding methods.
The same should be true for plugin's

[BUG] I discover that the value of the field: "Interpreter class name"
does not successfully save/load in the vmm.config files

I have attempted to test this but subclassing LargeIntegerPlugin twice
and generating both, and it seemed to work, though I have not fully
tested it.

As a trival example use case: The comment in
LargeIntegersPlugin-buildCodeGeneratorUpTo: someClass indicates that
there is no UI means to enable debugCode generation. Making
LargeIntegersPluginDebug a subclass of LargeIntegersPlugin could add the
single needed method. Both options would appear in the available plugins
enabling the user to select the debug/nondebug versions.

now.. back to normalizing!

Keith

 
The plan:

Mercurial installations have the following in their hgrc configuration file:

A)

[encode]
**.image :tempfile squeaknormalize -normalizeNoStart INFILE OUTFILE

B)
[encode]
**.image :tempfile squeaknormalize -headless -normalize INFILE OUTFILE
/path/to/script.st

Whenever mercurial commits a *.image file to the repository it will pass
it through this filter.
squeaknormalize being a modified vm.

Option A) reads, writes and quits the image without even starting the
interpreter
Option B) runs a script, which is then responsible for snapshotting and
quitting

The script could perform some simple image tidying such as flushing MC
caches etc.

When reading parameters, both options sneakily grab the first parameter
(INFILE) into "normalizingInputImageName", the normal document (OUTFILE)
is the next parameter and this becomes the current "imageName"

Typically mercurial uses temp files of the form /tmp/hg-filter-123
without an extension to support this I have had to patch out the
automatic appending of 'image' to the document parameter.

This scheme described above is implemented but not tested or debugged.

I still haven't succeeded in getting the image to save littleEndian on
my mac..

best regards

Keith

 
Having just watched a video about OLPC and seen how little storage they
have. Have any OLPC folks done this normalizing the image already?

Is the OLPC big or little endian?

I may be having a go at testing my latest this weekend, would anoyne be
willing to help out with the windows version since I lack tools/machine etc?

best regards

Keith

 
(a) The OLPC actually uses a compressed file system already, because  
of that Published Sophie Books for the OLPC use a zipped file with no  
compression. The zipped file will live in the journal at some point.

(b) Does setting oops values back to start of zero actually make it  
more compressable, versus say a start value of 0x00004000? Otherwise
the computational energy used to traverse the image is technically  
wasted.

(c) ittle/big endian?  OLPC is based on the geod aka intel  
instruction set.

http://wiki.laptop.org/go/Hardware_specification



On Oct 4, 2007, at 6:53 PM, Keith Hodges wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

 
John M McIntosh wrote:
>
> (a) The OLPC actually uses a compressed file system already, because
> of that Published Sophie Books for the OLPC use a zipped file with no
> compression. The zipped file will live in the journal at some point.
>
> (b) Does setting oops values back to start of zero actually make it
> more compressable, versus say a start value of 0x00004000?
We are not actually setting the oops back to zero, but back
relative-to-zero, They are just numbers, just different numbers. I would
not imagine that this has any effect upon the compressibility of any
single image. However diff algorithms working with two or more images
are more likely to see that much of a saved image is the same as the
last time it was saved. From the little I know it seems that the OLPC
file system (as well as mercurial binary storage) is designed with this
in mind.

Indeed you make a good point, if we can pick a good number that runtimes
can use without traversing the image then great. What number should it be?

Since OLPC applications have a virtual linux box to themselves can we
just pick a number that suits us/them?
> Otherwise
> the computational energy used to traverse the image is technically
> wasted.
>
> (c) ittle/big endian?  OLPC is based on the geod aka intel instruction
> set.
So it looks like little-endian is the favoured choice for the normalized
image.
> http://wiki.laptop.org/go/Hardware_specification
>
best regards

Keith
 

 

On Oct 5, 2007, at 5:05 , Keith Hodges wrote:

I don't think any OS (except for DOS maybe) would guarantee you a  
specific start address.

> Since OLPC applications have a virtual linux box to themselves can  
> we just pick a number that suits us/them?

I wouldn't think so ... but it is an interesting idea. I will ask the  
OLPC guys.

- Bert -

 
Bert Freudenberg schrieb: It does not need to guarantee a start address but just pick the same one
most of the time (for example, as long as the VM binary stays the same).
I think Squeak already does this. See below for another option.
>
>> Since OLPC applications have a virtual linux box to themselves can we
>> just pick a number that suits us/them?
>
> I wouldn't think so ... but it is an interesting idea. I will ask the
> OLPC guys.
It's not difficult for a VM to use mmap() on /dev/null (or was it
/dev/zero?) to allocate memory for the image to any given address under
linux (as long as the CPU/OS version supports memory at that address).
If a VM did this, it could have a fixed image start address even across
VM versions. Even cooler would be to mmap() the image file directly with
copy-on-write behavior for almost instantaneous startup.

Cheers,
Hans-Martin

 
On Fri, Oct 05, 2007 at 02:43:21PM +0200, Hans-Martin Mosner wrote: Actually the Unix VM already does use mmap() on /dev/zero to obtain the
base address, so it is easy to request a specific base address. Ian pointed
this out to me when I was doing the "32bit clean" patches, and that is
how I was able to test those changes.

The attached diff shows where to make the change in sqUnixMemory.c.

Note that mmap() will assign its own address if it does not agree with
the one that you requested. You will need to pick an address that works
on various versions of the kernel, and on both 32 bit and 64 bit systems
(or perhaps provide a list of addresses, and let the VM find one that
works on that particular platform).  The value 0x4000000 seems reasonable
on a 64 bit AMD Linux 2.6 system, but I did not check to see if it works
on a 32 bit system, so you may need to experiment a bit.

The OLPC platform has well defined hardware and OS, so it would be good
to pick a number that works on OLPC and make that the default.

Dave

 
On Oct 6, 2007, at 15:08 , David T. Lewis wrote:

Interesting - I also found that on Linux I can modify the executable  
to disable randomization using the -R flag of setarch:

http://www.linuxcommand.org/man_pages/setarch8.html

Have not tested this yet. And does the image loading logic actually  
check if a remap is necessary?

- Bert -