Nuking VM ImageSegment support (was Re: [squeak-dev] Daily Commit Log; System-bf.916)

On Thu, Sep 15, 2016 at 2:55 PM, <[hidden email]> wrote:

[snip]

http://lists.squeakfoundation.org/pipermail/packages/2016-September/068930.html

Name: System-bf.916
Ancestors: System-bf.915

Replace VM-level ImageSegment loading with a Smalltalk implementation for old (interpreter-era) projects.

Also removes support for writing segments.

This overrides the Spur support introduced in System-eem.758.

Hi Bert, Hi Tim,

On Thu, Sep 15, 2016 at 2:55 PM, <[hidden email]> wrote:

[snip]

http://lists.squeakfoundation.org/pipermail/packages/2016-September/068930.html

Name: System-bf.916
Ancestors: System-bf.915

Replace VM-level ImageSegment loading with a Smalltalk implementation for old (interpreter-era) projects.

Also removes support for writing segments.

This overrides the Spur support introduced in System-eem.758.

I am currently putting back the ImageSegment support for saving projects and loading native segments produced in this way.

  I''ve added subclasses of ImageSegment called NativeImageSegment and LegacyImageSegment.   I have a few questions.

1. I'm curious about the ImageSegment>>scanFrom: method:

!ImageSegment methodsFor: 'fileIn/Out' stamp: 'RAA 6/22/2000 17:49'!

scanFrom: aStream

    "Move source code from a fileIn to the changes file for classes in an ImageSegment.  Do not compile the methods.  They already came in via the image segment.  After the ImageSegment in the file, !!ImageSegment new!! captures control, and scanFrom: is called."

    | val chunk |

    [aStream atEnd] whileFalse:

        [aStream skipSeparators.

        val := (aStream peekFor: $!!)

            ifTrue: ["Move (aStream nextChunk), find the method or class

                        comment, and install the file location bytes"

                    (Compiler evaluate: aStream nextChunk logged: false)

                        scanFromNoCompile: aStream forSegment: self]

            ifFalse: [chunk := aStream nextChunk.

                    aStream checkForPreamble: chunk.

                    Compiler evaluate: chunk logged: true].

        aStream skipStyleChunk].

    "regular fileIn will close the file"

    ^ val! !

I don't see this in the new (legacy, System-bf.916) image segment loading code.  Do Etoys projects not include source code and hence its irrelevant? 

The question boils down to whether this should be a method on ImageSegment, and hence inherited by both NativeImageSegment and LegacyImageSegment, or just NativeImageSegment.  Also, I don't see how this method attaches sources to the methods brought in.  Is it in fact useless and hence that's why you've left it out?  If so, would it not be useful to add code that does update the source pointers in loaded methods correctly?

2. Should I pull in the Etoys extensions deleted in EToys-bf.234?  (ImageSegment>>(classOrganizersBeRoots:,rehashDictionaries:,rehashMethodDictionaries:)?  I'm guessing they'll be needed for native loading.

3. Same for the SMBase extension ImageSegment>>writeForExportOn:.  I guess this should come back in too.

4.  Where should I look for tests that may have been removed?

On Mon, Jul 3, 2017 at 11:43 PM, Eliot Miranda <[hidden email]> wrote:

Hi Bert, Hi Tim,

On Thu, Sep 15, 2016 at 2:55 PM, <[hidden email]> wrote:

[snip]

http://lists.squeakfoundation.org/pipermail/packages/2016-September/068930.html

Name: System-bf.916
Ancestors: System-bf.915

Replace VM-level ImageSegment loading with a Smalltalk implementation for old (interpreter-era) projects.

Also removes support for writing segments.

This overrides the Spur support introduced in System-eem.758.

I am currently putting back the ImageSegment support for saving projects and loading native segments produced in this way.

​Why?​ I thought we agreed that ImageSegments are not a great idea as a file format, for their lack of compatibility. Serialization speed isn't an issue anymore thanks to your fast jit.

What ImageSegments *are* good at is for splitting up an image into segments that do not have to be in memory at the same time - but in that case they would not be an independent entity but be considered part of one particular image. And for that we need only very few methods (that I removed too, see storeSegment).

  I''ve added subclasses of ImageSegment called NativeImageSegment and LegacyImageSegment.   I have a few questions.

1. I'm curious about the ImageSegment>>scanFrom: method:

!ImageSegment methodsFor: 'fileIn/Out' stamp: 'RAA 6/22/2000 17:49'!

scanFrom: aStream

    "Move source code from a fileIn to the changes file for classes in an ImageSegment.  Do not compile the methods.  They already came in via the image segment.  After the ImageSegment in the file, !!ImageSegment new!! captures control, and scanFrom: is called."

    | val chunk |

    [aStream atEnd] whileFalse:

        [aStream skipSeparators.

        val := (aStream peekFor: $!!)

            ifTrue: ["Move (aStream nextChunk), find the method or class

                        comment, and install the file location bytes"

                    (Compiler evaluate: aStream nextChunk logged: false)

                        scanFromNoCompile: aStream forSegment: self]

            ifFalse: [chunk := aStream nextChunk.

                    aStream checkForPreamble: chunk.

                    Compiler evaluate: chunk logged: true].

        aStream skipStyleChunk].

    "regular fileIn will close the file"

    ^ val! !

I don't see this in the new (legacy, System-bf.916) image segment loading code.  Do Etoys projects not include source code and hence its irrelevant? 

​They do, but the code path to load them does not go through this method.

The question boils down to whether this should be a method on ImageSegment, and hence inherited by both NativeImageSegment and LegacyImageSegment, or just NativeImageSegment.  Also, I don't see how this method attaches sources to the methods brought in.  Is it in fact useless and hence that's why you've left it out?  If so, would it not be useful to add code that does update the source pointers in loaded methods correctly?

The ImageSegment nowadays only stores uniclasses as objects. Other changes to the system are simply filed in, and not included in the segment. This method appears to be unused. I tried to trim everything not strictly need​ed for project loading

2. Should I pull in the Etoys extensions deleted in EToys-bf.234?  (ImageSegment>>(classOrganizersBeRoots:,rehashDictionaries:,rehashMethodDictionaries:)?  I'm guessing they'll be needed for native loading.

rehashDictionaries

​: is still there, not sure if we need the others.​

 

3. Same for the SMBase extension ImageSegment>>writeForExportOn:.  I guess this should come back in too.

​SqueakMap uses ImageSegments, yes. No idea if it's a good idea to write a new native segment.​

4.  Where should I look for tests that may have been removed?

​There are no tests, unfortunately ...

The way I was testing is taking a project written by Etoys (from squeakland.org) and dropping it into a trunk image. Before your changes this did work, more or less: If you revert to System-pre.956 and make allocateMethodContext: and allocateBlockContext: simply return nil, you can load old projects again. These contexts aren't needed anyways for Etoys projects (if we needed them I'd recreate by recompiling). 

I'm attaching a simple project that I just tested with this.