How does ImageSegmentLoader>>readObject work?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How does ImageSegmentLoader>>readObject work?

Hannes Hirzel
Hello

this is a follow up mail on loading a project file (*.pr) saved in
Squeak 3.10.2 [1] into Squeak 6.0a-#18000  [2]

The code breaks at

    ImageSegmentLoader>>readObject [3]

I'd like to find out which class causes this problem.


readObject
        | header oop nWords class format |
        header := self readUint32.
        (header bitAnd: HeaderTypeMask) caseOf: {
                [HeaderTypeSizeAndClass] ->
                        [nWords := header >> 2. class := self readUint32. header := self readUint32].
                [HeaderTypeClass] ->


The header is a 32 bit integer denoting the class, I assume.
I'd like to know which class causes the problem.

How can I find out which class this 32 bit integer refers to?

Regards

Hannes


--------------------------------------------------------------------------------------------------------------------------------

[1] The project file has a workspace in it in addition to some
RectangleMorphs and SimpleButtonMorphs which were loaded succesfully
in another test.

http://forum.world.st/The-Trunk-Morphic-kfr-1435-mcz-tp5077266p5077476.html
The reference has a test project *.pr file attached.

[2]
http://lists.squeakfoundation.org/pipermail/squeak-dev/2018-May/198848.html

New method added

SmartRefStream  >> in the conversion protocol

methodPropertiespps0

       ^ AdditionalMethodState




[3] readObject
        | header oop nWords class format |
        header := self readUint32.
        (header bitAnd: HeaderTypeMask) caseOf: {
                [HeaderTypeSizeAndClass] ->
                        [nWords := header >> 2. class := self readUint32. header := self readUint32].
                [HeaderTypeClass] ->
                        [class := header - 1. header := self readUint32. nWords := header
>> 2 bitAnd: 63].
                [HeaderTypeShort] ->
                        [nWords := header >> 2 bitAnd: 63. class := header >> 12 bitAnd: 31].
        } otherwise: [self error: 'unexpected free chunk'].
        nWords := nWords - 1. "nWords includes 1 header word"
        oop := position.
        ^[oopMap at: oop ifAbsentPut:
                [format := header >> 8 bitAnd: 15.
                "hash := header >> 17 bitAnd: 4095."
                self allocateObject: format class: class size: nWords]]
                        ensure: [position := oop + (nWords * 4)]

Reply | Threaded
Open this post in threaded view
|

Re: How does ImageSegmentLoader>>readObject work?

Eliot Miranda-2
Hi Hannes,

On Wed, May 23, 2018 at 4:15 AM, H. Hirzel <[hidden email]> wrote:
Hello

this is a follow up mail on loading a project file (*.pr) saved in
Squeak 3.10.2 [1] into Squeak 6.0a-#18000  [2]

The code breaks at

    ImageSegmentLoader>>readObject [3]

I'd like to find out which class causes this problem.


readObject
        | header oop nWords class format |
        header := self readUint32.
        (header bitAnd: HeaderTypeMask) caseOf: {
                [HeaderTypeSizeAndClass] ->
                        [nWords := header >> 2. class := self readUint32. header := self readUint32].
                [HeaderTypeClass] ->


The header is a 32 bit integer denoting the class, I assume.
I'd like to know which class causes the problem.

How can I find out which class this 32 bit integer refers to?

There are three header formats, selected by a two bit field, the fourth value identifying free objects, and hence never seen in an image segment.  The formats are

HeaderTypeSizeAndClass (0) The header is three words; the first is a size field, the second is the class up, and the third is the header (containing format, GC bits, etc)

HeaderTypeClass (1) The header is two words; the first is the class oop, the second is the header (containing format, GC bits, etc, and in this case a 6 bit word size field)

(HeaderTypeFree (2))

HeaderTypeShort (3): The header is one word, being the header including a 5 bit field (31 bitShift: 12) that is a zero-relative index into an array of 32 possible compact classes that in pre-Spur Squeaks are held in Smalltalk compactClasses.  In ImageSegmentLoader they are in CompactClasses, initialized in the class-side initialize method.  I would be willing to bet real money that this is the kind of header that is failing, and that the issue is missing compact classes such as MethodProperties.

Here's the layout of the header word in all three cases (taken from ObjectMemory's class comment in the VMMaker/VMMaker.oscog package):

MSB 3 bits reserved for gc (mark, root, unused)
12 bits object hash (for HashSets) 5 bits compact class index 4 bits object format 6 bits object size in 32-bit words LSB: 2 bits header type (0: 3-word, 1: 2-word, 2: forbidden, 3: 1-word)


Here's the initialization of CompactClasses from ImageSegmentLoader class>>initialize:

CompactClasses := {CompiledMethod. nil. Array. nil.
LargePositiveInteger. Float. MethodDictionary. Association.
Point. Rectangle. ByteString. nil.
nil "was BlockContext; needs special handling". Context. nil. Bitmap. 
nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil}.

Looking at the V39 sources I can't see an initialization of the compactClassesArray; recreateSpecialObjectsArray simply copies the existing compactClassesArray.  Each image can have its own compact classes (Behavior>>becomeCompact).  Therefore there has to be a copy of the compact classes array saved in the image segment.  I can see traces of this but don't know how the code works.  Look for compactClassesArray in the 3.9/3.10 sources and you may be able to find where the compact classes are in an image segment.

Once you've located the compact classes you'll have to decide how to handle the mapping.  The most important two things that have changed from 3.10 to Spur are

1. the bytecode set and block implementation; 3.10 used Smalltalk-80 style non-reentrant BlockContexts; as of Cog (4.0? 4.1?) we moved to reentrant BlockClosure, where MethodContext (now Context) is used for block and method activations.  There are 6 new byte codes used to implement closures; see the class comment of EncoderForV3PlusClosures

2. 3.10 had an instance of MethodProperties as the penultimate literal in all methods.  As of Cog (but this has nothing to do with the VM) methods either have a selector in the penultimate literal, or an instance of AdditionalMethodState (if the method has a pragma or properties), hence saving considerable space.


To map from BlockContext to BlockClosure you'll have to revive the decompiler for pre-closure bytecodes, decompile the block's method, recompile to closure byte codes, and map PCs and temp vars appropriately.  This isn't easy in general, but it is possible, at least in theory, and simple blocks may be simple (e.g. blocks that only take arguments).  (Think about the problem as you would in understanding how the Debugger maps from a context's pc and temporary variables to its display of the currently executing expression, and the temporary variables and back.  At least in 4.0 and/or 4.1 we had the pc/variable mapping system working for both, and we still have remnants of this; see DebuggerMethodMap's two subclasses, DebuggerMethodMapForBlueBookMethods & DebuggerMethodMapForClosureCompiledMethods.  You should be able to use these to perform the mapping.

To map from MethodProperties to AdditionalMethodState in a CompiledMethod is essentially trivial.  See if the MethodProperties has only a selector and then simply use the selector in place of the MethodProperties.  If there are additional properties, create an equivalent AdditionalMethodState, copying across the properties, and use that in place of the MethodProperties.

This is low-level enough that you may want to pair or at least consult with myself or Bert.  I was the one who wrought all this change.  I apologise for the severe inconvenience, but the current relative performance between Cog and the 3.10 interpreter in part depends on these changes.


HTH


Regards

Hannes


--------------------------------------------------------------------------------------------------------------------------------

[1] The project file has a workspace in it in addition to some
RectangleMorphs and SimpleButtonMorphs which were loaded succesfully
in another test.

http://forum.world.st/The-Trunk-Morphic-kfr-1435-mcz-tp5077266p5077476.html
The reference has a test project *.pr file attached.

[2]
http://lists.squeakfoundation.org/pipermail/squeak-dev/2018-May/198848.html

New method added

SmartRefStream  >> in the conversion protocol

methodPropertiespps0

       ^ AdditionalMethodState




[3] readObject
        | header oop nWords class format |
        header := self readUint32.
        (header bitAnd: HeaderTypeMask) caseOf: {
                [HeaderTypeSizeAndClass] ->
                        [nWords := header >> 2. class := self readUint32. header := self readUint32].
                [HeaderTypeClass] ->
                        [class := header - 1. header := self readUint32. nWords := header
>> 2 bitAnd: 63].
                [HeaderTypeShort] ->
                        [nWords := header >> 2 bitAnd: 63. class := header >> 12 bitAnd: 31].
        } otherwise: [self error: 'unexpected free chunk'].
        nWords := nWords - 1.   "nWords includes 1 header word"
        oop := position.
        ^[oopMap at: oop ifAbsentPut:
                [format := header >> 8 bitAnd: 15.
                "hash := header >> 17 bitAnd: 4095."
                self allocateObject: format class: class size: nWords]]
                        ensure: [position := oop + (nWords * 4)]




--
_,,,^..^,,,_
best, Eliot


Reply | Threaded
Open this post in threaded view
|

Re: How does ImageSegmentLoader>>readObject work?

Karl Ramberg
Thanks for the write up :-)
I really enjoy using the speed up that your effort has brought and are impressed by what you and others have achieved.
That some parts like image segments need care is expected and I personally see it as opportunity to learn more about the new 
system.

Best,
Karl



On Wed, May 23, 2018 at 8:25 PM, Eliot Miranda <[hidden email]> wrote:
Hi Hannes,

On Wed, May 23, 2018 at 4:15 AM, H. Hirzel <[hidden email]> wrote:
Hello

this is a follow up mail on loading a project file (*.pr) saved in
Squeak 3.10.2 [1] into Squeak 6.0a-#18000  [2]

The code breaks at

    ImageSegmentLoader>>readObject [3]

I'd like to find out which class causes this problem.


readObject
        | header oop nWords class format |
        header := self readUint32.
        (header bitAnd: HeaderTypeMask) caseOf: {
                [HeaderTypeSizeAndClass] ->
                        [nWords := header >> 2. class := self readUint32. header := self readUint32].
                [HeaderTypeClass] ->


The header is a 32 bit integer denoting the class, I assume.
I'd like to know which class causes the problem.

How can I find out which class this 32 bit integer refers to?

There are three header formats, selected by a two bit field, the fourth value identifying free objects, and hence never seen in an image segment.  The formats are

HeaderTypeSizeAndClass (0) The header is three words; the first is a size field, the second is the class up, and the third is the header (containing format, GC bits, etc)

HeaderTypeClass (1) The header is two words; the first is the class oop, the second is the header (containing format, GC bits, etc, and in this case a 6 bit word size field)

(HeaderTypeFree (2))

HeaderTypeShort (3): The header is one word, being the header including a 5 bit field (31 bitShift: 12) that is a zero-relative index into an array of 32 possible compact classes that in pre-Spur Squeaks are held in Smalltalk compactClasses.  In ImageSegmentLoader they are in CompactClasses, initialized in the class-side initialize method.  I would be willing to bet real money that this is the kind of header that is failing, and that the issue is missing compact classes such as MethodProperties.

Here's the layout of the header word in all three cases (taken from ObjectMemory's class comment in the VMMaker/VMMaker.oscog package):

MSB 3 bits reserved for gc (mark, root, unused)
12 bits object hash (for HashSets) 5 bits compact class index 4 bits object format 6 bits object size in 32-bit words LSB: 2 bits header type (0: 3-word, 1: 2-word, 2: forbidden, 3: 1-word)


Here's the initialization of CompactClasses from ImageSegmentLoader class>>initialize:

CompactClasses := {CompiledMethod. nil. Array. nil.
LargePositiveInteger. Float. MethodDictionary. Association.
Point. Rectangle. ByteString. nil.
nil "was BlockContext; needs special handling". Context. nil. Bitmap. 
nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil}.

Looking at the V39 sources I can't see an initialization of the compactClassesArray; recreateSpecialObjectsArray simply copies the existing compactClassesArray.  Each image can have its own compact classes (Behavior>>becomeCompact).  Therefore there has to be a copy of the compact classes array saved in the image segment.  I can see traces of this but don't know how the code works.  Look for compactClassesArray in the 3.9/3.10 sources and you may be able to find where the compact classes are in an image segment.

Once you've located the compact classes you'll have to decide how to handle the mapping.  The most important two things that have changed from 3.10 to Spur are

1. the bytecode set and block implementation; 3.10 used Smalltalk-80 style non-reentrant BlockContexts; as of Cog (4.0? 4.1?) we moved to reentrant BlockClosure, where MethodContext (now Context) is used for block and method activations.  There are 6 new byte codes used to implement closures; see the class comment of EncoderForV3PlusClosures

2. 3.10 had an instance of MethodProperties as the penultimate literal in all methods.  As of Cog (but this has nothing to do with the VM) methods either have a selector in the penultimate literal, or an instance of AdditionalMethodState (if the method has a pragma or properties), hence saving considerable space.


To map from BlockContext to BlockClosure you'll have to revive the decompiler for pre-closure bytecodes, decompile the block's method, recompile to closure byte codes, and map PCs and temp vars appropriately.  This isn't easy in general, but it is possible, at least in theory, and simple blocks may be simple (e.g. blocks that only take arguments).  (Think about the problem as you would in understanding how the Debugger maps from a context's pc and temporary variables to its display of the currently executing expression, and the temporary variables and back.  At least in 4.0 and/or 4.1 we had the pc/variable mapping system working for both, and we still have remnants of this; see DebuggerMethodMap's two subclasses, DebuggerMethodMapForBlueBookMethods & DebuggerMethodMapForClosureCompiledMethods.  You should be able to use these to perform the mapping.

To map from MethodProperties to AdditionalMethodState in a CompiledMethod is essentially trivial.  See if the MethodProperties has only a selector and then simply use the selector in place of the MethodProperties.  If there are additional properties, create an equivalent AdditionalMethodState, copying across the properties, and use that in place of the MethodProperties.

This is low-level enough that you may want to pair or at least consult with myself or Bert.  I was the one who wrought all this change.  I apologise for the severe inconvenience, but the current relative performance between Cog and the 3.10 interpreter in part depends on these changes.


HTH


Regards

Hannes


--------------------------------------------------------------------------------------------------------------------------------

[1] The project file has a workspace in it in addition to some
RectangleMorphs and SimpleButtonMorphs which were loaded succesfully
in another test.

http://forum.world.st/The-Trunk-Morphic-kfr-1435-mcz-tp5077266p5077476.html
The reference has a test project *.pr file attached.

[2]
http://lists.squeakfoundation.org/pipermail/squeak-dev/2018-May/198848.html

New method added

SmartRefStream  >> in the conversion protocol

methodPropertiespps0

       ^ AdditionalMethodState




[3] readObject
        | header oop nWords class format |
        header := self readUint32.
        (header bitAnd: HeaderTypeMask) caseOf: {
                [HeaderTypeSizeAndClass] ->
                        [nWords := header >> 2. class := self readUint32. header := self readUint32].
                [HeaderTypeClass] ->
                        [class := header - 1. header := self readUint32. nWords := header
>> 2 bitAnd: 63].
                [HeaderTypeShort] ->
                        [nWords := header >> 2 bitAnd: 63. class := header >> 12 bitAnd: 31].
        } otherwise: [self error: 'unexpected free chunk'].
        nWords := nWords - 1.   "nWords includes 1 header word"
        oop := position.
        ^[oopMap at: oop ifAbsentPut:
                [format := header >> 8 bitAnd: 15.
                "hash := header >> 17 bitAnd: 4095."
                self allocateObject: format class: class size: nWords]]
                        ensure: [position := oop + (nWords * 4)]




--
_,,,^..^,,,_
best, Eliot






Reply | Threaded
Open this post in threaded view
|

Re: How does ImageSegmentLoader>>readObject work?

Hannes Hirzel
Hi Eliot

In the meantime quite a number of projects with Squeak 3 style object
memory load fine [1].
Time to look at the issue of the blocks again which affects some of them

You write

<citation>
The most important two things that have changed from Squeak version
3.10 to Spur are

1. the bytecode set and block implementation; 3.10 used Smalltalk-80
style non-reentrant BlockContexts; as of Cog (4.0? 4.1?) we moved to
reentrant BlockClosure, where MethodContext (now Context) is used for
block and method activations.  There are 6 new byte codes used to
implement closures;

see the class comment of EncoderForV3PlusClosures

2. 3.10 had an instance of MethodProperties as the penultimate literal
in all methods.  As of Cog (but this has nothing to do with the VM)
methods either have a selector in the penultimate literal, or an
instance of AdditionalMethodState (if the method has a pragma or
properties), hence saving considerable space.
</citation>

I think we need a simple test case (*.pr file from e.g. Squeak 3.10.2)
with BlockContexts and then a way how to deal with it. What is the
minimal test case for this?

As a first solution to be included in 5.2 the simplest thing  might be
'graceful degradation', i.e. just informing the user that the project
has old style BlockContexts and cannot be loaded yet.

Maybe more is possible ...

Kind regards
Hannes


[1] See thread test plan.
Etoys ProjectLoading tests in 5.2
And http://wiki.squeak.org/squeak/1183

On 5/23/18, karl ramberg <[hidden email]> wrote:

> Thanks for the write up :-)
> I really enjoy using the speed up that your effort has brought and are
> impressed by what you and others have achieved.
> That some parts like image segments need care is expected and I personally
> see it as opportunity to learn more about the new
> system.
>
> Best,
> Karl
>
>
>
> On Wed, May 23, 2018 at 8:25 PM, Eliot Miranda <[hidden email]>
> wrote:
>
>> Hi Hannes,
>>
>> On Wed, May 23, 2018 at 4:15 AM, H. Hirzel <[hidden email]>
>> wrote:
>>
>>> Hello
>>>
>>> this is a follow up mail on loading a project file (*.pr) saved in
>>> Squeak 3.10.2 [1] into Squeak 6.0a-#18000  [2]
>>>
>>> The code breaks at
>>>
>>>     ImageSegmentLoader>>readObject [3]
>>>
>>> I'd like to find out which class causes this problem.
>>>
>>>
>>> readObject
>>>         | header oop nWords class format |
>>>         header := self readUint32.
>>>         (header bitAnd: HeaderTypeMask) caseOf: {
>>>                 [HeaderTypeSizeAndClass] ->
>>>                         [nWords := header >> 2. class := self
>>> readUint32.
>>> header := self readUint32].
>>>                 [HeaderTypeClass] ->
>>>
>>>
>>> The header is a 32 bit integer denoting the class, I assume.
>>> I'd like to know which class causes the problem.
>>>
>>> How can I find out which class this 32 bit integer refers to?
>>>
>>
>> There are three header formats, selected by a two bit field, the fourth
>> value identifying free objects, and hence never seen in an image segment.
>> The formats are
>>
>> HeaderTypeSizeAndClass (0) The header is three words; the first is a size
>> field, the second is the class up, and the third is the header
>> (containing
>> format, GC bits, etc)
>>
>> HeaderTypeClass (1) The header is two words; the first is the class oop,
>> the second is the header (containing format, GC bits, etc, and in this
>> case
>> a 6 bit word size field)
>>
>> (HeaderTypeFree (2))
>>
>> HeaderTypeShort (3): The header is one word, being the header including a
>> 5 bit field (31 bitShift: 12) that is a zero-relative index into an array
>> of 32 possible compact classes that in pre-Spur Squeaks are held in
>> Smalltalk compactClasses.  In ImageSegmentLoader they are in
>> CompactClasses, initialized in the class-side initialize method.  I would
>> be willing to bet real money that this is the kind of header that is
>> failing, and that the issue is missing compact classes such as
>> MethodProperties.
>>
>> Here's the layout of the header word in all three cases (taken from
>> ObjectMemory's class comment in the VMMaker/VMMaker.oscog package):
>>
>> MSB 3 bits reserved for gc (mark, root, unused)
>> 12 bits object hash (for HashSets) 5 bits compact class index 4 bits
>> object format 6 bits object size in 32-bit words LSB: 2 bits header type
>> (0: 3-word, 1: 2-word, 2: forbidden, 3: 1-word)
>>
>>
>> Here's the initialization of CompactClasses from ImageSegmentLoader
>> class>>initialize:
>>
>> CompactClasses := {CompiledMethod. nil. Array. nil.
>> LargePositiveInteger. Float. MethodDictionary. Association.
>> Point. Rectangle. ByteString. nil.
>> nil "was BlockContext; needs special handling". Context. nil. Bitmap.
>> nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil.
>> nil}.
>>
>> Looking at the V39 sources I can't see an initialization of the
>> compactClassesArray; recreateSpecialObjectsArray simply copies the
>> existing
>> compactClassesArray.  Each image can have its own compact classes
>> (Behavior>>becomeCompact).  Therefore there has to be a copy of the
>> compact
>> classes array saved in the image segment.  I can see traces of this but
>> don't know how the code works.  Look for compactClassesArray in the
>> 3.9/3.10 sources and you may be able to find where the compact classes
>> are
>> in an image segment.
>>
>> Once you've located the compact classes you'll have to decide how to
>> handle the mapping.  The most important two things that have changed from
>> 3.10 to Spur are
>>
>> 1. the bytecode set and block implementation; 3.10 used Smalltalk-80
>> style
>> non-reentrant BlockContexts; as of Cog (4.0? 4.1?) we moved to reentrant
>> BlockClosure, where MethodContext (now Context) is used for block and
>> method activations.  There are 6 new byte codes used to implement
>> closures;
>> see the class comment of EncoderForV3PlusClosures
>>
>> 2. 3.10 had an instance of MethodProperties as the penultimate literal in
>> all methods.  As of Cog (but this has nothing to do with the VM) methods
>> either have a selector in the penultimate literal, or an instance of
>> AdditionalMethodState (if the method has a pragma or properties), hence
>> saving considerable space.
>>
>>
>> To map from BlockContext to BlockClosure you'll have to revive the
>> decompiler for pre-closure bytecodes, decompile the block's method,
>> recompile to closure byte codes, and map PCs and temp vars appropriately.
>> This isn't easy in general, but it is possible, at least in theory, and
>> simple blocks may be simple (e.g. blocks that only take arguments).
>> (Think
>> about the problem as you would in understanding how the Debugger maps
>> from
>> a context's pc and temporary variables to its display of the currently
>> executing expression, and the temporary variables and back.  At least in
>> 4.0 and/or 4.1 we had the pc/variable mapping system working for both,
>> and
>> we still have remnants of this; see DebuggerMethodMap's two subclasses,
>> DebuggerMethodMapForBlueBookMethods &
>> DebuggerMethodMapForClosureCompiledMethods.
>> You should be able to use these to perform the mapping.
>>
>> To map from MethodProperties to AdditionalMethodState in a CompiledMethod
>> is essentially trivial.  See if the MethodProperties has only a selector
>> and then simply use the selector in place of the MethodProperties.  If
>> there are additional properties, create an equivalent
>> AdditionalMethodState, copying across the properties, and use that in
>> place
>> of the MethodProperties.
>>
>> This is low-level enough that you may want to pair or at least consult
>> with myself or Bert.  I was the one who wrought all this change.  I
>> apologise for the severe inconvenience, but the current relative
>> performance between Cog and the 3.10 interpreter in part depends on these
>> changes.
>>
>>
>> HTH
>>
>>
>>> Regards
>>>
>>> Hannes
>>>
>>>
>>> ------------------------------------------------------------
>>> --------------------------------------------------------------------
>>>
>>> [1] The project file has a workspace in it in addition to some
>>> RectangleMorphs and SimpleButtonMorphs which were loaded succesfully
>>> in another test.
>>>
>>> http://forum.world.st/The-Trunk-Morphic-kfr-1435-mcz-tp50772
>>> 66p5077476.html
>>> The reference has a test project *.pr file attached.
>>>
>>> [2]
>>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2018-
>>> May/198848.html
>>>
>>> New method added
>>>
>>> SmartRefStream  >> in the conversion protocol
>>>
>>> methodPropertiespps0
>>>
>>>        ^ AdditionalMethodState
>>>
>>>
>>>
>>>
>>> [3] readObject
>>>         | header oop nWords class format |
>>>         header := self readUint32.
>>>         (header bitAnd: HeaderTypeMask) caseOf: {
>>>                 [HeaderTypeSizeAndClass] ->
>>>                         [nWords := header >> 2. class := self
>>> readUint32.
>>> header := self readUint32].
>>>                 [HeaderTypeClass] ->
>>>                         [class := header - 1. header := self readUint32.
>>> nWords := header
>>> >> 2 bitAnd: 63].
>>>                 [HeaderTypeShort] ->
>>>                         [nWords := header >> 2 bitAnd: 63. class :=
>>> header >> 12 bitAnd: 31].
>>>         } otherwise: [self error: 'unexpected free chunk'].
>>>         nWords := nWords - 1.   "nWords includes 1 header word"
>>>         oop := position.
>>>         ^[oopMap at: oop ifAbsentPut:
>>>                 [format := header >> 8 bitAnd: 15.
>>>                 "hash := header >> 17 bitAnd: 4095."
>>>                 self allocateObject: format class: class size: nWords]]
>>>                         ensure: [position := oop + (nWords * 4)]
>>>
>>>
>>
>>
>> --
>> _,,,^..^,,,_
>> best, Eliot
>>
>>
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: How does ImageSegmentLoader>>readObject work?

Hannes Hirzel
A minimal test case is

a) Take a Squeak 3.10.2 image such as the
http://folk.uio.no/trygver/assets/BabyIDE.zip version from Trygve Reenskaug

use an interpreter VM to open it.

b) open a new empty project (no flaps, no trash can): Label it
'SimplBtnMrphWthBlkAsTgt1'
   (the name needs to be less than 24 characters, unfortunately)

c) Use the second example of page http://wiki.squeak.org/squeak/6413
    a SimpleButtonMorph with a block as target.

d) Choose 'world menu' / 'projects' / 'save project on local file only'

e)  Get the file 'SimplBtnMrphWthBlkAsTgt1.001.pr' from the Squeaklets folder

f) Drop the project file onto the desktop of a Squeak5.2alpha latest
update: #18117 image

Note: the test file is attached, so actually points a) to e) are not
necessary unless you want to verify yourself.



On 6/28/18, H. Hirzel <[hidden email]> wrote:

> Hi Eliot
>
> In the meantime quite a number of projects with Squeak 3 style object
> memory load fine [1].
> Time to look at the issue of the blocks again which affects some of them
>
> You write
>
> <citation>
> The most important two things that have changed from Squeak version
> 3.10 to Spur are
>
> 1. the bytecode set and block implementation; 3.10 used Smalltalk-80
> style non-reentrant BlockContexts; as of Cog (4.0? 4.1?) we moved to
> reentrant BlockClosure, where MethodContext (now Context) is used for
> block and method activations.  There are 6 new byte codes used to
> implement closures;
>
> see the class comment of EncoderForV3PlusClosures
>
> 2. 3.10 had an instance of MethodProperties as the penultimate literal
> in all methods.  As of Cog (but this has nothing to do with the VM)
> methods either have a selector in the penultimate literal, or an
> instance of AdditionalMethodState (if the method has a pragma or
> properties), hence saving considerable space.
> </citation>
>
> I think we need a simple test case (*.pr file from e.g. Squeak 3.10.2)
> with BlockContexts and then a way how to deal with it. What is the
> minimal test case for this?
>
> As a first solution to be included in 5.2 the simplest thing  might be
> 'graceful degradation', i.e. just informing the user that the project
> has old style BlockContexts and cannot be loaded yet.
>
> Maybe more is possible ...
>
> Kind regards
> Hannes
>
>
> [1] See thread test plan.
> Etoys ProjectLoading tests in 5.2
> And http://wiki.squeak.org/squeak/1183
>
> On 5/23/18, karl ramberg <[hidden email]> wrote:
>> Thanks for the write up :-)
>> I really enjoy using the speed up that your effort has brought and are
>> impressed by what you and others have achieved.
>> That some parts like image segments need care is expected and I
>> personally
>> see it as opportunity to learn more about the new
>> system.
>>
>> Best,
>> Karl
>>
>>
>>
>> On Wed, May 23, 2018 at 8:25 PM, Eliot Miranda <[hidden email]>
>> wrote:
>>
>>> Hi Hannes,
>>>
>>> On Wed, May 23, 2018 at 4:15 AM, H. Hirzel <[hidden email]>
>>> wrote:
>>>
>>>> Hello
>>>>
>>>> this is a follow up mail on loading a project file (*.pr) saved in
>>>> Squeak 3.10.2 [1] into Squeak 6.0a-#18000  [2]
>>>>
>>>> The code breaks at
>>>>
>>>>     ImageSegmentLoader>>readObject [3]
>>>>
>>>> I'd like to find out which class causes this problem.
>>>>
>>>>
>>>> readObject
>>>>         | header oop nWords class format |
>>>>         header := self readUint32.
>>>>         (header bitAnd: HeaderTypeMask) caseOf: {
>>>>                 [HeaderTypeSizeAndClass] ->
>>>>                         [nWords := header >> 2. class := self
>>>> readUint32.
>>>> header := self readUint32].
>>>>                 [HeaderTypeClass] ->
>>>>
>>>>
>>>> The header is a 32 bit integer denoting the class, I assume.
>>>> I'd like to know which class causes the problem.
>>>>
>>>> How can I find out which class this 32 bit integer refers to?
>>>>
>>>
>>> There are three header formats, selected by a two bit field, the fourth
>>> value identifying free objects, and hence never seen in an image
>>> segment.
>>> The formats are
>>>
>>> HeaderTypeSizeAndClass (0) The header is three words; the first is a
>>> size
>>> field, the second is the class up, and the third is the header
>>> (containing
>>> format, GC bits, etc)
>>>
>>> HeaderTypeClass (1) The header is two words; the first is the class oop,
>>> the second is the header (containing format, GC bits, etc, and in this
>>> case
>>> a 6 bit word size field)
>>>
>>> (HeaderTypeFree (2))
>>>
>>> HeaderTypeShort (3): The header is one word, being the header including
>>> a
>>> 5 bit field (31 bitShift: 12) that is a zero-relative index into an
>>> array
>>> of 32 possible compact classes that in pre-Spur Squeaks are held in
>>> Smalltalk compactClasses.  In ImageSegmentLoader they are in
>>> CompactClasses, initialized in the class-side initialize method.  I
>>> would
>>> be willing to bet real money that this is the kind of header that is
>>> failing, and that the issue is missing compact classes such as
>>> MethodProperties.
>>>
>>> Here's the layout of the header word in all three cases (taken from
>>> ObjectMemory's class comment in the VMMaker/VMMaker.oscog package):
>>>
>>> MSB 3 bits reserved for gc (mark, root, unused)
>>> 12 bits object hash (for HashSets) 5 bits compact class index 4 bits
>>> object format 6 bits object size in 32-bit words LSB: 2 bits header type
>>> (0: 3-word, 1: 2-word, 2: forbidden, 3: 1-word)
>>>
>>>
>>> Here's the initialization of CompactClasses from ImageSegmentLoader
>>> class>>initialize:
>>>
>>> CompactClasses := {CompiledMethod. nil. Array. nil.
>>> LargePositiveInteger. Float. MethodDictionary. Association.
>>> Point. Rectangle. ByteString. nil.
>>> nil "was BlockContext; needs special handling". Context. nil. Bitmap.
>>> nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil.
>>> nil}.
>>>
>>> Looking at the V39 sources I can't see an initialization of the
>>> compactClassesArray; recreateSpecialObjectsArray simply copies the
>>> existing
>>> compactClassesArray.  Each image can have its own compact classes
>>> (Behavior>>becomeCompact).  Therefore there has to be a copy of the
>>> compact
>>> classes array saved in the image segment.  I can see traces of this but
>>> don't know how the code works.  Look for compactClassesArray in the
>>> 3.9/3.10 sources and you may be able to find where the compact classes
>>> are
>>> in an image segment.
>>>
>>> Once you've located the compact classes you'll have to decide how to
>>> handle the mapping.  The most important two things that have changed
>>> from
>>> 3.10 to Spur are
>>>
>>> 1. the bytecode set and block implementation; 3.10 used Smalltalk-80
>>> style
>>> non-reentrant BlockContexts; as of Cog (4.0? 4.1?) we moved to reentrant
>>> BlockClosure, where MethodContext (now Context) is used for block and
>>> method activations.  There are 6 new byte codes used to implement
>>> closures;
>>> see the class comment of EncoderForV3PlusClosures
>>>
>>> 2. 3.10 had an instance of MethodProperties as the penultimate literal
>>> in
>>> all methods.  As of Cog (but this has nothing to do with the VM) methods
>>> either have a selector in the penultimate literal, or an instance of
>>> AdditionalMethodState (if the method has a pragma or properties), hence
>>> saving considerable space.
>>>
>>>
>>> To map from BlockContext to BlockClosure you'll have to revive the
>>> decompiler for pre-closure bytecodes, decompile the block's method,
>>> recompile to closure byte codes, and map PCs and temp vars
>>> appropriately.
>>> This isn't easy in general, but it is possible, at least in theory, and
>>> simple blocks may be simple (e.g. blocks that only take arguments).
>>> (Think
>>> about the problem as you would in understanding how the Debugger maps
>>> from
>>> a context's pc and temporary variables to its display of the currently
>>> executing expression, and the temporary variables and back.  At least in
>>> 4.0 and/or 4.1 we had the pc/variable mapping system working for both,
>>> and
>>> we still have remnants of this; see DebuggerMethodMap's two subclasses,
>>> DebuggerMethodMapForBlueBookMethods &
>>> DebuggerMethodMapForClosureCompiledMethods.
>>> You should be able to use these to perform the mapping.
>>>
>>> To map from MethodProperties to AdditionalMethodState in a
>>> CompiledMethod
>>> is essentially trivial.  See if the MethodProperties has only a selector
>>> and then simply use the selector in place of the MethodProperties.  If
>>> there are additional properties, create an equivalent
>>> AdditionalMethodState, copying across the properties, and use that in
>>> place
>>> of the MethodProperties.
>>>
>>> This is low-level enough that you may want to pair or at least consult
>>> with myself or Bert.  I was the one who wrought all this change.  I
>>> apologise for the severe inconvenience, but the current relative
>>> performance between Cog and the 3.10 interpreter in part depends on
>>> these
>>> changes.
>>>
>>>
>>> HTH
>>>
>>>
>>>> Regards
>>>>
>>>> Hannes
>>>>
>>>>
>>>> ------------------------------------------------------------
>>>> --------------------------------------------------------------------
>>>>
>>>> [1] The project file has a workspace in it in addition to some
>>>> RectangleMorphs and SimpleButtonMorphs which were loaded succesfully
>>>> in another test.
>>>>
>>>> http://forum.world.st/The-Trunk-Morphic-kfr-1435-mcz-tp50772
>>>> 66p5077476.html
>>>> The reference has a test project *.pr file attached.
>>>>
>>>> [2]
>>>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2018-
>>>> May/198848.html
>>>>
>>>> New method added
>>>>
>>>> SmartRefStream  >> in the conversion protocol
>>>>
>>>> methodPropertiespps0
>>>>
>>>>        ^ AdditionalMethodState
>>>>
>>>>
>>>>
>>>>
>>>> [3] readObject
>>>>         | header oop nWords class format |
>>>>         header := self readUint32.
>>>>         (header bitAnd: HeaderTypeMask) caseOf: {
>>>>                 [HeaderTypeSizeAndClass] ->
>>>>                         [nWords := header >> 2. class := self
>>>> readUint32.
>>>> header := self readUint32].
>>>>                 [HeaderTypeClass] ->
>>>>                         [class := header - 1. header := self
>>>> readUint32.
>>>> nWords := header
>>>> >> 2 bitAnd: 63].
>>>>                 [HeaderTypeShort] ->
>>>>                         [nWords := header >> 2 bitAnd: 63. class :=
>>>> header >> 12 bitAnd: 31].
>>>>         } otherwise: [self error: 'unexpected free chunk'].
>>>>         nWords := nWords - 1.   "nWords includes 1 header word"
>>>>         oop := position.
>>>>         ^[oopMap at: oop ifAbsentPut:
>>>>                 [format := header >> 8 bitAnd: 15.
>>>>                 "hash := header >> 17 bitAnd: 4095."
>>>>                 self allocateObject: format class: class size: nWords]]
>>>>                         ensure: [position := oop + (nWords * 4)]
>>>>
>>>>
>>>
>>>
>>> --
>>> _,,,^..^,,,_
>>> best, Eliot
>>>
>>>
>>>
>>>
>>
>



SimplBtnMrphWthBlkAsTgt1.001.pr (11K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How does ImageSegmentLoader>>readObject work?

Edgar De Cleene
BabyIDE is not plain 3.10.2.
Test should be made with this link
http://ftp.squeak.org/3.10gamma/Squeak3.10.gamma.7159.zip and apropiate
Stack VM

On linux the default loaded via Synaptic works and on Mac the JMM Squeak
4.2.5beta1U.
I do not know Stack VM working on Windows 10, so feedback is welcome.


Edgar
@morplenauta




On 28/06/2018, 04:09, "H. Hirzel" <[hidden email]> wrote:

> A minimal test case is
>
> a) Take a Squeak 3.10.2 image such as the
> http://folk.uio.no/trygver/assets/BabyIDE.zip version from Trygve Reenskaug
>
> use an interpreter VM to open it.
>
> b) open a new empty project (no flaps, no trash can): Label it
> 'SimplBtnMrphWthBlkAsTgt1'
>    (the name needs to be less than 24 characters, unfortunately)
>
> c) Use the second example of page http://wiki.squeak.org/squeak/6413
>     a SimpleButtonMorph with a block as target.
>
> d) Choose 'world menu' / 'projects' / 'save project on local file only'
>
> e)  Get the file 'SimplBtnMrphWthBlkAsTgt1.001.pr' from the Squeaklets folder
>
> f) Drop the project file onto the desktop of a Squeak5.2alpha latest
> update: #18117 image
>
> Note: the test file is attached, so actually points a) to e) are not
> necessary unless you want to verify yourself.