How does ImageSegmentLoader>>readObject work?

> Hi Eliot
>
> In the meantime quite a number of projects with Squeak 3 style object
> memory load fine [1].
> Time to look at the issue of the blocks again which affects some of them
>
> You write
>
> <citation>
> The most important two things that have changed from Squeak version
> 3.10 to Spur are
>
> 1. the bytecode set and block implementation; 3.10 used Smalltalk-80
> style non-reentrant BlockContexts; as of Cog (4.0? 4.1?) we moved to
> reentrant BlockClosure, where MethodContext (now Context) is used for
> block and method activations.  There are 6 new byte codes used to
> implement closures;
>
> see the class comment of EncoderForV3PlusClosures
>
> 2. 3.10 had an instance of MethodProperties as the penultimate literal
> in all methods.  As of Cog (but this has nothing to do with the VM)
> methods either have a selector in the penultimate literal, or an
> instance of AdditionalMethodState (if the method has a pragma or
> properties), hence saving considerable space.
> </citation>
>
> I think we need a simple test case (*.pr file from e.g. Squeak 3.10.2)
> with BlockContexts and then a way how to deal with it. What is the
> minimal test case for this?
>
> As a first solution to be included in 5.2 the simplest thing  might be
> 'graceful degradation', i.e. just informing the user that the project
> has old style BlockContexts and cannot be loaded yet.
>
> Maybe more is possible ...
>
> Kind regards
> Hannes
>
>
> [1] See thread test plan.
> Etoys ProjectLoading tests in 5.2
> And http://wiki.squeak.org/squeak/1183
>
> On 5/23/18, karl ramberg <[hidden email]> wrote:
>> Thanks for the write up :-)
>> I really enjoy using the speed up that your effort has brought and are
>> impressed by what you and others have achieved.
>> That some parts like image segments need care is expected and I
>> personally
>> see it as opportunity to learn more about the new
>> system.
>>
>> Best,
>> Karl
>>
>>
>>
>> On Wed, May 23, 2018 at 8:25 PM, Eliot Miranda <[hidden email]>
>> wrote:
>>
>>> Hi Hannes,
>>>
>>> On Wed, May 23, 2018 at 4:15 AM, H. Hirzel <[hidden email]>
>>> wrote:
>>>
>>>> Hello
>>>>
>>>> this is a follow up mail on loading a project file (*.pr) saved in
>>>> Squeak 3.10.2 [1] into Squeak 6.0a-#18000  [2]
>>>>
>>>> The code breaks at
>>>>
>>>>     ImageSegmentLoader>>readObject [3]
>>>>
>>>> I'd like to find out which class causes this problem.
>>>>
>>>>
>>>> readObject
>>>>         | header oop nWords class format |
>>>>         header := self readUint32.
>>>>         (header bitAnd: HeaderTypeMask) caseOf: {
>>>>                 [HeaderTypeSizeAndClass] ->
>>>>                         [nWords := header >> 2. class := self
>>>> readUint32.
>>>> header := self readUint32].
>>>>                 [HeaderTypeClass] ->
>>>>
>>>>
>>>> The header is a 32 bit integer denoting the class, I assume.
>>>> I'd like to know which class causes the problem.
>>>>
>>>> How can I find out which class this 32 bit integer refers to?
>>>>
>>>
>>> There are three header formats, selected by a two bit field, the fourth
>>> value identifying free objects, and hence never seen in an image
>>> segment.
>>> The formats are
>>>
>>> HeaderTypeSizeAndClass (0) The header is three words; the first is a
>>> size
>>> field, the second is the class up, and the third is the header
>>> (containing
>>> format, GC bits, etc)
>>>
>>> HeaderTypeClass (1) The header is two words; the first is the class oop,
>>> the second is the header (containing format, GC bits, etc, and in this
>>> case
>>> a 6 bit word size field)
>>>
>>> (HeaderTypeFree (2))
>>>
>>> HeaderTypeShort (3): The header is one word, being the header including
>>> a
>>> 5 bit field (31 bitShift: 12) that is a zero-relative index into an
>>> array
>>> of 32 possible compact classes that in pre-Spur Squeaks are held in
>>> Smalltalk compactClasses.  In ImageSegmentLoader they are in
>>> CompactClasses, initialized in the class-side initialize method.  I
>>> would
>>> be willing to bet real money that this is the kind of header that is
>>> failing, and that the issue is missing compact classes such as
>>> MethodProperties.
>>>
>>> Here's the layout of the header word in all three cases (taken from
>>> ObjectMemory's class comment in the VMMaker/VMMaker.oscog package):
>>>
>>> MSB 3 bits reserved for gc (mark, root, unused)
>>> 12 bits object hash (for HashSets) 5 bits compact class index 4 bits
>>> object format 6 bits object size in 32-bit words LSB: 2 bits header type
>>> (0: 3-word, 1: 2-word, 2: forbidden, 3: 1-word)
>>>
>>>
>>> Here's the initialization of CompactClasses from ImageSegmentLoader
>>> class>>initialize:
>>>
>>> CompactClasses := {CompiledMethod. nil. Array. nil.
>>> LargePositiveInteger. Float. MethodDictionary. Association.
>>> Point. Rectangle. ByteString. nil.
>>> nil "was BlockContext; needs special handling". Context. nil. Bitmap.
>>> nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil.
>>> nil}.
>>>
>>> Looking at the V39 sources I can't see an initialization of the
>>> compactClassesArray; recreateSpecialObjectsArray simply copies the
>>> existing
>>> compactClassesArray.  Each image can have its own compact classes
>>> (Behavior>>becomeCompact).  Therefore there has to be a copy of the
>>> compact
>>> classes array saved in the image segment.  I can see traces of this but
>>> don't know how the code works.  Look for compactClassesArray in the
>>> 3.9/3.10 sources and you may be able to find where the compact classes
>>> are
>>> in an image segment.
>>>
>>> Once you've located the compact classes you'll have to decide how to
>>> handle the mapping.  The most important two things that have changed
>>> from
>>> 3.10 to Spur are
>>>
>>> 1. the bytecode set and block implementation; 3.10 used Smalltalk-80
>>> style
>>> non-reentrant BlockContexts; as of Cog (4.0? 4.1?) we moved to reentrant
>>> BlockClosure, where MethodContext (now Context) is used for block and
>>> method activations.  There are 6 new byte codes used to implement
>>> closures;
>>> see the class comment of EncoderForV3PlusClosures
>>>
>>> 2. 3.10 had an instance of MethodProperties as the penultimate literal
>>> in
>>> all methods.  As of Cog (but this has nothing to do with the VM) methods
>>> either have a selector in the penultimate literal, or an instance of
>>> AdditionalMethodState (if the method has a pragma or properties), hence
>>> saving considerable space.
>>>
>>>
>>> To map from BlockContext to BlockClosure you'll have to revive the
>>> decompiler for pre-closure bytecodes, decompile the block's method,
>>> recompile to closure byte codes, and map PCs and temp vars
>>> appropriately.
>>> This isn't easy in general, but it is possible, at least in theory, and
>>> simple blocks may be simple (e.g. blocks that only take arguments).
>>> (Think
>>> about the problem as you would in understanding how the Debugger maps
>>> from
>>> a context's pc and temporary variables to its display of the currently
>>> executing expression, and the temporary variables and back.  At least in
>>> 4.0 and/or 4.1 we had the pc/variable mapping system working for both,
>>> and
>>> we still have remnants of this; see DebuggerMethodMap's two subclasses,
>>> DebuggerMethodMapForBlueBookMethods &
>>> DebuggerMethodMapForClosureCompiledMethods.
>>> You should be able to use these to perform the mapping.
>>>
>>> To map from MethodProperties to AdditionalMethodState in a
>>> CompiledMethod
>>> is essentially trivial.  See if the MethodProperties has only a selector
>>> and then simply use the selector in place of the MethodProperties.  If
>>> there are additional properties, create an equivalent
>>> AdditionalMethodState, copying across the properties, and use that in
>>> place
>>> of the MethodProperties.
>>>
>>> This is low-level enough that you may want to pair or at least consult
>>> with myself or Bert.  I was the one who wrought all this change.  I
>>> apologise for the severe inconvenience, but the current relative
>>> performance between Cog and the 3.10 interpreter in part depends on
>>> these
>>> changes.
>>>
>>>
>>> HTH
>>>
>>>
>>>> Regards
>>>>
>>>> Hannes
>>>>
>>>>
>>>> ------------------------------------------------------------
>>>> --------------------------------------------------------------------
>>>>
>>>> [1] The project file has a workspace in it in addition to some
>>>> RectangleMorphs and SimpleButtonMorphs which were loaded succesfully
>>>> in another test.
>>>>
>>>> http://forum.world.st/The-Trunk-Morphic-kfr-1435-mcz-tp50772
>>>> 66p5077476.html
>>>> The reference has a test project *.pr file attached.
>>>>
>>>> [2]
>>>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2018-
>>>> May/198848.html
>>>>
>>>> New method added
>>>>
>>>> SmartRefStream  >> in the conversion protocol
>>>>
>>>> methodPropertiespps0
>>>>
>>>>        ^ AdditionalMethodState
>>>>
>>>>
>>>>
>>>>
>>>> [3] readObject
>>>>         | header oop nWords class format |
>>>>         header := self readUint32.
>>>>         (header bitAnd: HeaderTypeMask) caseOf: {
>>>>                 [HeaderTypeSizeAndClass] ->
>>>>                         [nWords := header >> 2. class := self
>>>> readUint32.
>>>> header := self readUint32].
>>>>                 [HeaderTypeClass] ->
>>>>                         [class := header - 1. header := self
>>>> readUint32.
>>>> nWords := header
>>>> >> 2 bitAnd: 63].
>>>>                 [HeaderTypeShort] ->
>>>>                         [nWords := header >> 2 bitAnd: 63. class :=
>>>> header >> 12 bitAnd: 31].
>>>>         } otherwise: [self error: 'unexpected free chunk'].
>>>>         nWords := nWords - 1.   "nWords includes 1 header word"
>>>>         oop := position.
>>>>         ^[oopMap at: oop ifAbsentPut:
>>>>                 [format := header >> 8 bitAnd: 15.
>>>>                 "hash := header >> 17 bitAnd: 4095."
>>>>                 self allocateObject: format class: class size: nWords]]
>>>>                         ensure: [position := oop + (nWords * 4)]
>>>>
>>>>
>>>
>>>
>>> --
>>> _,,,^..^,,,_
>>> best, Eliot
>>>
>>>
>>>
>>>
>>
>