Probably pointless endianness question re: Float changes circa 4.6

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Probably pointless endianness question re: Float changes circa 4.6

Tim Johnson-2
 
Hi all,

I finally achieved something I have wanted to do for ten years, though  
its relevance has steadily declined over that time:  build the Carbon  
and Cocoa VMs on a PowerPC Mac.

- Carbon VM (4.2.5b1, from VMMaker 4.2.4 on Squeak 4.1)
- iOS/ObjC/Cocoa VM (5.7b2, from VMMaker 4.2.6 on Squeak 4.1)

After building a successful product, I took the Carbon VM and began  
incrementing the non-oscog VMMaker's version in a Squeak 4.5 image.  I  
was able to build products along the way from 4.3.21 past 4.4.12 and  
beyond.  (I had to stop when [as could be expected] some plugins  
wouldn't export and some support code was missing.)

Eventually I had a Mac Carbon VM which could load 4.6's format 6505  
image, and it began displaying the world, but it also had a few  
drawing errors and didn't respond to events.  A snip from  
SqueakDebug.log is at the end of this message, but probably isn't  
worth anyone's time to evaluate!

Where I think my efforts hit a dead end is with the Float changes [2]  
in the 6505 image format used by Squeak 4.6 [1].

But I'm just hoping to verify here that the Float changes in image  
format 6505 could have fatal implications for a big-endian platform  
like PowerPC (the formerly-native format for the Floats ;) ) on a  
"vintage" stack/interpreter VM, and that perhaps that would have been /
expected/ to be the last straw for it.  (A further question might be:  
how big would the lift have been to keep it going?  Or was that work  
done and just would require some extra attention?)

I'm sure none of this is relevant or worth time.  I guess my main  
question is whether this was supposed to have "kept working" but  
didn't — and whether that could have lingering implications on  
Squeak's effort (thanks David T Lewis!) to continue supporting the  
stack/interpreter VMMaker (though specifically only affecting big-
endian platforms, which admittedly is a very small population).  I  
suppose there could hypothetically be some big-endian platform out  
there which gets some Linux package of the 4.10.2 VM and fails to  
launch any image past 4.5 for similar reasons...?

FWIW, I did try the old Darwin/PPC port of 4.10.2 [3]  with both 4.6  
and David T Lewis's "Squeak 5.2-in-V3 format" image (which is no  
longer available on the 'net!) and it exhibited similar behavior:  
couldn't display the world, ate 100% CPU, had to be forced to quit.

Thanks,
a Tim

[1] http://wiki.squeak.org/squeak/6290
[2] http://forum.world.st/float-word-order-td106936.html
[3] http://squeakvm.org/unix/

Error: Cannot truncate this number
3 July 2020 8:11:03.001 am

VM: unix - Smalltalk
Image: Squeak4.6 [latest update: #13700]

SecurityManager state:
Restricted: false
FileAccess: true
SocketAccess: true
Working Dir /Users/ADMIN/Downloads/Squeak4.6-13700
Trusted Dir /Users/ADMIN/Downloads/Squeak4.6-13700/secure
Untrusted Dir /Users/ADMIN/Downloads/Squeak4.6-13700/My Squeak

Float(Object)>>error:
        Receiver: NaN
        Arguments and temporary variables:
                aString: 'Cannot truncate this number'
        Receiver's instance variables:
NaN
Float>>truncated
        Receiver: NaN
        Arguments and temporary variables:
                di: nil
                df: nil
                q: nil
                r: nil
        Receiver's instance variables:
NaN
Float>>rounded
        Receiver: NaN
        Arguments and temporary variables:

        Receiver's instance variables:
NaN
Color>>setRed:green:blue:
        Receiver: <<error during printing>>
        Arguments and temporary variables:
                r: NaN
                g: NaN
                b: NaN
        Receiver's instance variables:
                rgb: nil
                cachedDepth: nil
                cachedBitPattern: nil

Color class>>r:g:b:
        Receiver: Color
        Arguments and temporary variables:
                r: NaN
                g: NaN
                b: NaN
        Receiver's instance variables:
                superclass: Object
                methodDict: a MethodDictionary(size 107)
                format: 136
                instanceVariables: #('rgb' 'cachedDepth' 'cachedBitPattern')
                organization: ('access' alpha blue brightness green hue luminance  
red saturatio...etc...
                subclasses: {TranslucentColor}
                name: #Color
                classPool: a Dictionary(#Black->Color black #Blue->Color blue  
#BlueShift->0 #Br...etc...
                sharedPools: nil
                environment: Smalltalk
                category: #'Graphics-Primitives'

Color class>>r:g:b:alpha:
        Receiver: Color
        Arguments and temporary variables:
                r: NaN
                g: NaN
                b: NaN
                alpha: 0.0
        Receiver's instance variables:
                superclass: Object
                methodDict: a MethodDictionary(size 107)
                format: 136
                instanceVariables: #('rgb' 'cachedDepth' 'cachedBitPattern')
                organization: ('access' alpha blue brightness green hue luminance  
red saturatio...etc...
                subclasses: {TranslucentColor}
                name: #Color
                classPool: a Dictionary(#Black->Color black #Blue->Color blue  
#BlueShift->0 #Br...etc...
                sharedPools: nil
                environment: Smalltalk
                category: #'Graphics-Primitives'

Color>>alphaMixed:with:
        Receiver: Color veryDarkGray
        Arguments and temporary variables:

Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

David T. Lewis
 
Hi Tim,

I cannot follow up on this right now, but will do so soon (maybe tomorrow).

The float changes for format 6505 are actually quite simple so I'm sure
we can get this sorted out.

Dave

On Sun, Jul 05, 2020 at 12:52:24PM -0700, Tim Johnson wrote:

>
> Hi all,
>
> I finally achieved something I have wanted to do for ten years, though  
> its relevance has steadily declined over that time:  build the Carbon  
> and Cocoa VMs on a PowerPC Mac.
>
> - Carbon VM (4.2.5b1, from VMMaker 4.2.4 on Squeak 4.1)
> - iOS/ObjC/Cocoa VM (5.7b2, from VMMaker 4.2.6 on Squeak 4.1)
>
> After building a successful product, I took the Carbon VM and began  
> incrementing the non-oscog VMMaker's version in a Squeak 4.5 image.  I  
> was able to build products along the way from 4.3.21 past 4.4.12 and  
> beyond.  (I had to stop when [as could be expected] some plugins  
> wouldn't export and some support code was missing.)
>
> Eventually I had a Mac Carbon VM which could load 4.6's format 6505  
> image, and it began displaying the world, but it also had a few  
> drawing errors and didn't respond to events.  A snip from  
> SqueakDebug.log is at the end of this message, but probably isn't  
> worth anyone's time to evaluate!
>
> Where I think my efforts hit a dead end is with the Float changes [2]  
> in the 6505 image format used by Squeak 4.6 [1].
>
> But I'm just hoping to verify here that the Float changes in image  
> format 6505 could have fatal implications for a big-endian platform  
> like PowerPC (the formerly-native format for the Floats ;) ) on a  
> "vintage" stack/interpreter VM, and that perhaps that would have been /
> expected/ to be the last straw for it.  (A further question might be:  
> how big would the lift have been to keep it going?  Or was that work  
> done and just would require some extra attention?)
>
> I'm sure none of this is relevant or worth time.  I guess my main  
> question is whether this was supposed to have "kept working" but  
> didn't ??? and whether that could have lingering implications on  
> Squeak's effort (thanks David T Lewis!) to continue supporting the  
> stack/interpreter VMMaker (though specifically only affecting big-
> endian platforms, which admittedly is a very small population).  I  
> suppose there could hypothetically be some big-endian platform out  
> there which gets some Linux package of the 4.10.2 VM and fails to  
> launch any image past 4.5 for similar reasons...?
>
> FWIW, I did try the old Darwin/PPC port of 4.10.2 [3]  with both 4.6  
> and David T Lewis's "Squeak 5.2-in-V3 format" image (which is no  
> longer available on the 'net!) and it exhibited similar behavior:  
> couldn't display the world, ate 100% CPU, had to be forced to quit.
>
> Thanks,
> a Tim
>
> [1] http://wiki.squeak.org/squeak/6290
> [2] http://forum.world.st/float-word-order-td106936.html
> [3] http://squeakvm.org/unix/
>
> Error: Cannot truncate this number
> 3 July 2020 8:11:03.001 am
>
> VM: unix - Smalltalk
> Image: Squeak4.6 [latest update: #13700]
>
> SecurityManager state:
> Restricted: false
> FileAccess: true
> SocketAccess: true
> Working Dir /Users/ADMIN/Downloads/Squeak4.6-13700
> Trusted Dir /Users/ADMIN/Downloads/Squeak4.6-13700/secure
> Untrusted Dir /Users/ADMIN/Downloads/Squeak4.6-13700/My Squeak
>
> Float(Object)>>error:
> Receiver: NaN
> Arguments and temporary variables:
> aString: 'Cannot truncate this number'
> Receiver's instance variables:
> NaN
> Float>>truncated
> Receiver: NaN
> Arguments and temporary variables:
> di: nil
> df: nil
> q: nil
> r: nil
> Receiver's instance variables:
> NaN
> Float>>rounded
> Receiver: NaN
> Arguments and temporary variables:
>
> Receiver's instance variables:
> NaN
> Color>>setRed:green:blue:
> Receiver: <<error during printing>>
> Arguments and temporary variables:
> r: NaN
> g: NaN
> b: NaN
> Receiver's instance variables:
> rgb: nil
> cachedDepth: nil
> cachedBitPattern: nil
>
> Color class>>r:g:b:
> Receiver: Color
> Arguments and temporary variables:
> r: NaN
> g: NaN
> b: NaN
> Receiver's instance variables:
> superclass: Object
> methodDict: a MethodDictionary(size 107)
> format: 136
> instanceVariables: #('rgb' 'cachedDepth'
> 'cachedBitPattern')
> organization: ('access' alpha blue brightness green hue
> luminance  red saturatio...etc...
> subclasses: {TranslucentColor}
> name: #Color
> classPool: a Dictionary(#Black->Color black
> #Blue->Color blue  #BlueShift->0 #Br...etc...
> sharedPools: nil
> environment: Smalltalk
> category: #'Graphics-Primitives'
>
> Color class>>r:g:b:alpha:
> Receiver: Color
> Arguments and temporary variables:
> r: NaN
> g: NaN
> b: NaN
> alpha: 0.0
> Receiver's instance variables:
> superclass: Object
> methodDict: a MethodDictionary(size 107)
> format: 136
> instanceVariables: #('rgb' 'cachedDepth'
> 'cachedBitPattern')
> organization: ('access' alpha blue brightness green hue
> luminance  red saturatio...etc...
> subclasses: {TranslucentColor}
> name: #Color
> classPool: a Dictionary(#Black->Color black
> #Blue->Color blue  #BlueShift->0 #Br...etc...
> sharedPools: nil
> environment: Smalltalk
> category: #'Graphics-Primitives'
>
> Color>>alphaMixed:with:
> Receiver: Color veryDarkGray
> Arguments and temporary variables:
>
Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Eliot Miranda-2
In reply to this post by Tim Johnson-2
 
Hi Tim,

> On Jul 5, 2020, at 12:52 PM, Tim Johnson <[hidden email]> wrote:
>
> Hi all,
>
> I finally achieved something I have wanted to do for ten years, though its relevance has steadily declined over that time:  build the Carbon and Cocoa VMs on a PowerPC Mac.
>
> - Carbon VM (4.2.5b1, from VMMaker 4.2.4 on Squeak 4.1)
> - iOS/ObjC/Cocoa VM (5.7b2, from VMMaker 4.2.6 on Squeak 4.1)
>
> After building a successful product, I took the Carbon VM and began incrementing the non-oscog VMMaker's version in a Squeak 4.5 image.  I was able to build products along the way from 4.3.21 past 4.4.12 and beyond.  (I had to stop when [as could be expected] some plugins wouldn't export and some support code was missing.)
>
> Eventually I had a Mac Carbon VM which could load 4.6's format 6505 image, and it began displaying the world, but it also had a few drawing errors and didn't respond to events.  A snip from SqueakDebug.log is at the end of this message, but probably isn't worth anyone's time to evaluate!
>
> Where I think my efforts hit a dead end is with the Float changes [2] in the 6505 image format used by Squeak 4.6 [1].
>
> But I'm just hoping to verify here that the Float changes in image format 6505 could have fatal implications for a big-endian platform like PowerPC (the formerly-native format for the Floats ;) ) on a "vintage" stack/interpreter VM, and that perhaps that would have been /expected/ to be the last straw for it.  (A further question might be:  how big would the lift have been to keep it going?  Or was that work done and just would require some extra attention?)

I designed the scheme so that it would *not* have fatal implications on any platform :-)

First let’s understand the representation.  A Float is a double-precision 64-bit ieee float represented as a two element word array.

Before the changes floats were always stored in the heap in big-endian format which meant:

- aFloat at: 1 accessed the most significant word of the float on both big & little median platforms
- on little endian platforms the vm therefore has to word reverse Float objects, perform arithmetic to obtain a new 64-big float, and word reverse it to store it back into a new Float

This unfairly penalizes little endian platforms.

My simple change was three-fold:
1. A bit in the image header specifies whether floats in the image are in big-endian format (bit unset) or in platform format (eg big endian on PPC, little endian on x86).

2. on load if required all Floats would be word reversed to ensure they are in platform order

3. Float at: was modified so that on little endian platforms Float at: 1 answered the most significant word, just as it does on big endian platforms, hence hiding from the image the change in layout on little endian platforms and avoiding having to do any word reversal in arithmetic on any platforms


So what I did leveled the playing field.  How does this have fatal consequences for big endian platforms?

Note that the word reversal on Float at: is for free because the primitive only takes 1&2 as valid indices, so the implementation is really cheap.  So in fact what I did was ensure that fp was as fast as possible on both platforms and (consciously) didn’t penalize either platform.

Does this explanation make the scheme seem more sensible now?

>
> I'm sure none of this is relevant or worth time.  I guess my main question is whether this was supposed to have "kept working" but didn't — and whether that could have lingering implications on Squeak's effort (thanks David T Lewis!) to continue supporting the stack/interpreter VMMaker (though specifically only affecting big-endian platforms, which admittedly is a very small population).  I suppose there could hypothetically be some big-endian platform out there which gets some Linux package of the 4.10.2 VM and fails to launch any image past 4.5 for similar reasons...?
>
> FWIW, I did try the old Darwin/PPC port of 4.10.2 [3]  with both 4.6 and David T Lewis's "Squeak 5.2-in-V3 format" image (which is no longer available on the 'net!) and it exhibited similar behavior:  couldn't display the world, ate 100% CPU, had to be forced to quit.
>
> Thanks,
> a Tim
>
> [1] http://wiki.squeak.org/squeak/6290
> [2] http://forum.world.st/float-word-order-td106936.html
> [3] http://squeakvm.org/unix/
>
> Error: Cannot truncate this number
> 3 July 2020 8:11:03.001 am
>
> VM: unix - Smalltalk
> Image: Squeak4.6 [latest update: #13700]
>
> SecurityManager state:
> Restricted: false
> FileAccess: true
> SocketAccess: true
> Working Dir /Users/ADMIN/Downloads/Squeak4.6-13700
> Trusted Dir /Users/ADMIN/Downloads/Squeak4.6-13700/secure
> Untrusted Dir /Users/ADMIN/Downloads/Squeak4.6-13700/My Squeak
>
> Float(Object)>>error:
>    Receiver: NaN
>    Arguments and temporary variables:
>        aString:    'Cannot truncate this number'
>    Receiver's instance variables:
> NaN
> Float>>truncated
>    Receiver: NaN
>    Arguments and temporary variables:
>        di:    nil
>        df:    nil
>        q:    nil
>        r:    nil
>    Receiver's instance variables:
> NaN
> Float>>rounded
>    Receiver: NaN
>    Arguments and temporary variables:
>
>    Receiver's instance variables:
> NaN
> Color>>setRed:green:blue:
>    Receiver: <<error during printing>>
>    Arguments and temporary variables:
>        r:    NaN
>        g:    NaN
>        b:    NaN
>    Receiver's instance variables:
>        rgb:    nil
>        cachedDepth:    nil
>        cachedBitPattern:    nil
>
> Color class>>r:g:b:
>    Receiver: Color
>    Arguments and temporary variables:
>        r:    NaN
>        g:    NaN
>        b:    NaN
>    Receiver's instance variables:
>        superclass:    Object
>        methodDict:    a MethodDictionary(size 107)
>        format:    136
>        instanceVariables:    #('rgb' 'cachedDepth' 'cachedBitPattern')
>        organization:    ('access' alpha blue brightness green hue luminance red saturatio...etc...
>        subclasses:    {TranslucentColor}
>        name:    #Color
>        classPool:    a Dictionary(#Black->Color black #Blue->Color blue #BlueShift->0 #Br...etc...
>        sharedPools:    nil
>        environment:    Smalltalk
>        category:    #'Graphics-Primitives'
>
> Color class>>r:g:b:alpha:
>    Receiver: Color
>    Arguments and temporary variables:
>        r:    NaN
>        g:    NaN
>        b:    NaN
>        alpha:    0.0
>    Receiver's instance variables:
>        superclass:    Object
>        methodDict:    a MethodDictionary(size 107)
>        format:    136
>        instanceVariables:    #('rgb' 'cachedDepth' 'cachedBitPattern')
>        organization:    ('access' alpha blue brightness green hue luminance red saturatio...etc...
>        subclasses:    {TranslucentColor}
>        name:    #Color
>        classPool:    a Dictionary(#Black->Color black #Blue->Color blue #BlueShift->0 #Br...etc...
>        sharedPools:    nil
>        environment:    Smalltalk
>        category:    #'Graphics-Primitives'
>
> Color>>alphaMixed:with:
>    Receiver: Color veryDarkGray
>    Arguments and temporary variables:
>
Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Eliot Miranda-2
In reply to this post by David T. Lewis
 
Tim, P.S. now is the best time ever to attempt a 64-bit or 32-bit PPC JIT. The VMMaker.oscog infrastructure is better than ever. I did the ARMv8 code generator in record time (about 6 weeks, ignoring misunderstandings on my part).  And a 32-bit JIT would speed up 6505 images considerably.  I don’t have time to write either of these but I’d love to mentor you if you felt like having a go.

_,,,^..^,,,_ (phone)

> On Jul 5, 2020, at 2:16 PM, David T. Lewis <[hidden email]> wrote:
>
> 
> Hi Tim,
>
> I cannot follow up on this right now, but will do so soon (maybe tomorrow).
>
> The float changes for format 6505 are actually quite simple so I'm sure
> we can get this sorted out.
>
> Dave
>
>> On Sun, Jul 05, 2020 at 12:52:24PM -0700, Tim Johnson wrote:
>>
>> Hi all,
>>
>> I finally achieved something I have wanted to do for ten years, though  
>> its relevance has steadily declined over that time:  build the Carbon  
>> and Cocoa VMs on a PowerPC Mac.
>>
>> - Carbon VM (4.2.5b1, from VMMaker 4.2.4 on Squeak 4.1)
>> - iOS/ObjC/Cocoa VM (5.7b2, from VMMaker 4.2.6 on Squeak 4.1)
>>
>> After building a successful product, I took the Carbon VM and began  
>> incrementing the non-oscog VMMaker's version in a Squeak 4.5 image.  I  
>> was able to build products along the way from 4.3.21 past 4.4.12 and  
>> beyond.  (I had to stop when [as could be expected] some plugins  
>> wouldn't export and some support code was missing.)
>>
>> Eventually I had a Mac Carbon VM which could load 4.6's format 6505  
>> image, and it began displaying the world, but it also had a few  
>> drawing errors and didn't respond to events.  A snip from  
>> SqueakDebug.log is at the end of this message, but probably isn't  
>> worth anyone's time to evaluate!
>>
>> Where I think my efforts hit a dead end is with the Float changes [2]  
>> in the 6505 image format used by Squeak 4.6 [1].
>>
>> But I'm just hoping to verify here that the Float changes in image  
>> format 6505 could have fatal implications for a big-endian platform  
>> like PowerPC (the formerly-native format for the Floats ;) ) on a  
>> "vintage" stack/interpreter VM, and that perhaps that would have been /
>> expected/ to be the last straw for it.  (A further question might be:  
>> how big would the lift have been to keep it going?  Or was that work  
>> done and just would require some extra attention?)
>>
>> I'm sure none of this is relevant or worth time.  I guess my main  
>> question is whether this was supposed to have "kept working" but  
>> didn't ??? and whether that could have lingering implications on  
>> Squeak's effort (thanks David T Lewis!) to continue supporting the  
>> stack/interpreter VMMaker (though specifically only affecting big-
>> endian platforms, which admittedly is a very small population).  I  
>> suppose there could hypothetically be some big-endian platform out  
>> there which gets some Linux package of the 4.10.2 VM and fails to  
>> launch any image past 4.5 for similar reasons...?
>>
>> FWIW, I did try the old Darwin/PPC port of 4.10.2 [3]  with both 4.6  
>> and David T Lewis's "Squeak 5.2-in-V3 format" image (which is no  
>> longer available on the 'net!) and it exhibited similar behavior:  
>> couldn't display the world, ate 100% CPU, had to be forced to quit.
>>
>> Thanks,
>> a Tim
>>
>> [1] http://wiki.squeak.org/squeak/6290
>> [2] http://forum.world.st/float-word-order-td106936.html
>> [3] http://squeakvm.org/unix/
>>
>> Error: Cannot truncate this number
>> 3 July 2020 8:11:03.001 am
>>
>> VM: unix - Smalltalk
>> Image: Squeak4.6 [latest update: #13700]
>>
>> SecurityManager state:
>> Restricted: false
>> FileAccess: true
>> SocketAccess: true
>> Working Dir /Users/ADMIN/Downloads/Squeak4.6-13700
>> Trusted Dir /Users/ADMIN/Downloads/Squeak4.6-13700/secure
>> Untrusted Dir /Users/ADMIN/Downloads/Squeak4.6-13700/My Squeak
>>
>> Float(Object)>>error:
>>    Receiver: NaN
>>    Arguments and temporary variables:
>>        aString:    'Cannot truncate this number'
>>    Receiver's instance variables:
>> NaN
>> Float>>truncated
>>    Receiver: NaN
>>    Arguments and temporary variables:
>>        di:    nil
>>        df:    nil
>>        q:    nil
>>        r:    nil
>>    Receiver's instance variables:
>> NaN
>> Float>>rounded
>>    Receiver: NaN
>>    Arguments and temporary variables:
>>
>>    Receiver's instance variables:
>> NaN
>> Color>>setRed:green:blue:
>>    Receiver: <<error during printing>>
>>    Arguments and temporary variables:
>>        r:    NaN
>>        g:    NaN
>>        b:    NaN
>>    Receiver's instance variables:
>>        rgb:    nil
>>        cachedDepth:    nil
>>        cachedBitPattern:    nil
>>
>> Color class>>r:g:b:
>>    Receiver: Color
>>    Arguments and temporary variables:
>>        r:    NaN
>>        g:    NaN
>>        b:    NaN
>>    Receiver's instance variables:
>>        superclass:    Object
>>        methodDict:    a MethodDictionary(size 107)
>>        format:    136
>>        instanceVariables:    #('rgb' 'cachedDepth'
>>        'cachedBitPattern')
>>        organization:    ('access' alpha blue brightness green hue
>>        luminance  red saturatio...etc...
>>        subclasses:    {TranslucentColor}
>>        name:    #Color
>>        classPool:    a Dictionary(#Black->Color black
>>        #Blue->Color blue  #BlueShift->0 #Br...etc...
>>        sharedPools:    nil
>>        environment:    Smalltalk
>>        category:    #'Graphics-Primitives'
>>
>> Color class>>r:g:b:alpha:
>>    Receiver: Color
>>    Arguments and temporary variables:
>>        r:    NaN
>>        g:    NaN
>>        b:    NaN
>>        alpha:    0.0
>>    Receiver's instance variables:
>>        superclass:    Object
>>        methodDict:    a MethodDictionary(size 107)
>>        format:    136
>>        instanceVariables:    #('rgb' 'cachedDepth'
>>        'cachedBitPattern')
>>        organization:    ('access' alpha blue brightness green hue
>>        luminance  red saturatio...etc...
>>        subclasses:    {TranslucentColor}
>>        name:    #Color
>>        classPool:    a Dictionary(#Black->Color black
>>        #Blue->Color blue  #BlueShift->0 #Br...etc...
>>        sharedPools:    nil
>>        environment:    Smalltalk
>>        category:    #'Graphics-Primitives'
>>
>> Color>>alphaMixed:with:
>>    Receiver: Color veryDarkGray
>>    Arguments and temporary variables:
>>
Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

David T. Lewis
In reply to this post by Eliot Miranda-2
 
On Sun, Jul 05, 2020 at 03:00:23PM -0700, Eliot Miranda wrote:

>  
> Hi Tim,
>
> > On Jul 5, 2020, at 12:52 PM, Tim Johnson <[hidden email]> wrote:
> >
> > ???Hi all,
> >
> > I finally achieved something I have wanted to do for ten years, though its relevance has steadily declined over that time:  build the Carbon and Cocoa VMs on a PowerPC Mac.
> >
> > - Carbon VM (4.2.5b1, from VMMaker 4.2.4 on Squeak 4.1)
> > - iOS/ObjC/Cocoa VM (5.7b2, from VMMaker 4.2.6 on Squeak 4.1)
> >
> > After building a successful product, I took the Carbon VM and began incrementing the non-oscog VMMaker's version in a Squeak 4.5 image.  I was able to build products along the way from 4.3.21 past 4.4.12 and beyond.  (I had to stop when [as could be expected] some plugins wouldn't export and some support code was missing.)
> >
> > Eventually I had a Mac Carbon VM which could load 4.6's format 6505 image, and it began displaying the world, but it also had a few drawing errors and didn't respond to events.  A snip from SqueakDebug.log is at the end of this message, but probably isn't worth anyone's time to evaluate!
> >
> > Where I think my efforts hit a dead end is with the Float changes [2] in the 6505 image format used by Squeak 4.6 [1].
> >
> > But I'm just hoping to verify here that the Float changes in image format 6505 could have fatal implications for a big-endian platform like PowerPC (the formerly-native format for the Floats ;) ) on a "vintage" stack/interpreter VM, and that perhaps that would have been /expected/ to be the last straw for it.  (A further question might be:  how big would the lift have been to keep it going?  Or was that work done and just would require some extra attention?)
>
> I designed the scheme so that it would *not* have fatal implications on any platform :-)
>
> First let???s understand the representation.  A Float is a double-precision 64-bit ieee float represented as a two element word array.
>
> Before the changes floats were always stored in the heap in big-endian format which meant:
>
> - aFloat at: 1 accessed the most significant word of the float on both big & little median platforms
> - on little endian platforms the vm therefore has to word reverse Float objects, perform arithmetic to obtain a new 64-big float, and word reverse it to store it back into a new Float
>
> This unfairly penalizes little endian platforms.
>
> My simple change was three-fold:
> 1. A bit in the image header specifies whether floats in the image are in big-endian format (bit unset) or in platform format (eg big endian on PPC, little endian on x86).
>
> 2. on load if required all Floats would be word reversed to ensure they are in platform order
>

To add some detail:

- In a 6504 image, Floats are stored in big-endian format (as Eliot explains above) regardless of
  the endianness of the current platform

- In a 6505 image, Floats are stored in the native platform word ordering for the current platform

- When a 6505 image is loaded into a classic interpreter VM the floats are converted big-endian
  format in ContextInterpreter>>normalizeFloatOrderingInImage which effectively turns it back
  into a 6504 image

- When as 6505 image that was saved from a little-endian machine is loaded into an oscog (Cog/Spur)
  VM running on a big-endian platform, it is loaded in the saved format (presumeably little endian)

So ... I am only guessing, but AFAIK we have never tried loading a 6505 image that was saved
on a little-endian machine into a Cog/Spur VM running on a big-endian platform. Quite possibly
in needs a word-reversal operation in that case.

Dave

Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Tim Johnson-2
In reply to this post by Eliot Miranda-2
 
Hi Eliot,

I had a decent draft of this response, and my mail client pooped out when I hit send.  Trying again.  If this comes across as terse (I doubt it ;) ), please forgive.

On Jul 5, 2020, at 3:00 PM, Eliot Miranda wrote:

But I'm just hoping to verify here that the Float changes in image format 6505 could have fatal implications for a big-endian platform like PowerPC (the formerly-native format for the Floats ;) ) on a "vintage" stack/interpreter VM, and that perhaps that would have been /expected/ to be the last straw for it.  (A further question might be:  how big would the lift have been to keep it going?  Or was that work done and just would require some extra attention?)

I designed the scheme so that it would *not* have fatal implications on any platform :-)

This is a good point.  It helps me to improve my question:  

Q: Could my build environment (platform support code, non-oscog VMMaker sources, Squeak 4.5 hosted-VMMaker) be missing something necessary for the scheme to work as designed?

(This question & others summarized below.)

First let’s understand the representation.  A Float is a double-precision 64-bit ieee float represented as a two element word array.

Before the changes floats were always stored in the heap in big-endian format which meant:

- aFloat at: 1 accessed the most significant word of the float on both big & little median platforms
- on little endian platforms the vm therefore has to word reverse Float objects, perform arithmetic to obtain a new 64-big float, and word reverse it to store it back into a new Float

This unfairly penalizes little endian platforms.

Yes -- no doubt this was an important move.


My simple change was three-fold:
1. A bit in the image header specifies whether floats in the image are in big-endian format (bit unset) or in platform format (eg big endian on PPC, little endian on x86).

2. on load if required all Floats would be word reversed to ensure they are in platform order

^^ this is something I'm hoping to instrument or measure;  more on that below.


3. Float at: was modified so that on little endian platforms Float at: 1 answered the most significant word, just as it does on big endian platforms, hence hiding from the image the change in layout on little endian platforms and avoiding having to do any word reversal in arithmetic on any platforms


So what I did leveled the playing field.  How does this have fatal consequences for big endian platforms?

Well, I mean, that's so similar to the question I was trying to get across:  could it?  And, if so, under which conditions?

I definitely didn't mean to sound conclusive or be finger-pointy, and I'm sorry if it read that way.

After getting the 6505 format image to load & the world to (mostly) display, it felt like a small victory.  But then seeing Float-related errors in SqueakDebug.log after force-quitting made me think this conversion process could be related, and I hoped to gather more info.

Note that the word reversal on Float at: is for free because the primitive only takes 1&2 as valid indices, so the implementation is really cheap.  So in fact what I did was ensure that fp was as fast as possible on both platforms and (consciously) didn’t penalize either platform.

Does this explanation make the scheme seem more sensible now?

I really appreciate this explanation.  All I could find previously on this subject  was that thread I linked in my prior message, which didn't indicate which path was actually chosen.  This helps make it clear.

From my easy chair, your scheme looks brilliant and thorough.  I never meant to suggest otherwise, and am also totally underprepared and underqualified to do so.  The only thing not sensible here is that I'm trying to build & update a ten-year-old snapshot of the code for the Mac VM on an antiquated machine based on a niche architecture.  :) 

Your & Dave's responses have helped me to hone my question(s):

Q:  Did this Float scheme for 6505 require a change to the VM in order to function properly?  If so, (similar to what I asked above) is it possible that the VM I am building is missing some part of that change?  (Whether that is something that exists in VMMaker-oscog but didn't make it to VMMaker, or whether it's some platform support code, or a changed version of a primitive...)  If not, I am probably barking up the wrong tree.

Q:  How long might I expect the Float conversion at load (#2 above) to take, on a slower image running on a slower computer?  Milliseconds, seconds, minutes, or longer?  If I wanted to log it, trace it, or debug it, how could I find it?  (Dave helped with this, thanks:  ContextInterpreter>>normalizeFloatOrderingInImage )

Q:  If I spend the time to figure this out and do something about it, is there any way it could benefit anyone other than myself?  That is the crux of my previous blathering about any hypothetical big-endian platforms of the future.  

I hope I haven't wasted your time or energy, Eliot.  Thanks a lot for your explanation.

Tim

Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Tim Johnson-2
In reply to this post by David T. Lewis
 
Hi Dave,

On Jul 5, 2020, at 5:48 PM, David T. Lewis wrote:

> To add some detail:
>
> - In a 6504 image, Floats are stored in big-endian format (as Eliot  
> explains above) regardless of
>  the endianness of the current platform
>
> - In a 6505 image, Floats are stored in the native platform word  
> ordering for the current platform
>
> - When a 6505 image is loaded into a classic interpreter VM the  
> floats are converted big-endian
>  format in ContextInterpreter>>normalizeFloatOrderingInImage which  
> effectively turns it back
>  into a 6504 image

Thank you.  This helps me know where to possibly dig into the  
conversion process.

Also, I wanted to try this.  So I just did:

1) I loaded Squeak4.6-13700 (one of the alpha releases) into the  
4.10.2-2614 VM on my Raspberry Pi, and then saved it out.

2) `ckformat` verifies that the original image was is a 6505 format,  
and the image I saved out is a 6504 format.

3) Transferred it over to my G4, and it very much runs!  This  
particular VM was built with VMMaker 4.4.12.

Amazing!  What an effective solution :)   Stepping stone-by-stone into  
the future...


> - When as 6505 image that was saved from a little-endian machine is  
> loaded into an oscog (Cog/Spur)
>  VM running on a big-endian platform, it is loaded in the saved  
> format (presumeably little endian)
>
> So ... I am only guessing, but AFAIK we have never tried loading a  
> 6505 image that was saved
> on a little-endian machine into a Cog/Spur VM running on a big-
> endian platform. Quite possibly
> in needs a word-reversal operation in that case.

Very interesting.

Are there any Cog/Spur VMs for big-endian platforms?

Thanks a lot,
Tim

Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Boris Shingarov
In reply to this post by Eliot Miranda-2
 

On 7/5/20 6:05 PM, Eliot Miranda wrote:
>  
> now is the best time ever to attempt a 64-bit or 32-bit PPC JIT

..."attempt"??  I have most of it working already.  I'll ping you when I
get it to the same point where we reached with i386 and arm (i.e.
booting the Reader and being able to evaluate "100 factorial").


Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

timrowledge
In reply to this post by Tim Johnson-2
 
Looks like you've got past the question, but I was going to point out that

a) I don't recall any of the various conversion at load things we've ever done taking a noticeable time; and remember I usually have the slowest machines

b) endianness of bitmaps may become an issue (again) if we try some of the possible graphics improvement ideas. For my original RISC OS port I actually did convert all bitmaps to little endian and the BitBlt and all the Form related code.

> On 2020-07-05, at 10:54 PM, Tim Johnson <[hidden email]> wrote:
>
> Hi Dave,
>
> On Jul 5, 2020, at 5:48 PM, David T. Lewis wrote:
>
>> To add some detail:
>>
>> - In a 6504 image, Floats are stored in big-endian format (as Eliot explains above) regardless of
>> the endianness of the current platform
>>
>> - In a 6505 image, Floats are stored in the native platform word ordering for the current platform
>>
>> - When a 6505 image is loaded into a classic interpreter VM the floats are converted big-endian
>> format in ContextInterpreter>>normalizeFloatOrderingInImage which effectively turns it back
>> into a 6504 image
>
> Thank you.  This helps me know where to possibly dig into the conversion process.
>
> Also, I wanted to try this.  So I just did:
>
> 1) I loaded Squeak4.6-13700 (one of the alpha releases) into the 4.10.2-2614 VM on my Raspberry Pi, and then saved it out.
>
> 2) `ckformat` verifies that the original image was is a 6505 format, and the image I saved out is a 6504 format.
>
> 3) Transferred it over to my G4, and it very much runs!  This particular VM was built with VMMaker 4.4.12.
>
> Amazing!  What an effective solution :)   Stepping stone-by-stone into the future...
>
>
>> - When as 6505 image that was saved from a little-endian machine is loaded into an oscog (Cog/Spur)
>> VM running on a big-endian platform, it is loaded in the saved format (presumeably little endian)
>>
>> So ... I am only guessing, but AFAIK we have never tried loading a 6505 image that was saved
>> on a little-endian machine into a Cog/Spur VM running on a big-endian platform. Quite possibly
>> in needs a word-reversal operation in that case.
>
> Very interesting.
>
> Are there any Cog/Spur VMs for big-endian platforms?
>
> Thanks a lot,
> Tim
>
>


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: SSAN: Stop, and See if Anyone Notices


Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

timrowledge
In reply to this post by Boris Shingarov
 


> On 2020-07-06, at 8:53 AM, Boris Shingarov <[hidden email]> wrote:
>
>
> On 7/5/20 6:05 PM, Eliot Miranda wrote:
>>  now is the best time ever to attempt a 64-bit or 32-bit PPC JIT
>
> ..."attempt"??  I have most of it working already.  I'll ping you when I get it to the same point where we reached with i386 and arm (i.e. booting the Reader and being able to evaluate "100 factorial").
>

Time for us to start designing an instruction set that properly supports dynamic translation and message sending etc. A Smalltalk that can run on trillions of cores!


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful random insult:- Has nothing to say, but delights in saying it.


Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Eliot Miranda-2
In reply to this post by Tim Johnson-2
 
Hi Tim,

On Sun, Jul 5, 2020 at 10:22 PM Tim Johnson <[hidden email]> wrote:
 
Hi Eliot,

I had a decent draft of this response, and my mail client pooped out when I hit send.  Trying again.  If this comes across as terse (I doubt it ;) ), please forgive.

On Jul 5, 2020, at 3:00 PM, Eliot Miranda wrote:

But I'm just hoping to verify here that the Float changes in image format 6505 could have fatal implications for a big-endian platform like PowerPC (the formerly-native format for the Floats ;) ) on a "vintage" stack/interpreter VM, and that perhaps that would have been /expected/ to be the last straw for it.  (A further question might be:  how big would the lift have been to keep it going?  Or was that work done and just would require some extra attention?)

I designed the scheme so that it would *not* have fatal implications on any platform :-)

This is a good point.  It helps me to improve my question:  

Q: Could my build environment (platform support code, non-oscog VMMaker sources, Squeak 4.5 hosted-VMMaker) be missing something necessary for the scheme to work as designed?

(This question & others summarized below.)

First let’s understand the representation.  A Float is a double-precision 64-bit ieee float represented as a two element word array.

Before the changes floats were always stored in the heap in big-endian format which meant:

- aFloat at: 1 accessed the most significant word of the float on both big & little median platforms
- on little endian platforms the vm therefore has to word reverse Float objects, perform arithmetic to obtain a new 64-big float, and word reverse it to store it back into a new Float

This unfairly penalizes little endian platforms.

Yes -- no doubt this was an important move.


My simple change was three-fold:
1. A bit in the image header specifies whether floats in the image are in big-endian format (bit unset) or in platform format (eg big endian on PPC, little endian on x86).

2. on load if required all Floats would be word reversed to ensure they are in platform order

^^ this is something I'm hoping to instrument or measure;  more on that below.


3. Float at: was modified so that on little endian platforms Float at: 1 answered the most significant word, just as it does on big endian platforms, hence hiding from the image the change in layout on little endian platforms and avoiding having to do any word reversal in arithmetic on any platforms


So what I did leveled the playing field.  How does this have fatal consequences for big endian platforms?

Well, I mean, that's so similar to the question I was trying to get across:  could it?  And, if so, under which conditions?

Since the scheme moves the cost into Float>>at:[put:] it would be I guess in serializing/deserializing floats etc.  But I doubt the difference would be measurable; it'll be swamped by other effects.
 

I definitely didn't mean to sound conclusive or be finger-pointy, and I'm sorry if it read that way.

And I don't mean to sound attacked either.  I was really just curious.  Had I not seem something that would affect big-endian platforms (Cog has yet to be compiled on a big endian platform)?
 

After getting the 6505 format image to load & the world to (mostly) display, it felt like a small victory.  But then seeing Float-related errors in SqueakDebug.log after force-quitting made me think this conversion process could be related, and I hoped to gather more info.

Yes.  The Interpreter VM may be missing the closure support, or correct implementation of the closure support.  But it may not.  I haven't used it. Dave would know what the state of play is.
 
Note that the word reversal on Float at: is for free because the primitive only takes 1&2 as valid indices, so the implementation is really cheap.  So in fact what I did was ensure that fp was as fast as possible on both platforms and (consciously) didn’t penalize either platform.

Does this explanation make the scheme seem more sensible now?

I really appreciate this explanation.  All I could find previously on this subject  was that thread I linked in my prior message, which didn't indicate which path was actually chosen.  This helps make it clear.

From my easy chair, your scheme looks brilliant and thorough.  I never meant to suggest otherwise, and am also totally underprepared and underqualified to do so.  The only thing not sensible here is that I'm trying to build & update a ten-year-old snapshot of the code for the Mac VM on an antiquated machine based on a niche architecture.  :) 

Sorry.  I should have tried to craft my tone netter.
 
Your & Dave's responses have helped me to hone my question(s):

Q:  Did this Float scheme for 6505 require a change to the VM in order to function properly?  If so, (similar to what I asked above) is it possible that the VM I am building is missing some part of that change?  (Whether that is something that exists in VMMaker-oscog but didn't make it to VMMaker, or whether it's some platform support code, or a changed version of a primitive...)  If not, I am probably barking up the wrong tree.

IIRC, yes.  There's a macro in sqwMemoryAccess.h to access floats that always did the reversal on little-endian platforms.  This isn't necessary with 6505.  So look at sqMemoryAccess.h in your platform sources and in the opensmalltalk-vm sources and that should help you decide what the right solution is.


Q:  How long might I expect the Float conversion at load (#2 above) to take, on a slower image running on a slower computer?  Milliseconds, seconds, minutes, or longer?  If I wanted to log it, trace it, or debug it, how could I find it?  (Dave helped with this, thanks:  ContextInterpreter>>normalizeFloatOrderingInImage )

Oh, nothing noticabl;e.  The conversion only happens when the endianness is changing.  So in normal use you shouldn't see any slow down.
 
Q:  If I spend the time to figure this out and do something about it, is there any way it could benefit anyone other than myself?  

Well any fix should go into the platform sources for the Interpreter VM.
 
That is the crux of my previous blathering about any hypothetical big-endian platforms of the future.  

I hope I haven't wasted your time or energy, Eliot.  Thanks a lot for your explanation.

Not at all.
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Eliot Miranda-2
In reply to this post by Boris Shingarov
 
Hi Boris,

On Mon, Jul 6, 2020 at 8:53 AM Boris Shingarov <[hidden email]> wrote:
 

On 7/5/20 6:05 PM, Eliot Miranda wrote:
>   
> now is the best time ever to attempt a 64-bit or 32-bit PPC JIT

..."attempt"??  I have most of it working already.  I'll ping you when I
get it to the same point where we reached with i386 and arm (i.e.
booting the Reader and being able to evaluate "100 factorial").

Wonderful news!!  Why haven;t you committed anything so we can see your work in progress??  Please, I'd love to see it.
 
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Jecel Assumpcao Jr
In reply to this post by timrowledge
 
Tim Rowledge wrote on Mon, 6 Jul 2020 10:34:31 -0700
> Time for us to start designing an instruction set that
> properly supports dynamic translation and message
> sending etc. A Smalltalk that can run on trillions of cores!

It is work in progress...

-- Jecel
Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Tim Johnson-2
In reply to this post by Eliot Miranda-2
 

On Jul 6, 2020, at 12:27 PM, Eliot Miranda wrote:

On Mon, Jul 6, 2020 at 8:53 AM Boris Shingarov <[hidden email]> wrote:
 

On 7/5/20 6:05 PM, Eliot Miranda wrote:
>   
> now is the best time ever to attempt a 64-bit or 32-bit PPC JIT

..."attempt"??  I have most of it working already.  I'll ping you when I
get it to the same point where we reached with i386 and arm (i.e.
booting the Reader and being able to evaluate "100 factorial").

Wonderful news!!  Why haven;t you committed anything so we can see your work in progress??  Please, I'd love to see it.

+1 Me too!

Happy to test & experiment on various systems I have here.  They're all old G3, G4, G5s though:  no Raptor Talos II or Blackbird POWER ISA workstations in the apartment (yet?).

Thanks,
Tim


Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

David T. Lewis
In reply to this post by Tim Johnson-2
 
Hi Tim,

On Sun, Jul 05, 2020 at 10:54:14PM -0700, Tim Johnson wrote:

>
> Hi Dave,
>
> On Jul 5, 2020, at 5:48 PM, David T. Lewis wrote:
>
> >To add some detail:
> >
> >- In a 6504 image, Floats are stored in big-endian format (as Eliot  
> >explains above) regardless of
> > the endianness of the current platform
> >
> >- In a 6505 image, Floats are stored in the native platform word  
> >ordering for the current platform
> >
> >- When a 6505 image is loaded into a classic interpreter VM the  
> >floats are converted big-endian
> > format in ContextInterpreter>>normalizeFloatOrderingInImage which  
> >effectively turns it back
> > into a 6504 image
>
> Thank you.  This helps me know where to possibly dig into the  
> conversion process.
>
> Also, I wanted to try this.  So I just did:
>
> 1) I loaded Squeak4.6-13700 (one of the alpha releases) into the  
> 4.10.2-2614 VM on my Raspberry Pi, and then saved it out.
>
> 2) `ckformat` verifies that the original image was is a 6505 format,  
> and the image I saved out is a 6504 format.
>
> 3) Transferred it over to my G4, and it very much runs!  This  
> particular VM was built with VMMaker 4.4.12.
>
> Amazing!  What an effective solution :)   Stepping stone-by-stone into  
> the future...
>

By way of a belated follow up - The first version of VMMaker that provided
support for loading 6505 images in a classic interpreter VM was VMMaker-dtl.190
(VMMaker versionString 4.3.4) from October 6, 2010. There were later
changes related to refactoring, but anything from 4.3.4 onwards should
successfully load and run V3 images saved from Cog.

Prior to your efforts, this had never been tested on a big-endian
machine, so I could not guarantee that it worked. But from your note
above, it sounds like it did work when you tried it (good!).

To summarize - the basic logic is that if the image file being loaded is
6505 and if the currently running VM is on a little-endian machine, then
swap words in all Float objects before entering the interpreter (i.e.
while loading the image).

>
> >- When as 6505 image that was saved from a little-endian machine is  
> >loaded into an oscog (Cog/Spur)
> > VM running on a big-endian platform, it is loaded in the saved  
> >format (presumeably little endian)
> >
> >So ... I am only guessing, but AFAIK we have never tried loading a  
> >6505 image that was saved
> >on a little-endian machine into a Cog/Spur VM running on a big-
> >endian platform. Quite possibly
> >in needs a word-reversal operation in that case.
>
> Very interesting.
>
> Are there any Cog/Spur VMs for big-endian platforms?
>

As far as I know, there are no big-ending Cog/Spur VMs. If somebody
creates one, we would very likely need to update the word-swapping
logic that I described above, because it would be saving the Float
data in native order, which would not be big-endian. Something for
future consideration I guess. The Cog/Spur VM would also need to
handle word swapping if platform endianness changed.

Dave

 
Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Eliot Miranda-2
 
Hi Dave,

> On Jul 7, 2020, at 10:59 AM, David T. Lewis <[hidden email]> wrote:
>
> 
> Hi Tim,
>
>> On Sun, Jul 05, 2020 at 10:54:14PM -0700, Tim Johnson wrote:
>>
>> Hi Dave,
>>
>>> On Jul 5, 2020, at 5:48 PM, David T. Lewis wrote:
>>>
>>> To add some detail:
>>>
>>> - In a 6504 image, Floats are stored in big-endian format (as Eliot  
>>> explains above) regardless of
>>> the endianness of the current platform
>>>
>>> - In a 6505 image, Floats are stored in the native platform word  
>>> ordering for the current platform
>>>
>>> - When a 6505 image is loaded into a classic interpreter VM the  
>>> floats are converted big-endian
>>> format in ContextInterpreter>>normalizeFloatOrderingInImage which  
>>> effectively turns it back
>>> into a 6504 image
>>
>> Thank you.  This helps me know where to possibly dig into the  
>> conversion process.
>>
>> Also, I wanted to try this.  So I just did:
>>
>> 1) I loaded Squeak4.6-13700 (one of the alpha releases) into the  
>> 4.10.2-2614 VM on my Raspberry Pi, and then saved it out.
>>
>> 2) `ckformat` verifies that the original image was is a 6505 format,  
>> and the image I saved out is a 6504 format.
>>
>> 3) Transferred it over to my G4, and it very much runs!  This  
>> particular VM was built with VMMaker 4.4.12.
>>
>> Amazing!  What an effective solution :)   Stepping stone-by-stone into  
>> the future...
>>
>
> By way of a belated follow up - The first version of VMMaker that provided
> support for loading 6505 images in a classic interpreter VM was VMMaker-dtl.190
> (VMMaker versionString 4.3.4) from October 6, 2010. There were later
> changes related to refactoring, but anything from 4.3.4 onwards should
> successfully load and run V3 images saved from Cog.
>
> Prior to your efforts, this had never been tested on a big-endian
> machine, so I could not guarantee that it worked. But from your note
> above, it sounds like it did work when you tried it (good!).
>
> To summarize - the basic logic is that if the image file being loaded is
> 6505 and if the currently running VM is on a little-endian machine, then
> swap words in all Float objects before entering the interpreter (i.e.
> while loading the image).
>
>>
>>> - When as 6505 image that was saved from a little-endian machine is  
>>> loaded into an oscog (Cog/Spur)
>>> VM running on a big-endian platform, it is loaded in the saved  
>>> format (presumeably little endian)
>>>
>>> So ... I am only guessing, but AFAIK we have never tried loading a  
>>> 6505 image that was saved
>>> on a little-endian machine into a Cog/Spur VM running on a big-
>>> endian platform. Quite possibly
>>> in needs a word-reversal operation in that case.
>>
>> Very interesting.
>>
>> Are there any Cog/Spur VMs for big-endian platforms?
>>
>
> As far as I know, there are no big-ending Cog/Spur VMs. If somebody
> creates one, we would very likely need to update the word-swapping
> logic that I described above, because it would be saving the Float
> data in native order, which would not be big-endian. Something for
> future consideration I guess. The Cog/Spur VM would also need to
> handle word swapping if platform endianness changed.

I’ve not removed any of the load-time byte swapping logic.  What’s missing is correct handling of big-end is ness in Spur and the JIT.  The ones I noticed as I wrote code are makes self flag: #endianness.  There are a few crucial ones to deal with.

Boris will likely have encountered these as he’s working on the POC code generator, unless, sensibly he decides to use it in little-endian mode while developing the code generator.

The issues are not hard to fix.  The issue is best how to select the right code, use cppIf: or ifTrue: or endian-specific subclasses.

> Dave


_,,,^..^,,,_ (phone)
Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

David T. Lewis
 
Hi Eliot,

On Tue, Jul 07, 2020 at 11:59:57AM -0700, Eliot Miranda wrote:

>  
> Hi Dave,
>
> > On Jul 7, 2020, at 10:59 AM, David T. Lewis <[hidden email]> wrote:
> >
> > ???
> > Hi Tim,
> >
> >> On Sun, Jul 05, 2020 at 10:54:14PM -0700, Tim Johnson wrote:
> >>
> >> Hi Dave,
> >>
> >>> On Jul 5, 2020, at 5:48 PM, David T. Lewis wrote:
> >>>
> >>> To add some detail:
> >>>
> >>> - In a 6504 image, Floats are stored in big-endian format (as Eliot  
> >>> explains above) regardless of
> >>> the endianness of the current platform
> >>>
> >>> - In a 6505 image, Floats are stored in the native platform word  
> >>> ordering for the current platform
> >>>
> >>> - When a 6505 image is loaded into a classic interpreter VM the  
> >>> floats are converted big-endian
> >>> format in ContextInterpreter>>normalizeFloatOrderingInImage which  
> >>> effectively turns it back
> >>> into a 6504 image
> >>
> >> Thank you.  This helps me know where to possibly dig into the  
> >> conversion process.
> >>
> >> Also, I wanted to try this.  So I just did:
> >>
> >> 1) I loaded Squeak4.6-13700 (one of the alpha releases) into the  
> >> 4.10.2-2614 VM on my Raspberry Pi, and then saved it out.
> >>
> >> 2) `ckformat` verifies that the original image was is a 6505 format,  
> >> and the image I saved out is a 6504 format.
> >>
> >> 3) Transferred it over to my G4, and it very much runs!  This  
> >> particular VM was built with VMMaker 4.4.12.
> >>
> >> Amazing!  What an effective solution :)   Stepping stone-by-stone into  
> >> the future...
> >>
> >
> > By way of a belated follow up - The first version of VMMaker that provided
> > support for loading 6505 images in a classic interpreter VM was VMMaker-dtl.190
> > (VMMaker versionString 4.3.4) from October 6, 2010. There were later
> > changes related to refactoring, but anything from 4.3.4 onwards should
> > successfully load and run V3 images saved from Cog.
> >
> > Prior to your efforts, this had never been tested on a big-endian
> > machine, so I could not guarantee that it worked. But from your note
> > above, it sounds like it did work when you tried it (good!).
> >
> > To summarize - the basic logic is that if the image file being loaded is
> > 6505 and if the currently running VM is on a little-endian machine, then
> > swap words in all Float objects before entering the interpreter (i.e.
> > while loading the image).
> >
> >>
> >>> - When as 6505 image that was saved from a little-endian machine is  
> >>> loaded into an oscog (Cog/Spur)
> >>> VM running on a big-endian platform, it is loaded in the saved  
> >>> format (presumeably little endian)
> >>>
> >>> So ... I am only guessing, but AFAIK we have never tried loading a  
> >>> 6505 image that was saved
> >>> on a little-endian machine into a Cog/Spur VM running on a big-
> >>> endian platform. Quite possibly
> >>> in needs a word-reversal operation in that case.
> >>
> >> Very interesting.
> >>
> >> Are there any Cog/Spur VMs for big-endian platforms?
> >>
> >
> > As far as I know, there are no big-ending Cog/Spur VMs. If somebody
> > creates one, we would very likely need to update the word-swapping
> > logic that I described above, because it would be saving the Float
> > data in native order, which would not be big-endian. Something for
> > future consideration I guess. The Cog/Spur VM would also need to
> > handle word swapping if platform endianness changed.
>
> I???ve not removed any of the load-time byte swapping logic.  What???s missing is correct handling of big-end is ness in Spur and the JIT.  The ones I noticed as I wrote code are makes self flag: #endianness.  There are a few crucial ones to deal with.
>

Indeed, I understand that doing the JIT for big-endian is not a trivial
thing, especially if there is no realistic market for it.

> Boris will likely have encountered these as he???s working on the POC code generator, unless, sensibly he decides to use it in little-endian mode while developing the code generator.
>
> The issues are not hard to fix.  The issue is best how to select the right code, use cppIf: or ifTrue: or endian-specific subclasses.
>

With respect to the Float order issue we were discussing, I think that
the only thing that needs to be addressed is to look at the endianness of
the saved image, and do the word swap if endianness is different in the
current VM. It's a small bit of logic, and it would be very similar to
the method that I put into interpreter VM, except that there is no need
to be concerned about converting back to 6504 format, just do the word
swap if current endianness is different.

All image formats after 6504 use native float ordering, so that logic
would apply everywhere, and it just needs to be done during image load.
The word swapping is fast, although I can't say I have ever actually
measured it. But even if it was slow it would be a very rare use case,
so nothing to worry about.

Dave
 
Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Boris Shingarov
In reply to this post by Eliot Miranda-2
 

Hi Eliot,

Wonderful news!!  Why haven;t you committed anything so we can see your work in progress??  Please, I'd love to see it.
There isn't really anything that's not public.  (Well ok, there are some exceptions -- some early-prototype experiments re: deriving Smalltalk from Sail, stuff that's definitely too rough yet to be useful to anyone but myself).  The problem is not publishing, the problem is integration.  We have all the pieces working but some only work in Squeak, some in Pharo, some in Smalltalk/X.  Multiply that by the dimension that some are applied to OpenSmalltalk VM and some to Bee DMR.  Sure I could bring a piece from (say) Smalltalk/X into Squeak and commit it on the squeaksource repo, but what's the point, other than to create confusion and frustration for others?  At least the stuff I committed here (VMMaker-over-ULD instead of Alien) actually works IN SQUEAK: so if someone is curious to try it, they can start from where it's reasonable to expect that they will be able to understand what the code does.  (And yes, I am always here available to answer questions).  If they are curious to see how I infer an assembler and disassembler from PDL, they can head over to the Pharo-ArchC repo and start from watching that code run in their Pharo image (because it's been working well for the past 5 years).  But to expect that someone will load this unfamiliar bunch of code into Squeak and struggle to make it work, I think this would be a totally unreasonable, simply idiotic, expectation.

There are reasons why we ended up in this state of fragmentation.  One is that I am interested in things like proving correctness of self-modifying code, or like auto-inferring a RISC-V toolchain from its Sail2 spec, much more than in Squeak or Pharo per se.  So if I know it's going to take me a week to build a proof-of-concept of something in Pharo vs a year in Smalltalk/X, I'll do that piece in Pharo no matter my preferences.  Unfortunately, there are many of these situations forcing me to work in non-preferred dialects.

At this point in time, though, we need to sit down and brainstorm how to change this, because the components have now matured enough to reach diminishing returns from working on them separately.

I'd like to somehow integrate to at least allow for short-term progress (now we can't even exchange code easily -- Tonel and whatnot).

By short-term, I mean, probably, start from plugging the target-agnostic assemblers: things like e.g. CogARMCompiler>>faddd:with:, this kind of hand-written source just doesn't belong in the source-codebase: this knowledge is trivially inferred from the machine-readable CPU spec.  (And yes, I know how to slang the inferred form, although I have not actually done it yet).  This should simplify the CogCompilers by an order of magnitude.

The second thing I want to do sooner than later is erase the black-and-white division between simulation and production: for example in Cogit>>compileCallFor:..., either ALL of them go the cCode: [aRoutine asUnsignedInteger] way, or ALL of them go the inSmalltalk: [self simulatedTrampolineFor: aRoutine] way.  John Sarkela already taught us how to do it granularly, i.e. one simulation run can have some runtime calls executed as native CPU code, and some others be simulated via fakeAddress.  That's not even hard, but what stands in the way is that we have two different address space layouts.  Obviously these will need to become unified.


You're doing 64-bit, 32-bit or both?  What are the significant PPC platforms these days?
There are basically two disjoint PPC worlds these days, Embedded/IOT (NXP/Freescale) and Server (IBM).

Right now, only PPC32 is actively moving forward.  I am testing in gem5 simulation, as well as on two different dev boards from Freescale (with e300 and e500) and on the 440 block within XC5VFX (those are past-end-of-life by Xilinx, but are *excellent* debug targets (esp. when semihosting like in OpenSmalltalkVM) and very inexpensive because they are "old tech").

PPC64 is kind of in a weird state.  In gem5 there are out-of-tree patches for PPC64 support, I've tried those and found it frustrating to, when something doesn't work, never be sure if it's their bug or mine.

On the hardware side, there are some open-source implementations, one of them from IBM, but it's not quite there yet: every time I configure an FPGA with it and try running my JIT, I bump into one issue or another.  I try to participate in resolving them, but at this stage it's not a very productive tool for JIT testing.  (It's an immensely fun mode to run a Smalltalk VM in, though: no-OS, with MMU switched off, so there is nothing to detect a fake address, these are MMIO to a purpose-built register pulling an interrupt line on access... fun toy but I hope it will mature soon).

- bgs


On 7/6/20 3:27 PM, Eliot Miranda wrote:
 

Hi Boris,

On Mon, Jul 6, 2020 at 8:53 AM Boris Shingarov <[hidden email]> wrote:
 

On 7/5/20 6:05 PM, Eliot Miranda wrote:
>   
> now is the best time ever to attempt a 64-bit or 32-bit PPC JIT

..."attempt"??  I have most of it working already.  I'll ping you when I
get it to the same point where we reached with i386 and arm (i.e.
booting the Reader and being able to evaluate "100 factorial").

Wonderful news!!  Why haven;t you committed anything so we can see your work in progress??  Please, I'd love to see it.
 
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

David T. Lewis
In reply to this post by Tim Johnson-2
 
On Sun, Jul 05, 2020 at 12:52:24PM -0700, Tim Johnson wrote:
>
> FWIW, I did try the old Darwin/PPC port of 4.10.2 [3]  with both 4.6  
> and David T Lewis's "Squeak 5.2-in-V3 format" image (which is no  
> longer available on the 'net!)

Following up on a detail from an old thread, the image referenced above
is now avaiable again:

  http://files.squeak.org/various_images/SqueakV3Images/

Dave
Reply | Threaded
Open this post in threaded view
|

Re: Probably pointless endianness question re: Float changes circa 4.6

Edgar De Cleene
 
Just download
Image
-----
/Users/edgardecleene/Documents/AtlantisSqueak/5.3/Squeak5.3-V3-19432/squeak-
5.3-v3.image
Squeak5.3rc3
latest update: #19432
Current Change Set: HomeProject
Image format 6505 (32 bit)

Virtual Machine
---------------
/Applications/Cog 3312.app/Contents/MacOS/Squeak
Croquet Closure Cog VM [CoInterpreterPrimitives VMMaker.oscog-eem.1207]
Squeak Cog 4.0.3312
Mac OS X built on Apr 17 2015 20:21:10 UTC Compiler: 4.2.1 (Apple Inc. build
5666) (dot 3)
platform sources revision VM: r3312
http://www.squeakvm.org/svn/squeak/branches/Cog Date: 2015-04-17 12:34:25
-0700 Plugins: r3275
http://squeakvm.org/svn/squeak/trunk/platforms/Cross/plugins
CoInterpreter VMMaker.oscog-eem.1207 uuid:
4a338677-029f-42ac-b38d-2bc5ecc213f1 Apr 17 2015
StackToRegisterMappingCogit VMMaker.oscog-eem.1207 uuid:
4a338677-029f-42ac-b38d-2bc5ecc213f1 Apr 17 2015

To Build A Similar Virtual Machine
----------------------------------
Visit https://github.com/OpenSmalltalk/opensmalltalk-vm; follow the
"Clone or download" instructions, then read the top-level README.md
and HowToBuild files in the top-level build directory for your
platform(s), build.macos64x64/HowToBuild, build.win32x86/HowToBuild, etc.

But raise some debuggers.
I like chase then and fix if possible.
How you prefer manage , send directly to you or to list

Edgar
@morplenauta