Re: [squeak-dev] Apple starting to alert users that it will end 32-bit app support on the Mac

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Apple starting to alert users that it will end 32-bit app support on the Mac

Eliot Miranda-2
 
Hi Bert,


On Apr 12, 2018, at 6:03 AM, Bert Freudenberg <[hidden email]> wrote:

If that's indeed the case we need to make a 64 bit VM for 32 bit images. I think this should be just a matter of picking the right compiler options.

How so?  AFAIA it is not.  It means, for example, changing every memory access for an instance variable from 64 to 32 bits.  It is non-trivial.

What /is/ trivial is detecting that an image's word size is unsupported on a platform and offering to convert it, a process that takes less than a minute for typical images.

If we go this route we would include a headless 64-bit image that converts 32-bit images to 64-bits on 64-bit only systems, and a headless 32-bit image that converts 64-bit images to 32-bits on 32-bit only systems.

IIRC 64 to 32 bit conversion needs to be written, but I think it is trivial compared to building a 64-bit vm for 32-bit images.

Of course this applies only to Spur images.  We don't have a way of automatically converting V3 images to Spur.  Yo do that going forward we may have to depend on OS virtual machines, but somehow I can't see an ARMv8 Mac supporting x86 VMs like Parallels.  This could be the end of an interoperability era.


Otherwise we won't have any way of opening regular 32 bit images on Macs.

- Bert -

On 12 April 2018 at 14:47, Fabio Niephaus <[hidden email]> wrote:
Hi all,

According to [1], Apple has started to alert users when they open a 32-bit app on the Mac and I've seen such a dialog today myself when opening a 32-bit Squeak image.

It looks like it is time to make 64-bit the default, maybe with the next release? The Pharo community is already doing that for quite some time, so I'm wondering if there is any reason not to do the same?

Best,
Fabio

[1] https://techcrunch.com/2018/04/11/apple-starting-to-alert-users-that-it-will-end-32-bit-app-support/





Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Apple starting to alert users that it will end 32-bit app support on the Mac

Clément Bera-4
 
Hi all, 

question inlined...

On Thu, Apr 12, 2018 at 4:05 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,


On Apr 12, 2018, at 6:03 AM, Bert Freudenberg <[hidden email]> wrote:

If that's indeed the case we need to make a 64 bit VM for 32 bit images. I think this should be just a matter of picking the right compiler options.

How so?  AFAIA it is not.  It means, for example, changing every memory access for an instance variable from 64 to 32 bits.  It is non-trivial.

What /is/ trivial is detecting that an image's word size is unsupported on a platform and offering to convert it, a process that takes less than a minute for typical images.
 
Why does this process take a full minute ? Could we do that in 1 second ? Is it only because of SmallFloat ?

I mean I could dream of a world where 64-bits image under 2Gb are saved as 32 bits images to be more compact on disk. Then at start-up, the 32 bits image is converted into a 64 bits image if started on a 64 bits VM, or not if started on a 32 bits VM. We could also save image under 65k as 16 bits images, and convert them back as start-up :-). Obviously that would work only if the conversion is fast, but I think it could be with tricks to handle SmallFloats.

Note that the code optimized by Sista is currently dependent on word size though due to overflow computation, but I guess we'll solve this problem later.

 

If we go this route we would include a headless 64-bit image that converts 32-bit images to 64-bits on 64-bit only systems, and a headless 32-bit image that converts 64-bit images to 32-bits on 32-bit only systems.

IIRC 64 to 32 bit conversion needs to be written, but I think it is trivial compared to building a 64-bit vm for 32-bit images.

Of course this applies only to Spur images.  We don't have a way of automatically converting V3 images to Spur.  Yo do that going forward we may have to depend on OS virtual machines, but somehow I can't see an ARMv8 Mac supporting x86 VMs like Parallels.  This could be the end of an interoperability era.


Otherwise we won't have any way of opening regular 32 bit images on Macs.

- Bert -

On 12 April 2018 at 14:47, Fabio Niephaus <[hidden email]> wrote:
Hi all,

According to [1], Apple has started to alert users when they open a 32-bit app on the Mac and I've seen such a dialog today myself when opening a 32-bit Squeak image.

It looks like it is time to make 64-bit the default, maybe with the next release? The Pharo community is already doing that for quite some time, so I'm wondering if there is any reason not to do the same?

Best,
Fabio

[1] https://techcrunch.com/2018/04/11/apple-starting-to-alert-users-that-it-will-end-32-bit-app-support/









--
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Apple starting to alert users that it will end 32-bit app support on the Mac

Eliot Miranda-2
 
Hi Clément,

On Apr 12, 2018, at 8:29 AM, Clément Bera <[hidden email]> wrote:

Hi all, 

question inlined...

On Thu, Apr 12, 2018 at 4:05 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,


On Apr 12, 2018, at 6:03 AM, Bert Freudenberg <[hidden email]> wrote:

If that's indeed the case we need to make a 64 bit VM for 32 bit images. I think this should be just a matter of picking the right compiler options.

How so?  AFAIA it is not.  It means, for example, changing every memory access for an instance variable from 64 to 32 bits.  It is non-trivial.

What /is/ trivial is detecting that an image's word size is unsupported on a platform and offering to convert it, a process that takes less than a minute for typical images.
 
Why does this process take a full minute ? Could we do that in 1 second ? Is it only because of SmallFloat ?

The classes affected and the reasons are
- SmallInteger <-> LargePositiveInteger, LargeNegativeInteger, since in 32-bits SmallInteger minVal & SmallInteger minVal encompass a 31-bit 2's compliment range, and in 64-bits, a 61-bit range.
- SmallFloat64 <-> BoxedFloat64, since in 32-bits there are no immediate floats and in 64-bits floats whose exponent fits in 8 bits are represented by SmallFloat64
- Context & BlockClosure since pcs must be offset by 4 bytes per literal as the CompiledCode instance's literals will be twice or half as wide
- any and all bits collections (ByteSymbol, ByteArray, DoubleWordArray et al) whose formats change because the number of odd elements (the number of elements to subtract from the product of number of slots and slot size) is different in 32 and 64 bits

So the process in going from 32 to 64 bits is
- load the 32-bit image into a 32-bit simulator
- clone all the objects that are preserved in a 64-bit simulator, creating a map
- pass through the 32-bit objects, updating the corresponding slot in the 64-bit simulator with the object at the map
- save the 64-bit image

And the process in going in the reverse direction would be to
- load the 64-bit image into a 64-bit simulator
- clone all the objects in a 32-bit simulator, creating a map
- pass through the 64-bit objects, updating the corresponding slot in the 32-bit simulator with the object at the map, or otherwise allocating a boxed object for the 64-bit immediate that has no equivalent immediate in 32-bits
- save the 32-bit image

I just timed converting a 34 Mb 32-bit Squeak 5.1 image to 64-bits on my 2.3Ghz Core i7 Mac Mini in a 63-bit Spur image.  It took 20.6 seconds to convert.  One would probably have to add another 0.5 seconds for start up costs.  And Pi is around 5x slower than Mac.  So conversion times would be anywhere from 15 seconds to 1 1/2 minutes for standard images, scaling more or less linearly with larger images.

I certainly don't see any way of doing this in orders of magnitude faster.  But that's my limitation ;-)


I mean I could dream of a world where 64-bits image under 2Gb are saved as 32 bits images to be more compact on disk. Then at start-up, the 32 bits image is converted into a 64 bits image if started on a 64 bits VM, or not if started on a 32 bits VM. We could also save image under 65k as 16 bits images, and convert them back as start-up :-). Obviously that would work only if the conversion is fast, but I think it could be with tricks to handle SmallFloats.

I don't think two passes (allocation & mapping) can be avoided because pointer objects, compiled code, and bits objects all map differently, so there's no way to compute the position (and hence the oop) of the corresponding object independently of all other objects in the heap.


Note that the code optimized by Sista is currently dependent on word size though due to overflow computation, but I guess we'll solve this problem later.

 

If we go this route we would include a headless 64-bit image that converts 32-bit images to 64-bits on 64-bit only systems, and a headless 32-bit image that converts 64-bit images to 32-bits on 32-bit only systems.

IIRC 64 to 32 bit conversion needs to be written, but I think it is trivial compared to building a 64-bit vm for 32-bit images.

Of course this applies only to Spur images.  We don't have a way of automatically converting V3 images to Spur.  Yo do that going forward we may have to depend on OS virtual machines, but somehow I can't see an ARMv8 Mac supporting x86 VMs like Parallels.  This could be the end of an interoperability era.


Otherwise we won't have any way of opening regular 32 bit images on Macs.

- Bert -

On 12 April 2018 at 14:47, Fabio Niephaus <[hidden email]> wrote:
Hi all,

According to [1], Apple has started to alert users when they open a 32-bit app on the Mac and I've seen such a dialog today myself when opening a 32-bit Squeak image.

It looks like it is time to make 64-bit the default, maybe with the next release? The Pharo community is already doing that for quite some time, so I'm wondering if there is any reason not to do the same?

Best,
Fabio

[1] https://techcrunch.com/2018/04/11/apple-starting-to-alert-users-that-it-will-end-32-bit-app-support/









--
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Apple starting to alert users that it will end 32-bit app support on the Mac

Eliot Miranda-2
 
Hi Clément,

On Thu, Apr 12, 2018 at 9:02 AM, Eliot Miranda <[hidden email]> wrote:
Hi Clément,

On Apr 12, 2018, at 8:29 AM, Clément Bera <[hidden email]> wrote:

Hi all, 

question inlined...

On Thu, Apr 12, 2018 at 4:05 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,


On Apr 12, 2018, at 6:03 AM, Bert Freudenberg <[hidden email]> wrote:

If that's indeed the case we need to make a 64 bit VM for 32 bit images. I think this should be just a matter of picking the right compiler options.

How so?  AFAIA it is not.  It means, for example, changing every memory access for an instance variable from 64 to 32 bits.  It is non-trivial.

What /is/ trivial is detecting that an image's word size is unsupported on a platform and offering to convert it, a process that takes less than a minute for typical images.
 
Why does this process take a full minute ? Could we do that in 1 second ? Is it only because of SmallFloat ?

The classes affected and the reasons are
- SmallInteger <-> LargePositiveInteger, LargeNegativeInteger, since in 32-bits SmallInteger minVal & SmallInteger minVal encompass a 31-bit 2's compliment range, and in 64-bits, a 61-bit range.
- SmallFloat64 <-> BoxedFloat64, since in 32-bits there are no immediate floats and in 64-bits floats whose exponent fits in 8 bits are represented by SmallFloat64
- Context & BlockClosure since pcs must be offset by 4 bytes per literal as the CompiledCode instance's literals will be twice or half as wide
- any and all bits collections (ByteSymbol, ByteArray, DoubleWordArray et al) whose formats change because the number of odd elements (the number of elements to subtract from the product of number of slots and slot size) is different in 32 and 64 bits

So the process in going from 32 to 64 bits is
- load the 32-bit image into a 32-bit simulator
- clone all the objects that are preserved in a 64-bit simulator, creating a map
- pass through the 32-bit objects, updating the corresponding slot in the 64-bit simulator with the object at the map
- save the 64-bit image

And the process in going in the reverse direction would be to
- load the 64-bit image into a 64-bit simulator
- clone all the objects in a 32-bit simulator, creating a map
- pass through the 64-bit objects, updating the corresponding slot in the 32-bit simulator with the object at the map, or otherwise allocating a boxed object for the 64-bit immediate that has no equivalent immediate in 32-bits
- save the 32-bit image

I just timed converting a 34 Mb 32-bit Squeak 5.1 image to 64-bits on my 2.3Ghz Core i7 Mac Mini in a 63-bit Spur image.  It took 20.6 seconds to convert.  One would probably have to add another 0.5 seconds for start up costs.  And Pi is around 5x slower than Mac.  So conversion times would be anywhere from 15 seconds to 1 1/2 minutes for standard images, scaling more or less linearly with larger images.

I certainly don't see any way of doing this in orders of magnitude faster.  But that's my limitation ;-)

But this computation is ideal for Scorch/Sista right?  It's a series of loops.  It only ever does the same thing because input images are extremely similar.  The image could be trained on a suitably general input set before being deployed.  It would be very interesting to see how much faster Scorch/Sista manages to make image mapping.

I mean I could dream of a world where 64-bits image under 2Gb are saved as 32 bits images to be more compact on disk. Then at start-up, the 32 bits image is converted into a 64 bits image if started on a 64 bits VM, or not if started on a 32 bits VM. We could also save image under 65k as 16 bits images, and convert them back as start-up :-). Obviously that would work only if the conversion is fast, but I think it could be with tricks to handle SmallFloats.

I don't think two passes (allocation & mapping) can be avoided because pointer objects, compiled code, and bits objects all map differently, so there's no way to compute the position (and hence the oop) of the corresponding object independently of all other objects in the heap.


Note that the code optimized by Sista is currently dependent on word size though due to overflow computation, but I guess we'll solve this problem later.

 

If we go this route we would include a headless 64-bit image that converts 32-bit images to 64-bits on 64-bit only systems, and a headless 32-bit image that converts 64-bit images to 32-bits on 32-bit only systems.

IIRC 64 to 32 bit conversion needs to be written, but I think it is trivial compared to building a 64-bit vm for 32-bit images.

Of course this applies only to Spur images.  We don't have a way of automatically converting V3 images to Spur.  Yo do that going forward we may have to depend on OS virtual machines, but somehow I can't see an ARMv8 Mac supporting x86 VMs like Parallels.  This could be the end of an interoperability era.


Otherwise we won't have any way of opening regular 32 bit images on Macs.

- Bert -

On 12 April 2018 at 14:47, Fabio Niephaus <[hidden email]> wrote:
Hi all,

According to [1], Apple has started to alert users when they open a 32-bit app on the Mac and I've seen such a dialog today myself when opening a 32-bit Squeak image.

It looks like it is time to make 64-bit the default, maybe with the next release? The Pharo community is already doing that for quite some time, so I'm wondering if there is any reason not to do the same?

Best,
Fabio

[1] https://techcrunch.com/2018/04/11/apple-starting-to-alert-users-that-it-will-end-32-bit-app-support/
--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Apple starting to alert users that it will end 32-bit app support on the Mac

timrowledge
 


> On 12-04-2018, at 9:45 AM, Eliot Miranda <[hidden email]> wrote:
>
> So the process in going from 32 to 64 bits is
> - load the 32-bit image into a 32-bit simulator
> - clone all the objects that are preserved in a 64-bit simulator, creating a map
> - pass through the 32-bit objects, updating the corresponding slot in the 64-bit simulator with the object at the map
> - save the 64-bit image
>
> And the process in going in the reverse direction would be to
> - load the 64-bit image into a 64-bit simulator
> - clone all the objects in a 32-bit simulator, creating a map
> - pass through the 64-bit objects, updating the corresponding slot in the 32-bit simulator with the object at the map, or otherwise allocating a boxed object for the 64-bit immediate that has no equivalent immediate in 32-bits
> - save the 32-bit image
>
> I just timed converting a 34 Mb 32-bit Squeak 5.1 image to 64-bits on my 2.3Ghz Core i7 Mac Mini in a 63-bit Spur image.  It took 20.6 seconds to convert.  One would probably have to add another 0.5 seconds for start up costs.  And Pi is around 5x slower than Mac.  So conversion times would be anywhere from 15 seconds to 1 1/2 minutes for standard images, scaling more or less linearly with larger images.
>
> I certainly don't see any way of doing this in orders of magnitude faster.  But that's my limitation ;-)
>
> But this computation is ideal for Scorch/Sista right?  It's a series of loops.  It only ever does the same thing because input images are extremely similar.  The image could be trained on a suitably general input set before being deployed.  It would be very interesting to see how much faster Scorch/Sista manages to make image mapping.

So I guess we would have a 'native' utility image included with the normal system, that is set up to do this conversion from 'other' to 'native' as fast as possible, then quit and start the now-native image? The VM would be unchanged for normal usage except just possibly returning an error code if a wrong format image is specified and then the startup shell script(s) could do the run-converter-image pass. Faintly possible this would be a tad quicker than a specific test-first pass. Ah, I imagine we might be clever enough to detect the wrong format in the VM and then just load the converter image anyway, since we'd want to know where it lives and no other VM setup would have been done. The only extra trick would be storing the name of the converted image to run after the processing.

Sure, on *nix it is simple enough to run a format-checker, run the converter if needed (remembering an UI to ask the user if they actually want to do that) and then run the real VM/image. Doing it under our own control seems more fun.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
From C:\*.* to shining C:\*.*


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Apple starting to alert users that it will end 32-bit app support on the Mac

Eliot Miranda-2
 
Hi Tim,

On Thu, Apr 12, 2018 at 11:50 AM, tim Rowledge <[hidden email]> wrote:



> On 12-04-2018, at 9:45 AM, Eliot Miranda <[hidden email]> wrote:
>
> So the process in going from 32 to 64 bits is
> - load the 32-bit image into a 32-bit simulator
> - clone all the objects that are preserved in a 64-bit simulator, creating a map
> - pass through the 32-bit objects, updating the corresponding slot in the 64-bit simulator with the object at the map
> - save the 64-bit image
>
> And the process in going in the reverse direction would be to
> - load the 64-bit image into a 64-bit simulator
> - clone all the objects in a 32-bit simulator, creating a map
> - pass through the 64-bit objects, updating the corresponding slot in the 32-bit simulator with the object at the map, or otherwise allocating a boxed object for the 64-bit immediate that has no equivalent immediate in 32-bits
> - save the 32-bit image
>
> I just timed converting a 34 Mb 32-bit Squeak 5.1 image to 64-bits on my 2.3Ghz Core i7 Mac Mini in a 63-bit Spur image.  It took 20.6 seconds to convert.  One would probably have to add another 0.5 seconds for start up costs.  And Pi is around 5x slower than Mac.  So conversion times would be anywhere from 15 seconds to 1 1/2 minutes for standard images, scaling more or less linearly with larger images.
>
> I certainly don't see any way of doing this in orders of magnitude faster.  But that's my limitation ;-)
>
> But this computation is ideal for Scorch/Sista right?  It's a series of loops.  It only ever does the same thing because input images are extremely similar.  The image could be trained on a suitably general input set before being deployed.  It would be very interesting to see how much faster Scorch/Sista manages to make image mapping.

So I guess we would have a 'native' utility image included with the normal system, that is set up to do this conversion from 'other' to 'native' as fast as possible, then quit and start the now-native image? The VM would be unchanged for normal usage except just possibly returning an error code if a wrong format image is specified and then the startup shell script(s) could do the run-converter-image pass. Faintly possible this would be a tad quicker than a specific test-first pass. Ah, I imagine we might be clever enough to detect the wrong format in the VM and then just load the converter image anyway, since we'd want to know where it lives and no other VM setup would have been done. The only extra trick would be storing the name of the converted image to run after the processing.

That's right.  All this is trivial form the command line.  Prompt the user with "the image is X bits.  The VM is Y bits.  Do you want to convert?" form the launch script and do the dirty if asked.  From the app its a bit more difficult because we have to at least prompt the user in the GUI domain.  Ideally the converter would be one and the same image.  When run from the command line the converter would be launched headless and interrogate the user through stdio.  When run from the app it would use its own GUI and then relaunch.  If at all possible this should all be written in Smalltalk apart from the VM launching the converter image.  Some fragment of ckformat.c could be compiled into the VM to see if the image conversion was appropriate and to launch the converter image in the right mode.

Volunteers?  I'm happy to have a go at this when my decks are cleared, but that could be weeks away.

Sure, on *nix it is simple enough to run a format-checker, run the converter if needed (remembering an UI to ask the user if they actually want to do that) and then run the real VM/image. Doing it under our own control seems more fun.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
From C:\*.* to shining C:\*.*

_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Apple starting to alert users that it will end 32-bit app support on the Mac

timrowledge
 


> On 12-04-2018, at 2:02 PM, Eliot Miranda <[hidden email]> wrote:
>

> That's right.  All this is trivial form the command line.  Prompt the user with "the image is X bits.  The VM is Y bits.  Do you want to convert?" form the launch script and do the dirty if asked.  From the app its a bit more difficult because we have to at least prompt the user in the GUI domain.  Ideally the converter would be one and the same image.  When run from the command line the converter would be launched headless and interrogate the user through stdio.  When run from the app it would use its own GUI and then relaunch.  If at all possible this should all be written in Smalltalk apart from the VM launching the converter image.  Some fragment of ckformat.c could be compiled into the VM to see if the image conversion was appropriate and to launch the converter image in the right mode.

We already have the image format checking in checkImageVersionFromstartingAt(), though currently it only reports whether the bytes need swapping. We could extend it to return more values if the 32-64 conversion were needed. (Indeed, one might consider dropping the byteswap from the normal vm code and delegating that to a convertor as well, presuming it runs fast enough at some point)

At that point, in readImageFromFileHeapSizeStartingAt() we could then divert to open and load the known-location convertor image instead, which would be able to find the original image filename because we would have used recordFullPathForImageName() previously.

The probably more painful part is after the conversion is completed and the new image file created. We'd have to do whatever is required to clean up all allocations and handles (ie essentially quit() ) and then re-commence the readImageFromFileHeapSizeStartingAt(). We would have the advantage of knowing absolutely that the image did not need any further checking. I suppose conceptually one might even be able to do magic(™) and keep the convertor object memory allocation, move the fixed image data down to memory base and "just start executing". There's a bit of hand-waving there I suspect...

Rather more mundanely (boo, hiss) we could extend the checkImage... so that the VM could exit with an error telling surrounding scripts that the conversion is required and thus trigger whatever.

Should the auto-conversion write a new image file? Should it query the user about where to put any such new file, or default somehow? How small can we make the convertor image - after all almost everything could be removed if we can be bothered. Do we make a single convertor image that can do all the jobs (I'm imagining doing byteswapping, 32->64, 64->32, etc) or separate ones for each junta? Should it be chocolate flavoured?

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
A paperless office has about as likely as a paperless bathroom.


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Apple starting to alert users that it will end 32-bit app support on the Mac

Ben Coman
In reply to this post by Eliot Miranda-2
 


On 13 April 2018 at 07:00, Jecel Assumpcao Jr. <[hidden email]> wrote:
Eliot Miranda wrote on Thu, 12 Apr 2018 14:10:38 -0700
> That's why for me by far the simplest thing is to convert.

Computers have become a hundred times faster since Squeak first came out
and while disks are also faster the difference is a lot less dramatic.

Same starting point, alternative path. 
Just brainstorming...

Consider that the very old 32-bit Images worked worked on 
much slower hardware, and maybe the StackInterpreter would
provide sufficient performance on today's hardware, and more so into the future.
How hard would it be to compile a 32-bit StackInterpreter and 64-bit Spur together into the same VM?
This would avoid needing to convert, and converting back when the saved Image 
is opened back on a original 32-bit system.

Performance of recent 32-bit Images may be problematic, but less so into the future.
Still this might not be much of a problem for short-time adhoc review of old Images. 

cheers -ben