Re: Numbered primitives in images

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Numbered primitives in images

Bert Freudenberg

(please reply to vm-dev only)

On 16.01.2010, at 03:46, K. K. Subramaniam wrote:
>
>
> The 3.10-4 VM has primitiveObsoleteIndexedPrimitive in slot 207 in
> platforms/unix/src/vm/interp.c while the 3.11.3 tar.gz and the trunk contain
> primitiveFail. The change crept in r1935 through vmm-dtl-126.
>
> Is this intentional or an oversight?
>
> Subbu

Looks like a bug to me ...

- Bert -


Reply | Threaded
Open this post in threaded view
|

Re: Numbered primitives in images

Bert Freudenberg

Fwd from John Maloney ...

- Bert -

Begin forwarded message:

>
> From: John Maloney <[hidden email]>
> Date: 16. Januar 2010 16:21:59 MEZ
> To: Bert Freudenberg <[hidden email]>
> Cc: Ian Piumarta <[hidden email]>
> Subject: Re: [Vm-dev] Re: [squeak-dev] Numbered primitives in images
>
> Hi, Bert.
>
> Thanks for the alert.
>
> Yep, this would be a problem for Scratch. It would prevent sharing projects on the website. Normally, I would not mind sticking with an older VM, but that there is also a change in the sound support on Linux from ALSA to Pulse Audio that we need, and work on that is still ongoing. So we may need to use a current VM, at least on Linux.
>
> Any chance there are some other numbered primitives that could be co-opted for the closure prims? As I recall there were lots of unused numbered primitives.
>
> -- John
>
> On Jan 16, 2010, at 9:57 AM, Bert Freudenberg wrote:
>> That may be a problem ...
>>
>> - Bert -
>>
>> Begin forwarded message:
>>>
>>> From: "David T. Lewis" <[hidden email]>
>>> Date: 16. Januar 2010 15:55:02 MEZ
>>> To: "K. K. Subramaniam" <[hidden email]>
>>> Cc: [hidden email]
>>> Subject: [Vm-dev] Re: [squeak-dev] Numbered primitives in images
>>> Reply-To: Squeak Virtual Machine Development Discussion <[hidden email]>
>>>
>>>
>>> On Sat, Jan 16, 2010 at 08:16:06AM +0530, K. K. Subramaniam wrote:
>>>> On Friday 15 January 2010 10:46:45 pm Bert Freudenberg wrote:
>>>>> On 15.01.2010, at 16:58, K. K. Subramaniam wrote:
>>>>>> Scratch 1.4 (Linux) of 30-Nov-2009 on 3.11-3 VM.
>>>>>> Break into Scratch and do "Socket initializeNetwork".
>>>>>
>>>>> So it does not occur in normal operation?
>>>> Only if you define "normal" as "offline" ;-). Any operation which uses networking
>>>> (url block) goes through this method and it will fail when run under the
>>>> 3.11.3 VM built from unix-3.11.3-vmm or svn trunk.
>>>>
>>>> BTW, Scratch image is just an example which has numbered primitives.
>>>>
>>>> The 3.10-4 VM has primitiveObsoleteIndexedPrimitive in slot 207 in
>>>> platforms/unix/src/vm/interp.c while the 3.11.3 tar.gz and the trunk contain
>>>> primitiveFail. The change crept in r1935 through vmm-dtl-126.
>>>>
>>>> Is this intentional or an oversight?
>>>
>>> Sorry I was not able to reply earlier. The change was intentional,
>>> but the effect on Scratch was not.
>>>
>>> Some of the obsolete primitives had to be dropped when we added the
>>> block closure support. This happened in February 2009 (VMMaker-eem.114
>>> in the SqS/VMMaker). Specifically, we changed from this, which redirects
>>> the numbered primitives:
>>>
>>> "Networking Primitives (200-225) - NO LONGER INDEXED"
>>> (200 225 primitiveObsoleteIndexedPrimitive)
>>>
>>> to this, which reallocates the numbers for closure support and for
>>> future COG integration:
>>>
>>> "new closure primitives (were Networking primitives)"
>>> (200 primitiveClosureCopyWithCopiedValues)
>>> (201 primitiveClosureValue) "value"
>>> (202 primitiveClosureValue) "value:"
>>> (203 primitiveClosureValue) "value:value:"
>>> (204 primitiveClosureValue) "value:value:value:"
>>> (205 primitiveClosureValue) "value:value:value:value:"
>>> (206 primitiveClosureValueWithArgs) "valueWithArguments:"
>>>
>>> (207 209 primitiveFail) "reserved for Cog primitives"
>>>
>>> The obsolete primitive redirection had been in place for at least five
>>> years, so backward compatibility had been preserved for some time.
>>>
>>> Question for Subbu: Is this an important issue may affect a large number
>>> of Scratch users? In other words, are there kids trying to run Scratch,
>>> and their Scratch image stops working properly after Squeak gets updated
>>> to use a new VM?
>>>
>>> And a thought for the vm-dev list: I don't know if there might be a
>>> technical solution to this, but it might be possible to use a strategy
>>> in which the some of the old primitive lookups are preserved (at least
>>> for the range 207 through 225) if and only if the image format is
>>> pre-closures. Follow ups to this on vm-only.
>>>
>>> Dave
>


Reply | Threaded
Open this post in threaded view
|

Re: Re: Numbered primitives in images

David T. Lewis
 
The issue is now on Mantis:

  http://bugs.squeak.org/view.php?id=7454

Moving the closure primitives would be difficult at this point, as
we have a large installed base of closure-enabled images by now.

In principle, it should be possible to update the primitiveTable at
run time based on imageFormatVersion, or perhaps as an option to be
supplied to the VM at startup time, or a combination of the two.

Doing it automatically based on imageFormatVersion would probably
work, but has a catch-22 for someone adding closures to an older
image (e.g. adding the closure support to Etoys). For that situation
you want to be able to turn on closure support even though the image
does not yet need them, so perhaps e.g. a "-closures" command line
option turn on closure support in that case.

Any other ideas?

Dave

On Sat, Jan 16, 2010 at 06:56:47PM +0100, Bert Freudenberg wrote:

>
> Fwd from John Maloney ...
>
> - Bert -
>
> Begin forwarded message:
> >
> > From: John Maloney <[hidden email]>
> > Date: 16. Januar 2010 16:21:59 MEZ
> > To: Bert Freudenberg <[hidden email]>
> > Cc: Ian Piumarta <[hidden email]>
> > Subject: Re: [Vm-dev] Re: [squeak-dev] Numbered primitives in images
> >
> > Hi, Bert.
> >
> > Thanks for the alert.
> >
> > Yep, this would be a problem for Scratch. It would prevent sharing projects on the website. Normally, I would not mind sticking with an older VM, but that there is also a change in the sound support on Linux from ALSA to Pulse Audio that we need, and work on that is still ongoing. So we may need to use a current VM, at least on Linux.
> >
> > Any chance there are some other numbered primitives that could be co-opted for the closure prims? As I recall there were lots of unused numbered primitives.
> >
> > -- John
> >
> > On Jan 16, 2010, at 9:57 AM, Bert Freudenberg wrote:
> >> That may be a problem ...
> >>
> >> - Bert -
> >>
> >> Begin forwarded message:
> >>>
> >>> From: "David T. Lewis" <[hidden email]>
> >>> Date: 16. Januar 2010 15:55:02 MEZ
> >>> To: "K. K. Subramaniam" <[hidden email]>
> >>> Cc: [hidden email]
> >>> Subject: [Vm-dev] Re: [squeak-dev] Numbered primitives in images
> >>> Reply-To: Squeak Virtual Machine Development Discussion <[hidden email]>
> >>>
> >>>
> >>> On Sat, Jan 16, 2010 at 08:16:06AM +0530, K. K. Subramaniam wrote:
> >>>> On Friday 15 January 2010 10:46:45 pm Bert Freudenberg wrote:
> >>>>> On 15.01.2010, at 16:58, K. K. Subramaniam wrote:
> >>>>>> Scratch 1.4 (Linux) of 30-Nov-2009 on 3.11-3 VM.
> >>>>>> Break into Scratch and do "Socket initializeNetwork".
> >>>>>
> >>>>> So it does not occur in normal operation?
> >>>> Only if you define "normal" as "offline" ;-). Any operation which uses networking
> >>>> (url block) goes through this method and it will fail when run under the
> >>>> 3.11.3 VM built from unix-3.11.3-vmm or svn trunk.
> >>>>
> >>>> BTW, Scratch image is just an example which has numbered primitives.
> >>>>
> >>>> The 3.10-4 VM has primitiveObsoleteIndexedPrimitive in slot 207 in
> >>>> platforms/unix/src/vm/interp.c while the 3.11.3 tar.gz and the trunk contain
> >>>> primitiveFail. The change crept in r1935 through vmm-dtl-126.
> >>>>
> >>>> Is this intentional or an oversight?
> >>>
> >>> Sorry I was not able to reply earlier. The change was intentional,
> >>> but the effect on Scratch was not.
> >>>
> >>> Some of the obsolete primitives had to be dropped when we added the
> >>> block closure support. This happened in February 2009 (VMMaker-eem.114
> >>> in the SqS/VMMaker). Specifically, we changed from this, which redirects
> >>> the numbered primitives:
> >>>
> >>> "Networking Primitives (200-225) - NO LONGER INDEXED"
> >>> (200 225 primitiveObsoleteIndexedPrimitive)
> >>>
> >>> to this, which reallocates the numbers for closure support and for
> >>> future COG integration:
> >>>
> >>> "new closure primitives (were Networking primitives)"
> >>> (200 primitiveClosureCopyWithCopiedValues)
> >>> (201 primitiveClosureValue) "value"
> >>> (202 primitiveClosureValue) "value:"
> >>> (203 primitiveClosureValue) "value:value:"
> >>> (204 primitiveClosureValue) "value:value:value:"
> >>> (205 primitiveClosureValue) "value:value:value:value:"
> >>> (206 primitiveClosureValueWithArgs) "valueWithArguments:"
> >>>
> >>> (207 209 primitiveFail) "reserved for Cog primitives"
> >>>
> >>> The obsolete primitive redirection had been in place for at least five
> >>> years, so backward compatibility had been preserved for some time.
> >>>
> >>> Question for Subbu: Is this an important issue may affect a large number
> >>> of Scratch users? In other words, are there kids trying to run Scratch,
> >>> and their Scratch image stops working properly after Squeak gets updated
> >>> to use a new VM?
> >>>
> >>> And a thought for the vm-dev list: I don't know if there might be a
> >>> technical solution to this, but it might be possible to use a strategy
> >>> in which the some of the old primitive lookups are preserved (at least
> >>> for the range 207 through 225) if and only if the image format is
> >>> pre-closures. Follow ups to this on vm-only.
> >>>
> >>> Dave
> >
>
Reply | Threaded
Open this post in threaded view
|

VM overwrites extraVmMemory value with junk on 64 bit image

David T. Lewis
 
John, this may be of some concern if your new VMs are making use
of the #extraVmMemory setting.

  http://bugs.squeak.org/view.php?id=7455

A 64 bit image (wordSize 8) is currently using a 128 byte image file header,
with all the integer values in the header stored as 8 bit ints. The VM
is writing the image data at position 64 rather than 128, resulting the
value of extraVmMemory being overwritten with garbage. ImageTracer64
produces the correct 128 byte offset, but this is undone by the VM after
the first image save.

This is producing no obvious symptoms when running on a unix VM, but is
certainly not a good thing.

Short term fix should be to make the VM use headerSize 128 rather than 64
when writing a 64 bit image. IMO the correct thing to do long term is to
always write header data as 32 bit int regardless of image wordSize, and
use 64 bit file headers in both cases. This however will require some way
to retain backward compatibility for reading 64 bit images that have the
current file header structure.

Dave

Reply | Threaded
Open this post in threaded view
|

Re: VM overwrites extraVmMemory value with junk on 64 bit image

johnmci

extraVmMemory was an option for macintosh os 9 and earlier machines.  Say September 27, 1996 era machines.
In checking the flow of this variable it's actually ignored by macintosh VM since the early part of the last decade.

I'd suggest the slot is marked as reserved/obsolete, and the extraVmMemory removed. I note the extraVmMemory
is a vm parm value so we can't drop it directly without effecting index, but it can be marked as obsolete.

On 2010-01-16, at 12:30 PM, David T. Lewis wrote:

> extraVmMemory

--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================




Reply | Threaded
Open this post in threaded view
|

Re: Re: Numbered primitives in images

Eliot Miranda-2
In reply to this post by David T. Lewis
 
On Sat, Jan 16, 2010 at 11:20 AM, David T. Lewis <[hidden email]> wrote:

The issue is now on Mantis:

 http://bugs.squeak.org/view.php?id=7454

Moving the closure primitives would be difficult at this point, as
we have a large installed base of closure-enabled images by now.

In principle, it should be possible to update the primitiveTable at
run time based on imageFormatVersion, or perhaps as an option to be
supplied to the VM at startup time, or a combination of the two.

Doing it automatically based on imageFormatVersion would probably
work, but has a catch-22 for someone adding closures to an older
image (e.g. adding the closure support to Etoys). For that situation
you want to be able to turn on closure support even though the image
does not yet need them, so perhaps e.g. a "-closures" command line
option turn on closure support in that case.

Any other ideas?

Would it be possible to write a script to replace the numbered primitives in the Scratch image with named prims and run that script on a suitable pre-closure VM?  Once the image is saved it should run on the closure-enabled VMs.

Surely this is better than rewriting the primitive table.  I would think we would want to move away from obsolete interfaces and if Scratch supports named prims then it works, is compatible with other Squeaks and avoids hacks.
 

Dave

On Sat, Jan 16, 2010 at 06:56:47PM +0100, Bert Freudenberg wrote:
>
> Fwd from John Maloney ...
>
> - Bert -
>
> Begin forwarded message:
> >
> > From: John Maloney <[hidden email]>
> > Date: 16. Januar 2010 16:21:59 MEZ
> > To: Bert Freudenberg <[hidden email]>
> > Cc: Ian Piumarta <[hidden email]>
> > Subject: Re: [Vm-dev] Re: [squeak-dev] Numbered primitives in images
> >
> > Hi, Bert.
> >
> > Thanks for the alert.
> >
> > Yep, this would be a problem for Scratch. It would prevent sharing projects on the website. Normally, I would not mind sticking with an older VM, but that there is also a change in the sound support on Linux from ALSA to Pulse Audio that we need, and work on that is still ongoing. So we may need to use a current VM, at least on Linux.
> >
> > Any chance there are some other numbered primitives that could be co-opted for the closure prims? As I recall there were lots of unused numbered primitives.
> >
> >     -- John
> >
> > On Jan 16, 2010, at 9:57 AM, Bert Freudenberg wrote:
> >> That may be a problem ...
> >>
> >> - Bert -
> >>
> >> Begin forwarded message:
> >>>
> >>> From: "David T. Lewis" <[hidden email]>
> >>> Date: 16. Januar 2010 15:55:02 MEZ
> >>> To: "K. K. Subramaniam" <[hidden email]>
> >>> Cc: [hidden email]
> >>> Subject: [Vm-dev] Re: [squeak-dev] Numbered primitives in images
> >>> Reply-To: Squeak Virtual Machine Development Discussion <[hidden email]>
> >>>
> >>>
> >>> On Sat, Jan 16, 2010 at 08:16:06AM +0530, K. K. Subramaniam wrote:
> >>>> On Friday 15 January 2010 10:46:45 pm Bert Freudenberg wrote:
> >>>>> On 15.01.2010, at 16:58, K. K. Subramaniam wrote:
> >>>>>> Scratch 1.4 (Linux) of 30-Nov-2009 on 3.11-3 VM.
> >>>>>> Break into Scratch and do "Socket initializeNetwork".
> >>>>>
> >>>>> So it does not occur in normal operation?
> >>>> Only if you define "normal" as "offline" ;-). Any operation which uses networking
> >>>> (url block) goes through this method and it will fail when run under the
> >>>> 3.11.3 VM built from unix-3.11.3-vmm or svn trunk.
> >>>>
> >>>> BTW, Scratch image is just an example which has numbered primitives.
> >>>>
> >>>> The 3.10-4 VM has primitiveObsoleteIndexedPrimitive in slot 207 in
> >>>> platforms/unix/src/vm/interp.c while the 3.11.3 tar.gz and the trunk contain
> >>>> primitiveFail. The change crept in r1935 through vmm-dtl-126.
> >>>>
> >>>> Is this intentional or an oversight?
> >>>
> >>> Sorry I was not able to reply earlier. The change was intentional,
> >>> but the effect on Scratch was not.
> >>>
> >>> Some of the obsolete primitives had to be dropped when we added the
> >>> block closure support. This happened in February 2009 (VMMaker-eem.114
> >>> in the SqS/VMMaker). Specifically, we changed from this, which redirects
> >>> the numbered primitives:
> >>>
> >>>           "Networking Primitives (200-225) - NO LONGER INDEXED"
> >>>           (200 225 primitiveObsoleteIndexedPrimitive)
> >>>
> >>> to this, which reallocates the numbers for closure support and for
> >>> future COG integration:
> >>>
> >>>           "new closure primitives (were Networking primitives)"
> >>>           (200 primitiveClosureCopyWithCopiedValues)
> >>>           (201 primitiveClosureValue) "value"
> >>>           (202 primitiveClosureValue) "value:"
> >>>           (203 primitiveClosureValue) "value:value:"
> >>>           (204 primitiveClosureValue) "value:value:value:"
> >>>           (205 primitiveClosureValue) "value:value:value:value:"
> >>>           (206 primitiveClosureValueWithArgs) "valueWithArguments:"
> >>>
> >>>           (207 209 primitiveFail) "reserved for Cog primitives"
> >>>
> >>> The obsolete primitive redirection had been in place for at least five
> >>> years, so backward compatibility had been preserved for some time.
> >>>
> >>> Question for Subbu: Is this an important issue may affect a large number
> >>> of Scratch users? In other words, are there kids trying to run Scratch,
> >>> and their Scratch image stops working properly after Squeak gets updated
> >>> to use a new VM?
> >>>
> >>> And a thought for the vm-dev list: I don't know if there might be a
> >>> technical solution to this, but it might be possible to use a strategy
> >>> in which the some of the old primitive lookups are preserved (at least
> >>> for the range 207 through 225) if and only if the image format is
> >>> pre-closures. Follow ups to this on vm-only.
> >>>
> >>> Dave
> >
>

Reply | Threaded
Open this post in threaded view
|

Re: Re: Numbered primitives in images

Bert Freudenberg
 
On 17.01.2010, at 02:23, Eliot Miranda wrote:
On Sat, Jan 16, 2010 at 11:20 AM, David T. Lewis <[hidden email]> wrote:

The issue is now on Mantis:

 http://bugs.squeak.org/view.php?id=7454

Moving the closure primitives would be difficult at this point, as
we have a large installed base of closure-enabled images by now.

In principle, it should be possible to update the primitiveTable at
run time based on imageFormatVersion, or perhaps as an option to be
supplied to the VM at startup time, or a combination of the two.

Doing it automatically based on imageFormatVersion would probably
work, but has a catch-22 for someone adding closures to an older
image (e.g. adding the closure support to Etoys). For that situation
you want to be able to turn on closure support even though the image
does not yet need them, so perhaps e.g. a "-closures" command line
option turn on closure support in that case.

Any other ideas?

Would it be possible to write a script to replace the numbered primitives in the Scratch image with named prims and run that script on a suitable pre-closure VM?  Once the image is saved it should run on the closure-enabled VMs.

Surely this is better than rewriting the primitive table.  I would think we would want to move away from obsolete interfaces and if Scratch supports named prims then it works, is compatible with other Squeaks and avoids hacks.

The Scratch image is from around Squeak 2.5 - you might take a look here:

- Bert -

 

Dave

On Sat, Jan 16, 2010 at 06:56:47PM +0100, Bert Freudenberg wrote:
>
> Fwd from John Maloney ...
>
> - Bert -
>
> Begin forwarded message:
> >
> > From: John Maloney <[hidden email]>
> > Date: 16. Januar 2010 16:21:59 MEZ
> > To: Bert Freudenberg <[hidden email]>
> > Cc: Ian Piumarta <[hidden email]>
> > Subject: Re: [Vm-dev] Re: [squeak-dev] Numbered primitives in images
> >
> > Hi, Bert.
> >
> > Thanks for the alert.
> >
> > Yep, this would be a problem for Scratch. It would prevent sharing projects on the website. Normally, I would not mind sticking with an older VM, but that there is also a change in the sound support on Linux from ALSA to Pulse Audio that we need, and work on that is still ongoing. So we may need to use a current VM, at least on Linux.
> >
> > Any chance there are some other numbered primitives that could be co-opted for the closure prims? As I recall there were lots of unused numbered primitives.
> >
> >     -- John
> >
> > On Jan 16, 2010, at 9:57 AM, Bert Freudenberg wrote:
> >> That may be a problem ...
> >>
> >> - Bert -
> >>
> >> Begin forwarded message:
> >>>
> >>> From: "David T. Lewis" <[hidden email]>
> >>> Date: 16. Januar 2010 15:55:02 MEZ
> >>> To: "K. K. Subramaniam" <[hidden email]>
> >>> Cc: [hidden email]
> >>> Subject: [Vm-dev] Re: [squeak-dev] Numbered primitives in images
> >>> Reply-To: Squeak Virtual Machine Development Discussion <[hidden email]>
> >>>
> >>>
> >>> On Sat, Jan 16, 2010 at 08:16:06AM +0530, K. K. Subramaniam wrote:
> >>>> On Friday 15 January 2010 10:46:45 pm Bert Freudenberg wrote:
> >>>>> On 15.01.2010, at 16:58, K. K. Subramaniam wrote:
> >>>>>> Scratch 1.4 (Linux) of 30-Nov-2009 on 3.11-3 VM.
> >>>>>> Break into Scratch and do "Socket initializeNetwork".
> >>>>>
> >>>>> So it does not occur in normal operation?
> >>>> Only if you define "normal" as "offline" ;-). Any operation which uses networking
> >>>> (url block) goes through this method and it will fail when run under the
> >>>> 3.11.3 VM built from unix-3.11.3-vmm or svn trunk.
> >>>>
> >>>> BTW, Scratch image is just an example which has numbered primitives.
> >>>>
> >>>> The 3.10-4 VM has primitiveObsoleteIndexedPrimitive in slot 207 in
> >>>> platforms/unix/src/vm/interp.c while the 3.11.3 tar.gz and the trunk contain
> >>>> primitiveFail. The change crept in r1935 through vmm-dtl-126.
> >>>>
> >>>> Is this intentional or an oversight?
> >>>
> >>> Sorry I was not able to reply earlier. The change was intentional,
> >>> but the effect on Scratch was not.
> >>>
> >>> Some of the obsolete primitives had to be dropped when we added the
> >>> block closure support. This happened in February 2009 (VMMaker-eem.114
> >>> in the SqS/VMMaker). Specifically, we changed from this, which redirects
> >>> the numbered primitives:
> >>>
> >>>           "Networking Primitives (200-225) - NO LONGER INDEXED"
> >>>           (200 225 primitiveObsoleteIndexedPrimitive)
> >>>
> >>> to this, which reallocates the numbers for closure support and for
> >>> future COG integration:
> >>>
> >>>           "new closure primitives (were Networking primitives)"
> >>>           (200 primitiveClosureCopyWithCopiedValues)
> >>>           (201 primitiveClosureValue) "value"
> >>>           (202 primitiveClosureValue) "value:"
> >>>           (203 primitiveClosureValue) "value:value:"
> >>>           (204 primitiveClosureValue) "value:value:value:"
> >>>           (205 primitiveClosureValue) "value:value:value:value:"
> >>>           (206 primitiveClosureValueWithArgs) "valueWithArguments:"
> >>>
> >>>           (207 209 primitiveFail) "reserved for Cog primitives"
> >>>
> >>> The obsolete primitive redirection had been in place for at least five
> >>> years, so backward compatibility had been preserved for some time.
> >>>
> >>> Question for Subbu: Is this an important issue may affect a large number
> >>> of Scratch users? In other words, are there kids trying to run Scratch,
> >>> and their Scratch image stops working properly after Squeak gets updated
> >>> to use a new VM?
> >>>
> >>> And a thought for the vm-dev list: I don't know if there might be a
> >>> technical solution to this, but it might be possible to use a strategy
> >>> in which the some of the old primitive lookups are preserved (at least
> >>> for the range 207 through 225) if and only if the image format is
> >>> pre-closures. Follow ups to this on vm-only.
> >>>
> >>> Dave
> >
>




Reply | Threaded
Open this post in threaded view
|

Re: Re: Numbered primitives in images

K. K. Subramaniam
In reply to this post by David T. Lewis
 
On Sunday 17 January 2010 12:50:17 am David T. Lewis wrote:
> Doing it automatically based on imageFormatVersion would probably
> work, but has a catch-22 for someone adding closures to an older
> image (e.g. adding the closure support to Etoys). For that situation
> you want to be able to turn on closure support even though the image
> does not yet need them, so perhaps e.g. a "-closures" command line
> option turn on closure support in that case.
>
> Any other ideas?
An image may run on a VM on multiple platforms and not all may be at the same
rev, breaking backward compatibility between 3.10 and 3.11 may cause more
grief than necessary.

How about patching the VM to check for env variable (say SQUEAK_PRIM311) in
addition to checking for -closures?

Subbu
Reply | Threaded
Open this post in threaded view
|

Re: Re: Numbered primitives in images

David T. Lewis
 
On Sun, Jan 17, 2010 at 08:43:37PM +0530, K. K. Subramaniam wrote:

> On Sunday 17 January 2010 12:50:17 am David T. Lewis wrote:
> > Doing it automatically based on imageFormatVersion would probably
> > work, but has a catch-22 for someone adding closures to an older
> > image (e.g. adding the closure support to Etoys). For that situation
> > you want to be able to turn on closure support even though the image
> > does not yet need them, so perhaps e.g. a "-closures" command line
> > option turn on closure support in that case.
> >
> > Any other ideas?
> An image may run on a VM on multiple platforms and not all may be at the same
> rev, breaking backward compatibility between 3.10 and 3.11 may cause more
> grief than necessary.
>
> How about patching the VM to check for env variable (say SQUEAK_PRIM311) in
> addition to checking for -closures?

I guess we're still looking for ideas at this point. As Eliot points
out, rewriting the primitive table (which is probably what we would
do based on s SQUEAK_PRIM311 flag) is a messy hack, and it would be
best not to put that sort of thing into the VM.

Perhaps a BackwardCompatibilityHacksPlugin to keep the messy stuff
out of the main VM would be possible. To extend the challenge a bit
further, it would be interesting to see how far back in time we could
go with a single VM. The ftp.squeak.org site has images going all the
way back to Squeak 1.1.

Dave

Reply | Threaded
Open this post in threaded view
|

Re: Re: Numbered primitives in images

John Maloney
In reply to this post by Bert Freudenberg
 
Elliot's idea -- just fix the Scratch image to use the named socket  
prims -- sounds like a winner. It's simple, it doesn't add any  
complexity to the VM's, and it doesn't break anything. And it puts the  
burden on Scratch, where it belongs.

Assuming the named socket primitives do exactly the same things the  
numbered ones did (and it sounds like they do, since the numbered  
prims have been re-directed to the named ones for about a decade :-))  
then nothing should break, right?

Note that Scratch comes bundled with a copy of the Squeak VM, and the  
bundled VM in the current release still has support for the numbered  
socket primitives. So this VM change only matters for future releases  
of Scratch (and the Linux package, which is in active development).

Thanks for thinking about this issue.

        -- John


On Jan 17, 2010, at 7:07 AM, Bert Freudenberg wrote:

> On 17.01.2010, at 02:23, Eliot Miranda wrote:
>> On Sat, Jan 16, 2010 at 11:20 AM, David T. Lewis  
>> <[hidden email]> wrote:
>>
>> The issue is now on Mantis:
>>
>>  http://bugs.squeak.org/view.php?id=7454
>>
>> Moving the closure primitives would be difficult at this point, as
>> we have a large installed base of closure-enabled images by now.
>>
>> In principle, it should be possible to update the primitiveTable at
>> run time based on imageFormatVersion, or perhaps as an option to be
>> supplied to the VM at startup time, or a combination of the two.
>>
>> Doing it automatically based on imageFormatVersion would probably
>> work, but has a catch-22 for someone adding closures to an older
>> image (e.g. adding the closure support to Etoys). For that situation
>> you want to be able to turn on closure support even though the image
>> does not yet need them, so perhaps e.g. a "-closures" command line
>> option turn on closure support in that case.
>>
>> Any other ideas?
>>
>> Would it be possible to write a script to replace the numbered  
>> primitives in the Scratch image with named prims and run that  
>> script on a suitable pre-closure VM?  Once the image is saved it  
>> should run on the closure-enabled VMs.
>>
>> Surely this is better than rewriting the primitive table.  I would  
>> think we would want to move away from obsolete interfaces and if  
>> Scratch supports named prims then it works, is compatible with  
>> other Squeaks and avoids hacks.
>
> The Scratch image is from around Squeak 2.5 - you might take a look  
> here:
> http://download.scratch.mit.edu/source-code/ScratchSource1.4.zip
>
> - Bert -
>
>>
>>
>> Dave
>>
>> On Sat, Jan 16, 2010 at 06:56:47PM +0100, Bert Freudenberg wrote:
>> >
>> > Fwd from John Maloney ...
>> >
>> > - Bert -
>> >
>> > Begin forwarded message:
>> > >
>> > > From: John Maloney <[hidden email]>
>> > > Date: 16. Januar 2010 16:21:59 MEZ
>> > > To: Bert Freudenberg <[hidden email]>
>> > > Cc: Ian Piumarta <[hidden email]>
>> > > Subject: Re: [Vm-dev] Re: [squeak-dev] Numbered primitives in  
>> images
>> > >
>> > > Hi, Bert.
>> > >
>> > > Thanks for the alert.
>> > >
>> > > Yep, this would be a problem for Scratch. It would prevent  
>> sharing projects on the website. Normally, I would not mind  
>> sticking with an older VM, but that there is also a change in the  
>> sound support on Linux from ALSA to Pulse Audio that we need, and  
>> work on that is still ongoing. So we may need to use a current VM,  
>> at least on Linux.
>> > >
>> > > Any chance there are some other numbered primitives that could  
>> be co-opted for the closure prims? As I recall there were lots of  
>> unused numbered primitives.
>> > >
>> > >     -- John
>> > >
>> > > On Jan 16, 2010, at 9:57 AM, Bert Freudenberg wrote:
>> > >> That may be a problem ...
>> > >>
>> > >> - Bert -
>> > >>
>> > >> Begin forwarded message:
>> > >>>
>> > >>> From: "David T. Lewis" <[hidden email]>
>> > >>> Date: 16. Januar 2010 15:55:02 MEZ
>> > >>> To: "K. K. Subramaniam" <[hidden email]>
>> > >>> Cc: [hidden email]
>> > >>> Subject: [Vm-dev] Re: [squeak-dev] Numbered primitives in  
>> images
>> > >>> Reply-To: Squeak Virtual Machine Development Discussion <[hidden email]
>> >
>> > >>>
>> > >>>
>> > >>> On Sat, Jan 16, 2010 at 08:16:06AM +0530, K. K. Subramaniam  
>> wrote:
>> > >>>> On Friday 15 January 2010 10:46:45 pm Bert Freudenberg wrote:
>> > >>>>> On 15.01.2010, at 16:58, K. K. Subramaniam wrote:
>> > >>>>>> Scratch 1.4 (Linux) of 30-Nov-2009 on 3.11-3 VM.
>> > >>>>>> Break into Scratch and do "Socket initializeNetwork".
>> > >>>>>
>> > >>>>> So it does not occur in normal operation?
>> > >>>> Only if you define "normal" as "offline" ;-). Any operation  
>> which uses networking
>> > >>>> (url block) goes through this method and it will fail when  
>> run under the
>> > >>>> 3.11.3 VM built from unix-3.11.3-vmm or svn trunk.
>> > >>>>
>> > >>>> BTW, Scratch image is just an example which has numbered  
>> primitives.
>> > >>>>
>> > >>>> The 3.10-4 VM has primitiveObsoleteIndexedPrimitive in slot  
>> 207 in
>> > >>>> platforms/unix/src/vm/interp.c while the 3.11.3 tar.gz and  
>> the trunk contain
>> > >>>> primitiveFail. The change crept in r1935 through vmm-dtl-126.
>> > >>>>
>> > >>>> Is this intentional or an oversight?
>> > >>>
>> > >>> Sorry I was not able to reply earlier. The change was  
>> intentional,
>> > >>> but the effect on Scratch was not.
>> > >>>
>> > >>> Some of the obsolete primitives had to be dropped when we  
>> added the
>> > >>> block closure support. This happened in February 2009  
>> (VMMaker-eem.114
>> > >>> in the SqS/VMMaker). Specifically, we changed from this,  
>> which redirects
>> > >>> the numbered primitives:
>> > >>>
>> > >>>           "Networking Primitives (200-225) - NO LONGER INDEXED"
>> > >>>           (200 225 primitiveObsoleteIndexedPrimitive)
>> > >>>
>> > >>> to this, which reallocates the numbers for closure support  
>> and for
>> > >>> future COG integration:
>> > >>>
>> > >>>           "new closure primitives (were Networking primitives)"
>> > >>>           (200 primitiveClosureCopyWithCopiedValues)
>> > >>>           (201 primitiveClosureValue) "value"
>> > >>>           (202 primitiveClosureValue) "value:"
>> > >>>           (203 primitiveClosureValue) "value:value:"
>> > >>>           (204 primitiveClosureValue) "value:value:value:"
>> > >>>           (205 primitiveClosureValue)  
>> "value:value:value:value:"
>> > >>>           (206 primitiveClosureValueWithArgs)  
>> "valueWithArguments:"
>> > >>>
>> > >>>           (207 209 primitiveFail) "reserved for Cog primitives"
>> > >>>
>> > >>> The obsolete primitive redirection had been in place for at  
>> least five
>> > >>> years, so backward compatibility had been preserved for some  
>> time.
>> > >>>
>> > >>> Question for Subbu: Is this an important issue may affect a  
>> large number
>> > >>> of Scratch users? In other words, are there kids trying to  
>> run Scratch,
>> > >>> and their Scratch image stops working properly after Squeak  
>> gets updated
>> > >>> to use a new VM?
>> > >>>
>> > >>> And a thought for the vm-dev list: I don't know if there  
>> might be a
>> > >>> technical solution to this, but it might be possible to use a  
>> strategy
>> > >>> in which the some of the old primitive lookups are preserved  
>> (at least
>> > >>> for the range 207 through 225) if and only if the image  
>> format is
>> > >>> pre-closures. Follow ups to this on vm-only.
>> > >>>
>> > >>> Dave
>> > >

Reply | Threaded
Open this post in threaded view
|

Re: Re: Numbered primitives in images

Bert Freudenberg
In reply to this post by David T. Lewis

On 17.01.2010, at 16:39, David T. Lewis wrote:

>
>
> On Sun, Jan 17, 2010 at 08:43:37PM +0530, K. K. Subramaniam wrote:
>> On Sunday 17 January 2010 12:50:17 am David T. Lewis wrote:
>>> Doing it automatically based on imageFormatVersion would probably
>>> work, but has a catch-22 for someone adding closures to an older
>>> image (e.g. adding the closure support to Etoys). For that situation
>>> you want to be able to turn on closure support even though the image
>>> does not yet need them, so perhaps e.g. a "-closures" command line
>>> option turn on closure support in that case.
>>>
>>> Any other ideas?
>> An image may run on a VM on multiple platforms and not all may be at the same
>> rev, breaking backward compatibility between 3.10 and 3.11 may cause more
>> grief than necessary.
>>
>> How about patching the VM to check for env variable (say SQUEAK_PRIM311) in
>> addition to checking for -closures?
>
> I guess we're still looking for ideas at this point. As Eliot points
> out, rewriting the primitive table (which is probably what we would
> do based on s SQUEAK_PRIM311 flag) is a messy hack, and it would be
> best not to put that sort of thing into the VM.
>
> Perhaps a BackwardCompatibilityHacksPlugin to keep the messy stuff
> out of the main VM would be possible. To extend the challenge a bit
> further, it would be interesting to see how far back in time we could
> go with a single VM. The ftp.squeak.org site has images going all the
> way back to Squeak 1.1.
>
> Dave

IIUC the unix 3.10-5 VM is still okay and 3.10-6 changed this. We're sort-of lucky that the latest VM package in Fedora is still 3.10-5 (meaning the new OLPC OS will still support Scratch) but it is unfortunate that a minor version would break this. At least I wasn't aware the closure support does break backwards compatibility, and apparently Ian neither, or he would have at least made the change in 3.11 (if not 4.x).

- Bert -


Reply | Threaded
Open this post in threaded view
|

Re: Re: Numbered primitives in images

David T. Lewis
 
On Sun, Jan 17, 2010 at 08:02:23PM +0100, Bert Freudenberg wrote:
>
> IIUC the unix 3.10-5 VM is still okay and 3.10-6 changed this. We're sort-of
> lucky that the latest VM package in Fedora is still 3.10-5 (meaning the new
> OLPC OS will still support Scratch) but it is unfortunate that a minor version
> would break this. At least I wasn't aware the closure support does break
> backwards compatibility, and apparently Ian neither, or he would have at
> least made the change in 3.11 (if not 4.x).

Agreed, and with hindsight I'm sorry that I did not communicate the extent
of the changes more clearly. Although to be clear, it was a mistake: I did
not expect that the closure changes would break compatibility with a 2.5
based image, because the primitives indices that we removed had been marked
obsolete for at least five years.

By way of preventing recurrence, the new VM version numbering approach that
Ian has since adopted may help prevent this kind of misunderstanding. The
version number is now based on VMMaker versionString combined with the
Subversion version number, so for example in the case of the closures changes,
the VM version would have gone from 3.9.5-nnnn to 3.10-nnnn (and yes, I should
have made it 3.10.0 rather than 3.10, I'll keep that in mind in the future).

Maintaining backward compatibility as we move towards COG or other VM
enhancements is going to be a challenge, and we'll have to do this
carefully with an eye towards supporting a rather diverse installed base
of Squeak images.

Dave
 
Reply | Threaded
Open this post in threaded view
|

Re: Re: Numbered primitives in images

Eliot Miranda-2
 


On Sun, Jan 17, 2010 at 2:42 PM, David T. Lewis <[hidden email]> wrote:

On Sun, Jan 17, 2010 at 08:02:23PM +0100, Bert Freudenberg wrote:
>
> IIUC the unix 3.10-5 VM is still okay and 3.10-6 changed this. We're sort-of
> lucky that the latest VM package in Fedora is still 3.10-5 (meaning the new
> OLPC OS will still support Scratch) but it is unfortunate that a minor version
> would break this. At least I wasn't aware the closure support does break
> backwards compatibility, and apparently Ian neither, or he would have at
> least made the change in 3.11 (if not 4.x).

Agreed, and with hindsight I'm sorry that I did not communicate the extent
of the changes more clearly. Although to be clear, it was a mistake: I did
not expect that the closure changes would break compatibility with a 2.5
based image, because the primitives indices that we removed had been marked
obsolete for at least five years.

By way of preventing recurrence, the new VM version numbering approach that
Ian has since adopted may help prevent this kind of misunderstanding. The
version number is now based on VMMaker versionString combined with the
Subversion version number, so for example in the case of the closures changes,
the VM version would have gone from 3.9.5-nnnn to 3.10-nnnn (and yes, I should
have made it 3.10.0 rather than 3.10, I'll keep that in mind in the future).

Maintaining backward compatibility as we move towards COG or other VM
enhancements is going to be a challenge, and we'll have to do this
carefully with an eye towards supporting a rather diverse installed base
of Squeak images.

The move to Cog will break backward-compatibility with all pre-closure images as the Cog VM only runs closures. It does not and cannot easily support old BlockContext-based images.  So the issue of backward-compatibility is moot.

The issue is thus one of forward migration.  Applying the closure bootstrap is not an easy process but is one that has been done by others; Juan Vuletich migrated Cuis to closures without any help at all; most impressive.  With a bit of polish an up-to-date version of the bootstrap should be able to take images forward without too much effort.  One needs a VM that runs both closures and old BlockContext, i.e. one of the current clutch of pre-Cog VMs.  One needs a bootstrap that has been updated with the relevant bug-fixes.

best
Eliot


Dave