Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5]

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5]

Eliot Miranda-2
Hi All, but especially Tobias,

    as we know, ImageSegments are broken in Spur. They work occasionally, but when stressed the VM ends up crashing.  The fundamental issue is that the ImageSegment code makes assumptions about object ordering that Spur violates.  For example, here's ImageSegment's install:

install
"This operation retrieves the segment if necessary from file storage, installs it in memory, and replaces (using become:) all the root stubs with the reconstructed roots of the segment."

| newRoots |
state = #onFile ifTrue: [self readFromFile].
state = #onFileWithSymbols ifTrue: [self readFromFileWithSymbols.
endMarker := segment nextObject. "for enumeration of objects"
endMarker == 0 ifTrue: [endMarker := 'End' clone]].
(state = #active) | (state = #imported) ifFalse: [self errorWrongState].
newRoots := self loadSegmentFrom: segment outPointers: outPointers.
state = #imported 
ifTrue: ["just came in from exported file"
arrayOfRoots := newRoots]
ifFalse: [
arrayOfRoots elementsForwardIdentityTo: newRoots].
state := #inactive.
Beeper beepPrimitive

So before the image segment bytes (the segment inst var) is loaded, the object after it is assigned to endMarker, and if there isn't an object after segment, a new object ('End' clone) is assigned to endMarker.

This makes the assumption that objects are allocated in a strict order, and therefore endMarker will always be the object after segment.

Loading the segment via "newRoots := self loadSegmentFrom: segment outPointers: outPointers" then turns segment into a zero-length WordArray and its contents into the objects loaded by the segment.  Therefore, in the V3 system, the objects loaded from segment can be enumerated starting at segment nextObject and repeating until endMarker is found:

allObjectsDo: aBlock
"Enumerate all objects that came from this segment.  NOTE this assumes that the segment was created (and extracted).  After the segment has been installed (install), this method allows you to enumerate its objects."
| obj |

endMarker == nil ifTrue: [
^ self error: 'Just extract and install, don''t writeToFile:'].
segment size ~= 1 ifTrue: [
^ self error: 'Vestigial segment size must be 1 (version word)'].

obj := segment nextObject.  "Start with the next object after the vestigial header"
[obj == endMarker] whileFalse:  "Stop at the next object after the full segment"
[aBlock value: obj.
obj := obj nextObject].  "Step through the objects installed from the segment."

Now, as written, this just won't work in Spur.

a) the only place where there is any kind of stable order to objects is in oldSpace, so segment /has/ to be forced to old space to have any chance of its objects being in order when it gets converted from bytes to objects.

b) 'End' clone will be in newSpace and so endMarker isn't reliable unless it was obtained by segment nextObject when segment was already in oldSpace.

So it is perhaps possible to fix ImageSegments in Spur by forcing segment to oldSpace and being more careful with endMarker.  But I think there is a better way.

If the set of objects the segment contains can be obtained some how then this set can be simply enumerated, not depending on nextObject.  The primitive has to answer the array of roots, so its result can't be changed to be the entire array.  But segment could be becomed into an Array of all the objects in segment prior to it being loaded, in which case the above would become


install
"This operation retrieves the segment if necessary from file storage, installs it in memory, and replaces (using become:) all the root stubs with the reconstructed roots of the segment."

| newRoots |
state = #onFile ifTrue: [self readFromFile].
state = #onFileWithSymbols ifTrue:
[self readFromFileWithSymbols].
(state = #active) | (state = #imported) ifFalse: [self errorWrongState].
newRoots := self loadSegmentFrom: segment outPointers: outPointers.
state = #imported 
ifTrue: "just came in from exported file"
[arrayOfRoots := newRoots]
ifFalse:
[arrayOfRoots elementsForwardIdentityTo: newRoots].
state := #inactive.
Beeper beepPrimitive

allObjectsDo: aBlock
"Enumerate all objects that came from this segment.  NOTE this assumes that the segment was created (and extracted).  After the segment has been installed (install), this method allows you to enumerate its objects."
| obj |

segment isArray ifFalse:
[^ self error: 'Segment hasn''t been loaded?'].

segment do: aBlock

and the endMarker instVar would be deleted.

I am willing and ready to modify the primitive to convert the segment correctly.  Who will volunteer to rewrite the image-level ImageSegment code to use the new primitive?



On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda <[hidden email]> wrote:
Hi Tobias,

On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape <[hidden email]> wrote:
Hi all
On 13.08.2015, at 02:15, Eliot Miranda <[hidden email]> wrote:

> Hi Tobias,
>
>    forget that. I found it.  THis right?
>
> Trunk test suite for Spur
> Using existing cogspur r.3410
> cp -r /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410 /tmp/d20150812-28620-etbikj
>   image test suite
> VM: /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
> /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak -version
> /var/lib/jenkins/workspace/Trunk/default/tests.st
> spawning command 0 with timeout 1800 seconds: "/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak" "-vm-sound-null" "-vm-display-null" "/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image" "../tests.st"
> (Command started with PID 28643)
> 2015-08-12T23:31:48.17+01:00: Loading Hudson build tools... from /var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
> 2015-08-12T23:31:48.388+01:00: Running tests...
> setsockopt: Protocol not available
> setsockopt: Protocol not available
> 28646:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:612:
>
> Recursive not understood error encountered
>
>
> I bet this is ImageSegment related.

It sure is.

        BitmapStreamTests>testMatrixTransform2x3WithImageSegment
sends
        BitmapStreamTests>validateImageSegment

 Basically the ImageSegment code has been working in Spur on a hope and a prayer.  The code assumes objects are allocated in chronological order and that this order is preserved, not so with Spur.  So post image segment loading someObject/nextObject is used to enumerate the objects loaded.  This can't work reliably in Spur.  I *think* (ok I hope) that I've implemented the segment load primitive in Spur to answer an Array of the objects loaded, so that these can be explicitly enumerated.

So the job is a) to check that I have indeed implemented the primitive to do this and b) to rewrite the image segment loading code in the light of this.

David, this is an example of something that isn't back portable and should not be back-ported.

The most strange thing is that I cannot reproduce this on my Mac…
There the test does not crash the image…

Well then it may be a signed/unsigned bug in image loading instead.  On linux the image typically gets loaded quite high in the address space and so a good portion of the heap ends up above 0x7FFFFFFF, or negative territory if misinterpreted as signed integers.  On Mac and Windows the image tends to get loaded quite low and so has to be pretty big to fall foul of signedness issues.


Busy right now but will check tomorrow.


Best regards
        -Tobias


>
> On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda <[hidden email]> wrote:
> Hi Tobias,
>
> On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape <[hidden email]> wrote:
>
> On 12.08.2015, at 20:55, Eliot Miranda <[hidden email]> wrote:
>
> > Hi All,
> >
> >
> >     Fabio's kindly done most of the changes.  But some questions remain for more general discussion.  See below.
> >
> > Fabio,  thanks so much for doing this, and so quickly!
> >
> > On Tue, Aug 11, 2015 at 8:51 PM, Eliot Miranda <[hidden email]> wrote:
> > Hi All,
> >
> >     who will update http://squeak.org/downloads/ to include the 5.0 release?
> >
> > a) I suggest that the 5.0 all-in-one have a line in the left-hand table that includes the 4.6 release and that it precede the 4.6 release in the list.
> >
> > Done.
> >
> >
> > b) the Trunk link points to TrunkImage.zip which contains a non-Spur image.
> >
> > So this is the biggie.  What should we do?
> >
> > - add a Trunk 5.0 build with a different link?  (Noooooo)
> >
> > - change  http://build.squeak.org/job/SqueakTrunk to build what it says (Yessss please, who can do this?)
>
> Done: build.squeak.org/job/Trunk/
> But as I said several times, Spur/trunk test just crash for month…
>
> can you point me to the crash?  I'm looking at http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink and see 8 test failures but no crash.
>
>
> >
> > - add a Squeak 4.6 build job?  Is that even worth it any more considering 4.6 is released and stable?  If it is then what should it build?  David?
> >
> >
> > c) Spur VMs need to be linked to by an added line in the Virtual Machines list.
> >
> > Not applicable.  the VM links point to the root of my site so they effectively point to both old and Spur VMs.
> >
> >
> > d) Spur imagers need to be included in the Image and Changes list under Custom Installation
> >
> > Done.
> >
> >
> >
> > e) The SqueakV50.sources file also needs to be included in the Current Sources list.
> >
> > Done.
> >
> >
> > f) we could start a History list for the 5.0 line
> >
> > This is probably quite a big reorg on the page and not urgent.
> >
> >
> > So there are a few things to fix before 5.0 is freely downloadable.
> >
> > On Tue, Aug 11, 2015 at 8:23 PM, Chris Muller <[hidden email]> wrote:
> > In the 17 months since Squeak 4.5 was released, a huge development
> > effort took place to create the next generation virtual-machine for
> > the Squeak / Pharo / Newspeak family of programming systems.  Squeak
> > is the modern incarnation of the Smalltalk-80 programming environment
> > originally developed at the Xerox PARC.
> >
> > "Squeak 5" introduces this new VM and associated new memory model,
> > collectively referred to as "Spur".  Presented [1] by Eliot Miranda
> > and Clément Béra at the 2015 International Symposium on Memory
> > Management, this new VM affords Squeak applications a significant
> > boost in performance and memory management.  Among other
> > optimizations, the #become operation no longer requires a memory scan.
> > Object pinning and ephemerons are also now supported.  The release
> > notes [2] provide more details.
> >
> > The new memory model requires a new image file format.  Although this
> > new format results in about a 15% increased memory requirement for the
> > same number of 4.x objects, a new segmented heap allows memory to be
> > given back to the OS when its no longer needed, a great benefit for
> > application servers.
> >
> > As forward compatibility is as important to the Squeak community as
> > backward compatibility, Squeak 5 is delivers an image with identical
> > content as the recent 4.6 release.  Although this new Squeak 5 VM
> > cannot open images saved under the prior 4.x Cog format, objects and
> > code can be easily exported from the 4.x image and then imported into
> > Squeak 5.  Applications whose code runs strictly above the Smalltalk
> > meta layer will prove remarkably compatible with the new format, most
> > applications will require no changes whatsotever.
> >
> > Squeak 5 is the result of monumental effort by a tiny group of very
> > talented people, but its also just the beginning of yet a new effort;
> > Spur is just a stepping stone to a more ambitious goals planned over
> > the next five years.
> >
> > [1] -- A Partial Read Barrier for Efficient Support of Live
> > Object-oriented Programming
> > http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
> >
> > [2] -- Squeak 5 Release Notes
> > http://wiki.squeak.org/squeak/6207







--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot


Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5]

David T. Lewis
I think that image segments are a worthwhile idea, regardless of whether
we are worried about support project saving and other image conversion
issues. Here is a post by Dan Ingalls from 1999 that summarizes the work
on image segments at that time:

  http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html

Dan also mentioned it in "the future of Squeak, 1999":

  http://wiki.squeak.org/squeak/393

It is quite clear that saving projects was a proof of concept to illustrate
what might be done with image segments, but the overall motivation had more
to do with exploring modularity and mechanisms for delivering minimal images.

Tim Rowledge added a comment to that effect here:

  http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html

I will also note that Spur is not the first time we have needed to think
about making image segments work on a new image format. When Dan Ingalls and
Ian Piumarta announced the first 64-bit Squeak image, they said "We may ask for
help from the Squeak community in converting the remaining plugins, and also
image segment code."

  http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-August/081383.html

Quite a few of the things that Dan and Ian asked for help on have since been
done, but I don't think that image segment support for the 64-bit image was
among them. With the arrival of Spur, and the upcoming Spur 64-bit image, it
would be great if we can put some thought and effort into doing this right.

$0.02,

Dave


On Wed, Aug 19, 2015 at 02:09:32PM -0700, Eliot Miranda wrote:

>  
> Hi All, but especially Tobias,
>
>     as we know, ImageSegments are broken in Spur. They work occasionally,
> but when stressed the VM ends up crashing.  The fundamental issue is that
> the ImageSegment code makes assumptions about object ordering that Spur
> violates.  For example, here's ImageSegment's install:
>
> install
> "This operation retrieves the segment if necessary from file storage,
> installs it in memory, and replaces (using become:) all the root stubs with
> the reconstructed roots of the segment."
>
> | newRoots |
> state = #onFile ifTrue: [self readFromFile].
> state = #onFileWithSymbols ifTrue: [self readFromFileWithSymbols.
> endMarker := segment nextObject. "for enumeration of objects"
> endMarker == 0 ifTrue: [endMarker := 'End' clone]].
> (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
> newRoots := self loadSegmentFrom: segment outPointers: outPointers.
> state = #imported
> ifTrue: ["just came in from exported file"
> arrayOfRoots := newRoots]
> ifFalse: [
> arrayOfRoots elementsForwardIdentityTo: newRoots].
> state := #inactive.
> Beeper beepPrimitive
>
> So before the image segment bytes (the segment inst var) is loaded, the
> object after it is assigned to endMarker, and if there isn't an object
> after segment, a new object ('End' clone) is assigned to endMarker.
>
> This makes the assumption that objects are allocated in a strict order, and
> therefore endMarker will always be the object after segment.
>
> Loading the segment via "newRoots := self loadSegmentFrom: segment
> outPointers: outPointers" then turns segment into a zero-length WordArray
> and its contents into the objects loaded by the segment.  Therefore, in the
> V3 system, the objects loaded from segment can be enumerated starting at
> segment nextObject and repeating until endMarker is found:
>
> allObjectsDo: aBlock
> "Enumerate all objects that came from this segment.  NOTE this assumes that
> the segment was created (and extracted).  After the segment has been
> installed (install), this method allows you to enumerate its objects."
> | obj |
>
> endMarker == nil ifTrue: [
> ^ self error: 'Just extract and install, don''t writeToFile:'].
> segment size ~= 1 ifTrue: [
> ^ self error: 'Vestigial segment size must be 1 (version word)'].
>
> obj := segment nextObject.  "Start with the next object after the vestigial
> header"
> [obj == endMarker] whileFalse:  "Stop at the next object after the full
> segment"
> [aBlock value: obj.
> obj := obj nextObject].  "Step through the objects installed from the
> segment."
>
> Now, as written, this just won't work in Spur.
>
> a) the only place where there is any kind of stable order to objects is in
> oldSpace, so segment /has/ to be forced to old space to have any chance of
> its objects being in order when it gets converted from bytes to objects.
>
> b) 'End' clone will be in newSpace and so endMarker isn't reliable unless
> it was obtained by segment nextObject when segment was already in oldSpace.
>
> So it is perhaps possible to fix ImageSegments in Spur by forcing segment
> to oldSpace and being more careful with endMarker.  But I think there is a
> better way.
>
> If the set of objects the segment contains can be obtained some how then
> this set can be simply enumerated, not depending on nextObject.  The
> primitive has to answer the array of roots, so its result can't be changed
> to be the entire array.  But segment could be becomed into an Array of all
> the objects in segment prior to it being loaded, in which case the above
> would become
>
>
> install
> "This operation retrieves the segment if necessary from file storage,
> installs it in memory, and replaces (using become:) all the root stubs with
> the reconstructed roots of the segment."
>
> | newRoots |
> state = #onFile ifTrue: [self readFromFile].
> state = #onFileWithSymbols ifTrue:
> [self readFromFileWithSymbols].
> (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
> newRoots := self loadSegmentFrom: segment outPointers: outPointers.
> state = #imported
> ifTrue: "just came in from exported file"
> [arrayOfRoots := newRoots]
> ifFalse:
> [arrayOfRoots elementsForwardIdentityTo: newRoots].
> state := #inactive.
> Beeper beepPrimitive
>
> allObjectsDo: aBlock
> "Enumerate all objects that came from this segment.  NOTE this assumes that
> the segment was created (and extracted).  After the segment has been
> installed (install), this method allows you to enumerate its objects."
> | obj |
>
> segment isArray ifFalse:
> [^ self error: 'Segment hasn''t been loaded?'].
>
> segment do: aBlock
>
> and the endMarker instVar would be deleted.
>
> I am willing and ready to modify the primitive to convert the segment
> correctly.  Who will volunteer to rewrite the image-level ImageSegment code
> to use the new primitive?
>
>
>
> On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda <[hidden email]>
> wrote:
>
> > Hi Tobias,
> >
> > On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape <[hidden email]> wrote:
> >
> >> Hi all
> >> On 13.08.2015, at 02:15, Eliot Miranda <[hidden email]> wrote:
> >>
> >> > Hi Tobias,
> >> >
> >> >    forget that. I found it.  THis right?
> >> >
> >> > Trunk test suite for Spur
> >> > Using existing cogspur r.3410
> >> > cp -r /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410
> >> /tmp/d20150812-28620-etbikj
> >> >   image test suite
> >> > VM: /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
> >> > /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
> >> -version
> >> > /var/lib/jenkins/workspace/Trunk/default/tests.st
> >> > spawning command 0 with timeout 1800 seconds:
> >> "/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak"
> >> "-vm-sound-null" "-vm-display-null"
> >> "/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image"
> >> "../tests.st"
> >> > (Command started with PID 28643)
> >> > 2015-08-12T23:31:48.17+01:00: Loading Hudson build tools... from
> >> /var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
> >> > 2015-08-12T23:31:48.388+01:00: Running tests...
> >> > setsockopt: Protocol not available
> >> > setsockopt: Protocol not available
> >> > 28646:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
> >> protocol:s23_clnt.c:612:
> >> >
> >> > Recursive not understood error encountered
> >> >
> >> >
> >> > I bet this is ImageSegment related.
> >>
> >> It sure is.
> >>
> >>         BitmapStreamTests>testMatrixTransform2x3WithImageSegment
> >> sends
> >>         BitmapStreamTests>validateImageSegment
> >>
> >
> >  Basically the ImageSegment code has been working in Spur on a hope and a
> > prayer.  The code assumes objects are allocated in chronological order and
> > that this order is preserved, not so with Spur.  So post image segment
> > loading someObject/nextObject is used to enumerate the objects loaded.
> > This can't work reliably in Spur.  I *think* (ok I hope) that I've
> > implemented the segment load primitive in Spur to answer an Array of the
> > objects loaded, so that these can be explicitly enumerated.
> >
> > So the job is a) to check that I have indeed implemented the primitive to
> > do this and b) to rewrite the image segment loading code in the light of
> > this.
> >
> > David, this is an example of something that isn't back portable and should
> > not be back-ported.
> >
> > The most strange thing is that I cannot reproduce this on my Mac???
> >> There the test does not crash the image???
> >>
> >
> > Well then it may be a signed/unsigned bug in image loading instead.  On
> > linux the image typically gets loaded quite high in the address space and
> > so a good portion of the heap ends up above 0x7FFFFFFF, or negative
> > territory if misinterpreted as signed integers.  On Mac and Windows the
> > image tends to get loaded quite low and so has to be pretty big to fall
> > foul of signedness issues.
> >
> >
> > Busy right now but will check tomorrow.
> >
> >
> >> Best regards
> >>         -Tobias
> >>
> >>
> >> >
> >> > On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda <[hidden email]>
> >> wrote:
> >> > Hi Tobias,
> >> >
> >> > On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape <[hidden email]> wrote:
> >> >
> >> > On 12.08.2015, at 20:55, Eliot Miranda <[hidden email]> wrote:
> >> >
> >> > > Hi All,
> >> > >
> >> > >
> >> > >     Fabio's kindly done most of the changes.  But some questions
> >> remain for more general discussion.  See below.
> >> > >
> >> > > Fabio,  thanks so much for doing this, and so quickly!
> >> > >
> >> > > On Tue, Aug 11, 2015 at 8:51 PM, Eliot Miranda <
> >> [hidden email]> wrote:
> >> > > Hi All,
> >> > >
> >> > >     who will update http://squeak.org/downloads/ to include the 5.0
> >> release?
> >> > >
> >> > > a) I suggest that the 5.0 all-in-one have a line in the left-hand
> >> table that includes the 4.6 release and that it precede the 4.6 release in
> >> the list.
> >> > >
> >> > > Done.
> >> > >
> >> > >
> >> > > b) the Trunk link points to TrunkImage.zip which contains a non-Spur
> >> image.
> >> > >
> >> > > So this is the biggie.  What should we do?
> >> > >
> >> > > - add a Trunk 5.0 build with a different link?  (Noooooo)
> >> > >
> >> > > - change  http://build.squeak.org/job/SqueakTrunk to build what it
> >> says (Yessss please, who can do this?)
> >> >
> >> > Done: build.squeak.org/job/Trunk/
> >> > But as I said several times, Spur/trunk test just crash for month???
> >> >
> >> > can you point me to the crash?  I'm looking at
> >> http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink
> >> and see 8 test failures but no crash.
> >> >
> >> >
> >> > >
> >> > > - add a Squeak 4.6 build job?  Is that even worth it any more
> >> considering 4.6 is released and stable?  If it is then what should it
> >> build?  David?
> >> > >
> >> > >
> >> > > c) Spur VMs need to be linked to by an added line in the Virtual
> >> Machines list.
> >> > >
> >> > > Not applicable.  the VM links point to the root of my site so they
> >> effectively point to both old and Spur VMs.
> >> > >
> >> > >
> >> > > d) Spur imagers need to be included in the Image and Changes list
> >> under Custom Installation
> >> > >
> >> > > Done.
> >> > >
> >> > >
> >> > >
> >> > > e) The SqueakV50.sources file also needs to be included in the
> >> Current Sources list.
> >> > >
> >> > > Done.
> >> > >
> >> > >
> >> > > f) we could start a History list for the 5.0 line
> >> > >
> >> > > This is probably quite a big reorg on the page and not urgent.
> >> > >
> >> > >
> >> > > So there are a few things to fix before 5.0 is freely downloadable.
> >> > >
> >> > > On Tue, Aug 11, 2015 at 8:23 PM, Chris Muller <[hidden email]>
> >> wrote:
> >> > > In the 17 months since Squeak 4.5 was released, a huge development
> >> > > effort took place to create the next generation virtual-machine for
> >> > > the Squeak / Pharo / Newspeak family of programming systems.  Squeak
> >> > > is the modern incarnation of the Smalltalk-80 programming environment
> >> > > originally developed at the Xerox PARC.
> >> > >
> >> > > "Squeak 5" introduces this new VM and associated new memory model,
> >> > > collectively referred to as "Spur".  Presented [1] by Eliot Miranda
> >> > > and Cl??ment B??ra at the 2015 International Symposium on Memory
> >> > > Management, this new VM affords Squeak applications a significant
> >> > > boost in performance and memory management.  Among other
> >> > > optimizations, the #become operation no longer requires a memory scan.
> >> > > Object pinning and ephemerons are also now supported.  The release
> >> > > notes [2] provide more details.
> >> > >
> >> > > The new memory model requires a new image file format.  Although this
> >> > > new format results in about a 15% increased memory requirement for the
> >> > > same number of 4.x objects, a new segmented heap allows memory to be
> >> > > given back to the OS when its no longer needed, a great benefit for
> >> > > application servers.
> >> > >
> >> > > As forward compatibility is as important to the Squeak community as
> >> > > backward compatibility, Squeak 5 is delivers an image with identical
> >> > > content as the recent 4.6 release.  Although this new Squeak 5 VM
> >> > > cannot open images saved under the prior 4.x Cog format, objects and
> >> > > code can be easily exported from the 4.x image and then imported into
> >> > > Squeak 5.  Applications whose code runs strictly above the Smalltalk
> >> > > meta layer will prove remarkably compatible with the new format, most
> >> > > applications will require no changes whatsotever.
> >> > >
> >> > > Squeak 5 is the result of monumental effort by a tiny group of very
> >> > > talented people, but its also just the beginning of yet a new effort;
> >> > > Spur is just a stepping stone to a more ambitious goals planned over
> >> > > the next five years.
> >> > >
> >> > > [1] -- A Partial Read Barrier for Efficient Support of Live
> >> > > Object-oriented Programming
> >> > >
> >> http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
> >> > >
> >> > > [2] -- Squeak 5 Release Notes
> >> > > http://wiki.squeak.org/squeak/6207
> >>
> >>
> >>
> >>
> >>
> >
> >
> > --
> > _,,,^..^,,,_
> > best, Eliot
> >
>
>
>
> --
> _,,,^..^,,,_
> best, Eliot


Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5]

Chris Cunnington-4
As I've never seen (even on the wiki) an even halfway decent
demonstration of using an ImageSegment, I'm providing on here.

     hex := Browser allInstances first.

     "let's check what we're saving to compare with later"
     (hex buildWith: ToolBuilder default) openInWorld

     exporting := (ImageSegment new copyFromRootsForExport: (Array with:
hex)).

     “let’s put it on disk”
     exporting  writeForExport: 'browser.extSeg'.

     "Quit your image without saving. Actually, to get the full effect
go to http://ftp.squeak.org and get a fresh image of the same kind you
exported with. Drag browser.extSeg into the new image directory"

     “let’s pull it in from the disk”
     importing := (FileDirectory default readOnlyFileNamed:
'browser.extSeg') fileInObjectAndCode.

     "let's check what we imported to see if it's what we saved"
     ((importing originalRoots first) buildWith: ToolBuilder default)
openInWorld

Chris


On 2015-08-19 8:12 PM, David T. Lewis wrote:

> I think that image segments are a worthwhile idea, regardless of whether
> we are worried about support project saving and other image conversion
> issues. Here is a post by Dan Ingalls from 1999 that summarizes the work
> on image segments at that time:
>
>    http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html
>
> Dan also mentioned it in "the future of Squeak, 1999":
>
>    http://wiki.squeak.org/squeak/393
>
> It is quite clear that saving projects was a proof of concept to illustrate
> what might be done with image segments, but the overall motivation had more
> to do with exploring modularity and mechanisms for delivering minimal images.
>
> Tim Rowledge added a comment to that effect here:
>
>    http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html
>
> I will also note that Spur is not the first time we have needed to think
> about making image segments work on a new image format. When Dan Ingalls and
> Ian Piumarta announced the first 64-bit Squeak image, they said "We may ask for
> help from the Squeak community in converting the remaining plugins, and also
> image segment code."
>
>    http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-August/081383.html
>
> Quite a few of the things that Dan and Ian asked for help on have since been
> done, but I don't think that image segment support for the 64-bit image was
> among them. With the arrival of Spur, and the upcoming Spur 64-bit image, it
> would be great if we can put some thought and effort into doing this right.
>
> $0.02,
>
> Dave
>
>
> On Wed, Aug 19, 2015 at 02:09:32PM -0700, Eliot Miranda wrote:
>>  
>> Hi All, but especially Tobias,
>>
>>      as we know, ImageSegments are broken in Spur. They work occasionally,
>> but when stressed the VM ends up crashing.  The fundamental issue is that
>> the ImageSegment code makes assumptions about object ordering that Spur
>> violates.  For example, here's ImageSegment's install:
>>
>> install
>> "This operation retrieves the segment if necessary from file storage,
>> installs it in memory, and replaces (using become:) all the root stubs with
>> the reconstructed roots of the segment."
>>
>> | newRoots |
>> state = #onFile ifTrue: [self readFromFile].
>> state = #onFileWithSymbols ifTrue: [self readFromFileWithSymbols.
>> endMarker := segment nextObject. "for enumeration of objects"
>> endMarker == 0 ifTrue: [endMarker := 'End' clone]].
>> (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
>> newRoots := self loadSegmentFrom: segment outPointers: outPointers.
>> state = #imported
>> ifTrue: ["just came in from exported file"
>> arrayOfRoots := newRoots]
>> ifFalse: [
>> arrayOfRoots elementsForwardIdentityTo: newRoots].
>> state := #inactive.
>> Beeper beepPrimitive
>>
>> So before the image segment bytes (the segment inst var) is loaded, the
>> object after it is assigned to endMarker, and if there isn't an object
>> after segment, a new object ('End' clone) is assigned to endMarker.
>>
>> This makes the assumption that objects are allocated in a strict order, and
>> therefore endMarker will always be the object after segment.
>>
>> Loading the segment via "newRoots := self loadSegmentFrom: segment
>> outPointers: outPointers" then turns segment into a zero-length WordArray
>> and its contents into the objects loaded by the segment.  Therefore, in the
>> V3 system, the objects loaded from segment can be enumerated starting at
>> segment nextObject and repeating until endMarker is found:
>>
>> allObjectsDo: aBlock
>> "Enumerate all objects that came from this segment.  NOTE this assumes that
>> the segment was created (and extracted).  After the segment has been
>> installed (install), this method allows you to enumerate its objects."
>> | obj |
>>
>> endMarker == nil ifTrue: [
>> ^ self error: 'Just extract and install, don''t writeToFile:'].
>> segment size ~= 1 ifTrue: [
>> ^ self error: 'Vestigial segment size must be 1 (version word)'].
>>
>> obj := segment nextObject.  "Start with the next object after the vestigial
>> header"
>> [obj == endMarker] whileFalse:  "Stop at the next object after the full
>> segment"
>> [aBlock value: obj.
>> obj := obj nextObject].  "Step through the objects installed from the
>> segment."
>>
>> Now, as written, this just won't work in Spur.
>>
>> a) the only place where there is any kind of stable order to objects is in
>> oldSpace, so segment /has/ to be forced to old space to have any chance of
>> its objects being in order when it gets converted from bytes to objects.
>>
>> b) 'End' clone will be in newSpace and so endMarker isn't reliable unless
>> it was obtained by segment nextObject when segment was already in oldSpace.
>>
>> So it is perhaps possible to fix ImageSegments in Spur by forcing segment
>> to oldSpace and being more careful with endMarker.  But I think there is a
>> better way.
>>
>> If the set of objects the segment contains can be obtained some how then
>> this set can be simply enumerated, not depending on nextObject.  The
>> primitive has to answer the array of roots, so its result can't be changed
>> to be the entire array.  But segment could be becomed into an Array of all
>> the objects in segment prior to it being loaded, in which case the above
>> would become
>>
>>
>> install
>> "This operation retrieves the segment if necessary from file storage,
>> installs it in memory, and replaces (using become:) all the root stubs with
>> the reconstructed roots of the segment."
>>
>> | newRoots |
>> state = #onFile ifTrue: [self readFromFile].
>> state = #onFileWithSymbols ifTrue:
>> [self readFromFileWithSymbols].
>> (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
>> newRoots := self loadSegmentFrom: segment outPointers: outPointers.
>> state = #imported
>> ifTrue: "just came in from exported file"
>> [arrayOfRoots := newRoots]
>> ifFalse:
>> [arrayOfRoots elementsForwardIdentityTo: newRoots].
>> state := #inactive.
>> Beeper beepPrimitive
>>
>> allObjectsDo: aBlock
>> "Enumerate all objects that came from this segment.  NOTE this assumes that
>> the segment was created (and extracted).  After the segment has been
>> installed (install), this method allows you to enumerate its objects."
>> | obj |
>>
>> segment isArray ifFalse:
>> [^ self error: 'Segment hasn''t been loaded?'].
>>
>> segment do: aBlock
>>
>> and the endMarker instVar would be deleted.
>>
>> I am willing and ready to modify the primitive to convert the segment
>> correctly.  Who will volunteer to rewrite the image-level ImageSegment code
>> to use the new primitive?
>>
>>
>>
>> On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda <[hidden email]>
>> wrote:
>>
>>> Hi Tobias,
>>>
>>> On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape <[hidden email]> wrote:
>>>
>>>> Hi all
>>>> On 13.08.2015, at 02:15, Eliot Miranda <[hidden email]> wrote:
>>>>
>>>>> Hi Tobias,
>>>>>
>>>>>     forget that. I found it.  THis right?
>>>>>
>>>>> Trunk test suite for Spur
>>>>> Using existing cogspur r.3410
>>>>> cp -r /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410
>>>> /tmp/d20150812-28620-etbikj
>>>>>    image test suite
>>>>> VM: /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
>>>>> /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
>>>> -version
>>>>> /var/lib/jenkins/workspace/Trunk/default/tests.st
>>>>> spawning command 0 with timeout 1800 seconds:
>>>> "/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak"
>>>> "-vm-sound-null" "-vm-display-null"
>>>> "/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image"
>>>> "../tests.st"
>>>>> (Command started with PID 28643)
>>>>> 2015-08-12T23:31:48.17+01:00: Loading Hudson build tools... from
>>>> /var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
>>>>> 2015-08-12T23:31:48.388+01:00: Running tests...
>>>>> setsockopt: Protocol not available
>>>>> setsockopt: Protocol not available
>>>>> 28646:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
>>>> protocol:s23_clnt.c:612:
>>>>> Recursive not understood error encountered
>>>>>
>>>>>
>>>>> I bet this is ImageSegment related.
>>>> It sure is.
>>>>
>>>>          BitmapStreamTests>testMatrixTransform2x3WithImageSegment
>>>> sends
>>>>          BitmapStreamTests>validateImageSegment
>>>>
>>>   Basically the ImageSegment code has been working in Spur on a hope and a
>>> prayer.  The code assumes objects are allocated in chronological order and
>>> that this order is preserved, not so with Spur.  So post image segment
>>> loading someObject/nextObject is used to enumerate the objects loaded.
>>> This can't work reliably in Spur.  I *think* (ok I hope) that I've
>>> implemented the segment load primitive in Spur to answer an Array of the
>>> objects loaded, so that these can be explicitly enumerated.
>>>
>>> So the job is a) to check that I have indeed implemented the primitive to
>>> do this and b) to rewrite the image segment loading code in the light of
>>> this.
>>>
>>> David, this is an example of something that isn't back portable and should
>>> not be back-ported.
>>>
>>> The most strange thing is that I cannot reproduce this on my Mac???
>>>> There the test does not crash the image???
>>>>
>>> Well then it may be a signed/unsigned bug in image loading instead.  On
>>> linux the image typically gets loaded quite high in the address space and
>>> so a good portion of the heap ends up above 0x7FFFFFFF, or negative
>>> territory if misinterpreted as signed integers.  On Mac and Windows the
>>> image tends to get loaded quite low and so has to be pretty big to fall
>>> foul of signedness issues.
>>>
>>>
>>> Busy right now but will check tomorrow.
>>>
>>>
>>>> Best regards
>>>>          -Tobias
>>>>
>>>>
>>>>> On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda <[hidden email]>
>>>> wrote:
>>>>> Hi Tobias,
>>>>>
>>>>> On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape <[hidden email]> wrote:
>>>>>
>>>>> On 12.08.2015, at 20:55, Eliot Miranda <[hidden email]> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>>
>>>>>>      Fabio's kindly done most of the changes.  But some questions
>>>> remain for more general discussion.  See below.
>>>>>> Fabio,  thanks so much for doing this, and so quickly!
>>>>>>
>>>>>> On Tue, Aug 11, 2015 at 8:51 PM, Eliot Miranda <
>>>> [hidden email]> wrote:
>>>>>> Hi All,
>>>>>>
>>>>>>      who will update http://squeak.org/downloads/ to include the 5.0
>>>> release?
>>>>>> a) I suggest that the 5.0 all-in-one have a line in the left-hand
>>>> table that includes the 4.6 release and that it precede the 4.6 release in
>>>> the list.
>>>>>> Done.
>>>>>>
>>>>>>
>>>>>> b) the Trunk link points to TrunkImage.zip which contains a non-Spur
>>>> image.
>>>>>> So this is the biggie.  What should we do?
>>>>>>
>>>>>> - add a Trunk 5.0 build with a different link?  (Noooooo)
>>>>>>
>>>>>> - change  http://build.squeak.org/job/SqueakTrunk to build what it
>>>> says (Yessss please, who can do this?)
>>>>> Done: build.squeak.org/job/Trunk/
>>>>> But as I said several times, Spur/trunk test just crash for month???
>>>>>
>>>>> can you point me to the crash?  I'm looking at
>>>> http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink
>>>> and see 8 test failures but no crash.
>>>>>
>>>>>> - add a Squeak 4.6 build job?  Is that even worth it any more
>>>> considering 4.6 is released and stable?  If it is then what should it
>>>> build?  David?
>>>>>>
>>>>>> c) Spur VMs need to be linked to by an added line in the Virtual
>>>> Machines list.
>>>>>> Not applicable.  the VM links point to the root of my site so they
>>>> effectively point to both old and Spur VMs.
>>>>>>
>>>>>> d) Spur imagers need to be included in the Image and Changes list
>>>> under Custom Installation
>>>>>> Done.
>>>>>>
>>>>>>
>>>>>>
>>>>>> e) The SqueakV50.sources file also needs to be included in the
>>>> Current Sources list.
>>>>>> Done.
>>>>>>
>>>>>>
>>>>>> f) we could start a History list for the 5.0 line
>>>>>>
>>>>>> This is probably quite a big reorg on the page and not urgent.
>>>>>>
>>>>>>
>>>>>> So there are a few things to fix before 5.0 is freely downloadable.
>>>>>>
>>>>>> On Tue, Aug 11, 2015 at 8:23 PM, Chris Muller <[hidden email]>
>>>> wrote:
>>>>>> In the 17 months since Squeak 4.5 was released, a huge development
>>>>>> effort took place to create the next generation virtual-machine for
>>>>>> the Squeak / Pharo / Newspeak family of programming systems.  Squeak
>>>>>> is the modern incarnation of the Smalltalk-80 programming environment
>>>>>> originally developed at the Xerox PARC.
>>>>>>
>>>>>> "Squeak 5" introduces this new VM and associated new memory model,
>>>>>> collectively referred to as "Spur".  Presented [1] by Eliot Miranda
>>>>>> and Cl??ment B??ra at the 2015 International Symposium on Memory
>>>>>> Management, this new VM affords Squeak applications a significant
>>>>>> boost in performance and memory management.  Among other
>>>>>> optimizations, the #become operation no longer requires a memory scan.
>>>>>> Object pinning and ephemerons are also now supported.  The release
>>>>>> notes [2] provide more details.
>>>>>>
>>>>>> The new memory model requires a new image file format.  Although this
>>>>>> new format results in about a 15% increased memory requirement for the
>>>>>> same number of 4.x objects, a new segmented heap allows memory to be
>>>>>> given back to the OS when its no longer needed, a great benefit for
>>>>>> application servers.
>>>>>>
>>>>>> As forward compatibility is as important to the Squeak community as
>>>>>> backward compatibility, Squeak 5 is delivers an image with identical
>>>>>> content as the recent 4.6 release.  Although this new Squeak 5 VM
>>>>>> cannot open images saved under the prior 4.x Cog format, objects and
>>>>>> code can be easily exported from the 4.x image and then imported into
>>>>>> Squeak 5.  Applications whose code runs strictly above the Smalltalk
>>>>>> meta layer will prove remarkably compatible with the new format, most
>>>>>> applications will require no changes whatsotever.
>>>>>>
>>>>>> Squeak 5 is the result of monumental effort by a tiny group of very
>>>>>> talented people, but its also just the beginning of yet a new effort;
>>>>>> Spur is just a stepping stone to a more ambitious goals planned over
>>>>>> the next five years.
>>>>>>
>>>>>> [1] -- A Partial Read Barrier for Efficient Support of Live
>>>>>> Object-oriented Programming
>>>>>>
>>>> http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
>>>>>> [2] -- Squeak 5 Release Notes
>>>>>> http://wiki.squeak.org/squeak/6207
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> _,,,^..^,,,_
>>> best, Eliot
>>>
>>
>>
>> --
>> _,,,^..^,,,_
>> best, Eliot
>


Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5]

Mariano Martinez Peck
Hi guys, 

As part of my PhD thesis I did take a deep look to ImageSegment before starting with Fuel. I wrote a journal paper about my experiments with ImageSegment which I thinks provides quite a documentation that is not written anywhere. Hope it helps: http://rmod.lille.inria.fr/archives/papers/Mart11c-COMLAN-ObjectSwapping.pdf

Best, 



On Wed, Aug 19, 2015 at 10:18 PM, Chris Cunnington <[hidden email]> wrote:
As I've never seen (even on the wiki) an even halfway decent demonstration of using an ImageSegment, I'm providing on here.

    hex := Browser allInstances first.

    "let's check what we're saving to compare with later"
    (hex buildWith: ToolBuilder default) openInWorld

    exporting := (ImageSegment new copyFromRootsForExport: (Array with: hex)).

    “let’s put it on disk”
    exporting  writeForExport: 'browser.extSeg'.

    "Quit your image without saving. Actually, to get the full effect go to http://ftp.squeak.org and get a fresh image of the same kind you exported with. Drag browser.extSeg into the new image directory"

    “let’s pull it in from the disk”
    importing := (FileDirectory default readOnlyFileNamed: 'browser.extSeg') fileInObjectAndCode.

    "let's check what we imported to see if it's what we saved"
    ((importing originalRoots first) buildWith: ToolBuilder default) openInWorld

Chris



On 2015-08-19 8:12 PM, David T. Lewis wrote:
I think that image segments are a worthwhile idea, regardless of whether
we are worried about support project saving and other image conversion
issues. Here is a post by Dan Ingalls from 1999 that summarizes the work
on image segments at that time:

   http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html

Dan also mentioned it in "the future of Squeak, 1999":

   http://wiki.squeak.org/squeak/393

It is quite clear that saving projects was a proof of concept to illustrate
what might be done with image segments, but the overall motivation had more
to do with exploring modularity and mechanisms for delivering minimal images.

Tim Rowledge added a comment to that effect here:

   http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html

I will also note that Spur is not the first time we have needed to think
about making image segments work on a new image format. When Dan Ingalls and
Ian Piumarta announced the first 64-bit Squeak image, they said "We may ask for
help from the Squeak community in converting the remaining plugins, and also
image segment code."

   http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-August/081383.html

Quite a few of the things that Dan and Ian asked for help on have since been
done, but I don't think that image segment support for the 64-bit image was
among them. With the arrival of Spur, and the upcoming Spur 64-bit image, it
would be great if we can put some thought and effort into doing this right.

$0.02,

Dave


On Wed, Aug 19, 2015 at 02:09:32PM -0700, Eliot Miranda wrote:
  Hi All, but especially Tobias,

     as we know, ImageSegments are broken in Spur. They work occasionally,
but when stressed the VM ends up crashing.  The fundamental issue is that
the ImageSegment code makes assumptions about object ordering that Spur
violates.  For example, here's ImageSegment's install:

install
"This operation retrieves the segment if necessary from file storage,
installs it in memory, and replaces (using become:) all the root stubs with
the reconstructed roots of the segment."

| newRoots |
state = #onFile ifTrue: [self readFromFile].
state = #onFileWithSymbols ifTrue: [self readFromFileWithSymbols.
endMarker := segment nextObject. "for enumeration of objects"
endMarker == 0 ifTrue: [endMarker := 'End' clone]].
(state = #active) | (state = #imported) ifFalse: [self errorWrongState].
newRoots := self loadSegmentFrom: segment outPointers: outPointers.
state = #imported
ifTrue: ["just came in from exported file"
arrayOfRoots := newRoots]
ifFalse: [
arrayOfRoots elementsForwardIdentityTo: newRoots].
state := #inactive.
Beeper beepPrimitive

So before the image segment bytes (the segment inst var) is loaded, the
object after it is assigned to endMarker, and if there isn't an object
after segment, a new object ('End' clone) is assigned to endMarker.

This makes the assumption that objects are allocated in a strict order, and
therefore endMarker will always be the object after segment.

Loading the segment via "newRoots := self loadSegmentFrom: segment
outPointers: outPointers" then turns segment into a zero-length WordArray
and its contents into the objects loaded by the segment.  Therefore, in the
V3 system, the objects loaded from segment can be enumerated starting at
segment nextObject and repeating until endMarker is found:

allObjectsDo: aBlock
"Enumerate all objects that came from this segment.  NOTE this assumes that
the segment was created (and extracted).  After the segment has been
installed (install), this method allows you to enumerate its objects."
| obj |

endMarker == nil ifTrue: [
^ self error: 'Just extract and install, don''t writeToFile:'].
segment size ~= 1 ifTrue: [
^ self error: 'Vestigial segment size must be 1 (version word)'].

obj := segment nextObject.  "Start with the next object after the vestigial
header"
[obj == endMarker] whileFalse:  "Stop at the next object after the full
segment"
[aBlock value: obj.
obj := obj nextObject].  "Step through the objects installed from the
segment."

Now, as written, this just won't work in Spur.

a) the only place where there is any kind of stable order to objects is in
oldSpace, so segment /has/ to be forced to old space to have any chance of
its objects being in order when it gets converted from bytes to objects.

b) 'End' clone will be in newSpace and so endMarker isn't reliable unless
it was obtained by segment nextObject when segment was already in oldSpace.

So it is perhaps possible to fix ImageSegments in Spur by forcing segment
to oldSpace and being more careful with endMarker.  But I think there is a
better way.

If the set of objects the segment contains can be obtained some how then
this set can be simply enumerated, not depending on nextObject.  The
primitive has to answer the array of roots, so its result can't be changed
to be the entire array.  But segment could be becomed into an Array of all
the objects in segment prior to it being loaded, in which case the above
would become


install
"This operation retrieves the segment if necessary from file storage,
installs it in memory, and replaces (using become:) all the root stubs with
the reconstructed roots of the segment."

| newRoots |
state = #onFile ifTrue: [self readFromFile].
state = #onFileWithSymbols ifTrue:
[self readFromFileWithSymbols].
(state = #active) | (state = #imported) ifFalse: [self errorWrongState].
newRoots := self loadSegmentFrom: segment outPointers: outPointers.
state = #imported
ifTrue: "just came in from exported file"
[arrayOfRoots := newRoots]
ifFalse:
[arrayOfRoots elementsForwardIdentityTo: newRoots].
state := #inactive.
Beeper beepPrimitive

allObjectsDo: aBlock
"Enumerate all objects that came from this segment.  NOTE this assumes that
the segment was created (and extracted).  After the segment has been
installed (install), this method allows you to enumerate its objects."
| obj |

segment isArray ifFalse:
[^ self error: 'Segment hasn''t been loaded?'].

segment do: aBlock

and the endMarker instVar would be deleted.

I am willing and ready to modify the primitive to convert the segment
correctly.  Who will volunteer to rewrite the image-level ImageSegment code
to use the new primitive?



On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda <[hidden email]>
wrote:

Hi Tobias,

On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape <[hidden email]> wrote:

Hi all
On 13.08.2015, at 02:15, Eliot Miranda <[hidden email]> wrote:

Hi Tobias,

    forget that. I found it.  THis right?

Trunk test suite for Spur
Using existing cogspur r.3410
cp -r /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410
/tmp/d20150812-28620-etbikj
   image test suite
VM: /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
-version
/var/lib/jenkins/workspace/Trunk/default/tests.st
spawning command 0 with timeout 1800 seconds:
"/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak"
"-vm-sound-null" "-vm-display-null"
"/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image"
"../tests.st"
(Command started with PID 28643)
2015-08-12T23:31:48.17+01:00: Loading Hudson build tools... from
/var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
2015-08-12T23:31:48.388+01:00: Running tests...
setsockopt: Protocol not available
setsockopt: Protocol not available
28646:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
protocol:s23_clnt.c:612:
Recursive not understood error encountered


I bet this is ImageSegment related.
It sure is.

         BitmapStreamTests>testMatrixTransform2x3WithImageSegment
sends
         BitmapStreamTests>validateImageSegment

  Basically the ImageSegment code has been working in Spur on a hope and a
prayer.  The code assumes objects are allocated in chronological order and
that this order is preserved, not so with Spur.  So post image segment
loading someObject/nextObject is used to enumerate the objects loaded.
This can't work reliably in Spur.  I *think* (ok I hope) that I've
implemented the segment load primitive in Spur to answer an Array of the
objects loaded, so that these can be explicitly enumerated.

So the job is a) to check that I have indeed implemented the primitive to
do this and b) to rewrite the image segment loading code in the light of
this.

David, this is an example of something that isn't back portable and should
not be back-ported.

The most strange thing is that I cannot reproduce this on my Mac???
There the test does not crash the image???

Well then it may be a signed/unsigned bug in image loading instead.  On
linux the image typically gets loaded quite high in the address space and
so a good portion of the heap ends up above 0x7FFFFFFF, or negative
territory if misinterpreted as signed integers.  On Mac and Windows the
image tends to get loaded quite low and so has to be pretty big to fall
foul of signedness issues.


Busy right now but will check tomorrow.


Best regards
         -Tobias


On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda <[hidden email]>
wrote:
Hi Tobias,

On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape <[hidden email]> wrote:

On 12.08.2015, at 20:55, Eliot Miranda <[hidden email]> wrote:

Hi All,


     Fabio's kindly done most of the changes.  But some questions
remain for more general discussion.  See below.
Fabio,  thanks so much for doing this, and so quickly!

On Tue, Aug 11, 2015 at 8:51 PM, Eliot Miranda <
[hidden email]> wrote:
Hi All,

     who will update http://squeak.org/downloads/ to include the 5.0
release?
a) I suggest that the 5.0 all-in-one have a line in the left-hand
table that includes the 4.6 release and that it precede the 4.6 release in
the list.
Done.


b) the Trunk link points to TrunkImage.zip which contains a non-Spur
image.
So this is the biggie.  What should we do?

- add a Trunk 5.0 build with a different link?  (Noooooo)

- change  http://build.squeak.org/job/SqueakTrunk to build what it
says (Yessss please, who can do this?)
Done: build.squeak.org/job/Trunk/
But as I said several times, Spur/trunk test just crash for month???

can you point me to the crash?  I'm looking at
http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink
and see 8 test failures but no crash.

- add a Squeak 4.6 build job?  Is that even worth it any more
considering 4.6 is released and stable?  If it is then what should it
build?  David?

c) Spur VMs need to be linked to by an added line in the Virtual
Machines list.
Not applicable.  the VM links point to the root of my site so they
effectively point to both old and Spur VMs.

d) Spur imagers need to be included in the Image and Changes list
under Custom Installation
Done.



e) The SqueakV50.sources file also needs to be included in the
Current Sources list.
Done.


f) we could start a History list for the 5.0 line

This is probably quite a big reorg on the page and not urgent.


So there are a few things to fix before 5.0 is freely downloadable.

On Tue, Aug 11, 2015 at 8:23 PM, Chris Muller <[hidden email]>
wrote:
In the 17 months since Squeak 4.5 was released, a huge development
effort took place to create the next generation virtual-machine for
the Squeak / Pharo / Newspeak family of programming systems.  Squeak
is the modern incarnation of the Smalltalk-80 programming environment
originally developed at the Xerox PARC.

"Squeak 5" introduces this new VM and associated new memory model,
collectively referred to as "Spur".  Presented [1] by Eliot Miranda
and Cl??ment B??ra at the 2015 International Symposium on Memory
Management, this new VM affords Squeak applications a significant
boost in performance and memory management.  Among other
optimizations, the #become operation no longer requires a memory scan.
Object pinning and ephemerons are also now supported.  The release
notes [2] provide more details.

The new memory model requires a new image file format.  Although this
new format results in about a 15% increased memory requirement for the
same number of 4.x objects, a new segmented heap allows memory to be
given back to the OS when its no longer needed, a great benefit for
application servers.

As forward compatibility is as important to the Squeak community as
backward compatibility, Squeak 5 is delivers an image with identical
content as the recent 4.6 release.  Although this new Squeak 5 VM
cannot open images saved under the prior 4.x Cog format, objects and
code can be easily exported from the 4.x image and then imported into
Squeak 5.  Applications whose code runs strictly above the Smalltalk
meta layer will prove remarkably compatible with the new format, most
applications will require no changes whatsotever.

Squeak 5 is the result of monumental effort by a tiny group of very
talented people, but its also just the beginning of yet a new effort;
Spur is just a stepping stone to a more ambitious goals planned over
the next five years.

[1] -- A Partial Read Barrier for Efficient Support of Live
Object-oriented Programming

http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
[2] -- Squeak 5 Release Notes
http://wiki.squeak.org/squeak/6207





--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot






--


Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5]

Chris Cunnington-4
Sweet.

Chris

On 2015-08-19 9:31 PM, Mariano Martinez Peck wrote:
Hi guys, 

As part of my PhD thesis I did take a deep look to ImageSegment before starting with Fuel. I wrote a journal paper about my experiments with ImageSegment which I thinks provides quite a documentation that is not written anywhere. Hope it helps: http://rmod.lille.inria.fr/archives/papers/Mart11c-COMLAN-ObjectSwapping.pdf

Best, 



On Wed, Aug 19, 2015 at 10:18 PM, Chris Cunnington <[hidden email]> wrote:
As I've never seen (even on the wiki) an even halfway decent demonstration of using an ImageSegment, I'm providing on here.

    hex := Browser allInstances first.

    "let's check what we're saving to compare with later"
    (hex buildWith: ToolBuilder default) openInWorld

    exporting := (ImageSegment new copyFromRootsForExport: (Array with: hex)).

    “let’s put it on disk”
    exporting  writeForExport: 'browser.extSeg'.

    "Quit your image without saving. Actually, to get the full effect go to http://ftp.squeak.org and get a fresh image of the same kind you exported with. Drag browser.extSeg into the new image directory"

    “let’s pull it in from the disk”
    importing := (FileDirectory default readOnlyFileNamed: 'browser.extSeg') fileInObjectAndCode.

    "let's check what we imported to see if it's what we saved"
    ((importing originalRoots first) buildWith: ToolBuilder default) openInWorld

Chris



On 2015-08-19 8:12 PM, David T. Lewis wrote:
I think that image segments are a worthwhile idea, regardless of whether
we are worried about support project saving and other image conversion
issues. Here is a post by Dan Ingalls from 1999 that summarizes the work
on image segments at that time:

   http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html

Dan also mentioned it in "the future of Squeak, 1999":

   http://wiki.squeak.org/squeak/393

It is quite clear that saving projects was a proof of concept to illustrate
what might be done with image segments, but the overall motivation had more
to do with exploring modularity and mechanisms for delivering minimal images.

Tim Rowledge added a comment to that effect here:

   http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html

I will also note that Spur is not the first time we have needed to think
about making image segments work on a new image format. When Dan Ingalls and
Ian Piumarta announced the first 64-bit Squeak image, they said "We may ask for
help from the Squeak community in converting the remaining plugins, and also
image segment code."

   http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-August/081383.html

Quite a few of the things that Dan and Ian asked for help on have since been
done, but I don't think that image segment support for the 64-bit image was
among them. With the arrival of Spur, and the upcoming Spur 64-bit image, it
would be great if we can put some thought and effort into doing this right.

$0.02,

Dave


On Wed, Aug 19, 2015 at 02:09:32PM -0700, Eliot Miranda wrote:
  Hi All, but especially Tobias,

     as we know, ImageSegments are broken in Spur. They work occasionally,
but when stressed the VM ends up crashing.  The fundamental issue is that
the ImageSegment code makes assumptions about object ordering that Spur
violates.  For example, here's ImageSegment's install:

install
"This operation retrieves the segment if necessary from file storage,
installs it in memory, and replaces (using become:) all the root stubs with
the reconstructed roots of the segment."

| newRoots |
state = #onFile ifTrue: [self readFromFile].
state = #onFileWithSymbols ifTrue: [self readFromFileWithSymbols.
endMarker := segment nextObject. "for enumeration of objects"
endMarker == 0 ifTrue: [endMarker := 'End' clone]].
(state = #active) | (state = #imported) ifFalse: [self errorWrongState].
newRoots := self loadSegmentFrom: segment outPointers: outPointers.
state = #imported
ifTrue: ["just came in from exported file"
arrayOfRoots := newRoots]
ifFalse: [
arrayOfRoots elementsForwardIdentityTo: newRoots].
state := #inactive.
Beeper beepPrimitive

So before the image segment bytes (the segment inst var) is loaded, the
object after it is assigned to endMarker, and if there isn't an object
after segment, a new object ('End' clone) is assigned to endMarker.

This makes the assumption that objects are allocated in a strict order, and
therefore endMarker will always be the object after segment.

Loading the segment via "newRoots := self loadSegmentFrom: segment
outPointers: outPointers" then turns segment into a zero-length WordArray
and its contents into the objects loaded by the segment.  Therefore, in the
V3 system, the objects loaded from segment can be enumerated starting at
segment nextObject and repeating until endMarker is found:

allObjectsDo: aBlock
"Enumerate all objects that came from this segment.  NOTE this assumes that
the segment was created (and extracted).  After the segment has been
installed (install), this method allows you to enumerate its objects."
| obj |

endMarker == nil ifTrue: [
^ self error: 'Just extract and install, don''t writeToFile:'].
segment size ~= 1 ifTrue: [
^ self error: 'Vestigial segment size must be 1 (version word)'].

obj := segment nextObject.  "Start with the next object after the vestigial
header"
[obj == endMarker] whileFalse:  "Stop at the next object after the full
segment"
[aBlock value: obj.
obj := obj nextObject].  "Step through the objects installed from the
segment."

Now, as written, this just won't work in Spur.

a) the only place where there is any kind of stable order to objects is in
oldSpace, so segment /has/ to be forced to old space to have any chance of
its objects being in order when it gets converted from bytes to objects.

b) 'End' clone will be in newSpace and so endMarker isn't reliable unless
it was obtained by segment nextObject when segment was already in oldSpace.

So it is perhaps possible to fix ImageSegments in Spur by forcing segment
to oldSpace and being more careful with endMarker.  But I think there is a
better way.

If the set of objects the segment contains can be obtained some how then
this set can be simply enumerated, not depending on nextObject.  The
primitive has to answer the array of roots, so its result can't be changed
to be the entire array.  But segment could be becomed into an Array of all
the objects in segment prior to it being loaded, in which case the above
would become


install
"This operation retrieves the segment if necessary from file storage,
installs it in memory, and replaces (using become:) all the root stubs with
the reconstructed roots of the segment."

| newRoots |
state = #onFile ifTrue: [self readFromFile].
state = #onFileWithSymbols ifTrue:
[self readFromFileWithSymbols].
(state = #active) | (state = #imported) ifFalse: [self errorWrongState].
newRoots := self loadSegmentFrom: segment outPointers: outPointers.
state = #imported
ifTrue: "just came in from exported file"
[arrayOfRoots := newRoots]
ifFalse:
[arrayOfRoots elementsForwardIdentityTo: newRoots].
state := #inactive.
Beeper beepPrimitive

allObjectsDo: aBlock
"Enumerate all objects that came from this segment.  NOTE this assumes that
the segment was created (and extracted).  After the segment has been
installed (install), this method allows you to enumerate its objects."
| obj |

segment isArray ifFalse:
[^ self error: 'Segment hasn''t been loaded?'].

segment do: aBlock

and the endMarker instVar would be deleted.

I am willing and ready to modify the primitive to convert the segment
correctly.  Who will volunteer to rewrite the image-level ImageSegment code
to use the new primitive?



On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda <[hidden email]>
wrote:

Hi Tobias,

On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape <[hidden email]> wrote:

Hi all
On 13.08.2015, at 02:15, Eliot Miranda <[hidden email]> wrote:

Hi Tobias,

    forget that. I found it.  THis right?

Trunk test suite for Spur
Using existing cogspur r.3410
cp -r /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410
/tmp/d20150812-28620-etbikj
   image test suite
VM: /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
-version
/var/lib/jenkins/workspace/Trunk/default/tests.st
spawning command 0 with timeout 1800 seconds:
"/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak"
"-vm-sound-null" "-vm-display-null"
"/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image"
"../tests.st"
(Command started with PID 28643)
2015-08-12T23:31:48.17+01:00: Loading Hudson build tools... from
/var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
2015-08-12T23:31:48.388+01:00: Running tests...
setsockopt: Protocol not available
setsockopt: Protocol not available
28646:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
protocol:s23_clnt.c:612:
Recursive not understood error encountered


I bet this is ImageSegment related.
It sure is.

         BitmapStreamTests>testMatrixTransform2x3WithImageSegment
sends
         BitmapStreamTests>validateImageSegment

  Basically the ImageSegment code has been working in Spur on a hope and a
prayer.  The code assumes objects are allocated in chronological order and
that this order is preserved, not so with Spur.  So post image segment
loading someObject/nextObject is used to enumerate the objects loaded.
This can't work reliably in Spur.  I *think* (ok I hope) that I've
implemented the segment load primitive in Spur to answer an Array of the
objects loaded, so that these can be explicitly enumerated.

So the job is a) to check that I have indeed implemented the primitive to
do this and b) to rewrite the image segment loading code in the light of
this.

David, this is an example of something that isn't back portable and should
not be back-ported.

The most strange thing is that I cannot reproduce this on my Mac???
There the test does not crash the image???

Well then it may be a signed/unsigned bug in image loading instead.  On
linux the image typically gets loaded quite high in the address space and
so a good portion of the heap ends up above 0x7FFFFFFF, or negative
territory if misinterpreted as signed integers.  On Mac and Windows the
image tends to get loaded quite low and so has to be pretty big to fall
foul of signedness issues.


Busy right now but will check tomorrow.


Best regards
         -Tobias


On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda <[hidden email]>
wrote:
Hi Tobias,

On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape <[hidden email]> wrote:

On 12.08.2015, at 20:55, Eliot Miranda <[hidden email]> wrote:

Hi All,


     Fabio's kindly done most of the changes.  But some questions
remain for more general discussion.  See below.
Fabio,  thanks so much for doing this, and so quickly!

On Tue, Aug 11, 2015 at 8:51 PM, Eliot Miranda <
[hidden email]> wrote:
Hi All,

     who will update http://squeak.org/downloads/ to include the 5.0
release?
a) I suggest that the 5.0 all-in-one have a line in the left-hand
table that includes the 4.6 release and that it precede the 4.6 release in
the list.
Done.


b) the Trunk link points to TrunkImage.zip which contains a non-Spur
image.
So this is the biggie.  What should we do?

- add a Trunk 5.0 build with a different link?  (Noooooo)

- change  http://build.squeak.org/job/SqueakTrunk to build what it
says (Yessss please, who can do this?)
Done: build.squeak.org/job/Trunk/
But as I said several times, Spur/trunk test just crash for month???

can you point me to the crash?  I'm looking at
http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink
and see 8 test failures but no crash.

- add a Squeak 4.6 build job?  Is that even worth it any more
considering 4.6 is released and stable?  If it is then what should it
build?  David?

c) Spur VMs need to be linked to by an added line in the Virtual
Machines list.
Not applicable.  the VM links point to the root of my site so they
effectively point to both old and Spur VMs.

d) Spur imagers need to be included in the Image and Changes list
under Custom Installation
Done.



e) The SqueakV50.sources file also needs to be included in the
Current Sources list.
Done.


f) we could start a History list for the 5.0 line

This is probably quite a big reorg on the page and not urgent.


So there are a few things to fix before 5.0 is freely downloadable.

On Tue, Aug 11, 2015 at 8:23 PM, Chris Muller <[hidden email]>
wrote:
In the 17 months since Squeak 4.5 was released, a huge development
effort took place to create the next generation virtual-machine for
the Squeak / Pharo / Newspeak family of programming systems.  Squeak
is the modern incarnation of the Smalltalk-80 programming environment
originally developed at the Xerox PARC.

"Squeak 5" introduces this new VM and associated new memory model,
collectively referred to as "Spur".  Presented [1] by Eliot Miranda
and Cl??ment B??ra at the 2015 International Symposium on Memory
Management, this new VM affords Squeak applications a significant
boost in performance and memory management.  Among other
optimizations, the #become operation no longer requires a memory scan.
Object pinning and ephemerons are also now supported.  The release
notes [2] provide more details.

The new memory model requires a new image file format.  Although this
new format results in about a 15% increased memory requirement for the
same number of 4.x objects, a new segmented heap allows memory to be
given back to the OS when its no longer needed, a great benefit for
application servers.

As forward compatibility is as important to the Squeak community as
backward compatibility, Squeak 5 is delivers an image with identical
content as the recent 4.6 release.  Although this new Squeak 5 VM
cannot open images saved under the prior 4.x Cog format, objects and
code can be easily exported from the 4.x image and then imported into
Squeak 5.  Applications whose code runs strictly above the Smalltalk
meta layer will prove remarkably compatible with the new format, most
applications will require no changes whatsotever.

Squeak 5 is the result of monumental effort by a tiny group of very
talented people, but its also just the beginning of yet a new effort;
Spur is just a stepping stone to a more ambitious goals planned over
the next five years.

[1] -- A Partial Read Barrier for Efficient Support of Live
Object-oriented Programming

http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
[2] -- Squeak 5 Release Notes
http://wiki.squeak.org/squeak/6207





--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot






--



    



Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5]

David T. Lewis
In reply to this post by Mariano Martinez Peck
On Wed, Aug 19, 2015 at 10:31:00PM -0300, Mariano Martinez Peck wrote:
>  
> Hi guys,
>
> As part of my PhD thesis I did take a deep look to ImageSegment before
> starting with Fuel. I wrote a journal paper about my experiments with
> ImageSegment which I thinks provides quite a documentation that is not
> written anywhere. Hope it helps:
> http://rmod.lille.inria.fr/archives/papers/Mart11c-COMLAN-ObjectSwapping.pdf

Hi Mariano,

Thanks for the reference to your ObjectSwapping.pdf paper. I think that it
provides a very helpful overview.

As an aside, we hold regular meetings of the Squeak oversight board, which
as you might expect sometimes diverge into discussion of technical issues of
one sort or another. In discussion earlier today, Eliot referred to your Fuel
implementation as "Parcels done right" and I agreed that I think it may help
with implementing image segments on new image formats (Spur, 32/64 bit images).
Personally, I like Fuel because it is easy to read and understand.

I would encourage folks to read the ObjectSwapping paper and also to consider
helping with implementation of image segments on Spur images. Mariano's paper
provides good documentation on the topic.

Dave


Reply | Threaded
Open this post in threaded view
|

ImageSegments and some OT (was: Re: Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5])

Tobias Pape
In reply to this post by Eliot Miranda-2
Hi all,

On 19.08.2015, at 23:09, Eliot Miranda <[hidden email]> wrote:

> Hi All, but especially Tobias,

just for reference, I only had time to skim this mail.
I'll work through it more in-depth in September (vacation upcoming, sorry).

Also, I like Fuel as a way forward. I'd also like to deprecate DataStreams in
favor of Fuel[1], I also had a prototype of Fuel-based Monticello packages
which loaded much faster for large packages…

Best regards
        -Tobias


[1]: PS: Hi Pharo folks, MCDataStream will break with Spur,
     Have a look at System-topa.741 unil System-topa.473 in the
     Squeak trunk or the attached change set, YMMV tho.





>
>     as we know, ImageSegments are broken in Spur. They work occasionally, but when stressed the VM ends up crashing.  The fundamental issue is that the ImageSegment code makes assumptions about object ordering that Spur violates.  For example, here's ImageSegment's install:
>
> install
> "This operation retrieves the segment if necessary from file storage, installs it in memory, and replaces (using become:) all the root stubs with the reconstructed roots of the segment."
>
> | newRoots |
> state = #onFile ifTrue: [self readFromFile].
> state = #onFileWithSymbols ifTrue: [self readFromFileWithSymbols.
> endMarker := segment nextObject. "for enumeration of objects"
> endMarker == 0 ifTrue: [endMarker := 'End' clone]].
> (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
> newRoots := self loadSegmentFrom: segment outPointers: outPointers.
> state = #imported
> ifTrue: ["just came in from exported file"
> arrayOfRoots := newRoots]
> ifFalse: [
> arrayOfRoots elementsForwardIdentityTo: newRoots].
> state := #inactive.
> Beeper beepPrimitive
>
> So before the image segment bytes (the segment inst var) is loaded, the object after it is assigned to endMarker, and if there isn't an object after segment, a new object ('End' clone) is assigned to endMarker.
>
> This makes the assumption that objects are allocated in a strict order, and therefore endMarker will always be the object after segment.
>
> Loading the segment via "newRoots := self loadSegmentFrom: segment outPointers: outPointers" then turns segment into a zero-length WordArray and its contents into the objects loaded by the segment.  Therefore, in the V3 system, the objects loaded from segment can be enumerated starting at segment nextObject and repeating until endMarker is found:
>
> allObjectsDo: aBlock
> "Enumerate all objects that came from this segment.  NOTE this assumes that the segment was created (and extracted).  After the segment has been installed (install), this method allows you to enumerate its objects."
> | obj |
>
> endMarker == nil ifTrue: [
> ^ self error: 'Just extract and install, don''t writeToFile:'].
> segment size ~= 1 ifTrue: [
> ^ self error: 'Vestigial segment size must be 1 (version word)'].
>
> obj := segment nextObject.  "Start with the next object after the vestigial header"
> [obj == endMarker] whileFalse:  "Stop at the next object after the full segment"
> [aBlock value: obj.
> obj := obj nextObject].  "Step through the objects installed from the segment."
>
> Now, as written, this just won't work in Spur.
>
> a) the only place where there is any kind of stable order to objects is in oldSpace, so segment /has/ to be forced to old space to have any chance of its objects being in order when it gets converted from bytes to objects.
>
> b) 'End' clone will be in newSpace and so endMarker isn't reliable unless it was obtained by segment nextObject when segment was already in oldSpace.
>
> So it is perhaps possible to fix ImageSegments in Spur by forcing segment to oldSpace and being more careful with endMarker.  But I think there is a better way.
>
> If the set of objects the segment contains can be obtained some how then this set can be simply enumerated, not depending on nextObject.  The primitive has to answer the array of roots, so its result can't be changed to be the entire array.  But segment could be becomed into an Array of all the objects in segment prior to it being loaded, in which case the above would become
>
>
> install
> "This operation retrieves the segment if necessary from file storage, installs it in memory, and replaces (using become:) all the root stubs with the reconstructed roots of the segment."
>
> | newRoots |
> state = #onFile ifTrue: [self readFromFile].
> state = #onFileWithSymbols ifTrue:
> [self readFromFileWithSymbols].
> (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
> newRoots := self loadSegmentFrom: segment outPointers: outPointers.
> state = #imported
> ifTrue: "just came in from exported file"
> [arrayOfRoots := newRoots]
> ifFalse:
> [arrayOfRoots elementsForwardIdentityTo: newRoots].
> state := #inactive.
> Beeper beepPrimitive
>
> allObjectsDo: aBlock
> "Enumerate all objects that came from this segment.  NOTE this assumes that the segment was created (and extracted).  After the segment has been installed (install), this method allows you to enumerate its objects."
> | obj |
>
> segment isArray ifFalse:
> [^ self error: 'Segment hasn''t been loaded?'].
>
> segment do: aBlock
>
> and the endMarker instVar would be deleted.
>
> I am willing and ready to modify the primitive to convert the segment correctly.  Who will volunteer to rewrite the image-level ImageSegment code to use the new primitive?
>
>
>
> On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda <[hidden email]> wrote:
> Hi Tobias,
>
> On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape <[hidden email]> wrote:
> Hi all
> On 13.08.2015, at 02:15, Eliot Miranda <[hidden email]> wrote:
>
> > Hi Tobias,
> >
> >    forget that. I found it.  THis right?
> >
> > Trunk test suite for Spur
> > Using existing cogspur r.3410
> > cp -r /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410 /tmp/d20150812-28620-etbikj
> >   image test suite
> > VM: /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
> > /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak -version
> > /var/lib/jenkins/workspace/Trunk/default/tests.st
> > spawning command 0 with timeout 1800 seconds: "/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak" "-vm-sound-null" "-vm-display-null" "/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image" "../tests.st"
> > (Command started with PID 28643)
> > 2015-08-12T23:31:48.17+01:00: Loading Hudson build tools... from /var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
> > 2015-08-12T23:31:48.388+01:00: Running tests...
> > setsockopt: Protocol not available
> > setsockopt: Protocol not available
> > 28646:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:612:
> >
> > Recursive not understood error encountered
> >
> >
> > I bet this is ImageSegment related.
>
> It sure is.
>
>         BitmapStreamTests>testMatrixTransform2x3WithImageSegment
> sends
>         BitmapStreamTests>validateImageSegment
>
>  Basically the ImageSegment code has been working in Spur on a hope and a prayer.  The code assumes objects are allocated in chronological order and that this order is preserved, not so with Spur.  So post image segment loading someObject/nextObject is used to enumerate the objects loaded.  This can't work reliably in Spur.  I *think* (ok I hope) that I've implemented the segment load primitive in Spur to answer an Array of the objects loaded, so that these can be explicitly enumerated.
>
> So the job is a) to check that I have indeed implemented the primitive to do this and b) to rewrite the image segment loading code in the light of this.
>
> David, this is an example of something that isn't back portable and should not be back-ported.
>
> The most strange thing is that I cannot reproduce this on my Mac…
> There the test does not crash the image…
>
> Well then it may be a signed/unsigned bug in image loading instead.  On linux the image typically gets loaded quite high in the address space and so a good portion of the heap ends up above 0x7FFFFFFF, or negative territory if misinterpreted as signed integers.  On Mac and Windows the image tends to get loaded quite low and so has to be pretty big to fall foul of signedness issues.
>
>
> Busy right now but will check tomorrow.
>
>
> Best regards
>         -Tobias
>
>
> >
> > On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda <[hidden email]> wrote:
> > Hi Tobias,
> >
> > On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape <[hidden email]> wrote:
> >
> > On 12.08.2015, at 20:55, Eliot Miranda <[hidden email]> wrote:
> >
> > > Hi All,
> > >
> > >
> > >     Fabio's kindly done most of the changes.  But some questions remain for more general discussion.  See below.
> > >
> > > Fabio,  thanks so much for doing this, and so quickly!
> > >
> > > On Tue, Aug 11, 2015 at 8:51 PM, Eliot Miranda <[hidden email]> wrote:
> > > Hi All,
> > >
> > >     who will update http://squeak.org/downloads/ to include the 5.0 release?
> > >
> > > a) I suggest that the 5.0 all-in-one have a line in the left-hand table that includes the 4.6 release and that it precede the 4.6 release in the list.
> > >
> > > Done.
> > >
> > >
> > > b) the Trunk link points to TrunkImage.zip which contains a non-Spur image.
> > >
> > > So this is the biggie.  What should we do?
> > >
> > > - add a Trunk 5.0 build with a different link?  (Noooooo)
> > >
> > > - change  http://build.squeak.org/job/SqueakTrunk to build what it says (Yessss please, who can do this?)
> >
> > Done: build.squeak.org/job/Trunk/
> > But as I said several times, Spur/trunk test just crash for month…
> >
> > can you point me to the crash?  I'm looking at http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink and see 8 test failures but no crash.
> >
> >
> > >
> > > - add a Squeak 4.6 build job?  Is that even worth it any more considering 4.6 is released and stable?  If it is then what should it build?  David?
> > >
> > >
> > > c) Spur VMs need to be linked to by an added line in the Virtual Machines list.
> > >
> > > Not applicable.  the VM links point to the root of my site so they effectively point to both old and Spur VMs.
> > >
> > >
> > > d) Spur imagers need to be included in the Image and Changes list under Custom Installation
> > >
> > > Done.
> > >
> > >
> > >
> > > e) The SqueakV50.sources file also needs to be included in the Current Sources list.
> > >
> > > Done.
> > >
> > >
> > > f) we could start a History list for the 5.0 line
> > >
> > > This is probably quite a big reorg on the page and not urgent.
> > >
> > >
> > > So there are a few things to fix before 5.0 is freely downloadable.
> > >
> > > On Tue, Aug 11, 2015 at 8:23 PM, Chris Muller <[hidden email]> wrote:
> > > In the 17 months since Squeak 4.5 was released, a huge development
> > > effort took place to create the next generation virtual-machine for
> > > the Squeak / Pharo / Newspeak family of programming systems.  Squeak
> > > is the modern incarnation of the Smalltalk-80 programming environment
> > > originally developed at the Xerox PARC.
> > >
> > > "Squeak 5" introduces this new VM and associated new memory model,
> > > collectively referred to as "Spur".  Presented [1] by Eliot Miranda
> > > and Clément Béra at the 2015 International Symposium on Memory
> > > Management, this new VM affords Squeak applications a significant
> > > boost in performance and memory management.  Among other
> > > optimizations, the #become operation no longer requires a memory scan.
> > > Object pinning and ephemerons are also now supported.  The release
> > > notes [2] provide more details.
> > >
> > > The new memory model requires a new image file format.  Although this
> > > new format results in about a 15% increased memory requirement for the
> > > same number of 4.x objects, a new segmented heap allows memory to be
> > > given back to the OS when its no longer needed, a great benefit for
> > > application servers.
> > >
> > > As forward compatibility is as important to the Squeak community as
> > > backward compatibility, Squeak 5 is delivers an image with identical
> > > content as the recent 4.6 release.  Although this new Squeak 5 VM
> > > cannot open images saved under the prior 4.x Cog format, objects and
> > > code can be easily exported from the 4.x image and then imported into
> > > Squeak 5.  Applications whose code runs strictly above the Smalltalk
> > > meta layer will prove remarkably compatible with the new format, most
> > > applications will require no changes whatsotever.
> > >
> > > Squeak 5 is the result of monumental effort by a tiny group of very
> > > talented people, but its also just the beginning of yet a new effort;
> > > Spur is just a stepping stone to a more ambitious goals planned over
> > > the next five years.
> > >
> > > [1] -- A Partial Read Barrier for Efficient Support of Live
> > > Object-oriented Programming
> > > http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
> > >
> > > [2] -- Squeak 5 Release Notes
> > > http://wiki.squeak.org/squeak/6207





SpurDataStreamChanges.1.cs (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-dev] ImageSegments and some OT (was: Re: Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5])

Mariano Martinez Peck


On Thu, Aug 20, 2015 at 11:37 AM, Tobias Pape <[hidden email]> wrote:
 
Hi all,

On 19.08.2015, at 23:09, Eliot Miranda <[hidden email]> wrote:

> Hi All, but especially Tobias,

just for reference, I only had time to skim this mail.
I'll work through it more in-depth in September (vacation upcoming, sorry).

Also, I like Fuel as a way forward. I'd also like to deprecate DataStreams in
favor of Fuel[1], I also had a prototype of Fuel-based Monticello packages
which loaded much faster for large packages…


Is this similar to Tanker in the way that classes and methods are binary serialized with Fuel? (no compiling)
Or you just used Fuel instead of the current usage of MCDataStream? If this is the case, then you would still be compiling at load time etc. We already did this experiment long ago and we didn't find a significant speedup. 

What were your findings?
 
Best regards
        -Tobias


[1]: PS: Hi Pharo folks, MCDataStream will break with Spur,
     Have a look at System-topa.741 unil System-topa.473 in the
     Squeak trunk or the attached change set, YMMV tho.


Thanks, 
 




>
>     as we know, ImageSegments are broken in Spur. They work occasionally, but when stressed the VM ends up crashing.  The fundamental issue is that the ImageSegment code makes assumptions about object ordering that Spur violates.  For example, here's ImageSegment's install:
>
> install
>       "This operation retrieves the segment if necessary from file storage, installs it in memory, and replaces (using become:) all the root stubs with the reconstructed roots of the segment."
>
>       | newRoots |
>       state = #onFile ifTrue: [self readFromFile].
>       state = #onFileWithSymbols ifTrue: [self readFromFileWithSymbols.
>               endMarker := segment nextObject.        "for enumeration of objects"
>               endMarker == 0 ifTrue: [endMarker := 'End' clone]].
>       (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
>       newRoots := self loadSegmentFrom: segment outPointers: outPointers.
>       state = #imported
>               ifTrue: ["just came in from exported file"
>                       arrayOfRoots := newRoots]
>               ifFalse: [
>                       arrayOfRoots elementsForwardIdentityTo: newRoots].
>       state := #inactive.
>       Beeper beepPrimitive
>
> So before the image segment bytes (the segment inst var) is loaded, the object after it is assigned to endMarker, and if there isn't an object after segment, a new object ('End' clone) is assigned to endMarker.
>
> This makes the assumption that objects are allocated in a strict order, and therefore endMarker will always be the object after segment.
>
> Loading the segment via "newRoots := self loadSegmentFrom: segment outPointers: outPointers" then turns segment into a zero-length WordArray and its contents into the objects loaded by the segment.  Therefore, in the V3 system, the objects loaded from segment can be enumerated starting at segment nextObject and repeating until endMarker is found:
>
> allObjectsDo: aBlock
>       "Enumerate all objects that came from this segment.  NOTE this assumes that the segment was created (and extracted).  After the segment has been installed (install), this method allows you to enumerate its objects."
>       | obj |
>
>       endMarker == nil ifTrue: [
>               ^ self error: 'Just extract and install, don''t writeToFile:'].
>       segment size ~= 1 ifTrue: [
>               ^ self error: 'Vestigial segment size must be 1 (version word)'].
>
>       obj := segment nextObject.  "Start with the next object after the vestigial header"
>       [obj == endMarker] whileFalse:  "Stop at the next object after the full segment"
>               [aBlock value: obj.
>               obj := obj nextObject].  "Step through the objects installed from the segment."
>
> Now, as written, this just won't work in Spur.
>
> a) the only place where there is any kind of stable order to objects is in oldSpace, so segment /has/ to be forced to old space to have any chance of its objects being in order when it gets converted from bytes to objects.
>
> b) 'End' clone will be in newSpace and so endMarker isn't reliable unless it was obtained by segment nextObject when segment was already in oldSpace.
>
> So it is perhaps possible to fix ImageSegments in Spur by forcing segment to oldSpace and being more careful with endMarker.  But I think there is a better way.
>
> If the set of objects the segment contains can be obtained some how then this set can be simply enumerated, not depending on nextObject.  The primitive has to answer the array of roots, so its result can't be changed to be the entire array.  But segment could be becomed into an Array of all the objects in segment prior to it being loaded, in which case the above would become
>
>
> install
>       "This operation retrieves the segment if necessary from file storage, installs it in memory, and replaces (using become:) all the root stubs with the reconstructed roots of the segment."
>
>       | newRoots |
>       state = #onFile ifTrue: [self readFromFile].
>       state = #onFileWithSymbols ifTrue:
>               [self readFromFileWithSymbols].
>       (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
>       newRoots := self loadSegmentFrom: segment outPointers: outPointers.
>       state = #imported
>               ifTrue: "just came in from exported file"
>                       [arrayOfRoots := newRoots]
>               ifFalse:
>                       [arrayOfRoots elementsForwardIdentityTo: newRoots].
>       state := #inactive.
>       Beeper beepPrimitive
>
> allObjectsDo: aBlock
>       "Enumerate all objects that came from this segment.  NOTE this assumes that the segment was created (and extracted).  After the segment has been installed (install), this method allows you to enumerate its objects."
>       | obj |
>
>       segment isArray ifFalse:
>               [^ self error: 'Segment hasn''t been loaded?'].
>
>       segment do: aBlock
>
> and the endMarker instVar would be deleted.
>
> I am willing and ready to modify the primitive to convert the segment correctly.  Who will volunteer to rewrite the image-level ImageSegment code to use the new primitive?
>
>
>
> On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda <[hidden email]> wrote:
> Hi Tobias,
>
> On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape <[hidden email]> wrote:
> Hi all
> On 13.08.2015, at 02:15, Eliot Miranda <[hidden email]> wrote:
>
> > Hi Tobias,
> >
> >    forget that. I found it.  THis right?
> >
> > Trunk test suite for Spur
> > Using existing cogspur r.3410
> > cp -r /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410 /tmp/d20150812-28620-etbikj
> >   image test suite
> > VM: /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
> > /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak -version
> > /var/lib/jenkins/workspace/Trunk/default/tests.st
> > spawning command 0 with timeout 1800 seconds: "/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak" "-vm-sound-null" "-vm-display-null" "/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image" "../tests.st"
> > (Command started with PID 28643)
> > 2015-08-12T23:31:48.17+01:00: Loading Hudson build tools... from /var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
> > 2015-08-12T23:31:48.388+01:00: Running tests...
> > setsockopt: Protocol not available
> > setsockopt: Protocol not available
> > 28646:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:612:
> >
> > Recursive not understood error encountered
> >
> >
> > I bet this is ImageSegment related.
>
> It sure is.
>
>         BitmapStreamTests>testMatrixTransform2x3WithImageSegment
> sends
>         BitmapStreamTests>validateImageSegment
>
>  Basically the ImageSegment code has been working in Spur on a hope and a prayer.  The code assumes objects are allocated in chronological order and that this order is preserved, not so with Spur.  So post image segment loading someObject/nextObject is used to enumerate the objects loaded.  This can't work reliably in Spur.  I *think* (ok I hope) that I've implemented the segment load primitive in Spur to answer an Array of the objects loaded, so that these can be explicitly enumerated.
>
> So the job is a) to check that I have indeed implemented the primitive to do this and b) to rewrite the image segment loading code in the light of this.
>
> David, this is an example of something that isn't back portable and should not be back-ported.
>
> The most strange thing is that I cannot reproduce this on my Mac…
> There the test does not crash the image…
>
> Well then it may be a signed/unsigned bug in image loading instead.  On linux the image typically gets loaded quite high in the address space and so a good portion of the heap ends up above 0x7FFFFFFF, or negative territory if misinterpreted as signed integers.  On Mac and Windows the image tends to get loaded quite low and so has to be pretty big to fall foul of signedness issues.
>
>
> Busy right now but will check tomorrow.
>
>
> Best regards
>         -Tobias
>
>
> >
> > On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda <[hidden email]> wrote:
> > Hi Tobias,
> >
> > On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape <[hidden email]> wrote:
> >
> > On 12.08.2015, at 20:55, Eliot Miranda <[hidden email]> wrote:
> >
> > > Hi All,
> > >
> > >
> > >     Fabio's kindly done most of the changes.  But some questions remain for more general discussion.  See below.
> > >
> > > Fabio,  thanks so much for doing this, and so quickly!
> > >
> > > On Tue, Aug 11, 2015 at 8:51 PM, Eliot Miranda <[hidden email]> wrote:
> > > Hi All,
> > >
> > >     who will update http://squeak.org/downloads/ to include the 5.0 release?
> > >
> > > a) I suggest that the 5.0 all-in-one have a line in the left-hand table that includes the 4.6 release and that it precede the 4.6 release in the list.
> > >
> > > Done.
> > >
> > >
> > > b) the Trunk link points to TrunkImage.zip which contains a non-Spur image.
> > >
> > > So this is the biggie.  What should we do?
> > >
> > > - add a Trunk 5.0 build with a different link?  (Noooooo)
> > >
> > > - change  http://build.squeak.org/job/SqueakTrunk to build what it says (Yessss please, who can do this?)
> >
> > Done: build.squeak.org/job/Trunk/
> > But as I said several times, Spur/trunk test just crash for month…
> >
> > can you point me to the crash?  I'm looking at http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink and see 8 test failures but no crash.
> >
> >
> > >
> > > - add a Squeak 4.6 build job?  Is that even worth it any more considering 4.6 is released and stable?  If it is then what should it build?  David?
> > >
> > >
> > > c) Spur VMs need to be linked to by an added line in the Virtual Machines list.
> > >
> > > Not applicable.  the VM links point to the root of my site so they effectively point to both old and Spur VMs.
> > >
> > >
> > > d) Spur imagers need to be included in the Image and Changes list under Custom Installation
> > >
> > > Done.
> > >
> > >
> > >
> > > e) The SqueakV50.sources file also needs to be included in the Current Sources list.
> > >
> > > Done.
> > >
> > >
> > > f) we could start a History list for the 5.0 line
> > >
> > > This is probably quite a big reorg on the page and not urgent.
> > >
> > >
> > > So there are a few things to fix before 5.0 is freely downloadable.
> > >
> > > On Tue, Aug 11, 2015 at 8:23 PM, Chris Muller <[hidden email]> wrote:
> > > In the 17 months since Squeak 4.5 was released, a huge development
> > > effort took place to create the next generation virtual-machine for
> > > the Squeak / Pharo / Newspeak family of programming systems.  Squeak
> > > is the modern incarnation of the Smalltalk-80 programming environment
> > > originally developed at the Xerox PARC.
> > >
> > > "Squeak 5" introduces this new VM and associated new memory model,
> > > collectively referred to as "Spur".  Presented [1] by Eliot Miranda
> > > and Clément Béra at the 2015 International Symposium on Memory
> > > Management, this new VM affords Squeak applications a significant
> > > boost in performance and memory management.  Among other
> > > optimizations, the #become operation no longer requires a memory scan.
> > > Object pinning and ephemerons are also now supported.  The release
> > > notes [2] provide more details.
> > >
> > > The new memory model requires a new image file format.  Although this
> > > new format results in about a 15% increased memory requirement for the
> > > same number of 4.x objects, a new segmented heap allows memory to be
> > > given back to the OS when its no longer needed, a great benefit for
> > > application servers.
> > >
> > > As forward compatibility is as important to the Squeak community as
> > > backward compatibility, Squeak 5 is delivers an image with identical
> > > content as the recent 4.6 release.  Although this new Squeak 5 VM
> > > cannot open images saved under the prior 4.x Cog format, objects and
> > > code can be easily exported from the 4.x image and then imported into
> > > Squeak 5.  Applications whose code runs strictly above the Smalltalk
> > > meta layer will prove remarkably compatible with the new format, most
> > > applications will require no changes whatsotever.
> > >
> > > Squeak 5 is the result of monumental effort by a tiny group of very
> > > talented people, but its also just the beginning of yet a new effort;
> > > Spur is just a stepping stone to a more ambitious goals planned over
> > > the next five years.
> > >
> > > [1] -- A Partial Read Barrier for Efficient Support of Live
> > > Object-oriented Programming
> > > http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
> > >
> > > [2] -- Squeak 5 Release Notes
> > > http://wiki.squeak.org/squeak/6207







--


Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] [Vm-dev] ImageSegments and some OT (was: Re: Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5])

Tobias Pape
Hi Mariano,

On 20.08.2015, at 16:41, Mariano Martinez Peck <[hidden email]> wrote:
> […]
> Is this similar to Tanker in the way that classes and methods are binary serialized with Fuel? (no compiling)
> Or you just used Fuel instead of the current usage of MCDataStream? If this is the case, then you would still be compiling at load time etc. We already did this experiment long ago and we didn't find a significant speedup.
>
> What were your findings?
[…]
Well that was 4 years ago: http://forum.world.st/Monticello-enhancing-td4179277.html
And I don't remember too well, just my feeling that it was a lot faster by replacing
the DataStream blob with a fuel blob (so, no, not like Tanker).
(No actual numbers written down, tho, sorry)

Best regards
        -Tobia



Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] [Vm-dev] ImageSegments and some OT (was: Re: Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5])

Mariano Martinez Peck


On Thu, Aug 20, 2015 at 11:47 AM, Tobias Pape <[hidden email]> wrote:
Hi Mariano,

On 20.08.2015, at 16:41, Mariano Martinez Peck <[hidden email]> wrote:
> […]
> Is this similar to Tanker in the way that classes and methods are binary serialized with Fuel? (no compiling)
> Or you just used Fuel instead of the current usage of MCDataStream? If this is the case, then you would still be compiling at load time etc. We already did this experiment long ago and we didn't find a significant speedup.
>
> What were your findings?
[…]
Well that was 4 years ago: http://forum.world.st/Monticello-enhancing-td4179277.html
And I don't remember too well, just my feeling that it was a lot faster by replacing
the DataStream blob with a fuel blob (so, no, not like Tanker).
(No actual numbers written down, tho, sorry)


OK, that's what I imagined. Yes, we did the same experiment too. As said, we haven't seen an speed up there because even if Fuel would be 100x faster writing that blob, the writing of that blob was insignificant in a Monticello load. Most of the time when loading with Monticello goes into the parsing/compiling, the class building thingies (may involve became: to update instances), the notifications, etc..

In fact, one of the things we found is that:
-  just having a bulk notification (and update all the observers) would speed up things (have you ever tried loading with Monticello with TestRunner opened???? takes a lifetime!)
- Spur would help us with the lazy become when updating instances

So probably maybe with those 2 things we already see an speedup even if we still compile from source...



--


Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] ImageSegments and some OT (was: Re: Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5])

Marcus Denker-4
In reply to this post by Tobias Pape

> On 20 Aug 2015, at 16:37, Tobias Pape <[hidden email]> wrote:
>
> Hi all,
>
> On 19.08.2015, at 23:09, Eliot Miranda <[hidden email]> wrote:
>
>> Hi All, but especially Tobias,
>
> just for reference, I only had time to skim this mail.
> I'll work through it more in-depth in September (vacation upcoming, sorry).
>
> Also, I like Fuel as a way forward. I'd also like to deprecate DataStreams in
> favor of Fuel[1], I also had a prototype of Fuel-based Monticello packages
> which loaded much faster for large packages…
>
> Best regards
> -Tobias
>
>
> [1]: PS: Hi Pharo folks, MCDataStream will break with Spur,
>     Have a look at System-topa.741 unil System-topa.473 in the
>     Squeak trunk or the attached change set, YMMV tho.
>

Very nice!

yes, back when we cleanup the 4+ different serialisers we decided to
not go Fuel for Monticello for staying compatible. We therefore just took
a simplified DataStream and added it as MCDataStream to Monticello itself.

It would be nice to retire that once and for all!

        Marcus


Reply | Threaded
Open this post in threaded view
|

Fwd: [Vm-dev] Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5]

Edgar De Cleene
In reply to this post by David T. Lewis

I forward this as I think Fuel deserves be part of Squeak 5.0.
And is time to move forward of some old code not needed now.

Edgar
@morplenauta

De: "David T. Lewis" <[hidden email]>
Asunto: Re: [Vm-dev] Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5]
Fecha: 20 de agosto de 2015 1:00:33 GMT-03:00
Para: Squeak Virtual Machine Development Discussion <[hidden email]>
Cc: The general-purpose Squeak developers list <[hidden email]>
Responder a: Squeak Virtual Machine Development Discussion <[hidden email]>


On Wed, Aug 19, 2015 at 10:31:00PM -0300, Mariano Martinez Peck wrote:

Hi guys,

As part of my PhD thesis I did take a deep look to ImageSegment before
starting with Fuel. I wrote a journal paper about my experiments with
ImageSegment which I thinks provides quite a documentation that is not
written anywhere. Hope it helps:
http://rmod.lille.inria.fr/archives/papers/Mart11c-COMLAN-ObjectSwapping.pdf

Hi Mariano,

Thanks for the reference to your ObjectSwapping.pdf paper. I think that it
provides a very helpful overview.

As an aside, we hold regular meetings of the Squeak oversight board, which
as you might expect sometimes diverge into discussion of technical issues of
one sort or another. In discussion earlier today, Eliot referred to your Fuel
implementation as "Parcels done right" and I agreed that I think it may help
with implementing image segments on new image formats (Spur, 32/64 bit images).
Personally, I like Fuel because it is easy to read and understand.

I would encourage folks to read the ObjectSwapping paper and also to consider
helping with implementation of image segments on Spur images. Mariano's paper
provides good documentation on the topic.

Dave