Re: [squeak-dev] New Spur trunk image available

Hi All,

    finally the Spur Squeak trunk image is updateable.  The image in http://www.mirandabanda.org/files/Cog/SpurImages/2014-10-16/ was created today and thanks to Bert Freudenberg's latest Monticello work can be updated independently of the non-Spur trunk.  Spur VMs are available in http://www.mirandabanda.org/files/Cog/VM/VM.r3105/ (and later as they appear).  Without wanting to appear too overconfident the Spur system looks to be ready for use apart from image segments (which I hope to have working some time next month).  I'm really interested in having this stress tested by as many people as possible.  Spur really does offer a significant performance and functionality improvement over the current system, but it needs testing to ensure its reliability.

Esteban Lorenzano is hard at work on the Pharo bootstrap for Spur so I hope Pharo 4 Spur will be available soon.

As far as trunk goes, using Spur alongside non-Spur trunk is going to be difficult to manage for the near future.  Right now, Spur modifies the Collections, Compiler, Kernel and System packages, and this is done by auto-editing the non-Spur versions of those packages, something I do periodically as new versions arrive.  I also auto-edit trunk configurations (the part of the image update scheme that ensures packages are loaded in the right order when there are dependencies between packages) from non-Spur "update" to Spur "update.spur" forms.  This at east means that Spur can keep up with trunk.  But it does /not/ provide a way of committing to Collections.spur, Compiler.spur, Kernel.spur or System.spur without getting out of sync with non-Spur trunk.  Note that apart from these packages, one /can/ safely commit any other package from a Spur image to trunk.

Right now the plan is to release both V3 (the pre-Spur format) and Spur versions of Squeak 4.6 (I hope it'll be called Squeak 5.0).  This isn't my preference.  I'd like to see just Spur released, once reliability is verified.  But I understand the safety and backward-compatibility concerns (Spur won't be able to load V3 image segments, and vice verse).  The issue is of course that we have this tricky package situation to manage where, to keep the two systems in sync, modifications to Collections, Compiler, Kernel and System need to be committed from V3 and auto-edited to Spur. I think that's too clumsy to be practicable.  Perhaps allowing the two systems to fork and doing a manual merge will be acceptable, but it'll be work to keep them in sync.

--
best,

Eliot

Hi Eliot. I will make time to install Spur (assuming it's presently working on the Mac.) I'll use it as my primary VM unless I run into problems, in which case I will report them.

Hi Eliot, Casey,

Quoting Eliot Miranda <[hidden email]>:

Hi Casey,

On Oct 17, 2014, at 12:37 AM, Casey Ransberger <[hidden email]> wrote:
 

Hi Eliot. I will make time to install Spur (assuming it's presently working on the Mac.) I'll use it as my primary VM unless I run into problems, in which case I will report them.

 

Yes, I build Spur VMs alongside Cog VMs, so they're available for Mac, x86 Windows and x86 linux.  (and soon stack VMs for ARM linux).

 

And thank you, both for using it and for these great questions.

 

I think I understood you, but I want to be clear; you're saying that (like Cog) changes needed to be made to the image to support the new VM. 

 

In this case more like to exploit the new VM.  So 32-bit Spur has immediate characters alongside immediate SmallIntegers, and 64-bit Spur will add immediate floats.  Spur has a 64k literal limit in methods vs 256 in V3 (pre-Spur).  There are slightly different versions of Behavior>>new, new:, basicNew & basicNew: to marry with the new GC and segmented heap, and there are primitives for allInstances and allObjects.  These are all benefits that could have been done without, but are clearly good and indeed the 64k literal limit exists in V3 Newspeak.

 

In fact, the 64k limit is also associated with support for two bytecode sets.  And that leads to support for Sista, the adaptive optimizer we hope will be available later next year.

 

The necessary change is to add Behavior>>identityHash to ensure a class's identityHash is its index in the VM's class table.  This is what keeps Spur objects relatively compact, and keeps their header uniform.  i.e. every object's header contains its class's index in a 22-bit field instead of a direct reference to its class object.

Thanks for all this detail. It will be really useful.

Query: can the Interpreter (either that or Stack) VM be modified to support images which run above Spur? Or is this necessarily a forking event? (CC Cuis folks.)

 

Yes.  Stack Spur VMs are easily generable from a VMMaker image.  In fact Stack Spur was the first working version.  But porting the context Interpreter means finishing the merge with trunk VMMaker for which the community (in the person of David Lewis and myself) have so few cycles (volunteers welcome, but they must be prepared to merge the Interpreter into the Cog branch, not the other way).

WRT Stack Spur VM, this is indeed great news! It means that Cuis can adopt the Spur image format and continue to be fully multi-platform, without needing to maintain two active branchs. Yay!

WRT the context Interpreter, it would be nice to merge it, essentially to ease the landscape to contributors.

Query: should incorporating the changes to the image that you have outlined below into Cuis allow Cuis to make use of the new VM? (I perhaps am digging very vaguely into what those changes are. Sorry I'm not presently equipped to ask smarter questions: I'm probably just beside myself with enthusiasm for your work and perchance "jumping the gun" a bit.)

 

Yes.  The bootstrap is written as an image editor.  Adding support for Cuis should be straightforward, provided Cuis's compiler is brought close to trunk's (to support and exploit the 64k literal limit and multiple bytecode sets).

 

The Cuis compiler has always been close to Squeak's, especially after Cuis adopted Cog. (For the record, AFAIK, I was the only person, besides Eliot, to migrate a context Interpreter image to Cog. That was Cuis, of course.)

Eliot, after taking a look at the Spur bootstrap some months ago (haven't found time yet to make it work on Cuis), and reading this message of yours, I wonder, wouldn't it make sense to do just the essential (i.e. mandatory) changes (like Behavior>>identityHash, rehashing, image header, etc) in the bootstrap? Then do all the rest afterwards, after starting the image with a Spur VM.  This should ease significantly adapting this second part to various Squeak images, and would encourage wider adoption by other projects. It would also ease keeing the Spur bootstrap updated, as the target images keep changing.

Anyone motivated to port Cuis should contact me to discuss and for me to show you around the code.

Thanks for the offer, Eliot. If nobody else in the Cuis community beats me to it (folks, you are welcome to do it!), I will. Right now all my spare cycles are devoted to advancing Morphic 3 (the pre-alpha code was released MIT recently). But I will have time to do this in maybe a month, and I will really appreciate your help.

TIA,

 

Thanks.  These questions have allowed me to clarify.  But do you (or anyone else) have any opinions or concerns about the parallel development model for trunk I outlined?

 

Casey

 

P.S.

 

Please forgive the pun, if you noticed it! :D

 

I missed it :-/

Eliot (phone)

 

Thanks Eliot for the thoughtful answer, and thanks Casey for asking the questions.

Cheers,
Juan Vuletich

On Oct 16, 2014, at 11:51 PM, Eliot Miranda <[hidden email]> wrote:
 

Hi All,

 

    finally the Spur Squeak trunk image is updateable.  The image in http://www.mirandabanda.org/files/Cog/SpurImages/2014-10-16/ was created today and thanks to Bert Freudenberg's latest Monticello work can be updated independently of the non-Spur trunk.  Spur VMs are available in http://www.mirandabanda.org/files/Cog/VM/VM.r3105/ (and later as they appear).  Without wanting to appear too overconfident the Spur system looks to be ready for use apart from image segments (which I hope to have working some time next month).  I'm really interested in having this stress tested by as many people as possible.  Spur really does offer a significant performance and functionality improvement over the current system, but it needs testing to ensure its reliability.

 

Esteban Lorenzano is hard at work on the Pharo bootstrap for Spur so I hope Pharo 4 Spur will be available soon.

 

As far as trunk goes, using Spur alongside non-Spur trunk is going to be difficult to manage for the near future.  Right now, Spur modifies the Collections, Compiler, Kernel and System packages, and this is done by auto-editing the non-Spur versions of those packages, something I do periodically as new versions arrive.  I also auto-edit trunk configurations (the part of the image update scheme that ensures packages are loaded in the right order when there are dependencies between packages) from non-Spur "update" to Spur "update.spur" forms.  This at east means that Spur can keep up with trunk.  But it does /not/ provide a way of committing to Collections.spur, Compiler.spur, Kernel.spur or System.spur without getting out of sync with non-Spur trunk.  Note that apart from these packages, one /can/ safely commit any other package from a Spur image to trunk.

 

Right now the plan is to release both V3 (the pre-Spur format) and Spur versions of Squeak 4.6 (I hope it'll be called Squeak 5.0).  This isn't my preference.  I'd like to see just Spur released, once reliability is verified.  But I understand the safety and backward-compatibility concerns (Spur won't be able to load V3 image segments, and vice verse).  The issue is of course that we have this tricky package situation to manage where, to keep the two systems in sync, modifications to Collections, Compiler, Kernel and System need to be committed from V3 and auto-edited to Spur. I think that's too clumsy to be practicable.  Perhaps allowing the two systems to fork and doing a manual merge will be acceptable, but it'll be work to keep them in sync.

--
best,

Eliot