3.9 vs. 3.10 : Closures, fixTemps

> Philippe Marschall wrote:
>>> there is really no reason whatsoever why a working closure solution
>>> couldn't be adopted to the current compiler (which would mitigate  
>>> the
>>> risk factors of adoption dramatically).
>> They why was the closure solution of the new compiler rejected?
>
> "The closure solution rejected"? Not that I know of. I haven't even  
> seen a thorough discussion about its merits (which should include  
> benchmarks etc) - what I did see was some talk that led me to  
> believe that there is still a ways to go here. To the best of my  
> knowledge this solution isn't "final" by any means.
>
> Or do you mean the compiler itself? In which case all I can say it's  
> hard to recommend a compiler which can't even recompile Object ;-)
>
> Cheers,
>  - Andreas
>
>

>
> On Dec 19, 2007, at 6:12 PM, Andreas Raab wrote:
>
>> Philippe Marschall wrote:
>>>> there is really no reason whatsoever why a working closure solution
>>>> couldn't be adopted to the current compiler (which would mitigate the
>>>> risk factors of adoption dramatically).
>>> They why was the closure solution of the new compiler rejected?
>>
>> "The closure solution rejected"? Not that I know of. I haven't even
>> seen a thorough discussion about its merits (which should include
>> benchmarks etc) - what I did see was some talk that led me to believe
>> that there is still a ways to go here. To the best of my knowledge
>> this solution isn't "final" by any means.
>>
>> Or do you mean the compiler itself? In which case all I can say it's
>> hard to recommend a compiler which can't even recompile Object ;-)
>>
>> Cheers,
>>  - Andreas
>>
>>
>
>     Mth
>
>
>
>
>

> stephane ducasse wrote:
>> The newcompiler can recompile the complete image. I do not which  
>> version you are using.
>
> * Start with 3.9.7067-final
> * From Squeakmap load the latest published version of AST
> * From Squeakmap load the latest published version of NewCompiler
> * Preferences enable: #compileUseNewCompiler
> * Object compileAll
> * Boom.
>
> Besides, when I tried to load NewCompiler yesterday evening via  
> Universes in 3.10 it failed to load entirely. Not sure why (and I  
> wasn't in a mood of trying to find out).
>
> Cheers,
>  - Andreas
>

>
> On 19-Dec-07, at 8:19 PM, Andreas Raab wrote:
>
>> Mathieu Suen wrote:
>>> Well so propose us your solution.
>>
>> Okay, I promised not to invent a solution but it's one of "those"
>> problems that I've thought about for a while in the past so let's see.
>> Here is how I would address the problem of implementing full closures
>> in Squeak:
>
> Well at a quick look it appears plausible - but I have a much simpler
> idea. Eliot solved this problems in 1987 - in fact I *think* he was the
> first so to do but wouldn't bet any large amount of money - and the
> basics are described in his seminal paper from the 87 OOPSLa proceedings.

> Mathieu Suen wrote:
>> Sorry I made a mistake in the universe browser. It is now fix
>
> Indeed much better now. I'm getting halfways through now with  
> recompiling everything but now it dies in the middle of recompiling  
> Preferences (you can try this by executing "Preferences  
> compileAll"). Any ideas?
>
> Cheers,
>  - Andreas
>

> Mathieu Suen wrote:
>> Well so propose us your solution.
>
> Okay, I promised not to invent a solution but it's one of "those"  
> problems that I've thought about for a while in the past so let's see.  
> Here is how I would address the problem of implementing full closures in  
> Squeak:
>
> 1) First, I would fix the "attempt to evaluate a block that is already  
> evaluated" problem. That's about ten lines of code in the VM - if you  
> look at primitiveValue[withArgs] you'll find that we fail if the  
> CallerIndex is non-nil. Instead of failing, I'd clone the original  
> context and continue as usual.
>
> At this point you'd be able to execute code like:
>
>    | v |
>    fac := [v > 0 ifTrue:[1] ifFalse:[v := v-1. fac value * (v+1)]].
>    fac value: 10.
>
> which allows recursive evaluation but without block args

> (since they get hacked in some terrible ways).
>
> 2) Next, I'd deal with block variables (temps and args). Instead of  
> sharing the method contexts, blocks would store their own temps and  
> access them via tempAt:. So that for example code like here:
>
>    self do:[:v| | v2 | v2 := v squared. v2].
>
> would be compiled into to:
>
>    self do:[:v|
>      thisContext push: nil. "allocate v2"
>      thisContext tempAt: 2 "v2" put: (thisContext tempAt: 1 "v") squared.
>      thisContext tempAt: 2.
>    ].

> The implementation would be pretty straightforward too, since all you'd  
> have to do is to add BlockVariableNode that generates tempAt: messages  
> (plus some initialization). Very, very straightforward.
>
> 3) Next, I would deal with "outer" state like here:
>
>    self do:[:x|
>      x do:[:y| x * y]
>    ].
>
> This is probably the hardest part since dealing with "non-method  
> closed-over state" can be tricky due to life-time issues. Anyway, for  
> starters I would just add a hidden temporary "outerContext" for all  
> blocks that is just a temp like any other such that the compiler can  
> generate code like:
>
>    self do:[:x|
>      x do:[:y|
>        ((thisContext tempAt: 1) "outerContext" tempAt: 1 "x") *
>          (thisContext tempAt: 2) "y"
>      ].
>    ].

> I would start with the obvious push/popBlockTemp bytecodes which would  
> bring all of the above up to par with current Squeak (except (3) which  
> could still be slower). Finally, one might look at the life-time of  
> closed-over state - it is certainly not necessary to preserve the entire  
> outer context so there may be other things that could be done to fix it.
>
> In any case, the above looks pretty simple and straightforward. Wouldn't  
> you agree?
>
> Cheers,
>    - Andreas
>
>

> Mathieu Suen wrote:
>> Well so propose us your solution.
>
> Okay, I promised not to invent a solution but it's one of "those"  
> problems that I've thought about for a while in the past so let's  
> see. Here is how I would address the problem of implementing full  
> closures in Squeak:
>
> 1) First, I would fix the "attempt to evaluate a block that is  
> already evaluated" problem. That's about ten lines of code in the VM  
> - if you look at primitiveValue[withArgs] you'll find that we fail  
> if the CallerIndex is non-nil. Instead of failing, I'd clone the  
> original context and continue as usual.
>
> At this point you'd be able to execute code like:
>
>  | v |
>  fac := [v > 0 ifTrue:[1] ifFalse:[v := v-1. fac value * (v+1)]].
>  fac value: 10.
>
> which allows recursive evaluation but without block args (since they  
> get hacked in some terrible ways).
>
> 2) Next, I'd deal with block variables (temps and args). Instead of  
> sharing the method contexts, blocks would store their own temps and  
> access them via tempAt:. So that for example code like here:
>
>  self do:[:v| | v2 | v2 := v squared. v2].
>
> would be compiled into to:
>
>  self do:[:v|
>    thisContext push: nil. "allocate v2"
>    thisContext tempAt: 2 "v2" put: (thisContext tempAt: 1 "v")  
> squared.
>    thisContext tempAt: 2.
>  ].
>
> This would be slower than what we have today but obviously, the  
> pattern "thisContext tempAt:" is a prime candidate for optimization.  
> At this point we'd be able to run the following correctly:
>
> fac [:v| v = 0 ifTrue:[1] ifFalse:[ v * (fac value: v-1)].
>
> The implementation would be pretty straightforward too, since all  
> you'd have to do is to add BlockVariableNode that generates tempAt:  
> messages (plus some initialization). Very, very straightforward.
>
> 3) Next, I would deal with "outer" state like here:
>
>  self do:[:x|
>    x do:[:y| x * y]
>  ].
>
> This is probably the hardest part since dealing with "non-method  
> closed-over state" can be tricky due to life-time issues. Anyway,  
> for starters I would just add a hidden temporary "outerContext" for  
> all blocks that is just a temp like any other such that the compiler  
> can generate code like:
>
>  self do:[:x|
>    x do:[:y|
>      ((thisContext tempAt: 1) "outerContext" tempAt: 1 "x") *
>        (thisContext tempAt: 2) "y"
>    ].
>  ].
>
> At this point (I think - please correct me if I'm missing something)  
> we have *complete* closure semantics in a straightforward way. What  
> remains is optimization.
>
> 4) Lastly, optimization. I would start with the obvious push/
> popBlockTemp bytecodes which would bring all of the above up to par  
> with current Squeak (except (3) which could still be slower).  
> Finally, one might look at the life-time of closed-over state - it  
> is certainly not necessary to preserve the entire outer context so  
> there may be other things that could be done to fix it.
>
> In any case, the above looks pretty simple and straightforward.  
> Wouldn't you agree?
>
> Cheers,
>  - Andreas
>

> On Thu, 20 Dec 2007 05:19:33 +0100, Andreas Raab wrote:
>
>> Mathieu Suen wrote:
>>> Well so propose us your solution.
>>
>> Okay, I promised not to invent a solution but it's one of "those"  
>> problems that I've thought about for a while in the past so let's  
>> see. Here is how I would address the problem of implementing full  
>> closures in Squeak:
>>
>> 1) First, I would fix the "attempt to evaluate a block that is  
>> already evaluated" problem. That's about ten lines of code in the  
>> VM - if you look at primitiveValue[withArgs] you'll find that we  
>> fail if the CallerIndex is non-nil. Instead of failing, I'd clone  
>> the original context and continue as usual.
>>
>> At this point you'd be able to execute code like:
>>
>>   | v |
>>   fac := [v > 0 ifTrue:[1] ifFalse:[v := v-1. fac value * (v+1)]].
>>   fac value: 10.
>>
>> which allows recursive evaluation but without block args
>
> Did you mean
>
>    | v |
>    v := 10.
>    fac := [v > 0 ifFalse:[1] ifTrue:[v := v-1. fac fixTemps clone  
> value * (v+1)]].
>    fac clone value
>
> I confess that I copied the idea from Self's view on method  
> activation (a clone of a template :)
>
> But then, how'd that be on Smalltalks other than Squeak, i.e. if a  
> Seasider creates+tests such code with Squeak and later runs it on  
> another Seaside-capable VM? Looks like opening the cross-Smalltalk  
> Hunting Season :(
>
>> (since they get hacked in some terrible ways).
>>
>> 2) Next, I'd deal with block variables (temps and args). Instead of  
>> sharing the method contexts, blocks would store their own temps and  
>> access them via tempAt:. So that for example code like here:
>>
>>   self do:[:v| | v2 | v2 := v squared. v2].
>>
>> would be compiled into to:
>>
>>   self do:[:v|
>>     thisContext push: nil. "allocate v2"
>>     thisContext tempAt: 2 "v2" put: (thisContext tempAt: 1 "v")  
>> squared.
>>     thisContext tempAt: 2.
>>   ].
>
> Sure, that'd be an easy one for the compiler+decompiler guild.
>
>> This would be slower than what we have today but obviously, the  
>> pattern "thisContext tempAt:" is a prime candidate for optimization.
>
> AFAIK there's no bytecode which pushes/pops temps in  
> *activeContext*, only in *homeContext* (and so the main reason  
> behind this discussion). That looks like a major change to VM.
>
>> At this point we'd be able to run the following correctly:
>>
>> fac [:v| v = 0 ifTrue:[1] ifFalse:[ v * (fac value: v-1)].
>
> This statement does not parse. Can you reformulate without syntax  
> error.
>
>> The implementation would be pretty straightforward too, since all  
>> you'd have to do is to add BlockVariableNode that generates tempAt:  
>> messages (plus some initialization). Very, very straightforward.
>>
>> 3) Next, I would deal with "outer" state like here:
>>
>>   self do:[:x|
>>     x do:[:y| x * y]
>>   ].
>>
>> This is probably the hardest part since dealing with "non-method  
>> closed-over state" can be tricky due to life-time issues. Anyway,  
>> for starters I would just add a hidden temporary "outerContext" for  
>> all blocks that is just a temp like any other such that the  
>> compiler can generate code like:
>>
>>   self do:[:x|
>>     x do:[:y|
>>       ((thisContext tempAt: 1) "outerContext" tempAt: 1 "x") *
>>         (thisContext tempAt: 2) "y"
>>     ].
>>   ].
>
> Sort of "outerContext" is already part of the BlockClosure handling  
> with NewCompiler, there it's an instance of ClosureEnvironment.
>
>> At this point (I think - please correct me if I'm missing  
>> something) we have *complete* closure semantics in a  
>> straightforward way.
>
> I would like to see comment(s) on the above from Marcus, but he's  
> currently not subscribed to squeak-dev (cc'ed him anyways).
>
>> What remains is optimization.
>
> Bryce & Marcus & Math & my discussed optimizing BlockClosure
> +bytecodes some time ago, if that's of interest
>
> - http://www.google.com/search?&q=Newcompiler+bytecode+recycling+closure
>
>> 4) Lastly, optimization.
>
> During the discussion we came to the point that context recycling,  
> which is interrupted by explicit and implicit #pushActiveContext,  
> would no longer be hindered because BlockClosure seems to be a  
> complete replacement for BlockContext. By that time it looked like a  
> very promising speed-up.
>
> /Klaus
>
>> I would start with the obvious push/popBlockTemp bytecodes which  
>> would bring all of the above up to par with current Squeak (except  
>> (3) which could still be slower). Finally, one might look at the  
>> life-time of closed-over state - it is certainly not necessary to  
>> preserve the entire outer context so there may be other things that  
>> could be done to fix it.
>>
>> In any case, the above looks pretty simple and straightforward.  
>> Wouldn't you agree?
>>
>> Cheers,
>>   - Andreas
>>
>>
>
>