[Compiler] #bindings: improvement: Global shadowing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

[Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4
Hi,

the #binding: API until now did not allow to shadow globals. But there is actually no reason to forbid that
(and it can actually be interesting, e.g. to compile code where one class ref is switched to another).

The Pull Request does:

- add comment to OCExtraBindingScope
- categorize tests
- change #lookupVar: to allow shadowing of Globals.
- change test to reflect the new behaviour

This means you can do:

result := Smalltalk compiler
                bindings: {(#Object -> Point)} asDictionary;
                evaluate: 'Object new'.

https://github.com/pharo-project/pharo/pull/1379

        Marcus
Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4


> On 18 May 2018, at 10:39, Marcus Denker <[hidden email]> wrote:
>
> Hi,
>
> the #binding: API until now did not allow to shadow globals. But there is actually no reason to forbid that
> (and it can actually be interesting, e.g. to compile code where one class ref is switched to another).
>
> The Pull Request does:
>
> - add comment to OCExtraBindingScope
> - categorize tests
> - change #lookupVar: to allow shadowing of Globals.
> - change test to reflect the new behaviour
>
> This means you can do:
>
> result := Smalltalk compiler
> bindings: {(#Object -> Point)} asDictionary;
> evaluate: 'Object new'.
>
> https://github.com/pharo-project/pharo/pull/1379
>

A second iteration now makes it work correctly for assignments:

        https://github.com/pharo-project/pharo/pull/1397

Smalltalk compiler
        bindings: {(#test -> Point)} asDictionary;
        evaluate: 'test := 42'.

        Marcus


Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Thierry Goubier
Interesting...

Thierry

2018-05-22 14:18 GMT+02:00 Marcus Denker <[hidden email]>:

>
>
>> On 18 May 2018, at 10:39, Marcus Denker <[hidden email]> wrote:
>>
>> Hi,
>>
>> the #binding: API until now did not allow to shadow globals. But there is actually no reason to forbid that
>> (and it can actually be interesting, e.g. to compile code where one class ref is switched to another).
>>
>> The Pull Request does:
>>
>> - add comment to OCExtraBindingScope
>> - categorize tests
>> - change #lookupVar: to allow shadowing of Globals.
>> - change test to reflect the new behaviour
>>
>> This means you can do:
>>
>> result := Smalltalk compiler
>>               bindings: {(#Object -> Point)} asDictionary;
>>               evaluate: 'Object new'.
>>
>> https://github.com/pharo-project/pharo/pull/1379
>>
>
> A second iteration now makes it work correctly for assignments:
>
>         https://github.com/pharo-project/pharo/pull/1397
>
> Smalltalk compiler
>         bindings: {(#test -> Point)} asDictionary;
>         evaluate: 'test := 42'.
>
>         Marcus
>
>

Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4
In reply to this post by Marcus Denker-4
>
> A second iteration now makes it work correctly for assignments:
>
> https://github.com/pharo-project/pharo/pull/1397
>
> Smalltalk compiler
>        bindings: {(#test -> Point)} asDictionary;
>        evaluate: 'test := 42’.

While #bindings: allows us now to add additional bindings or override global bindings from Smalltalk globals, there is another feature missing that would be nice: be able to set the global environment

I want to be able to tell the compiler: Do not use Smalltalk globals, but this SystemDictionary instead.

        https://github.com/pharo-project/pharo/pull/1406

with this, we can create a SystemDictionary with just one binding in it:

        environment := SystemDictionary new.
        environment at: #MyClass put: Point.

then we compile a method:

        method := Smalltalk compiler
                                        environment: environment;
                                        compile: 'tt ^MyClass’.

and if we execute it, it shows that compilation used the environment:

        return := method valueWithReceiver: nil arguments: #().
        self assert: return equals: Point.

If we try to compile a ref to Object, we get a nil ref:

        method := Smalltalk compiler
                                        environment: environment;
                                        compile: 'tt ^Object'.
        return := method valueWithReceiver: nil arguments: #().
        self assert: return equals: nil.

But, as the Undeclared mechanism is not aware of it, it puts an undeclared for Object there.

In the same direction, all objects the compiler creates come still from Smalltalk globals (CompiledMethod, all Literals…).

But all this can be improved, we do pass the environment down the compiler chain already so that it is even available in the Parser/Scanner. Thus we could e.g. create Literals as instances from the classes in the environment. But how exactly has to be seen.

        Marcus



Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Thierry Goubier
Marcus,

can you do inheritence / masking of a hierarchy of system dictionaries ?

Could be used to make "kind of" sandboxes, around, say, user code so
that it does not have the ability to override a certain system core.

Thierry

2018-05-23 11:45 GMT+02:00 Marcus Denker <[hidden email]>:

>>
>> A second iteration now makes it work correctly for assignments:
>>
>>       https://github.com/pharo-project/pharo/pull/1397
>>
>> Smalltalk compiler
>>        bindings: {(#test -> Point)} asDictionary;
>>        evaluate: 'test := 42’.
>
> While #bindings: allows us now to add additional bindings or override global bindings from Smalltalk globals, there is another feature missing that would be nice: be able to set the global environment
>
> I want to be able to tell the compiler: Do not use Smalltalk globals, but this SystemDictionary instead.
>
>         https://github.com/pharo-project/pharo/pull/1406
>
> with this, we can create a SystemDictionary with just one binding in it:
>
>         environment := SystemDictionary new.
>         environment at: #MyClass put: Point.
>
> then we compile a method:
>
>         method := Smalltalk compiler
>                                         environment: environment;
>                                         compile: 'tt ^MyClass’.
>
> and if we execute it, it shows that compilation used the environment:
>
>         return := method valueWithReceiver: nil arguments: #().
>         self assert: return equals: Point.
>
> If we try to compile a ref to Object, we get a nil ref:
>
>         method := Smalltalk compiler
>                                         environment: environment;
>                                         compile: 'tt ^Object'.
>         return := method valueWithReceiver: nil arguments: #().
>         self assert: return equals: nil.
>
> But, as the Undeclared mechanism is not aware of it, it puts an undeclared for Object there.
>
> In the same direction, all objects the compiler creates come still from Smalltalk globals (CompiledMethod, all Literals…).
>
> But all this can be improved, we do pass the environment down the compiler chain already so that it is even available in the Parser/Scanner. Thus we could e.g. create Literals as instances from the classes in the environment. But how exactly has to be seen.
>
>         Marcus
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4


> On 23 May 2018, at 11:49, Thierry Goubier <[hidden email]> wrote:
>
> Marcus,
>
> can you do inheritence / masking of a hierarchy of system dictionaries ?
>
> Could be used to make "kind of" sandboxes, around, say, user code so
> that it does not have the ability to override a certain system core.
>

Yes, I was starting to think about exactly that… I will do some experiments.

        Marcus


Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Sean P. DeNigris
Administrator
In reply to this post by Marcus Denker-4
Marcus Denker-4 wrote
> method := Smalltalk compiler
> environment: environment;
> compile: 'tt ^MyClass’.

Awesome! Is it possible to override a global from /within/ a method (e.g. in
a test)?



-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Thierry Goubier
2018-05-23 13:20 GMT+02:00 Sean P. DeNigris <[hidden email]>:
> Marcus Denker-4 wrote
>>       method := Smalltalk compiler
>>                                       environment: environment;
>>                                       compile: 'tt ^MyClass’.
>
> Awesome! Is it possible to override a global from /within/ a method (e.g. in
> a test)?

I think at the class level by manipulating that class compiler (and
forcing a method recompile...)

Thierry

>
>
> -----
> Cheers,
> Sean
> --
> Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>

Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4


> On 23 May 2018, at 13:30, Thierry Goubier <[hidden email]> wrote:
>
> 2018-05-23 13:20 GMT+02:00 Sean P. DeNigris <[hidden email]>:
>> Marcus Denker-4 wrote
>>>      method := Smalltalk compiler
>>>                                      environment: environment;
>>>                                      compile: 'tt ^MyClass’.
>>
>> Awesome! Is it possible to override a global from /within/ a method (e.g. in
>> a test)?
>
> I think at the class level by manipulating that class compiler

Yes. You can override the class side method:

in Behaviour, it is defined as

compiler
        "Answer a compiler appropriate for source methods of this class."

        ^self compilerClass new
                environment: self environment;
                class: self

so you can just

compiler
        “Change the compiler environment "

        ^super compiler
                environment: self myEnvironment


and it should work.

an a per method bases, we can only manipulate the compiler via Pragmas to set options:

        <compilerOptions: #(+ optionInlineTimesRepeat)>

for example.

> (and
> forcing a method recompile...)
>
indeed.. I still believe that we should late-bind global access :-)

        Marcus
Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4


> On 23 May 2018, at 13:47, Marcus Denker <[hidden email]> wrote:
>
>
>
>> On 23 May 2018, at 13:30, Thierry Goubier <[hidden email]> wrote:
>>
>> 2018-05-23 13:20 GMT+02:00 Sean P. DeNigris <[hidden email]>:
>>> Marcus Denker-4 wrote
>>>>     method := Smalltalk compiler
>>>>                                     environment: environment;
>>>>                                     compile: 'tt ^MyClass’.
>>>
>>> Awesome! Is it possible to override a global from /within/ a method (e.g. in
>>> a test)?
>>
>> I think at the class level by manipulating that class compiler
>
> Yes. You can override the class side method:
>
> in Behaviour, it is defined as
>
> compiler
> "Answer a compiler appropriate for source methods of this class."
>
> ^self compilerClass new
> environment: self environment;
> class: self
>
> so you can just
>
> compiler
> “Change the compiler environment "
>
> ^super compiler
> environment: self myEnvironment
>
>
> and it should work.

Here we can see, too, that we need to do more work. In Behaviour:

environment
        "Return the environment in which the receiver is visible"
        ^Smalltalk globals

For classes to automatically compile with the environment they are in, we somehow need to return here
that environment…

        Marcus


Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Sean P. DeNigris
Administrator
In reply to this post by Marcus Denker-4
Marcus Denker-4 wrote
> indeed.. I still believe that we should late-bind global access :-)

I'll keep +1-ing every time you mention this ha ha



-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4
In reply to this post by Marcus Denker-4


On 23 May 2018, at 13:53, Marcus Denker <[hidden email]> wrote:



On 23 May 2018, at 13:47, Marcus Denker <[hidden email]> wrote:



On 23 May 2018, at 13:30, Thierry Goubier <[hidden email]> wrote:

2018-05-23 13:20 GMT+02:00 Sean P. DeNigris <[hidden email]>:
Marcus Denker-4 wrote
   method := Smalltalk compiler
                                   environment: environment;
                                   compile: 'tt ^MyClass’.

Awesome! Is it possible to override a global from /within/ a method (e.g. in
a test)?

I think at the class level by manipulating that class compiler

Yes. You can override the class side method:

in Behaviour, it is defined as

compiler
"Answer a compiler appropriate for source methods of this class."

^self compilerClass new
environment: self environment;
class: self

so you can just 

compiler
“Change the compiler environment "

^super compiler
environment: self myEnvironment


and it should work.

Here we can see, too, that we need to do more work. In Behaviour:

environment
"Return the environment in which the receiver is visible"
^Smalltalk globals

For classes to automatically compile with the environment they are in, we somehow need to return here
that environment… 

This is already correct (Class overrides it, all classes have the ivar environment correctly set)

Marcus
Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4
In reply to this post by Marcus Denker-4


> On 18 May 2018, at 10:39, Marcus Denker <[hidden email]> wrote:
>
> Hi,
>
> the #binding: API until now did not allow to shadow globals. But there is actually no reason to forbid that
> (and it can actually be interesting, e.g. to compile code where one class ref is switched to another).
>
> The Pull Request does:
>
> - add comment to OCExtraBindingScope
> - categorize tests
> - change #lookupVar: to allow shadowing of Globals.
> - change test to reflect the new behaviour
>
> This means you can do:
>
> result := Smalltalk compiler
> bindings: {(#Object -> Point)} asDictionary;
> evaluate: 'Object new’.

So what can we do with this? If you develop a compiler, it is always very annoying that any change that you do actually breaks the compiler if you do it wrong. Larger changes over multiple methods get quite hard to do.

One thing that we will soon have is the “accept for test” feature that I posted recently.
Another idea is to use the #bindings: feature of the compiler.

I did small experiment, I added this method to the OpalCompiler class on the class side:

prepareForDevelopment
        "this method sets up an overlay in the global var CompilerOverlay"
        <script>
        | overlayEnvironment |
       
        overlayEnvironment := Dictionary new.
        Smalltalk globals at: #CompilerOverlay put: overlayEnvironment.
       
        "now we put a copy of all the classes of the compiler package into the environment"
        self package definedClasses do: [ :class |
                overlayEnvironment at: class name put: class copy].

        “then we recompile the classes in the environment with itself as an overlay"
        overlayEnvironment valuesDo: [ :class |
                class methods do: [ :method | |newMethod|
                 newMethod := class compiler
                                bindings: overlayEnvironment;
                                compile: method sourceCode.
                 class addSelectorSilently: method selector withMethod: newMethod.
                ]].

After that is done, we have CompilerOverlay global that contains a copy of all the classes of the compiler recompiled so that they use all the classes inside this copy.

So we now can set this copy as the main compiler for the image:

SmalltalkImage compilerClass: (CompilerOverlay at: #OpalCompiler)

And after that, you can put a #halt deep in the compiler (or do some change that needs multiple steps), yet compilation will not be affected.
Most tests (not all) are writing in a way that they use classes from the package directly (they do not go via the global #compilerClass setting).

So this means that one can now change the compiler as much as one wants, test it by running tests and then be quite sure that it will not break anything.

Shortcomings:
        -> extensions are not treated, only the main package
        -> Not used much, for sure needs testing to see if this really works.
        -> Can we get rid of the #CompilerOverlay global?

        Marcus

 




Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4
>
> Shortcomings:
> -> extensions are not treated, only the main package
> -> Not used much, for sure needs testing to see if this really works.
 
       —> there might be other places in addition global #compilerClass: where we need to point to the overlay version
             For the complete “compile a method” it works, but I think there are other cases where it is not enough.
           
> -> Can we get rid of the #CompilerOverlay global?
>
     I think it can be a class variable of OpalCompiler, this way we can turn the whole thing into a Setting (if it turns out to be usable)

        Marcus
Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4
In reply to this post by Marcus Denker-4


So what can we do with this? If you develop a compiler, it is always very annoying that any change that you do actually breaks the compiler if you do it wrong. Larger changes over multiple methods get quite hard to do.

One thing that we will soon have is the “accept for test” feature that I posted recently. 
Another idea is to use the #bindings: feature of the compiler.

I did small experiment, I added this method to the OpalCompiler class on the class side:

…..


And after that, you can put a #halt deep in the compiler (or do some change that needs multiple steps), yet compilation will not be affected.
Most tests (not all) are writing in a way that they use classes from the package directly (they do not go via the global #compilerClass setting).

So this means that one can now change the compiler as much as one wants, test it by running tests and then be quite sure that it will not break anything.

Shortcomings:
-> extensions are not treated, only the main package
-> Not used much, for sure needs testing to see if this really works.
-> Can we get rid of the #CompilerOverlay global?

I made a pull request:  
Marcus
Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4
In reply to this post by Marcus Denker-4


On 28 May 2018, at 12:07, Marcus Denker <[hidden email]> wrote:


Shortcomings:
-> extensions are not treated, only the main package
-> Not used much, for sure needs testing to see if this really works.

      —> there might be other places in addition global #compilerClass: where we need to point to the overlay version
            For the complete “compile a method” it works, but I think there are other cases where it is not enough.

I think this one makes it really usable:


Now I can even put a halt in the semantic analysis which is called by syntax highlighting in the browser at each keystroke.


Marcus
Reply | Threaded
Open this post in threaded view
|

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus Denker-4


On 29 May 2018, at 11:41, Marcus Denker <[hidden email]> wrote:



On 28 May 2018, at 12:07, Marcus Denker <[hidden email]> wrote:


Shortcomings:
-> extensions are not treated, only the main package
-> Not used much, for sure needs testing to see if this really works.

      —> there might be other places in addition global #compilerClass: where we need to point to the overlay version
            For the complete “compile a method” it works, but I think there are other cases where it is not enough.

I think this one makes it really usable:


Now I can even put a halt in the semantic analysis which is called by syntax highlighting in the browser at each keystroke.

This is now merged with a small addition to provide a UI via the Settings:



If this is selected, the overlay is active.

Marcus