Smalltalk › Pharo › Pharo Smalltalk Developers

[Compiler] #bindings: improvement: Global shadowing

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

17 messages Options

Marcus Denker-4

[Compiler] #bindings: improvement: Global shadowing

Hi,

the #binding: API until now did not allow to shadow globals. But there is actually no reason to forbid that
(and it can actually be interesting, e.g. to compile code where one class ref is switched to another).

The Pull Request does:

- add comment to OCExtraBindingScope
- categorize tests
- change #lookupVar: to allow shadowing of Globals.
- change test to reflect the new behaviour

This means you can do:

result := Smalltalk compiler
bindings: {(#Object -> Point)} asDictionary;
evaluate: 'Object new'.

https://github.com/pharo-project/pharo/pull/1379

Marcus

Marcus Denker-4

Re: [Compiler] #bindings: improvement: Global shadowing

> On 18 May 2018, at 10:39, Marcus Denker <[hidden email]> wrote:
>
> Hi,
>
> the #binding: API until now did not allow to shadow globals. But there is actually no reason to forbid that
> (and it can actually be interesting, e.g. to compile code where one class ref is switched to another).
>
> The Pull Request does:
>
> - add comment to OCExtraBindingScope
> - categorize tests
> - change #lookupVar: to allow shadowing of Globals.
> - change test to reflect the new behaviour
>
> This means you can do:
>
> result := Smalltalk compiler
> bindings: {(#Object -> Point)} asDictionary;
> evaluate: 'Object new'.
>
> https://github.com/pharo-project/pharo/pull/1379
>

A second iteration now makes it work correctly for assignments:

https://github.com/pharo-project/pharo/pull/1397

Smalltalk compiler
bindings: {(#test -> Point)} asDictionary;
evaluate: 'test := 42'.

Marcus

Thierry Goubier

Re: [Compiler] #bindings: improvement: Global shadowing

Interesting...

Thierry

2018-05-22 14:18 GMT+02:00 Marcus Denker <[hidden email]>:

>
>
>> On 18 May 2018, at 10:39, Marcus Denker <[hidden email]> wrote:
>>
>> Hi,
>>
>> the #binding: API until now did not allow to shadow globals. But there is actually no reason to forbid that
>> (and it can actually be interesting, e.g. to compile code where one class ref is switched to another).
>>
>> The Pull Request does:
>>
>> - add comment to OCExtraBindingScope
>> - categorize tests
>> - change #lookupVar: to allow shadowing of Globals.
>> - change test to reflect the new behaviour
>>
>> This means you can do:
>>
>> result := Smalltalk compiler
>> bindings: {(#Object -> Point)} asDictionary;
>> evaluate: 'Object new'.
>>
>> https://github.com/pharo-project/pharo/pull/1379
>>
>
> A second iteration now makes it work correctly for assignments:
>
> https://github.com/pharo-project/pharo/pull/1397
>
> Smalltalk compiler
> bindings: {(#test -> Point)} asDictionary;
> evaluate: 'test := 42'.
>
> Marcus
>
>

Marcus Denker-4

Re: [Compiler] #bindings: improvement: Global shadowing

In reply to this post by Marcus Denker-4

>
> A second iteration now makes it work correctly for assignments:
>
> https://github.com/pharo-project/pharo/pull/1397
>
> Smalltalk compiler
> bindings: {(#test -> Point)} asDictionary;
> evaluate: 'test := 42’.

While #bindings: allows us now to add additional bindings or override global bindings from Smalltalk globals, there is another feature missing that would be nice: be able to set the global environment

I want to be able to tell the compiler: Do not use Smalltalk globals, but this SystemDictionary instead.

https://github.com/pharo-project/pharo/pull/1406

with this, we can create a SystemDictionary with just one binding in it:

environment := SystemDictionary new.
environment at: #MyClass put: Point.

then we compile a method:

method := Smalltalk compiler
environment: environment;
compile: 'tt ^MyClass’.

and if we execute it, it shows that compilation used the environment:

return := method valueWithReceiver: nil arguments: #().
self assert: return equals: Point.

If we try to compile a ref to Object, we get a nil ref:

method := Smalltalk compiler
environment: environment;
compile: 'tt ^Object'.
return := method valueWithReceiver: nil arguments: #().
self assert: return equals: nil.

But, as the Undeclared mechanism is not aware of it, it puts an undeclared for Object there.

In the same direction, all objects the compiler creates come still from Smalltalk globals (CompiledMethod, all Literals…).

But all this can be improved, we do pass the environment down the compiler chain already so that it is even available in the Parser/Scanner. Thus we could e.g. create Literals as instances from the classes in the environment. But how exactly has to be seen.

Marcus

Thierry Goubier

Re: [Compiler] #bindings: improvement: Global shadowing

Marcus,

can you do inheritence / masking of a hierarchy of system dictionaries ?

Could be used to make "kind of" sandboxes, around, say, user code so
that it does not have the ability to override a certain system core.

Thierry

2018-05-23 11:45 GMT+02:00 Marcus Denker <[hidden email]>:

>>
>> A second iteration now makes it work correctly for assignments:
>>
>> https://github.com/pharo-project/pharo/pull/1397
>>
>> Smalltalk compiler
>> bindings: {(#test -> Point)} asDictionary;
>> evaluate: 'test := 42’.
>
> While #bindings: allows us now to add additional bindings or override global bindings from Smalltalk globals, there is another feature missing that would be nice: be able to set the global environment
>
> I want to be able to tell the compiler: Do not use Smalltalk globals, but this SystemDictionary instead.
>
> https://github.com/pharo-project/pharo/pull/1406
>
> with this, we can create a SystemDictionary with just one binding in it:
>
> environment := SystemDictionary new.
> environment at: #MyClass put: Point.
>
> then we compile a method:
>
> method := Smalltalk compiler
> environment: environment;
> compile: 'tt ^MyClass’.
>
> and if we execute it, it shows that compilation used the environment:
>
> return := method valueWithReceiver: nil arguments: #().
> self assert: return equals: Point.
>
> If we try to compile a ref to Object, we get a nil ref:
>
> method := Smalltalk compiler
> environment: environment;
> compile: 'tt ^Object'.
> return := method valueWithReceiver: nil arguments: #().
> self assert: return equals: nil.
>
> But, as the Undeclared mechanism is not aware of it, it puts an undeclared for Object there.
>
> In the same direction, all objects the compiler creates come still from Smalltalk globals (CompiledMethod, all Literals…).
>
> But all this can be improved, we do pass the environment down the compiler chain already so that it is even available in the Parser/Scanner. Thus we could e.g. create Literals as instances from the classes in the environment. But how exactly has to be seen.
>
> Marcus
>
>
>

Marcus Denker-4

Re: [Compiler] #bindings: improvement: Global shadowing

> On 23 May 2018, at 11:49, Thierry Goubier <[hidden email]> wrote:
>
> Marcus,
>
> can you do inheritence / masking of a hierarchy of system dictionaries ?
>
> Could be used to make "kind of" sandboxes, around, say, user code so
> that it does not have the ability to override a certain system core.
>

Yes, I was starting to think about exactly that… I will do some experiments.

Marcus

Sean P. DeNigris

Re: [Compiler] #bindings: improvement: Global shadowing

Administrator

In reply to this post by Marcus Denker-4

Marcus Denker-4 wrote
> method := Smalltalk compiler
> environment: environment;
> compile: 'tt ^MyClass’.

Awesome! Is it possible to override a global from /within/ a method (e.g. in
a test)?

-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Cheers,
Sean

Thierry Goubier

Re: [Compiler] #bindings: improvement: Global shadowing

2018-05-23 13:20 GMT+02:00 Sean P. DeNigris <[hidden email]>:
> Marcus Denker-4 wrote
>> method := Smalltalk compiler
>> environment: environment;
>> compile: 'tt ^MyClass’.
>
> Awesome! Is it possible to override a global from /within/ a method (e.g. in
> a test)?

I think at the class level by manipulating that class compiler (and
forcing a method recompile...)

Thierry

>
>
> -----
> Cheers,
> Sean
> --
> Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>

Marcus Denker-4

Re: [Compiler] #bindings: improvement: Global shadowing

> On 23 May 2018, at 13:30, Thierry Goubier <[hidden email]> wrote:
>
> 2018-05-23 13:20 GMT+02:00 Sean P. DeNigris <[hidden email]>:
>> Marcus Denker-4 wrote
>>> method := Smalltalk compiler
>>> environment: environment;
>>> compile: 'tt ^MyClass’.
>>
>> Awesome! Is it possible to override a global from /within/ a method (e.g. in
>> a test)?
>
> I think at the class level by manipulating that class compiler

Yes. You can override the class side method:

in Behaviour, it is defined as

compiler
"Answer a compiler appropriate for source methods of this class."

^self compilerClass new
environment: self environment;
class: self

so you can just

compiler
“Change the compiler environment "

^super compiler
environment: self myEnvironment

and it should work.

an a per method bases, we can only manipulate the compiler via Pragmas to set options:

<compilerOptions: #(+ optionInlineTimesRepeat)>

for example.

> (and
> forcing a method recompile...)
>
indeed.. I still believe that we should late-bind global access :-)

Marcus

Marcus Denker-4

Re: [Compiler] #bindings: improvement: Global shadowing

> On 23 May 2018, at 13:47, Marcus Denker <[hidden email]> wrote:
>
>
>
>> On 23 May 2018, at 13:30, Thierry Goubier <[hidden email]> wrote:
>>
>> 2018-05-23 13:20 GMT+02:00 Sean P. DeNigris <[hidden email]>:
>>> Marcus Denker-4 wrote
>>>> method := Smalltalk compiler
>>>> environment: environment;
>>>> compile: 'tt ^MyClass’.
>>>
>>> Awesome! Is it possible to override a global from /within/ a method (e.g. in
>>> a test)?
>>
>> I think at the class level by manipulating that class compiler
>
> Yes. You can override the class side method:
>
> in Behaviour, it is defined as
>
> compiler
> "Answer a compiler appropriate for source methods of this class."
>
> ^self compilerClass new
> environment: self environment;
> class: self
>
> so you can just
>
> compiler
> “Change the compiler environment "
>
> ^super compiler
> environment: self myEnvironment
>
>
> and it should work.

Here we can see, too, that we need to do more work. In Behaviour:

environment
"Return the environment in which the receiver is visible"
^Smalltalk globals

For classes to automatically compile with the environment they are in, we somehow need to return here
that environment…

Marcus

Sean P. DeNigris

Re: [Compiler] #bindings: improvement: Global shadowing

Administrator

In reply to this post by Marcus Denker-4

Marcus Denker-4 wrote
> indeed.. I still believe that we should late-bind global access :-)

I'll keep +1-ing every time you mention this ha ha

-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Cheers,
Sean

Marcus Denker-4

Re: [Compiler] #bindings: improvement: Global shadowing

In reply to this post by Marcus Denker-4

On 23 May 2018, at 13:53, Marcus Denker <[hidden email]> wrote:

On 23 May 2018, at 13:47, Marcus Denker <[hidden email]> wrote:

On 23 May 2018, at 13:30, Thierry Goubier <[hidden email]> wrote:

2018-05-23 13:20 GMT+02:00 Sean P. DeNigris <[hidden email]>:
Marcus Denker-4 wrote
   method := Smalltalk compiler
                                   environment: environment;
                                   compile: 'tt ^MyClass’.

Awesome! Is it possible to override a global from /within/ a method (e.g. in
a test)?

I think at the class level by manipulating that class compiler

Yes. You can override the class side method:

in Behaviour, it is defined as

compiler
"Answer a compiler appropriate for source methods of this class."

^self compilerClass new
environment: self environment;
class: self

so you can just

compiler
“Change the compiler environment "

^super compiler
environment: self myEnvironment

and it should work.

Here we can see, too, that we need to do more work. In Behaviour:

environment
"Return the environment in which the receiver is visible"
^Smalltalk globals

For classes to automatically compile with the environment they are in, we somehow need to return here
that environment…

This is already correct (Class overrides it, all classes have the ivar environment correctly set)

Marcus

Marcus Denker-4

Re: [Compiler] #bindings: improvement: Global shadowing

In reply to this post by Marcus Denker-4

So what can we do with this? If you develop a compiler, it is always very annoying that any change that you do actually breaks the compiler if you do it wrong. Larger changes over multiple methods get quite hard to do.

One thing that we will soon have is the “accept for test” feature that I posted recently.
Another idea is to use the #bindings: feature of the compiler.

I did small experiment, I added this method to the OpalCompiler class on the class side:

prepareForDevelopment
"this method sets up an overlay in the global var CompilerOverlay"
<script>
| overlayEnvironment |

overlayEnvironment := Dictionary new.
Smalltalk globals at: #CompilerOverlay put: overlayEnvironment.

"now we put a copy of all the classes of the compiler package into the environment"
self package definedClasses do: [ :class |
overlayEnvironment at: class name put: class copy].

“then we recompile the classes in the environment with itself as an overlay"
overlayEnvironment valuesDo: [ :class |
class methods do: [ :method | |newMethod|
newMethod := class compiler
bindings: overlayEnvironment;
compile: method sourceCode.
class addSelectorSilently: method selector withMethod: newMethod.
]].

After that is done, we have CompilerOverlay global that contains a copy of all the classes of the compiler recompiled so that they use all the classes inside this copy.

So we now can set this copy as the main compiler for the image:

SmalltalkImage compilerClass: (CompilerOverlay at: #OpalCompiler)

And after that, you can put a #halt deep in the compiler (or do some change that needs multiple steps), yet compilation will not be affected.
Most tests (not all) are writing in a way that they use classes from the package directly (they do not go via the global #compilerClass setting).

So this means that one can now change the compiler as much as one wants, test it by running tests and then be quite sure that it will not break anything.

Shortcomings:
-> extensions are not treated, only the main package
-> Not used much, for sure needs testing to see if this really works.
-> Can we get rid of the #CompilerOverlay global?

Marcus

Marcus Denker-4

Re: [Compiler] #bindings: improvement: Global shadowing

>
> Shortcomings:
> -> extensions are not treated, only the main package
> -> Not used much, for sure needs testing to see if this really works.

—> there might be other places in addition global #compilerClass: where we need to point to the overlay version
For the complete “compile a method” it works, but I think there are other cases where it is not enough.

> -> Can we get rid of the #CompilerOverlay global?
>
I think it can be a class variable of OpalCompiler, this way we can turn the whole thing into a Setting (if it turns out to be usable)

Marcus

Marcus Denker-4

Re: [Compiler] #bindings: improvement: Global shadowing

In reply to this post by Marcus Denker-4

So what can we do with this? If you develop a compiler, it is always very annoying that any change that you do actually breaks the compiler if you do it wrong. Larger changes over multiple methods get quite hard to do.

One thing that we will soon have is the “accept for test” feature that I posted recently.
Another idea is to use the #bindings: feature of the compiler.

I did small experiment, I added this method to the OpalCompiler class on the class side:

…..

And after that, you can put a #halt deep in the compiler (or do some change that needs multiple steps), yet compilation will not be affected.
Most tests (not all) are writing in a way that they use classes from the package directly (they do not go via the global #compilerClass setting).

So this means that one can now change the compiler as much as one wants, test it by running tests and then be quite sure that it will not break anything.

Shortcomings:
-> extensions are not treated, only the main package
-> Not used much, for sure needs testing to see if this really works.
-> Can we get rid of the #CompilerOverlay global?

I made a pull request:

https://github.com/pharo-project/pharo/pull/1449

Marcus

Marcus Denker-4

Re: [Compiler] #bindings: improvement: Global shadowing

In reply to this post by Marcus Denker-4

On 28 May 2018, at 12:07, Marcus Denker <[hidden email]> wrote:

Shortcomings:
-> extensions are not treated, only the main package
-> Not used much, for sure needs testing to see if this really works.

—> there might be other places in addition global #compilerClass: where we need to point to the overlay version
For the complete “compile a method” it works, but I think there are other cases where it is not enough.

I think this one makes it really usable:

https://github.com/pharo-project/pharo/pull/1455

Now I can even put a halt in the semantic analysis which is called by syntax highlighting in the browser at each keystroke.

Marcus

Marcus Denker-4

Re: [Compiler] #bindings: improvement: Global shadowing

On 29 May 2018, at 11:41, Marcus Denker <[hidden email]> wrote:

On 28 May 2018, at 12:07, Marcus Denker <[hidden email]> wrote:

Shortcomings:
-> extensions are not treated, only the main package
-> Not used much, for sure needs testing to see if this really works.

—> there might be other places in addition global #compilerClass: where we need to point to the overlay version
For the complete “compile a method” it works, but I think there are other cases where it is not enough.

I think this one makes it really usable:

https://github.com/pharo-project/pharo/pull/1455

Now I can even put a halt in the semantic analysis which is called by syntax highlighting in the browser at each keystroke.

This is now merged with a small addition to provide a UI via the Settings:

If this is selected, the overlay is active.

Marcus