Smalltalk › Pharo › Pharo Smalltalk Developers

Code generation for ASTs with syntax errors

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

21 messages Options

Marcus Denker-4

Code generation for ASTs with syntax errors

We can parse code with syntax errors: it generates partial ASTs with the faulty part of the input as a RBParseErrorNode.

Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification
is raised at runtime instead of compile time.

testEvalSimpleMethodWithError
| ast cm |
ast := OpalCompiler new
source: 'method 3+';
useFaultyForParsing: true;
parse.

self assert: ast isMethod.
self assert: ast isFaulty.

cm := ast compiledMethod.
self should: [cm valueWithReceiver: nil arguments: #()] raise: SyntaxErrorNotification

The syntax error instance is compiled in via a literal variable, in the end the only thing needed was:

visitParseErrorNode: anErrorNode
methodBuilder
pushLiteralVariable: #error -> anErrorNode asSyntaxErrorNotification;
send: #signal.

This is in #50044.

Future work: introduce a RuntimeSyntaxError that can be turned off per error and globally.

Marcus

Damien Pollet

Re: Code generation for ASTs with syntax errors

On 13 May 2015 at 08:35, Marcus Denker <[hidden email]> wrote:

Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification
is raised at runtime instead of compile time.

I can hear the noise of eyebrows raising in the compiler community :)

Syntax errors shouldn't require immediate fixing during compilation per se, but there should be a way to deal with them "at compile time" —whatever that means for us. For instance, faulty methods could have a flashy icon in the browser; I think I'd even color the whole package-class-protocol-selector in red if either of them contain faulty methods.

Damien Pollet
type less, do more [ | ] http://people.untyped.org/damien.pollet

Denis Kudriashov

Re: Code generation for ASTs with syntax errors

In reply to this post by Marcus Denker-4

Thanks's Marcus. It is beautiful thing.

Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one

Best regards,

Denis

2015-05-13 9:35 GMT+03:00 Marcus Denker <[hidden email]>:

We can parse code with syntax errors: it generates partial ASTs with the faulty part of the input as a RBParseErrorNode.

Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification
is raised at runtime instead of compile time.

testEvalSimpleMethodWithError
| ast cm |
ast := OpalCompiler new
source: 'method 3+';
useFaultyForParsing: true;
parse.

self assert: ast isMethod.
self assert: ast isFaulty.

cm := ast compiledMethod.
self should: [cm valueWithReceiver: nil arguments: #()] raise: SyntaxErrorNotification

The syntax error instance is compiled in via a literal variable, in the end the only thing needed was:

visitParseErrorNode: anErrorNode
methodBuilder
pushLiteralVariable: #error -> anErrorNode asSyntaxErrorNotification;
send: #signal.

This is in #50044.

Future work: introduce a RuntimeSyntaxError that can be turned off per error and globally.

Marcus

Marcus Denker-4

Re: Code generation for ASTs with syntax errors

In reply to this post by Damien Pollet

On 13 May 2015, at 10:41, Damien Pollet <[hidden email]> wrote:

On 13 May 2015 at 08:35, Marcus Denker <[hidden email]> wrote:
Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification
is raised at runtime instead of compile time.

I can hear the noise of eyebrows raising in the compiler community :)

Yes! :-)

Syntax errors shouldn't require immediate fixing during compilation per se, but there should be a way to deal with them "at compile time" —whatever that means for us. For instance, faulty methods could have a flashy icon in the browser; I think I'd even color the whole package-class-protocol-selector in red if either of them contain faulty methods.

I think that GT does it already right: it does not insert the error message in the text, but instead shows it using a tiny window undefendend of the text.

Combined with syntax highlighting, this should be very close to what we need (the devil is in the detail, of course).

As for finding them: the nice property of syntax errors is that they are trivial to detect statically… so e.g. we can add that people never can commit code like that, and

of course code critics can find it trivially… the rule just needs to check for #isFaulty (maybe we should range that to #hasSyntaxErrors).

Marcus

Ben Coman

Re: Code generation for ASTs with syntax errors

In reply to this post by Damien Pollet

On Wed, May 13, 2015 at 4:41 PM, Damien Pollet <[hidden email]> wrote:

On 13 May 2015 at 08:35, Marcus Denker <[hidden email]> wrote:
Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification
is raised at runtime instead of compile time.

What is the advantage of allowing syntax errors into the running system, when they can be trapped statically before that ? It seems to me the advantage of this change is that it allows compilation to complete rather than bombing out part way through. But just because compilation finishes, doesn't mean it needs to be committed into the runtime system.

Syntax errors shouldn't require immediate fixing during compilation per se, but there should be a way to deal with them "at compile time" —whatever that means for us.

For instance, faulty methods could have a flashy icon in the browser; I think I'd even color the whole package-class-protocol-selector in red if either of them contain faulty methods.

I guess it *should* mean that the faulty-compiled-method is not committed but stored in a temporary buffer similar to how editing source is effectively done in a temporary buffer. Indication of the fault would only be needed in the window that compiled the fault.

cheers -ben

Marcus Denker-4

Re: Code generation for ASTs with syntax errors

On 19 May 2015, at 01:51, Ben Coman <[hidden email]> wrote:

On Wed, May 13, 2015 at 4:41 PM, Damien Pollet <[hidden email]> wrote:
On 13 May 2015 at 08:35, Marcus Denker <[hidden email]> wrote:
Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification
is raised at runtime instead of compile time.

What is the advantage of allowing syntax errors into the running system, when they can be trapped statically before that ? It seems to me the advantage of this change is that it allows compilation to complete rather than bombing out part way through. But just because compilation finishes, doesn't mean it needs to be committed into the runtime system.
Syntax errors shouldn't require immediate fixing during compilation per se, but there should be a way to deal with them "at compile time" —whatever that means for us.
For instance, faulty methods could have a flashy icon in the browser; I think I'd even color the whole package-class-protocol-selector in red if either of them contain faulty methods.

I guess it *should* mean that the faulty-compiled-method is not committed but stored in a temporary buffer similar to how editing source is effectively done in a temporary buffer. Indication of the fault would only be needed in the window that compiled the fault.

No, we once experimented with adding support to Nautilus for that. But the result is that it is very easy to loose code. And you never know what is “really saved” and what not.

But in general: I know that this is one of those things that I will not be able to convince anyone that it is useful before it is done… which is good: I like to explore directions that everyone else rejects immediately.

Marcus

Marcus Denker-4

Re: Code generation for ASTs with syntax errors

In reply to this post by Denis Kudriashov

On 13 May 2015, at 11:21, Denis Kudriashov <[hidden email]> wrote:

Thanks's Marcus. It is beautiful thing.

Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one

You can just look at

OpalCompiler new

source: 'method 3+. 1+2';

useFaultyForParsing: true;

parse.

with the GT inspector, it has a very nice AST view:

Thierry Goubier

Re: Code generation for ASTs with syntax errors

2015-05-19 7:49 GMT+02:00 Marcus Denker <[hidden email]>:

On 13 May 2015, at 11:21, Denis Kudriashov <[hidden email]> wrote:

Thanks's Marcus. It is beautiful thing.

Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one

You can just look at

OpalCompiler new
source: 'method 3+. 1+2';
useFaultyForParsing: true;
parse.

Impressive !

with the GT inspector, it has a very nice AST view:

The AST view of the GT Inspector is useless for serious work.

I've been spending the past weeks and months looking at ASTs, SSA and CFGs of code and I allways need a twin view: node + source code (and selecting one of those node select the relevant portion of the code).

Thierry

Jan Vrany

Re: Code generation for ASTs with syntax errors

>
> with the GT inspector, it has a very nice AST view:
>
>
> The AST view of the GT Inspector is useless for serious work.
>
>
> I've been spending the past weeks and months looking at ASTs, SSA and
> CFGs of code and I allways need a twin view: node + source code (and
> selecting one of those node select the relevant portion of the code).

Agree, something like this would be even nicer...

Jan

stx_screenshot_011.png (346K) Download Attachment

Andrei Chis

Re: Code generation for ASTs with syntax errors

In reply to this post by Thierry Goubier

On Tue, May 19, 2015 at 10:02 AM, Thierry Goubier <[hidden email]> wrote:

2015-05-19 7:49 GMT+02:00 Marcus Denker <[hidden email]>:

On 13 May 2015, at 11:21, Denis Kudriashov <[hidden email]> wrote:

Thanks's Marcus. It is beautiful thing.

Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one

You can just look at

OpalCompiler new
source: 'method 3+. 1+2';
useFaultyForParsing: true;
parse.

Impressive !

with the GT inspector, it has a very nice AST view:

The AST view of the GT Inspector is useless for serious work.

I've been spending the past weeks and months looking at ASTs, SSA and CFGs of code and I allways need a twin view: node + source code (and selecting one of those node select the relevant portion of the code).

Switching to the 'Source code' view in the second pane gives you exactly that

Thierry

Tudor Girba-2

Re: Code generation for ASTs with syntax errors

In reply to this post by Jan Vrany

It's already available :).

Doru

On Tue, May 19, 2015 at 10:18 AM, Jan Vrany <[hidden email]> wrote:

>
> with the GT inspector, it has a very nice AST view:
>
>
> The AST view of the GT Inspector is useless for serious work.
>
>
> I've been spending the past weeks and months looking at ASTs, SSA and
> CFGs of code and I allways need a twin view: node + source code (and
> selecting one of those node select the relevant portion of the code).

Agree, something like this would be even nicer...

Jan

www.tudorgirba.com

"Every thing has its own flow"

Thierry Goubier

Re: Code generation for ASTs with syntax errors

In reply to this post by Andrei Chis

2015-05-19 10:16 GMT+02:00 Andrei Chis <[hidden email]>:

On Tue, May 19, 2015 at 10:02 AM, Thierry Goubier <[hidden email]> wrote:

2015-05-19 7:49 GMT+02:00 Marcus Denker <[hidden email]>:

On 13 May 2015, at 11:21, Denis Kudriashov <[hidden email]> wrote:

Thanks's Marcus. It is beautiful thing.

Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one

You can just look at

OpalCompiler new
source: 'method 3+. 1+2';
useFaultyForParsing: true;
parse.

Impressive !

with the GT inspector, it has a very nice AST view:

The AST view of the GT Inspector is useless for serious work.

I've been spending the past weeks and months looking at ASTs, SSA and CFGs of code and I allways need a twin view: node + source code (and selecting one of those node select the relevant portion of the code).

Switching to the 'Source code' view in the second pane gives you exactly that

And that view get the selection wrong ;)

Thierry

Thierry Goubier

Re: Code generation for ASTs with syntax errors

In reply to this post by Tudor Girba-2

Ok.

Now I need to evaluate if the time spent writing a GT presentation is worth the performance impact of using the GT Inspector use compared to writing a dedicated version of a faster tree viewer ;)

Thierry

2015-05-19 10:18 GMT+02:00 Tudor Girba <[hidden email]>:

It's already available :).

Doru

On Tue, May 19, 2015 at 10:18 AM, Jan Vrany <[hidden email]> wrote:

>
> with the GT inspector, it has a very nice AST view:
>
>
> The AST view of the GT Inspector is useless for serious work.
>
>
> I've been spending the past weeks and months looking at ASTs, SSA and
> CFGs of code and I allways need a twin view: node + source code (and
> selecting one of those node select the relevant portion of the code).

Agree, something like this would be even nicer...

Jan

--
www.tudorgirba.com

"Every thing has its own flow"

Marcus Denker-4

Re: Code generation for ASTs with syntax errors

In reply to this post by Thierry Goubier

And that view get the selection wrong ;)

No, the ErrorNode has “all the rest” as it’s extend for Shout so it can color the error just the same as people are used to.

Marcus

Thierry Goubier

Re: Code generation for ASTs with syntax errors

2015-05-19 10:27 GMT+02:00 Marcus Denker <[hidden email]>:

And that view get the selection wrong ;)

No, the ErrorNode has “all the rest” as it’s extend for Shout so it can color the error just the same as people are used to.

Ok.

But I'd really like Shout not to color red the rest of the code when I start to modify code half way through a method :)

Is that inherent to the way ErrorNode instances are generated or is it just done to maintain as it was?

Thierry

Marcus

Marcus Denker-4

Re: Code generation for ASTs with syntax errors

On 19 May 2015, at 10:32, Thierry Goubier <[hidden email]> wrote:

2015-05-19 10:27 GMT+02:00 Marcus Denker <[hidden email]>:

And that view get the selection wrong ;)

No, the ErrorNode has “all the rest” as it’s extend for Shout so it can color the error just the same as people are used to.

Ok.

But I'd really like Shout not to color red the rest of the code when I start to modify code half way through a method :)

Is that inherent to the way ErrorNode instances are generated or is it just done to maintain as it was?

I am not entirely sure… it sets the code of the ErrorNode to be just from where we are to the end of the to be parsed text.

At that point it is hard to know the real extend of the error… needs to be checked how to do it nicer.

I think we should first get it in a state that we can replace the shout parser (both for syntax highlight and code completion).

Marcus

Damien Pollet

Re: Code generation for ASTs with syntax errors

In reply to this post by Marcus Denker-4

On 18 May 2015 at 21:33, Marcus Denker <[hidden email]> wrote:

As for finding them: the nice property of syntax errors is that they are trivial to detect statically… so e.g. we can add that people never can commit code like that

NOPE.

The problem I had was that NBExternalStructure was changed from a variable bytes class to a normal one. So subclasses were committed without error but still couldn't be loaded with a more recent NB. Errors WILL happen, and fixing those is where plain text beats objects…

Damien Pollet
type less, do more [ | ] http://people.untyped.org/damien.pollet

Thierry Goubier

Re: Code generation for ASTs with syntax errors

In reply to this post by Marcus Denker-4

2015-05-19 11:22 GMT+02:00 Marcus Denker <[hidden email]>:

On 19 May 2015, at 10:32, Thierry Goubier <[hidden email]> wrote:

2015-05-19 10:27 GMT+02:00 Marcus Denker <[hidden email]>:

And that view get the selection wrong ;)

No, the ErrorNode has “all the rest” as it’s extend for Shout so it can color the error just the same as people are used to.

Ok.

But I'd really like Shout not to color red the rest of the code when I start to modify code half way through a method :)

Is that inherent to the way ErrorNode instances are generated or is it just done to maintain as it was?

I am not entirely sure… it sets the code of the ErrorNode to be just from where we are to the end of the to be parsed text.
At that point it is hard to know the real extend of the error… needs to be checked how to do it nicer.

In the example you give, the parser is able to trigger an error, go up the ast tree, and restart correct parsing further down the source code, no? Is that the way you are doing it?

I'm interested in a general parsing theory for that kind of error detection / recovery.

Thierry

I think we should first get it in a state that we can replace the shout parser (both for syntax highlight and code completion).

Marcus

Denis Kudriashov

Re: Code generation for ASTs with syntax errors

In reply to this post by Marcus Denker-4

Perfect!

19 мая 2015 г. 7:49 пользователь "Marcus Denker" <[hidden email]> написал:

On 13 May 2015, at 11:21, Denis Kudriashov <[hidden email]> wrote:

Thanks's Marcus. It is beautiful thing.

Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one

You can just look at

OpalCompiler new
source: 'method 3+. 1+2';
useFaultyForParsing: true;
parse.

with the GT inspector, it has a very nice AST view:

Denis Kudriashov

Re: Code generation for ASTs with syntax errors

In reply to this post by Marcus Denker-4

But statement after error is parsed and ast node exists for them.
Is it not correct to make end of error equal to start of next ast node?

19 мая 2015 г. 11:22 пользователь "Marcus Denker" <[hidden email]> написал:

On 19 May 2015, at 10:32, Thierry Goubier <[hidden email]> wrote:

2015-05-19 10:27 GMT+02:00 Marcus Denker <[hidden email]>:

And that view get the selection wrong ;)

No, the ErrorNode has “all the rest” as it’s extend for Shout so it can color the error just the same as people are used to.

Ok.

But I'd really like Shout not to color red the rest of the code when I start to modify code half way through a method :)

Is that inherent to the way ErrorNode instances are generated or is it just done to maintain as it was?

I am not entirely sure… it sets the code of the ErrorNode to be just from where we are to the end of the to be parsed text.
At that point it is hard to know the real extend of the error… needs to be checked how to do it nicer.

I think we should first get it in a state that we can replace the shout parser (both for syntax highlight and code completion).

Marcus