#parseMethod:onError: isn't as obvious as expected - is it me?

Hi Tim,

The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.

For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).

Regards,

Thierry

2017-09-01 8:50 GMT+02:00 Tim Mackinnon <[hidden email]>:

Marcus - I'd be interested in how you thought to solve this.

My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).

Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:"  to signal that it's going to reuse the results?

Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.

I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.

Tim

Sent from my iPhone

> On 31 Aug 2017, at 19:11, Marcus Denker <[hidden email]> wrote:
>
>
>> On 31 Aug 2017, at 19:07, Stephane Ducasse <[hidden email]> wrote:
>>
>> On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <[hidden email]> wrote:
>>> I’m looking for some feedback on the usage of  RBParser>>#parseMethod:onError:.
>>>
>>> I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
>>
>> Excellent!!!
>>
>>
>>> Anyway - step one (which I think I’ve come close on), was to teach senders/implementors to look at the AST so you don’t have to highlight exactly what you want to search on - instead it can infer from your cursor… however my next step was to improve how we can select code to ease refactoring, bracketing etc…
>>
>> Yes but we have to handle broken code then.
>> Did you check how smart suggestions did it for code that compiled?
>>
>
> I looked some weeks ago and made some notes what to do… but I fear this has to wait for next week,
> there is no way that I can do anything till ESUG…
>
>    Marcus

there is #parseFaultyMethod:, too. We use RBParser for syntax highlighting.

We do already have “senders” and “implementors working in the AST when using Suggestions.

See SugsMenuBuilder>>#findBestNodeFor: 

in the case of non-accepted code, it uses the compiler chain with #optionParseErrors (which uses parseFaultyMethod: to parse).

context hasUnacceptedEdits

ifTrue: [ 

root := Smalltalk compiler

source: context code;

options: #(+ optionParseErrors);

parse.

context selectedClass ifNotNil: [ :theClass | root methodNode methodClass: theClass ].

root doSemanticAnalysis ]

This works well but fails when you actually are in an error node.. for the same code the “dumb” way we use

with standard “senders of” just takes line, looks for spaces and feeds that word into the “senders of”..

So to make this work better we need to combine both approaches and deal better with the Error Node content.

(e.g. trying to fix it and re-parse or just use the dumb “get the work under the cursor” way as a first step).

Marcus

On 8 Sep 2017, at 03:43, Tim Mackinnon <[hidden email]> wrote:

Damn you smalltalk… you suck me in to little side problems….

RBParser is actually a lot more flexible than I had realised - and not so difficult to understand for a mere mortal…

It naively seems that (what I still consider a confusingly named method) is quite simple to extend to handle at least medium sized errors just by doing something like:

ast := [RBParser parseMethod: source onError: [ :msg :pos :parser | |seqNode errorNode|

errorNode := RBParseErrorNode

errorMessage: msg value: parser currentToken value printString at: pos. 

seqNode := parser sequenceNodeClass new.

parser 

step; 

parseStatements: true into: seqNode.

seqNode 

addNodeFirst: errorNode;

yourself

]] onDNU: #body: do: [ nil “Handle non method source fragments"].

NOTE: I did have to add >>#currentToken as a method to RBParser as it doesn’t expose very much for the onError blocks, which I think it should to make them a bit more useful.

Tim

On 7 Sep 2017, at 14:39, Tim Mackinnon <[hidden email]> wrote:

Thanks Thierry - I’ve been noodling in the background and next I hit the question of “comments” - which I found RB does handle… and then as per your point - how to skip past an error and try further down (which it seems that the scanner might have the info to let me skip simple things). You seem to get sucked in more and more when trying to to this properly - which does make it sound like a very good internship or GSoc project.

I think I may have enough to get something polished enough to release to the wider world. I certainly love having better keyboard support when I’m coding - as we type a lot (even though we have simple syntax) - and having efficient tools takes away the friction.

Next on my hit list, it add the same support to the refactoring tools - which while sophisticated are so clunky in use when compared to IntelliJ.

Tim

On 7 Sep 2017, at 08:02, Thierry Goubier <[hidden email]> wrote:

Hi Tim,

2017-09-01 13:39 GMT+02:00 Tim Mackinnon <[hidden email]>:

Thanks Thierry - this is proving an interesting problem domain ...

I too am using #bestNodeFor: although I don't find it always gives the best node (it does if you are clearly in the target range, but if you are on the outer boundary like the next position after a selector or next to a "." or ";" it favours the parent and not the closest node. So in usage I find I need to tweak it a bit.

Yes, when you are on a boundary, then finding the right node may be difficult. When developping the ancestor to smart suggestion, I considered #bestNodeFor: as good enough, but I considered looking into "up" and "down" ast navigation at a time, and I wasn't alone (I think it was active in the Pharo editor at a point).

 

I'll look at smacc though - also there is that experimental setting to allow parse errors, I don't know if it helps in any way.

There is a general question there, which is how you try to parse as much as possible while jumping over the error, making the error area as small as possible (that would be really great for syntax highlighting, by the way).

The problem is that in hand-written parsers, you need to hard-code the error management in the parser. With a table-driven parser (as is SmaCC), then you can explore in a systematic way what are the possible states that would allow the parsing to continue correctly after making the error area as small as possible.

This would make for a very nice internship subject...

All this said, I think I have a workable suggestion that is not much code that might be open to scrutiny from those wiser than me.

Looking at the keyboard handling in the system - it's quite sprawling and tricky to follow. I think there is lots of room for refactoring.

It is very easy in Pharo to implement a nice and clean keyboard handling; all the necessary components have been in place for years, and we already have implementations (using the KM API).

Now, most Pharo developpers just jump into hardcoding keyboard handling instead.

Thierry

 

I'm also not sure if I like the pragma usage either - I find it quite difficult to follow what's calling what.

Tim

Sent from my iPhone

On 1 Sep 2017, at 09:18, Thierry Goubier <[hidden email]> wrote:

Hi Tim,

The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.

For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).

Regards,

Thierry

2017-09-01 8:50 GMT+02:00 Tim Mackinnon <[hidden email]>:

Marcus - I'd be interested in how you thought to solve this.

My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).

Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:"  to signal that it's going to reuse the results?

Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.

I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.

Tim

Sent from my iPhone

> On 31 Aug 2017, at 19:11, Marcus Denker <[hidden email]> wrote:
>
>
>> On 31 Aug 2017, at 19:07, Stephane Ducasse <[hidden email]> wrote:
>>
>> On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <[hidden email]> wrote:
>>> I’m looking for some feedback on the usage of  RBParser>>#parseMethod:onError:.
>>>
>>> I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
>>
>> Excellent!!!
>>
>>
>>> Anyway - step one (which I think I’ve come close on), was to teach senders/implementors to look at the AST so you don’t have to highlight exactly what you want to search on - instead it can infer from your cursor… however my next step was to improve how we can select code to ease refactoring, bracketing etc…
>>
>> Yes but we have to handle broken code then.
>> Did you check how smart suggestions did it for code that compiled?
>>
>
> I looked some weeks ago and made some notes what to do… but I fear this has to wait for next week,
> there is no way that I can do anything till ESUG…
>
>    Marcus

Thanks for the pointer Marcus, I'll have a look - it's a fascinating area that I didn't know much about until now ;)

All I wanted was to not have to select specific code to do senders (& refactoring), as well as have better statement selection so I can more easily put brackets around things...

But now it dawns on me I'd like better code completion after I've already completed but maybe want the fuller keyword method.... 

Even the refactoring of local method changes shouldn't need you to save your method to rename a local variable or extract a statement to a variable. And I hate having to declare local variables painfully one by one - and I miss the option to detect a simple typo and correct it.

These are all things that I love in IntelliJ - and I think we have fantastic infrastructure to do these things too. So I'll keep plugging away at lunch times to see what I can do.

I've rewritten it about 3 times already ;)

Tim

Sent from my iPhone

On 9 Sep 2017, at 08:42, Marcus Denker <[hidden email]> wrote:

there is #parseFaultyMethod:, too. We use RBParser for syntax highlighting.

We do already have “senders” and “implementors working in the AST when using Suggestions.

See SugsMenuBuilder>>#findBestNodeFor: 

in the case of non-accepted code, it uses the compiler chain with #optionParseErrors (which uses parseFaultyMethod: to parse).

context hasUnacceptedEdits

ifTrue: [ 

root := Smalltalk compiler

source: context code;

options: #(+ optionParseErrors);

parse.

context selectedClass ifNotNil: [ :theClass | root methodNode methodClass: theClass ].

root doSemanticAnalysis ]

This works well but fails when you actually are in an error node.. for the same code the “dumb” way we use

with standard “senders of” just takes line, looks for spaces and feeds that word into the “senders of”..

So to make this work better we need to combine both approaches and deal better with the Error Node content.

(e.g. trying to fix it and re-parse or just use the dumb “get the work under the cursor” way as a first step).

Marcus

On 8 Sep 2017, at 03:43, Tim Mackinnon <[hidden email]> wrote:

Damn you smalltalk… you suck me in to little side problems….

RBParser is actually a lot more flexible than I had realised - and not so difficult to understand for a mere mortal…

It naively seems that (what I still consider a confusingly named method) is quite simple to extend to handle at least medium sized errors just by doing something like:

ast := [RBParser parseMethod: source onError: [ :msg :pos :parser | |seqNode errorNode|

errorNode := RBParseErrorNode

errorMessage: msg value: parser currentToken value printString at: pos. 

seqNode := parser sequenceNodeClass new.

parser 

step; 

parseStatements: true into: seqNode.

seqNode 

addNodeFirst: errorNode;

yourself

]] onDNU: #body: do: [ nil “Handle non method source fragments"].

NOTE: I did have to add >>#currentToken as a method to RBParser as it doesn’t expose very much for the onError blocks, which I think it should to make them a bit more useful.

Tim

On 7 Sep 2017, at 14:39, Tim Mackinnon <[hidden email]> wrote:

Thanks Thierry - I’ve been noodling in the background and next I hit the question of “comments” - which I found RB does handle… and then as per your point - how to skip past an error and try further down (which it seems that the scanner might have the info to let me skip simple things). You seem to get sucked in more and more when trying to to this properly - which does make it sound like a very good internship or GSoc project.

I think I may have enough to get something polished enough to release to the wider world. I certainly love having better keyboard support when I’m coding - as we type a lot (even though we have simple syntax) - and having efficient tools takes away the friction.

Next on my hit list, it add the same support to the refactoring tools - which while sophisticated are so clunky in use when compared to IntelliJ.

Tim

On 7 Sep 2017, at 08:02, Thierry Goubier <[hidden email]> wrote:

Hi Tim,

2017-09-01 13:39 GMT+02:00 Tim Mackinnon <[hidden email]>:

Thanks Thierry - this is proving an interesting problem domain ...

I too am using #bestNodeFor: although I don't find it always gives the best node (it does if you are clearly in the target range, but if you are on the outer boundary like the next position after a selector or next to a "." or ";" it favours the parent and not the closest node. So in usage I find I need to tweak it a bit.

Yes, when you are on a boundary, then finding the right node may be difficult. When developping the ancestor to smart suggestion, I considered #bestNodeFor: as good enough, but I considered looking into "up" and "down" ast navigation at a time, and I wasn't alone (I think it was active in the Pharo editor at a point).

 

I'll look at smacc though - also there is that experimental setting to allow parse errors, I don't know if it helps in any way.

There is a general question there, which is how you try to parse as much as possible while jumping over the error, making the error area as small as possible (that would be really great for syntax highlighting, by the way).

The problem is that in hand-written parsers, you need to hard-code the error management in the parser. With a table-driven parser (as is SmaCC), then you can explore in a systematic way what are the possible states that would allow the parsing to continue correctly after making the error area as small as possible.

This would make for a very nice internship subject...

All this said, I think I have a workable suggestion that is not much code that might be open to scrutiny from those wiser than me.

Looking at the keyboard handling in the system - it's quite sprawling and tricky to follow. I think there is lots of room for refactoring.

It is very easy in Pharo to implement a nice and clean keyboard handling; all the necessary components have been in place for years, and we already have implementations (using the KM API).

Now, most Pharo developpers just jump into hardcoding keyboard handling instead.

Thierry

 

I'm also not sure if I like the pragma usage either - I find it quite difficult to follow what's calling what.

Tim

Sent from my iPhone

On 1 Sep 2017, at 09:18, Thierry Goubier <[hidden email]> wrote:

Hi Tim,

The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.

For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).

Regards,

Thierry

2017-09-01 8:50 GMT+02:00 Tim Mackinnon <[hidden email]>:

Marcus - I'd be interested in how you thought to solve this.

My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).

Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:"  to signal that it's going to reuse the results?

Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.

I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.

Tim

Sent from my iPhone

> On 31 Aug 2017, at 19:11, Marcus Denker <[hidden email]> wrote:
>
>
>> On 31 Aug 2017, at 19:07, Stephane Ducasse <[hidden email]> wrote:
>>
>> On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <[hidden email]> wrote:
>>> I’m looking for some feedback on the usage of  RBParser>>#parseMethod:onError:.
>>>
>>> I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
>>
>> Excellent!!!
>>
>>
>>> Anyway - step one (which I think I’ve come close on), was to teach senders/implementors to look at the AST so you don’t have to highlight exactly what you want to search on - instead it can infer from your cursor… however my next step was to improve how we can select code to ease refactoring, bracketing etc…
>>
>> Yes but we have to handle broken code then.
>> Did you check how smart suggestions did it for code that compiled?
>>
>
> I looked some weeks ago and made some notes what to do… but I fear this has to wait for next week,
> there is no way that I can do anything till ESUG…
>
>    Marcus