Binary selector and special characters

See RBParserTest>>#testBinarySelectors

It's based on the draft ANSI Smalltalk-80 standard. You integrated it. It tests the RBParser's parsing of binary method definitions and message sends of all binary selectors from 1 char upto 3 chars. (The Blue Book is more restrictive than ANSI, limiting them to 2 chars max IIRC.)

I wrote the test because of issues I had with the OldCompiler's handling of selectors containing "|" and issues on other platforms like GemStone, so the behavior I need and think is correct won't get broken without warning.

Hi Stef,

2016-08-29 8:35 GMT+02:00 stepharo <[hidden email]>:

This is a really nice and important question.
I would really have a clear answer because it will make the system more stable.

If you can build an analysis and let us know it would be really great.


Something related but not on the same topic is that I would love to have a syntax for nested comments.

This is really annoying to have to uncomment parts when we have to comment a large part. We discussed this back in 2007-2009 but we never did it.

If your need is that (uncomment while commenting and the reverse), then the answer is not a syntax change, but a better comment/uncomment command in the editor.

Now that you say that, I also have the issue. I'll have a try in AltBrowser if I can get the behavior you'd like.

Thierry

 

Stef


Le 28/8/16 à 12:17, Nicolai Hess a écrit :

Hi,

where can I find a good reference about what characters are allowed as
binary selectors (from old syntax definition) and what is nowadays allowed
by the implementations.

And whether the current set of allowed binaries selector includes some additions on
purpose or if this is just a bug of the parser.

From what I found out,  (Blue book and some other smalltalk syntax definitions)
the current set of allowed characters includes the "special characters":
$! $% $& $* $+ $, $- $/ $< $= $> $? $@ $\ $| $~
(some implementation do not allow $@ and some calls $- not a special character
but allowed as binary selector character)

And this is what String>>#numArgs uses. Therefore

'-' numArgs "->1".
'!' numArgs "->1".
And for example:
'§' numArgs "-> -1 (the -1 is indicating "not even a valid selector")"

But I am interested in the characters not called "special characters and
not even in the range 0-126.

The scanner allowes much more characters to be used as a selector name
(From the scanners typeTable) :

{Character value: 1 . Character value: 2 . Character value: 3 . Character value: 4 . Character value: 5 . Character value: 6 . Character value: 7 . Character backspace . Character value: 11 . Character value: 14 . Character value: 15 . Character value: 16 . Character value: 17 . Character value: 18 . Character value: 19 . Character value: 20 . Character value: 21 . Character value: 22 . Character value: 23 . Character value: 24 . Character value: 25 . Character value: 26 . Character escape . Character value: 28 . Character value: 29 . Character value: 30 . Character value: 31 . $! . $% . $& . $* . $+ . $, . $- . $/ . $< . $= . $> . $? . $@ . $\ . $` . $~ . Character delete . $€ . $ . $‚ . $ƒ . $„ . $… . $† . $‡ . $ˆ . $‰ . $Š . $‹ . $Œ . $ . $Ž . $ . $ . $‘ . $’ . $“ . $” . $• . $– . $— . $˜ . $™ . $š . $› . $œ . $ . $ž . $Ÿ . $  . $¡ . $¢ . $£ . $¤ . $¥ . $¦ . $§ . $¨ . $© . $« . $¬ . $­ . $® . $¯ . $° . $± . $² . $³ . $´ . $¶ . $· . $¸ . $¹ . $» . $¼ . $½ . $¾ . $¿ . $× . $÷}

This means you can define a method with for example the name "÷".

So , the question I want to ask. What do we want to allow as a binary selector (character).
All that is nowadays "parseable" as binary selector, or only the set of "special characters"
or something between both, and where to put this information, the "this is an allowed binary
selector character" information?

Thanks
Nicolai

Thierry

If you have a better editor control even better :)

Syntax wise, one could consider "" to be inside a comment (i.e. do not split into two comments if encountered inside a comment, as it is done now).

This one could be nice too :)

Thierry

If you have a better editor control even better :)

Syntax wise, one could consider "" to be inside a comment (i.e. do not split into two comments if encountered inside a comment, as it is done now).

This one could be nice too :)

On Mon, Aug 29, 2016 at 11:42 AM, stepharo <[hidden email]> wrote:

Thierry

If you have a better editor control even better :)

Syntax wise, one could consider "" to be inside a comment (i.e. do not split into two comments if encountered inside a comment, as it is done now).

This one could be nice too :)

This would be also consistent with how single quotes are handled in a string, which would make it doublenice. :)

Peter

Le 29/08/2016 à 21:28, stepharo a écrit :

Le 29/8/16 à 17:45, Thierry Goubier a écrit :

Hi Stef,

2016-08-29 11:42 GMT+02:00 stepharo <[hidden email]
<mailto:[hidden email]>>:

    Thierry

    If you have a better editor control even better :)

    Syntax wise, one could consider "" to be inside a comment (i.e.
    do not split into two comments if encountered inside a comment,
    as it is done now).

    This one could be nice too :)


 https://pharo.fogbugz.com/f/cases/19011/Integrate-two-double-quotes-inside-comments

I'll have the slice ready soon. Any comments on what that would mean
regarding the Smalltalk commonly accepted syntax if that feature is
integrated?

It will break compatibility for people using it now we should raise the
topic and lets a chance to people to discuss about it. We could check
before publishing if code contain nested comments.

Hum. The slice should parse anything legal Smalltalk; just that it may show less comments intervals (because in fact it will coalesce adjacent comments).

For example, standard parse will say that:

'"this ""test"' is a token with two comments, intervals 1 to: 7 and 8 to: 13.

The slice makes that a single comment:

'"this ""test"' is a token with one comment, interval 1 to: 13.

Now, this has probably no impact on parsing smalltalk code. But it changes a bit the language definition, so that's why I'd like comments on it.

I think that I would use them only when developing.

Up to you :)

The most interesting is to have the correct comment/uncomment behavior in an editor... that one works independently and is quite cool.

Thierry

Stef

Thierry

2016-08-29 21:38 GMT+02:00 Thierry Goubier <[hidden email]>:

Le 29/08/2016 à 21:28, stepharo a écrit :

Le 29/8/16 à 17:45, Thierry Goubier a écrit :

Hi Stef,

2016-08-29 11:42 GMT+02:00 stepharo <[hidden email]
<mailto:[hidden email]>>:

    Thierry

    If you have a better editor control even better :)

    Syntax wise, one could consider "" to be inside a comment (i.e.
    do not split into two comments if encountered inside a comment,
    as it is done now).

    This one could be nice too :)


 https://pharo.fogbugz.com/f/cases/19011/Integrate-two-double-quotes-inside-comments

I'll have the slice ready soon. Any comments on what that would mean
regarding the Smalltalk commonly accepted syntax if that feature is
integrated?

It will break compatibility for people using it now we should raise the
topic and lets a chance to people to discuss about it. We could check
before publishing if code contain nested comments.

Hum. The slice should parse anything legal Smalltalk; just that it may show less comments intervals (because in fact it will coalesce adjacent comments).

Yes, I think the change for RBScanner is fine, it does not changes what kind of comments are accepted, only how they are assigned to

the AST nodes (one vs. multiple comments).

(BTW. do we have a function that would do the coalescing of intervals:

(1 to:99) (100 to: 199) -> (1 to:199)

? )

 

For example, standard parse will say that:

'"this ""test"' is a token with two comments, intervals 1 to: 7 and 8 to: 13.

The slice makes that a single comment:

'"this ""test"' is a token with one comment, interval 1 to: 13.

Now, this has probably no impact on parsing smalltalk code. But it changes a bit the language definition, so that's why I'd like comments on it.

I think that I would use them only when developing.

Up to you :)

The most interesting is to have the correct comment/uncomment behavior in an editor... that one works independently and is quite cool.

Thierry

Stef

Thierry

2016-08-29 21:38 GMT+02:00 Thierry Goubier <[hidden email]>:

Le 29/08/2016 à 21:28, stepharo a écrit :

Le 29/8/16 à 17:45, Thierry Goubier a écrit :

Hi Stef,

2016-08-29 11:42 GMT+02:00 stepharo <[hidden email]
<mailto:[hidden email]>>:

    Thierry

    If you have a better editor control even better :)

    Syntax wise, one could consider "" to be inside a comment (i.e.
    do not split into two comments if encountered inside a comment,
    as it is done now).

    This one could be nice too :)


 https://pharo.fogbugz.com/f/cases/19011/Integrate-two-double-quotes-inside-comments

I'll have the slice ready soon. Any comments on what that would mean
regarding the Smalltalk commonly accepted syntax if that feature is
integrated?

It will break compatibility for people using it now we should raise the
topic and lets a chance to people to discuss about it. We could check
before publishing if code contain nested comments.

Hum. The slice should parse anything legal Smalltalk; just that it may show less comments intervals (because in fact it will coalesce adjacent comments).

Yes, I think the change for RBScanner is fine, it does not changes what kind of comments are accepted, only how they are assigned to

the AST nodes (one vs. multiple comments).

(BTW. do we have a function that would do the coalescing of intervals:

(1 to:99) (100 to: 199) -> (1 to:199)

? )

For example, standard parse will say that:

'"this ""test"' is a token with two comments, intervals 1 to: 7 and 8 to: 13.

The slice makes that a single comment:

'"this ""test"' is a token with one comment, interval 1 to: 13.

Now, this has probably no impact on parsing smalltalk code. But it changes a bit the language definition, so that's why I'd like comments on it.

I think that I would use them only when developing.

Up to you :)

The most interesting is to have the correct comment/uncomment behavior in an editor... that one works independently and is quite cool.

Thierry

Stef

Thierry

Oops.  No need to add a step method; the increment method already exists:

 

On Wed, Aug 31, 2016 at 9:12 AM, Eliot Miranda <[hidden email]> wrote:

On Tue, Aug 30, 2016 at 8:14 AM, Nicolai Hess <[hidden email]> wrote:

2016-08-29 21:38 GMT+02:00 Thierry Goubier <[hidden email]>:

Le 29/08/2016 à 21:28, stepharo a écrit :

Le 29/8/16 à 17:45, Thierry Goubier a écrit :

Hi Stef,

2016-08-29 11:42 GMT+02:00 stepharo <[hidden email]
<mailto:[hidden email]>>:

    Thierry

    If you have a better editor control even better :)

    Syntax wise, one could consider "" to be inside a comment (i.e.
    do not split into two comments if encountered inside a comment,
    as it is done now).

    This one could be nice too :)


 https://pharo.fogbugz.com/f/cases/19011/Integrate-two-double-quotes-inside-comments

I'll have the slice ready soon. Any comments on what that would mean
regarding the Smalltalk commonly accepted syntax if that feature is
integrated?

It will break compatibility for people using it now we should raise the
topic and lets a chance to people to discuss about it. We could check
before publishing if code contain nested comments.

Hum. The slice should parse anything legal Smalltalk; just that it may show less comments intervals (because in fact it will coalesce adjacent comments).

Yes, I think the change for RBScanner is fine, it does not changes what kind of comments are accepted, only how they are assigned to

the AST nodes (one vs. multiple comments).

(BTW. do we have a function that would do the coalescing of intervals:

(1 to:99) (100 to: 199) -> (1 to:199)

? )

Find attached something that works in Squeak 5

For example, standard parse will say that:

'"this ""test"' is a token with two comments, intervals 1 to: 7 and 8 to: 13.

The slice makes that a single comment:

'"this ""test"' is a token with one comment, interval 1 to: 13.

Now, this has probably no impact on parsing smalltalk code. But it changes a bit the language definition, so that's why I'd like comments on it.

I think that I would use them only when developing.

Up to you :)

The most interesting is to have the correct comment/uncomment behavior in an editor... that one works independently and is quite cool.

Thierry

Stef

Thierry

 

_,,,^..^,,,_

best, Eliot

--

_,,,^..^,,,_

best, Eliot

2016-08-28 13:41 GMT+02:00 monty <[hidden email]>:

See RBParserTest>>#testBinarySelectors

It's based on the draft ANSI Smalltalk-80 standard. You integrated it. It tests the RBParser's parsing of binary method definitions and message sends of all binary selectors from 1 char upto 3 chars. (The Blue Book is more restrictive than ANSI, limiting them to 2 chars max IIRC.)

I wrote the test because of issues I had with the OldCompiler's handling of selectors containing "|" and issues on other platforms like GemStone, so the behavior I need and think is correct won't get broken without warning.

Hi Monty,

yes, but I am just wondering why the scanner interprets some characters as binary selector token, whereas they are not allowed as binary selectors.

In the old scanner, the initialization of the type table just sets " binary token" as the default for all characters and changes some of them explicit to for example

 ($0 asciiValue to: $9 asciiValue) -> digit tokens

 #(9 10 12 13 32 )  -> delimiter  token
...

But RBScanner on the other hand explicitly sets some non-ascii characters  to be used as "binary tokens"

    classificationTable at: 177 put: #binary.    "plus-or-minus"
    classificationTable at: 183 put: #binary.    "centered dot"
    classificationTable at: 215 put: #binary.    "times"
    classificationTable at: 247 put: #binary.    "divide"

It looks like someone ( or somewhere )  it should be allowed to use these characters as a binary selector
 "#($± $· $× $÷)"

Although later at the parsing step, using this tokens for binary message selectors isn't allowed.

I think I will exclude these characters as binary selector tokens.