Issue 4650 in pharo: Strange $ evaluates to Character arrowUp

Status: New
Owner: ----

New issue 4650 by [hidden email]: Strange $ evaluates to Character  
arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

a
        ^ $

is parsed wrongly and return Character arrowUp when evaluated. The same  
happens when evaluating a single $ in the Workspace.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: FixReviewNeeded

Comment #1 on issue 4650 by [hidden email]: Strange $ evaluates to  
Character arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

fixed in slice

  SLICE-Issue-4650-dollar-compiler-fix-CamilloBruni.1


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Labels: Milestone-1.4

Comment #2 on issue 4650 by [hidden email]: Strange $ evaluates to  
Character arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

(No comment was entered for this change.)


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: FixToInclude
        Labels: Type-Bug

Comment #3 on issue 4650 by [hidden email]: Strange $ evaluates to  
Character arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

ok, looks good


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: Integrated

Comment #4 on issue 4650 by [hidden email]: Strange $ evaluates to  
Character arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

in 14322


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #5 on issue 4650 by [hidden email]: Strange $ evaluates to  
Character arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

Thanks for taking some dust off!
I gave it another round in Squeak and came to a bit more clean-up

Nicolas Cellier uploaded a new version of Compiler to project The Trunk:
http://source.squeak.org/trunk/Compiler-nice.223.mcz

==================== Summary ====================

Name: Compiler-nice.223
Author: nice
Time: 9 February 2012, 4:05:21.21 pm
UUID: 8c21622a-6317-4c71-ad1a-607ba1683e6c
Ancestors: Compiler-nice.222

Correct a bug (see http://code.google.com/p/pharo/issues/detail?id=4650)

self should: [Compiler evaluate: '$'] raise: Error.

I also simplified all the redundancy (aheadChar == DoItCharacter and:  
[source atEnd]) and replaced with aheadChar == DoItCharacter.
Indeed, these were uggly incomplete guards trying to distinguish a  
DoItCharacter marking an end of stream from a (Character value: 30 "ASCII  
character RS means Record Separator") encountered in source...
But they would not even handle the case when DoItCharacter is the last  
source character., except maybe the uggliest contorsions in xDigit...

Instead I replaced the DoItCharacter marking the endOfStream by a character  
that we should never encounter in source.
I have chosen 16r10FFFF which is the last unicode and will never be used to  
encode a character (as all ending in FFFE and FFFF).
A different strategy would be to use a value greater than the last unicode,  
like 16r110000, and would also work...
Or use a different Object. In this later case, the object would have to  
understand charCode or we would have to change more Scanner methods (at  
least typeTableAt:).

Note that with current Character implementation, (Character value:  
16r10FFFF) ~~ (Character value: 16r10FFFF).
Since all tests are written with identity test aheadChar or hereChar ==  
DoItCharacter, even if such Character were encountered in source, it  
wouldn't be interpreted as an endOfStream mark, thus any Character code >  
255 could have been used, but this would be more fragile.

Consequently, I also modified the character type table to not interpret  
(Character value: 30) as end of source (#doIt).
I think that it was previously possible to insert a (Character value: 30)  
in source, and everything after that would have been ignored (if not  
between quotes) and could potentially store meta information. But this was  
both undocumented and AFAIK unused. It's easy to go back if we want to, by  
restoring previous version of #initializeTypeTable.

While doing this, I noticed that (Character value: 30) will now be  
interpreted as binary as many other characters including invisible control  
characters...
It could thus be used in a binary selector! That's crazy and I suggest  
using xIllegal in the Character typeTable, since I had previously prepared  
this method...
But one change at a time...

=============== Diff against Compiler-nice.222 ===============

Item was changed:
  ----- Method: Parser>>advance (in category 'scanning') -----
  advance
        | this |
        prevMark := hereMark.
        prevEnd := hereEnd.
        this := here.
        here := token.
        hereType := tokenType.
        hereMark := mark.
+       hereEnd := source position - (aheadChar == DoItCharacter
+               ifTrue: [hereChar == DoItCharacter
-       hereEnd := source position - ((aheadChar == 30 asCharacter and:  
[source atEnd])
-               ifTrue: [hereChar == 30 asCharacter
                        ifTrue: [0]
                        ifFalse: [1]]
                ifFalse: [2]).
        self scanToken.
        "Transcript show: 'here: ', here printString, ' mark: ', hereMark  
printString, ' end: ', hereEnd printString; cr."
        ^this!

Item was changed:
  ----- Method: Scanner class>>initialize (in category 'initialization')  
-----
  initialize

        self initializeTypeTable.
+       "The unicode ending with FFFE or FFFF are non characters and can be  
used by applications if they wish.
+       We use last legal unicode 16r10FFFF to encode the end of source  
stream"
+       DoItCharacter := Character value: 16r10FFFF!
-       DoItCharacter := Character value: 30!

Item was changed:
  ----- Method: Scanner class>>initializeTypeTable (in  
category 'initialization') -----
  initializeTypeTable
        "self initializeTypeTable"

        | newTable |
        newTable := Array new: 256 withAll: #xBinary. "default"
        newTable atAll: #(9 10 12 13 32 ) put: #xDelimiter. "tab lf ff cr  
space"
        newTable atAll: ($0 asciiValue to: $9 asciiValue) put: #xDigit.

        1 to: 255
                do: [:index |
                        (Character value: index) isLetter
                                ifTrue: [newTable at: index put: #xLetter]].

-       newTable at: 30 put: #doIt.
        newTable at: $" asciiValue put: #xDoubleQuote.
        newTable at: $# asciiValue put: #xLitQuote.
        newTable at: $$ asciiValue put: #xDollar.
        newTable at: $' asciiValue put: #xSingleQuote.
        newTable at: $: asciiValue put: #xColon.
        newTable at: $( asciiValue put: #leftParenthesis.
        newTable at: $) asciiValue put: #rightParenthesis.
        newTable at: $. asciiValue put: #period.
        newTable at: $; asciiValue put: #semicolon.
        newTable at: $[ asciiValue put: #leftBracket.
        newTable at: $] asciiValue put: #rightBracket.
        newTable at: ${ asciiValue put: #leftBrace.
        newTable at: $} asciiValue put: #rightBrace.
        newTable at: $^ asciiValue put: #upArrow.
        newTable at: $_ asciiValue put: #xUnderscore.
        newTable at: $| asciiValue put: #verticalBar.
        TypeTable := newTable "bon voyage!!"!

Item was changed:
  ----- Method: Scanner>>scanToken (in category 'expression types') -----
  scanToken

        [(tokenType := self typeTableAt: hereChar) == #xDelimiter]
                whileTrue: [self step].  "Skip delimiters fast, there almost  
always is one."
+       mark := aheadChar == DoItCharacter
-       mark := (aheadChar == DoItCharacter and: [source atEnd])
                ifTrue: [source position]
                ifFalse: [source position - 1].
        (tokenType at: 1) == $x "x as first letter"
                ifTrue: [self perform: tokenType "means perform to compute  
token & type"]
                ifFalse: [token := self step asSymbol "else just unique the  
first char"].
        ^token!

Item was changed:
  ----- Method: Scanner>>typeTableAt: (in category 'multi-character scans')  
-----
  typeTableAt: aCharacter
+       ^typeTable
+               at: aCharacter charCode
+               ifAbsent:
+                       [aCharacter == DoItCharacter
+                               ifTrue: [#doIt]
+                               ifFalse: [#xLetter]]!
-       ^typeTable at: aCharacter charCode ifAbsent:[#xLetter]!

Item was changed:
  ----- Method: Scanner>>xDigit (in category 'multi-character scans') -----
  xDigit
        "Form a number."

        tokenType := #number.
+       aheadChar == DoItCharacter
-       (aheadChar == DoItCharacter and: [source atEnd
-                       and:  [source skip: -1. source next ~~  
DoItCharacter]])
                ifTrue: [source skip: -1 "Read off the end last time"]
                ifFalse: [source skip: -2].
        token := (SqNumberParser on: source)
                failBlock: [:errorString :position | self notify:  
errorString at:position];
                nextNumber.
        self step; step!

Item was changed:
  ----- Method: Scanner>>xDollar (in category 'multi-character scans') -----
  xDollar
        "Form a Character literal."

+       aheadChar == DoItCharacter ifTrue: [^self offEnd: 'A Character was  
expected'].
        self step. "pass over $"
        token := self step.
        tokenType := #character!

Item was changed:
  ----- Method: Scanner>>xDoubleQuote (in category 'multi-character scans')  
-----
  xDoubleQuote
      "Collect a comment."
-     "wod 1/10/98: Allow 'empty' comments by testing the first character
- for $"" rather than blindly adding it to the comment being collected."

        buffer reset.
        self step.
        [ hereChar == $" ] whileFalse: [
+               hereChar == DoItCharacter ifTrue: [
-               (hereChar == DoItCharacter and: [ source atEnd ]) ifTrue: [
                        ^self offEnd: 'Unmatched comment quote' ].
                buffer nextPut: self step ].
        self step.
        (currentComment ifNil: [
                currentComment := OrderedCollection new ])
                        add: buffer contents.
        self scanToken!

Item was changed:
  ----- Method: Scanner>>xLetter (in category 'multi-character scans') -----
  xLetter
        "Form a word or keyword."

        | type |
        buffer reset.
        [(type := self typeTableAt: hereChar) == #xLetter
                or: [type == #xDigit
                or: [type == #xUnderscore and:[self  
allowUnderscoreSelectors]]]] whileTrue:
                        ["open code step for speed"
                        buffer nextPut: hereChar.
                        hereChar := aheadChar.
                        aheadChar := source atEnd
+                                                       ifTrue:  
[DoItCharacter]
-                                                       ifTrue: [30  
asCharacter "doit"]
                                                        ifFalse: [source  
next]].
+       tokenType := (type == #xColon and: [aheadChar ~~ $=])
-       tokenType := (type == #colon or: [type == #xColon and: [aheadChar  
~~ $=]])
                                        ifTrue:
                                                [buffer nextPut: self step.
                                                "Allow any number of  
embedded colons in literal symbols"
                                                [(self typeTableAt:  
hereChar) == #xColon] whileTrue:
                                                        [buffer nextPut:  
self step].
                                                #keyword]
                                        ifFalse:
                                                [#word].
        token := buffer contents!

Item was changed:
  ----- Method: Scanner>>xSingleQuote (in category 'multi-character scans')  
-----
  xSingleQuote
        "String."

        self step.
        buffer reset.
        [hereChar == $'
                and: [aheadChar == $'
                                ifTrue: [self step. false]
                                ifFalse: [true]]]
                whileFalse:
+                       [hereChar == DoItCharacter
+                               ifTrue: [^self offEnd: 'Unmatched string  
quote'].
+                       buffer nextPut: self step].
-                       [buffer nextPut: self step.
-                       (hereChar == DoItCharacter and: [source atEnd])
-                               ifTrue: [^self offEnd: 'Unmatched string  
quote']].
        self step.
        token := buffer contents.
        tokenType := #string!


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: Workneeded

Comment #6 on issue 4650 by [hidden email]: Strange $ evaluates to  
Character arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

(No comment was entered for this change.)


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: FixReviewNeeded

Comment #7 on issue 4650 by [hidden email]: Strange $ evaluates to  
Character arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

Name:  
SLICE-Issue-4650-Strange--evaluates-to-Character-arrowUp-secondRound-nice.1
Dependencies: Compiler-nice.314

Second round clean-up while fixing (Compiler evaluate: '^$').
Apply the clean up of Squeak with a DoItCharacter class var.
While at it, as Squeak did, and since this change touch approximately the  
same methods, transform a few Symbol comparison = into == to marginally  
fast up the Scanner.

WARNING: IF ATOMIC LOADING IS UNABLE TO LOAD ALL CODE IN ONCE
(I didn't test and have no more time slot this evening)
THEN WE SHALL INITIALIZE DoItCharacter with value: 30 IN A FIRST SLICE
THEN RELOAD THIS SLICE...

TODO(IT): clean up Character typeTable which has moved to String: remove 30  
-> #doIt


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #8 on issue 4650 by [hidden email]: Strange $ evaluates to  
Character arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

I tried to merge versus Compiler.MarcusDenker.315 and did not get any  
problem. But a change is missing, #xDoubleQuote still checks 30 asCharacter  
instead of DoItCharacter...

This fix is superseded by the one in Issue 3439 because they overlap, and  
Issue 3439 does contain both the xDoubleQuote change and a bunch of tests  
including one covering this issue, so I think I will not commit yet another  
SLICE for this issue.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #9 on issue 4650 by [hidden email]: Strange $ evaluates to  
Character arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

ok so we close this an merge it with 3439?


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: Closed

Comment #10 on issue 4650 by [hidden email]: Strange $ evaluates to  
Character arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

OK


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: Duplicate
        Mergedinto: 3439

Comment #11 on issue 4650 by [hidden email]: Strange $ evaluates to  
Character arrowUp
http://code.google.com/p/pharo/issues/detail?id=4650

(No comment was entered for this change.)


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker