RBScanner and compile time constants

Hi,

I finally understand the issue (I have already beeng debugging this
in autumn).

  STInST.RBParser parseExpression: '#(##(1/2))'   -> error

This is because of:

  (STInST.RBScanner on: '#(##(1/2))') next next   -> $)

and this is because of...


    scanLiteralArrayParts [
        <category: 'private-scanning'>
        currentCharacter == $# ifTrue: [^self scanLiteral].

...

    scanLiteral [
        <category: 'private-scanning'>
        self step.
        self stripSeparators.
...
        currentCharacter == $# ifTrue: [^self scanExtendedLiterals].



    scanExtendedLiterals [
        <category: 'private-scanning numbers'>
        | token |
        self step.
        currentCharacter == $(
            ifTrue:
                [self step.
                ^RBOptimizedToken start: tokenStart].
        self scannerError: 'Expecting parentheses'
    ]


...


So scanExtendedLiterals will consume the $( of ##( but the $) of the literal will
not be consumed leaving the extra token.

Now my question is if the optimized token should recursively scan the
literals? Or should the RBScanner count $( and $) to know that it needs
to read one more item?


holger

_______________________________________________
help-smalltalk mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-smalltalk

Il 20/12/2013 22:21, Holger Hans Peter Freyther ha scritto:
> Now my question is if the optimized token should recursively scan the
> literals? Or should the RBScanner count $( and $) to know that it needs
> to read one more item?

Keeping track of the number of parentheses, and exiting scanLiteralArray
when it matches the count at entry, sounds like the easiest solution.

Only #scanLiteralArray and #scanSpecialCharacter deal with parentheses.

Paolo

_______________________________________________
help-smalltalk mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-smalltalk

On Fri, Dec 20, 2013 at 11:11:16PM +0100, Paolo Bonzini wrote:
> Il 20/12/2013 22:21, Holger Hans Peter Freyther ha scritto:
> > Now my question is if the optimized token should recursively scan the
> > literals? Or should the RBScanner count $( and $) to know that it needs
> > to read one more item?
>
> Keeping track of the number of parentheses, and exiting scanLiteralArray
> when it matches the count at entry, sounds like the easiest solution.
>
> Only #scanLiteralArray and #scanSpecialCharacter deal with parentheses.


Before reading your mail I came up with that. I can change it to add
counting of currentCharacter==$( and currentCharacter==$) in RBScanner>>#step
instead?

With the below diff I can tokenize/parse #(##(1/2)) the only issue is that
compiling it doesn't work...

Works:
        res := STEvaluationDriver new
            parseSmalltalkStream: '##(1/2)' readStream
            with: GSTFileInParser.
        self assert: res equals: 1/2.

Doesn't:

        res := STEvaluationDriver new
            parseSmalltalkStream: '#(##(1/2))' readStream
            with: GSTFileInParser.
        res inspect.

STInST.RBOptimizedToken(Object)>>doesNotUnderstand: #realValue (SysExcept.st:1408)
optimized [] in STInST.RBLiteralToken>>realValue (RBToken.st:227)
Array(ArrayedCollection)>>collect: (ArrayColl.st:307)
STInST.RBLiteralToken>>realValue (RBToken.st:227)
STInST.RBLiteralNode>>value (RBParseNodes.st:2616)
STInST.STCompiler>>acceptLiteralNode: (STCompiler.st:626)
STInST.RBLiteralNode>>acceptVisitor: (RBParseNodes.st:2654)
STInST.STCompiler>>acceptReturnNode: (STCompiler.st:896)
STInST.RBReturnNode>>acceptVisitor: (RBParseNodes.st:1594)
optimized [] in STInST.STCompiler>>compileStatements: (STCompiler.st:570)
Array(SequenceableCollection)>>keysAndValuesDo: (SeqCollect.st:886)
STInST.STCompiler>>compileStatements: (STCompiler.st:571)
STInST.STCompiler>>acceptSequenceNode: (STCompiler.st:447)
STInST.RBSequenceNode>>acceptVisitor: (RBParseNodes.st:981)
STInST.STCompiler(STInST.RBProgramNodeVisitor)>>visitNode: (RBParseNodes.st:49)
STInST.STCompiler class>>compile:asMethodOf:classified:parser:environment: (STCompiler.st:161)
STInST.STEvaluationDriver>>evaluate: (STEvaluationDriver.st:209)




diff --git a/packages/stinst/parser/RBParser.st b/packages/stinst/parser/RBParser.st
index b953762..3314ff9 100644
--- a/packages/stinst/parser/RBParser.st
+++ b/packages/stinst/parser/RBParser.st
@@ -1215,16 +1215,20 @@ Stream subclass: RBScanner [
 
     scanLiteralArray [
        <category: 'private-scanning'>
-       | arrayStream start |
+       | arrayStream start optimizedNodes |
        arrayStream := WriteStream on: (Array new: 10).
        self step.
        start := tokenStart.
+       optimizedNodes := 0.
       
        [self stripSeparators.
        tokenStart := stream position.
-       currentCharacter == $)]
+       currentCharacter == $) and: [optimizedNodes = 0]]
                whileFalse:
-                   [arrayStream nextPut: self scanLiteralArrayParts.
+                   [ |res|
+                   res := arrayStream nextPut: self scanLiteralArrayParts.
+                   res isOptimized ifTrue: [optimizedNodes := optimizedNodes + 1].
+                   res isSpecial ifTrue: [optimizedNodes := optimizedNodes - 1].
                    buffer reset].
        self step.
        ^RBLiteralToken
@@ -1254,6 +1258,8 @@ Stream subclass: RBScanner [
        currentCharacter == $$ ifTrue: [^self scanLiteralCharacter].
        currentCharacter == $( ifTrue: [^self scanLiteralArray].
        currentCharacter == $[ ifTrue: [^self scanByteArray].
+       "In the case of RBOptimizedToken we scan for $) as well"
+       currentCharacter == $) ifTrue: [^self scanSpecialCharacter].
        ^self scannerError: 'Unknown character in literal array'
     ]


_______________________________________________
help-smalltalk mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-smalltalk