Refactoring Browser error

> Hi All,
>
> If I attempt to execute the following:
>
> | rules |
>
> rules := RBParseTreeRewriter new.
>
> rules replace: 'state at: `@index put: `@expr' with: 'state at:
> `@index put: `@expr asC_unsigned_int'.

>
> (rules executeTree: (MD5Plugin parseTreeFor:
> #md5ProcessBuffer:withState:)) ifTrue:
> [rules tree newSource inspect].
> self error: 'parse tree rewrite failed'
>
>
> I get "Error: End of statement list encountered" (stack trace below).
> The same works and produces the expected output in Pharo 7.
>
> It looks like the Refactoring Browser has diverged quite a bit between
> Squeak and Pharo.
>
> Who looks after RB in Squeak, and is there a plan to keep in sync with
> Pharo, or the other way around, or should this be fixed independently?
>
> Thanks,
> Alistair
>
>
>
>
> 4 November 2018 8:01:29.599623 pm
>
> VM: unix - Smalltalk
> Image: Squeak5.3alpha [latest update: #18261]
>
> SecurityManager state:
> Restricted: false
> FileAccess: true
> SocketAccess: true
> Working Dir /home/alistair/vmmaker/opensmalltalk-vm/sqvmm.03
> Trusted Dir /home/alistair/vmmaker/opensmalltalk-vm/sqvmm.03/secure
> Untrusted Dir /home/alistair/vmmaker/opensmalltalk-vm/sqvmm.03/My Squeak
>
> RBPatternParser(Object)>>error:
> Receiver: a RBPatternParser
> Arguments and temporary variables:
> aString: 'End of statement list encounted'
> Receiver's instance variables:
> scanner: a RBPatternScanner
> currentToken: RBShortAssignmentToken
> nextToken: nil
> errorBlock: nil
> source: 'state at: `@index put: `@expr asC_unsigned_int'
> comments: an OrderedCollection()
> pragmas: nil
>
> RBPatternParser(RBParser)>>parserError:
> Receiver: a RBPatternParser
> Arguments and temporary variables:
> aString: 'End of statement list encounted'
> Receiver's instance variables:
> scanner: a RBPatternScanner
> currentToken: RBShortAssignmentToken
> nextToken: nil
> errorBlock: nil
> source: 'state at: `@index put: `@expr asC_unsigned_int'
> comments: an OrderedCollection()
> pragmas: nil
>
> RBPatternParser(RBParser)>>parseStatementList:into:
> Receiver: a RBPatternParser
> Arguments and temporary variables:
> pragmaBoolean: false
> sequenceNode: RBSequenceNode()
> statements: an OrderedCollection(RBMessageNode(state at: `@index put:
> `@expr as...etc...
> return: true
> periods: an OrderedCollection()
> returnPosition: nil
> node: RBMessageNode(state at: `@index put: `@expr asC)
> Receiver's instance variables:
> scanner: a RBPatternScanner
> currentToken: RBShortAssignmentToken
> nextToken: nil
> errorBlock: nil
> source: 'state at: `@index put: `@expr asC_unsigned_int'
> comments: an OrderedCollection()
> pragmas: nil
>
> RBPatternParser(RBParser)>>parseStatements:
> Receiver: a RBPatternParser
> Arguments and temporary variables:
> pragmaBoolean: false
> args: #()
> leftBar: nil
> rightBar: nil
> Receiver's instance variables:
> scanner: a RBPatternScanner
> currentToken: RBShortAssignmentToken
> nextToken: nil
> errorBlock: nil
> source: 'state at: `@index put: `@expr asC_unsigned_int'
> comments: an OrderedCollection()
> pragmas: nil
>
> RBPatternParser(RBParser)>>parseExpression:
> Receiver: a RBPatternParser
> Arguments and temporary variables:
> aString: 'state at: `@index put: `@expr asC_unsigned_int'
> node: nil
> Receiver's instance variables:
> scanner: a RBPatternScanner
> currentToken: RBShortAssignmentToken
> nextToken: nil
> errorBlock: nil
> source: 'state at: `@index put: `@expr asC_unsigned_int'
> comments: an OrderedCollection()
> pragmas: nil
>
> RBPatternParser class(RBParser class)>>parseExpression:onError:
> Receiver: RBPatternParser
> Arguments and temporary variables:
> aString: 'state at: `@index put: `@expr asC_unsigned_int'
> aBlock: nil
> node: nil
> parser: a RBPatternParser
> Receiver's instance variables:
> superclass: RBParser
> methodDict: a MethodDictionary(#messageNodeClass->(RBPatternParser>>#messageNod...etc...
> format: 65543
> instanceVariables: nil
> organization: ('private-classes' messageNodeClass methodNodeClass
> pragmaNodeClass...etc...
> subclasses: nil
> name: #RBPatternParser
> classPool: nil
> sharedPools: nil
> environment: Smalltalk
> category: #'AST-Core-Parser'
>
> RBParser class>>parseRewriteExpression:onError:
> Receiver: RBParser
> Arguments and temporary variables:
> aString: 'state at: `@index put: `@expr asC_unsigned_int'
> aBlock: nil
> Receiver's instance variables:
> superclass: Object
> methodDict: a MethodDictionary(#addCommentsTo:->(RBParser>>#addCommentsTo:
> "a CompiledMethod...etc...
> format: 65543
> instanceVariables: #('scanner' 'currentToken' 'nextToken' 'errorBlock'
> 'source'...etc...
> organization: ('private' addCommentsTo: nextToken
> patchLiteralArrayToken patchLiteralMessage...etc...
> subclasses: {RBPatternParser}
> name: #RBParser
> classPool: nil
> sharedPools: nil
> environment: Smalltalk
> category: #'AST-Core-Parser'
>
> RBParser class>>parseRewriteExpression:
> Receiver: RBParser
> Arguments and temporary variables:
> aString: 'state at: `@index put: `@expr asC_unsigned_int'
> Receiver's instance variables:
> superclass: Object
> methodDict: a MethodDictionary(#addCommentsTo:->(RBParser>>#addCommentsTo:
> "a CompiledMethod...etc...
> format: 65543
> instanceVariables: #('scanner' 'currentToken' 'nextToken' 'errorBlock'
> 'source'...etc...
> organization: ('private' addCommentsTo: nextToken
> patchLiteralArrayToken patchLiteralMessage...etc...
> subclasses: {RBPatternParser}
> name: #RBParser
> classPool: nil
> sharedPools: nil
> environment: Smalltalk
> category: #'AST-Core-Parser'
>
> RBStringReplaceRule>>replaceString:
> Receiver: a RBStringReplaceRule
> Arguments and temporary variables:
> replaceString: 'state at: `@index put: `@expr asC_unsigned_int'
> Receiver's instance variables:
> searchTree: RBMessageNode(state at: `@index put: `@expr)
> owner: nil
> verificationBlock: [closure] in RBStringReplaceRule(RBReplaceRule)>>initialize
> replaceTree: nil
>
> RBStringReplaceRule>>searchFor:replaceWith:
> Receiver: a RBStringReplaceRule
> Arguments and temporary variables:
> searchString: 'state at: `@index put: `@expr'
> replaceString: 'state at: `@index put: `@expr asC_unsigned_int'
> Receiver's instance variables:
> searchTree: RBMessageNode(state at: `@index put: `@expr)
> owner: nil
> verificationBlock: [closure] in RBStringReplaceRule(RBReplaceRule)>>initialize
> replaceTree: nil
>
> RBStringReplaceRule class>>searchFor:replaceWith:
> Receiver: RBStringReplaceRule
> Arguments and temporary variables:
> searchString: 'state at: `@index put: `@expr'
> replaceString: 'state at: `@index put: `@expr asC_unsigned_int'
> Receiver's instance variables:
> superclass: RBReplaceRule
> methodDict: a MethodDictionary(#foundMatchFor:->(RBStringReplaceRule>>#foundMat...etc...
> format: 65540
> instanceVariables: #('replaceTree')
> organization: ('matching' foundMatchFor:)
> ('initialize-release' methodReplaceString:...etc...
> subclasses: nil
> name: #RBStringReplaceRule
> classPool: nil
> sharedPools: nil
> environment: Smalltalk
> category: #'AST-Core-Matching'
>
> RBParseTreeRewriter>>replace:with:
> Receiver: a RBParseTreeRewriter
> Arguments and temporary variables:
> searchString: 'state at: `@index put: `@expr'
> replaceString: 'state at: `@index put: `@expr asC_unsigned_int'
> Receiver's instance variables:
> searches: an OrderedCollection()
> answer: nil
> argumentSearches: an OrderedCollection()
> context: a RBSmallDictionary
> messages: nil
> tree: nil
>
> UndefinedObject>>DoIt
> Receiver: nil
> Arguments and temporary variables:
> rules: a RBParseTreeRewriter
> Receiver's instance variables:
> nil
>
>
> --- The full stack ---
> RBPatternParser(Object)>>error:
> RBPatternParser(RBParser)>>parserError:
> RBPatternParser(RBParser)>>parseStatementList:into:
> RBPatternParser(RBParser)>>parseStatements:
> RBPatternParser(RBParser)>>parseExpression:
> RBPatternParser class(RBParser class)>>parseExpression:onError:
> RBParser class>>parseRewriteExpression:onError:
> RBParser class>>parseRewriteExpression:
> RBStringReplaceRule>>replaceString:
> RBStringReplaceRule>>searchFor:replaceWith:
> RBStringReplaceRule class>>searchFor:replaceWith:
> RBParseTreeRewriter>>replace:with:
> UndefinedObject>>DoIt

>
> On Sun, 4 Nov 2018, Alistair Grant wrote:
>
> > Hi All,
> >
> > If I attempt to execute the following:
> >
> > | rules |
> >
> > rules := RBParseTreeRewriter new.
> >
> > rules replace: 'state at: `@index put: `@expr' with: 'state at:
> > `@index put: `@expr asC_unsigned_int'.
>
> I presume the version you use doesn't support underscores in selectors.
> Does it work if you use a selector with no underscores in it?

> Levente
>
> >
> > (rules executeTree: (MD5Plugin parseTreeFor:
> > #md5ProcessBuffer:withState:)) ifTrue:
> > [rules tree newSource inspect].
> > self error: 'parse tree rewrite failed'
> >
> >
> > I get "Error: End of statement list encountered" (stack trace below).
> > The same works and produces the expected output in Pharo 7.
> >
> > It looks like the Refactoring Browser has diverged quite a bit between
> > Squeak and Pharo.
> >
> > Who looks after RB in Squeak, and is there a plan to keep in sync with
> > Pharo, or the other way around, or should this be fixed independently?
> >
> > Thanks,
> > Alistair

> Hi Levente,
>
> On Mon, 5 Nov 2018 at 00:35, Levente Uzonyi <[hidden email]> wrote:
>>
>> On Sun, 4 Nov 2018, Alistair Grant wrote:
>>
>> > Hi All,
>> >
>> > If I attempt to execute the following:
>> >
>> > | rules |
>> >
>> > rules := RBParseTreeRewriter new.
>> >
>> > rules replace: 'state at: `@index put: `@expr' with: 'state at:
>> > `@index put: `@expr asC_unsigned_int'.
>>
>> I presume the version you use doesn't support underscores in selectors.
>> Does it work if you use a selector with no underscores in it?
>
> That's it, thanks!  (it works without the underscores)
>
> I hate it when I forget to include version information, sorry:
>
> Squeak5.3alpha
> latest update: #18261
> Image format 68021 (64 bit)
>
> AST-Core-eem.97
> Refactoring-Changes-eem.22
> Refactoring-Core-ul.162
> Refactoring-Environment-eem.11
> Refactoring-Squeak-Platform-cwp.2
> Refactoring-Tests-Changes-lr.16
> Refactoring-Tests-Core-lr.56
> Refactoring-Tests-Environment-eem.5
>
>
>> How did you load RB? Did you use the Metacello configuration? (it may
>> not be up-to-date)
>
> All the above appear to be the latest versions on
> http://www.squeaksource.com/rb.
>
> This is a vmmaker image, they were all loaded through:
>
> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/BuildSqueakSpurTrunkVMMakerImage.st
>
> It sounds like there may be an updated version of RB that already
> addresses this?
>
> Thanks again,
> Alistair
>
>
>> Levente
>>
>> >
>> > (rules executeTree: (MD5Plugin parseTreeFor:
>> > #md5ProcessBuffer:withState:)) ifTrue:
>> > [rules tree newSource inspect].
>> > self error: 'parse tree rewrite failed'
>> >
>> >
>> > I get "Error: End of statement list encountered" (stack trace below).
>> > The same works and produces the expected output in Pharo 7.
>> >
>> > It looks like the Refactoring Browser has diverged quite a bit between
>> > Squeak and Pharo.
>> >
>> > Who looks after RB in Squeak, and is there a plan to keep in sync with
>> > Pharo, or the other way around, or should this be fixed independently?
>> >
>> > Thanks,
>> > Alistair

> The cause of the error is that RBScanner considers _ to be #special
> instead of #alphabetic when Scanner class >> #allowUnderscoreAsAssignment
> returns true.
> In Pharo this is not an issue, because underscore assignments have been
> abandoned and underscores in selectors are allowed permanently.
> But in Squeak everything can happen (so underscore can either be #special
> or #alphabetic depending on its context) and RBScanner would need some
> rewrite to support those cases.
>
> I suggest the line
>
>         Scanner allowUnderscoreAsAssignment: false
>
> be added to BuildSqueakSpurTrunkVMMakerImage.st before RB is loaded to
> avoid this issue in future images (because VMMaker code shouldn't rely
> on underscore assignments being supported).

> Levente
>
> On Mon, 5 Nov 2018, Alistair Grant wrote:
>
> > Hi Levente,
> >
> > On Mon, 5 Nov 2018 at 00:35, Levente Uzonyi <[hidden email]> wrote:
> >>
> >> On Sun, 4 Nov 2018, Alistair Grant wrote:
> >>
> >> > Hi All,
> >> >
> >> > If I attempt to execute the following:
> >> >
> >> > | rules |
> >> >
> >> > rules := RBParseTreeRewriter new.
> >> >
> >> > rules replace: 'state at: `@index put: `@expr' with: 'state at:
> >> > `@index put: `@expr asC_unsigned_int'.
> >>
> >> I presume the version you use doesn't support underscores in selectors.
> >> Does it work if you use a selector with no underscores in it?
> >
> > That's it, thanks!  (it works without the underscores)
> >
> > I hate it when I forget to include version information, sorry:
> >
> > Squeak5.3alpha
> > latest update: #18261
> > Image format 68021 (64 bit)
> >
> > AST-Core-eem.97
> > Refactoring-Changes-eem.22
> > Refactoring-Core-ul.162
> > Refactoring-Environment-eem.11
> > Refactoring-Squeak-Platform-cwp.2
> > Refactoring-Tests-Changes-lr.16
> > Refactoring-Tests-Core-lr.56
> > Refactoring-Tests-Environment-eem.5
> >
> >
> >> How did you load RB? Did you use the Metacello configuration? (it may
> >> not be up-to-date)
> >
> > All the above appear to be the latest versions on
> > http://www.squeaksource.com/rb.
> >
> > This is a vmmaker image, they were all loaded through:
> >
> > https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/BuildSqueakSpurTrunkVMMakerImage.st
> >
> > It sounds like there may be an updated version of RB that already
> > addresses this?
> >
> > Thanks again,
> > Alistair
> >
> >
> >> Levente
> >>
> >> >
> >> > (rules executeTree: (MD5Plugin parseTreeFor:
> >> > #md5ProcessBuffer:withState:)) ifTrue:
> >> > [rules tree newSource inspect].
> >> > self error: 'parse tree rewrite failed'
> >> >
> >> >
> >> > I get "Error: End of statement list encountered" (stack trace below).
> >> > The same works and produces the expected output in Pharo 7.
> >> >
> >> > It looks like the Refactoring Browser has diverged quite a bit between
> >> > Squeak and Pharo.
> >> >
> >> > Who looks after RB in Squeak, and is there a plan to keep in sync with
> >> > Pharo, or the other way around, or should this be fixed independently?
> >> >
> >> > Thanks,
> >> > Alistair
>