Underscore selectors
Locked
 
|
Hi all,
I'd like to have the preference to allow underscore selectors working in 4.4. To do that, we'll need to fix Character>>tokenish to honor the preference, ideally without degrading the performance of fileIn and package loading. I don't remember all the issues that came up when the preference was introduced, though. Is there anything else that needs to be done? Colin |
|
|
On Mon, 13 Aug 2012, Colin Putney wrote:
> Hi all, > > I'd like to have the preference to allow underscore selectors working > in 4.4. To do that, we'll need to fix Character>>tokenish to honor the > preference, ideally without degrading the performance of fileIn and > package loading. I don't remember all the issues that came up when the > preference was introduced, though. Is there anything else that needs > to be done? I started to collect the todos here: http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-July/164980.html One more thing that needs to be done is to implement a small tool, which can convert code with underscore assignments to ansi assignments. I have a prototype, but it's not fully working due to a bug in Scanner >> #scanTokenPositionsIn:into:. Here's a snippet that demonstrates the issue: Array streamContents: [ :stream | | source scanner | source := 'x := 1. x := 2'. scanner := Scanner new. scanner scanTokenPositionsIn: source into: [ :start :end | stream nextPut: { start. end. source copyFrom: start to: end. scanner instVarNamed: #tokenType } ] ] The first assignment x := 1 is tokenized properly to #word, #leftArrow and #number, but the second one is not. The last 2 tokens are combined into a #doIt, so the code cannot find fix last assignment in a method. Levente > > Colin > > |
|
|
Yep, this code is highly questionnable:
"Compensate for the fact that the parser uses two character lookahead. Normally we must remove the extra two characters. But this mustn't happen for the last token at the end of stream." aBlock value: mark value: (source atEnd ifTrue: [tokenType := #doIt. "to cause an immediate ^self" source position] "ABOVE LINE WILL CAUSE A PREMATURE ^self SINCE THE SCANNER SCANS TWO CHARACTER AHEAD" ifFalse: [source position - 2])]. Nicolas 2012/8/13 Levente Uzonyi <[hidden email]>: > On Mon, 13 Aug 2012, Colin Putney wrote: > >> Hi all, >> >> I'd like to have the preference to allow underscore selectors working >> in 4.4. To do that, we'll need to fix Character>>tokenish to honor the >> preference, ideally without degrading the performance of fileIn and >> package loading. I don't remember all the issues that came up when the >> preference was introduced, though. Is there anything else that needs >> to be done? > > > I started to collect the todos here: > http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-July/164980.html > > One more thing that needs to be done is to implement a small tool, which can > convert code with underscore assignments to ansi assignments. I have a > prototype, but it's not fully working due to a bug in Scanner >> > #scanTokenPositionsIn:into:. Here's a snippet that demonstrates the issue: > > Array streamContents: [ :stream | > | source scanner | > source := 'x := 1. x := 2'. > scanner := Scanner new. > scanner scanTokenPositionsIn: source into: [ :start :end | > stream nextPut: { start. end. source copyFrom: start to: > end. scanner instVarNamed: #tokenType } ] ] > > The first assignment x := 1 is tokenized properly to #word, #leftArrow and > #number, but the second one is not. The last 2 tokens are combined into a > #doIt, so the code cannot find fix last assignment in a method. > > > Levente > >> >> Colin >> >> > |
|
|
It should more or less use the same logic than Parser>>advance
hereEnd := source position - (aheadChar == DoItCharacter ifTrue: [hereChar == DoItCharacter ifTrue: [0] ifFalse: [1]] ifFalse: [2]). Nicolas 2012/8/13 Nicolas Cellier <[hidden email]>: > Yep, this code is highly questionnable: > > "Compensate for the fact that the parser uses two character lookahead. > Normally we must > remove the extra two characters. But this mustn't happen for the > last token at the end of stream." > aBlock > value: mark > value: (source atEnd > ifTrue: [tokenType := #doIt. "to cause an immediate ^self" source position] > "ABOVE LINE WILL CAUSE A PREMATURE ^self SINCE THE SCANNER SCANS TWO > CHARACTER AHEAD" > ifFalse: [source position - 2])]. > > Nicolas > > 2012/8/13 Levente Uzonyi <[hidden email]>: >> On Mon, 13 Aug 2012, Colin Putney wrote: >> >>> Hi all, >>> >>> I'd like to have the preference to allow underscore selectors working >>> in 4.4. To do that, we'll need to fix Character>>tokenish to honor the >>> preference, ideally without degrading the performance of fileIn and >>> package loading. I don't remember all the issues that came up when the >>> preference was introduced, though. Is there anything else that needs >>> to be done? >> >> >> I started to collect the todos here: >> http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-July/164980.html >> >> One more thing that needs to be done is to implement a small tool, which can >> convert code with underscore assignments to ansi assignments. I have a >> prototype, but it's not fully working due to a bug in Scanner >> >> #scanTokenPositionsIn:into:. Here's a snippet that demonstrates the issue: >> >> Array streamContents: [ :stream | >> | source scanner | >> source := 'x := 1. x := 2'. >> scanner := Scanner new. >> scanner scanTokenPositionsIn: source into: [ :start :end | >> stream nextPut: { start. end. source copyFrom: start to: >> end. scanner instVarNamed: #tokenType } ] ] >> >> The first assignment x := 1 is tokenized properly to #word, #leftArrow and >> #number, but the second one is not. The last 2 tokens are combined into a >> #doIt, so the code cannot find fix last assignment in a method. >> >> >> Levente >> >>> >>> Colin >>> >>> >> |
|
|
That would answer:
#(#(1 1 'x' #word) #(3 4 ':=' #leftArrow) #(6 6 '1' #number) #(7 7 '.' #period) #(9 9 'x' #word) #(11 12 ':=' #leftArrow) #(14 14 '2' #number) #(15 14 '' #doIt)) Is it OK to have a final empty #doit ? Nicolas 2012/8/13 Nicolas Cellier <[hidden email]>: > It should more or less use the same logic than Parser>>advance > > hereEnd := source position - (aheadChar == DoItCharacter > ifTrue: [hereChar == DoItCharacter > ifTrue: [0] > ifFalse: [1]] > ifFalse: [2]). > > Nicolas > > 2012/8/13 Nicolas Cellier <[hidden email]>: >> Yep, this code is highly questionnable: >> >> "Compensate for the fact that the parser uses two character lookahead. >> Normally we must >> remove the extra two characters. But this mustn't happen for the >> last token at the end of stream." >> aBlock >> value: mark >> value: (source atEnd >> ifTrue: [tokenType := #doIt. "to cause an immediate ^self" source position] >> "ABOVE LINE WILL CAUSE A PREMATURE ^self SINCE THE SCANNER SCANS TWO >> CHARACTER AHEAD" >> ifFalse: [source position - 2])]. >> >> Nicolas >> >> 2012/8/13 Levente Uzonyi <[hidden email]>: >>> On Mon, 13 Aug 2012, Colin Putney wrote: >>> >>>> Hi all, >>>> >>>> I'd like to have the preference to allow underscore selectors working >>>> in 4.4. To do that, we'll need to fix Character>>tokenish to honor the >>>> preference, ideally without degrading the performance of fileIn and >>>> package loading. I don't remember all the issues that came up when the >>>> preference was introduced, though. Is there anything else that needs >>>> to be done? >>> >>> >>> I started to collect the todos here: >>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-July/164980.html >>> >>> One more thing that needs to be done is to implement a small tool, which can >>> convert code with underscore assignments to ansi assignments. I have a >>> prototype, but it's not fully working due to a bug in Scanner >> >>> #scanTokenPositionsIn:into:. Here's a snippet that demonstrates the issue: >>> >>> Array streamContents: [ :stream | >>> | source scanner | >>> source := 'x := 1. x := 2'. >>> scanner := Scanner new. >>> scanner scanTokenPositionsIn: source into: [ :start :end | >>> stream nextPut: { start. end. source copyFrom: start to: >>> end. scanner instVarNamed: #tokenType } ] ] >>> >>> The first assignment x := 1 is tokenized properly to #word, #leftArrow and >>> #number, but the second one is not. The last 2 tokens are combined into a >>> #doIt, so the code cannot find fix last assignment in a method. >>> >>> >>> Levente >>> >>>> >>>> Colin >>>> >>>> >>> |
|
|
On Mon, 13 Aug 2012, Nicolas Cellier wrote:
> That would answer: > > #(#(1 1 'x' #word) #(3 4 ':=' #leftArrow) #(6 6 '1' #number) #(7 7 > '.' #period) #(9 9 'x' #word) #(11 12 ':=' #leftArrow) #(14 14 '2' > #number) #(15 14 '' #doIt)) > > Is it OK to have a final empty #doit ? I guess it is. If #doIt is there (which used to be), then it should be empty. Levente > > Nicolas > > 2012/8/13 Nicolas Cellier <[hidden email]>: >> It should more or less use the same logic than Parser>>advance >> >> hereEnd := source position - (aheadChar == DoItCharacter >> ifTrue: [hereChar == DoItCharacter >> ifTrue: [0] >> ifFalse: [1]] >> ifFalse: [2]). >> >> Nicolas >> >> 2012/8/13 Nicolas Cellier <[hidden email]>: >>> Yep, this code is highly questionnable: >>> >>> "Compensate for the fact that the parser uses two character lookahead. >>> Normally we must >>> remove the extra two characters. But this mustn't happen for the >>> last token at the end of stream." >>> aBlock >>> value: mark >>> value: (source atEnd >>> ifTrue: [tokenType := #doIt. "to cause an immediate ^self" source position] >>> "ABOVE LINE WILL CAUSE A PREMATURE ^self SINCE THE SCANNER SCANS TWO >>> CHARACTER AHEAD" >>> ifFalse: [source position - 2])]. >>> >>> Nicolas >>> >>> 2012/8/13 Levente Uzonyi <[hidden email]>: >>>> On Mon, 13 Aug 2012, Colin Putney wrote: >>>> >>>>> Hi all, >>>>> >>>>> I'd like to have the preference to allow underscore selectors working >>>>> in 4.4. To do that, we'll need to fix Character>>tokenish to honor the >>>>> preference, ideally without degrading the performance of fileIn and >>>>> package loading. I don't remember all the issues that came up when the >>>>> preference was introduced, though. Is there anything else that needs >>>>> to be done? >>>> >>>> >>>> I started to collect the todos here: >>>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-July/164980.html >>>> >>>> One more thing that needs to be done is to implement a small tool, which can >>>> convert code with underscore assignments to ansi assignments. I have a >>>> prototype, but it's not fully working due to a bug in Scanner >> >>>> #scanTokenPositionsIn:into:. Here's a snippet that demonstrates the issue: >>>> >>>> Array streamContents: [ :stream | >>>> | source scanner | >>>> source := 'x := 1. x := 2'. >>>> scanner := Scanner new. >>>> scanner scanTokenPositionsIn: source into: [ :start :end | >>>> stream nextPut: { start. end. source copyFrom: start to: >>>> end. scanner instVarNamed: #tokenType } ] ] >>>> >>>> The first assignment x := 1 is tokenized properly to #word, #leftArrow and >>>> #number, but the second one is not. The last 2 tokens are combined into a >>>> #doIt, so the code cannot find fix last assignment in a method. >>>> >>>> >>>> Levente >>>> >>>>> >>>>> Colin >>>>> >>>>> >>>> > > |
|
|
OK, I just commited, you can check if it helps...
Nicolas 2012/8/13 Levente Uzonyi <[hidden email]>: > On Mon, 13 Aug 2012, Nicolas Cellier wrote: > >> That would answer: >> >> #(#(1 1 'x' #word) #(3 4 ':=' #leftArrow) #(6 6 '1' #number) #(7 7 >> '.' #period) #(9 9 'x' #word) #(11 12 ':=' #leftArrow) #(14 14 '2' >> #number) #(15 14 '' #doIt)) >> >> Is it OK to have a final empty #doit ? > > > I guess it is. If #doIt is there (which used to be), then it should be > empty. > > > Levente > > >> >> Nicolas >> >> 2012/8/13 Nicolas Cellier <[hidden email]>: >>> >>> It should more or less use the same logic than Parser>>advance >>> >>> hereEnd := source position - (aheadChar == DoItCharacter >>> ifTrue: [hereChar == DoItCharacter >>> ifTrue: [0] >>> ifFalse: [1]] >>> ifFalse: [2]). >>> >>> Nicolas >>> >>> 2012/8/13 Nicolas Cellier <[hidden email]>: >>>> >>>> Yep, this code is highly questionnable: >>>> >>>> "Compensate for the fact that the parser uses two character lookahead. >>>> Normally we must >>>> remove the extra two characters. But this mustn't >>>> happen for the >>>> last token at the end of stream." >>>> aBlock >>>> value: mark >>>> value: (source atEnd >>>> ifTrue: [tokenType := #doIt. "to >>>> cause an immediate ^self" source position] >>>> "ABOVE LINE WILL CAUSE A PREMATURE ^self SINCE THE SCANNER SCANS TWO >>>> CHARACTER AHEAD" >>>> ifFalse: [source position - >>>> 2])]. >>>> >>>> Nicolas >>>> >>>> 2012/8/13 Levente Uzonyi <[hidden email]>: >>>>> >>>>> On Mon, 13 Aug 2012, Colin Putney wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I'd like to have the preference to allow underscore selectors working >>>>>> in 4.4. To do that, we'll need to fix Character>>tokenish to honor the >>>>>> preference, ideally without degrading the performance of fileIn and >>>>>> package loading. I don't remember all the issues that came up when the >>>>>> preference was introduced, though. Is there anything else that needs >>>>>> to be done? >>>>> >>>>> >>>>> >>>>> I started to collect the todos here: >>>>> >>>>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-July/164980.html >>>>> >>>>> One more thing that needs to be done is to implement a small tool, >>>>> which can >>>>> convert code with underscore assignments to ansi assignments. I have a >>>>> prototype, but it's not fully working due to a bug in Scanner >> >>>>> #scanTokenPositionsIn:into:. Here's a snippet that demonstrates the >>>>> issue: >>>>> >>>>> Array streamContents: [ :stream | >>>>> | source scanner | >>>>> source := 'x := 1. x := 2'. >>>>> scanner := Scanner new. >>>>> scanner scanTokenPositionsIn: source into: [ :start :end | >>>>> stream nextPut: { start. end. source copyFrom: start >>>>> to: >>>>> end. scanner instVarNamed: #tokenType } ] ] >>>>> >>>>> The first assignment x := 1 is tokenized properly to #word, #leftArrow >>>>> and >>>>> #number, but the second one is not. The last 2 tokens are combined into >>>>> a >>>>> #doIt, so the code cannot find fix last assignment in a method. >>>>> >>>>> >>>>> Levente >>>>> >>>>>> >>>>>> Colin >>>>>> >>>>>> >>>>> >> >> > |
|
|
On Mon, 13 Aug 2012, Nicolas Cellier wrote:
> OK, I just commited, you can check if it helps... Great, thanks. Levente > > Nicolas > > 2012/8/13 Levente Uzonyi <[hidden email]>: >> On Mon, 13 Aug 2012, Nicolas Cellier wrote: >> >>> That would answer: >>> >>> #(#(1 1 'x' #word) #(3 4 ':=' #leftArrow) #(6 6 '1' #number) #(7 7 >>> '.' #period) #(9 9 'x' #word) #(11 12 ':=' #leftArrow) #(14 14 '2' >>> #number) #(15 14 '' #doIt)) >>> >>> Is it OK to have a final empty #doit ? >> >> >> I guess it is. If #doIt is there (which used to be), then it should be >> empty. >> >> >> Levente >> >> >>> >>> Nicolas >>> >>> 2012/8/13 Nicolas Cellier <[hidden email]>: >>>> >>>> It should more or less use the same logic than Parser>>advance >>>> >>>> hereEnd := source position - (aheadChar == DoItCharacter >>>> ifTrue: [hereChar == DoItCharacter >>>> ifTrue: [0] >>>> ifFalse: [1]] >>>> ifFalse: [2]). >>>> >>>> Nicolas >>>> >>>> 2012/8/13 Nicolas Cellier <[hidden email]>: >>>>> >>>>> Yep, this code is highly questionnable: >>>>> >>>>> "Compensate for the fact that the parser uses two character lookahead. >>>>> Normally we must >>>>> remove the extra two characters. But this mustn't >>>>> happen for the >>>>> last token at the end of stream." >>>>> aBlock >>>>> value: mark >>>>> value: (source atEnd >>>>> ifTrue: [tokenType := #doIt. "to >>>>> cause an immediate ^self" source position] >>>>> "ABOVE LINE WILL CAUSE A PREMATURE ^self SINCE THE SCANNER SCANS TWO >>>>> CHARACTER AHEAD" >>>>> ifFalse: [source position - >>>>> 2])]. >>>>> >>>>> Nicolas >>>>> >>>>> 2012/8/13 Levente Uzonyi <[hidden email]>: >>>>>> >>>>>> On Mon, 13 Aug 2012, Colin Putney wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I'd like to have the preference to allow underscore selectors working >>>>>>> in 4.4. To do that, we'll need to fix Character>>tokenish to honor the >>>>>>> preference, ideally without degrading the performance of fileIn and >>>>>>> package loading. I don't remember all the issues that came up when the >>>>>>> preference was introduced, though. Is there anything else that needs >>>>>>> to be done? >>>>>> >>>>>> >>>>>> >>>>>> I started to collect the todos here: >>>>>> >>>>>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-July/164980.html >>>>>> >>>>>> One more thing that needs to be done is to implement a small tool, >>>>>> which can >>>>>> convert code with underscore assignments to ansi assignments. I have a >>>>>> prototype, but it's not fully working due to a bug in Scanner >> >>>>>> #scanTokenPositionsIn:into:. Here's a snippet that demonstrates the >>>>>> issue: >>>>>> >>>>>> Array streamContents: [ :stream | >>>>>> | source scanner | >>>>>> source := 'x := 1. x := 2'. >>>>>> scanner := Scanner new. >>>>>> scanner scanTokenPositionsIn: source into: [ :start :end | >>>>>> stream nextPut: { start. end. source copyFrom: start >>>>>> to: >>>>>> end. scanner instVarNamed: #tokenType } ] ] >>>>>> >>>>>> The first assignment x := 1 is tokenized properly to #word, #leftArrow >>>>>> and >>>>>> #number, but the second one is not. The last 2 tokens are combined into >>>>>> a >>>>>> #doIt, so the code cannot find fix last assignment in a method. >>>>>> >>>>>> >>>>>> Levente >>>>>> >>>>>>> >>>>>>> Colin >>>>>>> >>>>>>> >>>>>> >>> >>> >> > > |
«
Return to Squeak - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |