The Trunk: Compiler-tobe.455.mcz

Marcel Taeumel uploaded a new version of Compiler to project The Trunk:
http://source.squeak.org/trunk/Compiler-tobe.455.mcz

==================== Summary ====================

Name: Compiler-tobe.455
Author: tobe
Time: 11 March 2021, 1:50:12.615385 pm
UUID: ba3e2f61-8c2a-4dcf-a288-f6e81db4eda1
Ancestors: Compiler-codefrau.454

Fix scanAllTokenPositionsInto: when the text of a comment happened to appear just before it. See testScanTokensRepeatComment for an example.

=============== Diff against Compiler-codefrau.454 ===============

Item was changed:
  ----- Method: Scanner>>scanAllTokenPositionsInto: (in category 'expression types') -----
  scanAllTokenPositionsInto: aBlock
  "Evaluate aBlock with the start and end positions of all separate non-white-space tokens, including comments."
 
  | lastMark |
  lastMark := 1.
  [currentComment ifNotNil:
  [currentComment do:
  [:cmnt| | idx |
+ idx := source originalContents findLastOccurrenceOfString: cmnt startingAt: lastMark.
- idx := source originalContents indexOfSubCollection: cmnt startingAt: lastMark.
  (idx > 0 and: [idx < mark]) ifTrue:
  [aBlock value: idx - 1 value: (lastMark := idx + cmnt size)]].
  currentComment := nil].
  mark ifNotNil:
  [(token == #-
   and: [(self typeTableAt: hereChar) == #xDigit]) ifTrue:
  [| savedMark |
  savedMark := mark.
  self scanToken.
  token := token negated.
  mark := savedMark].
  "Compensate for the fact that the parser uses two character lookahead.  Normally we must
   remove the extra two characters.  But this mustn't happen for the last token at the end of stream."
  aBlock
  value: mark
  value: source position - (aheadChar == DoItCharacter
  ifTrue: [hereChar == DoItCharacter
  ifTrue: [0]
  ifFalse: [1]]
  ifFalse: [2])].
  (tokenType == #rightParenthesis
   or: [tokenType == #doIt]) ifTrue:
  [^self].
  tokenType == #leftParenthesis
  ifTrue:
  [self scanToken; scanAllTokenPositionsInto: aBlock]
  ifFalse:
  [(tokenType == #word or: [tokenType == #keyword or: [tokenType == #colon]])
  ifTrue:
  [self scanLitWord.
  token == #true ifTrue: [token := true].
  token == #false ifTrue: [token := false].
  token == #nil ifTrue: [token := nil]]
  ifFalse:
  [(token == #-
   and: [(self typeTableAt: hereChar) == #xDigit])
  ifTrue:
  [self scanToken.
  token := token negated]]].
  self scanToken ] repeat!