The Trunk: Compiler-eem.381.mcz

Eliot Miranda uploaded a new version of Compiler to project The Trunk:
http://source.squeak.org/trunk/Compiler-eem.381.mcz

==================== Summary ====================

Name: Compiler-eem.381
Author: eem
Time: 31 May 2018, 4:59:26.014435 pm
UUID: 6612bdf4-e29a-4cc5-a550-139de9e3c42d
Ancestors: Compiler-eem.380

Fix typedScanTokens: for underscore-prefixed symbols such as #_WIN32.

=============== Diff against Compiler-eem.380 ===============

Item was changed:
  ----- Method: Scanner>>initScannerForTokenization (in category 'initialize-release') -----
  initScannerForTokenization
  "Use a version of typeTable that doesn't raise xIllegal when enocuntering an _"
  | underscoreIndex |
  underscoreIndex := typeTable identityIndexOf: #xUnderscore ifAbsent: [^self].
  typeTable := typeTable copy.
+ typeTable at: underscoreIndex put: #xUnderscoreForTokenization.
+ typeTable at: (typeTable identityIndexOf: #xLitQuote) put: #xLitQuoteForTokenization!
- typeTable at: underscoreIndex put: #xUnderscoreForTokenization!

Item was added:
+ ----- Method: Scanner>>xLitQuoteForTokenization (in category 'multi-character scans') -----
+ xLitQuoteForTokenization
+ "Parse underscores as per the regular parser when following a # symbol, so that e.g. #_WIN32 is correctly tokenized."
+ | index |
+ index :=  typeTable identityIndexOf: #xUnderscoreForTokenization.
+ typeTable at: index put: #xUnderscore.
+ [self xLitQuote] ensure:
+ [typeTable at: index put: #xUnderscoreForTokenization]!

Item was changed:
  ----- Method: Scanner>>xUnderscoreForTokenization (in category 'multi-character scans') -----
  xUnderscoreForTokenization
+ "Parse underscores as assignments so as to be able to correctly tokenize ancient source code."
  self step.
  tokenType := #leftArrow.
  ^token := #'_'!