The Trunk: Multilingual-ul.238.mcz

Levente Uzonyi uploaded a new version of Multilingual to project The Trunk:
http://source.squeak.org/trunk/Multilingual-ul.238.mcz

==================== Summary ====================

Name: Multilingual-ul.238
Author: ul
Time: 13 June 2018, 5:07:40.804457 pm
UUID: 8f89eae0-1eca-48c6-a3f7-ebe13afa2dbe
Ancestors: Multilingual-nice.237

Changes to various TextConverters:
#nextPut:toStream: can not be used to initialize latin1Map and latin1Encodings anymore (because most implementations reference them), so
 - moved the current implementation of #initializeLatin1MapAndEncodings to ByteTextConverter, because only those converters understand #encode:.
 - TextConverter >> #initializeLatin1MapAndEncodings has become subclass responsibility
 - implemented #initializeLatin1MapAndEncodings in UTF* text converters
 - removed class-side #initialize implementations. Will add a new one to TextConverter once all subclasses can reinitialize their tables.
 - added class side #initializeTables which is expected to initialize the various class side tables in the correct order
 - postscript will reinitialize class side tables where possible (all converters except for  EUCTextConverter, CNGBTextConverter, EUCJPTextConverter, EUCKRTextConverter, ShiftJISTextConverter, CompoundTextConverter, KOI8RTextConverter)
 - latin1Map and latin1Encodings are not initialized lazily anymore

UTF16TextConverter will take endianness into account when writing the BOM. Same thing is probably missing from UTF32TextConverter too.

=============== Diff against Multilingual-nice.237 ===============

Item was removed:
- ----- Method: ByteTextConverter class>>initialize (in category 'class initialization') -----
- initialize
-
-       self == ByteTextConverter
- ifTrue: [self allSubclassesDo: [:c | c initialize]]
- ifFalse: [self
- initializeDecodeTable;
- initializeEncodeTable;
- initializeLatin1MapAndEncodings]
- !

Item was added:
+ ----- Method: ByteTextConverter class>>initializeLatin1MapAndEncodings (in category 'class initialization') -----
+ initializeLatin1MapAndEncodings
+ "Initialize the latin1Map and latin1Encodings.
+ These variables ensure that conversions from latin1 ByteString is reasonably fast.
+ This implementation assumes that encodeTable is initialized."
+
+ latin1Map := ByteArray new: 256.
+ latin1Encodings := Array new: 256.
+ 0 to: 255 do:[:i |
+ | latin1 encoded |
+ latin1 := String with: (Character value: i).
+ [encoded := String with: (Character value: (self new encode: latin1 first charCode))]
+ ifError: [].
+ latin1 = encoded ifTrue:[
+ latin1Map at: i+1 put: 0. "no translation needed"
+ ] ifFalse: [
+ latin1Map at: i+1 put: 1. "translation needed"
+ latin1Encodings at: i+1 put: encoded.
+ ]].!

Item was added:
+ ----- Method: ByteTextConverter class>>initializeTables (in category 'class initialization') -----
+ initializeTables
+
+       self == ByteTextConverter ifTrue: [ ^self ].
+ self
+ initializeDecodeTable;
+ initializeEncodeTable;
+ initializeLatin1MapAndEncodings
+ !

Item was changed:
+ ----- Method: TextConverter class>>initializeLatin1MapAndEncodings (in category 'class initialization') -----
- ----- Method: TextConverter class>>initializeLatin1MapAndEncodings (in category 'accessing') -----
  initializeLatin1MapAndEncodings
  "Initialize the latin1Map and latin1Encodings.
+ These variables ensure that conversions from latin1 ByteString is reasonably fast."
- These variables ensure that conversions from latin1 ByteString is reasonably fast"
 
+ self subclassResponsibility
+
+ !
- latin1Map := ByteArray new: 256.
- latin1Encodings := Array new: 256.
- 0 to: 255 do:[:i |
- | latin1 encoded |
- latin1 := String with: (Character value: i).
- [encoded := String with: (Character value: (self new encode: latin1 first charCode))]
- ifError: [].
- latin1 = encoded ifTrue:[
- latin1Map at: i+1 put: 0. "no translation needed"
- ] ifFalse: [
- latin1Map at: i+1 put: 1. "translation needed"
- latin1Encodings at: i+1 put: encoded.
- ]].!

Item was added:
+ ----- Method: TextConverter class>>initializeTables (in category 'class initialization') -----
+ initializeTables
+
+ self initializeLatin1MapAndEncodings!

Item was changed:
  ----- Method: TextConverter class>>latin1Encodings (in category 'accessing') -----
  latin1Encodings
  "Answer an Array mapping latin1 characters to conversion string"
 
+ ^latin1Encodings!
- ^latin1Encodings ifNil:
- [self initializeLatin1MapAndEncodings.
- latin1Encodings]!

Item was changed:
  ----- Method: TextConverter class>>latin1Map (in category 'accessing') -----
  latin1Map
  "Answer a ByteArray map telling if latin1 characters needs conversion or not"
 
+ ^latin1Map!
- ^latin1Map ifNil:
- [self initializeLatin1MapAndEncodings.
- latin1Map]!

Item was added:
+ ----- Method: UTF16TextConverter class>>initializeLatin1MapAndEncodings (in category 'accessing') -----
+ initializeLatin1MapAndEncodings
+ "Initialize the latin1Map and latin1Encodings.
+ These variables ensure that conversions from latin1 ByteString is reasonably fast."
+
+ latin1Map := ByteArray new: 256 withAll: 1.
+ latin1Encodings := (0 to: 255) collect: [ :i | { 0. i } asByteArray asString ]!

Item was changed:
  ----- Method: UTF16TextConverter>>nextPut:toStream: (in category 'conversion') -----
  nextPut: aCharacter toStream: aStream
 
  | charCode |
  aStream isBinary ifTrue: [ ^aCharacter storeBinaryOn: aStream ].
  (useByteOrderMark and: [ byteOrderMarkDone not ]) ifTrue: [
+ self next16BitValue: (useLittleEndian ifTrue: [ 16rFFFE ] ifFalse: [ 16rFEFF ]) toStream: aStream.
- self next16BitValue: 16rFEFF toStream: aStream.
  byteOrderMarkDone := true ].
  (charCode := aCharacter charCode) < 256
  ifTrue: [
  (latin1Encodings at: charCode + 1)
  ifNil: [ self next16BitValue: charCode toStream: aStream ]
  ifNotNil: [ :encodedString | aStream basicNextPutAll: encodedString ] ]
  ifFalse: [
  charCode <= 16rFFFF
  ifTrue: [ self next16BitValue: charCode toStream: aStream ]
  ifFalse: [
  | low high |
  charCode := charCode - 16r10000.
  low := charCode \\ 16r400 + 16rDC00.
  high := charCode // 16r400 + 16rD800.
  self
  next16BitValue: high toStream: aStream;
  next16BitValue: low toStream: aStream ] ].
  ^aCharacter!

Item was changed:
  ----- Method: UTF32TextConverter class>>initializeLatin1MapAndEncodings (in category 'utilities') -----
  initializeLatin1MapAndEncodings
  "Initialize the latin1Map and latin1Encodings.
+ These variables ensure that conversions from latin1 ByteString is reasonably fast."
- These variables ensure that conversions from latin1 ByteString is reasonably fast"
 
+ latin1Map := ByteArray new: 256 withAll: 1.
+ latin1Encodings := (0 to: 255) collect: [ :i | { 0. 0. 0. i } asByteArray asString ]
+ !
- latin1Map := (ByteArray new: 256) atAllPut: 1.
- latin1Encodings := (0 to: 255) collect: [:i | (ByteArray newFrom: {0 . 0 . 0 . i}) asString]!

Item was added:
+ ----- Method: UTF8TextConverter class>>initializeLatin1MapAndEncodings (in category 'as yet unclassified') -----
+ initializeLatin1MapAndEncodings
+ "Initialize the latin1Map and latin1Encodings. These variables ensure that conversions from latin1 ByteString is reasonably fast."
+
+ latin1Map := (0 to: 255) collect: [ :each | each bitShift: -7 ] as: ByteArray.
+ latin1Encodings := (0 to: 255) collect: [ :each |
+ each <= 127
+ ifTrue: [ nil ]
+ ifFalse: [
+ { 192 bitOr: (each bitShift: -6). (each bitAnd: 63) bitOr: 128 } asByteArray asString ] ]!

Item was changed:
+ (PackageInfo named: 'Multilingual') postscript: '"Initialize the tables in all TextConverters that support it."
+ TextConverter allSubclassesDo: [ :each | [ each initializeTables ] ifError: [] ]'!
- (PackageInfo named: 'Multilingual') postscript: 'Unicode
- initializeCaseFolding;
- initializeCompositionMappings'!