The Trunk: Multilingual-ul.115.mcz

Levente Uzonyi uploaded a new version of Multilingual to project The Trunk:
http://source.squeak.org/trunk/Multilingual-ul.115.mcz

==================== Summary ====================

Name: Multilingual-ul.115
Author: ul
Time: 27 March 2010, 9:30:44.52 pm
UUID: efcad9da-606e-ea4b-9cb8-a19d6e18a164
Ancestors: Multilingual-ul.114

- move separator skip methods to the TextConverters

=============== Diff against Multilingual-ul.114 ===============

Item was added:
+ ----- Method: TextConverter>>skipSeparatorsAndPeekNextFrom: (in category 'fileIn/Out') -----
+ skipSeparatorsAndPeekNextFrom: aStream
+ "Same as #skipSeparators, but returns the next character after the separators if such exists."
+
+ | state character |
+ [
+ state := self saveStateOf: aStream.
+ (character := aStream next)
+ ifNil: [ false ]
+ ifNotNil: [ character isSeparator ] ] whileTrue.
+ character ifNotNil: [
+ self restoreStateOf: aStream with: state.
+ ^character ].
+ ^aStream
+ !

Item was changed:
  ----- Method: UTF8TextConverter>>nextChunkFromStream: (in category 'fileIn/Out') -----
  nextChunkFromStream: input
  "Answer the contents of input, up to the next terminator character. Doubled terminators indicate an embedded terminator character."
 
+ self skipSeparatorsFrom: input.
- input skipSeparators.
  ^self
  parseLangTagFor: (
+ self class decodeByteString: (
+ String new: 1000 streamContents: [ :stream |
+ [
+ stream nextPutAll: (input basicUpTo: $!!).
+ input basicNext == $!! ]
+ whileTrue: [
+ stream nextPut: $!! ].
+ input atEnd ifFalse: [ input skip: -1 ] ]))
- String new: 1000 streamContents: [ :stream |
- [
- stream nextPutAll: (input basicUpTo: $!!).
- input basicNext == $!! ]
- whileTrue: [
- stream nextPut: $!! ].
- input atEnd ifFalse: [ input skip: -1 ] ]) utf8ToSqueak
  fromStream: input!

Item was added:
+ ----- Method: UTF8TextConverter>>skipSeparatorsAndPeekNextFrom: (in category 'fileIn/Out') -----
+ skipSeparatorsAndPeekNextFrom: aStream
+ "Same as #skipSeparators, but returns the next character after the separators if such exists."
+
+ | character |
+ [
+ ((character := aStream basicNext)
+ ifNil: [ ^aStream "backwards compatibility, should be nil" ])
+ isSeparator ] whileTrue.
+ aStream skip: -1.
+ ^character
+ !

Item was changed:
  ----- Method: MultiByteFileStream>>skipSeparatorsAndPeekNext (in category 'public') -----
  skipSeparatorsAndPeekNext
+ "A special function to make nextChunk fast. Same as #skipSeparators, but returns the next character after the separators if such exists."
+
+ ^converter skipSeparatorsAndPeekNextFrom: self!
- "Same as #skipSeparators, but returns the next character after the separators if such exists."
-
- | state character |
- [
- state := converter saveStateOf: self.
- (character := self next)
- ifNil: [ false ]
- ifNotNil: [ character isSeparator ] ] whileTrue.
- character ifNotNil: [
- converter restoreStateOf: self with: state.
- ^character ].
- !

Item was added:
+ ----- Method: TextConverter>>skipSeparatorsFrom: (in category 'fileIn/Out') -----
+ skipSeparatorsFrom: aStream
+
+ | state character |
+ [
+ state := self saveStateOf: aStream.
+ (character := aStream next)
+ ifNil: [ false ]
+ ifNotNil: [ character isSeparator ] ] whileTrue.
+ character ifNotNil: [
+ self restoreStateOf: aStream with: state ]
+ !

Item was added:
+ ----- Method: UTF8TextConverter>>skipSeparatorsFrom: (in category 'fileIn/Out') -----
+ skipSeparatorsFrom: aStream
+
+ [ (aStream basicNext ifNil: [ ^self ]) isSeparator ] whileTrue.
+ aStream skip: -1!

Item was changed:
  ----- Method: MultiByteFileStream>>skipSeparators (in category 'public') -----
  skipSeparators
 
+ converter skipSeparatorsFrom: self!
- | state character |
- [
- state := converter saveStateOf: self.
- (character := self next)
- ifNil: [ false ]
- ifNotNil: [ character isSeparator ] ] whileTrue.
- character ifNotNil: [
- converter restoreStateOf: self with: state ]!

Item was changed:
  ----- Method: UTF8TextConverter>>parseLangTagFor:fromStream: (in category 'fileIn/Out') -----
  parseLangTagFor: aString fromStream: stream
 
+ | position |
+ position := stream position.
- | state |
- state := self saveStateOf: stream.
  "Test for ]lang[ tag"
+ ((self skipSeparatorsAndPeekNextFrom: stream) == $] and: [
- (stream skipSeparatorsAndPeekNext == $] and: [
  (stream basicNext: 6) = ']lang[' ]) ifTrue: [
  ^stream
  decodeString: aString
  andRuns: (self nextChunkFromStream: stream) ].
  "no tag"
+ stream position: position.
- self restoreStateOf: stream with: state.
  ^aString!

Item was changed:
  ----- Method: UTF8TextConverter>>nextChunkTextFromStream: (in category 'fileIn/Out') -----
  nextChunkTextFromStream: input
  "Deliver the next chunk as a Text.  Decode the following ]style[ chunk if present.  Position at start of next real chunk."
 
+ | chunk position runs |
- | chunk state runs |
  chunk := self nextChunkFromStream: input.
+ position := input position.
+ ((self skipSeparatorsAndPeekNextFrom: input) == $] and: [
- state := self saveStateOf: input.
- (input skipSeparatorsAndPeekNext == $] and: [
  (input basicNext: 7) = ']style[' ])
  ifTrue: [
  runs := RunArray scanFrom: (self nextChunkFromStream: input) readStream ]
  ifFalse: [
+ input position: position.
- self restoreStateOf: input with: state.
  runs := RunArray new: chunk size withAll: #() ].
  ^Text string: chunk runs: runs!