What is with the UTF7 encoder?

Tamara Kogan

Smalltalk Development

Cincom Systems

-Boris

Tamara Kogan

Smalltalk Development

Cincom Systems

Tamara Kogan

Smalltalk Development

Cincom Systems

-Boris

Tamara Kogan

Smalltalk Development

Cincom Systems

-Boris

Tamara Kogan

Smalltalk Development

Cincom Systems

-Boris

Tamara Kogan

Smalltalk Development

Cincom Systems

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Here is an example from our SUnit tests:

                str := ('testöäü' asByteArrayEncoding: #'utf_7') asByteString.

                stream := (ByteArray new withEncoding: #UTF_7) writeStream.

                stream nextPutAll: 'testöäü'.

                stream close.                                                                                    

                self assert: str = (stream encodedContents withEncoding: #ASCII) readStream contents

I need a utf 7 encoder/decoder, so I loaded NetClientBase, tried the following

and got an exception;

('some text'  asByteArrayEncoding: #utf_7) asStringEncoding: #utf_7

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Here is an example from our SUnit tests:

                str := ('testöäü' asByteArrayEncoding: #'utf_7') asByteString.

                stream := (ByteArray new withEncoding: #UTF_7) writeStream.

                stream nextPutAll: 'testöäü'.

                stream close.                                                                                    

                self assert: str = (stream encodedContents withEncoding: #ASCII) readStream contents

I need a utf 7 encoder/decoder, so I loaded NetClientBase, tried the following

and got an exception;

('some text'  asByteArrayEncoding: #utf_7) asStringEncoding: #utf_7

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Tamara Kogan

Smalltalk Development

Cincom Systems

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Here is an example from our SUnit tests:

                str := ('testöäü' asByteArrayEncoding: #'utf_7') asByteString.

                stream := (ByteArray new withEncoding: #UTF_7) writeStream.

                stream nextPutAll: 'testöäü'.

                stream close.                                                                                    

                self assert: str = (stream encodedContents withEncoding: #ASCII) readStream contents

I need a utf 7 encoder/decoder, so I loaded NetClientBase, tried the following

and got an exception;

('some text'  asByteArrayEncoding: #utf_7) asStringEncoding: #utf_7

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

I guess because since 1999 the method String class>>fromIntegerArray: encoding: has been implemented to decode in to ByteString not ByteArray.

That doesn’t quite answer the question of why Terry can’t just go to bytes and back with an identical encoder?

Here is an example from our SUnit tests:

                str := ('testöäü' asByteArrayEncoding: #'utf_7') asByteString.

                stream := (ByteArray new withEncoding: #UTF_7) writeStream.

                stream nextPutAll: 'testöäü'.

                stream close.                                                                                    

                self assert: str = (stream encodedContents withEncoding: #ASCII) readStream contents

I need a utf 7 encoder/decoder, so I loaded NetClientBase, tried the following

and got an exception;

('some text'  asByteArrayEncoding: #utf_7) asStringEncoding: #utf_7

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Tamara Kogan

Smalltalk Development

Cincom Systems

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

I guess because since 1999 the method String class>>fromIntegerArray: encoding: has been implemented to decode in to ByteString not ByteArray.

That doesn’t quite answer the question of why Terry can’t just go to bytes and back with an identical encoder?

Here is an example from our SUnit tests:

                str := ('testöäü' asByteArrayEncoding: #'utf_7') asByteString.

                stream := (ByteArray new withEncoding: #UTF_7) writeStream.

                stream nextPutAll: 'testöäü'.

                stream close.                                                                                    

                self assert: str = (stream encodedContents withEncoding: #ASCII) readStream contents

I need a utf 7 encoder/decoder, so I loaded NetClientBase, tried the following

and got an exception;

('some text'  asByteArrayEncoding: #utf_7) asStringEncoding: #utf_7

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Terry,

UTF8 encoder was reworked completely since it became the main internet encoding.  Since no one complained (or used )  utf7 encoding the encoder was neglected.

I will create an AR but it would be interesting to know why do you need utf7. It will help to define the AR priority.

Tamara

These are the bugs I found in UTF7StreamEncoder

1. UTF7StreamEnoder>>nextFrom:  answers the numerical code, whereas the class protocol expects it to answer the character.

2. UTF7StreamEncoder>>isNotASCII:  treats character codes greater than 255 as ASCII, that does not work.

3. UTF7StremEncoder>>fillNibbleFrom:

  the line

                (aStream atEnd or: [aStream peek == self shiftOutCode]) ifTrue: [ ^nSextets ].

  should be changed so it consumes the shiftOutCode before returning.

                (aStream atEnd or: [aStream peek == self shiftOutCode]) ifTrue: [ aStream next. ^nSextets ].

Georg Heeg eK, Dortmund und Köthen, HR Dortmund A 12812

Wallstraße 22, 06366 Köthen

Tel. +49-3496-214328, Fax +49-3496-214712

-Boris

Tamara Kogan

Smalltalk Development

Cincom Systems

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Terry and Boris,

you are perfectly right.

There are actually quite a few bugs in UTF7StreamEncoder.

1.       Any character > 255 are not encoded correctly, e.g. $€, an exception is thrown during encoding [I do not have a fix for that]

2.       To fix your problem UTF7StreamEncoder>nextFrom: must be fixed. Here is the fix which works for me:

nextFrom: aStream

               "Decode the next byte(s) in the stream and answer the code."

               | code c1 |

               aStream peek isNil ifTrue: [^nil].

               code := aStream peek.

               code = self shiftInCode

                             ifTrue:

                                            [code := aStream next.

                                            (code = self shiftInCode and: [aStream peek = self shiftOutCode])

                                                           ifTrue:

                                                                          [aStream next.

                                                                          ^code asCharacter].

                                            self shifting: true].

               (code = self shiftOutCode and: [self shifting])

                             ifTrue:

                                            [(c1 := self shiftInNextFrom: aStream) notNil ifTrue: [^c1 asCharacter].

                                            self shifting: false.

                                            aStream next.

                                            aStream peek isNil ifTrue: [^nil]].

               ^(self shifting

                             ifTrue: [self shiftInNextFrom: aStream]

                             ifFalse: [aStream next]) asCharacter

3.       If the string ends with a non ASCII character like $ö another exception is thrown. Here is the fix of UTF7StreamEncoder> shiftInNextFrom: which works for me:

shiftInNextFrom: aStream

               | c1 c2 |

               c1 := super nextFrom: aStream.

               c1 == nil ifTrue: [^nil].

               c2 := super nextFrom: aStream.

               c2 == nil ifTrue: [^nil].

               aStream peek = self shiftOutCode

                             ifTrue:

                                            [aStream next.

                                            self shifting: false].

               ^(c1 bitShift: 8) + c2

Have fun

Georg

That doesn’t quite answer the question of why Terry can’t just go to bytes and back with an identical encoder?

Here is an example from our SUnit tests:

                str := ('testöäü' asByteArrayEncoding: #'utf_7') asByteString.

                stream := (ByteArray new withEncoding: #UTF_7) writeStream.

                stream nextPutAll: 'testöäü'.

                stream close.                                                                                    

                self assert: str = (stream encodedContents withEncoding: #ASCII) readStream contents

I need a utf 7 encoder/decoder, so I loaded NetClientBase, tried the following

and got an exception;

('some text'  asByteArrayEncoding: #utf_7) asStringEncoding: #utf_7

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Georg Heeg eK, Dortmund und Köthen, HR Dortmund A 12812

Wallstraße 22, 06366 Köthen

Tel. +49-3496-214328, Fax +49-3496-214712

Terry

===========================================================

Terry Raymond

Crafted Smalltalk

80 Lazywood Ln.

Tiverton, RI  02878

(401) 624-4517      [hidden email]

===========================================================

Terry,

UTF8 encoder was reworked completely since it became the main internet encoding.  Since no one complained (or used )  utf7 encoding the encoder was neglected.

I will create an AR but it would be interesting to know why do you need utf7. It will help to define the AR priority.

Tamara

These are the bugs I found in UTF7StreamEncoder

1. UTF7StreamEnoder>>nextFrom:  answers the numerical code, whereas the class protocol expects it to answer the character.

2. UTF7StreamEncoder>>isNotASCII:  treats character codes greater than 255 as ASCII, that does not work.

3. UTF7StremEncoder>>fillNibbleFrom:

  the line

                (aStream atEnd or: [aStream peek == self shiftOutCode]) ifTrue: [ ^nSextets ].

  should be changed so it consumes the shiftOutCode before returning.

                (aStream atEnd or: [aStream peek == self shiftOutCode]) ifTrue: [ aStream next. ^nSextets ].