String>>#base64Encoded

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

String>>#base64Encoded

Sven Van Caekenberghe
Hi,

It took me quite some time today to figure this out and it might be relevant for others too:

it turns out that String>>#base64Encoded introduces newlines which you definitively do not want when doing Basic HTTP Encoding for example. The tricky part is that it works when you stay below a certain length.

This can easily be solved by invoking Base64MimeConvertor with the #mimeEncode: <string> multiLine: false

Zinc HTTP Components now does this consistently through ZnUtils class>>#encodeBase64:

Sven


Reply | Threaded
Open this post in threaded view
|

Re: String>>#base64Encoded

Stéphane Ducasse

On Mar 21, 2011, at 9:42 PM, Sven Van Caekenberghe wrote:

> Hi,
>
> It took me quite some time today to figure this out and it might be relevant for others too:
>
> it turns out that String>>#base64Encoded introduces newlines which you definitively do not want when doing Basic HTTP Encoding for example. The tricky part is that it works when you stay below a certain length.

when would it makes sense?

>
> This can easily be solved by invoking Base64MimeConvertor with the #mimeEncode: <string> multiLine: false
>
> Zinc HTTP Components now does this consistently through ZnUtils class>>#encodeBase64:
>
> Sven
>
>


Reply | Threaded
Open this post in threaded view
|

Re: String>>#base64Encoded

Sven Van Caekenberghe

On 21 Mar 2011, at 21:52, Stéphane Ducasse wrote:

> On Mar 21, 2011, at 9:42 PM, Sven Van Caekenberghe wrote:
>
>> Hi,
>>
>> It took me quite some time today to figure this out and it might be relevant for others too:
>>
>> it turns out that String>>#base64Encoded introduces newlines which you definitively do not want when doing Basic HTTP Encoding for example. The tricky part is that it works when you stay below a certain length.
>
> when would it makes sense?

Base64 encoding ( http://en.wikipedia.org/wiki/Base64 ) is a techique to encode binary data in ASCII strings using a radix 64 representation. It came from the early days of the internet where is was used to encode binary stuff in mail messages (part of MIME encoding). For these usages, line breaks make sense, since some email client break lines themselves, and by preformatting you prevent that, for example:

(Base64MimeConverter
        mimeEncode: (ZnClient get: 'http://www.pharo-project.org/favicon.ico') entity readStream
        multiLine: true) contents
       
'AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAABMLAAATCwAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA0K1+/+2tF//lpRj/0J1F/86x
kP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADQso7/
1Zw1/+ekF//koBf/zZdF/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAADLrIr/yI05/9SLG//UvKP/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAA3s2//8qrjf8AAAAAAAAAAAAAAADClmr/28m4/wAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACpayz/rH1S/9K9q/8AAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAxKeP/4lWLv9+Tiz/oXtc/97P
xf8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADYxrj/
oX1j/2RBNP+QbFT/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAC6m4H/1MGz/+7m4v/Ltqb/tJR3/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAsJWF/1tHVf+qkIH/4tTL/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAANG8rf9GPGj/IiJn/19MYP/Mt6n/AAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANXEuP9NQGj/IiiB/youf/8NFnb/Qztx/9PC
t/8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADczcL/mIGA/y8ziP8nMZj/
GSGF/4l2hP/Tw7n/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMy4
q/8pMZj/JzKi/yUxov+znJb/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAADezsT/d3Om/1dt0f9cZrz/v6mf/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAJqapP+i3f//iazZ/9/Pwv8AAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADBqIv/lYps/7Caev8AAAAAAAAAAAAA
AAAAAAAA/g8AAP8H8L//wwAA/nMAAP8fNcL/BwAA/4cAAP8HAAD/DwAA/wc4FP4DAAD+AwAA
/wf///8H////hzp0/48AAA=='

Base64 encoding is also used a lot in cryptography to encode binary data like hashes or even for silly obfuscation. For example, in HTTP, Basic Authentication is a defined as a HTTP header called 'Authorization' with as value 'Basic' <space> <credentials> where credentials is obtained by Base64 encoding <username>:<password>.

ZnRequest>>#setBasicAuthenticationUsername: username password: password
        self headers
                at: 'Authorization'
                put: 'Basic ', (ZnUtils encodeBase64: (username, ':', password))

ZnUtils class>>#encodeBase64: string
        ^ (Base64MimeConverter mimeEncode: string readStream multiLine: false) contents

Here you don't want line breaks since it would break the common assumption that a header is a single line (technically, multiline headers do exist, but that is another story).

Now, as long as the input string is below 55 characters in length, no line breaks will be introduced. In code (assuming no line line breaks will be introduced by our email clients or the mailing list software inbetween; re-execute in a workspace if necessary):

(Base64MimeConverter
        mimeEncode: ((String new: 54) atAllPut: $A) readStream
        multiLine: true) contents
       
 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFB'

(Base64MimeConverter
        mimeEncode: ((String new: 54) atAllPut: $A) readStream
        multiLine: false) contents
       
 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFB'

(Base64MimeConverter
        mimeEncode: ((String new: 55) atAllPut: $A) readStream
        multiLine: true) contents

 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFB
QQ=='
       
(Base64MimeConverter
        mimeEncode: ((String new: 55) atAllPut: $A) readStream
        multiLine: false) contents
       
 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQQ=='

Please note, this is not a bug, just a gotcha that can go unnoticed for quite some time.

Sven

PS: Base64 decoding is clever enough to transparently deal with line breaks.

>> This can easily be solved by invoking Base64MimeConvertor with the #mimeEncode: <string> multiLine: false
>>
>> Zinc HTTP Components now does this consistently through ZnUtils class>>#encodeBase64:
>>
>> Sven


Reply | Threaded
Open this post in threaded view
|

Re: String>>#base64Encoded

Stéphane Ducasse
Thanks

may be we should record this nice explanation in the help or something like that.


On Mar 22, 2011, at 7:45 AM, Sven Van Caekenberghe wrote:

>
> On 21 Mar 2011, at 21:52, Stéphane Ducasse wrote:
>
>> On Mar 21, 2011, at 9:42 PM, Sven Van Caekenberghe wrote:
>>
>>> Hi,
>>>
>>> It took me quite some time today to figure this out and it might be relevant for others too:
>>>
>>> it turns out that String>>#base64Encoded introduces newlines which you definitively do not want when doing Basic HTTP Encoding for example. The tricky part is that it works when you stay below a certain length.
>>
>> when would it makes sense?
>
> Base64 encoding ( http://en.wikipedia.org/wiki/Base64 ) is a techique to encode binary data in ASCII strings using a radix 64 representation. It came from the early days of the internet where is was used to encode binary stuff in mail messages (part of MIME encoding). For these usages, line breaks make sense, since some email client break lines themselves, and by preformatting you prevent that, for example:
>
> (Base64MimeConverter
> mimeEncode: (ZnClient get: 'http://www.pharo-project.org/favicon.ico') entity readStream
> multiLine: true) contents
>
> 'AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAABMLAAATCwAA
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA0K1+/+2tF//lpRj/0J1F/86x
> kP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADQso7/
> 1Zw1/+ekF//koBf/zZdF/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> AAAAAAAAAAAAAAAAAADLrIr/yI05/9SLG//UvKP/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> AAAAAAAAAAAAAAAA3s2//8qrjf8AAAAAAAAAAAAAAADClmr/28m4/wAAAAAAAAAAAAAAAAAA
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACpayz/rH1S/9K9q/8AAAAAAAAAAAAAAAAAAAAA
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAxKeP/4lWLv9+Tiz/oXtc/97P
> xf8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADYxrj/
> oX1j/2RBNP+QbFT/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> AAC6m4H/1MGz/+7m4v/Ltqb/tJR3/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> AAAAAAAAAAAAAAAAsJWF/1tHVf+qkIH/4tTL/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAANG8rf9GPGj/IiJn/19MYP/Mt6n/AAAAAAAAAAAAAAAA
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANXEuP9NQGj/IiiB/youf/8NFnb/Qztx/9PC
> t/8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADczcL/mIGA/y8ziP8nMZj/
> GSGF/4l2hP/Tw7n/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMy4
> q/8pMZj/JzKi/yUxov+znJb/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> AAAAAAAAAADezsT/d3Om/1dt0f9cZrz/v6mf/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAJqapP+i3f//iazZ/9/Pwv8AAAAAAAAAAAAAAAAAAAAA
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADBqIv/lYps/7Caev8AAAAAAAAAAAAA
> AAAAAAAA/g8AAP8H8L//wwAA/nMAAP8fNcL/BwAA/4cAAP8HAAD/DwAA/wc4FP4DAAD+AwAA
> /wf///8H////hzp0/48AAA=='
>
> Base64 encoding is also used a lot in cryptography to encode binary data like hashes or even for silly obfuscation. For example, in HTTP, Basic Authentication is a defined as a HTTP header called 'Authorization' with as value 'Basic' <space> <credentials> where credentials is obtained by Base64 encoding <username>:<password>.
>
> ZnRequest>>#setBasicAuthenticationUsername: username password: password
> self headers
> at: 'Authorization'
> put: 'Basic ', (ZnUtils encodeBase64: (username, ':', password))
>
> ZnUtils class>>#encodeBase64: string
> ^ (Base64MimeConverter mimeEncode: string readStream multiLine: false) contents
>
> Here you don't want line breaks since it would break the common assumption that a header is a single line (technically, multiline headers do exist, but that is another story).
>
> Now, as long as the input string is below 55 characters in length, no line breaks will be introduced. In code (assuming no line line breaks will be introduced by our email clients or the mailing list software inbetween; re-execute in a workspace if necessary):
>
> (Base64MimeConverter
> mimeEncode: ((String new: 54) atAllPut: $A) readStream
> multiLine: true) contents
>
> 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFB'
>
> (Base64MimeConverter
> mimeEncode: ((String new: 54) atAllPut: $A) readStream
> multiLine: false) contents
>
> 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFB'
>
> (Base64MimeConverter
> mimeEncode: ((String new: 55) atAllPut: $A) readStream
> multiLine: true) contents
>
> 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFB
> QQ=='
>
> (Base64MimeConverter
> mimeEncode: ((String new: 55) atAllPut: $A) readStream
> multiLine: false) contents
>
> 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQQ=='
>
> Please note, this is not a bug, just a gotcha that can go unnoticed for quite some time.
>
> Sven
>
> PS: Base64 decoding is clever enough to transparently deal with line breaks.
>
>>> This can easily be solved by invoking Base64MimeConvertor with the #mimeEncode: <string> multiLine: false
>>>
>>> Zinc HTTP Components now does this consistently through ZnUtils class>>#encodeBase64:
>>>
>>> Sven
>
>


Reply | Threaded
Open this post in threaded view
|

Re: String>>#base64Encoded

tfleig
In reply to this post by Sven Van Caekenberghe
In other language environments I have seen base64 encoders where the line break is optional, or where a maximum line length can be specified.

In Smalltalk it might look line:

String>>#base64Encoded

String>>#base64EncodedWithLineBreaks
or
String>>#base64EncodedMaxLineLength:

TF

On Mon, Mar 21, 2011 at 11:45 PM, Sven Van Caekenberghe <[hidden email]> wrote:

On 21 Mar 2011, at 21:52, Stéphane Ducasse wrote:

> On Mar 21, 2011, at 9:42 PM, Sven Van Caekenberghe wrote:
>
>> Hi,
>>
>> It took me quite some time today to figure this out and it might be relevant for others too:
>>
>> it turns out that String>>#base64Encoded introduces newlines which you definitively do not want when doing Basic HTTP Encoding for example. The tricky part is that it works when you stay below a certain length.
>
> when would it makes sense?

Base64 encoding ( http://en.wikipedia.org/wiki/Base64 ) is a techique to encode binary data in ASCII strings using a radix 64 representation. It came from the early days of the internet where is was used to encode binary stuff in mail messages (part of MIME encoding). For these usages, line breaks make sense, since some email client break lines themselves, and by preformatting you prevent that, for example:

(Base64MimeConverter
       mimeEncode: (ZnClient get: 'http://www.pharo-project.org/favicon.ico') entity readStream
       multiLine: true) contents

'AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAABMLAAATCwAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA0K1+/+2tF//lpRj/0J1F/86x
kP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADQso7/
1Zw1/+ekF//koBf/zZdF/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAADLrIr/yI05/9SLG//UvKP/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAA3s2//8qrjf8AAAAAAAAAAAAAAADClmr/28m4/wAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACpayz/rH1S/9K9q/8AAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAxKeP/4lWLv9+Tiz/oXtc/97P
xf8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADYxrj/
oX1j/2RBNP+QbFT/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAC6m4H/1MGz/+7m4v/Ltqb/tJR3/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAsJWF/1tHVf+qkIH/4tTL/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAANG8rf9GPGj/IiJn/19MYP/Mt6n/AAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANXEuP9NQGj/IiiB/youf/8NFnb/Qztx/9PC
t/8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADczcL/mIGA/y8ziP8nMZj/
GSGF/4l2hP/Tw7n/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMy4
q/8pMZj/JzKi/yUxov+znJb/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAADezsT/d3Om/1dt0f9cZrz/v6mf/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAJqapP+i3f//iazZ/9/Pwv8AAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADBqIv/lYps/7Caev8AAAAAAAAAAAAA
AAAAAAAA/g8AAP8H8L//wwAA/nMAAP8fNcL/BwAA/4cAAP8HAAD/DwAA/wc4FP4DAAD+AwAA
/wf///8H////hzp0/48AAA=='

Base64 encoding is also used a lot in cryptography to encode binary data like hashes or even for silly obfuscation. For example, in HTTP, Basic Authentication is a defined as a HTTP header called 'Authorization' with as value 'Basic' <space> <credentials> where credentials is obtained by Base64 encoding <username>:<password>.

ZnRequest>>#setBasicAuthenticationUsername: username password: password
       self headers
               at: 'Authorization'
               put: 'Basic ', (ZnUtils encodeBase64: (username, ':', password))

ZnUtils class>>#encodeBase64: string
       ^ (Base64MimeConverter mimeEncode: string readStream multiLine: false) contents

Here you don't want line breaks since it would break the common assumption that a header is a single line (technically, multiline headers do exist, but that is another story).

Now, as long as the input string is below 55 characters in length, no line breaks will be introduced. In code (assuming no line line breaks will be introduced by our email clients or the mailing list software inbetween; re-execute in a workspace if necessary):

(Base64MimeConverter
       mimeEncode: ((String new: 54) atAllPut: $A) readStream
       multiLine: true) contents

 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFB'

(Base64MimeConverter
       mimeEncode: ((String new: 54) atAllPut: $A) readStream
       multiLine: false) contents

 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFB'

(Base64MimeConverter
       mimeEncode: ((String new: 55) atAllPut: $A) readStream
       multiLine: true) contents

 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFB
QQ=='

(Base64MimeConverter
       mimeEncode: ((String new: 55) atAllPut: $A) readStream
       multiLine: false) contents

 'QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQQ=='

Please note, this is not a bug, just a gotcha that can go unnoticed for quite some time.

Sven

PS: Base64 decoding is clever enough to transparently deal with line breaks.

>> This can easily be solved by invoking Base64MimeConvertor with the #mimeEncode: <string> multiLine: false
>>
>> Zinc HTTP Components now does this consistently through ZnUtils class>>#encodeBase64:
>>
>> Sven