Smalltalk › Pharo › Pharo Smalltalk Users

ZnEasy / UTF-8

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

10 messages Options

volkert-2

ZnEasy / UTF-8

Dear all,

i am trying to get a JSON Document with ZnEasy. The Document is UTF-8 encoded.

(ZnEasy get: '<a class="moz-txt-link-freetext" href="http://">http://<removed>/api/atm/801813') entity string.

Any hints?

Volkert

Sven Van Caekenberghe-2

Re: ZnEasy / UTF-8

Hi Volkert,

Tip 1: use ZnClient and inspect the response in detail, more specifically to see whether the response content type specifies the encoding.

ZnClient new get: 'http://<removed>/api/atm/801813'; yourself.

Tip 2: you could try to force override the default encoding using ZnDefaultCharacterEncoder (see the class comment).

But recent Zn code defaults to UTF8 (see #withDefaultUTF8Decoding: and #withDefaultUTF8Encoding:). So I am curious to see the response headers.

HTH,

Sven

> On 01 May 2015, at 11:54, volkert <[hidden email]> wrote:
>
> Dear all,
>
> i am trying to get a JSON Document with ZnEasy. The Document is UTF-8 encoded.
>
> (ZnEasy get: 'http://<removed>/api/atm/801813') entity string.
>
> <gccacjeh.png>
>
> Any hints?
>
> Volkert
>
>

volkert-2

Re: ZnEasy / UTF-8

I tried ZnClient new get: '<a class="moz-txt-link-freetext" href="http://">http://<removed>/api/atm/801813'; yourself and got.

This is the Doc i see in Firefox.

The JSON is served with ZnServer / Teapot. ;-)

Volkert

Am 01.05.2015 um 12:21 schrieb Sven Van Caekenberghe:

Hi Volkert,

Tip 1: use ZnClient and inspect the response in detail, more specifically to see whether the response content type specifies the encoding.

  ZnClient new get: '<a class="moz-txt-link-freetext" href="http://">http://<removed>/api/atm/801813'; yourself.

Tip 2: you could try to force override the default encoding using ZnDefaultCharacterEncoder (see the class comment).

But recent Zn code defaults to UTF8 (see #withDefaultUTF8Decoding: and #withDefaultUTF8Encoding:). So I am curious to see the response headers.

HTH,

Sven

On 01 May 2015, at 11:54, volkert [hidden email] wrote:

Dear all,

i am trying to get a JSON Document with ZnEasy. The Document is UTF-8 encoded.

(ZnEasy get: '<a class="moz-txt-link-freetext" href="http://">http://<removed>/api/atm/801813') entity string.
    
<gccacjeh.png>

Any hints?

Volkert

NorbertHartl

Re: ZnEasy / UTF-8

You need to look at the headers of the request. I would assume the sender sets an Content-Type: application/json without charset. In that moment it is uncertain what has to be done because the server needs to assume what charset it might be and this tasks seldomly works out good.

In order to make it work set a content type of your entity this way

ZnMimeType applicationJson setCharSetUTF8

Now you send a Content-Type of application/json;charset=utf-8 and the server knows which character decoder to use. That should be it. So please always assign a char set to your sending data to make it more reliable what happens on the other end.

hope this helps,

Norbert

Am 01.05.2015 um 14:13 schrieb volkert <[hidden email]>:

I tried ZnClient new get: 'http://<removed>/api/atm/801813'; yourself and got.

<fdchdjab.png>

This is the Doc i see in Firefox.

<fceagiec.png>

The JSON is served with ZnServer / Teapot. ;-)

Volkert

Am 01.05.2015 um 12:21 schrieb Sven Van Caekenberghe:

Hi Volkert,

Tip 1: use ZnClient and inspect the response in detail, more specifically to see whether the response content type specifies the encoding.

  ZnClient new get: 'http://<removed>/api/atm/801813'; yourself.

Tip 2: you could try to force override the default encoding using ZnDefaultCharacterEncoder (see the class comment).

But recent Zn code defaults to UTF8 (see #withDefaultUTF8Decoding: and #withDefaultUTF8Encoding:). So I am curious to see the response headers.

HTH,

Sven

On 01 May 2015, at 11:54, volkert [hidden email] wrote:

Dear all,

i am trying to get a JSON Document with ZnEasy. The Document is UTF-8 encoded.

(ZnEasy get: 'http://<removed>/api/atm/801813') entity string.
    
<gccacjeh.png>

Any hints?

Volkert

volkert-2

Re: ZnEasy / UTF-8

I thought UTF-8 is the default encoding in case JSON Documents, and so for the Content-Type: application/json.

I added "setCharSetUTF8" this Teapot-Output and all works fine now. :-)

Danke,
Volkert

Am 01.05.2015 um 14:20 schrieb Norbert Hartl:

You need to look at the headers of the request. I would assume the sender sets an Content-Type: application/json without charset. In that moment it is uncertain what has to be done because the server needs to assume what charset it might be and this tasks seldomly works out good.
In order to make it work set a content type of your entity this way

ZnMimeType applicationJson setCharSetUTF8

Now you send a Content-Type of application/json;charset=utf-8 and the server knows which character decoder to use. That should be it. So please always assign a char set to your sending data to make it more reliable what happens on the other end.

hope this helps,
Norbert
Am 01.05.2015 um 14:13 schrieb volkert <[hidden email]>:
I tried ZnClient new get: 'http://<removed>/api/atm/801813'; yourself and got.

<fdchdjab.png>

This is the Doc i see in Firefox.

<fceagiec.png>

The JSON is served with ZnServer / Teapot. ;-)

Volkert

Am 01.05.2015 um 12:21 schrieb Sven Van Caekenberghe:
Hi Volkert,

Tip 1: use ZnClient and inspect the response in detail, more specifically to see whether the response content type specifies the encoding.

  ZnClient new get: 'http://<removed>/api/atm/801813'; yourself.

Tip 2: you could try to force override the default encoding using ZnDefaultCharacterEncoder (see the class comment).

But recent Zn code defaults to UTF8 (see #withDefaultUTF8Decoding: and #withDefaultUTF8Encoding:). So I am curious to see the response headers.

HTH,

Sven
On 01 May 2015, at 11:54, volkert [hidden email] wrote:

Dear all,

i am trying to get a JSON Document with ZnEasy. The Document is UTF-8 encoded.

(ZnEasy get: 'http://<removed>/api/atm/801813') entity string.
    
<gccacjeh.png>

Any hints?

Volkert

Sven Van Caekenberghe-2

Re: ZnEasy / UTF-8

> On 01 May 2015, at 15:15, volkert <[hidden email]> wrote:
>
> I thought UTF-8 is the default encoding in case JSON Documents, and so for the Content-Type: application/json.
>
> I added "setCharSetUTF8" this Teapot-Output and all works fine now. :-)

Fine, but I still think this is strange, it should work by default too.
But to understand what went wrong, I need an example (server+client) that I can run.

> Danke,
> Volkert
>
> Am 01.05.2015 um 14:20 schrieb Norbert Hartl:
>> You need to look at the headers of the request. I would assume the sender sets an Content-Type: application/json without charset. In that moment it is uncertain what has to be done because the server needs to assume what charset it might be and this tasks seldomly works out good.
>> In order to make it work set a content type of your entity this way
>>
>> ZnMimeType applicationJson setCharSetUTF8
>>
>> Now you send a Content-Type of application/json;charset=utf-8 and the server knows which character decoder to use. That should be it. So please always assign a char set to your sending data to make it more reliable what happens on the other end.
>>
>> hope this helps,
>>
>> Norbert
>>> Am 01.05.2015 um 14:13 schrieb volkert <[hidden email]>:
>>>
>>> I tried ZnClient new get: 'http://<removed>/api/atm/801813'; yourself and got.
>>>
>>> <fdchdjab.png>
>>>
>>> This is the Doc i see in Firefox.
>>>
>>> <fceagiec.png>
>>>
>>> The JSON is served with ZnServer / Teapot. ;-)
>>>
>>> Volkert
>>>
>>>
>>> Am 01.05.2015 um 12:21 schrieb Sven Van Caekenberghe:
>>>> Hi Volkert,
>>>>
>>>> Tip 1: use ZnClient and inspect the response in detail, more specifically to see whether the response content type specifies the encoding.
>>>>
>>>> ZnClient new get: '
>>>> http://
>>>> <removed>/api/atm/801813'; yourself.
>>>>
>>>> Tip 2: you could try to force override the default encoding using ZnDefaultCharacterEncoder (see the class comment).
>>>>
>>>> But recent Zn code defaults to UTF8 (see #withDefaultUTF8Decoding: and #withDefaultUTF8Encoding:). So I am curious to see the response headers.
>>>>
>>>> HTH,
>>>>
>>>> Sven
>>>>
>>>>
>>>>> On 01 May 2015, at 11:54, volkert <[hidden email]>
>>>>> wrote:
>>>>>
>>>>> Dear all,
>>>>>
>>>>> i am trying to get a JSON Document with ZnEasy. The Document is UTF-8 encoded.
>>>>>
>>>>> (ZnEasy get: '
>>>>> http://
>>>>> <removed>/api/atm/801813') entity string.
>>>>>
>>>>> <gccacjeh.png>
>>>>>
>>>>> Any hints?
>>>>>
>>>>> Volkert
>>>>>
>>>>>
>>>>>
>>>
>>
>

NorbertHartl

Re: ZnEasy / UTF-8

In reply to this post by volkert-2

Am 01.05.2015 um 15:15 schrieb volkert <[hidden email]>:

I thought UTF-8 is the default encoding in case JSON Documents, and so for the Content-Type: application/json.

I added "setCharSetUTF8" this Teapot-Output and all works fine now. :-)

The combination assumption-character encoding is never a good one. It is something that is hard to debug if not done right. Character encoding should _always_ be negotiated. So like in most problems being explicit is superior to act on implicit things.

Danke,

Bitte.

Norbert

Volkert

Am 01.05.2015 um 14:20 schrieb Norbert Hartl:
You need to look at the headers of the request. I would assume the sender sets an Content-Type: application/json without charset. In that moment it is uncertain what has to be done because the server needs to assume what charset it might be and this tasks seldomly works out good.
In order to make it work set a content type of your entity this way

ZnMimeType applicationJson setCharSetUTF8

Now you send a Content-Type of application/json;charset=utf-8 and the server knows which character decoder to use. That should be it. So please always assign a char set to your sending data to make it more reliable what happens on the other end.

hope this helps,
Norbert
Am 01.05.2015 um 14:13 schrieb volkert <[hidden email]>:
I tried ZnClient new get: 'http://<removed>/api/atm/801813'; yourself and got.

<fdchdjab.png>

This is the Doc i see in Firefox.

<fceagiec.png>

The JSON is served with ZnServer / Teapot. ;-)

Volkert

Am 01.05.2015 um 12:21 schrieb Sven Van Caekenberghe:
Hi Volkert,

Tip 1: use ZnClient and inspect the response in detail, more specifically to see whether the response content type specifies the encoding.

  ZnClient new get: 'http://<removed>/api/atm/801813'; yourself.

Tip 2: you could try to force override the default encoding using ZnDefaultCharacterEncoder (see the class comment).

But recent Zn code defaults to UTF8 (see #withDefaultUTF8Decoding: and #withDefaultUTF8Encoding:). So I am curious to see the response headers.

HTH,

Sven
On 01 May 2015, at 11:54, volkert [hidden email] wrote:

Dear all,

i am trying to get a JSON Document with ZnEasy. The Document is UTF-8 encoded.

(ZnEasy get: 'http://<removed>/api/atm/801813') entity string.
    
<gccacjeh.png>

Any hints?

Volkert

volkert-2

Re: ZnEasy / UTF-8

In reply to this post by Sven Van Caekenberghe-2

Here a stripped down example

d := { 'bank' -> 'Région Genève Rhône'. } asDictionary.

teapot := Teapot configure: {
#defaultOutput -> #json.
#port -> 8081.
#debugMode -> true
}.

teapot
GET: '/api/d' -> d;
exception: KeyNotFound -> (TeaResponse notFound body: 'asd');
start.

(ZnEasy get: 'http://localhost:8081/api/d') entity string.

Pharo 4.0 / Teapot

BW,
Volkert

Am 01.05.2015 um 15:35 schrieb Sven Van Caekenberghe:

>> On 01 May 2015, at 15:15, volkert <[hidden email]> wrote:
>>
>> I thought UTF-8 is the default encoding in case JSON Documents, and so for the Content-Type: application/json.
>>
>> I added "setCharSetUTF8" this Teapot-Output and all works fine now. :-)
> Fine, but I still think this is strange, it should work by default too.
> But to understand what went wrong, I need an example (server+client) that I can run.
>
>> Danke,
>> Volkert
>>
>> Am 01.05.2015 um 14:20 schrieb Norbert Hartl:
>>> You need to look at the headers of the request. I would assume the sender sets an Content-Type: application/json without charset. In that moment it is uncertain what has to be done because the server needs to assume what charset it might be and this tasks seldomly works out good.
>>> In order to make it work set a content type of your entity this way
>>>
>>> ZnMimeType applicationJson setCharSetUTF8
>>>
>>> Now you send a Content-Type of application/json;charset=utf-8 and the server knows which character decoder to use. That should be it. So please always assign a char set to your sending data to make it more reliable what happens on the other end.
>>>
>>> hope this helps,
>>>
>>> Norbert
>>>> Am 01.05.2015 um 14:13 schrieb volkert <[hidden email]>:
>>>>
>>>> I tried ZnClient new get: 'http://<removed>/api/atm/801813'; yourself and got.
>>>>
>>>> <fdchdjab.png>
>>>>
>>>> This is the Doc i see in Firefox.
>>>>
>>>> <fceagiec.png>
>>>>
>>>> The JSON is served with ZnServer / Teapot. ;-)
>>>>
>>>> Volkert
>>>>
>>>>
>>>> Am 01.05.2015 um 12:21 schrieb Sven Van Caekenberghe:
>>>>> Hi Volkert,
>>>>>
>>>>> Tip 1: use ZnClient and inspect the response in detail, more specifically to see whether the response content type specifies the encoding.
>>>>>
>>>>> ZnClient new get: '
>>>>> http://
>>>>> <removed>/api/atm/801813'; yourself.
>>>>>
>>>>> Tip 2: you could try to force override the default encoding using ZnDefaultCharacterEncoder (see the class comment).
>>>>>
>>>>> But recent Zn code defaults to UTF8 (see #withDefaultUTF8Decoding: and #withDefaultUTF8Encoding:). So I am curious to see the response headers.
>>>>>
>>>>> HTH,
>>>>>
>>>>> Sven
>>>>>
>>>>>
>>>>>> On 01 May 2015, at 11:54, volkert <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> i am trying to get a JSON Document with ZnEasy. The Document is UTF-8 encoded.
>>>>>>
>>>>>> (ZnEasy get: '
>>>>>> http://
>>>>>> <removed>/api/atm/801813') entity string.
>>>>>>
>>>>>> <gccacjeh.png>
>>>>>>
>>>>>> Any hints?
>>>>>>
>>>>>> Volkert
>>>>>>
>>>>>>
>>>>>>
>

Sven Van Caekenberghe-2

Re: ZnEasy / UTF-8

Hi Volkert,

Thanks for the test case, it helped me figure out what the problem was.

Basically, what Norbert said was 100% correct: the server code (Teapot) should have specified the charset or encoding of the mime-type it was using (so application/json;charset=utf-8 instead of just application/json).

However, I thought I had changed the default to be utf-8, so I was surprised it was not picked up.

It turned out utf-8 was the default for reading/writing http messages and their entities, but not so when manually constructing instances (like Teapot did).

In code,

json := '{''bank'':''Région Genève Rhône''}'.
entity := ZnEntity with: json type: 'application/json'.
response := ZnResponse ok: entity.
response encoder.

did not return a ZnUTF8Encoder but a ZnNullEncoder.

I changed that now.

===
Name: Zinc-HTTP-SvenVanCaekenberghe.428
Author: SvenVanCaekenberghe
Time: 2 May 2015, 7:09:55.832482 pm
UUID: 6376bd91-2a95-4265-bfff-e3c6626aee79
Ancestors: Zinc-HTTP-SvenVanCaekenberghe.427

Change ZnStringEntity>>#initializeEncoder to default to ZnUTF8Encoder unless ZnDefaultCharacterEncoder is set

Change ZnEntityReader>>#readEntity to no longer use #withDefaultUtf8Decoder: (and remove it)

Change ZnEntityWriter>>#writeEntity: to no longer use #withDefaultUtf8Encoder: (and remove it)

Add ZnResponseTests>>#testDefaultUTF8Encoding
===
Name: Zinc-Tests-SvenVanCaekenberghe.226
Author: SvenVanCaekenberghe
Time: 2 May 2015, 7:10:12.331593 pm
UUID: 9010fbd6-bf9d-400e-b463-cb73920b79f6
Ancestors: Zinc-Tests-SvenVanCaekenberghe.225

Change ZnStringEntity>>#initializeEncoder to default to ZnUTF8Encoder unless ZnDefaultCharacterEncoder is set

Change ZnEntityReader>>#readEntity to no longer use #withDefaultUtf8Decoder: (and remove it)

Change ZnEntityWriter>>#writeEntity: to no longer use #withDefaultUtf8Encoder: (and remove it)

Add ZnResponseTests>>#testDefaultUTF8Encoding
===

for now, in #bleedingEdge of Zinc HTTP Components.

HTH,

Sven

> On 01 May 2015, at 15:53, volkert <[hidden email]> wrote:
>
> Here a stripped down example
>
> d := { 'bank' -> 'Région Genève Rhône'. } asDictionary.
>
> teapot := Teapot configure: {
> #defaultOutput -> #json.
> #port -> 8081.
> #debugMode -> true
> }.
>
> teapot
> GET: '/api/d' -> d;
> exception: KeyNotFound -> (TeaResponse notFound body: 'asd');
> start.
>
> (ZnEasy get: 'http://localhost:8081/api/d') entity string.
>
> Pharo 4.0 / Teapot
>
> BW,
> Volkert
>
>
>
> Am 01.05.2015 um 15:35 schrieb Sven Van Caekenberghe:
>>> On 01 May 2015, at 15:15, volkert <[hidden email]> wrote:
>>>
>>> I thought UTF-8 is the default encoding in case JSON Documents, and so for the Content-Type: application/json.
>>>
>>> I added "setCharSetUTF8" this Teapot-Output and all works fine now. :-)
>> Fine, but I still think this is strange, it should work by default too.
>> But to understand what went wrong, I need an example (server+client) that I can run.
>>
>>> Danke,
>>> Volkert
>>>
>>> Am 01.05.2015 um 14:20 schrieb Norbert Hartl:
>>>> You need to look at the headers of the request. I would assume the sender sets an Content-Type: application/json without charset. In that moment it is uncertain what has to be done because the server needs to assume what charset it might be and this tasks seldomly works out good.
>>>> In order to make it work set a content type of your entity this way
>>>>
>>>> ZnMimeType applicationJson setCharSetUTF8
>>>>
>>>> Now you send a Content-Type of application/json;charset=utf-8 and the server knows which character decoder to use. That should be it. So please always assign a char set to your sending data to make it more reliable what happens on the other end.
>>>>
>>>> hope this helps,
>>>>
>>>> Norbert
>>>>> Am 01.05.2015 um 14:13 schrieb volkert <[hidden email]>:
>>>>>
>>>>> I tried ZnClient new get: 'http://<removed>/api/atm/801813'; yourself and got.
>>>>>
>>>>> <fdchdjab.png>
>>>>>
>>>>> This is the Doc i see in Firefox.
>>>>>
>>>>> <fceagiec.png>
>>>>>
>>>>> The JSON is served with ZnServer / Teapot. ;-)
>>>>>
>>>>> Volkert
>>>>>
>>>>>
>>>>> Am 01.05.2015 um 12:21 schrieb Sven Van Caekenberghe:
>>>>>> Hi Volkert,
>>>>>>
>>>>>> Tip 1: use ZnClient and inspect the response in detail, more specifically to see whether the response content type specifies the encoding.
>>>>>>
>>>>>> ZnClient new get: '
>>>>>> http://
>>>>>> <removed>/api/atm/801813'; yourself.
>>>>>>
>>>>>> Tip 2: you could try to force override the default encoding using ZnDefaultCharacterEncoder (see the class comment).
>>>>>>
>>>>>> But recent Zn code defaults to UTF8 (see #withDefaultUTF8Decoding: and #withDefaultUTF8Encoding:). So I am curious to see the response headers.
>>>>>>
>>>>>> HTH,
>>>>>>
>>>>>> Sven
>>>>>>
>>>>>>
>>>>>>> On 01 May 2015, at 11:54, volkert <[hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> i am trying to get a JSON Document with ZnEasy. The Document is UTF-8 encoded.
>>>>>>>
>>>>>>> (ZnEasy get: '
>>>>>>> http://
>>>>>>> <removed>/api/atm/801813') entity string.
>>>>>>> <gccacjeh.png>
>>>>>>>
>>>>>>> Any hints?
>>>>>>>
>>>>>>> Volkert
>>>>>>>
>>>>>>>
>>>>>>>
>>
>
>

volkert-2

Re: ZnEasy / UTF-8

Understand. Anyway, in Teapot it should possible to set the UTF-8
charset for JSON documents.

Thanks,
Volkert

Am 02.05.2015 um 20:34 schrieb Sven Van Caekenberghe:

> Hi Volkert,
>
> Thanks for the test case, it helped me figure out what the problem was.
>
> Basically, what Norbert said was 100% correct: the server code (Teapot) should have specified the charset or encoding of the mime-type it was using (so application/json;charset=utf-8 instead of just application/json).
>
> However, I thought I had changed the default to be utf-8, so I was surprised it was not picked up.
>
> It turned out utf-8 was the default for reading/writing http messages and their entities, but not so when manually constructing instances (like Teapot did).
>
> In code,
>
> json := '{''bank'':''Région Genève Rhône''}'.
> entity := ZnEntity with: json type: 'application/json'.
> response := ZnResponse ok: entity.
> response encoder.
>
> did not return a ZnUTF8Encoder but a ZnNullEncoder.
>
> I changed that now.
>
> ===
> Name: Zinc-HTTP-SvenVanCaekenberghe.428
> Author: SvenVanCaekenberghe
> Time: 2 May 2015, 7:09:55.832482 pm
> UUID: 6376bd91-2a95-4265-bfff-e3c6626aee79
> Ancestors: Zinc-HTTP-SvenVanCaekenberghe.427
>
> Change ZnStringEntity>>#initializeEncoder to default to ZnUTF8Encoder unless ZnDefaultCharacterEncoder is set
>
> Change ZnEntityReader>>#readEntity to no longer use #withDefaultUtf8Decoder: (and remove it)
>
> Change ZnEntityWriter>>#writeEntity: to no longer use #withDefaultUtf8Encoder: (and remove it)
>
> Add ZnResponseTests>>#testDefaultUTF8Encoding
> ===
> Name: Zinc-Tests-SvenVanCaekenberghe.226
> Author: SvenVanCaekenberghe
> Time: 2 May 2015, 7:10:12.331593 pm
> UUID: 9010fbd6-bf9d-400e-b463-cb73920b79f6
> Ancestors: Zinc-Tests-SvenVanCaekenberghe.225
>
> Change ZnStringEntity>>#initializeEncoder to default to ZnUTF8Encoder unless ZnDefaultCharacterEncoder is set
>
> Change ZnEntityReader>>#readEntity to no longer use #withDefaultUtf8Decoder: (and remove it)
>
> Change ZnEntityWriter>>#writeEntity: to no longer use #withDefaultUtf8Encoder: (and remove it)
>
> Add ZnResponseTests>>#testDefaultUTF8Encoding
> ===
>
> for now, in #bleedingEdge of Zinc HTTP Components.
>
> HTH,
>
> Sven
>