Issue 5063 in pharo: Zinc uses default encoding of utf-8 when encoding 'url safe encoded' strings

Status: Accepted
Owner: ----
Labels: Milestone-1.3 Type-Bug

New issue 5063 by norbert.hartl: Zinc uses default encoding of utf-8 when  
encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

I've discovered a problem in Zinc. If you execute this piece of code

| client entity |
client := ZnNeoClient new.
entity := client applicationFormUrlEncodedEntity.
entity at: 'testkey' put: 'ö'.
Transcript show: 'entity type: ', entity contentType printString; cr.
Transcript show: 'encoded body: ', (String streamContents: [:stream|
        entity writeRepresentationOn: stream ])

You'll get (in pharo #13315)

entity type: application/x-www-form-urlencoded;charset=iso-8859-1
encoded body: testkey=%C3%B6

As you can see the content type contains latin-1 as encoding while the body  
is utf-8 encoded. The utf-8 encoding comes from

ZnUtils class>>writeQueryFields:on:

that uses encodeForHTTP which uses utf-8 by default.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #1 on issue 5063 by norbert.hartl: Zinc uses default encoding of  
utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

It is not really clear where the iso-8859-1 encoding comes from. But the  
behaviour is wrong anyway. I just comment this to not forget to check the  
cause of the iso addition


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #2 on issue 5063 by norbert.hartl: Zinc uses default encoding of  
utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

Fix uploaded in ZincHTTPComponents repository:

Name: Zinc-HTTP-NorbertHartl.225
Author: NorbertHartl
Time: 6 December 2011, 6:30:33 pm
UUID: 18a28639-5e95-4594-937c-268df69987ec
Ancestors: Zinc-HTTP-SvenVanCaekenberghe.224

changed ZnApplicationFormUrlEncodedEntity to check encoding of contentType.  
If an encoding is present to presentation is written using the specified  
encoding

---

Name: Zinc-Tests-NorbertHartl.117
Author: NorbertHartl
Time: 6 December 2011, 6:35:28 pm
UUID: 4971e3f3-ba21-46ab-8674-560d00dfe751
Ancestors: Zinc-Tests-SvenVanCaekenberghe.116

second commit. Last time I added the test to an old version. Redid for  
newest version:

added two tests to check encoding handling of  
ZnApplicationFormUrlEncodedEntity when writing representation


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #3 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

Super, Norbert!

Thanks a lot.

Everything looks OK to me.

Yes, I will think a bit about the ZnMimeType sharing/immutability problem,  
I knew that was a bit weak, but I have to find a good solution.

Sven



_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #4 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

I commited the following changes to ZnMimeType to deal with your other  
issue.


Name: Zinc-HTTP-SvenVanCaekenberghe.226
Author: SvenVanCaekenberghe
Time: 6 December 2011, 8:56:22 pm
UUID: 52cc1692-4d4f-4c43-813c-21b61e386eaf
Ancestors: Zinc-HTTP-NorbertHartl.225

changed ZnMimeType such that the 'constants' returned by the methods in the  
class side convenience protocol can now be freely modified by returning a  
copy;
implemented ZnMimeType>>#postCopy;
ZnMimeType>>#parameters will now lazy initialize to a SmallDictionary  
instead of a regular Dictionary


Name: Zinc-Tests-SvenVanCaekenberghe.118
Author: SvenVanCaekenberghe
Time: 6 December 2011, 8:58:47 pm
UUID: ce3f20f9-22d1-4b52-afe0-9c3a04813224
Ancestors: Zinc-Tests-NorbertHartl.117

added ZnMimeTypeTests>>testCopying to test whether the 'constants' returned  
by the class side convenience methods of ZnMimeType can be freely modified;
modified ZnMimeType>>#testDefault and #testIdentity to not longer assume  
the 'constants' returned by the class side convenience methods of  
ZnMimeType are #==



_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: FixToInclude

Comment #5 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

Zinc-HTTP-SvenVanCaekenberghe.226 and Zinc-Tests-SvenVanCaekenberghe.118  
from http://www.squeaksource.com/ZincHTTPComponents


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Labels: Milestone-1.4

Comment #6 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

Thanks


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #7 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

So should we update 1.3, too? We did not update zinc there since a looong  
time


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #8 on issue 5063 by norbert.hartl: Zinc uses default encoding of  
utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

I opt for it. The last change with exchanging ZnNeoClient with ZnClient was  
rather heavy. This fix is a pure fix that does not change any interface


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: Integrated

Comment #10 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

I update Zinc in 1.3 to the latest version

13321


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: Workneeded

Comment #11 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

Nope. After doing that, i can not publish the updates. It always times out  
when uploading the mcz files.




_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Cc: [hidden email]

Comment #12 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

(No comment was entered for this change.)


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #13 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

That's no good, and it should not happen (I work in 1.3 all the time).

How can I reproduce this myself ?

I take what version, #13315 and load the latest Zn ?


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #14 on issue 5063 by norbert.hartl: Zinc uses default encoding of  
utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

Ah, maybe that is the problem I was experiencing. If you upgrade pharo via  
update stream you end up with ZnClient having no methods on the instance  
side. I could image this has something to do with the renaming of  
ZnNeoClient and ZnClient with removing ZnNeoClient at the same time. I  
don't know you the atomic loading behaviour works in pharo but I think it  
is suspect to helped in this breakage


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Labels: -Milestone-1.4

Comment #15 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

(No comment was entered for this change.)


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #16 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

Updating to latest Zn should fix this.

See also:

   http://code.google.com/p/pharo/issues/detail?id=5122

Don't forget to do

   ZnMimeType initialize


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: FixToInclude

Comment #17 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

Another problem is that 1.3 now is finally too large for squeak source.

I need to upload a new ScriptLoader package, but the UI times out *and*  
with MC it does not work either.

So before doing anything on 1.3, we need to move it away from squeak source.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #18 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

Marcus,

I didn't notice this issue was tagged 1.3

For an insane timeout of 15 minutes in 1.3 #13315 you could do:

ZnNetworkingUtils defaultSocketStreamTimeout: 900

HTH,

Sven


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #19 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

I'm lost. Should this fix be included in 1.4?


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #20 on issue 5063 by [hidden email]: Zinc uses default  
encoding of utf-8 when encoding 'url safe encoded' strings
http://code.google.com/p/pharo/issues/detail?id=5063

No, 1.4 already has this code.

Zn needs to be updated for 1.3 but that seems to be difficult.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker