GRInvalidUtf8Error: Invalid UTF-8 input

Hi,

I have this pdf file (joined here) which always produce this error when
I try to upload it in Seaside.

It seems to be true whenever the filename comes with non ascii characters.

All in all, it looks likes GRPharoUtf8Codec is wrong in decoding
filename with non ascii characters.

It happens with Seaside3.1, Pharo4, newest Grease-Pharo30-Core package

When using the Seaside Functional Test Suite / Upload test, the same
problem occurs.

What did I miss?

Thanks

Hilaire

--
Dr. Geo
http://drgeo.eu

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

On Tue, Jun 21, 2016 at 4:46 PM, Hilaire <[hidden email]> wrote:
A stack trace would be helpful so that we know where error happens.

Also some other information:
- is the web site utf-8?
- what adapter do you use?
- what's the encoding on the adapter?
- what operating system and browser do you use?
- I assume the strange language in the file name is french.

Super helpful would be a tcpdump of the http session but we can do without.

Cheers
Philippe
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

It works with pure Zn in Pharo:
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi,

Le 21/06/2016 19:01, Philippe Marschall a écrit :
> A stack trace would be helpful so that we know where error happens.

Sadly there is no trace as the debugger does not popup. Only this error
raised, from GRPharoUtf8Codec>>decode: with a call to the invalidUtf8
method:

GRPharoUtf8Codec>>invalidUtf8
GRInvalidUtf8Error signal: 'Invalid UTF-8 input'

Strangely no debugger pop up, only the error message printed on the
browser, although in other seaside situation I have error. So I insert
an halt to have the attached trace.


> Also some other information:
> - is the web site utf-8?

Yes

> - what adapter do you use?

Type: ZnZincServerAdaptor
Port: 8080
Encoding: utf-8
zinc on port 8080 [running]

> - what's the encoding on the adapter?



> - what operating system and browser do you use?

Linux for the server.
I experienced the issue with Chrome under Windows and Linux
As well Firefox on Linux

> - I assume the strange language in the file name is french.

Indeed :)

Thanks


> Super helpful would be a tcpdump of the http session but we can do without.

I have to learn to do that.


--
Dr. Geo
http://drgeo.eu

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

On Tue, Jun 21, 2016 at 7:31 PM, Hilaire <[hidden email]> wrote:
Ok, it's likely in the server adapter before Seaside actually kicks in
then. Can you set a break point in GRPharoUtf8Codec >> #invalidUtf8?

My suspect would be ZnZincServerAdaptor >> #convertMultipart:

If you can send us the string it's trying to convert that would be helpful.

Cheers
Philippe
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Le 22/06/2016 09:40, Philippe Marschall a écrit :
> Ok, it's likely in the server adapter before Seaside actually kicks in
> then. Can you set a break point in GRPharoUtf8Codec >> #invalidUtf8?
>
> My suspect would be ZnZincServerAdaptor >> #convertMultipart:
>
> If you can send us the string it's trying to convert that would be helpful.


The string argument of GRPharoUtf8Codec>>decode:

is

aString ->'Identités certifiées.pdf'

printed as this in the Debugger. As we know Pharo does not use UTF8
internally it is suspect to see an utf8 string correctly printed in
Pharo, right?

Does it looks like a Latin1 ?:

aString  asByteArray do:  [:each|  Transcript show: each hex ;  space]

16r49 16r64 16r65 16r6E 16r74 16r69 16r74 16rE9 16r73 16r20 16r63 16r65
16r72 16r74 16r69 16r66 16r69 16rE9 16r65 16r73 16r2E 16r70 16r64 16r66

So indeed, GRPharoUtf8Codec>>decode: already received a decoded utf8
string to latin1, then obviously fail.

Now looking back in the stack as you suggested, then decoding already
took place at:

ZnMimePart>>fileName
"Pathnames are often silenty encoded using UTF-8,
this is a no-op for ASCII, but will fail on Latin-1 and others"
^ (self detectContentDispositionValue: 'filename')
        ifNotNil: [ :encodedFileName | encodedFileName asByteArray utf8Decoded ]


The timecode of this method is 10/10/2014 from Sven

The second place where the decode takes place is (Zinc-Seaside package):

ZNZincServerAdaptor>>convertMultipartFileField: part
| file |
(file := WAFile new)
        fileName: (self codec decode: part fileName);
        contentType: part contentType printString;
        contents: part contents asByteArray.
^ file

Timecode is 11/14/2014 from Johan, where the decode was added.


This two methods use too different decoding methods (duplication?), one
from the Grease package, the other from ZN package.

My opinion is the Zinc-Seaside package should not try to decode, or
preferably use the ZN decode method (utf8Decoded), but it will bring an
error on already decoded string.


Hilaire

--
Dr. Geo
http://drgeo.eu

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

My suggestion is to remove the decode: part, this change looks safe as
part fileName already decodes.


Hilaire


Le 22/06/2016 13:33, Hilaire a écrit :
> ZNZincServerAdaptor>>convertMultipartFileField: part
> | file |
> (file := WAFile new)
> fileName: (self codec decode: part fileName);
> contentType: part contentType printString;
> contents: part contents asByteArray.
> ^ file
>
> Timecode is 11/14/2014 from Johan, where the decode was added.

--
Dr. Geo
http://drgeo.eu

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

My suggestion is to remove the decode: in this method. I am not sure
about the whole implication.

Hilaire

Le 22/06/2016 13:33, Hilaire a écrit :
> ZNZincServerAdaptor>>convertMultipartFileField: part
> | file |
> (file := WAFile new)
> fileName: (self codec decode: part fileName);
> contentType: part contentType printString;
> contents: part contents asByteArray.
> ^ file
>
> Timecode is 11/14/2014 from Johan, where the decode was added.

--
Dr. Geo
http://drgeo.eu

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> On 22 Jun 2016, at 14:01, Hilaire <[hidden email]> wrote:
>
> My suggestion is to remove the decode: in this method.

I think that too.

> I am not sure about the whole implication.

Me neither ;-)

In any case, here is an older discussion about the same topic:

http://forum.world.st/File-upload-encoding-issue-tt4783446.html

Sven

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Indeed, but looking at the time code of the two methods I mentioned
(both after this October 2014 discussion), it looks like you guys
concurrently fixed the problem in two different parts, but resulting in
a mural break :) Hopefully easy to get fixed.

But I am surprise this problem did not show up till then. Or is it fixed
in newer ZN-* packages?

Hilaire



Le 22/06/2016 15:28, Sven Van Caekenberghe a écrit :
> Me neither ;-)
>
> In any case, here is an older discussion about the same topic:
>
> http://forum.world.st/File-upload-encoding-issue-tt4783446.html
>
> Sven

--
Dr. Geo
http://drgeo.eu

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

I don't remember every detail of the discussions, but by the look of it, your solution seems correct: Zn did already decode it, the Zn adaptor should not do it again.

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

If one grants me write access, I can commit the change.
Of course I would like to know the opinion of Johan, I don't want to
break anything.

Hilaire

Le 22/06/2016 22:01, Sven Van Caekenberghe a écrit :
> I don't remember every detail of the discussions, but by the look of it, your solution seems correct: Zn did already decode it, the Zn adaptor should not do it again.

--
Dr. Geo
http://drgeo.eu

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi Hilaire,

I am curious which package version of Zinc-Seaside you are using.

I vaguely remember something about fixing an issue in that area but when I look at Zinc-Seaside-JohanBrichau.43 (which is the latest version) the send of #decode: is not there.

cheers
Johan

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi Johan,

When I browse directly from Monticello the Zinc-Seaside-JohanBrichau.43
package I can read the #decode: message.

Is it possible you fixed it locally without a commit?

The repo I have been using is http://mc.stfx.eu/ZincHTTPComponents

Hilaire

Le 25/06/2016 08:37, Johan Brichau a écrit :
> Hi Hilaire,
>
> I am curious which package version of Zinc-Seaside you are using.
>
> I vaguely remember something about fixing an issue in that area but when I look at Zinc-Seaside-JohanBrichau.43 (which is the latest version) the send of #decode: is not there.
>
> cheers
> Johan

--
Dr. Geo
http://drgeo.eu

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Are there multiple package versions out there with the same version number?
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hilaire,
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Le 25/06/2016 09:16, Johan Brichau a écrit :
> Are there multiple package versions out there with the same version number?
> Can you check if the UUID is the same as the one below?

The description of the package I have looks strange (see the Time):

Name: Zinc-Seaside-JohanBrichau.43
Author: JohanBrichau
Time: 22 Juin 2016, 8:25:10.061265 pm
UUID: d2edc44e-eebb-454f-b309-f6d49d639770
Ancestors: Zinc-Seaside-pmm.42


When I flush the Monticello cache and refresh, it looks like (the Time
is shifted to now):

Name: Zinc-Seaside-JohanBrichau.43
Author: JohanBrichau
Time: 25 Juin 2016, 8:25:10.061265 pm
UUID: d2edc44e-eebb-454f-b309-f6d49d639770
Ancestors: Zinc-Seaside-pmm.42


> Here is a screenshot when I look at Zinc-Seaside-JohanBrichau.43 in
> http://mc.stfx.eu/ZincHTTPComponents
> I am looking at the correct method, right?

Yes.


--
Dr. Geo
http://drgeo.eu

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

When downloading

http://mc.stfx.eu/ZincHTTPComponents/Zinc-Seaside-JohanBrichau.43.mcz

and viewing the source code in a text editor, the #decode: message is
present in the convertMultipartFileField: method.

Hilaire

Le 25/06/2016 09:24, Johan Brichau a écrit :
--
Dr. Geo
http://drgeo.eu

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hilaire,

Your observation that Sven and I both fixed it in different places is correct.
I did revert the fix on December 26th but I somehow created the package below with the same version number and it never got uploaded correctly (for reasons I cannot really explain).

Name: Zinc-Seaside-JohanBrichau.43
Author: JohanBrichau
Time: 26 December 2014, 4:00:06.580211 pm
UUID: 62fd0b62-9e07-498e-aac8-3432614b39fc
Ancestors: Zinc-Seaside-pmm.42

I already created a new package .44 and will email it to Sven but...

The “funny” part right now is that the upload file test does not break in my image when using the published code for Zinc-Seaside!
So… I am a bit baffled and would like to understand first what is going in.

Johan

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside