random maddening bug - "corrupted" file upload contents

Cyril,

If you think something is wrong with the low level Zinc implementation, you could try without Seaside.

ZnServer startDefaultOn: 1701.

http://localhost:1701/form-test-3

You could try to debug/inspect from #formTest3:

HTH,

Sven

> On 22 Dec 2017, at 11:38, Cyrille Delaunay <[hidden email]> wrote:
>
> Hi everyone,
>
> Since some times, we are experimenting a random bug while using seaside file upload.
> Not sure at all about the exact origin of this issue (Application specific, Zinc, Seaside, Pharo ?).
> But as it start to drive us mad, I'm sharing our problem with you.
> Maybe someone would already have experimented something similar or would have a good feeling about the issue.
>
> High level description:
>
> We have a file upload element rendered with seaside.
> The callback of this file upload element is triggered when the form is submitted via a submit button.
> Problem: Within this callback, the raw contents of the WAFile does not match the expected contents of the selected file.
> The first 4 bytes are "wrong", and rest of the byte array seems ok.
> This issue occurs randomly and we haven't been able to determine / guess any context criteria.
>
>
> Debugging:
>
> In order to figure out what was wrong, we tried to catch a place in pharo where file contents started to differ from the expected file contents.
>
> The earlier upload step in which we have been able to notice this delta, is method
>
> ZnMultiThreadedServer >> executeOneRequestResponseOn:
>
> In this method, a http request is read and materialized from an input stream,
> From my naive point of vue, this stream looks like a kind of communication entry point between the web browser and pharo.
>
> The zip input file I am trying to upload is expected to begin with the following bytes:
> #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 ...
>
> The request object built within #executeOneRequestResponseOn:  points to an"entity" variable, which points to a "parts" variable, which points to a single ZnMimePart.
> This ZnMimePart points to a byte array beginning with:
> #[0 0 0 0 10 0 0 0 0 0 125 83 67 73 ...
>
> I also experimented another case where
> The request object built within #executeOneRequestResponseOn:  points to an"entity" variable, which points to a "parts" variable, which points to multiple ZnMimePart.
> One of the ZnMimePart points to a byte array beginning with:
> #[138 67 80 75 10 0 0 0 0 0 125 83 67 73
>
> We can see that in both cases the first 4 bytes do not match the expected first 4 bytes of the file.
> As a consequence, later in our process,
> pharo gets a error while trying to unzip this upload file, complaining about a "'bad LH signature at 0'".
>
>
> Does this speak to anyone ?
>
>
>
> --
> Cyrille Delaunay

> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Well, I can't repeat it, for now.

Try to make a minimal self-contained example that fails (reliably), like how I did it, you can even try to do the upload in Pharo too (see #testFormTest3).

> On 22 Dec 2017, at 18:00, Cyril Ferlicot <[hidden email]> wrote:
>
>
> On ven. 22 déc. 2017 at 17:50, Sven Van Caekenberghe <[hidden email]> wrote:
> Cyrille,
>
> I wrote a file like this:
>
> FileLocator temp / 'myzip.zip' writeStreamDo: [ :out |
>   out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0] ].
>
> Which I can read as follows:
>
> FileLocator temp / 'myzip.zip' binaryReadStreamDo: [ :in | in upToEnd ].
>
>   => #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0]
>
> If I upload this file to http://localhost:1701/form-test-3 this works perfectly (macOS 10.13.2 Pharo #60402, Safari. Firefox & Chrome), first time and second time.
>
>
>
> Maybe you should try on a co-workers machine just to make sure it is not a local problem on your machine ?
>
> Hi Sven,
>
> It happen on Cyrille's machine, my machine, Yann's machine and our boss's machine.
>
> We first saw it on Windows and never on mac/Linux. But now we see it also on mac.
>
> I did not see it yet on Linux but we use it less for now.
>
> Also, when it happen, it tend to happen a lot afterward.
>
> Also, by random it means that we can have 1/2 week without problem, then 4 failure in 2h.
>
>
>
> Sven
>
> --
> Cyril Ferlicot
> https://ferlicot.fr
>
> http://www.synectique.eu
> 2 rue Jacques Prévert 01,
> 59650 Villeneuve d'ascq France
> <Screen Shot 2017-12-22 at 17.48.31.png>_______________________________________________

> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi Sven,

I am back on this issue :)

I have kind of a "clean" scenario that could help to reproduce.

It is still random, but at one point, it shows up for me.

Here is how I am doing:

Download Pharo 6.1 on my Mac (Sierra 10.12.6): https://pharo.org/web/download
Then, iterate over this process till spotting the issue:

=> start the pharo image
=> execute this piece of code in a playground

ZnServer startDefaultOn: 1701.
ZnServer default maximumEntitySize: 80* 1024 * 1024. 
'/Users/cdelaunay/myzip.zip' asFileReference writeStreamDo: [ :out | 
out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0].
18202065 timesRepeat: [ out nextPut: 0 ]
].

=> Open a web browser page on: http://localhost:1701/form-test-3
=> Upload the file zip file previously generated ('myzip.zip').
=> If the web page displays: "contents=000000000a00..." (or anything unexpected), THIS IS THE ISSUE !
=> If the web page displays: "contents=504b03040a00..", the upload worked fine. You can close the image (without saving)

Mettre à jour Saisir temps Surveiller Copier Supprimer

Usually, after starting 3 / 4 times pharo image, I am getting the problem.

I have been able to reproduce with chrome and firefox.

Seing that I was able to reproduce within a fresh pharo image, I tried to automate this scenario.

I wrote the following test:

ZnServerTests >>testFormTest4

| inputs client part fileReference fileContents entity |

fileReference := 'test-zip.zip' asFileReference.

fileReference

writeStreamDo: [ :out | 

out

binary;

nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0].

18202065 timesRepeat: [ out nextPut: 0 ] ].

fileContents := fileReference binaryReadStreamDo: [ :in | in upToEnd ].

15

timesRepeat: [ self

withServerDo: [ :server | 

server maximumEntitySize: 80 * 1024 * 1024.

server debugMode: true.

(client := ZnClient new)

url: server localUrl;

addPathSegment: 'form-test-3'.

client timeout: 120000.

entity := (ZnByteArrayEntity type: (ZnMimeType main: 'application' sub: 'zip') length: fileContents size) bytes: fileContents.

part := ZnMimePart fieldName: 'file' fileName: fileReference basename entity: entity.

client

resetEntity;

addPart: part;

post.

self assert: client isSuccess.

self assert: (client contents includesSubstring: '504b03040a') ] ]

In order to make this test run correctly I had to modify this method.

Without this change, I always got an empty html page as response

ZnDefaultServerDelegate >> formTest3: request

| contents filename contentType page |

contents := filename := contentType := ''.

(request hasEntity and: [ request contentType matches: ZnMimeType multiPartFormData ])

ifTrue: [ 

(request entity partNamed: #file ifNone: [ nil ]) 

ifNotNil: [ :part |

filename := part fileName.

contents := part contents.

contentType := part  contentType.

contentType isBinary ifTrue: [ contents := contents hex ] ] ].

page := ZnHtmlOutputStream streamContents: [ :html |

html page: 'Form Test 3' do: [ 

html 

tag: #form 

attributes: #(action 'form-test-3' 'accept-charset' 'utf-8' 

enctype 'multipart/form-data' method POST) 

do: [ 

html 

str: 'File'; space;

tag: #input attributes: #(type file name file); space;

tag: #input attributes: #(type submit) ];

tag: #p do: [ html str: 'filename = '; str: filename ];

tag: #p do: [ html str: 'content-type = '; str: contentType asString ];

tag: #p do: [ html str: 'contents = '; str: (contents copyFrom: 1 to: 20) asString ] ] ].

^ ZnResponse ok: (ZnEntity html: page)

Unfortunately, the tests always passed till now !

I will try to go on refining the context parameters that could make the issue show up or not

 

2017-12-22 18:05 GMT+01:00 Sven Van Caekenberghe <[hidden email]>:

Well, I can't repeat it, for now.

Try to make a minimal self-contained example that fails (reliably), like how I did it, you can even try to do the upload in Pharo too (see #testFormTest3).

> On 22 Dec 2017, at 18:00, Cyril Ferlicot <[hidden email]> wrote:
>
>
> On ven. 22 déc. 2017 at 17:50, Sven Van Caekenberghe <[hidden email]> wrote:
> Cyrille,
>
> I wrote a file like this:
>
> FileLocator temp / 'myzip.zip' writeStreamDo: [ :out |
>   out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0] ].
>
> Which I can read as follows:
>
> FileLocator temp / 'myzip.zip' binaryReadStreamDo: [ :in | in upToEnd ].
>
>   => #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0]
>
> If I upload this file to http://localhost:1701/form-test-3 this works perfectly (macOS 10.13.2 Pharo #60402, Safari. Firefox & Chrome), first time and second time.
>
>
>
> Maybe you should try on a co-workers machine just to make sure it is not a local problem on your machine ?
>
> Hi Sven,
>
> It happen on Cyrille's machine, my machine, Yann's machine and our boss's machine.
>
> We first saw it on Windows and never on mac/Linux. But now we see it also on mac.
>
> I did not see it yet on Linux but we use it less for now.
>
> Also, when it happen, it tend to happen a lot afterward.
>
> Also, by random it means that we can have 1/2 week without problem, then 4 failure in 2h.
>
>
>
> Sven
>
> --
> Cyril Ferlicot
> https://ferlicot.fr
>
> http://www.synectique.eu
> 2 rue Jacques Prévert 01,
> 59650 Villeneuve d'ascq France
> <Screen Shot 2017-12-22 at 17.48.31.png>_______________________________________________

> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

--

Cyrille Delaunay

2018-01-02 17:37 GMT+01:00 Sven Van Caekenberghe <[hidden email]>:

> On 2 Jan 2018, at 12:53, Cyrille Delaunay <[hidden email]> wrote:
>
> Hi Sven,
>
> I am back on this issue :)
>
> I have kind of a "clean" scenario that could help to reproduce.
> It is still random, but at one point, it shows up for me.
>
> Here is how I am doing:
> Download Pharo 6.1 on my Mac (Sierra 10.12.6): https://pharo.org/web/download
> Then, iterate over this process till spotting the issue:
>
> => start the pharo image
> => execute this piece of code in a playground
>
> ZnServer startDefaultOn: 1701.
> ZnServer default maximumEntitySize: 80* 1024 * 1024.
> '/Users/cdelaunay/myzip.zip' asFileReference writeStreamDo: [ :out |
> out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0].
> 18202065 timesRepeat: [ out nextPut: 0 ]
> ].
>
> => Open a web browser page on: http://localhost:1701/form-test-3
> => Upload the file zip file previously generated ('myzip.zip').
> => If the web page displays: "contents=000000000a00..." (or anything unexpected), THIS IS THE ISSUE !
> => If the web page displays: "contents=504b03040a00..", the upload worked fine. You can close the image (without saving)
>
> Mettre à jour Saisir temps Surveiller Copier Supprimer
>
>
> Usually, after starting 3 / 4 times pharo image, I am getting the problem.
> I have been able to reproduce with chrome and firefox.
>
>
> Seing that I was able to reproduce within a fresh pharo image, I tried to automate this scenario.
> I wrote the following test:
>
>
> ZnServerTests >>testFormTest4
>       | inputs client part fileReference fileContents entity |
>       fileReference := 'test-zip.zip' asFileReference.
>       fileReference
>               writeStreamDo: [ :out |
>                       out
>                               binary;
>                               nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0].
>                       18202065 timesRepeat: [ out nextPut: 0 ] ].
>       fileContents := fileReference binaryReadStreamDo: [ :in | in upToEnd ].
>       15
>               timesRepeat: [ self
>                               withServerDo: [ :server |
>                                       server maximumEntitySize: 80 * 1024 * 1024.
>                                       server debugMode: true.
>                                       (client := ZnClient new)
>                                               url: server localUrl;
>                                               addPathSegment: 'form-test-3'.
>                                       client timeout: 120000.
>                                       entity := (ZnByteArrayEntity type: (ZnMimeType main: 'application' sub: 'zip') length: fileContents size) bytes: fileContents.
>                                       part := ZnMimePart fieldName: 'file' fileName: fileReference basename entity: entity.
>                                       client
>                                               resetEntity;
>                                               addPart: part;
>                                               post.
>                                       self assert: client isSuccess.
>                                       self assert: (client contents includesSubstring: '504b03040a') ] ]
>
>
>
>
> In order to make this test run correctly I had to modify this method.
> Without this change, I always got an empty html page as response
>
>
>
> ZnDefaultServerDelegate >> formTest3: request
>       | contents filename contentType page |
>       contents := filename := contentType := ''.
>       (request hasEntity and: [ request contentType matches: ZnMimeType multiPartFormData ])
>               ifTrue: [
>                       (request entity partNamed: #file ifNone: [ nil ])
>                               ifNotNil: [ :part |
>                                       filename := part fileName.
>                                       contents := part contents.
>                                       contentType := part  contentType.
>                                       contentType isBinary ifTrue: [ contents := contents hex ] ] ].
>       page := ZnHtmlOutputStream streamContents: [ :html |
>               html page: 'Form Test 3' do: [
>                       html
>                               tag: #form
>                               attributes: #(action 'form-test-3' 'accept-charset' 'utf-8'
>                                                                       enctype 'multipart/form-data' method POST)
>                               do: [
>                                       html
>                                               str: 'File'; space;
>                                               tag: #input attributes: #(type file name file); space;
>                                               tag: #input attributes: #(type submit) ];
>                               tag: #p do: [ html str: 'filename = '; str: filename ];
>                               tag: #p do: [ html str: 'content-type = '; str: contentType asString ];
>                               tag: #p do: [ html str: 'contents = '; str: (contents copyFrom: 1 to: 20) asString ] ] ].
>       ^ ZnResponse ok: (ZnEntity html: page)
>
>
>
> Unfortunately, the tests always passed till now !

So you have to upload 80Mb ?

Sometimes yes.

But here the file I build and use to reproduce is only 18 mb. 

(the same size as the original file highlighting the issue on my machine).

 

And if you script it in Pharo and repeat it 15 times it keeps on working ?
Really ?

Yes indeed.

But again, it is completely random.

I could not ensure that the problem would never show up with the script.

I will try to make more tries tomorrow.

 

It seems that it only fails when a manually operated browser is involved.

Yes, exactly

 

Have you tried doing the upload with curl ?

No.

How would I do that ?

 

> I will try to go on refining the context parameters that could make the issue show up or not
>
>
>
> 2017-12-22 18:05 GMT+01:00 Sven Van Caekenberghe <[hidden email]>:
> Well, I can't repeat it, for now.
>
> Try to make a minimal self-contained example that fails (reliably), like how I did it, you can even try to do the upload in Pharo too (see #testFormTest3).
>
> > On 22 Dec 2017, at 18:00, Cyril Ferlicot <[hidden email]> wrote:
> >
> >
> > On ven. 22 déc. 2017 at 17:50, Sven Van Caekenberghe <[hidden email]> wrote:
> > Cyrille,
> >
> > I wrote a file like this:
> >
> > FileLocator temp / 'myzip.zip' writeStreamDo: [ :out |
> >   out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0] ].
> >
> > Which I can read as follows:
> >
> > FileLocator temp / 'myzip.zip' binaryReadStreamDo: [ :in | in upToEnd ].
> >
> >   => #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0]
> >
> > If I upload this file to http://localhost:1701/form-test-3 this works perfectly (macOS 10.13.2 Pharo #60402, Safari. Firefox & Chrome), first time and second time.
> >
> >
> >
> > Maybe you should try on a co-workers machine just to make sure it is not a local problem on your machine ?
> >
> > Hi Sven,
> >
> > It happen on Cyrille's machine, my machine, Yann's machine and our boss's machine.
> >
> > We first saw it on Windows and never on mac/Linux. But now we see it also on mac.
> >
> > I did not see it yet on Linux but we use it less for now.
> >
> > Also, when it happen, it tend to happen a lot afterward.
> >
> > Also, by random it means that we can have 1/2 week without problem, then 4 failure in 2h.
> >
> >
> >
> > Sven
> >
> > --
> > Cyril Ferlicot
> > https://ferlicot.fr
> >
> > http://www.synectique.eu
> > 2 rue Jacques Prévert 01,
> > 59650 Villeneuve d'ascq France
> > <Screen Shot 2017-12-22 at 17.48.31.png>_______________________________________________
> > seaside mailing list
> > [hidden email]
> > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
>
>
> --
> Cyrille Delaunay
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

--

Cyrille Delaunay

I tried several times and the test I wrote always passed.

All I have been able to do, is to experiment some failures by playing with the client timeout.

My test initially contains the following statement: client timeout: 120000.

If I comment this statement, sometimes the server received a request with a nil entity.

The test then get an unexpected response.

But I am not sure it an unexpected behavior, as there is a lot of data being sent.

2018-01-02 20:28 GMT+01:00 Cyrille Delaunay <[hidden email]>:

2018-01-02 17:37 GMT+01:00 Sven Van Caekenberghe <[hidden email]>:

> On 2 Jan 2018, at 12:53, Cyrille Delaunay <[hidden email]> wrote:
>
> Hi Sven,
>
> I am back on this issue :)
>
> I have kind of a "clean" scenario that could help to reproduce.
> It is still random, but at one point, it shows up for me.
>
> Here is how I am doing:
> Download Pharo 6.1 on my Mac (Sierra 10.12.6): https://pharo.org/web/download
> Then, iterate over this process till spotting the issue:
>
> => start the pharo image
> => execute this piece of code in a playground
>
> ZnServer startDefaultOn: 1701.
> ZnServer default maximumEntitySize: 80* 1024 * 1024.
> '/Users/cdelaunay/myzip.zip' asFileReference writeStreamDo: [ :out |
> out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0].
> 18202065 timesRepeat: [ out nextPut: 0 ]
> ].
>
> => Open a web browser page on: http://localhost:1701/form-test-3
> => Upload the file zip file previously generated ('myzip.zip').
> => If the web page displays: "contents=000000000a00..." (or anything unexpected), THIS IS THE ISSUE !
> => If the web page displays: "contents=504b03040a00..", the upload worked fine. You can close the image (without saving)
>
> Mettre à jour Saisir temps Surveiller Copier Supprimer
>
>
> Usually, after starting 3 / 4 times pharo image, I am getting the problem.
> I have been able to reproduce with chrome and firefox.
>
>
> Seing that I was able to reproduce within a fresh pharo image, I tried to automate this scenario.
> I wrote the following test:
>
>
> ZnServerTests >>testFormTest4
>       | inputs client part fileReference fileContents entity |
>       fileReference := 'test-zip.zip' asFileReference.
>       fileReference
>               writeStreamDo: [ :out |
>                       out
>                               binary;
>                               nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0].
>                       18202065 timesRepeat: [ out nextPut: 0 ] ].
>       fileContents := fileReference binaryReadStreamDo: [ :in | in upToEnd ].
>       15
>               timesRepeat: [ self
>                               withServerDo: [ :server |
>                                       server maximumEntitySize: 80 * 1024 * 1024.
>                                       server debugMode: true.
>                                       (client := ZnClient new)
>                                               url: server localUrl;
>                                               addPathSegment: 'form-test-3'.
>                                       client timeout: 120000.
>                                       entity := (ZnByteArrayEntity type: (ZnMimeType main: 'application' sub: 'zip') length: fileContents size) bytes: fileContents.
>                                       part := ZnMimePart fieldName: 'file' fileName: fileReference basename entity: entity.
>                                       client
>                                               resetEntity;
>                                               addPart: part;
>                                               post.
>                                       self assert: client isSuccess.
>                                       self assert: (client contents includesSubstring: '504b03040a') ] ]
>
>
>
>
> In order to make this test run correctly I had to modify this method.
> Without this change, I always got an empty html page as response
>
>
>
> ZnDefaultServerDelegate >> formTest3: request
>       | contents filename contentType page |
>       contents := filename := contentType := ''.
>       (request hasEntity and: [ request contentType matches: ZnMimeType multiPartFormData ])
>               ifTrue: [
>                       (request entity partNamed: #file ifNone: [ nil ])
>                               ifNotNil: [ :part |
>                                       filename := part fileName.
>                                       contents := part contents.
>                                       contentType := part  contentType.
>                                       contentType isBinary ifTrue: [ contents := contents hex ] ] ].
>       page := ZnHtmlOutputStream streamContents: [ :html |
>               html page: 'Form Test 3' do: [
>                       html
>                               tag: #form
>                               attributes: #(action 'form-test-3' 'accept-charset' 'utf-8'
>                                                                       enctype 'multipart/form-data' method POST)
>                               do: [
>                                       html
>                                               str: 'File'; space;
>                                               tag: #input attributes: #(type file name file); space;
>                                               tag: #input attributes: #(type submit) ];
>                               tag: #p do: [ html str: 'filename = '; str: filename ];
>                               tag: #p do: [ html str: 'content-type = '; str: contentType asString ];
>                               tag: #p do: [ html str: 'contents = '; str: (contents copyFrom: 1 to: 20) asString ] ] ].
>       ^ ZnResponse ok: (ZnEntity html: page)
>
>
>
> Unfortunately, the tests always passed till now !

So you have to upload 80Mb ?

Sometimes yes.

But here the file I build and use to reproduce is only 18 mb. 

(the same size as the original file highlighting the issue on my machine).

 

And if you script it in Pharo and repeat it 15 times it keeps on working ?
Really ?

Yes indeed.

But again, it is completely random.

I could not ensure that the problem would never show up with the script.

I will try to make more tries tomorrow.

 

It seems that it only fails when a manually operated browser is involved.

Yes, exactly

 

Have you tried doing the upload with curl ?

No.

How would I do that ?

 

> I will try to go on refining the context parameters that could make the issue show up or not
>
>
>
> 2017-12-22 18:05 GMT+01:00 Sven Van Caekenberghe <[hidden email]>:
> Well, I can't repeat it, for now.
>
> Try to make a minimal self-contained example that fails (reliably), like how I did it, you can even try to do the upload in Pharo too (see #testFormTest3).
>
> > On 22 Dec 2017, at 18:00, Cyril Ferlicot <[hidden email]> wrote:
> >
> >
> > On ven. 22 déc. 2017 at 17:50, Sven Van Caekenberghe <[hidden email]> wrote:
> > Cyrille,
> >
> > I wrote a file like this:
> >
> > FileLocator temp / 'myzip.zip' writeStreamDo: [ :out |
> >   out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0] ].
> >
> > Which I can read as follows:
> >
> > FileLocator temp / 'myzip.zip' binaryReadStreamDo: [ :in | in upToEnd ].
> >
> >   => #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0]
> >
> > If I upload this file to http://localhost:1701/form-test-3 this works perfectly (macOS 10.13.2 Pharo #60402, Safari. Firefox & Chrome), first time and second time.
> >
> >
> >
> > Maybe you should try on a co-workers machine just to make sure it is not a local problem on your machine ?
> >
> > Hi Sven,
> >
> > It happen on Cyrille's machine, my machine, Yann's machine and our boss's machine.
> >
> > We first saw it on Windows and never on mac/Linux. But now we see it also on mac.
> >
> > I did not see it yet on Linux but we use it less for now.
> >
> > Also, when it happen, it tend to happen a lot afterward.
> >
> > Also, by random it means that we can have 1/2 week without problem, then 4 failure in 2h.
> >
> >
> >
> > Sven
> >
> > --
> > Cyril Ferlicot
> > https://ferlicot.fr
> >
> > http://www.synectique.eu
> > 2 rue Jacques Prévert 01,
> > 59650 Villeneuve d'ascq France
> > <Screen Shot 2017-12-22 at 17.48.31.png>_______________________________________________
> > seaside mailing list
> > [hidden email]
> > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
>
>
> --
> Cyrille Delaunay
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

--

Cyrille Delaunay

--

Cyrille Delaunay

I made some experiments which might help to restrict the scope:

I applied my little "manual" scenario with a Pharo 5 image (https://files.pharo.org/get-files/50/)

After 7 tries, the issue never show up.

With a Pharo 6.1 image, the issue shows up after maximum 3 / 4 tries.

My feeling: something changed between these 2 versions.

This idea would tend to be confirmed by the period we started to experiment the issue: We recently migrated some production images from Pharo 4 to Pharo 6.

Is it Zinc or some Pharo inside bricks used by Zinc ? I could not say.

2018-01-04 16:48 GMT+01:00 Cyrille Delaunay <[hidden email]>:

I tried several times and the test I wrote always passed.

All I have been able to do, is to experiment some failures by playing with the client timeout.

My test initially contains the following statement: client timeout: 120000.

If I comment this statement, sometimes the server received a request with a nil entity.

The test then get an unexpected response.

But I am not sure it an unexpected behavior, as there is a lot of data being sent.

2018-01-02 20:28 GMT+01:00 Cyrille Delaunay <[hidden email]>:

2018-01-02 17:37 GMT+01:00 Sven Van Caekenberghe <[hidden email]>:

> On 2 Jan 2018, at 12:53, Cyrille Delaunay <[hidden email]> wrote:
>
> Hi Sven,
>
> I am back on this issue :)
>
> I have kind of a "clean" scenario that could help to reproduce.
> It is still random, but at one point, it shows up for me.
>
> Here is how I am doing:
> Download Pharo 6.1 on my Mac (Sierra 10.12.6): https://pharo.org/web/download
> Then, iterate over this process till spotting the issue:
>
> => start the pharo image
> => execute this piece of code in a playground
>
> ZnServer startDefaultOn: 1701.
> ZnServer default maximumEntitySize: 80* 1024 * 1024.
> '/Users/cdelaunay/myzip.zip' asFileReference writeStreamDo: [ :out |
> out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0].
> 18202065 timesRepeat: [ out nextPut: 0 ]
> ].
>
> => Open a web browser page on: http://localhost:1701/form-test-3
> => Upload the file zip file previously generated ('myzip.zip').
> => If the web page displays: "contents=000000000a00..." (or anything unexpected), THIS IS THE ISSUE !
> => If the web page displays: "contents=504b03040a00..", the upload worked fine. You can close the image (without saving)
>
> Mettre à jour Saisir temps Surveiller Copier Supprimer
>
>
> Usually, after starting 3 / 4 times pharo image, I am getting the problem.
> I have been able to reproduce with chrome and firefox.
>
>
> Seing that I was able to reproduce within a fresh pharo image, I tried to automate this scenario.
> I wrote the following test:
>
>
> ZnServerTests >>testFormTest4
>       | inputs client part fileReference fileContents entity |
>       fileReference := 'test-zip.zip' asFileReference.
>       fileReference
>               writeStreamDo: [ :out |
>                       out
>                               binary;
>                               nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0].
>                       18202065 timesRepeat: [ out nextPut: 0 ] ].
>       fileContents := fileReference binaryReadStreamDo: [ :in | in upToEnd ].
>       15
>               timesRepeat: [ self
>                               withServerDo: [ :server |
>                                       server maximumEntitySize: 80 * 1024 * 1024.
>                                       server debugMode: true.
>                                       (client := ZnClient new)
>                                               url: server localUrl;
>                                               addPathSegment: 'form-test-3'.
>                                       client timeout: 120000.
>                                       entity := (ZnByteArrayEntity type: (ZnMimeType main: 'application' sub: 'zip') length: fileContents size) bytes: fileContents.
>                                       part := ZnMimePart fieldName: 'file' fileName: fileReference basename entity: entity.
>                                       client
>                                               resetEntity;
>                                               addPart: part;
>                                               post.
>                                       self assert: client isSuccess.
>                                       self assert: (client contents includesSubstring: '504b03040a') ] ]
>
>
>
>
> In order to make this test run correctly I had to modify this method.
> Without this change, I always got an empty html page as response
>
>
>
> ZnDefaultServerDelegate >> formTest3: request
>       | contents filename contentType page |
>       contents := filename := contentType := ''.
>       (request hasEntity and: [ request contentType matches: ZnMimeType multiPartFormData ])
>               ifTrue: [
>                       (request entity partNamed: #file ifNone: [ nil ])
>                               ifNotNil: [ :part |
>                                       filename := part fileName.
>                                       contents := part contents.
>                                       contentType := part  contentType.
>                                       contentType isBinary ifTrue: [ contents := contents hex ] ] ].
>       page := ZnHtmlOutputStream streamContents: [ :html |
>               html page: 'Form Test 3' do: [
>                       html
>                               tag: #form
>                               attributes: #(action 'form-test-3' 'accept-charset' 'utf-8'
>                                                                       enctype 'multipart/form-data' method POST)
>                               do: [
>                                       html
>                                               str: 'File'; space;
>                                               tag: #input attributes: #(type file name file); space;
>                                               tag: #input attributes: #(type submit) ];
>                               tag: #p do: [ html str: 'filename = '; str: filename ];
>                               tag: #p do: [ html str: 'content-type = '; str: contentType asString ];
>                               tag: #p do: [ html str: 'contents = '; str: (contents copyFrom: 1 to: 20) asString ] ] ].
>       ^ ZnResponse ok: (ZnEntity html: page)
>
>
>
> Unfortunately, the tests always passed till now !

So you have to upload 80Mb ?

Sometimes yes.

But here the file I build and use to reproduce is only 18 mb. 

(the same size as the original file highlighting the issue on my machine).

 

And if you script it in Pharo and repeat it 15 times it keeps on working ?
Really ?

Yes indeed.

But again, it is completely random.

I could not ensure that the problem would never show up with the script.

I will try to make more tries tomorrow.

 

It seems that it only fails when a manually operated browser is involved.

Yes, exactly

 

Have you tried doing the upload with curl ?

No.

How would I do that ?

 

> I will try to go on refining the context parameters that could make the issue show up or not
>
>
>
> 2017-12-22 18:05 GMT+01:00 Sven Van Caekenberghe <[hidden email]>:
> Well, I can't repeat it, for now.
>
> Try to make a minimal self-contained example that fails (reliably), like how I did it, you can even try to do the upload in Pharo too (see #testFormTest3).
>
> > On 22 Dec 2017, at 18:00, Cyril Ferlicot <[hidden email]> wrote:
> >
> >
> > On ven. 22 déc. 2017 at 17:50, Sven Van Caekenberghe <[hidden email]> wrote:
> > Cyrille,
> >
> > I wrote a file like this:
> >
> > FileLocator temp / 'myzip.zip' writeStreamDo: [ :out |
> >   out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0] ].
> >
> > Which I can read as follows:
> >
> > FileLocator temp / 'myzip.zip' binaryReadStreamDo: [ :in | in upToEnd ].
> >
> >   => #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0]
> >
> > If I upload this file to http://localhost:1701/form-test-3 this works perfectly (macOS 10.13.2 Pharo #60402, Safari. Firefox & Chrome), first time and second time.
> >
> >
> >
> > Maybe you should try on a co-workers machine just to make sure it is not a local problem on your machine ?
> >
> > Hi Sven,
> >
> > It happen on Cyrille's machine, my machine, Yann's machine and our boss's machine.
> >
> > We first saw it on Windows and never on mac/Linux. But now we see it also on mac.
> >
> > I did not see it yet on Linux but we use it less for now.
> >
> > Also, when it happen, it tend to happen a lot afterward.
> >
> > Also, by random it means that we can have 1/2 week without problem, then 4 failure in 2h.
> >
> >
> >
> > Sven
> >
> > --
> > Cyril Ferlicot
> > https://ferlicot.fr
> >
> > http://www.synectique.eu
> > 2 rue Jacques Prévert 01,
> > 59650 Villeneuve d'ascq France
> > <Screen Shot 2017-12-22 at 17.48.31.png>_______________________________________________
> > seaside mailing list
> > [hidden email]
> > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
>
>
> --
> Cyrille Delaunay
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

--

Cyrille Delaunay

--

Cyrille Delaunay

--

Cyrille Delaunay

I've been able to boil it down to one method. This fails sometimes and succeeds sometimes. Whichever it does, it tends to keep doing that for a while. Quitting and restarting may get it to change.

test4
"
self test4
"
    | species bufferSize buffer totalRead outputStream answer inputStream ba byte1 |
   
    false ifTrue: [
        ba := (FileStream readOnlyFileNamed: '/Users/bob/z99.zip') binary; contentsOfEntireFile.
     ] ifFalse: [
        ba := ByteArray new: 18202085.
        ba atAllPut: 99.
        1 to: 20 do: [  :i | ba at: i put: (#[80 75 3 4 10 7 7 7 7 7 125 83 67 73 7 7 7 7 7 7] at: i) ].
     ].
    inputStream := ba readStream.
    bufferSize := 16384.
    species := ByteArray.
    buffer := species new: bufferSize.
    totalRead := 0.
    outputStream := nil.
    [ inputStream atEnd ] whileFalse: [ | readCount |
        readCount := inputStream readInto: buffer startingAt: 1 count: bufferSize.
        totalRead = 0 ifTrue: [
            byte1 := buffer first.
        ].
        totalRead := totalRead + readCount.
       
        outputStream ifNil: [
            inputStream atEnd
                ifTrue: [ ^ buffer copyFrom: 1 to: readCount ]
                ifFalse: [ outputStream := (species new: bufferSize) writeStream ] ].
        outputStream next: readCount putAll: buffer startingAt: 1.
        byte1 = outputStream firstByte ifFalse: [ self halt ].
    ].
    answer := outputStream ifNil: [ species new ] ifNotNil: [ outputStream contents ].
    byte1 = answer first ifFalse: [ self halt ].
    ^answer

On 1/10/18 4:42 AM, Cyrille Delaunay wrote:

Has anyone been able to reproduce my little scenario ?

As a reminder, suspicion is that something "wrong" has been introduced with Pharo6

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside