[ENH] Large file upload patch (remake)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[ENH] Large file upload patch (remake)

Göran Krampe
Hi folks!

Ok, this came up once again on IRC - how to do chunkwise uploading and
thus continously write to a file and handle LARGE files.

I googled and first I found Colin Curtin writing code for this (and also
streamed downloading code) back in 2004! Might be worth looking at more
closely. I also found Colin Putney's code from aug 2007, which seems to
originate from DabbleDB (I guess) - Squeak 3.7.

I more or less took Colin's approach, rewrote the low level bit (so that
it does not end up doing bytewise searching as I presume Colin's code
does) and more or less copied WATempFile as is. The multipartFormFieldsDo:
code now ALWAYS uses WATempFile, regardless if it has a file name or not -
and regardless of how small they are. This could of course be discussed.

But the MAIN THING is this:

Upload of 295Mb file (locally) on a box with tinyBenchmarks ('117972350
bytecodes/sec; 4788828 sends/sec') took around 45 seconds. Uploading
squeak.image which was 16Mb took 2 seconds or so. CPU load kinda high
(profile shows that it is the copying from the buffer into FileStream) but
no visible memory consumption.

I also noted that this is being recently discussed too - but hey, I had
already started coding and wanted to finish it :).

regards, Göran
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

StreamedUpload.4.cs.gz (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [ENH] Large file upload patch (remake)

Philippe Marschall
Looks interesting. If Giovanni can take care of the Kom side it can
integrate it into the Squeak side of Seaside 2.9.

Cheers
Philippe

2008/4/7, Göran Krampe <[hidden email]>:

> Hi folks!
>
>  Ok, this came up once again on IRC - how to do chunkwise uploading and
>  thus continously write to a file and handle LARGE files.
>
>  I googled and first I found Colin Curtin writing code for this (and also
>  streamed downloading code) back in 2004! Might be worth looking at more
>  closely. I also found Colin Putney's code from aug 2007, which seems to
>  originate from DabbleDB (I guess) - Squeak 3.7.
>
>  I more or less took Colin's approach, rewrote the low level bit (so that
>  it does not end up doing bytewise searching as I presume Colin's code
>  does) and more or less copied WATempFile as is. The multipartFormFieldsDo:
>  code now ALWAYS uses WATempFile, regardless if it has a file name or not -
>  and regardless of how small they are. This could of course be discussed.
>
>  But the MAIN THING is this:
>
>  Upload of 295Mb file (locally) on a box with tinyBenchmarks ('117972350
>  bytecodes/sec; 4788828 sends/sec') took around 45 seconds. Uploading
>  squeak.image which was 16Mb took 2 seconds or so. CPU load kinda high
>  (profile shows that it is the copying from the buffer into FileStream) but
>  no visible memory consumption.
>
>  I also noted that this is being recently discussed too - but hey, I had
>  already started coding and wanted to finish it :).
>
>  regards, Göran
> _______________________________________________
>  seaside mailing list
>  [hidden email]
>  http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
>
>

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: [ENH] Large file upload patch (remake)

Göran Krampe
Hi!

> Looks interesting. If Giovanni can take care of the Kom side it can
> integrate it into the Squeak side of Seaside 2.9.
>
> Cheers
> Philippe

Yeah, the ideal approach is probably:

1. Add new method in SocketStream as a Kom class extension for now and
modify the method in MultipartChunk that uses it. This is easy. I can make
this happen and talk to Giovanni etc.

2. Add/modify the bits in Seaside 2.9.

Then we can eventually move the new method in SocketStream into Squeak,
where it belongs.

I can extent the file upload test example for this, and add some class
comments to show how to use WATempFile etc, thus making this more
polished.

regards, Göran

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside