Smalltalk › Frameworks & Tools › Seaside › Seaside General

Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

29 messages Options

Philippe Marschall

Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo

2008/7/23 "S.J.Chun" <[hidden email]>:
> Hmm,
>
> it seems that with current Seaside(I'm using 2.8) large file upload mught be hard
> task.
>
> And for file size, not all files have that size.(Sorry for my poor english) But some of
> the files will have that size.

10 GB is massive. There are whole lot of 32bit integers that can
overflow and a lot of other things that can go wrong at any stage
including the browser and Apache.

> My current idea is that I should create module(?) for Comanche or Swazoo for File
> Upload (this just process multipart/form request for file upload with additional
> parameters) and small web application or module for processing download url.(This
> might be possible with Seaside with RESTable URL support or simple module).

I only tried the upload streaming for Kom on localhost. While it could
handle largish files (40 MB) it wasn't able to get close to 400 MB. So
Kom might not be your tool. I have not seen an upload streaming API
for Swazoo (which doesn't mean it's not there). Squeak might not be
the best tool for the job where. HTTP might not be as well.

For the download I would simply set a X-Sendfile header and let Apache
handle it.

Cheers
Philippe
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Janko Mivšek

Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo

Philippe Marschall wrote:

> 10 GB is massive. There are whole lot of 32bit integers that can
> overflow and a lot of other things that can go wrong at any stage
> including the browser and Apache.
>
>> My current idea is that I should create module(?) for Comanche or Swazoo for File
>> Upload (this just process multipart/form request for file upload with additional
>> parameters) and small web application or module for processing download url.(This
>> might be possible with Seaside with RESTable URL support or simple module).
>
> I only tried the upload streaming for Kom on localhost. While it could
> handle largish files (40 MB) it wasn't able to get close to 400 MB. So
> Kom might not be your tool. I have not seen an upload streaming API
> for Swazoo (which doesn't mean it's not there). Squeak might not be
> the best tool for the job where. HTTP might not be as well.

Swazoo 2.x has streaming upload and download capability and can easily
handle such big files. I tested with few GB by myself. On both Squeak
and VW.

Maybe it is a time to add that streaming to your Swazoo adaptor too? See
http://www.swazoo.org/streaming.html for more. See also HTTPRequestTest
with examples, how to use it (testing-posts).

Best regards
Janko

--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall

Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo

2008/7/23 Janko Mivšek <[hidden email]>:

>
> Philippe Marschall wrote:
>
>> 10 GB is massive. There are whole lot of 32bit integers that can
>> overflow and a lot of other things that can go wrong at any stage
>> including the browser and Apache.
>>
>>> My current idea is that I should create module(?) for Comanche or Swazoo
>>> for File
>>> Upload (this just process multipart/form request for file upload with
>>> additional
>>> parameters) and small web application or module for processing download
>>> url.(This
>>> might be possible with Seaside with RESTable URL support or simple
>>> module).
>>
>> I only tried the upload streaming for Kom on localhost. While it could
>> handle largish files (40 MB) it wasn't able to get close to 400 MB. So
>> Kom might not be your tool. I have not seen an upload streaming API
>> for Swazoo (which doesn't mean it's not there). Squeak might not be
>> the best tool for the job where. HTTP might not be as well.
>
> Swazoo 2.x has streaming upload and download capability and can easily
> handle such big files. I tested with few GB by myself. On both Squeak and
> VW.
>
> Maybe it is a time to add that streaming to your Swazoo adaptor too? See
> http://www.swazoo.org/streaming.html for more. See also HTTPRequestTest with
> examples, how to use it (testing-posts).

That assumes you already know the names of the fields containing files
which I don't. I can only see #postDataKeys and #postKeysAndValuesDo:
which seem to read the full request. I also couldn't find a test (or
sender) for #postDataAt:beforeStreamingDo:.

Cheers
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Janko Mivšek

Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo

Philippe Marschall wrote:

>> Swazoo 2.x has streaming upload and download capability and can easily
>> handle such big files. I tested with few GB by myself. On both Squeak and
>> VW.
>>
>> Maybe it is a time to add that streaming to your Swazoo adaptor too? See
>> http://www.swazoo.org/streaming.html for more. See also HTTPRequestTest with
>> examples, how to use it (testing-posts).

> That assumes you already know the names of the fields containing files
> which I don't. I can only see #postDataKeys and #postKeysAndValuesDo:
> which seem to read the full request. I also couldn't find a test (or
> sender) for #postDataAt:beforeStreamingDo:.

Not knowing names of the fields is the whole point of "deferring"
parsing the POST request until this information is known. Deferring
means that request is not read and completely parsed already by Swazoo
but lately, when some post data is first needed. That allows you to
delay streaming too, until you have enough info where to stream and to
open the destination file accordingly.

See this example from http://www.swazoo.org/streaming.html:

Site>>answerTo: aHTTPRequest
| stream |
aHTTPRequest
postDataAt: 'file'
beforeStreamingDo: [:datum |
stream := (SpFilename named: datum filename) writeStream.
datum writeStream: stream].

aHTTPRequest ensureFullRead. "to be sure streaming occurs"
stream close. "mandatory, to close open stream! "

^HTTPResponse ok

In that example we open a file named as the original file, which is
sent in request as post data, just before a file content.

I hope that helps

Best regards
Janko

--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall

Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo

2008/7/23 Janko Mivšek <[hidden email]>:

> Philippe Marschall wrote:
>
>>> Swazoo 2.x has streaming upload and download capability and can easily
>>> handle such big files. I tested with few GB by myself. On both Squeak and
>>> VW.
>>>
>>> Maybe it is a time to add that streaming to your Swazoo adaptor too? See
>>> http://www.swazoo.org/streaming.html for more. See also HTTPRequestTest
>>> with
>>> examples, how to use it (testing-posts).
>
>> That assumes you already know the names of the fields containing files
>> which I don't. I can only see #postDataKeys and #postKeysAndValuesDo:
>> which seem to read the full request. I also couldn't find a test (or
>> sender) for #postDataAt:beforeStreamingDo:.
>
> Not knowing names of the fields is the whole point of "deferring" parsing
> the POST request until this information is known. Deferring means that
> request is not read and completely parsed already by Swazoo but lately, when
> some post data is first needed. That allows you to delay streaming too,
> until you have enough info where to stream and to open the destination file
> accordingly.
>
> See this example from http://www.swazoo.org/streaming.html:
>
> Site>>answerTo: aHTTPRequest
> | stream |
> aHTTPRequest
> postDataAt: 'file'
> beforeStreamingDo: [:datum |
> stream := (SpFilename named: datum filename) writeStream.
> datum writeStream: stream].
>
> aHTTPRequest ensureFullRead. "to be sure streaming occurs"
> stream close. "mandatory, to close open stream! "
>
> ^HTTPResponse ok
>
>
> In that example we open a file named as the original file, which is sent in
> request as post data, just before a file content.
>
> I hope that helps

Nope sorry. See the attached code. Your example and the attachment
work only if there is an upload filed with the name 'file'. That's the
only upload filed that will be streamed. That's why the attached code
breaks. If you change the name of the upload field to 'file' then the
code breaks somewhere in MultiByteFileStream (Squeak 3.9 Mac VM).

Cheers
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

STStreamSite.st (2K) Download Attachment

Göran Krampe

Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo

In reply to this post by Philippe Marschall

Hi!

Philippe Marschall wrote:

> 2008/7/23 "S.J.Chun" <[hidden email]>:
>> Hmm,
>>
>> it seems that with current Seaside(I'm using 2.8) large file upload mught be hard
>> task.
>>
>> And for file size, not all files have that size.(Sorry for my poor english) But some of
>> the files will have that size.
>
> 10 GB is massive. There are whole lot of 32bit integers that can
> overflow and a lot of other things that can go wrong at any stage
> including the browser and Apache.
>
>> My current idea is that I should create module(?) for Comanche or Swazoo for File
>> Upload (this just process multipart/form request for file upload with additional
>> parameters) and small web application or module for processing download url.(This
>> might be possible with Seaside with RESTable URL support or simple module).
>
> I only tried the upload streaming for Kom on localhost. While it could
> handle largish files (40 MB) it wasn't able to get close to 400 MB. So
> Kom might not be your tool. I have not seen an upload streaming API

Hmmm, not sure - but IIRC I uploaded 300Mb files easily using the
streaming code I made. I don't have it handy now for testing - but I am
not sure why it would not work?

regards, Göran
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Janko Mivšek

Re: [Q] File Upload/Download Server, Comanche or Swazoo

In reply to this post by Philippe Marschall

Philippe Marschall wrote:

>> See this example from http://www.swazoo.org/streaming.html:
>>
>> Site>>answerTo: aHTTPRequest
>> | stream |
>> aHTTPRequest
>> postDataAt: 'file'
>> beforeStreamingDo: [:datum |
>> stream := (SpFilename named: datum filename) writeStream.
>> datum writeStream: stream].
>>
>> aHTTPRequest ensureFullRead. "to be sure streaming occurs"
>> stream close. "mandatory, to close open stream! "
>>
>> ^HTTPResponse ok
>>
>>
>> In that example we open a file named as the original file, which is sent in
>> request as post data, just before a file content.
>>
>> I hope that helps
>
> Nope sorry. See the attached code. Your example and the attachment
> work only if there is an upload filed with the name 'file'. That's the
> only upload filed that will be streamed. That's why the attached code
> breaks. If you change the name of the upload field to 'file' then the
> code breaks somewhere in MultiByteFileStream (Squeak 3.9 Mac VM).

Philippe, first sorry for late answer. Time pressures ...

I looked at your code, where you make a subclass of Site where is a test
form with file input field , then post via streaming to output file.

Problem is that you named file input field as "upload" while in my
example it is expected to be "file". So, just adjust those two names and
your example should work.

Best regards
Janko

--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall

Re: [Q] File Upload/Download Server, Comanche or Swazoo

2008/7/29 Janko Mivšek <[hidden email]>:

> Philippe Marschall wrote:
>
>>> See this example from http://www.swazoo.org/streaming.html:
>>>
>>> Site>>answerTo: aHTTPRequest
>>> | stream |
>>> aHTTPRequest
>>> postDataAt: 'file'
>>> beforeStreamingDo: [:datum |
>>> stream := (SpFilename named: datum filename) writeStream.
>>> datum writeStream: stream].
>>>
>>> aHTTPRequest ensureFullRead. "to be sure streaming occurs"
>>> stream close. "mandatory, to close open stream! "
>>>
>>> ^HTTPResponse ok
>>>
>>>
>>> In that example we open a file named as the original file, which is sent
>>> in
>>> request as post data, just before a file content.
>>>
>>> I hope that helps
>>
>> Nope sorry. See the attached code. Your example and the attachment
>> work only if there is an upload filed with the name 'file'. That's the
>> only upload filed that will be streamed. That's why the attached code
>> breaks. If you change the name of the upload field to 'file' then the
>> code breaks somewhere in MultiByteFileStream (Squeak 3.9 Mac VM).
>
> Philippe, first sorry for late answer. Time pressures ...
>
> I looked at your code, where you make a subclass of Site where is a test
> form with file input field , then post via streaming to output file.
>
> Problem is that you named file input field as "upload" while in my example
> it is expected to be "file". So, just adjust those two names and your
> example should work.

Nope sorry, the example code was written explicitly to illustrate the
problem. Upload streaming with Swazoo works only iff you know
beforehand which fields will be files. There is now way how I can know
this. Compare this to the upload streaming of Kom which detects which
fields are files.

And even then the Swazoo code still breaks in MultiByteFileStream.

Cheers
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Janko Mivšek

Re: [Q] File Upload/Download Server, Comanche or Swazoo

Philippe Marschall wrote:

>> I looked at your code, where you make a subclass of Site where is a test
>> form with file input field , then post via streaming to output file.
>>
>> Problem is that you named file input field as "upload" while in my example
>> it is expected to be "file". So, just adjust those two names and your
>> example should work.
>
> Nope sorry, the example code was written explicitly to illustrate the
> problem. Upload streaming with Swazoo works only iff you know
> beforehand which fields will be files. There is now way how I can know
> this. Compare this to the upload streaming of Kom which detects which
> fields are files.

I don't understand how there is no way to know which input fields will
be file ones. Web framework is naming them during HTML generation
anyway, I suppose Seaside too?

Bigger problem is that you don't have that knowledge until request come
quite deeply in the web framework and here the idea of "deferred"
request parsing, until information of input field names is known.

> And even then the Swazoo code still breaks in MultiByteFileStream.

I'm otherwise using Sport's Filenane and there is no signs of any broken
code or corruption of uploaded files.

Best regards
Janko

--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall

Re: [Q] File Upload/Download Server, Comanche or Swazoo

2008/7/29 Janko Mivšek <[hidden email]>:

Yes, in a non-predicable way (for the server adapter). Even if it
would generate them in a predictable way there would still be a
problem if you have multiple file upload fields in the same form.

Cheers
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Janko Mivšek

Re: [Q] File Upload/Download Server, Comanche or Swazoo

Philippe Marschall wrote:

>> I don't understand how there is no way to know which input fields will be
>> file ones. Web framework is naming them during HTML generation anyway, I
>> suppose Seaside too?
>
> Yes, in a non-predicable way (for the server adapter). Even if it
> would generate them in a predictable way there would still be a
> problem if you have multiple file upload fields in the same form.

In Aida there is completely predictable way to generate names and pair
them with forthcoming POST. I therefore don't see the reason why Seaside
can't do something similar.

See for example image upload in http://squeaksite.aidaweb.si/About/
(login as demo/demo, then edit that page and see Attach image IFRAME).
If we look at source of this IFRAME, there we have the following input
fields:

...
<form method=post action="/About/?view=images&version=2"
enctype="multipart/form-data">
...
<input type="hidden" name="id5273" value="images">
<input type="file" name="field3">
<input type="submit" name="field4" value="Upload">
...
</form>

As you can see field names are auto generated and Aida register them
internally so that fields can be paired with post data in forthcoming POST.

For streaming this pairing must not occur until you don't know the
destination file, so you must not read post data from POST until then.
This is maybe your problem, you read too fast post data in your adapter.
End result in that case is that you stream the upload "in the air",
because target stream is unknown.

Janko

--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall

Re: [Q] File Upload/Download Server, Comanche or Swazoo

In reply to this post by Janko Mivšek

2008/7/29 Janko Mivšek <[hidden email]>:

> Philippe Marschall wrote:
>
>>> I looked at your code, where you make a subclass of Site where is a test
>>> form with file input field , then post via streaming to output file.
>>>
>>> Problem is that you named file input field as "upload" while in my
>>> example
>>> it is expected to be "file". So, just adjust those two names and your
>>> example should work.
>>
>> Nope sorry, the example code was written explicitly to illustrate the
>> problem. Upload streaming with Swazoo works only iff you know
>> beforehand which fields will be files. There is now way how I can know
>> this. Compare this to the upload streaming of Kom which detects which
>> fields are files.
>
> I don't understand how there is no way to know which input fields will be
> file ones. Web framework is naming them during HTML generation anyway, I
> suppose Seaside too?
>
> Bigger problem is that you don't have that knowledge until request come
> quite deeply in the web framework and here the idea of "deferred" request
> parsing, until information of input field names is known.
>
>> And even then the Swazoo code still breaks in MultiByteFileStream.
>
> I'm otherwise using Sport's Filenane and there is no signs of any broken
> code or corruption of uploaded files.

I use the posted code which uses Sport's Filename

29 July 2008 8:18:43 pm

VM: Mac OS - a SmalltalkImage
Image: Squeak3.9.1 [latest update: #7075]

SecurityManager state:
Restricted: false
FileAccess: true
SocketAccess: true

ByteString(Object)>>error:
Receiver: ''

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall

Re: [Q] File Upload/Download Server, Comanche or Swazoo

2008/7/29 Philippe Marschall <[hidden email]>:

> 2008/7/29 Janko Mivšek <[hidden email]>:
>> Philippe Marschall wrote:
>>
>>>> I looked at your code, where you make a subclass of Site where is a test
>>>> form with file input field , then post via streaming to output file.
>>>>
>>>> Problem is that you named file input field as "upload" while in my
>>>> example
>>>> it is expected to be "file". So, just adjust those two names and your
>>>> example should work.
>>>
>>> Nope sorry, the example code was written explicitly to illustrate the
>>> problem. Upload streaming with Swazoo works only iff you know
>>> beforehand which fields will be files. There is now way how I can know
>>> this. Compare this to the upload streaming of Kom which detects which
>>> fields are files.
>>
>> I don't understand how there is no way to know which input fields will be
>> file ones. Web framework is naming them during HTML generation anyway, I
>> suppose Seaside too?
>>
>> Bigger problem is that you don't have that knowledge until request come
>> quite deeply in the web framework and here the idea of "deferred" request
>> parsing, until information of input field names is known.
>>
>>> And even then the Swazoo code still breaks in MultiByteFileStream.
>>
>> I'm otherwise using Sport's Filenane and there is no signs of any broken
>> code or corruption of uploaded files.
>
> I use the posted code which uses Sport's Filename
>
> 29 July 2008 8:18:43 pm
>
> VM: Mac OS - a SmalltalkImage
> Image: Squeak3.9.1 [latest update: #7075]
>
> SecurityManager state:
> Restricted: false
> FileAccess: true
> SocketAccess: true
>
> ByteString(Object)>>error:
> Receiver: ''
>

I attached the stack trace. I didn't seem to be send correctly.

Cheers
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

stacktrace.txt (3K) Download Attachment

Janko Mivšek

Re: [Q] File Upload/Download Server, Comanche or Swazoo

Philippe,

Try with newest 2.2beta, you are obviously still using 2.1.

SpFileStream>>nextPut: "nextPutAll: in 2.2"
HTTPPost>>nextPut:to:afterTwoTo: "no more this method in 2.2"
HTTPPost>>readEntityFrom:datum:boundary:
HTTPPost>>partFromStream:boundary:

Best regards
Janko

>>>> And even then the Swazoo code still breaks in MultiByteFileStream.

>>> I'm otherwise using Sport's Filenane and there is no signs of any broken
>>> code or corruption of uploaded files.

>> I use the posted code which uses Sport's Filename

--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall

Re: [Q] File Upload/Download Server, Comanche or Swazoo

2008/7/29 Janko Mivšek <[hidden email]>:
> Philippe,
>
> Try with newest 2.2beta, you are obviously still using 2.1.

That seems to fix it.

Thanks
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall

Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo

In reply to this post by Göran Krampe

2008/7/29 Göran Krampe <[hidden email]>:

> Hi!
>
> Philippe Marschall wrote:
>> 2008/7/23 "S.J.Chun" <[hidden email]>:
>>> Hmm,
>>>
>>> it seems that with current Seaside(I'm using 2.8) large file upload mught be hard
>>> task.
>>>
>>> And for file size, not all files have that size.(Sorry for my poor english) But some of
>>> the files will have that size.
>>
>> 10 GB is massive. There are whole lot of 32bit integers that can
>> overflow and a lot of other things that can go wrong at any stage
>> including the browser and Apache.
>>
>>> My current idea is that I should create module(?) for Comanche or Swazoo for File
>>> Upload (this just process multipart/form request for file upload with additional
>>> parameters) and small web application or module for processing download url.(This
>>> might be possible with Seaside with RESTable URL support or simple module).
>>
>> I only tried the upload streaming for Kom on localhost. While it could
>> handle largish files (40 MB) it wasn't able to get close to 400 MB. So
>> Kom might not be your tool. I have not seen an upload streaming API
>
> Hmmm, not sure - but IIRC I uploaded 300Mb files easily using the
> streaming code I made. I don't have it handy now for testing - but I am
> not sure why it would not work?

That would interest me too.

I made some very limited testing on Squeak 3.9.1 and the Mac VM, these
are my observations:

300 MB, hogs CPU badly, image grows dangerously but works
600 MB, hogs CPU badly, low space semaphore, image unusable
4 GB, Squeak image crashes, system unusable, reboot required

Cheers
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Michael Rueger-6

Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo

Philippe Marschall wrote:

> That would interest me too.
>
> I made some very limited testing on Squeak 3.9.1 and the Mac VM, these
> are my observations:
>
> 300 MB, hogs CPU badly, image grows dangerously but works
> 600 MB, hogs CPU badly, low space semaphore, image unusable
> 4 GB, Squeak image crashes, system unusable, reboot required

Clearly what you get when streaming into a memory stream first.

Michael
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall

Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo

2008/7/30 Michael Rueger <[hidden email]>:

> Philippe Marschall wrote:
>
>> That would interest me too.
>>
>> I made some very limited testing on Squeak 3.9.1 and the Mac VM, these
>> are my observations:
>>
>> 300 MB, hogs CPU badly, image grows dangerously but works
>> 600 MB, hogs CPU badly, low space semaphore, image unusable
>> 4 GB, Squeak image crashes, system unusable, reboot required
>
> Clearly what you get when streaming into a memory stream first.

That's certainly what it looks like. But clearly not the intention ;-)

Cheers
Philippe
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Göran Krampe

Yes, it works - and fast too! ( was (Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo)

Hi!

Ok, I just tested this again using this setup on my Dell D420 (small
Dell laptop, bogomips: 2394 according to /proc/cpuinfo) using:

- 3.8.1-6747 image
- gokr@yoda:~$ squeakvm -version
3.9-8 #1 Tue Mar 25 22:39:11 UTC 2008 gcc 4.2.3
Squeak3.9alpha of 4 July 2005 [latest update: #7021]
Linux vernadsky 2.6.15.7 #1 SMP
- Xubuntu Linux yoda 2.6.24-19-generic #1 SMP

I tested by loading my changeset into a Gjallar image and then using a
firefox 3 browser on:
http://localhost:8080/seaside/tests/alltests

...where there is an upload test. Ok the numbers:

Local test with a file of size 731654144 took circa 138 seconds to
upload giving approximately 5Mb/sec upload speed. Since Janko mentions
1.5Mb/sec for Swazoo this seems kinda good. :)

The image size does not move an inch during this - but the CPU shows 95%
load in top - looking closer at this it seems to be a side effect
because Process browser tells me that the actual Process in Squeak doing
the work is showing about 3% time! It is instead Morphic/idle that goes
up. Squeak is also fully responsive as is the OS during the upload.

I am sure this is related to other mechanisms in Squeak. And oh, yes,
also tried a file of size 1463308288 (double size) giving eh, slightly
larger upload time (not sure why).

Did not succeed with a 2Gb+ file, not sure why, haven't investigated
further. Code attached, it downloads files into ./seasidetmp

regards, Göran

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

StreamedUpload.4.cs.gz (3K) Download Attachment

Philippe Marschall

Re: Yes, it works - and fast too! ( was (Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo)

2008/8/1 Göran Krampe <[hidden email]>:

> Hi!
>
> Ok, I just tested this again using this setup on my Dell D420 (small
> Dell laptop, bogomips: 2394 according to /proc/cpuinfo) using:
>
> - 3.8.1-6747 image
> - gokr@yoda:~$ squeakvm -version
> 3.9-8 #1 Tue Mar 25 22:39:11 UTC 2008 gcc 4.2.3
> Squeak3.9alpha of 4 July 2005 [latest update: #7021]
> Linux vernadsky 2.6.15.7 #1 SMP
> - Xubuntu Linux yoda 2.6.24-19-generic #1 SMP
>
> I tested by loading my changeset into a Gjallar image and then using a
> firefox 3 browser on:
> http://localhost:8080/seaside/tests/alltests
>
> ...where there is an upload test. Ok the numbers:
>
> Local test with a file of size 731654144 took circa 138 seconds to
> upload giving approximately 5Mb/sec upload speed. Since Janko mentions
> 1.5Mb/sec for Swazoo this seems kinda good. :)
>
> The image size does not move an inch during this - but the CPU shows 95%
> load in top - looking closer at this it seems to be a side effect
> because Process browser tells me that the actual Process in Squeak doing
> the work is showing about 3% time! It is instead Morphic/idle that goes
> up. Squeak is also fully responsive as is the OS during the upload.
>
> I am sure this is related to other mechanisms in Squeak. And oh, yes,
> also tried a file of size 1463308288 (double size) giving eh, slightly
> larger upload time (not sure why).
>
> Did not succeed with a 2Gb+ file, not sure why, haven't investigated
> further. Code attached, it downloads files into ./seasidetmp

Interesting. At first glance this looks like the code you posted some
months ago (KomHttpServer-gk.31 + friends). I can also not make out
any differences to the code in Seaside 2.9 except from some cleanup.

On a side note it would be cool if Squeak supported the creation of
temporary files. Eg. a unique file in /var/tmp or equivalent.

Cheers
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside