Smalltalk › Frameworks & Tools › Seaside › Seaside General

Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

29 messages Options

Philippe Marschall

Re: [Q] File Upload/Download Server, Comanche or Swazoo

2008/7/29 Janko Mivšek <[hidden email]>:

> Philippe Marschall wrote:
>
>>> I don't understand how there is no way to know which input fields will be
>>> file ones. Web framework is naming them during HTML generation anyway, I
>>> suppose Seaside too?
>>
>> Yes, in a non-predicable way (for the server adapter). Even if it
>> would generate them in a predictable way there would still be a
>> problem if you have multiple file upload fields in the same form.
>
> In Aida there is completely predictable way to generate names and pair them
> with forthcoming POST. I therefore don't see the reason why Seaside can't do
> something similar.
>
> See for example image upload in http://squeaksite.aidaweb.si/About/ (login
> as demo/demo, then edit that page and see Attach image IFRAME). If we look
> at source of this IFRAME, there we have the following input fields:
>
> ...
> <form method=post action="/About/?view=images&version=2"
> enctype="multipart/form-data">
> ...
> <input type="hidden" name="id5273" value="images">
> <input type="file" name="field3">
> <input type="submit" name="field4" value="Upload">
> ...
> </form>
>
> As you can see field names are auto generated and Aida register them
> internally so that fields can be paired with post data in forthcoming POST.

I wonder how this works together with the back button. If we'd do that
we'd end up implementing half of Seaside in the server adapter. All
the session, callback probably even state and continuation logic would
have to be there.

> For streaming this pairing must not occur until you don't know the
> destination file, so you must not read post data from POST until then. This
> is maybe your problem, you read too fast post data in your adapter. End
> result in that case is that you stream the upload "in the air", because
> target stream is unknown.

A Seaside server adapter does very little work. All it does is it
converts a "native" request to a Seaside request and a Seaside
response to a "native" response. It doesn't know anything about
sessions, continuations, callbacks, components, applications or any
other Seaside implementation detail. It just delegates to an object
that understands #handleRequest:. This makes server adapters quite
small and easy to implement. It allows us the change the way Seaside
works without changing the server adapter (mostly). You can also take
this and build something completely different than Seaside and have it
running on different dialects (Squeak, Gemstone, CST, GST, Dolphin,
VAST, ...) and different servers (Kom, Swazoo, Swzaoo 2, OpenTalk,
FastCGI, ...) without any code change. That's for example what RSRSS
does.

All I want to do is to stream the fields that are files. Which fields
are files is encoded in the request. Why for example can't I just
iterate over the POST fields while they are being read? Look at each
one and see whether it is a file upload or not and if it is then
stream it. All this information can be parsed incrementally from the
request without the need to fully parse it first. In fact this is what
Kom and commons-fileupload do.

Cheers
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Göran Krampe

Re: Yes, it works - and fast too! ( was (Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo)

In reply to this post by Philippe Marschall

Hi!

Philippe Marschall wrote:
> Interesting. At first glance this looks like the code you posted some
> months ago (KomHttpServer-gk.31 + friends). I can also not make out
> any differences to the code in Seaside 2.9 except from some cleanup.

Ah, forgot that it was included nowadays, right - I can take a look at a
vanilla Seaside 2.9 and see why it doesn't seem to work in there. I will
get back to you ... in a day or two. :)

> On a side note it would be cool if Squeak supported the creation of
> temporary files. Eg. a unique file in /var/tmp or equivalent.

Agreed.

regards, Göran
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Janko Mivšek

Re: Yes, it works - and fast too! (was [Q] File Upload/Download Server, Comanche or Swazoo)

In reply to this post by Göran Krampe

Hi all,

Göran's work encouraged me to run benchmarks more carefully and
repeatably to minimize impact of OS file system performance and as you
can see, Kom and Swazoo are actually close in upload performance, about
5-6MB/s.

Big apology to Squeak community for that original 1.5MB/s figure, which
obviously came out from only one test, instead of repeatable to avoid
other impacts. In new tests I didn't change Swazoo code, I just made an
additional upload demo on Aida.

Here are results in seconds of repeatable test for two files, 572MB and
102MB ones:

572MB file (600.037.00 bytes)

Squeak:
Kom/Seaside 92 98 117 (avg 5.9MB/s)
Swazoo/Aida 108 111 107 (avg 5.6MB/s)
VW:
Swazoo/Aida 67 68 66 (avg 9MB/s)

102MB file (106.491.000 bytes)

Squeak:
Kom/Seaside 18 35 25 32 (avg 3.9MB/s)
Swazoo/Aida 16 16 17 16 (avg 6.6MB/s)
VW:
Swazoo/Aida 5 5 6 5 (avg 20MB/s)

As you can see for larger files we are very close while for smaller file
Swazoo is still faster. Also you can see that Kom results vary a lot
while Swazoo ones are stable. It seems that this is due to bigger amount
of garbage generation by Kom (this can also be seen from memory
consumption variations) while Swazoo garbage is minimal.

In any case we can conclude with quite certainty that on Squeak we
reached the upload speed limit near 6MB/sec. To be faster we will need
to optimize TCP socket and file VM modules.

Tests were done on Damien's squeak-dev 3.9 with both Aida and Seaside
installed, with Swazoo 2.2beta and Göran's Kom patch. Done on openSuse
10.0 64bit on 3.2GHz PC, bogomips: 6400.

Best regards
Janko

Göran Krampe wrote:

> Ok, I just tested this again using this setup on my Dell D420 (small
> Dell laptop, bogomips: 2394 according to /proc/cpuinfo) using:
>
> - 3.8.1-6747 image
> - gokr@yoda:~$ squeakvm -version
> 3.9-8 #1 Tue Mar 25 22:39:11 UTC 2008 gcc 4.2.3
> Squeak3.9alpha of 4 July 2005 [latest update: #7021]
> Linux vernadsky 2.6.15.7 #1 SMP
> - Xubuntu Linux yoda 2.6.24-19-generic #1 SMP
>
> I tested by loading my changeset into a Gjallar image and then using a
> firefox 3 browser on:
> http://localhost:8080/seaside/tests/alltests
>
> ...where there is an upload test. Ok the numbers:
>
> Local test with a file of size 731654144 took circa 138 seconds to
> upload giving approximately 5Mb/sec upload speed. Since Janko mentions
> 1.5Mb/sec for Swazoo this seems kinda good. :)
>
> The image size does not move an inch during this - but the CPU shows 95%
> load in top - looking closer at this it seems to be a side effect
> because Process browser tells me that the actual Process in Squeak doing
> the work is showing about 3% time! It is instead Morphic/idle that goes
> up. Squeak is also fully responsive as is the OS during the upload.
>
> I am sure this is related to other mechanisms in Squeak. And oh, yes,
> also tried a file of size 1463308288 (double size) giving eh, slightly
> larger upload time (not sure why).
>
> Did not succeed with a 2Gb+ file, not sure why, haven't investigated
> further. Code attached, it downloads files into ./seasidetmp
>
> regards, Göran

--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall

Re: Yes, it works - and fast too! ( was (Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo)

In reply to this post by Göran Krampe

2008/8/2 Göran Krampe <[hidden email]>:

> Hi!
>
> Philippe Marschall wrote:
>> Interesting. At first glance this looks like the code you posted some
>> months ago (KomHttpServer-gk.31 + friends). I can also not make out
>> any differences to the code in Seaside 2.9 except from some cleanup.
>
> Ah, forgot that it was included nowadays, right - I can take a look at a
> vanilla Seaside 2.9 and see why it doesn't seem to work in there. I will
> get back to you

The code is in: Seaside-Squeak-Kom. The two main changes are:
- upload streaming can be switched on and off
- WATemporaryFile is a subclass of WAFile

> ... in a day or two. :)

No hurry.

Cheers
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall

Re: Yes, it works - and fast too! ( was (Re: Re: Re: [squeak-dev] [Q] File Upload/Download Server, Comanche or Swazoo)

2008/8/2 Philippe Marschall <[hidden email]>:

> 2008/8/2 Göran Krampe <[hidden email]>:
>> Hi!
>>
>> Philippe Marschall wrote:
>>> Interesting. At first glance this looks like the code you posted some
>>> months ago (KomHttpServer-gk.31 + friends). I can also not make out
>>> any differences to the code in Seaside 2.9 except from some cleanup.
>>
>> Ah, forgot that it was included nowadays, right - I can take a look at a
>> vanilla Seaside 2.9 and see why it doesn't seem to work in there. I will
>> get back to you
>
> The code is in: Seaside-Squeak-Kom. The two main changes are:
> - upload streaming can be switched on and off
> - WATemporaryFile is a subclass of WAFile

And WAFileUploadTest has been changed to show the contents of the file
only if it is smaller than a certain limit. Else it simply shows the
size. So you can use it for testing.

Cheers
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Göran Krampe

Re: Re: Yes, it works - and fast too! (was [Q] File Upload/Download Server, Comanche or Swazoo)

In reply to this post by Janko Mivšek

Hi again!

Janko Mivšek wrote:

> Hi all,
>
> Göran's work encouraged me to run benchmarks more carefully and
> repeatably to minimize impact of OS file system performance and as you
> can see, Kom and Swazoo are actually close in upload performance, about
> 5-6MB/s.
>
> Big apology to Squeak community for that original 1.5MB/s figure, which
> obviously came out from only one test, instead of repeatable to avoid
> other impacts. In new tests I didn't change Swazoo code, I just made an
> additional upload demo on Aida.
>
> Here are results in seconds of repeatable test for two files, 572MB and
> 102MB ones:
>
> 572MB file (600.037.00 bytes)
>
> Squeak:
> Kom/Seaside 92 98 117 (avg 5.9MB/s)
> Swazoo/Aida 108 111 107 (avg 5.6MB/s)
> VW:
> Swazoo/Aida 67 68 66 (avg 9MB/s)
>
>
> 102MB file (106.491.000 bytes)
>
> Squeak:
> Kom/Seaside 18 35 25 32 (avg 3.9MB/s)
> Swazoo/Aida 16 16 17 16 (avg 6.6MB/s)
> VW:
> Swazoo/Aida 5 5 6 5 (avg 20MB/s)
>
> As you can see for larger files we are very close while for smaller file
> Swazoo is still faster. Also you can see that Kom results vary a lot
> while Swazoo ones are stable. It seems that this is due to bigger amount
> of garbage generation by Kom (this can also be seen from memory
> consumption variations) while Swazoo garbage is minimal.
>
> In any case we can conclude with quite certainty that on Squeak we
> reached the upload speed limit near 6MB/sec. To be faster we will need
> to optimize TCP socket and file VM modules.

Hmmm, ok, I did two more things:
- Refactored away one buffer copy. Now the Kom code copies straight from
the SocketStream inBuffer into the FileStream.
- Raised the buffer size from 1 Mb to 10 (or 20, 30, it does improve
numbers with a large buffer).

This gave the following numbers on my box for some files:

Time: 11.613 Size: 81601394 MB/s: 6.70121016453518
Time: 0.064 Size: 642560 MB/s: 9.57489013671875
Time: 12.024 Size: 81601394 MB/s: 6.4721518330628
Time: 0.111 Size: 828738 MB/s: 7.12023554621516
Time: 130.331 Size: 731654144 MB/s: 5.353751337939554
Time: 126.742 Size: 731654144 MB/s: 5.50535549087911

I also profiled this a bit and now there is very little gc going on and
I can't really easily see how to improve this much further. Small files
seem to generally be faster, but as you see the 81Mb file lands
regularly between 6.5 and 7 MB/s. Peaks around 10Mb for small files.

Latest version attached, this was tested with Seaside 2.8.

regards, Göran

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

StreamedUploadZeroCopy.1.cs.gz (4K) Download Attachment

Philippe Marschall

Re: Re: Yes, it works - and fast too! (was [Q] File Upload/Download Server, Comanche or Swazoo)

2008/8/3 Göran Krampe <[hidden email]>:

> Hi again!
>
> Janko Mivšek wrote:
>> Hi all,
>>
>> Göran's work encouraged me to run benchmarks more carefully and
>> repeatably to minimize impact of OS file system performance and as you
>> can see, Kom and Swazoo are actually close in upload performance, about
>> 5-6MB/s.
>>
>> Big apology to Squeak community for that original 1.5MB/s figure, which
>> obviously came out from only one test, instead of repeatable to avoid
>> other impacts. In new tests I didn't change Swazoo code, I just made an
>> additional upload demo on Aida.
>>
>> Here are results in seconds of repeatable test for two files, 572MB and
>> 102MB ones:
>>
>> 572MB file (600.037.00 bytes)
>>
>> Squeak:
>> Kom/Seaside 92 98 117 (avg 5.9MB/s)
>> Swazoo/Aida 108 111 107 (avg 5.6MB/s)
>> VW:
>> Swazoo/Aida 67 68 66 (avg 9MB/s)
>>
>>
>> 102MB file (106.491.000 bytes)
>>
>> Squeak:
>> Kom/Seaside 18 35 25 32 (avg 3.9MB/s)
>> Swazoo/Aida 16 16 17 16 (avg 6.6MB/s)
>> VW:
>> Swazoo/Aida 5 5 6 5 (avg 20MB/s)
>>
>> As you can see for larger files we are very close while for smaller file
>> Swazoo is still faster. Also you can see that Kom results vary a lot
>> while Swazoo ones are stable. It seems that this is due to bigger amount
>> of garbage generation by Kom (this can also be seen from memory
>> consumption variations) while Swazoo garbage is minimal.
>>
>> In any case we can conclude with quite certainty that on Squeak we
>> reached the upload speed limit near 6MB/sec. To be faster we will need
>> to optimize TCP socket and file VM modules.
>
> Hmmm, ok, I did two more things:

Could you put these in a new version of Kom, let's say KomHttpServer-gk.32?

> - Refactored away one buffer copy. Now the Kom code copies straight from
> the SocketStream inBuffer into the FileStream.
> - Raised the buffer size from 1 Mb to 10 (or 20, 30, it does improve
> numbers with a large buffer).

That sounds a bit extreme. We're talking about allocating a 10Mb
buffer for each upload, right?

Cheers
Philippe

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Janko Mivšek

Re: Yes, it works - and fast too! (was [Q] File Upload/Download Server, Comanche or Swazoo)

In reply to this post by Göran Krampe

Dear all,

New tests, with Göran's new Kom patch with direct copying and updated
Swazoo with large buffer support. Here I tested impact of buffer size to
the performance, with 8K (only Swazoo), 1M and 10M buffers.

572MB file (600.037.000 bytes)

Squeak:

Kom/Seaside 1M 110 104 104 (avg 5.7MB/s)
10M 100 104 98 (avg 5.9MB/s) (+4%)

Swazoo/Aida 8K 102 107 105 (avg 5.7MB/s)
1M 93 92 93 (avg 6.5MB/s) (+20%)
10M 90 89 91 (avg 6.7MB/s
VW:
Swazoo/Aida 8K 63 63 63 (avg 9.5MB/s)
1M 75 72 73 (avg 8.2MB/s) (-15%)
10M 70 70 70 (avg 8.6MB/s

102MB file (106.491.000 bytes)

Squeak:

Kom/Seaside 1M 22 20 21 (avg 5MB/s)
10M 16 21 16 (avg 6MB/s) (+20%)

Swazoo/Aida 8K 14 15 14 (avg 7.5MB/s)
1M 11 11 12 (avg 9MB/s) (+20%)
10M 12 11 12 (avg 9MB/s)
VW:
Swazoo/Aida 8K 5 5 6 (avg 20MB/s)
1M 6 6 6 (avg 18MB/s) (-10%)
10M 6 6 6 (avg 18MB/s)

Kom results are now more stable from run to run and better for 100MB
file where impact of large buffer is also biggest, +20%.

Swazoo on Squeak is 20% better with 1M buffer while 10M one improves
upload just slightly. Interesting are VisualWorks results, which are
with bigger buffer 10% worse !

Above tests were done with server and browser on the same machine, now
it is a time to rerun them on the real network. Buffer sizes can have
different impact then!

Best regards
Janko

Göran Krampe wrote:

> Janko Mivšek wrote:

>> Göran's work encouraged me to run benchmarks more carefully and
>> repeatably to minimize impact of OS file system performance and as you
>> can see, Kom and Swazoo are actually close in upload performance, about
>> 5-6MB/s.
>>
>> Big apology to Squeak community for that original 1.5MB/s figure, which
>> obviously came out from only one test, instead of repeatable to avoid
>> other impacts. In new tests I didn't change Swazoo code, I just made an
>> additional upload demo on Aida.
>>
>> Here are results in seconds of repeatable test for two files, 572MB and
>> 102MB ones:
>>
>> 572MB file (600.037.00 bytes)
>>
>> Squeak:
>> Kom/Seaside 92 98 117 (avg 5.9MB/s)
>> Swazoo/Aida 108 111 107 (avg 5.6MB/s)
>> VW:
>> Swazoo/Aida 67 68 66 (avg 9MB/s)
>>
>>
>> 102MB file (106.491.000 bytes)
>>
>> Squeak:
>> Kom/Seaside 18 35 25 32 (avg 3.9MB/s)
>> Swazoo/Aida 16 16 17 16 (avg 6.6MB/s)
>> VW:
>> Swazoo/Aida 5 5 6 5 (avg 20MB/s)
>>
>> As you can see for larger files we are very close while for smaller file
>> Swazoo is still faster. Also you can see that Kom results vary a lot
>> while Swazoo ones are stable. It seems that this is due to bigger amount
>> of garbage generation by Kom (this can also be seen from memory
>> consumption variations) while Swazoo garbage is minimal.
>>
>> In any case we can conclude with quite certainty that on Squeak we
>> reached the upload speed limit near 6MB/sec. To be faster we will need
>> to optimize TCP socket and file VM modules.
>
> Hmmm, ok, I did two more things:
> - Refactored away one buffer copy. Now the Kom code copies straight from
> the SocketStream inBuffer into the FileStream.
> - Raised the buffer size from 1 Mb to 10 (or 20, 30, it does improve
> numbers with a large buffer).
>
> This gave the following numbers on my box for some files:
>
> Time: 11.613 Size: 81601394 MB/s: 6.70121016453518
> Time: 0.064 Size: 642560 MB/s: 9.57489013671875
> Time: 12.024 Size: 81601394 MB/s: 6.4721518330628
> Time: 0.111 Size: 828738 MB/s: 7.12023554621516
> Time: 130.331 Size: 731654144 MB/s: 5.353751337939554
> Time: 126.742 Size: 731654144 MB/s: 5.50535549087911
>
> I also profiled this a bit and now there is very little gc going on and
> I can't really easily see how to improve this much further. Small files
> seem to generally be faster, but as you see the 81Mb file lands
> regularly between 6.5 and 7 MB/s. Peaks around 10Mb for small files.
>
> Latest version attached, this was tested with Seaside 2.8.
>
> regards, Göran
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Göran Krampe

Re: Re: Yes, it works - and fast too! (was [Q] File Upload/Download Server, Comanche or Swazoo)

Hi all!

The race continues - I was disappointed with the numbers below:

Janko Mivšek wrote:

> Squeak:
>
> Kom/Seaside 1M 110 104 104 (avg 5.7MB/s)
> 10M 100 104 98 (avg 5.9MB/s) (+4%)
>
> Swazoo/Aida 8K 102 107 105 (avg 5.7MB/s)
> 1M 93 92 93 (avg 6.5MB/s) (+20%)
> 10M 90 89 91 (avg 6.7MB/s

...and especially for the 102 Mb file (not included above), so I took a
look and yes, of course I found silly stuff going on in SocketStream. My
fixes aren't proper yet but it seems Kom can do just as good - I did
reach the 9Mb for a 81Mb file, now I am tuning it. Will post when I get
it proper.

regards, Göran
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside