Benchmark proposal

Hi,

As a follow-up to the discussion about performances, I'de like to
present the benchmark we have performed to compare Seaside to Java and
PHP. All comments and critics welcomed. The idea is to try to define a
benchmark that we could use to compare different web server solutions
using Smalltalk, so we'd be able to advice people about which tools to
use in what situation.

The benchmark we drove was rather small, and perhaps not the best we
could have thought of, but here it is.

1st, we measured the pure response time of the solution: using the same
server (Debian sarge, 1 GB RAM, SATA2 drives, PentiumIV 3GHz, don't
known the cache size, 100Mb/s LAN), we built a very simple page (hello
world type) using Java, PHP and Seaside. We then attacked the server
from another machine on our private LAN, using siege. We changed the
number of simultaneous users and the number of queries. Then we used the
results of siege to deduce the page availability. In parallell, we
measured the CPU and memory consumption using top with refresh rate of 0.1s.

2nd, we performed the same test, but with 3 machines running siege in
parallell. This changes a little bit, because now we have true
parallelism of requests. As before, we used siege results to deduce the
respoinse rate and time.

3rd, we built a simple web page that performs a huge query to a MySQL
database. Again, we measured the response time. Note that we did not
develop something particular, aka connection pools or stuff like that.

4th, we built a complex web site (similar to the shopping cart example
of Seaside) in the 3b languages. We launched 48 instances of a program
that browses the site automatically (can't remember its name), and then
measured the remaining load using siege.


I'll post the results as an answer, I need a bit of reformatting before.

Vincent
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi all!

Just wanted to mention that there is IIRC a bit of work that can be done
in the lower levels of Squeak web serving - at least if you use
KomHttpServer. I profiled and tweaked it quite a bit (2-3 years ago) for
fun and managed to squeeze out a substantial improvement.

After the first obvious bottlenecks I think it boiled down to header
parsing etc and now it is getting tempting to see if Exupery can improve
it.

Anyway, some day I might get around looking at all that again, if there
is interest.

regards, Göran

PS. I also think benchmarking for pure speed is almost totally
uninteresting when it comes to Seaside vs Others. But a benchmark can be
useful when tuning and looking for bottlenecks. In real apps there are
almost always other bottlenecks to look at - like db access, proper
caching etc.
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

There's always interest in tuning bottlenecks and making them faster.

Ramon Leon
http://onsmalltalk.com 

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> > Just wanted to mention that there is IIRC a bit of work that can be
> > done in the lower levels of Squeak web serving - at least
> if you use
> > KomHttpServer. I profiled and tweaked it quite a bit (2-3
> years ago)
> > for fun and managed to squeeze out a substantial improvement.

For fun...  How about a look at this... For fun... I'm having some speed
issues with SoapCore in conjunction with Seaside, i.e., using it with
Seaside so it's kind of on topic....

Using Soap against .Net.  It's my only access to my SqlServer database, and
recently I've noticed that a large part of my performance issues seem to
stem from Soap Decoding.  I'd like to know what you think.  Here's a Message
tally on a result set that takes about 1.7 seconds to return from the
service, but nearly a minute to decode 192 objects...

- 54494 tallies, 54557 msec.

**Tree**
100.0% {54557ms} WBBookingEngine>>soapCallBackDoor:sql:
  100.0% {54557ms} SoapCall>>invokeAndReturn
    97.7% {53302ms} SoapResponse>>returnValue
      |97.7% {53302ms} SoapResponse>>buildAdditionalResults
      |  97.7% {53302ms} SoapResponse>>extractDecodedValueFrom:
      |    97.5% {53193ms} SoapDecoder>>decodeXmlElement:
      |      96.2% {52484ms} SoapDecoder>>resolve
      |        92.0% {50192ms} Dictionary>>valuesDo:
      |          |92.0% {50192ms} Dictionary>>associationsDo:
      |          |  92.0% {50192ms} Dictionary(Set)>>do:
      |        2.7% {1473ms} SoapHref>>resolve
    2.3% {1255ms} SoapCall>>invoke
      2.3% {1255ms} SoapHttpConnector>>send:to:with:

**Leaves**
92.1% {50247ms} Dictionary(Set)>>do:

**Memory**
        old -1,547,728 bytes
        young -72,492 bytes
        used -1,620,220 bytes
        free +1,420,712 bytes

**GCs**
        full 1 totalling 229ms (0.0% uptime), avg 229.0ms
        incr 623 totalling 361ms (1.0% uptime), avg 1.0ms
        tenures 33 (avg 18 GCs/tenure)
        root table 0 overflows

Which leaves me staring at #do: wondering where to proceed from here.  I
emailed the author hoping maybe he'd have a better clue, but if anyone likes
tweaking code... Here's some code that needs tweaking.

Ramon Leon
http://onsmalltalk.com 

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi Ramon,

in the case you reported below method #do: is innocent, instead the sender  
of #valuesDo: has to be blamed for doing a sequential access to the  
elements of a dictionary (which may have and almost always has, to skip  
all the nil entries). Moreover, just the SoapHref members are filtered by  
SoapDecoder>>resolve and again, this might be a futile noop but  
aDictionary must check all its "raw" elements if its [the dictionary's]  
size > 0.

It may perhaps be possible to cache members of SoapHref in an array (or  
so) for speeding up SoapDecoder>>resolve drastically but this is only  
speculative because the relevant portions of SoapDecoder lack  
documentation 100%[tm].

/Klaus

On Thu, 18 Jan 2007 01:13:19 +0100, Ramon wrote:

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi!

"Ramon Leon" <[hidden email]> wrote:
[SNIP]
>       |        92.0% {50192ms} Dictionary>>valuesDo:
>       |          |92.0% {50192ms} Dictionary>>associationsDo:
>       |          |  92.0% {50192ms} Dictionary(Set)>>do:
>       |        2.7% {1473ms} SoapHref>>resolve
>     2.3% {1255ms} SoapCall>>invoke
>       2.3% {1255ms} SoapHttpConnector>>send:to:with:
> 
> **Leaves**
> 92.1% {50247ms} Dictionary(Set)>>do:
[SNIP]
> Which leaves me staring at #do: wondering where to proceed from here.  I
> emailed the author hoping maybe he'd have a better clue, but if anyone likes
> tweaking code... Here's some code that needs tweaking.

Since I just can't stay away from tuning stuff (it is so fun!) I found
this:

First, I wondered why #do: ended up as a leaf, when it is clearly taking
so much time.
Set>>do: is just iterating over an array, so the #do: itself is probably
not the problem.
Looking at senders of #resolve we discover it does #isMemberOf: on each
element - but... that is not slow either.

But what about SoapHref>>resolve? Aha! Well, I have NO idea if I am
right - but my eyes quickly homed in on the #becomeForward:. :)
You could throw in some timing code to see if that is actually the spot
where it all goes.

regards, Göran
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside