Call for help: Seaside performances

Hi everyone,

I have to study performances of Seaside compared to other solutions
(namely Java, PHP). The target would be a large application with many
services and very high number of users (possibly several 100,000). The
web servers would be deployed in a web farm with load balancing, etc.
The goal is to have high availability and very good response time.

I'm interested in any return of experience, and pointers to performance
tests and architecture definitions (even sparse). I'll try to avoid the
troll of wether seaside is better than java or PHP (that's not the
debate: seaside rules but I have to prove it ;-) ).

As some of you already know, I've made a first application using
Squeak/Seaside, but the performances are quite deceiving (a 1/12 in
favor of PHP, 1/6 for Java). Perhaps Squeak is not the best choice for
the VM ?

Avi, I heard you use Seaside to server dabbledb.com. Could you provide
some hints on the architecture for achieving good performances ?

All information greatly welcomed. Thanks for your help,


Vincent
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

On 1/11/07, Vincent Girard-Reydet <[hidden email]> wrote:

> Avi, I heard you use Seaside to server dabbledb.com. Could you provide
> some hints on the architecture for achieving good performances ?

The main hint I have is this: a large number of Squeak VMs each
handling a small number of sessions scales much better than a large
number of sessions on a single or small number of Squeak VMs.

Cheers,
Avi
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

I'm hosting http://www.reservetravel.com/v6 on a 3 server farm with each
server running one image on Windows Server 2003.  I'm running into
scalability issues when clients try and do a pay per click campaign and
Yahoo crawls me for 10000 keywords way too fast, each one starting a new
session, all hitting the same image because of a bug in our F5.  

To keep the site up, I have a script running every minute which will restart
the image if it fails to respond within an appropriate amount of time.  The
F5 brings it in and out of the pool as it goes up or down.  It sends me an
email when it does so, only happened a handful of times in the past few
weeks, not too bad.

Part of the problem is a bug in our BigIP F5 not load balancing correctly,
I'm waiting for the upgrade, but part of it does seem related to too many
sessions on an image.  I'm coming to the conclusion that I need to run
several images on a box as you're saying here.

Avi, any chance you can elaborate with some detail on what it takes to
dynamically bring up and take down images as necessary to handle the load?
Also, when using so many images, how does one manage keeping them all up to
date with the latest code?  Do you just have the images upgrade to the
latest version of a package on startup?  I know you're using Apache to fire
off some Ruby scripts to kick up the image, but knowing and doing are two
different things.  How do you have Apache dynamically proxy to a dynamic
port and make squeak start up on that port?  Any help in establishing some
best practices for production deployments would be benificial to many I'm
sure.

Ramon Leon
http://onsmalltalk.com

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

On 1/11/07, Ramon Leon <[hidden email]> wrote:

> Avi, any chance you can elaborate with some detail on what it takes to
> dynamically bring up and take down images as necessary to handle the load?

Well, in the case of Dabble there's an obvious partitioning - each
subdomain (foo.dabbledb.com) gets its own image.  When someone makes a
request to that subdomain, we start a new image up for it if we don't
already have one running.  If an image is idle for a few minutes, we
save it (including all the session info) and take it down.  FWIW, our
most heavily loaded server has about 7000 images, of which usually
about 15 are running at a time.  The average total footprint of one of
our images is about 100MB, which means that we use under 1.5GB for
this.  The highest I've seen is 22 out of the 7000 running at once.
We have 4GB of RAM in the server and have never come close to
thrashing.

> Also, when using so many images, how does one manage keeping them all up to
> date with the latest code?  Do you just have the images upgrade to the
> latest version of a package on startup?

No, when we update we just upgrade a single image using MC and then
clone it.  This means that pushing new code out requires blowing away
everyone's current sessions.

> How do you have Apache dynamically proxy to a dynamic
> port and make squeak start up on that port?

What, can't we keep any secrets? :)  I'll write that part up later.

Cheers,
Avi
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hey you got us into this. :-)
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> On Jan 11, 2007, at 12:57 PM, Avi Bryant wrote:
>> What, can't we keep any secrets? :)  I'll write that
>>  part up later.
>
>
> Todd Blanchard
> Hey you got us into this. :-)

Yea... What he said! LOL, but seriously, anxiously awaiting that write-up...
I have to get the kinks worked out before the travel season hits and the
real load starts coming.

Ramon Leon
http://onsmalltalk.com 

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

On 1/11/07, Ramon Leon <[hidden email]> wrote:

> Yea... What he said! LOL, but seriously, anxiously awaiting that write-up...
> I have to get the kinks worked out before the travel season hits and the
> real load starts coming.

So, here's the basic idea.  Each image has a unique port assigned to
it.  When a request comes into apache, you do the following:

1. Map the URL (or for us, the HTTP_HOST) to the right port.
2. Check if that port is open.
3. If the port is open, proxy the request to that port.
4. If the port isn't open, start up the image, and keep watching the
port until it opens.  Then proxy the request to it.

Steps 1 and 2 can be done using a program as a RewriteMap
(prg:/usr/bin/...).  I have one that takes a hostname and returns,
say, "open123" or "closed345" or "none" depending on what port it
finds and whether it's open.  This is easy to match in a subsequent
rewrite rule.

Step 3 is done by a RewriteRule that uses the [P] flag and the dynamic
port returned by the RewriteMap.

Step 4 is done by a CGI which shells out to start the image, does a
busywait loop for the port to open and then (in our case) uses the
Ruby HTTP library to do the final proxy (this is slow but only
necessary on the first request).

The specifics are pretty dependent on environment but the general
strategy works well.

Avi
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

That's a pretty neat way of doing it when you partition the application
in a predictable manner, be it the hostname or the url token. In our
case we'll just have a fat stateful load balancer in front of an army of
medium sized servers (multiple VMs across a few servers, actually) that
proxies requests round the pool for new sessions and to the matching
machine for existing sessions based on their session cookie (or _s
parameter, since we use both now), in which case scaling is achieved by
adding more machines to the pool and spreading the load evenly across.
You can also get away with doing simpler IP based load balancing in most
cases which saves you cycles on the balancer that are needed to extract
the session id and serve as an SSL proxy, exception being the clients
from networks where outside gateway keeps changing all the time, which
AOL used to do quite a bit, not sure about the state of things nowadays.

Cheers!

-Boris

--
+1.604.689.0322
DeepCove Labs Ltd.
4th floor 595 Howe Street
Vancouver, Canada V6C 2T5
http://tinyurl.com/r7uw4

[hidden email]

CONFIDENTIALITY NOTICE

This email is intended only for the persons named in the message
header. Unless otherwise indicated, it contains information that is
private and confidential. If you have received it in error, please
notify the sender and delete the entire message including any
attachments.

Thank you.

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Avi
Bryant
Sent: Thursday, January 11, 2007 3:34 PM
To: The Squeak Enterprise Aubergines Server - general discussion.
Subject: Re: [Seaside] Call for help: Seaside performances

On 1/11/07, Ramon Leon <[hidden email]> wrote:

> Yea... What he said! LOL, but seriously, anxiously awaiting that
write-up...
> I have to get the kinks worked out before the travel season hits and
the
> real load starts coming.

So, here's the basic idea.  Each image has a unique port assigned to
it.  When a request comes into apache, you do the following:

1. Map the URL (or for us, the HTTP_HOST) to the right port.
2. Check if that port is open.
3. If the port is open, proxy the request to that port.
4. If the port isn't open, start up the image, and keep watching the
port until it opens.  Then proxy the request to it.

Steps 1 and 2 can be done using a program as a RewriteMap
(prg:/usr/bin/...).  I have one that takes a hostname and returns,
say, "open123" or "closed345" or "none" depending on what port it
finds and whether it's open.  This is easy to match in a subsequent
rewrite rule.

Step 3 is done by a RewriteRule that uses the [P] flag and the dynamic
port returned by the RewriteMap.

Step 4 is done by a CGI which shells out to start the image, does a
busywait loop for the port to open and then (in our case) uses the
Ruby HTTP library to do the final proxy (this is slow but only
necessary on the first request).

The specifics are pretty dependent on environment but the general
strategy works well.

Avi
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Care to share your Apache config?  I don't need to partition my app like
Dabble, I'm thinking the same as you're doing here, just run a bunch of
images and use Apache to load balance them.  I just discovered
mod_proxy_balancer today, looks like exactly what I need, but I have to
upgrade to 2.2 to use it.

Ramon Leon
http://onsmalltalk.com

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

I certainly would, but it's not the Apache that does the balancing here,
rather a separate piece of hardware that acts as an SSL proxy and does
the balancing.

Cheers,

-Boris

--
+1.604.689.0322
DeepCove Labs Ltd.
4th floor 595 Howe Street
Vancouver, Canada V6C 2T5
http://tinyurl.com/r7uw4

[hidden email]

CONFIDENTIALITY NOTICE

This email is intended only for the persons named in the message
header. Unless otherwise indicated, it contains information that is
private and confidential. If you have received it in error, please
notify the sender and delete the entire message including any
attachments.

Thank you.

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Ramon
Leon
Sent: Thursday, January 11, 2007 4:21 PM
To: 'The Squeak Enterprise Aubergines Server - general discussion.'
Subject: RE: [Seaside] Call for help: Seaside performances

Care to share your Apache config?  I don't need to partition my app like
Dabble, I'm thinking the same as you're doing here, just run a bunch of
images and use Apache to load balance them.  I just discovered
mod_proxy_balancer today, looks like exactly what I need, but I have to
upgrade to 2.2 to use it.

Ramon Leon
http://onsmalltalk.com

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Many thanks for all those usefull informations!
For our application things are a bit more hard to partition than
dabbdle. Basically we have one namespace with many services inside, but
only 2 or 3 entrypoints. We expect heavy load on each of those entrypoints.

The prototype we have done uses apache first to serve the domain name (3
virtual hosts on the same server), then proxies all Seaside requests to
a software load balancer on the same server. The load balancer we use
for now is HAProxy, but I think the trick of Avi (having an external
program used in apache) is worth trying. We ran into problems using
apache proxying and cookies, namely apache's mod_proxy_balancer isn't
able to stick sessions on the same server. That's why we use HAProxy as
the load balancer.

Does any one of you have benchmarks of seaside performances? My problems
is we have very short planning, and Avi's solution seems to require
quite a lot of development (write the load balancer, write the scripts
that start/stop the image dynamically, configure apache and debug!!). I
have to know if it's worth the try, compared to using Java/JBoss (we
have guys here than push on it).
 From what I understand, the goal is to have each VM serve about 10-50
connections, and start a new image if more connections happen? Also, do
you use Squeak or VW for the VM ?

Vincent
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Are you sure about this?  From what I can see, you can pass a cookie name to
ProxyPass when using mod_proxy_balancer and it will do sticky sessions.

    ProxyPass / balancer://ramonOne/seaside/wbv6/$1 stickysession=_s
    ProxyPassReverse / balancer://ramonOne/seaside/wbv6/$1

Though, now I'm fighting a bug with cookies and infinite redirects because I
have to enable session cookie to get this to work.  I saw this discussed,
but I think the recent proposed hack doesn't take into account a basepath of
/ when running behind Apache.

Ramon Leon
http://onsmalltalk.com 

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Thanks vincent. I'm really happy that you followed my advice and I  
hope seasiders will help you.

Stef

On 11 janv. 07, at 17:31, Vincent Girard-Reydet wrote:

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Does anybody have benchmarks comparing Squeak and VW for serving web  
applications?
I would really be interested to know. People often told me that VW  
was good for serving but
I never got any real data.

On 12 janv. 07, at 09:31, Vincent Girard-Reydet wrote:

> Also, do you use Squeak or VW for the VM ?

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

This thread has been great.
Another questions I would like to ask:
Is anyone using memcached?
Is anyone using lighttpd?
What do you guys do when an image hangs? do you just restart it and
lose the sessions it might had?
Which database are you guys using on the backend?
This one has been asked before, but I did not see a consensous about
it:  Precisely which version of VM, image, Seaside (and whatever other
tricks like keeping Morphic windows closed) did you find to be the
most stable?


Thanks :)
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

I propose to define a reference benchmark (and a tool) and then run it
on several smalltalk web app servers, like Seaside, Aida/Web, Web
Toolkit etc. I can benchmark Aida/Web on VW.

Best regards
Janko

stephane ducasse wrote: _______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi,

I just finished benchmarking of Aida/Web web app server and if someone
repeats benchmarking of Seaside, then we'll have something to compare.

I measured memory consumption and throughput on an Aida/Web dynamically
generated login web page (http://aida.eranova.si/admin.html?view=login),
which has 4KB of HTML.

Tool named httperf was used for generating web requests. First I
generated 10.000 requests, which generated 10.000 new sessions and then
measured memory consumption:

- 220MB, that is 23KB per session

After optimizing memory consumption (gradually deleting more and more
session state, which is normally done every night) I achieved:

- 17KB per session, when last requests were removed from every session
- 400B per session when all WebApplication instances were removed.

Above page create 3 WebAplication instances, which hold session state
for representing domain objects and each such instance consumes 1KB and
above. WebAdminApp which generates that web page consumes 14KB, but with
some simple optimization it can be decreased to 1KB too.

Throughput which Aida can sustain is up to 30 requests/s for above test
page.

Benchmarking was done on P4 3.20GHz with 2GB memory running SuSE Linux
10.0 64bit, on VisualWorks 7.4.1.

More about benchmark procedure on:
- http://www.aidaweb.si/aidaweb-benchmarks.html

Best regards
Janko


Janko Mivšek wrote:
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> I just finished benchmarking of Aida/Web web app server and if someone
> repeats benchmarking of Seaside, then we'll have something to compare.

<rant>
I just finished benchmarking Seaside. I measured the time and the
lines of code to implement a dynamically generated login web
application which has 1KB of HTML.

A development environment named Squeak was used to implement the task.
First I created a new component (including the rendering code),
registered it as an application entry point and then I counted the
lines of code:

- 6 lines of code (renderContentOn:)
- 1 do it (application registration)

I could do all this in less than 2 minutes, including the startup of
Squeak. I think that could further be optimized by using a ready made
component.

Benchmarking was done on 2.16 GHz Intel Core Duo with 2 GB memory
running Mac OS X 10.4 on Squeak 3.9.
</rant>

Cheers,
Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi Lukas,

What of the memory consumption and the throughput ?

Lukas Renggli wrote:
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi lukas

I still think that this is important to recognize that people like  
vincent has serious concerns about
seaside scalability. Vincent seemed to be really enthousiast about  
seaside. Now they will certainly
switch to PhP because of speed. Vincent could you let us know.

Stef





On 17 janv. 07, at 07:49, Lukas Renggli wrote:

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside