swazoo presents problems under load

Dale,

Under load, when proxying to swazoo, if you have long running
processes ( like talking to a merchant processor - that could take a
few seconds ),
the queue can get full for listeners and the front end proxy will get
connection refused.

How hard would it be to create a 'master swazoo' that takes in
connections and hands them off to child processes for handling, much
like how
apache operates when you run it in fork mode. A single main apache
listens on port 80 ( or whatever ), keeps track of its children and
their availability
and then hands off incoming connections to them.

I've updated from 9 to 18 swazoos to see if we can get around the
connection refused issue, but in the end its just a band aid over a
larger problem.

Sean Allen wrote:
Sean,

As you know, I haven't spent a lot of time load testing swazoo, nor have
I spent time looking at the internals of Swazoo.

I've done all my load testing for Seaside using FastCGI and (ultimately)
  lightppd.

For FastCGI, I did find that it was necessary to limit the number of
accepts I did in a single Gem (using the famous gateSemaphore). In
FastCGI a thread is forked on each accept() so the requests are queued
within the gem (forked threads) not sitting waiting to be accepted - so
they aren't subject to timeouts. However, without the limit, the FastCGI
gem could accept connections so fast that we ran out oftempObj space
with queued up requests waiting to run...

Another thing that I observed with Apache was that it didn't seem to
evenly distribute requests evenly to all gems "in the pool". When I
started using lighttpd it appeared that the requests were distributed
more evenly...

So here's the check list of things to do.

   1. Are the connection requests being evenly distributed to all
      of the gems, or the gems at the "beginning of the list"
      getting too many requests?

If the requests aren't evenly distributed, then you need a better "load
balancer." My experience with lightppd is that it does a good job of
load balancing as it appears to pay attention to the number of connects
that are pending for each gem and it keeps the connection count evenly
balanced. Apache seems to have a "more sophisticated" algorithm that
uses somethin other than raw connection counts ...

Assuming that connections are now evenly distributed, but there are
still too many requests coming in to be handled by the available gems.

   2. Are the gems busy consuming cpu cycles or are they blocked waiting
      on external resources?

If the gems are consuming cpu cycles, then the correct option is to add
more gems, because you have more concurrent requests than you can handle
with the current number of gems.

If the cpu(s) upon which the gems are running is saturated, then you
need to add more CPUs to ensure that gems aren't block upon cpu cycles.

Assuming that you've got plenty of cpu cycles then presumably the gems
are blocked on an external resource (as in your case). For this one you
have a couple of choices.

The first choice is to add enough gems to handle your peak connection
load ...

With 9 gems and 2 seconds of wait time per request you are only able to
handle requests at a _steady state_ rate of 4.5 requests/second. With 18
gems you're up to 9 requests/second sustained. Keep in mind that this
rate is measuring the blocking transactions/second not the overall
requests/second, since I assume that the vast majority of requests are
handled quickly.

If you're experiencing peakiness with your load, then you need to
consider the capacity of your gems in a different way ... With FastCGI
and the gateSemaphore each gem is capable of queuing up 10 requests or
with 18 gems you'd have capacity for queueing 180 requests...For the
capacity we have count the fast requests along with the blocking requests.

If you are queuing requests (and it sounds like you are), it is critical
that you have a good load balancer and that you have enough gems to
cover the maximum number of concurrent blocking requests plus several
additional gems to handle the fast requests (a good load balancer will
always pick an idle gem over a gem that has a single connection)...you
don't want the fast requests to have to get in line behind a blocking
request unless you are in the peak of the peak...

If you collect the numbers for your situation and find that the number
of gems you need is reasonable for your hardware (the cost of an idle
gem is primarily it's memory footprint) ... with idle gems you can
afford to be running more gems than you have real memory for, since each
of the idle gems will be idle for several seconds...

If the number is unreasonable then the solution will have to involve
monkey business. I've gone through all of the analysis to this point
trying to make sure that we've done due diligence on alternate solutions
before resorting to monkey business:)

I recall that several months ago we talked about different solutions and
I think that at the time I suggested that you would want to route the
request off to another vm whose job was to sit and process the request.
I also recall that you guys were considering to commits in the middle of
  processing seaside requests and I recall that I didn't like that idea...

I've got an idea that is similar to the idea of creating a service gem,
except that instead of passing the request along to separate vm to
process (where _it_ would sit and wait on the external resource), you
essentialy defer the processing of the request until the external
resource has responded ... it would still require that you be careful
about not updating persistent state until you are at the point where you
  are ready to complete the request, but it wouldn't involve passing off
information to a separate gem for processsing and then doing polling
from within seaside ... Here's a rough overview of the steps:

   1. start processing standard seaside request
   2. at the point where you're read to read from the socket, you
      fork off a process that waits for the response and signals a
      semaphore when bytes are available on the socket.
   3. return a 'special' WAResponse that includes the semaphore in it's
      payload along with a WARequest that will pick up processing where
      you left off...you can save some of the information that you need
      in the session state ... in this scheme the session lock is not
      released and will to the outside world look like a single long
      running HTTP request
   4. When this special WAResponse is processed at the top-level code
      (i.e., the point at which the WAResponse is converted to a Swazoo
      response) the code recognizes that the WAResponse is special and
      instead of committing and returning the HTTP response, the code
      commits and then the thread is blocked until the semaphore is
      signalled.
   5. When the semaphore is signalled, the WARequest is put into the
      incoming request queue (i.e., waiting on the transaction mutex) so
      that it can resume processing "where it left off"

In essence the process is very similar to the processing that occurs
when you have a "lock not acquired" event, with a few tweaks here and there.

There are some rough edges to the idea that would have to be worked out,
but I'd imagine that this kind of facility would be useful for anyone
that needs to block on external resource...

I think this mail has gone on long enough:)

Dale

Hi,

I could have the wrong end of the stick here, but we found that
reverse proxying using lighttpd uses "fair balancing". A request is
not forwarded to one of the hyper servers if it is already busy with
one. Lighttpd will queue requests up and only dispatch when one of the
hypers finishes with a request. I'll go right now and check it out.

Cheers
Otto

On Thu, May 20, 2010 at 9:47 PM, Sean Allen <[hidden email]> wrote:

Seems to be working as advertised

On Fri, May 21, 2010 at 1:13 PM, Otto Behrens <[hidden email]> wrote:

That doesnt address the problem. The problem is that the OS listen
call has a maximum size
it will allow a queue to grow to. Once all your backend servers hit
that size, then you start
getting connection refused and most front end server will mark the
server as down.

Some frontends will fall over to other proxies, which ours did just
last night AND...
it proceeded to overload the listen queue for each swazoo, at which
point, the front end
can either return a variety of 500 error because it cant complete the
request or it can
queue up the connections until a backend proxy becomes free or until
the frontend
runs out of space in its listener queue.

In the case of the lighttpd scenario you are talking about, if it does
what you say in that fashion.
only handing a single request off to the proxy, you would hit this
problem even sooner than I am
because, only one connection at a time would be given to each swazoo
so the listener queque for
your front end server would being growing faster than if it just tries
to hand off to a backend.

If you cant add more backend application servers on the fly, this or
some variation will happen
under load. Eventually of course, on a single machine you would run
into memory/cpu contention
issues with all the backend application servers you might spawn, but
that is at least a farther
down the road problem than listen queques getting full.

On Fri, May 21, 2010 at 7:33 AM, Otto Behrens <[hidden email]> wrote:

On Fri, May 21, 2010 at 7:58 AM, Sean Allen <[hidden email]> wrote:

On Fri, May 21, 2010 at 10:03 AM, Chris Curtis <[hidden email]> wrote:
128 is the standard on linux.

but applications can set for themselves when they do a listen call so
even if you update the
kernel max value, your application could still be using less. Apache
for example will
max out at 511 unless you change the value.

Providing a negative value to the listen call for backlog will use the
kernel level max.

Btw, for everyone's edification. The swazoo that is in Gemstone &
Pharo maxes out at a backlog of 50 connections for a listen backlog.

See HTTPServer>>start

self socket listenFor: 50.

I assume that is probably the max for any of the swazoo ports.

Unfortunately, the listen implementation somewhere in that chain
doesnt accept a negative value like the standard c call
to say 'use the max available on the machine'. So you have to tune
that by hand inside swazoo rather than just tuning your
OS installation.

On Fri, May 21, 2010 at 10:35 AM, Sean Allen
<[hidden email]> wrote:

Good to know. It's probably also worth making it explicit that this makes THREE points where the backlog limit can be constrained:

Nginx defaults to -1 ie, it maxes out at whatever the OS max is.

Apache by default is 511 or was the last time I looked at the source.

On Fri, May 21, 2010 at 11:42 AM, Chris Curtis <[hidden email]> wrote:

Right now, I believe the root cause of "overflowing backlog" is that one or more gems are becoming unresponsive ... In a healthy system with enough gems to meet the load, you should only see connection attempts bouncing off the backlog queue because of extreme peak loading ... in this case with gems effectively dropping out of service (but keeping their socket connection open so that they continue to attrract connection requests) ... you will be guaranteed to have certain percentage of requests timing out  and no backlog limit will be big enough:)

Dale
________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Sean Allen [[hidden email]]
Sent: Friday, May 21, 2010 8:48 AM
To: GemStone Seaside beta discussion
Subject: Re: [GS/SS Beta] swazoo presents problems under load

Nginx defaults to -1 ie, it maxes out at whatever the OS max is.

Apache by default is 511 or was the last time I looked at the source.

On Fri, May 21, 2010 at 11:42 AM, Chris Curtis <[hidden email]> wrote: