[Fwd: Socket I/O Problems with UNIX Squeak 3.8a1 VM]

-------- Original Message --------
Subject: Socket I/O Problems with UNIX Squeak 3.8a1 VM
Date: Sat, 20 Aug 2005 11:33:09 -0400 (EDT)
From: David Shaffer <[hidden email]>
Reply-To: The general-purpose Squeak developers list
<[hidden email]>
Newsgroups: gmane.comp.lang.smalltalk.squeak.general

I have been trying to figure out why Squeak performs so badly as a
webserver.  With some help from John McIntosh (thanks John!) I was able
to track down a problem in the Unix VM's socket I/O support.  The
problem is the standard "missed signal" when using select().  When the
Squeak VM enters aioPoll() it uses a select() to check for activity on
various sockets (and other file descriptors -- but I'll stick to TCP
sockets here).  For server sockets (passive TCP sockets) who have
received new connections it proceeds to call accept() once and then
signal the appropriate semaphore.  The problem is that more than one new
connection may have arrived during (and before) that first select and
standard POSIX practices say you have to call accept() until it returns
-1 with errno=EWOULDBLOCK.  I am quite certain of this...because should
multiple connections arrive during the first select(), a subsequent
select() will not detect them.  Where I'm a little less certain, not
having any books with me, is if you need to do similar things for read()
and write()...I believe that you do.  In any case, at the very least,
this causes the SqueakVM to miss connections when they occur close
together.  I won't go into details but the problem cascades causing UNIX
Squeak-based servers to behave very badly when receiving large numbers
of closely spaced connections.  I don't have a production-ready solution
but right now I have a working image which uses an array like so:

typedef struct privateSocketStruct
{
   int s;            /* Unix socket */
   int connSema;            /* connection io notification semaphore */
   int readSema;            /* read io notification semaphore */
   int writeSema;        /* write io notification semaphore */
   int sockState;        /* connection + data state */
   int sockError;        /* errno after socket error */
   struct sockaddr_in peer;    /* default send/recv address for UDP */
   int multiListen;        /* whether to listen for multiple connections */
   int acceptedSock[1024];        /* a connection that has been accepted */
   int acceptedSockCount;
} privateSocketStruct;


Anyway, I patched the code to loop over accept() and fill the
acceptedSock array and repeatedly signal the semaphore.  Of course I
also added the logic on the other end of the accept to deal with
decrementing this count.  Clearly this is a stop-gap solution.  I don't
think we need to allocate 1024 ints for every socket.   Anyway after a
lot of pounding with httperf the net result seems to be a much more
reliable Squeak/UNIX web server.  I don't believe, though, that it
performs much faster than it did before...just that it doesn't
mysteriously drop connections.  So, I'm only part way to my ultimate
goal which is to improve performance.  I thought this would be a good
time to bring this up and get feedback (especially from Ian) about how
you would like me to shape the actual fix to this problem.  Add a single
void * member to privateSocketStruct which, for server sockets points,
to a "struct privateServerSocket" which holds the count and maybe a
linked list of socket file descriptors.

When I return to the U.S. I'll look into the details of the interactions
of select() with non-blocking read's and write's.  I suspect that this
code might need a good looking-at too.  After that I'd like to work with
Ian to put together a patch against the 3.8g VM that Ian has provided
since many of us use this VM in production Seaside images.

Also, if someone wants to forward this on to the VM list, I'd appreciate
it.  I'm not a list member and have a very poor connection at the moment.

BTW folks...ESUG was excellent!  Thanks to everyone who put it together.

David