code compatibility

A couple of thoughts on code compatibility:

====
HTTPServer

HTTPServer>>newSocket: is a utility method with dialect specific code.
HTTPServer>>interactWithNewConnectionOver: is the only sender and also
contains dialect specific code.

Suggestion1: Make these methods subclass responsibilities for dialect
specific HTTPServer classes.

Suggestion2:
Extract the methods to SwazooSocket(class)

The method would look something like:

SwazooSocket(class)>>accept: aListenerSocket onError: operation
^[self newSocket: aListenerSocket ] on: OsError
      do:
       [:ex |
       Transcript
        show: 'Socket accept error: ';
        show: ex errorString;
        cr.
       ObjectMemory compactingGC.
       operation value]).

HTTPServer>>interactWithNewConnectionOver: socket
 | conn |
 conn := HTTPConnection socket: (SwazooSocket
    accept: socket
    onError: [^self]).
 self addConnection: conn.
 ^conn interact

====
HTTPConnection

Any thoughts on dealing with "bad" connections?

I can not implement HTTPConnection>>getAndDispatchMessages's forked process
interrupting the loop process in Dolphin.

Instead of VW forking a new process to check each HTTPConnection instance,
could HTTPServer have a single low priority process that waits on a delay
and wakes up to enumerate over all connections checking for "bad"
connections?

This still leaves the question of what is a good test for a bad connection?


====
HTTPPost

HTTPPost>>readFrom: makes the assumption that if the contentType is not
'multipart/form-data' it is
'application/x-www-form-urlencoded'. Could this be explicitly checked for,
and if neither, the entity is stored without any decoding taking place?

====
Resources

I can live without resources having a parent pointer, but it would either
make my configuration files nasty, or I would need to add a custom
CompositeResource. The main reason is that I use the urls for namespaces and
many specs are not allowing relative urls as namespaces. I have found it to
be generally useful

Is this a no-go?


Thanks,
Steve
www.dolphinharbor.org

Steve -- good thoughts.  My time is somewhat limited right now, but I've
been chipping away at some of these things in spare moments.  My project
deadlines are just pushing me towards other areas of Swazoo (thus my
recent work has been on integration of CacheControl, SwazooActivePages,
etc.).

Here's where I think I'd like to see things go WRT your comments:

HTTPServer: We'll move the socket accept code into SwazooSocket.  I'd
also like to make the master server socket be an instance variable of
HTTPServer so that we don't have to pass it around in awkwardly-named
messages like #interactWithNewConnectionOver: (would #acceptConnection
be better?).  The only ramifications I can see are that the socket has
to be managed properly if ever a live HTTPServer is saved in an image
snapshot.  On my server, I don't do that, but I think Janko does.  Do
you, Steve?  Janko, speak up.  :)

HTTPConnection: I think a short timeout on the reading of a message
header (combined with a longer timeout on the reading of the entity)
should be sufficient.  The #interruptWith: block is a kludge, to say the
least, and all it does is implement a timeout on the combined reading of
the message and header.  There is a very rudimentary start of support
for this in SwazooSocket, but it wiil probably also require some support
in HTTPConnection (i.e., HTTPConnection will have to know the header /
entity timeouts and send SwazooSocket>>#read:timeout: repeatedly until
the header or entity arrives).  This would eliminate the need for a
low-priority "cleanup" process IMHO, but we'd have to verify that
empirically.

HTTPPost: What does the spec say on decoding?  I haven't looked at it in
a while.  Whatever it takes to comply with the spec, let's do that.

Resources: I'm fine with the backpointer, but I haven't had a driving
need for it.  I'll try to integrate your code soon.  Can you tell me
which methods are most significantly affected?


Ken

Hi Ken

Ken Treis wrote:

I agree completely here :)

> The only ramifications I can see are that the socket has to be managed
> properly if ever a live HTTPServer is saved in an image snapshot.  On
> my server, I don't do that, but I think Janko does.  Do you, Steve?  
> Janko, speak up.  :)

Yes, that's a good thing to add. I did that in one of my versions
loooong ago. I would put snapshoot event handling  in dialect specific
package and only provide hooks in core. Start/stop (which by the way
don't work ok right now) of entire Swazoo and any Site will probably
suffice as hooks.

I think we need to introduce start/stop into Resources too. In order to
start/stop aSite, some resources also need to know about that start/stop
event (to cleanup caches for instances, to initialize, to save someting
etc.)

I think a keeper process will be need sooner or later. For hardening
Swazoo I'm thinking of using it for changing priority of long-lived
requests (to reduce a endless loop inpact for instance), to timeout
whole connection, to timeout bogus requests etc etc.

I'm also proposing to introduce a connection state, that is, in which
state is connection at that moment. States that come into my mind are:
#idle #receiving #processing #sending

States and timestamps will then help keeper process to decide what to do
with a connection.

>
> HTTPPost: What does the spec say on decoding?  I haven't looked at it
> in a while.  Whatever it takes to comply with the spec, let's do that.
>
> Resources: I'm fine with the backpointer, but I haven't had a driving
> need for it.  I'll try to integrate your code soon.  Can you tell me
> which methods are most significantly affected?


I vote for backpointers too :)

JAnko

Hi Ken,

> be better?).  The only ramifications I can see are that the socket has
> to be managed properly if ever a live HTTPServer is saved in an image
> snapshot.  On my server, I don't do that, but I think Janko does.  Do
> you, Steve?  Janko, speak up.  :)

I also do not save live HTTPServer's in an image, but not being able to was
reported as a bug. I register SwazooSiteRegistry to receive an event
from the system on image startUp, and stop and start any HTTPServer
instances. The code to do this can go into the Dolphin compatibility
package.

Timing out on the Socket level would not be portable to Dolphin. FWIW: the
SwazooSocketTest>>testReadTimeout fails to timeout on my Win2k VW5.4nc
image. I have not looked into why, but WinSock1.1 did not support timeouts.


> HTTPPost: What does the spec say on decoding?  I haven't looked at it in
> a while.  Whatever it takes to comply with the spec, let's do that.

The html spec [1] says that we must handle both
application/x-www-form-urlencoded and multipart/form-data. The HTTP spec at
[2] talks about other multipart encodings, but these only look relevant to
clients.

Other content-types are probably best left to a resource to process.

I fixed a couple of bugs I had in Dolphin's HTTPPost since the version
included in the Spray download, and after having another look at the specs
today, I think I can clean it up further.

... I was going to add a fileout, but I have given Store publish a go. You
should see my changes as version 9 with a blessing of "To Review". Store is
a very useful system! Please feel free to integrate, or not, the changes as
you see fit. I added an instance variable to hold the entityBody if no
decoding takes place as that is how I have implemented it in Dolphin. The
alternative would be to use the postData dictionary with a "magic key". I
prefer the instance variable.

Thanks,
Steve
www.dolphinharbor.org


[1] http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13.4
[2] http://www.w3.org/Protocols/HTTP/1.1/spec.html#Content-Codings

----- Original Message -----
From: "Janko Mivsek" <[hidden email]>

Hi Janko,

> I think we need to introduce start/stop into Resources too. In order to
> start/stop aSite, some resources also need to know about that start/stop
> event (to cleanup caches for instances, to initialize, to save someting
> etc.)

I have added "start" as part of creating a Site from a configuration file.
When a resource has it's parent pointer set, it is sent #onResourceCreated.
This is where I do any initialization that depends on knowledge of the url.

I have not needed a "stop" as I have tended to rely on finalization.

>
> I think a keeper process will be need sooner or later. For hardening
> Swazoo I'm thinking of using it for changing priority of long-lived
> requests (to reduce a endless loop inpact for instance), to timeout
> whole connection, to timeout bogus requests etc etc.

The keeper process is the best option for Dolphin portability.

Dynamically changing the priority of processes is an interesting idea. I
have fiddled with changing connection and server priorities under stress. If
the server processes priority is higher than the connection priority the
server is very responsive for lower number of connections and will clear the
listen backlog when hit with a high number of concurrent requests.

If the priorities are equal, the connections are served faster, but it is
too easy to fill the backlog and start returning connection refused errors.

I have thought about a throttle that normally keeps the server priority
higher than the connection priority, but drops it down under certain
circumstances. I have not been able to think of a good way to determine what
would trigger the drop. Counting the number of connections seems arbitrary.
Dolphin's profiling tools are not really suited to process profiling. Do you
have profiling tools in VW that help to study the connection and server
processes?


> I'm also proposing to introduce a connection state, that is, in which
> state is connection at that moment. States that come into my mind are:
> #idle #receiving #processing #sending

I have one state that I use for testing whether a connection is bad.
#isWaiting is true when a message starts to be read. It changes to false
once the first line has been read.

I kill a connection's process if #isWaiting for longer than 70 seconds. This
is probably too conservative. The closest thing I have found to a best
practice on this issue is:

http://www-old.ics.uci.edu/pub/ietf/http/draft-ietf-http-connection-00.txt

Thanks,
Steve
www.dolphinharbor.org

----- Original Message -----
From: "Ken Treis" <[hidden email]>

> Resources: I'm fine with the backpointer, but I haven't had a driving
> need for it.  I'll try to integrate your code soon.  Can you tell me
> which methods are most significantly affected?

I have merged these changes and uploaded them to Store as "To Review"
Swazoo-Core(10,stevew) and Swazoo-Tests(10,stevew)

Thanks,
Steve
www.dolphinharbor.org