session lock denied causes swazoo seaside 3 adaptor to become unresponsive

Is there an easy fix for this?

If a swazoo adaptor for seaside 3 is hit by a session lock issue, it
becomes unresponsive and doesnt answer additional requests
Trying to connect like lynx http://127.0.0.1:9001/ just results the
connection not being answered. basically, connection refused.

Sean,

I don't know if there is an easy fix or not ...

Do you have a test bed system where you can put your system under load outside of a production environment?

The only way we'll get our hands around this is to characterize the problem in detail ... I need stack traces from the gems to know exactly _why_ ...

I have seen logs with lots of session lock denied messages, so it is not the first instance of a session lock that causes the problem, so you must mean that the gem goes wacko when the the retry limit is exceeded...

I have to build 1.0-beta.6 system, so once I've got that system up I will look at the swazoo/retry limit code and see if there is an obvious fix or not...

Dale
________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Sean Allen [[hidden email]]
Sent: Friday, May 21, 2010 7:08 AM
To: GemStone Seaside beta discussion
Subject: [GS/SS Beta] session lock denied causes swazoo seaside 3 adaptor to    become unresponsive

Is there an easy fix for this?

If a swazoo adaptor for seaside 3 is hit by a session lock issue, it
becomes unresponsive and doesnt answer additional requests
Trying to connect like lynx http://127.0.0.1:9001/ just results the
connection not being answered. basically, connection refused.

I can sometimes reproduce. It doesnt always happen when I do the same
thing each time.
I think there might be a timing factor that I cant always get right.
This is a stack trace from a
swazoo hangs one:


 WAGsSwazooAdaptor Server started on port 9003
----------- Lock not acquired - retrying LOG ENTRY: Session lock
denied: 2075-----------
----------- Lock not acquired - retrying LOG ENTRY: Session lock
denied: 2075-----------
----------- Lock not acquired - retrying LOG ENTRY: Session lock
denied: 2075-----------
----------- Lock not acquired - retrying LOG ENTRY: Session lock
denied: 2075-----------
----------- Lock not acquired - retrying LOG ENTRY: Session lock
denied: 2075-----------
----------- Lock not acquired - retrying LOG ENTRY: Session lock
denied: 2075-----------
----------- Lock not acquired - retrying LOG ENTRY: Session lock
denied: 2075-----------
----------- Lock not acquired - retrying LOG ENTRY: Session lock
denied: 2075-----------
----------- Lock not acquired - retrying LOG ENTRY: Session lock
denied: 2075-----------
----------- Lock not acquired - retrying LOG ENTRY: Session lock
denied: 2075-----------
----------- Internal Swazoo ERROR Encountered: 2010-05-21T13:07:12-07:00
Too many retries: 11
1 GRGemStonePlatform >> logError:title: @2 line 4  [GsMethod OOP 124136961]
2 WAGsSwazooAdaptor >> internalServerErrorFor:message: @10 line 12
[GsMethod OOP 148484865]
3 ComplexBlock in GRGemStonePlatform >>
seasideProcessRequest:adaptor:resultBlock: @17 line 14  [GsMethod OOP
150105857]
4 ExceptionHandler >> doTryBlock: @9 line 7  [GsMethod OOP 11916289]
5 ExceptionHandler >> try:on:do: @15 line 18  [GsMethod OOP 11913985]
6 ComplexBlock in ExecutableBlock >> on:do: @2 line 8  [GsMethod OOP 9449985]
7 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: @25
line 16  [GsMethod OOP 150105857]
8 WAGsSwazooAdaptor >> process: @4 line 4  [GsMethod OOP 148485121]
9 WAPluggableSite >> answerTo: @2 line 2  [GsMethod OOP 130287617]
10 WAPluggableSite >> helpResolve: @7 line 3  [GsMethod OOP 130287105]
11 URIResolution >> visitResource: @1 line 2  [GsMethod OOP 130668545]
12 ComplexBlock in URIResolution >> visitChildrenOf:advancing: @10
line 7  [GsMethod OOP 130667009]
13 Collection >> do: @5 line 10  [GsMethod OOP 1544961]
14 URIResolution >> visitChildrenOf:advancing: @15 line 5  [GsMethod
OOP 130667009]
15 URIResolution >> resolveTransparentComposite: @2 line 2  [GsMethod
OOP 130670337]
16 URIResolution >> resolveServerRoot: @1 line 2  [GsMethod OOP 130668289]
17 ServerRootComposite >> helpResolve: @1 line 2  [GsMethod OOP 131127041]
18 URIResolution >> visitResource: @1 line 2  [GsMethod OOP 130668545]
19 URIResolution class >> resolveRequest:startingAt: @3 line 2
[GsMethod OOP 130223617]
20 HTTPServer >> answerTo: @3 line 3  [GsMethod OOP 130320897]
21 ComplexBlock in HTTPConnection >> produceResponseFor: @9 line 5
[GsMethod OOP 130933761]
22 ExceptionHandler >> doTryBlock: @9 line 7  [GsMethod OOP 11916289]
23 ExceptionHandler >> try:on:do: @15 line 18  [GsMethod OOP 11913985]
24 ComplexBlock in ExecutableBlock >> on:do: @2 line 8  [GsMethod OOP 9449985]
25 HTTPConnection >> produceResponseFor: @23 line 6  [GsMethod OOP 130933761]
26 HTTPConnection >> getAndDispatchMessages @9 line 9  [GsMethod OOP 130934529]
27 ComplexBlock in HTTPConnection >> interact @7 line 6  [GsMethod OOP
130934785]
28 ExceptionHandler >> doTryBlock: @9 line 7  [GsMethod OOP 11916289]
29 ExceptionHandler >> try:on:do: @15 line 18  [GsMethod OOP 11913985]
30 ComplexBlock in ExecutableBlock >> on:do: @2 line 8  [GsMethod OOP 9449985]
31 ComplexVCBlock in HTTPConnection >> interact @12 line 9  [GsMethod
OOP 130934785]
32 ComplexBlock in ExecutableBlock >> ifCurtailed: @4 line 6
[GsMethod OOP 67997953]
33 ComplexBlock in ExecutableBlock >> ensure: @4 line 11  [GsMethod OOP 2305281]
34 ComplexBlock in ExecutableBlock >> ensure: @6 line 11  [GsMethod OOP 2305281]
35 ComplexVCBlock in ExecutableBlock >> ifCurtailed: @8 line 8
[GsMethod OOP 67997953]
36 ComplexVCBlock in HTTPConnection >> interact @19 line 13  [GsMethod
OOP 130934785]
37 HTTPConnection >> interact @26 line 18  [GsMethod OOP 130934785]
38 HTTPServer >> acceptConnection @15 line 11  [GsMethod OOP 148109313]
39 ComplexBlock in HTTPServer >> start @13 line 6  [GsMethod OOP 1266430721]
40 ComplexBlock in BlockClosure >> repeat @3 line 5  [GsMethod OOP 18247425]
41 ComplexVCBlock in HTTPServer >> start @14 line 6  [GsMethod OOP 1266430721]
42 GsProcess >> _startPart2 @15 line 17  [GsMethod OOP 4503553]
43 GsProcess >> _start @1 line 9  [GsMethod OOP 4504065]


On Fri, May 21, 2010 at 2:18 PM, Dale Henrichs <[hidden email]> wrote:

Haha,

the mail that I sent this morning just came through, but my build finished and I just got back to looking at swazoo, so this was oddly good timing:)

The problem you are seeing when the retry limit is hit must be related to the internal error handling for the swazoo adaptor.

WAGsSwazooAdaptor>>internalServerError:for: returns an HTTPResponse with a code of 500 ... which is basically what a normal response from Swazoo returns ... so something downstream from the response is having a problem and swazoo is not handling _that_ problem well ... so in your testing you could try triggering an internalServerError explicitly and see what blows up ...

Dale
________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Dale Henrichs [[hidden email]]
Sent: Friday, May 21, 2010 11:18 AM
To: GemStone Seaside beta discussion
Subject: Re: [GS/SS Beta] session lock denied causes swazoo seaside 3 adaptor to        become unresponsive

Sean,

I don't know if there is an easy fix or not ...

Do you have a test bed system where you can put your system under load outside of a production environment?

The only way we'll get our hands around this is to characterize the problem in detail ... I need stack traces from the gems to know exactly _why_ ...

I have seen logs with lots of session lock denied messages, so it is not the first instance of a session lock that causes the problem, so you must mean that the gem goes wacko when the the retry limit is exceeded...

I have to build 1.0-beta.6 system, so once I've got that system up I will look at the swazoo/retry limit code and see if there is an obvious fix or not...

Dale
________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Sean Allen [[hidden email]]
Sent: Friday, May 21, 2010 7:08 AM
To: GemStone Seaside beta discussion
Subject: [GS/SS Beta] session lock denied causes swazoo seaside 3 adaptor to    become unresponsive

Is there an easy fix for this?

If a swazoo adaptor for seaside 3 is hit by a session lock issue, it
becomes unresponsive and doesnt answer additional requests
Trying to connect like lynx http://127.0.0.1:9001/ just results the
connection not being answered. basically, connection refused.