Smalltalk › Gemtalk › GLASS

debug help for gem

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

24 messages Options

NorbertHartl

Re: debug help for gem

Dale,

I produced a stack trace from a gem. I had two vms running and the stack trace ends up being the same for both of them. The stack trace is attached.
To figure out the problem I would need to get more into how swazoo works.

I'm just wondering about

WAGsSwazooAdaptor>>start
"blocks main thread, does not return"
super start.
[true] whileTrue: [ (Delay forSeconds: 10) wait ].

So at least I would assume that "super start" triggers the server listening. After that the adaptor will go into the endless loop. If an exception ocurrs in the code triggered by "super start" how can this exception break the whileTrue: loop? The stack trace shows that the process is still hanging in the loop. So my weak assumption would be that an ocurring exception ends the piece of code that is started by "super start" but the gem still hangs in the loop.

Norbert

On 22.09.2010, at 20:14, Dale Henrichs wrote:

> Norbert,
>
> If I'm following things, it seems that the swazoo vm is alive, but stops listening on the port?
>
> A 'kill -USR1' should give you a smalltalk stack trace in the server log so that you can tell what is going on inside the server....
>
> This seems to be similar to a problem that Sean had with Swazoo? If I recall he had gems running hot and not listening/responding...
>
> At this point in time, I'm wondering if I should create a Seaside 3.0 adaptor for Hyper. Hyper didn't have these kinds of bugs (I don't think). I remember that when I was porting Swazoo I noticed that the structure of the server was quite different between Hyper and Swazoo...
>
> I've got a couple of projects stacked up right now, but a Hyper adaptor might make sense ... what do you think?
>
> Dale
>
>
> Norbert Hartl wrote:
>> I use the swazoo gs adapter.
>> Norbert
>> Am 21.09.2010 um 16:32 schrieb Sean Allen <[hidden email] <mailto:[hidden email]>>:
>>> Are you using the FastCGI or Swazoo adapter?
>>>
>>> On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl < <mailto:[hidden email]>[hidden email] <mailto:[hidden email]>> wrote:
>>>
>>> I'm using daemontools for three weeks now and I discover severe
>>> problems. One of my services is not fully ported to seaside 3.0
>>> and throws some exceptions. What I experience is that all of my
>>> vms go just _offline_. I mean there is just no socket listening on
>>> the configured port. Looking at svstat from daemontools it seems
>>> that the vms do not restart.
>>> I thought the vms are restartet and something prevents opening a
>>> socket. But now it seems that the listening socket goes away but
>>> the vm does not stop and therefor doesn't get restarted.
>>> My scripts are roughly the ones described in the gemstone
>>> daemontools howto. I have a run script the invokes another script
>>> that the seaside 30 run script modified for logging and foreground
>>> processing.
>>>
>>> What is the best way to check this kind of behaviour. I looked at
>>> the log files but didn't find anything useful.
>>>
>>> Any ideas?
>>>
>>> Norbert
>>>
>>>
>>>
>

NorbertHartl

Re: debug help for gem

And the attachment of the log

Norbert

On 23.09.2010, at 09:57, Norbert Hartl wrote:

> Dale,
>
> I produced a stack trace from a gem. I had two vms running and the stack trace ends up being the same for both of them. The stack trace is attached.
> To figure out the problem I would need to get more into how swazoo works.
>
> I'm just wondering about
>
> WAGsSwazooAdaptor>>start
> "blocks main thread, does not return"
> super start.
> [true] whileTrue: [ (Delay forSeconds: 10) wait ].
>
> So at least I would assume that "super start" triggers the server listening. After that the adaptor will go into the endless loop. If an exception ocurrs in the code triggered by "super start" how can this exception break the whileTrue: loop? The stack trace shows that the process is still hanging in the loop. So my weak assumption would be that an ocurring exception ends the piece of code that is started by "super start" but the gem still hangs in the loop.
>
> Norbert
>
> On 22.09.2010, at 20:14, Dale Henrichs wrote:
>
>> Norbert,
>>
>> If I'm following things, it seems that the swazoo vm is alive, but stops listening on the port?
>>
>> A 'kill -USR1' should give you a smalltalk stack trace in the server log so that you can tell what is going on inside the server....
>>
>> This seems to be similar to a problem that Sean had with Swazoo? If I recall he had gems running hot and not listening/responding...
>>
>> At this point in time, I'm wondering if I should create a Seaside 3.0 adaptor for Hyper. Hyper didn't have these kinds of bugs (I don't think). I remember that when I was porting Swazoo I noticed that the structure of the server was quite different between Hyper and Swazoo...
>>
>> I've got a couple of projects stacked up right now, but a Hyper adaptor might make sense ... what do you think?
>>
>> Dale
>>
>>
>> Norbert Hartl wrote:
>>> I use the swazoo gs adapter.
>>> Norbert
>>> Am 21.09.2010 um 16:32 schrieb Sean Allen <[hidden email] <mailto:[hidden email]>>:
>>>> Are you using the FastCGI or Swazoo adapter?
>>>>
>>>> On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl < <mailto:[hidden email]>[hidden email] <mailto:[hidden email]>> wrote:
>>>>
>>>> I'm using daemontools for three weeks now and I discover severe
>>>> problems. One of my services is not fully ported to seaside 3.0
>>>> and throws some exceptions. What I experience is that all of my
>>>> vms go just _offline_. I mean there is just no socket listening on
>>>> the configured port. Looking at svstat from daemontools it seems
>>>> that the vms do not restart.
>>>> I thought the vms are restartet and something prevents opening a
>>>> socket. But now it seems that the listening socket goes away but
>>>> the vm does not stop and therefor doesn't get restarted.
>>>> My scripts are roughly the ones described in the gemstone
>>>> daemontools howto. I have a run script the invokes another script
>>>> that the seaside 30 run script modified for logging and foreground
>>>> processing.
>>>>
>>>> What is the best way to check this kind of behaviour. I looked at
>>>> the log files but didn't find anything useful.
>>>>
>>>> Any ideas?
>>>>
>>>> Norbert
>>>>
>>>>
>>>>
>>
>

swazoo-hang.log (10K) Download Attachment

NorbertHartl

Re: debug help for gem

In reply to this post by NorbertHartl

I played a little more with cherokee and fastcgi (stubborn enough not to let it go). Finally I got something working and it is far from being bad. Cherokee has an interpreter mode for fastcgi processes. That means it cares about starting external processes for fastcgi. I created three instances and assigned them to one virtual server in round robin load balancing mode.

Everything I configured from the cherokee-admin web interface. Cherokee starts on request all of the three defined gems and accesses them in a load balancing fashion. So there is no need for daemontools. If a instance dies cherokee starts it automatically. I didn't test but the setting does not seem fastcgi bound.

I try to iron out the setting and hopefully I'll find some time to blog about it.

Norbert

On 22.09.2010, at 22:24, Norbert Hartl wrote:

Thanks,

that might be the way to go. I'll just spend a couple of vms and a restart mechanism. Cherokee is quite good in detecting none working sources.
I don't get fastcgi working with cherokee. I would be interested to test how this all works if cherokee starts the vm instances. But either I'm too stupid to set it up or I don't know enough about fastcgi.

Norbert

On 22.09.2010, at 20:58, Sean Allen wrote:

I actually had a ruby script that made a connection to each gem and if it didnt respond properly, would do a system call to restart the swazoo listeners.
I dont use daemontools, can you not have it try to get a particular url? Monit and God amongst others both provide that functionality.

On Wed, Sep 22, 2010 at 2:17 PM, Dale Henrichs <[hidden email]> wrote:
Yes, daemontools doesn't help if the gem doesn't fall over when it is sick ...

Dale

Sean Allen wrote:

I ran into problems with the swazoo adapter becoming unresponsive but not actually dying. From talking to Dale a few months ago that was on the radar but wasn't an immediate fix. If using daemontools to restart doesnt work ( it seems you have that addressed now ), the proscribed solution is to use the FastCGI adapter as it is the battle tested method.

On Tue, Sep 21, 2010 at 6:41 PM, Norbert Hartl <[hidden email] <mailto:[hidden email]>> wrote:

I use the swazoo gs adapter.

Norbert

Am 21.09.2010 um 16:32 schrieb Sean Allen
<[hidden email] <mailto:[hidden email]>>:

Are you using the FastCGI or Swazoo adapter?

On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl <
<mailto:[hidden email]>[hidden email]
<mailto:[hidden email]>> wrote:

I'm using daemontools for three weeks now and I discover
severe problems. One of my services is not fully ported to
seaside 3.0 and throws some exceptions. What I experience is
that all of my vms go just _offline_. I mean there is just no
socket listening on the configured port. Looking at svstat
from daemontools it seems that the vms do not restart.
I thought the vms are restartet and something prevents opening
a socket. But now it seems that the listening socket goes away
but the vm does not stop and therefor doesn't get restarted.
My scripts are roughly the ones described in the gemstone
daemontools howto. I have a run script the invokes another
script that the seaside 30 run script modified for logging and
foreground processing.

What is the best way to check this kind of behaviour. I looked
at the log files but didn't find anything useful.

Any ideas?

Norbert

Dale Henrichs

Re: debug help for gem

In reply to this post by NorbertHartl

Norbert,

This looks the source of the hang and from the stack trace the vm is
indeed sitting in that delay loop... It looks like instead of an
unconditional infinite loop, the true should be replaced by a variable
that is set to false if an error is encountered ... an ensure block
around the listener might do the trick...

I've submitted Issue 181
(http://code.google.com/p/glassdb/issues/detail?id=181) to track the issue.

Dale

Norbert Hartl wrote: