debug help for gem

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

debug help for gem

NorbertHartl
I'm using daemontools for three weeks now and I discover severe problems. One of my services is not fully ported to seaside 3.0 and throws some exceptions. What I experience is that all of my vms go just _offline_. I mean there is just no socket listening on the configured port. Looking at svstat from daemontools it seems that the vms do not restart.
I thought the vms are restartet and something prevents opening a socket. But now it seems that the listening socket goes away but the vm does not stop and therefor doesn't get restarted.
My scripts are roughly the ones described in the gemstone daemontools howto. I have a run script the invokes another script that the seaside 30 run script modified for logging and foreground processing.

What is the best way to check this kind of behaviour. I looked at the log files but didn't find anything useful.

Any ideas?

Norbert

 
Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

Adrian Lienhard
Hi Norbert,

We have seen a similar problem (although very seldom) with Squeak and Daemontools. Since this is not Gemstone it may not be related at all but I thought I would mention it anyway.

The image suddenly stops listening on the port on which WAKom was started, however, as one can see with lsof the image listens on some other port, but it hasn't been restarted. This is really strange and we haven't figured out why that can happen.

Cheers,
Adrian

On Sep 21, 2010, at 09:34 , Norbert Hartl wrote:

> I'm using daemontools for three weeks now and I discover severe problems. One of my services is not fully ported to seaside 3.0 and throws some exceptions. What I experience is that all of my vms go just _offline_. I mean there is just no socket listening on the configured port. Looking at svstat from daemontools it seems that the vms do not restart.
> I thought the vms are restartet and something prevents opening a socket. But now it seems that the listening socket goes away but the vm does not stop and therefor doesn't get restarted.
> My scripts are roughly the ones described in the gemstone daemontools howto. I have a run script the invokes another script that the seaside 30 run script modified for logging and foreground processing.
>
> What is the best way to check this kind of behaviour. I looked at the log files but didn't find anything useful.
>
> Any ideas?
>
> Norbert
>

Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

Tobias Pape
In reply to this post by NorbertHartl
Am 2010-09-21 um 09:34 schrieb Norbert Hartl:
> I'm using daemontools for three weeks now and I discover severe problems. One of my services is not fully ported to seaside 3.0 and throws some exceptions. What I experience is that all of my vms go just _offline_. I mean there is just no socket listening on the configured port. Looking at svstat from daemontools it seems that the vms do not restart.
> I thought the vms are restartet and something prevents opening a socket. But now it seems that the listening socket goes away but the vm does not stop and therefor doesn't get restarted.
> My scripts are roughly the ones described in the gemstone daemontools howto. I have a run script the invokes another script that the seaside 30 run script modified for logging and foreground processing.
>
> What is the best way to check this kind of behaviour. I looked at the log files but didn't find anything useful.
>
> Any ideas?


Jep.

But I need a bit of information:

a) the output of svstat
b) the output of 'ps -ef |grep topaz' when the gems are apparently not restarting
c) the ./run -script you use to start the gem
d) the line containing 'topaz' from the script started from the run script.

So long,
        -Tobias

Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

NorbertHartl

On 21.09.2010, at 11:13, Tobias Pape wrote:

> Am 2010-09-21 um 09:34 schrieb Norbert Hartl:
>> I'm using daemontools for three weeks now and I discover severe problems. One of my services is not fully ported to seaside 3.0 and throws some exceptions. What I experience is that all of my vms go just _offline_. I mean there is just no socket listening on the configured port. Looking at svstat from daemontools it seems that the vms do not restart.
>> I thought the vms are restartet and something prevents opening a socket. But now it seems that the listening socket goes away but the vm does not stop and therefor doesn't get restarted.
>> My scripts are roughly the ones described in the gemstone daemontools howto. I have a run script the invokes another script that the seaside 30 run script modified for logging and foreground processing.
>>
>> What is the best way to check this kind of behaviour. I looked at the log files but didn't find anything useful.
>>
>> Any ideas?
>
>
> Jep.
>
> But I need a bit of information:
>
> a) the output of svstat

svstat lists all instances as running and the uptime matches the last manual restart time.

> b) the output of 'ps -ef |grep topaz' when the gems are apparently not restarting

All vms are still running. Means they look normal no zombie state or similar

> c) the ./run -script you use to start the gem

I have in /etc/service/[service]-[portnumber]/run script that looks like

#!/bin/sh

source /opt/gemstone/product/seaside/defSeaside
exec setuidgid gemstone /opt/gemstone/bin/start-[gemstone user] "www" 9005

> d) the line containing 'topaz' from the script started from the run script.
>
The start-* scripts are basically a copied version of startSeaside30_Adaptor. I only changed the execution contained in the shipped script to

exec $GEMSTONE/bin/topaz -l -T50000 2>&1 << EOF

Ok, that is the information you wanted. I'm not 100% sure how the svstat uptimes relate to the process life time. But having exec all the way down should make the uptime that svstat reports the uptime of the topaz process. If this is assured then I could say the stale topaz process is still the one that was initially started. But that would mean to me that something (probably an exception) forces the gem out of the listen loop without shutting the whole vm down.

thanks,

Norbert  

Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

Tobias Pape
>>
>> Jep.
>>
>> But I need a bit of information:
>>
>> a) the output of svstat
>
> svstat lists all instances as running and the uptime matches the last manual restart time.

thats good.

>
>> b) the output of 'ps -ef |grep topaz' when the gems are apparently not restarting
>
> All vms are still running. Means they look normal no zombie state or similar

ok,

>
>> c) the ./run -script you use to start the gem
>
> I have in /etc/service/[service]-[portnumber]/run script that looks like
>
> #!/bin/sh
>
> source /opt/gemstone/product/seaside/defSeaside
> exec setuidgid gemstone /opt/gemstone/bin/start-[gemstone user] "www" 9005
>
>> d) the line containing 'topaz' from the script started from the run script.
>>
> The start-* scripts are basically a copied version of startSeaside30_Adaptor. I only changed the execution contained in the shipped script to
>
> exec $GEMSTONE/bin/topaz -l -T50000 2>&1 << EOF


both look fine to me

>
> Ok, that is the information you wanted. I'm not 100% sure how the svstat uptimes relate to the process life time. But having exec all the way down should make the uptime that svstat reports the uptime of the topaz process.

should.

> If this is assured then I could say the stale topaz process is still the one that was initially started.

that is, there is one topaz not relating to any daemontools-started gem?
if it is not the maintainance-gem, kill it, it probably keeps the port open.
use lsof -i to see all open and listening ports.

> But that would mean to me that something (probably an exception) forces the gem out of the listen loop without shutting the whole vm down.

Depends on the exception (whether caught by the topaz-script or your application) and the code
that is following the "server starting" command (Swazoo/FCGI/etc..)


So Long,
        -Tobias
Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

SeanTAllen
In reply to this post by NorbertHartl
Are you using the FastCGI or Swazoo adapter?

On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl <[hidden email]> wrote:
I'm using daemontools for three weeks now and I discover severe problems. One of my services is not fully ported to seaside 3.0 and throws some exceptions. What I experience is that all of my vms go just _offline_. I mean there is just no socket listening on the configured port. Looking at svstat from daemontools it seems that the vms do not restart.
I thought the vms are restartet and something prevents opening a socket. But now it seems that the listening socket goes away but the vm does not stop and therefor doesn't get restarted.
My scripts are roughly the ones described in the gemstone daemontools howto. I have a run script the invokes another script that the seaside 30 run script modified for logging and foreground processing.

What is the best way to check this kind of behaviour. I looked at the log files but didn't find anything useful.

Any ideas?

Norbert

 

Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

NorbertHartl
I use the swazoo gs adapter.

Norbert



Am 21.09.2010 um 16:32 schrieb Sean Allen <[hidden email]>:

Are you using the FastCGI or Swazoo adapter?

On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl <[hidden email]> wrote:
I'm using daemontools for three weeks now and I discover severe problems. One of my services is not fully ported to seaside 3.0 and throws some exceptions. What I experience is that all of my vms go just _offline_. I mean there is just no socket listening on the configured port. Looking at svstat from daemontools it seems that the vms do not restart.
I thought the vms are restartet and something prevents opening a socket. But now it seems that the listening socket goes away but the vm does not stop and therefor doesn't get restarted.
My scripts are roughly the ones described in the gemstone daemontools howto. I have a run script the invokes another script that the seaside 30 run script modified for logging and foreground processing.

What is the best way to check this kind of behaviour. I looked at the log files but didn't find anything useful.

Any ideas?

Norbert

 

Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

NorbertHartl
In reply to this post by Adrian Lienhard

On 21.09.2010, at 09:46, Adrian Lienhard wrote:

> Hi Norbert,
>
> We have seen a similar problem (although very seldom) with Squeak and Daemontools. Since this is not Gemstone it may not be related at all but I thought I would mention it anyway.
>
> The image suddenly stops listening on the port on which WAKom was started, however, as one can see with lsof the image listens on some other port, but it hasn't been restarted. This is really strange and we haven't figured out why that can happen.
>
>
In my case it is different. The socket on the port just vanishes completely but the process has not been restarted. Well, I trying to monitor this in more depth.

Norbert

>
> On Sep 21, 2010, at 09:34 , Norbert Hartl wrote:
>
>> I'm using daemontools for three weeks now and I discover severe problems. One of my services is not fully ported to seaside 3.0 and throws some exceptions. What I experience is that all of my vms go just _offline_. I mean there is just no socket listening on the configured port. Looking at svstat from daemontools it seems that the vms do not restart.
>> I thought the vms are restartet and something prevents opening a socket. But now it seems that the listening socket goes away but the vm does not stop and therefor doesn't get restarted.
>> My scripts are roughly the ones described in the gemstone daemontools howto. I have a run script the invokes another script that the seaside 30 run script modified for logging and foreground processing.
>>
>> What is the best way to check this kind of behaviour. I looked at the log files but didn't find anything useful.
>>
>> Any ideas?
>>
>> Norbert
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

NorbertHartl
In reply to this post by Tobias Pape

On 21.09.2010, at 13:36, Tobias Pape wrote:

>>>
>>> Jep.
>>>
>>> But I need a bit of information:
>>>
>>> a) the output of svstat
>>
>> svstat lists all instances as running and the uptime matches the last manual restart time.
>
> thats good.
>
>>
>>> b) the output of 'ps -ef |grep topaz' when the gems are apparently not restarting
>>
>> All vms are still running. Means they look normal no zombie state or similar
>
> ok,
>
>>
>>> c) the ./run -script you use to start the gem
>>
>> I have in /etc/service/[service]-[portnumber]/run script that looks like
>>
>> #!/bin/sh
>>
>> source /opt/gemstone/product/seaside/defSeaside
>> exec setuidgid gemstone /opt/gemstone/bin/start-[gemstone user] "www" 9005
>>
>>> d) the line containing 'topaz' from the script started from the run script.
>>>
>> The start-* scripts are basically a copied version of startSeaside30_Adaptor. I only changed the execution contained in the shipped script to
>>
>> exec $GEMSTONE/bin/topaz -l -T50000 2>&1 << EOF
>
>
> both look fine to me
>
>>
>> Ok, that is the information you wanted. I'm not 100% sure how the svstat uptimes relate to the process life time. But having exec all the way down should make the uptime that svstat reports the uptime of the topaz process.
>
> should.
>
>> If this is assured then I could say the stale topaz process is still the one that was initially started.
>
> that is, there is one topaz not relating to any daemontools-started gem?

No.

> if it is not the maintainance-gem, kill it, it probably keeps the port open.
> use lsof -i to see all open and listening ports.
>
If the image would have been restarted the case would be more clear to me. There are no additional processes that can sit on a certain port. I think there is hardly a chance to get a second process started if you are using daemontools. But the same problem could appear even if there is no second process. After listening sockets go away there can be already established connections to this port. Unless the operating system got the chance to clean up (fin_wait2 problem) you might get in trouble opening the same port again. I checked that and can't be the problem because nothing is restarted.

>> But that would mean to me that something (probably an exception) forces the gem out of the listen loop without shutting the whole vm down.
>
> Depends on the exception (whether caught by the topaz-script or your application) and the code
> that is following the "server starting" command (Swazoo/FCGI/etc..)
>
I think we are talking about uncaught exceptions. I forgot again what the installStaticException part in the topaz script exactly does. But apart from this it seems to work like before. If you have an unhandled exception then this breaks out of the

WAGemStoneRunSeasideGems startGemServerOn: $2

and should end the script and therefor exit the process.

Norbert

Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

NorbertHartl
I found something. Daemontools logging wasn't set up properly so I missed important stuff.

There are two commandline parameter to the script startSeaside30_Adaptor. The second one is the port and the first one is string that is just used for generating filenames and log messages. A good idea could be that the first argument choses the adaptor being used by the gem. But it doesn't.

I have as run script for daemontools the following:

---
#!/bin/sh

source /opt/gemstone/product/seaside/defSeaside
exec setuidgid gemstone /opt/gemstone/bin/start-rosso3 "www" 9005
---
As you can see I just use any "label" as first argument to the start script. The start script looks like this at the end:

---
 1 WAGemStoneRunSeasideGems startGemServerOn: $2.
 2 "does not return"
 3 %
 4 run
 5 GemToGemAnnouncement uninstallStaticHandler.
 6 System beginTransaction.
 7 (ObjectLogEntry
 8   fatal: '$2: topaz exit'
 9   object:
10     'port: ', $1 printString, ' ',
11     'pid: ', (System gemVersionReport at: 'processId') printString) addToLog.
12 System commitTransaction.
---

The first line starts the blocking server adaptor. If there is an exception it returns. While creating the objectlog entry there ocurrs another exception. After shell replacement line 10 reads this way

'port: ', www printString, '',

and www is an undefined symbol. My assumption would be that the second exception keeps the gem from exiting somehow. So the server adaptor has been shutdown and the process is stale. That would at least explain my situation completely. Well, I think I know by tomorrow or the day after. In the meantime I'm curious what the people that know what is happening have to say.

thanks,

Norbert

On 22.09.2010, at 11:47, Norbert Hartl wrote:

>
> On 21.09.2010, at 13:36, Tobias Pape wrote:
>
>>>>
>>>> Jep.
>>>>
>>>> But I need a bit of information:
>>>>
>>>> a) the output of svstat
>>>
>>> svstat lists all instances as running and the uptime matches the last manual restart time.
>>
>> thats good.
>>
>>>
>>>> b) the output of 'ps -ef |grep topaz' when the gems are apparently not restarting
>>>
>>> All vms are still running. Means they look normal no zombie state or similar
>>
>> ok,
>>
>>>
>>>> c) the ./run -script you use to start the gem
>>>
>>> I have in /etc/service/[service]-[portnumber]/run script that looks like
>>>
>>> #!/bin/sh
>>>
>>> source /opt/gemstone/product/seaside/defSeaside
>>> exec setuidgid gemstone /opt/gemstone/bin/start-[gemstone user] "www" 9005
>>>
>>>> d) the line containing 'topaz' from the script started from the run script.
>>>>
>>> The start-* scripts are basically a copied version of startSeaside30_Adaptor. I only changed the execution contained in the shipped script to
>>>
>>> exec $GEMSTONE/bin/topaz -l -T50000 2>&1 << EOF
>>
>>
>> both look fine to me
>>
>>>
>>> Ok, that is the information you wanted. I'm not 100% sure how the svstat uptimes relate to the process life time. But having exec all the way down should make the uptime that svstat reports the uptime of the topaz process.
>>
>> should.
>>
>>> If this is assured then I could say the stale topaz process is still the one that was initially started.
>>
>> that is, there is one topaz not relating to any daemontools-started gem?
>
> No.
>
>> if it is not the maintainance-gem, kill it, it probably keeps the port open.
>> use lsof -i to see all open and listening ports.
>>
> If the image would have been restarted the case would be more clear to me. There are no additional processes that can sit on a certain port. I think there is hardly a chance to get a second process started if you are using daemontools. But the same problem could appear even if there is no second process. After listening sockets go away there can be already established connections to this port. Unless the operating system got the chance to clean up (fin_wait2 problem) you might get in trouble opening the same port again. I checked that and can't be the problem because nothing is restarted.
>
>>> But that would mean to me that something (probably an exception) forces the gem out of the listen loop without shutting the whole vm down.
>>
>> Depends on the exception (whether caught by the topaz-script or your application) and the code
>> that is following the "server starting" command (Swazoo/FCGI/etc..)
>>
> I think we are talking about uncaught exceptions. I forgot again what the installStaticException part in the topaz script exactly does. But apart from this it seems to work like before. If you have an unhandled exception then this breaks out of the
>
> WAGemStoneRunSeasideGems startGemServerOn: $2
>
> and should end the script and therefor exit the process.
>
> Norbert
>

Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

SeanTAllen
In reply to this post by NorbertHartl
I ran into problems with the swazoo adapter becoming unresponsive but not actually dying. From talking to Dale a few months ago that was on the radar but wasn't an immediate fix. If using daemontools to restart doesnt work ( it seems you have that addressed now ), the proscribed solution is to use the FastCGI adapter as it is the battle tested method.

On Tue, Sep 21, 2010 at 6:41 PM, Norbert Hartl <[hidden email]> wrote:
I use the swazoo gs adapter.

Norbert



Am 21.09.2010 um 16:32 schrieb Sean Allen <[hidden email]>:

Are you using the FastCGI or Swazoo adapter?

On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl <[hidden email][hidden email]> wrote:
I'm using daemontools for three weeks now and I discover severe problems. One of my services is not fully ported to seaside 3.0 and throws some exceptions. What I experience is that all of my vms go just _offline_. I mean there is just no socket listening on the configured port. Looking at svstat from daemontools it seems that the vms do not restart.
I thought the vms are restartet and something prevents opening a socket. But now it seems that the listening socket goes away but the vm does not stop and therefor doesn't get restarted.
My scripts are roughly the ones described in the gemstone daemontools howto. I have a run script the invokes another script that the seaside 30 run script modified for logging and foreground processing.

What is the best way to check this kind of behaviour. I looked at the log files but didn't find anything useful.

Any ideas?

Norbert

 


Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

Dale Henrichs
In reply to this post by NorbertHartl
Norbert,

If I'm following things, it seems that the swazoo vm is alive, but stops
listening on the port?

A 'kill -USR1' should give you a smalltalk stack trace in the server log
so that you can tell what is going on inside the server....

This seems to be similar to a problem that Sean had with Swazoo? If I
recall he had gems running hot and not listening/responding...

At this point in time, I'm wondering if I should create a Seaside 3.0
adaptor for Hyper. Hyper didn't have these kinds of bugs (I don't
think). I remember that when I was porting Swazoo I noticed that the
structure of the server was quite different between Hyper and Swazoo...

I've got a couple of projects stacked up right now, but a Hyper adaptor
might make sense ... what do you think?

Dale


  Norbert Hartl wrote:

> I use the swazoo gs adapter.
>
> Norbert
>
>
>
> Am 21.09.2010 um 16:32 schrieb Sean Allen <[hidden email]
> <mailto:[hidden email]>>:
>
>> Are you using the FastCGI or Swazoo adapter?
>>
>> On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl <
>> <mailto:[hidden email]>[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     I'm using daemontools for three weeks now and I discover severe
>>     problems. One of my services is not fully ported to seaside 3.0
>>     and throws some exceptions. What I experience is that all of my
>>     vms go just _offline_. I mean there is just no socket listening on
>>     the configured port. Looking at svstat from daemontools it seems
>>     that the vms do not restart.
>>     I thought the vms are restartet and something prevents opening a
>>     socket. But now it seems that the listening socket goes away but
>>     the vm does not stop and therefor doesn't get restarted.
>>     My scripts are roughly the ones described in the gemstone
>>     daemontools howto. I have a run script the invokes another script
>>     that the seaside 30 run script modified for logging and foreground
>>     processing.
>>
>>     What is the best way to check this kind of behaviour. I looked at
>>     the log files but didn't find anything useful.
>>
>>     Any ideas?
>>
>>     Norbert
>>
>>      
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

Dale Henrichs
In reply to this post by SeanTAllen
Yes, daemontools doesn't help if the gem doesn't fall over when it is
sick ...

Dale

Sean Allen wrote:

> I ran into problems with the swazoo adapter becoming unresponsive but
> not actually dying. From talking to Dale a few months ago that was on
> the radar but wasn't an immediate fix. If using daemontools to restart
> doesnt work ( it seems you have that addressed now ), the proscribed
> solution is to use the FastCGI adapter as it is the battle tested method.
>
> On Tue, Sep 21, 2010 at 6:41 PM, Norbert Hartl <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I use the swazoo gs adapter.
>
>     Norbert
>
>
>
>     Am 21.09.2010 um 16:32 schrieb Sean Allen
>     <[hidden email] <mailto:[hidden email]>>:
>
>>     Are you using the FastCGI or Swazoo adapter?
>>
>>     On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl <
>>     <mailto:[hidden email]>[hidden email]
>>     <mailto:[hidden email]>> wrote:
>>
>>         I'm using daemontools for three weeks now and I discover
>>         severe problems. One of my services is not fully ported to
>>         seaside 3.0 and throws some exceptions. What I experience is
>>         that all of my vms go just _offline_. I mean there is just no
>>         socket listening on the configured port. Looking at svstat
>>         from daemontools it seems that the vms do not restart.
>>         I thought the vms are restartet and something prevents opening
>>         a socket. But now it seems that the listening socket goes away
>>         but the vm does not stop and therefor doesn't get restarted.
>>         My scripts are roughly the ones described in the gemstone
>>         daemontools howto. I have a run script the invokes another
>>         script that the seaside 30 run script modified for logging and
>>         foreground processing.
>>
>>         What is the best way to check this kind of behaviour. I looked
>>         at the log files but didn't find anything useful.
>>
>>         Any ideas?
>>
>>         Norbert
>>
>>          
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

SeanTAllen
I actually had a ruby script that made a connection to each gem and if it didnt respond properly, would do a system call to restart the swazoo listeners.
I dont use daemontools, can you not have it try to get a particular url? Monit and God amongst others both provide that functionality.

On Wed, Sep 22, 2010 at 2:17 PM, Dale Henrichs <[hidden email]> wrote:
Yes, daemontools doesn't help if the gem doesn't fall over when it is sick ...

Dale

Sean Allen wrote:
I ran into problems with the swazoo adapter becoming unresponsive but not actually dying. From talking to Dale a few months ago that was on the radar but wasn't an immediate fix. If using daemontools to restart doesnt work ( it seems you have that addressed now ), the proscribed solution is to use the FastCGI adapter as it is the battle tested method.

On Tue, Sep 21, 2010 at 6:41 PM, Norbert Hartl <[hidden email] <mailto:[hidden email]>> wrote:

   I use the swazoo gs adapter.

   Norbert



   Am 21.09.2010 um 16:32 schrieb Sean Allen
   <[hidden email] <mailto:[hidden email]>>:

   Are you using the FastCGI or Swazoo adapter?

   On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl <
   <mailto:[hidden email]>> wrote:

       I'm using daemontools for three weeks now and I discover
       severe problems. One of my services is not fully ported to
       seaside 3.0 and throws some exceptions. What I experience is
       that all of my vms go just _offline_. I mean there is just no
       socket listening on the configured port. Looking at svstat
       from daemontools it seems that the vms do not restart.
       I thought the vms are restartet and something prevents opening
       a socket. But now it seems that the listening socket goes away
       but the vm does not stop and therefor doesn't get restarted.
       My scripts are roughly the ones described in the gemstone
       daemontools howto. I have a run script the invokes another
       script that the seaside 30 run script modified for logging and
       foreground processing.

       What is the best way to check this kind of behaviour. I looked
       at the log files but didn't find anything useful.

       Any ideas?

       Norbert

       




Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

SeanTAllen
In reply to this post by Dale Henrichs
I would go with hyper compared to swazoo. I got rather familiar with their guts last winter when I had to get them both to run in the small image ( SOAP requires Hyper ).
Swazoo has a bunch more features than Hyper that arent needed for a GLASS deployment. That is just extra code to possible have a bug hiding in it. I'd be curious to know
what features a GLASS webserver really needs. At least the 90 use case. I've been contemplating writing my own just because well, i've written http servers before and it might be fun.

On Wed, Sep 22, 2010 at 2:14 PM, Dale Henrichs <[hidden email]> wrote:
Norbert,

If I'm following things, it seems that the swazoo vm is alive, but stops listening on the port?

A 'kill -USR1' should give you a smalltalk stack trace in the server log so that you can tell what is going on inside the server....

This seems to be similar to a problem that Sean had with Swazoo? If I recall he had gems running hot and not listening/responding...

At this point in time, I'm wondering if I should create a Seaside 3.0 adaptor for Hyper. Hyper didn't have these kinds of bugs (I don't think). I remember that when I was porting Swazoo I noticed that the structure of the server was quite different between Hyper and Swazoo...

I've got a couple of projects stacked up right now, but a Hyper adaptor might make sense ... what do you think?

Dale


 Norbert Hartl wrote:
I use the swazoo gs adapter.

Norbert



Am 21.09.2010 um 16:32 schrieb Sean Allen <[hidden email] <mailto:[hidden email]>>:

Are you using the FastCGI or Swazoo adapter?

On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl < <mailto:[hidden email]>[hidden email] <mailto:[hidden email]>> wrote:

   I'm using daemontools for three weeks now and I discover severe
   problems. One of my services is not fully ported to seaside 3.0
   and throws some exceptions. What I experience is that all of my
   vms go just _offline_. I mean there is just no socket listening on
   the configured port. Looking at svstat from daemontools it seems
   that the vms do not restart.
   I thought the vms are restartet and something prevents opening a
   socket. But now it seems that the listening socket goes away but
   the vm does not stop and therefor doesn't get restarted.
   My scripts are roughly the ones described in the gemstone
   daemontools howto. I have a run script the invokes another script
   that the seaside 30 run script modified for logging and foreground
   processing.

   What is the best way to check this kind of behaviour. I looked at
   the log files but didn't find anything useful.

   Any ideas?

   Norbert

   



Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

Dale Henrichs
I went with Swazoo in December because the Seaside30 adaptor had already
been written for Swazoo, so it was expedient to go with Swazoo.

Now, I'm pretty familiar with the structure of the adaptors, so I should
be able to put one together for Hyper...

I'm not sure what to say about features:) Single listening port per vm,
a thread forked off per per request, pluggable handling of
request/response. Built in logging (with pluggability, so Object log can
be involved)...

The Web Client work going on for Pharo could be interesting to port to
GemStone, although I haven't had time to look at it and compare the
features to what's available with Hyper.

With that said, I think Bruce has more recent updates for Hyper, but it
is such a pain to get code from VW/Store to GLASS/Monticello that I
consider calling my dentist for a root canal, first:)

Dale


Sean Allen wrote:

> I would go with hyper compared to swazoo. I got rather familiar with
> their guts last winter when I had to get them both to run in the small
> image ( SOAP requires Hyper ).
> Swazoo has a bunch more features than Hyper that arent needed for a
> GLASS deployment. That is just extra code to possible have a bug hiding
> in it. I'd be curious to know
> what features a GLASS webserver really needs. At least the 90 use case.
> I've been contemplating writing my own just because well, i've written
> http servers before and it might be fun.
>
> On Wed, Sep 22, 2010 at 2:14 PM, Dale Henrichs <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Norbert,
>
>     If I'm following things, it seems that the swazoo vm is alive, but
>     stops listening on the port?
>
>     A 'kill -USR1' should give you a smalltalk stack trace in the server
>     log so that you can tell what is going on inside the server....
>
>     This seems to be similar to a problem that Sean had with Swazoo? If
>     I recall he had gems running hot and not listening/responding...
>
>     At this point in time, I'm wondering if I should create a Seaside
>     3.0 adaptor for Hyper. Hyper didn't have these kinds of bugs (I
>     don't think). I remember that when I was porting Swazoo I noticed
>     that the structure of the server was quite different between Hyper
>     and Swazoo...
>
>     I've got a couple of projects stacked up right now, but a Hyper
>     adaptor might make sense ... what do you think?
>
>     Dale
>
>
>      Norbert Hartl wrote:
>
>         I use the swazoo gs adapter.
>
>         Norbert
>
>
>
>         Am 21.09.2010 um 16:32 schrieb Sean Allen
>         <[hidden email]
>         <mailto:[hidden email]>
>         <mailto:[hidden email]
>         <mailto:[hidden email]>>>:
>
>             Are you using the FastCGI or Swazoo adapter?
>
>             On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl <
>             <mailto:[hidden email]
>             <mailto:[hidden email]>>[hidden email]
>             <mailto:[hidden email]> <mailto:[hidden email]
>             <mailto:[hidden email]>>> wrote:
>
>                I'm using daemontools for three weeks now and I discover
>             severe
>                problems. One of my services is not fully ported to
>             seaside 3.0
>                and throws some exceptions. What I experience is that all
>             of my
>                vms go just _offline_. I mean there is just no socket
>             listening on
>                the configured port. Looking at svstat from daemontools
>             it seems
>                that the vms do not restart.
>                I thought the vms are restartet and something prevents
>             opening a
>                socket. But now it seems that the listening socket goes
>             away but
>                the vm does not stop and therefor doesn't get restarted.
>                My scripts are roughly the ones described in the gemstone
>                daemontools howto. I have a run script the invokes
>             another script
>                that the seaside 30 run script modified for logging and
>             foreground
>                processing.
>
>                What is the best way to check this kind of behaviour. I
>             looked at
>                the log files but didn't find anything useful.
>
>                Any ideas?
>
>                Norbert
>
>                
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

SeanTAllen
If I remember correctly, expedient would have been the way to go as some people were waiting on an adapter.

Insert smile, wave and thanks again here.

On Wed, Sep 22, 2010 at 3:26 PM, Dale Henrichs <[hidden email]> wrote:
I went with Swazoo in December because the Seaside30 adaptor had already been written for Swazoo, so it was expedient to go with Swazoo.

Now, I'm pretty familiar with the structure of the adaptors, so I should be able to put one together for Hyper...

I'm not sure what to say about features:) Single listening port per vm, a thread forked off per per request, pluggable handling of request/response. Built in logging (with pluggability, so Object log can be involved)...

The Web Client work going on for Pharo could be interesting to port to GemStone, although I haven't had time to look at it and compare the features to what's available with Hyper.

With that said, I think Bruce has more recent updates for Hyper, but it is such a pain to get code from VW/Store to GLASS/Monticello that I consider calling my dentist for a root canal, first:)

Dale


Sean Allen wrote:
I would go with hyper compared to swazoo. I got rather familiar with their guts last winter when I had to get them both to run in the small image ( SOAP requires Hyper ).
Swazoo has a bunch more features than Hyper that arent needed for a GLASS deployment. That is just extra code to possible have a bug hiding in it. I'd be curious to know
what features a GLASS webserver really needs. At least the 90 use case. I've been contemplating writing my own just because well, i've written http servers before and it might be fun.

On Wed, Sep 22, 2010 at 2:14 PM, Dale Henrichs <[hidden email] <mailto:[hidden email]>> wrote:

   Norbert,

   If I'm following things, it seems that the swazoo vm is alive, but
   stops listening on the port?

   A 'kill -USR1' should give you a smalltalk stack trace in the server
   log so that you can tell what is going on inside the server....

   This seems to be similar to a problem that Sean had with Swazoo? If
   I recall he had gems running hot and not listening/responding...

   At this point in time, I'm wondering if I should create a Seaside
   3.0 adaptor for Hyper. Hyper didn't have these kinds of bugs (I
   don't think). I remember that when I was porting Swazoo I noticed
   that the structure of the server was quite different between Hyper
   and Swazoo...

   I've got a couple of projects stacked up right now, but a Hyper
   adaptor might make sense ... what do you think?

   Dale


    Norbert Hartl wrote:

       I use the swazoo gs adapter.

       Norbert



       Am 21.09.2010 um 16:32 schrieb Sean Allen
       <[hidden email]
       <mailto:[hidden email]>
       <mailto:[hidden email]

       <mailto:[hidden email]>>>:

           Are you using the FastCGI or Swazoo adapter?

           On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl <
           <mailto:[hidden email]
           <mailto:[hidden email]>>[hidden email]
           <mailto:[hidden email]> <mailto:[hidden email]

           <mailto:[hidden email]>>> wrote:

              I'm using daemontools for three weeks now and I discover
           severe
              problems. One of my services is not fully ported to
           seaside 3.0
              and throws some exceptions. What I experience is that all
           of my
              vms go just _offline_. I mean there is just no socket
           listening on
              the configured port. Looking at svstat from daemontools
           it seems
              that the vms do not restart.
              I thought the vms are restartet and something prevents
           opening a
              socket. But now it seems that the listening socket goes
           away but
              the vm does not stop and therefor doesn't get restarted.
              My scripts are roughly the ones described in the gemstone
              daemontools howto. I have a run script the invokes
           another script
              that the seaside 30 run script modified for logging and
           foreground
              processing.

              What is the best way to check this kind of behaviour. I
           looked at
              the log files but didn't find anything useful.

              Any ideas?

              Norbert

             




Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

NorbertHartl
In reply to this post by Dale Henrichs

On 22.09.2010, at 20:14, Dale Henrichs wrote:

Norbert,

If I'm following things, it seems that the swazoo vm is alive, but stops listening on the port?

yep

A 'kill -USR1' should give you a smalltalk stack trace in the server log so that you can tell what is going on inside the server....

I forget this all the time.

This seems to be similar to a problem that Sean had with Swazoo? If I recall he had gems running hot and not listening/responding...

At this point in time, I'm wondering if I should create a Seaside 3.0 adaptor for Hyper. Hyper didn't have these kinds of bugs (I don't think). I remember that when I was porting Swazoo I noticed that the structure of the server was quite different between Hyper and Swazoo...

I can't tell really. I spent an hour to convince cherokee to work with fastcgi but I don't get it working. And the other half of the story is that I don't like fastcgi at all and I don't even know what it is good for in this environment. 

I've got a couple of projects stacked up right now, but a Hyper adaptor might make sense ... what do you think?

Well, I'm facing either a fastcgi adapter that I don't like or a swazoo adapter that doesn't work that good. From this particular point of view hyper seems a good thing to me :) At least I know that hyper is the first server that dies happily (and fast) if something goes wrong.

Norbert


Norbert Hartl wrote:
I use the swazoo gs adapter.
Norbert
Am 21.09.2010 um 16:32 schrieb Sean Allen <[hidden email] <[hidden email]>>:
Are you using the FastCGI or Swazoo adapter?

On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl < <[hidden email]>[hidden email] <[hidden email]>> wrote:

   I'm using daemontools for three weeks now and I discover severe
   problems. One of my services is not fully ported to seaside 3.0
   and throws some exceptions. What I experience is that all of my
   vms go just _offline_. I mean there is just no socket listening on
   the configured port. Looking at svstat from daemontools it seems
   that the vms do not restart.
   I thought the vms are restartet and something prevents opening a
   socket. But now it seems that the listening socket goes away but
   the vm does not stop and therefor doesn't get restarted.
   My scripts are roughly the ones described in the gemstone
   daemontools howto. I have a run script the invokes another script
   that the seaside 30 run script modified for logging and foreground
   processing.

   What is the best way to check this kind of behaviour. I looked at
   the log files but didn't find anything useful.

   Any ideas?

   Norbert

    



Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

NorbertHartl
In reply to this post by SeanTAllen
Thanks,

that might be the way to go. I'll just spend a couple of vms and a restart mechanism. Cherokee is quite good in detecting none working sources. 
I don't get fastcgi working with cherokee. I would be interested to test how this all works if cherokee starts the vm instances. But either I'm too stupid to set it up or I don't know enough about fastcgi.

Norbert

On 22.09.2010, at 20:58, Sean Allen wrote:

I actually had a ruby script that made a connection to each gem and if it didnt respond properly, would do a system call to restart the swazoo listeners.
I dont use daemontools, can you not have it try to get a particular url? Monit and God amongst others both provide that functionality.

On Wed, Sep 22, 2010 at 2:17 PM, Dale Henrichs <[hidden email]> wrote:
Yes, daemontools doesn't help if the gem doesn't fall over when it is sick ...

Dale

Sean Allen wrote:
I ran into problems with the swazoo adapter becoming unresponsive but not actually dying. From talking to Dale a few months ago that was on the radar but wasn't an immediate fix. If using daemontools to restart doesnt work ( it seems you have that addressed now ), the proscribed solution is to use the FastCGI adapter as it is the battle tested method.

On Tue, Sep 21, 2010 at 6:41 PM, Norbert Hartl <[hidden email] <mailto:[hidden email]>> wrote:

   I use the swazoo gs adapter.

   Norbert



   Am 21.09.2010 um 16:32 schrieb Sean Allen
   <[hidden email] <mailto:[hidden email]>>:

   Are you using the FastCGI or Swazoo adapter?

   On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl <
   <mailto:[hidden email]>> wrote:

       I'm using daemontools for three weeks now and I discover
       severe problems. One of my services is not fully ported to
       seaside 3.0 and throws some exceptions. What I experience is
       that all of my vms go just _offline_. I mean there is just no
       socket listening on the configured port. Looking at svstat
       from daemontools it seems that the vms do not restart.
       I thought the vms are restartet and something prevents opening
       a socket. But now it seems that the listening socket goes away
       but the vm does not stop and therefor doesn't get restarted.
       My scripts are roughly the ones described in the gemstone
       daemontools howto. I have a run script the invokes another
       script that the seaside 30 run script modified for logging and
       foreground processing.

       What is the best way to check this kind of behaviour. I looked
       at the log files but didn't find anything useful.

       Any ideas?

       Norbert

       





Reply | Threaded
Open this post in threaded view
|

Re: debug help for gem

Dale Henrichs
In reply to this post by SeanTAllen
Sean Allen wrote:
> If I remember correctly, expedient would have been the way to go as some
> people were waiting on an adapter.
>
> Insert smile, wave and thanks again here.

Haha! Exactamudo and you're welcome:)

Dale

>
> On Wed, Sep 22, 2010 at 3:26 PM, Dale Henrichs <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I went with Swazoo in December because the Seaside30 adaptor had
>     already been written for Swazoo, so it was expedient to go with Swazoo.
>
>     Now, I'm pretty familiar with the structure of the adaptors, so I
>     should be able to put one together for Hyper...
>
>     I'm not sure what to say about features:) Single listening port per
>     vm, a thread forked off per per request, pluggable handling of
>     request/response. Built in logging (with pluggability, so Object log
>     can be involved)...
>
>     The Web Client work going on for Pharo could be interesting to port
>     to GemStone, although I haven't had time to look at it and compare
>     the features to what's available with Hyper.
>
>     With that said, I think Bruce has more recent updates for Hyper, but
>     it is such a pain to get code from VW/Store to GLASS/Monticello that
>     I consider calling my dentist for a root canal, first:)
>
>     Dale
>
>
>     Sean Allen wrote:
>
>         I would go with hyper compared to swazoo. I got rather familiar
>         with their guts last winter when I had to get them both to run
>         in the small image ( SOAP requires Hyper ).
>         Swazoo has a bunch more features than Hyper that arent needed
>         for a GLASS deployment. That is just extra code to possible have
>         a bug hiding in it. I'd be curious to know
>         what features a GLASS webserver really needs. At least the 90
>         use case. I've been contemplating writing my own just because
>         well, i've written http servers before and it might be fun.
>
>         On Wed, Sep 22, 2010 at 2:14 PM, Dale Henrichs
>         <[hidden email] <mailto:[hidden email]>
>         <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>
>            Norbert,
>
>            If I'm following things, it seems that the swazoo vm is
>         alive, but
>            stops listening on the port?
>
>            A 'kill -USR1' should give you a smalltalk stack trace in the
>         server
>            log so that you can tell what is going on inside the server....
>
>            This seems to be similar to a problem that Sean had with
>         Swazoo? If
>            I recall he had gems running hot and not listening/responding...
>
>            At this point in time, I'm wondering if I should create a Seaside
>            3.0 adaptor for Hyper. Hyper didn't have these kinds of bugs (I
>            don't think). I remember that when I was porting Swazoo I noticed
>            that the structure of the server was quite different between
>         Hyper
>            and Swazoo...
>
>            I've got a couple of projects stacked up right now, but a Hyper
>            adaptor might make sense ... what do you think?
>
>            Dale
>
>
>             Norbert Hartl wrote:
>
>                I use the swazoo gs adapter.
>
>                Norbert
>
>
>
>                Am 21.09.2010 um 16:32 schrieb Sean Allen
>                <[hidden email]
>         <mailto:[hidden email]>
>                <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                <mailto:[hidden email]
>         <mailto:[hidden email]>
>
>                <mailto:[hidden email]
>         <mailto:[hidden email]>>>>:
>
>                    Are you using the FastCGI or Swazoo adapter?
>
>                    On Tue, Sep 21, 2010 at 3:34 AM, Norbert Hartl <
>                    <mailto:[hidden email] <mailto:[hidden email]>
>                    <mailto:[hidden email]
>         <mailto:[hidden email]>>>[hidden email]
>         <mailto:[hidden email]>
>                    <mailto:[hidden email]
>         <mailto:[hidden email]>> <mailto:[hidden email]
>         <mailto:[hidden email]>
>
>                    <mailto:[hidden email]
>         <mailto:[hidden email]>>>> wrote:
>
>                       I'm using daemontools for three weeks now and I
>         discover
>                    severe
>                       problems. One of my services is not fully ported to
>                    seaside 3.0
>                       and throws some exceptions. What I experience is
>         that all
>                    of my
>                       vms go just _offline_. I mean there is just no socket
>                    listening on
>                       the configured port. Looking at svstat from
>         daemontools
>                    it seems
>                       that the vms do not restart.
>                       I thought the vms are restartet and something prevents
>                    opening a
>                       socket. But now it seems that the listening socket
>         goes
>                    away but
>                       the vm does not stop and therefor doesn't get
>         restarted.
>                       My scripts are roughly the ones described in the
>         gemstone
>                       daemontools howto. I have a run script the invokes
>                    another script
>                       that the seaside 30 run script modified for
>         logging and
>                    foreground
>                       processing.
>
>                       What is the best way to check this kind of
>         behaviour. I
>                    looked at
>                       the log files but didn't find anything useful.
>
>                       Any ideas?
>
>                       Norbert
>
>                      
>
>
>
>

12