Web request problem on server with two Glass environment

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Web request problem on server with two Glass environment

dario trussardi
Ciao,

i have a server based on Ubuntu and Lighttpd.

On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.

One work with :
Starting WAFastCGIAdaptor gem on port 9030
Starting WAFastCGIAdaptor gem on port 9031
Starting WAFastCGIAdaptor gem on port 9032
and other with:

Starting WAFastCGIAdaptor gem on port 9020
Starting WAFastCGIAdaptor gem on port 9021
Starting WAFastCGIAdaptor gem on port 9022

All work fine for more time and the lighttpd redirect the request to the specific environment.

 This is one example of lighttpd configuration:

$HTTP["host"] =~ "dariospizza.com$" {

  url.redirect = (

    "^/$"  => "http://www.dariospizza.com/dariospizza"

  )

  $HTTP["url"] =~ "^/dariospizza" {

    fastcgi.server = (

      "/dariospizza" =>

      (

        ("host" => "192.168.1.100","port" => 9020,"check-local" => "disable","mode" => "responder" ),

        ("host" => "192.168.1.100","port" => 9021,"check-local" => "disable","mode" => "responder" ),

        ("host" => "192.168.1.100","port" => 9022,"check-local" => "disable","mode" => "responder" )

      )

    )

  }

  $HTTP["url"] =~ "^/Dariospizza/" {

    server.document-root  = "/mnt/ISTDataBase/dbbase"

  }

  $HTTP["url"] =~ "^/filesdbbase" {

    fastcgi.server = (

      "/filesdbbase" =>

      (

        ("host" => "192.168.1.100","port" => 9020,"check-local" => "disable","mode" => "responder" ),

        ("host" => "192.168.1.100","port" => 9021,"check-local" => "disable","mode" => "responder" ),

        ("host" => "192.168.1.100","port" => 9022,"check-local" => "disable","mode" => "responder" )

      )

    )

  }

}

 



But sometime the system becomes unstable .
The request for FastCGI 9020 are sent to the Glass environment  that manages 9030 port.

I don't understund because it and what is the original problem.

For solve it i need to stoop and restart the two Glass environment.

Any considerations, idea are welcome.

Thanks,

Dario


Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

James Foster-8
Dario,

I don't know anything about lighttpd configurations, but the one you provided below has both domains going to 902x ports.

James

----- Original Message -----
From: "Dario Trussardi" <[hidden email]>
To: "beta discussion Gemstone Seaside" <[hidden email]>
Sent: Friday, March 30, 2012 7:06:30 AM
Subject: [GS/SS Beta] Web request problem on server with two Glass environment

Ciao,

        i have a server based on Ubuntu and Lighttpd.

        On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.

        One work with :
                Starting WAFastCGIAdaptor gem on port 9030
                Starting WAFastCGIAdaptor gem on port 9031
                Starting WAFastCGIAdaptor gem on port 9032
               
        and other with:

                Starting WAFastCGIAdaptor gem on port 9020
                Starting WAFastCGIAdaptor gem on port 9021
                Starting WAFastCGIAdaptor gem on port 9022
               

        All work fine for more time and the lighttpd redirect the request to the specific environment.
 This is one example of lighttpd configuration:

> $HTTP["host"] =~ "dariospizza.com$" {
>
>   url.redirect = (
>
>     "^/$"  => "http://www.dariospizza.com/dariospizza"
>
>   )
>
>   $HTTP["url"] =~ "^/dariospizza" {
>
>     fastcgi.server = (
>
>       "/dariospizza" =>
>
>       (
>
>         ("host" => "192.168.1.100","port" => 9020,"check-local" => "disable","mode" => "responder" ),
>
>         ("host" => "192.168.1.100","port" => 9021,"check-local" => "disable","mode" => "responder" ),
>
>         ("host" => "192.168.1.100","port" => 9022,"check-local" => "disable","mode" => "responder" )
>
>       )
>
>     )
>
>   }
>
>   $HTTP["url"] =~ "^/Dariospizza/" {
>
>     server.document-root  = "/mnt/ISTDataBase/dbbase"
>
>   }
>
>   $HTTP["url"] =~ "^/filesdbbase" {
>
>     fastcgi.server = (
>
>       "/filesdbbase" =>
>
>       (
>
>         ("host" => "192.168.1.100","port" => 9020,"check-local" => "disable","mode" => "responder" ),
>
>         ("host" => "192.168.1.100","port" => 9021,"check-local" => "disable","mode" => "responder" ),
>
>         ("host" => "192.168.1.100","port" => 9022,"check-local" => "disable","mode" => "responder" )
>
>       )
>
>     )
>
>   }
>
> }
>
 

       


        But sometime the system becomes unstable .
       
        The request for FastCGI 9020 are sent to the Glass environment  that manages 9030 port.

        I don't understund because it and what is the original problem.

        For solve it i need to stoop and restart the two Glass environment.

Any considerations, idea are welcome.

        Thanks,

                Dario


       
Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

dario trussardi

James,

yes, it manage the all request ( WArequest - WAFileHandler - static files reference )  relative to dariospizza site.

This is one lighttpd configuration relative to FastCGI 9030 port in the same lighttpd configuration file.


$HTTP["host"] =~ "clusone.org$" {
  url.redirect = (
        )
  $HTTP["url"] =~ "^/clusone" {
    fastcgi.server = (
      "/clusone" =>
      (
        ("host" => "192.168.1.100","port" => 9030,"check-local" => "disable","mode" => "responder" ),
        ("host" => "192.168.1.100","port" => 9031,"check-local" => "disable","mode" => "responder" ),
        ("host" => "192.168.1.100","port" => 9032,"check-local" => "disable","mode" => "responder" )
      )
    )
  }
  $HTTP["url"] =~ "^/Clusone/" {
    server.document-root  = "/mnt/ISTDataBase/dbgofame"
        }
  $HTTP["url"] =~ "^/filesdbgofame" {
    fastcgi.server = (
      "/filesdbgofame" =>
      (
        ("host" => "192.168.1.100","port" => 9030,"check-local" => "disable","mode" => "responder" ),
        ("host" => "192.168.1.100","port" => 9031,"check-local" => "disable","mode" => "responder" ),
        ("host" => "192.168.1.100","port" => 9032,"check-local" => "disable","mode" => "responder" )
      )
    )
  }
}

All work fine for more time  ( 30 - 60 - 90 days )

But after  ????????      i found the problem.

Thank,
Ciao

Dario



Dario,

I don't know anything about lighttpd configurations, but the one you provided below has both domains going to 902x ports.

James

----- Original Message -----
From: "Dario Trussardi" <[hidden email]>
To: "beta discussion Gemstone Seaside" <[hidden email]>
Sent: Friday, March 30, 2012 7:06:30 AM
Subject: [GS/SS Beta] Web request problem on server with two Glass environment

Ciao,

i have a server based on Ubuntu and Lighttpd.

On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.

One work with :
Starting WAFastCGIAdaptor gem on port 9030
Starting WAFastCGIAdaptor gem on port 9031
Starting WAFastCGIAdaptor gem on port 9032

and other with:

Starting WAFastCGIAdaptor gem on port 9020
Starting WAFastCGIAdaptor gem on port 9021
Starting WAFastCGIAdaptor gem on port 9022


All work fine for more time and the lighttpd redirect the request to the specific environment.
This is one example of lighttpd configuration:

$HTTP["host"] =~ "dariospizza.com$" {

 url.redirect = (

   "^/$"  => "http://www.dariospizza.com/dariospizza"

 )

 $HTTP["url"] =~ "^/dariospizza" {

   fastcgi.server = (

     "/dariospizza" =>

     (

       ("host" => "192.168.1.100","port" => 9020,"check-local" => "disable","mode" => "responder" ),

       ("host" => "192.168.1.100","port" => 9021,"check-local" => "disable","mode" => "responder" ),

       ("host" => "192.168.1.100","port" => 9022,"check-local" => "disable","mode" => "responder" )

     )

   )

 }

 $HTTP["url"] =~ "^/Dariospizza/" {

   server.document-root  = "/mnt/ISTDataBase/dbbase"

 }

 $HTTP["url"] =~ "^/filesdbbase" {

   fastcgi.server = (

     "/filesdbbase" =>

     (

       ("host" => "192.168.1.100","port" => 9020,"check-local" => "disable","mode" => "responder" ),

       ("host" => "192.168.1.100","port" => 9021,"check-local" => "disable","mode" => "responder" ),

       ("host" => "192.168.1.100","port" => 9022,"check-local" => "disable","mode" => "responder" )

     )

   )

 }

}






But sometime the system becomes unstable .

The request for FastCGI 9020 are sent to the Glass environment  that manages 9030 port.

I don't understund because it and what is the original problem.

For solve it i need to stoop and restart the two Glass environment.

Any considerations, idea are welcome.

Thanks,

Dario




Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

Dale Henrichs
In reply to this post by dario trussardi
Dario,

It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.

I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.

If you have a gem in the unresponsive state, we'd like to get a couple of data points:

  kill -USR1 <pid>

to get the smalltalk stack dumped to the gem log and:

  $GEMSTONE/bin/gstack <pid>

to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.

Dale

----- Original Message -----
| From: "Dario Trussardi" <[hidden email]>
| To: "beta discussion Gemstone Seaside" <[hidden email]>
| Sent: Friday, March 30, 2012 7:06:30 AM
| Subject: [GS/SS Beta] Web request problem on server with two Glass environment
|
| Ciao,
|
|
| i have a server based on Ubuntu and Lighttpd.
|
|
| On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
|
|
| One work with :
| Starting WAFastCGIAdaptor gem on port 9030
| Starting WAFastCGIAdaptor gem on port 9031
| Starting WAFastCGIAdaptor gem on port 9032
|
| and other with:
|
|
| Starting WAFastCGIAdaptor gem on port 9020
| Starting WAFastCGIAdaptor gem on port 9021
| Starting WAFastCGIAdaptor gem on port 9022
|
|
|
| All work fine for more time and the lighttpd redirect the request to
| the specific environment.
|
|
| This is one example of lighttpd configuration:
|
|
|
|
|
| $HTTP["host"] =~ "dariospizza.com$" {
|
| url.redirect = (
|
| "^/$" => " http://www.dariospizza.com/dariospizza "
|
| )
|
| $HTTP["url"] =~ "^/dariospizza" {
|
| fastcgi.server = (
|
| "/dariospizza" =>
|
| (
|
| ("host" => "192.168.1.100","port" => 9020,"check-local" =>
| "disable","mode" => "responder" ),
|
| ("host" => "192.168.1.100","port" => 9021,"check-local" =>
| "disable","mode" => "responder" ),
|
| ("host" => "192.168.1.100","port" => 9022,"check-local" =>
| "disable","mode" => "responder" )
|
| )
|
| )
|
| }
|
| $HTTP["url"] =~ "^/Dariospizza/" {
|
| server.document-root = "/mnt/ISTDataBase/dbbase"
|
| }
|
| $HTTP["url"] =~ "^/filesdbbase" {
|
| fastcgi.server = (
|
| "/filesdbbase" =>
|
| (
|
| ("host" => "192.168.1.100","port" => 9020,"check-local" =>
| "disable","mode" => "responder" ),
|
| ("host" => "192.168.1.100","port" => 9021,"check-local" =>
| "disable","mode" => "responder" ),
|
| ("host" => "192.168.1.100","port" => 9022,"check-local" =>
| "disable","mode" => "responder" )
|
| )
|
| )
|
| }
|
| }
|
|
|
|
|
|
|
| But sometime the system becomes unstable .
|
| The request for FastCGI 9020 are sent to the Glass environment that
| manages 9030 port.
|
|
| I don't understund because it and what is the original problem.
|
|
| For solve it i need to stoop and restart the two Glass environment.
|
|
| Any considerations, idea are welcome.
|
|
| Thanks,
|
|
| Dario
|
|
|
|
|
Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

dario trussardi
Ciao folks,

the next time when i found   the problem, i will make attention to document the error with more detail.

For now thanks,

Dario

Dario,

It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.

I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.

If you have a gem in the unresponsive state, we'd like to get a couple of data points:

 kill -USR1 <pid>

to get the smalltalk stack dumped to the gem log and:

 $GEMSTONE/bin/gstack <pid>

to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.

Dale

----- Original Message -----
| From: "Dario Trussardi" <[hidden email]>
| To: "beta discussion Gemstone Seaside" <[hidden email]>
| Sent: Friday, March 30, 2012 7:06:30 AM
| Subject: [GS/SS Beta] Web request problem on server with two Glass environment
|
| Ciao,
|
|
| i have a server based on Ubuntu and Lighttpd.
|
|
| On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
|
|
| One work with :
| Starting WAFastCGIAdaptor gem on port 9030
| Starting WAFastCGIAdaptor gem on port 9031
| Starting WAFastCGIAdaptor gem on port 9032
|
| and other with:
|
|
| Starting WAFastCGIAdaptor gem on port 9020
| Starting WAFastCGIAdaptor gem on port 9021
| Starting WAFastCGIAdaptor gem on port 9022
|
|
|
| All work fine for more time and the lighttpd redirect the request to
| the specific environment.
|
|
| This is one example of lighttpd configuration:
|
|
|
|
|
| $HTTP["host"] =~ "dariospizza.com$" {
|
| url.redirect = (
|
| "^/$" => " http://www.dariospizza.com/dariospizza "
|
| )
|
| $HTTP["url"] =~ "^/dariospizza" {
|
| fastcgi.server = (
|
| "/dariospizza" =>
|
| (
|
| ("host" => "192.168.1.100","port" => 9020,"check-local" =>
| "disable","mode" => "responder" ),
|
| ("host" => "192.168.1.100","port" => 9021,"check-local" =>
| "disable","mode" => "responder" ),
|
| ("host" => "192.168.1.100","port" => 9022,"check-local" =>
| "disable","mode" => "responder" )
|
| )
|
| )
|
| }
|
| $HTTP["url"] =~ "^/Dariospizza/" {
|
| server.document-root = "/mnt/ISTDataBase/dbbase"
|
| }
|
| $HTTP["url"] =~ "^/filesdbbase" {
|
| fastcgi.server = (
|
| "/filesdbbase" =>
|
| (
|
| ("host" => "192.168.1.100","port" => 9020,"check-local" =>
| "disable","mode" => "responder" ),
|
| ("host" => "192.168.1.100","port" => 9021,"check-local" =>
| "disable","mode" => "responder" ),
|
| ("host" => "192.168.1.100","port" => 9022,"check-local" =>
| "disable","mode" => "responder" )
|
| )
|
| )
|
| }
|
| }
|
|
|
|
|
|
|
| But sometime the system becomes unstable .
|
| The request for FastCGI 9020 are sent to the Glass environment that
| manages 9030 port.
|
|
| I don't understund because it and what is the original problem.
|
|
| For solve it i need to stoop and restart the two Glass environment.
|
|
| Any considerations, idea are welcome.
|
|
| Thanks,
|
|
| Dario
|
|
|
|
|

Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

Johan Brichau-2
In reply to this post by Dale Henrichs
Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request.
Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone.

I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0.

I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this.

that's all I know so far...

Johan

On 30 Mar 2012, at 18:14, Dale Henrichs wrote:

> Dario,
>
> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.
>
> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.
>
> If you have a gem in the unresponsive state, we'd like to get a couple of data points:
>
>  kill -USR1 <pid>
>
> to get the smalltalk stack dumped to the gem log and:
>
>  $GEMSTONE/bin/gstack <pid>
>
> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.
>
> Dale
>
> ----- Original Message -----
> | From: "Dario Trussardi" <[hidden email]>
> | To: "beta discussion Gemstone Seaside" <[hidden email]>
> | Sent: Friday, March 30, 2012 7:06:30 AM
> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment
> |
> | Ciao,
> |
> |
> | i have a server based on Ubuntu and Lighttpd.
> |
> |
> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
> |
> |
> | One work with :
> | Starting WAFastCGIAdaptor gem on port 9030
> | Starting WAFastCGIAdaptor gem on port 9031
> | Starting WAFastCGIAdaptor gem on port 9032
> |
> | and other with:
> |
> |
> | Starting WAFastCGIAdaptor gem on port 9020
> | Starting WAFastCGIAdaptor gem on port 9021
> | Starting WAFastCGIAdaptor gem on port 9022
> |
> |
> |
> | All work fine for more time and the lighttpd redirect the request to
> | the specific environment.
> |
> |
> | This is one example of lighttpd configuration:
> |
> |
> |
> |
> |
> | $HTTP["host"] =~ "dariospizza.com$" {
> |
> | url.redirect = (
> |
> | "^/$" => " http://www.dariospizza.com/dariospizza "
> |
> | )
> |
> | $HTTP["url"] =~ "^/dariospizza" {
> |
> | fastcgi.server = (
> |
> | "/dariospizza" =>
> |
> | (
> |
> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
> | "disable","mode" => "responder" ),
> |
> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
> | "disable","mode" => "responder" ),
> |
> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
> | "disable","mode" => "responder" )
> |
> | )
> |
> | )
> |
> | }
> |
> | $HTTP["url"] =~ "^/Dariospizza/" {
> |
> | server.document-root = "/mnt/ISTDataBase/dbbase"
> |
> | }
> |
> | $HTTP["url"] =~ "^/filesdbbase" {
> |
> | fastcgi.server = (
> |
> | "/filesdbbase" =>
> |
> | (
> |
> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
> | "disable","mode" => "responder" ),
> |
> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
> | "disable","mode" => "responder" ),
> |
> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
> | "disable","mode" => "responder" )
> |
> | )
> |
> | )
> |
> | }
> |
> | }
> |
> |
> |
> |
> |
> |
> |
> | But sometime the system becomes unstable .
> |
> | The request for FastCGI 9020 are sent to the Glass environment that
> | manages 9030 port.
> |
> |
> | I don't understund because it and what is the original problem.
> |
> |
> | For solve it i need to stoop and restart the two Glass environment.
> |
> |
> | Any considerations, idea are welcome.
> |
> |
> | Thanks,
> |
> |
> | Dario
> |
> |
> |
> |
> |

Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

Larry Kellogg

On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote:

> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request.
> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone.


Hello Johan,
I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem  if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible?

Regards,

Larry


>
> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0.
>
> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this.
>
> that's all I know so far...
>
> Johan
>
> On 30 Mar 2012, at 18:14, Dale Henrichs wrote:
>
>> Dario,
>>
>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.
>>
>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.
>>
>> If you have a gem in the unresponsive state, we'd like to get a couple of data points:
>>
>> kill -USR1 <pid>
>>
>> to get the smalltalk stack dumped to the gem log and:
>>
>> $GEMSTONE/bin/gstack <pid>
>>
>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.
>>
>> Dale
>>
>> ----- Original Message -----
>> | From: "Dario Trussardi" <[hidden email]>
>> | To: "beta discussion Gemstone Seaside" <[hidden email]>
>> | Sent: Friday, March 30, 2012 7:06:30 AM
>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment
>> |
>> | Ciao,
>> |
>> |
>> | i have a server based on Ubuntu and Lighttpd.
>> |
>> |
>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
>> |
>> |
>> | One work with :
>> | Starting WAFastCGIAdaptor gem on port 9030
>> | Starting WAFastCGIAdaptor gem on port 9031
>> | Starting WAFastCGIAdaptor gem on port 9032
>> |
>> | and other with:
>> |
>> |
>> | Starting WAFastCGIAdaptor gem on port 9020
>> | Starting WAFastCGIAdaptor gem on port 9021
>> | Starting WAFastCGIAdaptor gem on port 9022
>> |
>> |
>> |
>> | All work fine for more time and the lighttpd redirect the request to
>> | the specific environment.
>> |
>> |
>> | This is one example of lighttpd configuration:
>> |
>> |
>> |
>> |
>> |
>> | $HTTP["host"] =~ "dariospizza.com$" {
>> |
>> | url.redirect = (
>> |
>> | "^/$" => " http://www.dariospizza.com/dariospizza "
>> |
>> | )
>> |
>> | $HTTP["url"] =~ "^/dariospizza" {
>> |
>> | fastcgi.server = (
>> |
>> | "/dariospizza" =>
>> |
>> | (
>> |
>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>> | "disable","mode" => "responder" ),
>> |
>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>> | "disable","mode" => "responder" ),
>> |
>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>> | "disable","mode" => "responder" )
>> |
>> | )
>> |
>> | )
>> |
>> | }
>> |
>> | $HTTP["url"] =~ "^/Dariospizza/" {
>> |
>> | server.document-root = "/mnt/ISTDataBase/dbbase"
>> |
>> | }
>> |
>> | $HTTP["url"] =~ "^/filesdbbase" {
>> |
>> | fastcgi.server = (
>> |
>> | "/filesdbbase" =>
>> |
>> | (
>> |
>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>> | "disable","mode" => "responder" ),
>> |
>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>> | "disable","mode" => "responder" ),
>> |
>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>> | "disable","mode" => "responder" )
>> |
>> | )
>> |
>> | )
>> |
>> | }
>> |
>> | }
>> |
>> |
>> |
>> |
>> |
>> |
>> |
>> | But sometime the system becomes unstable .
>> |
>> | The request for FastCGI 9020 are sent to the Glass environment that
>> | manages 9030 port.
>> |
>> |
>> | I don't understund because it and what is the original problem.
>> |
>> |
>> | For solve it i need to stoop and restart the two Glass environment.
>> |
>> |
>> | Any considerations, idea are welcome.
>> |
>> |
>> | Thanks,
>> |
>> |
>> | Dario
>> |
>> |
>> |
>> |
>> |
>

Reply | Threaded
Open this post in threaded view
|

Sending Email from Seaside/GLASS

Larry Kellogg
Hello,
  I'm finally getting around to trying to send email from Seaside under GLASS. I have the following method.

sendEmailVerificationMessage
 | msg |
        SendMail mailhost: 'smtp.gmail.com'.
        msg := WAEmailMessage
          from: (WAEmailAddress address: '[hidden email]' username: 'PracticeMusic Help')
           to: (WAEmailAddress address: self user emailAddress username: self user screenName)
            subject: 'Email Address Verification'.
   msg body: 'Test email message'.
        GRGemStonePlatform current seasideDeliverEmailMessage: msg
       

I went through and defined all of the smptp parameters for GMail based on this page: http://email.about.com/od/accessinggmail/f/Gmail_SMTP_Settings.htm

When I try to call my method, the image just hangs, and this is written to the terminal window, over and over:

Waiting for server to write...
Waiting for server to write...
Waiting for server to write...
Waiting for server to write...
Waiting for server to write...
Waiting for server to writeā€¦


Any idea what I'm doing wrong?

Regards,

Larry Kellogg
www.practicemusic.com
Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

Johan Brichau-2
In reply to this post by Larry Kellogg
Larry,

Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/
If you have any questions, I'm happy to answer them.

Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem.

check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid
        start program = "<start gem 9001>" as uid sites and gid sites
        stop program = "<stop gem 9001>" as uid sites and gid sites
                # Empty FastCGI request
                if failed port 9001
                  # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09)
                  # padding 8 bytes (0x08), followed by 8xNULLs padding
                  send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
                  # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A)
                  expect "\0x01\0x0A"
                  timeout 20 seconds
                then restart

On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote:

>
> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote:
>
>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request.
>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone.
>
>
> Hello Johan,
> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem  if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible?
>
> Regards,
>
> Larry
>
>
>>
>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0.
>>
>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this.
>>
>> that's all I know so far...
>>
>> Johan
>>
>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote:
>>
>>> Dario,
>>>
>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.
>>>
>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.
>>>
>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points:
>>>
>>> kill -USR1 <pid>
>>>
>>> to get the smalltalk stack dumped to the gem log and:
>>>
>>> $GEMSTONE/bin/gstack <pid>
>>>
>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.
>>>
>>> Dale
>>>
>>> ----- Original Message -----
>>> | From: "Dario Trussardi" <[hidden email]>
>>> | To: "beta discussion Gemstone Seaside" <[hidden email]>
>>> | Sent: Friday, March 30, 2012 7:06:30 AM
>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment
>>> |
>>> | Ciao,
>>> |
>>> |
>>> | i have a server based on Ubuntu and Lighttpd.
>>> |
>>> |
>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
>>> |
>>> |
>>> | One work with :
>>> | Starting WAFastCGIAdaptor gem on port 9030
>>> | Starting WAFastCGIAdaptor gem on port 9031
>>> | Starting WAFastCGIAdaptor gem on port 9032
>>> |
>>> | and other with:
>>> |
>>> |
>>> | Starting WAFastCGIAdaptor gem on port 9020
>>> | Starting WAFastCGIAdaptor gem on port 9021
>>> | Starting WAFastCGIAdaptor gem on port 9022
>>> |
>>> |
>>> |
>>> | All work fine for more time and the lighttpd redirect the request to
>>> | the specific environment.
>>> |
>>> |
>>> | This is one example of lighttpd configuration:
>>> |
>>> |
>>> |
>>> |
>>> |
>>> | $HTTP["host"] =~ "dariospizza.com$" {
>>> |
>>> | url.redirect = (
>>> |
>>> | "^/$" => " http://www.dariospizza.com/dariospizza "
>>> |
>>> | )
>>> |
>>> | $HTTP["url"] =~ "^/dariospizza" {
>>> |
>>> | fastcgi.server = (
>>> |
>>> | "/dariospizza" =>
>>> |
>>> | (
>>> |
>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>> | "disable","mode" => "responder" ),
>>> |
>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>> | "disable","mode" => "responder" ),
>>> |
>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>> | "disable","mode" => "responder" )
>>> |
>>> | )
>>> |
>>> | )
>>> |
>>> | }
>>> |
>>> | $HTTP["url"] =~ "^/Dariospizza/" {
>>> |
>>> | server.document-root = "/mnt/ISTDataBase/dbbase"
>>> |
>>> | }
>>> |
>>> | $HTTP["url"] =~ "^/filesdbbase" {
>>> |
>>> | fastcgi.server = (
>>> |
>>> | "/filesdbbase" =>
>>> |
>>> | (
>>> |
>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>> | "disable","mode" => "responder" ),
>>> |
>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>> | "disable","mode" => "responder" ),
>>> |
>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>> | "disable","mode" => "responder" )
>>> |
>>> | )
>>> |
>>> | )
>>> |
>>> | }
>>> |
>>> | }
>>> |
>>> |
>>> |
>>> |
>>> |
>>> |
>>> |
>>> | But sometime the system becomes unstable .
>>> |
>>> | The request for FastCGI 9020 are sent to the Glass environment that
>>> | manages 9030 port.
>>> |
>>> |
>>> | I don't understund because it and what is the original problem.
>>> |
>>> |
>>> | For solve it i need to stoop and restart the two Glass environment.
>>> |
>>> |
>>> | Any considerations, idea are welcome.
>>> |
>>> |
>>> | Thanks,
>>> |
>>> |
>>> | Dario
>>> |
>>> |
>>> |
>>> |
>>> |
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

Larry Kellogg

On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote:

> Larry,
>
> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/
> If you have any questions, I'm happy to answer them.
>
> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem.
>
> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid
>        start program = "<start gem 9001>" as uid sites and gid sites
>        stop program = "<stop gem 9001>" as uid sites and gid sites
> # Empty FastCGI request
> if failed port 9001
>  # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09)
>  # padding 8 bytes (0x08), followed by 8xNULLs padding
>  send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
>  # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A)
>  expect "\0x01\0x0A"
>  timeout 20 seconds
> then restart
>

  Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests.

  Regards,

  Larry

> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote:
>
>>
>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote:
>>
>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request.
>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone.
>>
>>
>> Hello Johan,
>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem  if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible?
>>
>> Regards,
>>
>> Larry
>>
>>
>>>
>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0.
>>>
>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this.
>>>
>>> that's all I know so far...
>>>
>>> Johan
>>>
>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote:
>>>
>>>> Dario,
>>>>
>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.
>>>>
>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.
>>>>
>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points:
>>>>
>>>> kill -USR1 <pid>
>>>>
>>>> to get the smalltalk stack dumped to the gem log and:
>>>>
>>>> $GEMSTONE/bin/gstack <pid>
>>>>
>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.
>>>>
>>>> Dale
>>>>
>>>> ----- Original Message -----
>>>> | From: "Dario Trussardi" <[hidden email]>
>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]>
>>>> | Sent: Friday, March 30, 2012 7:06:30 AM
>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment
>>>> |
>>>> | Ciao,
>>>> |
>>>> |
>>>> | i have a server based on Ubuntu and Lighttpd.
>>>> |
>>>> |
>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
>>>> |
>>>> |
>>>> | One work with :
>>>> | Starting WAFastCGIAdaptor gem on port 9030
>>>> | Starting WAFastCGIAdaptor gem on port 9031
>>>> | Starting WAFastCGIAdaptor gem on port 9032
>>>> |
>>>> | and other with:
>>>> |
>>>> |
>>>> | Starting WAFastCGIAdaptor gem on port 9020
>>>> | Starting WAFastCGIAdaptor gem on port 9021
>>>> | Starting WAFastCGIAdaptor gem on port 9022
>>>> |
>>>> |
>>>> |
>>>> | All work fine for more time and the lighttpd redirect the request to
>>>> | the specific environment.
>>>> |
>>>> |
>>>> | This is one example of lighttpd configuration:
>>>> |
>>>> |
>>>> |
>>>> |
>>>> |
>>>> | $HTTP["host"] =~ "dariospizza.com$" {
>>>> |
>>>> | url.redirect = (
>>>> |
>>>> | "^/$" => " http://www.dariospizza.com/dariospizza "
>>>> |
>>>> | )
>>>> |
>>>> | $HTTP["url"] =~ "^/dariospizza" {
>>>> |
>>>> | fastcgi.server = (
>>>> |
>>>> | "/dariospizza" =>
>>>> |
>>>> | (
>>>> |
>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>> | "disable","mode" => "responder" ),
>>>> |
>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>> | "disable","mode" => "responder" ),
>>>> |
>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>> | "disable","mode" => "responder" )
>>>> |
>>>> | )
>>>> |
>>>> | )
>>>> |
>>>> | }
>>>> |
>>>> | $HTTP["url"] =~ "^/Dariospizza/" {
>>>> |
>>>> | server.document-root = "/mnt/ISTDataBase/dbbase"
>>>> |
>>>> | }
>>>> |
>>>> | $HTTP["url"] =~ "^/filesdbbase" {
>>>> |
>>>> | fastcgi.server = (
>>>> |
>>>> | "/filesdbbase" =>
>>>> |
>>>> | (
>>>> |
>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>> | "disable","mode" => "responder" ),
>>>> |
>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>> | "disable","mode" => "responder" ),
>>>> |
>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>> | "disable","mode" => "responder" )
>>>> |
>>>> | )
>>>> |
>>>> | )
>>>> |
>>>> | }
>>>> |
>>>> | }
>>>> |
>>>> |
>>>> |
>>>> |
>>>> |
>>>> |
>>>> |
>>>> | But sometime the system becomes unstable .
>>>> |
>>>> | The request for FastCGI 9020 are sent to the Glass environment that
>>>> | manages 9030 port.
>>>> |
>>>> |
>>>> | I don't understund because it and what is the original problem.
>>>> |
>>>> |
>>>> | For solve it i need to stoop and restart the two Glass environment.
>>>> |
>>>> |
>>>> | Any considerations, idea are welcome.
>>>> |
>>>> |
>>>> | Thanks,
>>>> |
>>>> |
>>>> | Dario
>>>> |
>>>> |
>>>> |
>>>> |
>>>> |
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Sending Email from Seaside/GLASS

NorbertHartl
In reply to this post by Larry Kellogg

Am 08.04.2012 um 21:31 schrieb Lawrence Kellogg:

> Hello,
>  I'm finally getting around to trying to send email from Seaside under GLASS. I have the following method.
>
> sendEmailVerificationMessage
> | msg |
> SendMail mailhost: 'smtp.gmail.com'.
> msg := WAEmailMessage
>          from: (WAEmailAddress address: '[hidden email]' username: 'PracticeMusic Help')
>           to: (WAEmailAddress address: self user emailAddress username: self user screenName)
>            subject: 'Email Address Verification'.
>   msg body: 'Test email message'.
> GRGemStonePlatform current seasideDeliverEmailMessage: msg
>
>
> I went through and defined all of the smptp parameters for GMail based on this page: http://email.about.com/od/accessinggmail/f/Gmail_SMTP_Settings.htm
>
> When I try to call my method, the image just hangs, and this is written to the terminal window, over and over:
>
> Waiting for server to write...
> Waiting for server to write...
> Waiting for server to write...
> Waiting for server to write...
> Waiting for server to write...
> Waiting for server to writeā€¦
>
>
> Any idea what I'm doing wrong?
>
You need TLS/SSL to send mails over gmail. It is on the mentioned web page. You cannot do this in gemstone at the moment.

Norbert

Reply | Threaded
Open this post in threaded view
|

Re: Sending Email from Seaside/GLASS

Larry Kellogg

On Apr 9, 2012, at 5:11 AM, Norbert Hartl wrote:

>
> Am 08.04.2012 um 21:31 schrieb Lawrence Kellogg:
>
>> Hello,
>> I'm finally getting around to trying to send email from Seaside under GLASS. I have the following method.
>>
>> sendEmailVerificationMessage
>> | msg |
>> SendMail mailhost: 'smtp.gmail.com'.
>> msg := WAEmailMessage
>>         from: (WAEmailAddress address: '[hidden email]' username: 'PracticeMusic Help')
>>          to: (WAEmailAddress address: self user emailAddress username: self user screenName)
>>           subject: 'Email Address Verification'.
>>  msg body: 'Test email message'.
>> GRGemStonePlatform current seasideDeliverEmailMessage: msg
>>
>>
>> I went through and defined all of the smptp parameters for GMail based on this page: http://email.about.com/od/accessinggmail/f/Gmail_SMTP_Settings.htm
>>
>> When I try to call my method, the image just hangs, and this is written to the terminal window, over and over:
>>
>> Waiting for server to write...
>> Waiting for server to write...
>> Waiting for server to write...
>> Waiting for server to write...
>> Waiting for server to write...
>> Waiting for server to writeā€¦
>>
>>
>> Any idea what I'm doing wrong?
>>
> You need TLS/SSL to send mails over gmail. It is on the mentioned web page. You cannot do this in gemstone at the moment.
>

Hello Norbert,
  Yes, I noticed the requirement for TLS/SSL, but I was hoping it was covered, magically. ;-) Thanks for letting me know.
Does this mean there s no way to send email from Gemstone, at the moment?

  I need a feature where users can request their own passwords, or a link to a page to reset their passwords.

  Regards,

  Larry
  www.practicemusic.com
 



> Norbert
>

Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

Larry Kellogg
In reply to this post by Larry Kellogg

On Apr 8, 2012, at 7:39 PM, Lawrence Kellogg wrote:

>
> On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote:
>
>> Larry,
>>
>> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/
>> If you have any questions, I'm happy to answer them.
>>
>> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem.
>>
>> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid
>>       start program = "<start gem 9001>" as uid sites and gid sites
>>       stop program = "<stop gem 9001>" as uid sites and gid sites
>> # Empty FastCGI request
>> if failed port 9001
>>  # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09)
>>  # padding 8 bytes (0x08), followed by 8xNULLs padding
>>  send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
>>  # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A)
>>  expect "\0x01\0x0A"
>>  timeout 20 seconds
>> then restart
>>

Johan,
  I used the configuration you specified above and just replaced these two lines:

       start program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001"
       stop program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 stop"
               
but it does not seem to work. I can see that monit is running. If I kill -9 my FastCGI Gem,  the system does not stop and restart all of the Gems. Also, monit does not seem to start everything up initially. What am I missing?

  Regards,

  Larry


>
>  Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests.
>
>  Regards,
>
>  Larry
>
>> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote:
>>
>>>
>>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote:
>>>
>>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request.
>>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone.
>>>
>>>
>>> Hello Johan,
>>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem  if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible?
>>>
>>> Regards,
>>>
>>> Larry
>>>
>>>
>>>>
>>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0.
>>>>
>>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this.
>>>>
>>>> that's all I know so far...
>>>>
>>>> Johan
>>>>
>>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote:
>>>>
>>>>> Dario,
>>>>>
>>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.
>>>>>
>>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.
>>>>>
>>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points:
>>>>>
>>>>> kill -USR1 <pid>
>>>>>
>>>>> to get the smalltalk stack dumped to the gem log and:
>>>>>
>>>>> $GEMSTONE/bin/gstack <pid>
>>>>>
>>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.
>>>>>
>>>>> Dale
>>>>>
>>>>> ----- Original Message -----
>>>>> | From: "Dario Trussardi" <[hidden email]>
>>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]>
>>>>> | Sent: Friday, March 30, 2012 7:06:30 AM
>>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment
>>>>> |
>>>>> | Ciao,
>>>>> |
>>>>> |
>>>>> | i have a server based on Ubuntu and Lighttpd.
>>>>> |
>>>>> |
>>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
>>>>> |
>>>>> |
>>>>> | One work with :
>>>>> | Starting WAFastCGIAdaptor gem on port 9030
>>>>> | Starting WAFastCGIAdaptor gem on port 9031
>>>>> | Starting WAFastCGIAdaptor gem on port 9032
>>>>> |
>>>>> | and other with:
>>>>> |
>>>>> |
>>>>> | Starting WAFastCGIAdaptor gem on port 9020
>>>>> | Starting WAFastCGIAdaptor gem on port 9021
>>>>> | Starting WAFastCGIAdaptor gem on port 9022
>>>>> |
>>>>> |
>>>>> |
>>>>> | All work fine for more time and the lighttpd redirect the request to
>>>>> | the specific environment.
>>>>> |
>>>>> |
>>>>> | This is one example of lighttpd configuration:
>>>>> |
>>>>> |
>>>>> |
>>>>> |
>>>>> |
>>>>> | $HTTP["host"] =~ "dariospizza.com$" {
>>>>> |
>>>>> | url.redirect = (
>>>>> |
>>>>> | "^/$" => " http://www.dariospizza.com/dariospizza "
>>>>> |
>>>>> | )
>>>>> |
>>>>> | $HTTP["url"] =~ "^/dariospizza" {
>>>>> |
>>>>> | fastcgi.server = (
>>>>> |
>>>>> | "/dariospizza" =>
>>>>> |
>>>>> | (
>>>>> |
>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>> | "disable","mode" => "responder" ),
>>>>> |
>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>> | "disable","mode" => "responder" ),
>>>>> |
>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>> | "disable","mode" => "responder" )
>>>>> |
>>>>> | )
>>>>> |
>>>>> | )
>>>>> |
>>>>> | }
>>>>> |
>>>>> | $HTTP["url"] =~ "^/Dariospizza/" {
>>>>> |
>>>>> | server.document-root = "/mnt/ISTDataBase/dbbase"
>>>>> |
>>>>> | }
>>>>> |
>>>>> | $HTTP["url"] =~ "^/filesdbbase" {
>>>>> |
>>>>> | fastcgi.server = (
>>>>> |
>>>>> | "/filesdbbase" =>
>>>>> |
>>>>> | (
>>>>> |
>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>> | "disable","mode" => "responder" ),
>>>>> |
>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>> | "disable","mode" => "responder" ),
>>>>> |
>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>> | "disable","mode" => "responder" )
>>>>> |
>>>>> | )
>>>>> |
>>>>> | )
>>>>> |
>>>>> | }
>>>>> |
>>>>> | }
>>>>> |
>>>>> |
>>>>> |
>>>>> |
>>>>> |
>>>>> |
>>>>> |
>>>>> | But sometime the system becomes unstable .
>>>>> |
>>>>> | The request for FastCGI 9020 are sent to the Glass environment that
>>>>> | manages 9030 port.
>>>>> |
>>>>> |
>>>>> | I don't understund because it and what is the original problem.
>>>>> |
>>>>> |
>>>>> | For solve it i need to stoop and restart the two Glass environment.
>>>>> |
>>>>> |
>>>>> | Any considerations, idea are welcome.
>>>>> |
>>>>> |
>>>>> | Thanks,
>>>>> |
>>>>> |
>>>>> | Dario
>>>>> |
>>>>> |
>>>>> |
>>>>> |
>>>>> |
>>>>
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Sending Email from Seaside/GLASS

NorbertHartl
In reply to this post by Larry Kellogg

Am 09.04.2012 um 11:30 schrieb Lawrence Kellogg:

>
> On Apr 9, 2012, at 5:11 AM, Norbert Hartl wrote:
>
>>
>> Am 08.04.2012 um 21:31 schrieb Lawrence Kellogg:
>>
>>> Hello,
>>> I'm finally getting around to trying to send email from Seaside under GLASS. I have the following method.
>>>
>>> sendEmailVerificationMessage
>>> | msg |
>>> SendMail mailhost: 'smtp.gmail.com'.
>>> msg := WAEmailMessage
>>>        from: (WAEmailAddress address: '[hidden email]' username: 'PracticeMusic Help')
>>>         to: (WAEmailAddress address: self user emailAddress username: self user screenName)
>>>          subject: 'Email Address Verification'.
>>> msg body: 'Test email message'.
>>> GRGemStonePlatform current seasideDeliverEmailMessage: msg
>>>
>>>
>>> I went through and defined all of the smptp parameters for GMail based on this page: http://email.about.com/od/accessinggmail/f/Gmail_SMTP_Settings.htm
>>>
>>> When I try to call my method, the image just hangs, and this is written to the terminal window, over and over:
>>>
>>> Waiting for server to write...
>>> Waiting for server to write...
>>> Waiting for server to write...
>>> Waiting for server to write...
>>> Waiting for server to write...
>>> Waiting for server to writeā€¦
>>>
>>>
>>> Any idea what I'm doing wrong?
>>>
>> You need TLS/SSL to send mails over gmail. It is on the mentioned web page. You cannot do this in gemstone at the moment.
>>
>
> Hello Norbert,
>  Yes, I noticed the requirement for TLS/SSL, but I was hoping it was covered, magically. ;-) Thanks for letting me know.
> Does this mean there s no way to send email from Gemstone, at the moment?
>
>  I need a feature where users can request their own passwords, or a link to a page to reset their passwords.
>
The directy usage of the gmail mail server is not possible at the moment. For this someone would need to add SSL support to gemstone. Gemstone 3.0 has an foreign function interface that makes it possible but nobody did it so far.
I think you have two options. Either use a static SSL connector [1] that wraps any connection into SSL. Or the most complicated and most reliable thing would be to have a local mail server configured that acts as a relay between your machine and google.

[1] http://www.stunnel.org/

Norbert

Reply | Threaded
Open this post in threaded view
|

Re: Sending Email from Seaside/GLASS

Tobias Pape

Am 2012-04-09 um 12:58 schrieb Norbert Hartl:

>
> Am 09.04.2012 um 11:30 schrieb Lawrence Kellogg:
>
>>
>> On Apr 9, 2012, at 5:11 AM, Norbert Hartl wrote:
>>
>>>
>>> Am 08.04.2012 um 21:31 schrieb Lawrence Kellogg:
>>>
>>>> Hello,
>>>> I'm finally getting around to trying to send email from Seaside under GLASS. I have the following method.
>>>>
>>>> sendEmailVerificationMessage
>>>> | msg |
>>>> SendMail mailhost: 'smtp.gmail.com'.
[ā€¦]

>>
> The directy usage of the gmail mail server is not possible at the moment. For this someone would need to add SSL support to gemstone. Gemstone 3.0 has an foreign function interface that makes it possible but nobody did it so far.
> I think you have two options. Either use a static SSL connector [1] that wraps any connection into SSL. Or the most complicated and most reliable thing would be to have a local mail server configured that acts as a relay between your machine and google.
>
> [1] http://www.stunnel.org/

Additionally, make sure that you put your mailhost into the
respective Seaside Configuration, so you can stay portable ;)

Best
        -Tobias
Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

Johan Brichau-2
In reply to this post by Larry Kellogg
Larry,

What is the output of the following command on your system?

sudo monit summary


On 09 Apr 2012, at 11:52, Lawrence Kellogg wrote:

>
> On Apr 8, 2012, at 7:39 PM, Lawrence Kellogg wrote:
>
>>
>> On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote:
>>
>>> Larry,
>>>
>>> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/
>>> If you have any questions, I'm happy to answer them.
>>>
>>> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem.
>>>
>>> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid
>>>      start program = "<start gem 9001>" as uid sites and gid sites
>>>      stop program = "<stop gem 9001>" as uid sites and gid sites
>>> # Empty FastCGI request
>>> if failed port 9001
>>>  # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09)
>>>  # padding 8 bytes (0x08), followed by 8xNULLs padding
>>>  send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
>>>  # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A)
>>>  expect "\0x01\0x0A"
>>>  timeout 20 seconds
>>> then restart
>>>
>
> Johan,
>  I used the configuration you specified above and just replaced these two lines:
>
>       start program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001"
>       stop program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 stop"
>
> but it does not seem to work. I can see that monit is running. If I kill -9 my FastCGI Gem,  the system does not stop and restart all of the Gems. Also, monit does not seem to start everything up initially. What am I missing?
>
>  Regards,
>
>  Larry
>
>
>>
>> Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests.
>>
>> Regards,
>>
>> Larry
>>
>>> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote:
>>>
>>>>
>>>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote:
>>>>
>>>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request.
>>>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone.
>>>>
>>>>
>>>> Hello Johan,
>>>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem  if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible?
>>>>
>>>> Regards,
>>>>
>>>> Larry
>>>>
>>>>
>>>>>
>>>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0.
>>>>>
>>>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this.
>>>>>
>>>>> that's all I know so far...
>>>>>
>>>>> Johan
>>>>>
>>>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote:
>>>>>
>>>>>> Dario,
>>>>>>
>>>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.
>>>>>>
>>>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.
>>>>>>
>>>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points:
>>>>>>
>>>>>> kill -USR1 <pid>
>>>>>>
>>>>>> to get the smalltalk stack dumped to the gem log and:
>>>>>>
>>>>>> $GEMSTONE/bin/gstack <pid>
>>>>>>
>>>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.
>>>>>>
>>>>>> Dale
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> | From: "Dario Trussardi" <[hidden email]>
>>>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]>
>>>>>> | Sent: Friday, March 30, 2012 7:06:30 AM
>>>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment
>>>>>> |
>>>>>> | Ciao,
>>>>>> |
>>>>>> |
>>>>>> | i have a server based on Ubuntu and Lighttpd.
>>>>>> |
>>>>>> |
>>>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
>>>>>> |
>>>>>> |
>>>>>> | One work with :
>>>>>> | Starting WAFastCGIAdaptor gem on port 9030
>>>>>> | Starting WAFastCGIAdaptor gem on port 9031
>>>>>> | Starting WAFastCGIAdaptor gem on port 9032
>>>>>> |
>>>>>> | and other with:
>>>>>> |
>>>>>> |
>>>>>> | Starting WAFastCGIAdaptor gem on port 9020
>>>>>> | Starting WAFastCGIAdaptor gem on port 9021
>>>>>> | Starting WAFastCGIAdaptor gem on port 9022
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> | All work fine for more time and the lighttpd redirect the request to
>>>>>> | the specific environment.
>>>>>> |
>>>>>> |
>>>>>> | This is one example of lighttpd configuration:
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> | $HTTP["host"] =~ "dariospizza.com$" {
>>>>>> |
>>>>>> | url.redirect = (
>>>>>> |
>>>>>> | "^/$" => " http://www.dariospizza.com/dariospizza "
>>>>>> |
>>>>>> | )
>>>>>> |
>>>>>> | $HTTP["url"] =~ "^/dariospizza" {
>>>>>> |
>>>>>> | fastcgi.server = (
>>>>>> |
>>>>>> | "/dariospizza" =>
>>>>>> |
>>>>>> | (
>>>>>> |
>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>>> | "disable","mode" => "responder" ),
>>>>>> |
>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>>> | "disable","mode" => "responder" ),
>>>>>> |
>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>>> | "disable","mode" => "responder" )
>>>>>> |
>>>>>> | )
>>>>>> |
>>>>>> | )
>>>>>> |
>>>>>> | }
>>>>>> |
>>>>>> | $HTTP["url"] =~ "^/Dariospizza/" {
>>>>>> |
>>>>>> | server.document-root = "/mnt/ISTDataBase/dbbase"
>>>>>> |
>>>>>> | }
>>>>>> |
>>>>>> | $HTTP["url"] =~ "^/filesdbbase" {
>>>>>> |
>>>>>> | fastcgi.server = (
>>>>>> |
>>>>>> | "/filesdbbase" =>
>>>>>> |
>>>>>> | (
>>>>>> |
>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>>> | "disable","mode" => "responder" ),
>>>>>> |
>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>>> | "disable","mode" => "responder" ),
>>>>>> |
>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>>> | "disable","mode" => "responder" )
>>>>>> |
>>>>>> | )
>>>>>> |
>>>>>> | )
>>>>>> |
>>>>>> | }
>>>>>> |
>>>>>> | }
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> | But sometime the system becomes unstable .
>>>>>> |
>>>>>> | The request for FastCGI 9020 are sent to the Glass environment that
>>>>>> | manages 9030 port.
>>>>>> |
>>>>>> |
>>>>>> | I don't understund because it and what is the original problem.
>>>>>> |
>>>>>> |
>>>>>> | For solve it i need to stoop and restart the two Glass environment.
>>>>>> |
>>>>>> |
>>>>>> | Any considerations, idea are welcome.
>>>>>> |
>>>>>> |
>>>>>> | Thanks,
>>>>>> |
>>>>>> |
>>>>>> | Dario
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>
>>>>
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

Larry Kellogg

On Apr 10, 2012, at 2:30 AM, Johan Brichau wrote:

> Larry,
>
> What is the output of the following command on your system?
>
> sudo monit summary
>

Well. I struggled with file/group ownership problems and changed it from seaside user group to root,

monit: The control file '/etc/monit.conf' must be owned by you.

So, I now have:

 ls -l monit.conf
-rw------- 1 root root 11526 Apr  8 23:58 monit.conf

sudo monit -t
Control file syntax OK

but I get

sudo monit summary
monit: error connecting to the monit daemon

Not very helpful. I see that monit is running but not really doing anything, as far as I can tell.
What am I doing wrong?

 Thanks for the help, Johan.

 Larry


>
> On 09 Apr 2012, at 11:52, Lawrence Kellogg wrote:
>
>>
>> On Apr 8, 2012, at 7:39 PM, Lawrence Kellogg wrote:
>>
>>>
>>> On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote:
>>>
>>>> Larry,
>>>>
>>>> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/
>>>> If you have any questions, I'm happy to answer them.
>>>>
>>>> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem.
>>>>
>>>> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid
>>>>     start program = "<start gem 9001>" as uid sites and gid sites
>>>>     stop program = "<stop gem 9001>" as uid sites and gid sites
>>>> # Empty FastCGI request
>>>> if failed port 9001
>>>>  # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09)
>>>>  # padding 8 bytes (0x08), followed by 8xNULLs padding
>>>>  send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
>>>>  # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A)
>>>>  expect "\0x01\0x0A"
>>>>  timeout 20 seconds
>>>> then restart
>>>>
>>
>> Johan,
>> I used the configuration you specified above and just replaced these two lines:
>>
>>      start program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001"
>>      stop program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 stop"
>>
>> but it does not seem to work. I can see that monit is running. If I kill -9 my FastCGI Gem,  the system does not stop and restart all of the Gems. Also, monit does not seem to start everything up initially. What am I missing?
>>
>> Regards,
>>
>> Larry
>>
>>
>>>
>>> Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests.
>>>
>>> Regards,
>>>
>>> Larry
>>>
>>>> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote:
>>>>
>>>>>
>>>>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote:
>>>>>
>>>>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request.
>>>>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone.
>>>>>
>>>>>
>>>>> Hello Johan,
>>>>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem  if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Larry
>>>>>
>>>>>
>>>>>>
>>>>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0.
>>>>>>
>>>>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this.
>>>>>>
>>>>>> that's all I know so far...
>>>>>>
>>>>>> Johan
>>>>>>
>>>>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote:
>>>>>>
>>>>>>> Dario,
>>>>>>>
>>>>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.
>>>>>>>
>>>>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.
>>>>>>>
>>>>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points:
>>>>>>>
>>>>>>> kill -USR1 <pid>
>>>>>>>
>>>>>>> to get the smalltalk stack dumped to the gem log and:
>>>>>>>
>>>>>>> $GEMSTONE/bin/gstack <pid>
>>>>>>>
>>>>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.
>>>>>>>
>>>>>>> Dale
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> | From: "Dario Trussardi" <[hidden email]>
>>>>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]>
>>>>>>> | Sent: Friday, March 30, 2012 7:06:30 AM
>>>>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment
>>>>>>> |
>>>>>>> | Ciao,
>>>>>>> |
>>>>>>> |
>>>>>>> | i have a server based on Ubuntu and Lighttpd.
>>>>>>> |
>>>>>>> |
>>>>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
>>>>>>> |
>>>>>>> |
>>>>>>> | One work with :
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9030
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9031
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9032
>>>>>>> |
>>>>>>> | and other with:
>>>>>>> |
>>>>>>> |
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9020
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9021
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9022
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> | All work fine for more time and the lighttpd redirect the request to
>>>>>>> | the specific environment.
>>>>>>> |
>>>>>>> |
>>>>>>> | This is one example of lighttpd configuration:
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> | $HTTP["host"] =~ "dariospizza.com$" {
>>>>>>> |
>>>>>>> | url.redirect = (
>>>>>>> |
>>>>>>> | "^/$" => " http://www.dariospizza.com/dariospizza "
>>>>>>> |
>>>>>>> | )
>>>>>>> |
>>>>>>> | $HTTP["url"] =~ "^/dariospizza" {
>>>>>>> |
>>>>>>> | fastcgi.server = (
>>>>>>> |
>>>>>>> | "/dariospizza" =>
>>>>>>> |
>>>>>>> | (
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>>>> | "disable","mode" => "responder" ),
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>>>> | "disable","mode" => "responder" ),
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>>>> | "disable","mode" => "responder" )
>>>>>>> |
>>>>>>> | )
>>>>>>> |
>>>>>>> | )
>>>>>>> |
>>>>>>> | }
>>>>>>> |
>>>>>>> | $HTTP["url"] =~ "^/Dariospizza/" {
>>>>>>> |
>>>>>>> | server.document-root = "/mnt/ISTDataBase/dbbase"
>>>>>>> |
>>>>>>> | }
>>>>>>> |
>>>>>>> | $HTTP["url"] =~ "^/filesdbbase" {
>>>>>>> |
>>>>>>> | fastcgi.server = (
>>>>>>> |
>>>>>>> | "/filesdbbase" =>
>>>>>>> |
>>>>>>> | (
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>>>> | "disable","mode" => "responder" ),
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>>>> | "disable","mode" => "responder" ),
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>>>> | "disable","mode" => "responder" )
>>>>>>> |
>>>>>>> | )
>>>>>>> |
>>>>>>> | )
>>>>>>> |
>>>>>>> | }
>>>>>>> |
>>>>>>> | }
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> | But sometime the system becomes unstable .
>>>>>>> |
>>>>>>> | The request for FastCGI 9020 are sent to the Glass environment that
>>>>>>> | manages 9030 port.
>>>>>>> |
>>>>>>> |
>>>>>>> | I don't understund because it and what is the original problem.
>>>>>>> |
>>>>>>> |
>>>>>>> | For solve it i need to stoop and restart the two Glass environment.
>>>>>>> |
>>>>>>> |
>>>>>>> | Any considerations, idea are welcome.
>>>>>>> |
>>>>>>> |
>>>>>>> | Thanks,
>>>>>>> |
>>>>>>> |
>>>>>>> | Dario
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>
>>>>>
>>>>
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

Larry Kellogg
In reply to this post by Johan Brichau-2

On Apr 10, 2012, at 2:30 AM, Johan Brichau wrote:

> Larry,
>
> What is the output of the following command on your system?
>
> sudo monit summary

I have also seen these messages...

$sudo monit
monit daemon at 7971 awakened

$ sudo monit summary
monit: error connecting to the monit daemon


Larry

>
>
> On 09 Apr 2012, at 11:52, Lawrence Kellogg wrote:
>
>>
>> On Apr 8, 2012, at 7:39 PM, Lawrence Kellogg wrote:
>>
>>>
>>> On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote:
>>>
>>>> Larry,
>>>>
>>>> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/
>>>> If you have any questions, I'm happy to answer them.
>>>>
>>>> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem.
>>>>
>>>> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid
>>>>     start program = "<start gem 9001>" as uid sites and gid sites
>>>>     stop program = "<stop gem 9001>" as uid sites and gid sites
>>>> # Empty FastCGI request
>>>> if failed port 9001
>>>>  # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09)
>>>>  # padding 8 bytes (0x08), followed by 8xNULLs padding
>>>>  send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
>>>>  # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A)
>>>>  expect "\0x01\0x0A"
>>>>  timeout 20 seconds
>>>> then restart
>>>>
>>
>> Johan,
>> I used the configuration you specified above and just replaced these two lines:
>>
>>      start program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001"
>>      stop program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 stop"
>>
>> but it does not seem to work. I can see that monit is running. If I kill -9 my FastCGI Gem,  the system does not stop and restart all of the Gems. Also, monit does not seem to start everything up initially. What am I missing?
>>
>> Regards,
>>
>> Larry
>>
>>
>>>
>>> Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests.
>>>
>>> Regards,
>>>
>>> Larry
>>>
>>>> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote:
>>>>
>>>>>
>>>>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote:
>>>>>
>>>>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request.
>>>>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone.
>>>>>
>>>>>
>>>>> Hello Johan,
>>>>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem  if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Larry
>>>>>
>>>>>
>>>>>>
>>>>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0.
>>>>>>
>>>>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this.
>>>>>>
>>>>>> that's all I know so far...
>>>>>>
>>>>>> Johan
>>>>>>
>>>>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote:
>>>>>>
>>>>>>> Dario,
>>>>>>>
>>>>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.
>>>>>>>
>>>>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.
>>>>>>>
>>>>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points:
>>>>>>>
>>>>>>> kill -USR1 <pid>
>>>>>>>
>>>>>>> to get the smalltalk stack dumped to the gem log and:
>>>>>>>
>>>>>>> $GEMSTONE/bin/gstack <pid>
>>>>>>>
>>>>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.
>>>>>>>
>>>>>>> Dale
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> | From: "Dario Trussardi" <[hidden email]>
>>>>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]>
>>>>>>> | Sent: Friday, March 30, 2012 7:06:30 AM
>>>>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment
>>>>>>> |
>>>>>>> | Ciao,
>>>>>>> |
>>>>>>> |
>>>>>>> | i have a server based on Ubuntu and Lighttpd.
>>>>>>> |
>>>>>>> |
>>>>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
>>>>>>> |
>>>>>>> |
>>>>>>> | One work with :
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9030
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9031
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9032
>>>>>>> |
>>>>>>> | and other with:
>>>>>>> |
>>>>>>> |
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9020
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9021
>>>>>>> | Starting WAFastCGIAdaptor gem on port 9022
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> | All work fine for more time and the lighttpd redirect the request to
>>>>>>> | the specific environment.
>>>>>>> |
>>>>>>> |
>>>>>>> | This is one example of lighttpd configuration:
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> | $HTTP["host"] =~ "dariospizza.com$" {
>>>>>>> |
>>>>>>> | url.redirect = (
>>>>>>> |
>>>>>>> | "^/$" => " http://www.dariospizza.com/dariospizza "
>>>>>>> |
>>>>>>> | )
>>>>>>> |
>>>>>>> | $HTTP["url"] =~ "^/dariospizza" {
>>>>>>> |
>>>>>>> | fastcgi.server = (
>>>>>>> |
>>>>>>> | "/dariospizza" =>
>>>>>>> |
>>>>>>> | (
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>>>> | "disable","mode" => "responder" ),
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>>>> | "disable","mode" => "responder" ),
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>>>> | "disable","mode" => "responder" )
>>>>>>> |
>>>>>>> | )
>>>>>>> |
>>>>>>> | )
>>>>>>> |
>>>>>>> | }
>>>>>>> |
>>>>>>> | $HTTP["url"] =~ "^/Dariospizza/" {
>>>>>>> |
>>>>>>> | server.document-root = "/mnt/ISTDataBase/dbbase"
>>>>>>> |
>>>>>>> | }
>>>>>>> |
>>>>>>> | $HTTP["url"] =~ "^/filesdbbase" {
>>>>>>> |
>>>>>>> | fastcgi.server = (
>>>>>>> |
>>>>>>> | "/filesdbbase" =>
>>>>>>> |
>>>>>>> | (
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>>>> | "disable","mode" => "responder" ),
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>>>> | "disable","mode" => "responder" ),
>>>>>>> |
>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>>>> | "disable","mode" => "responder" )
>>>>>>> |
>>>>>>> | )
>>>>>>> |
>>>>>>> | )
>>>>>>> |
>>>>>>> | }
>>>>>>> |
>>>>>>> | }
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> | But sometime the system becomes unstable .
>>>>>>> |
>>>>>>> | The request for FastCGI 9020 are sent to the Glass environment that
>>>>>>> | manages 9030 port.
>>>>>>> |
>>>>>>> |
>>>>>>> | I don't understund because it and what is the original problem.
>>>>>>> |
>>>>>>> |
>>>>>>> | For solve it i need to stoop and restart the two Glass environment.
>>>>>>> |
>>>>>>> |
>>>>>>> | Any considerations, idea are welcome.
>>>>>>> |
>>>>>>> |
>>>>>>> | Thanks,
>>>>>>> |
>>>>>>> |
>>>>>>> | Dario
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>
>>>>>
>>>>
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

Johan Brichau-2
If there's an error connecting to the monit daemon, then it's not running normally.

There should be a start/stop script in /etc/init.d/

sudo /etc/init.d/monit start

After 4-5 minutes, monit actually starts and only then you can get results from 'sudo monit summary'

If that really fails, look for errors in /var/log/messages (and family)

On 10 Apr 2012, at 11:38, Lawrence Kellogg wrote:

>
> On Apr 10, 2012, at 2:30 AM, Johan Brichau wrote:
>
>> Larry,
>>
>> What is the output of the following command on your system?
>>
>> sudo monit summary
>
> I have also seen these messages...
>
> $sudo monit
> monit daemon at 7971 awakened
>
> $ sudo monit summary
> monit: error connecting to the monit daemon
>
>
> Larry
>
>>
>>
>> On 09 Apr 2012, at 11:52, Lawrence Kellogg wrote:
>>
>>>
>>> On Apr 8, 2012, at 7:39 PM, Lawrence Kellogg wrote:
>>>
>>>>
>>>> On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote:
>>>>
>>>>> Larry,
>>>>>
>>>>> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/
>>>>> If you have any questions, I'm happy to answer them.
>>>>>
>>>>> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem.
>>>>>
>>>>> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid
>>>>>    start program = "<start gem 9001>" as uid sites and gid sites
>>>>>    stop program = "<stop gem 9001>" as uid sites and gid sites
>>>>> # Empty FastCGI request
>>>>> if failed port 9001
>>>>>  # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09)
>>>>>  # padding 8 bytes (0x08), followed by 8xNULLs padding
>>>>>  send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
>>>>>  # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A)
>>>>>  expect "\0x01\0x0A"
>>>>>  timeout 20 seconds
>>>>> then restart
>>>>>
>>>
>>> Johan,
>>> I used the configuration you specified above and just replaced these two lines:
>>>
>>>     start program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001"
>>>     stop program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 stop"
>>>
>>> but it does not seem to work. I can see that monit is running. If I kill -9 my FastCGI Gem,  the system does not stop and restart all of the Gems. Also, monit does not seem to start everything up initially. What am I missing?
>>>
>>> Regards,
>>>
>>> Larry
>>>
>>>
>>>>
>>>> Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests.
>>>>
>>>> Regards,
>>>>
>>>> Larry
>>>>
>>>>> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote:
>>>>>
>>>>>>
>>>>>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote:
>>>>>>
>>>>>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request.
>>>>>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone.
>>>>>>
>>>>>>
>>>>>> Hello Johan,
>>>>>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem  if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Larry
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0.
>>>>>>>
>>>>>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this.
>>>>>>>
>>>>>>> that's all I know so far...
>>>>>>>
>>>>>>> Johan
>>>>>>>
>>>>>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote:
>>>>>>>
>>>>>>>> Dario,
>>>>>>>>
>>>>>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.
>>>>>>>>
>>>>>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.
>>>>>>>>
>>>>>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points:
>>>>>>>>
>>>>>>>> kill -USR1 <pid>
>>>>>>>>
>>>>>>>> to get the smalltalk stack dumped to the gem log and:
>>>>>>>>
>>>>>>>> $GEMSTONE/bin/gstack <pid>
>>>>>>>>
>>>>>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.
>>>>>>>>
>>>>>>>> Dale
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>> | From: "Dario Trussardi" <[hidden email]>
>>>>>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]>
>>>>>>>> | Sent: Friday, March 30, 2012 7:06:30 AM
>>>>>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment
>>>>>>>> |
>>>>>>>> | Ciao,
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | i have a server based on Ubuntu and Lighttpd.
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | One work with :
>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9030
>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9031
>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9032
>>>>>>>> |
>>>>>>>> | and other with:
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9020
>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9021
>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9022
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | All work fine for more time and the lighttpd redirect the request to
>>>>>>>> | the specific environment.
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | This is one example of lighttpd configuration:
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | $HTTP["host"] =~ "dariospizza.com$" {
>>>>>>>> |
>>>>>>>> | url.redirect = (
>>>>>>>> |
>>>>>>>> | "^/$" => " http://www.dariospizza.com/dariospizza "
>>>>>>>> |
>>>>>>>> | )
>>>>>>>> |
>>>>>>>> | $HTTP["url"] =~ "^/dariospizza" {
>>>>>>>> |
>>>>>>>> | fastcgi.server = (
>>>>>>>> |
>>>>>>>> | "/dariospizza" =>
>>>>>>>> |
>>>>>>>> | (
>>>>>>>> |
>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>>>>> | "disable","mode" => "responder" ),
>>>>>>>> |
>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>>>>> | "disable","mode" => "responder" ),
>>>>>>>> |
>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>>>>> | "disable","mode" => "responder" )
>>>>>>>> |
>>>>>>>> | )
>>>>>>>> |
>>>>>>>> | )
>>>>>>>> |
>>>>>>>> | }
>>>>>>>> |
>>>>>>>> | $HTTP["url"] =~ "^/Dariospizza/" {
>>>>>>>> |
>>>>>>>> | server.document-root = "/mnt/ISTDataBase/dbbase"
>>>>>>>> |
>>>>>>>> | }
>>>>>>>> |
>>>>>>>> | $HTTP["url"] =~ "^/filesdbbase" {
>>>>>>>> |
>>>>>>>> | fastcgi.server = (
>>>>>>>> |
>>>>>>>> | "/filesdbbase" =>
>>>>>>>> |
>>>>>>>> | (
>>>>>>>> |
>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>>>>> | "disable","mode" => "responder" ),
>>>>>>>> |
>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>>>>> | "disable","mode" => "responder" ),
>>>>>>>> |
>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>>>>> | "disable","mode" => "responder" )
>>>>>>>> |
>>>>>>>> | )
>>>>>>>> |
>>>>>>>> | )
>>>>>>>> |
>>>>>>>> | }
>>>>>>>> |
>>>>>>>> | }
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | But sometime the system becomes unstable .
>>>>>>>> |
>>>>>>>> | The request for FastCGI 9020 are sent to the Glass environment that
>>>>>>>> | manages 9030 port.
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | I don't understund because it and what is the original problem.
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | For solve it i need to stoop and restart the two Glass environment.
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | Any considerations, idea are welcome.
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | Thanks,
>>>>>>>> |
>>>>>>>> |
>>>>>>>> | Dario
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>> |
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Web request problem on server with two Glass environment

Larry Kellogg

On Apr 10, 2012, at 8:20 AM, Johan Brichau wrote:

> If there's an error connecting to the monit daemon, then it's not running normally.
>
> There should be a start/stop script in /etc/init.d/
>
> sudo /etc/init.d/monit start

I am getting monit to start but it just says failed to start

$ sudo /etc/init.d/monit start
Starting monit:                                            [  OK  ]

and the log output

[UTC Apr 10 15:19:44] info     : 'system_ip-10-189-103-87.ec2.internal' Monit started
[UTC Apr 10 15:19:44] error    : 'fastcgi_9001' process is not running
[UTC Apr 10 15:19:44] info     : 'fastcgi_9001' trying to restart
[UTC Apr 10 15:19:44] info     : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems
[UTC Apr 10 15:20:14] error    : 'fastcgi_9001' failed to start
[UTC Apr 10 15:21:14] error    : 'fastcgi_9001' process is not running
[UTC Apr 10 15:21:14] info     : 'fastcgi_9001' trying to restart
[UTC Apr 10 15:21:14] info     : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems
[UTC Apr 10 15:21:44] error    : 'fastcgi_9001' failed to start

my startAllGems script is just

runSeasideGems30 start FastCGI 9001

and works fine when I run it from the command line.

my stopAllGems script is just

runSeasideGems30 stop

I'm stuckā€¦..

Larry




>
> After 4-5 minutes, monit actually starts and only then you can get results from 'sudo monit summary'
>
> If that really fails, look for errors in /var/log/messages (and family)
>
> On 10 Apr 2012, at 11:38, Lawrence Kellogg wrote:
>
>>
>> On Apr 10, 2012, at 2:30 AM, Johan Brichau wrote:
>>
>>> Larry,
>>>
>>> What is the output of the following command on your system?
>>>
>>> sudo monit summary
>>
>> I have also seen these messages...
>>
>> $sudo monit
>> monit daemon at 7971 awakened
>>
>> $ sudo monit summary
>> monit: error connecting to the monit daemon
>>
>>
>> Larry
>>
>>>
>>>
>>> On 09 Apr 2012, at 11:52, Lawrence Kellogg wrote:
>>>
>>>>
>>>> On Apr 8, 2012, at 7:39 PM, Lawrence Kellogg wrote:
>>>>
>>>>>
>>>>> On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote:
>>>>>
>>>>>> Larry,
>>>>>>
>>>>>> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/
>>>>>> If you have any questions, I'm happy to answer them.
>>>>>>
>>>>>> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem.
>>>>>>
>>>>>> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid
>>>>>>   start program = "<start gem 9001>" as uid sites and gid sites
>>>>>>   stop program = "<stop gem 9001>" as uid sites and gid sites
>>>>>> # Empty FastCGI request
>>>>>> if failed port 9001
>>>>>>  # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09)
>>>>>>  # padding 8 bytes (0x08), followed by 8xNULLs padding
>>>>>>  send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
>>>>>>  # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A)
>>>>>>  expect "\0x01\0x0A"
>>>>>>  timeout 20 seconds
>>>>>> then restart
>>>>>>
>>>>
>>>> Johan,
>>>> I used the configuration you specified above and just replaced these two lines:
>>>>
>>>>    start program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001"
>>>>    stop program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 stop"
>>>>
>>>> but it does not seem to work. I can see that monit is running. If I kill -9 my FastCGI Gem,  the system does not stop and restart all of the Gems. Also, monit does not seem to start everything up initially. What am I missing?
>>>>
>>>> Regards,
>>>>
>>>> Larry
>>>>
>>>>
>>>>>
>>>>> Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Larry
>>>>>
>>>>>> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote:
>>>>>>
>>>>>>>
>>>>>>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote:
>>>>>>>
>>>>>>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request.
>>>>>>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone.
>>>>>>>
>>>>>>>
>>>>>>> Hello Johan,
>>>>>>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem  if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible?
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Larry
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0.
>>>>>>>>
>>>>>>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this.
>>>>>>>>
>>>>>>>> that's all I know so far...
>>>>>>>>
>>>>>>>> Johan
>>>>>>>>
>>>>>>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote:
>>>>>>>>
>>>>>>>>> Dario,
>>>>>>>>>
>>>>>>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences.
>>>>>>>>>
>>>>>>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes.
>>>>>>>>>
>>>>>>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points:
>>>>>>>>>
>>>>>>>>> kill -USR1 <pid>
>>>>>>>>>
>>>>>>>>> to get the smalltalk stack dumped to the gem log and:
>>>>>>>>>
>>>>>>>>> $GEMSTONE/bin/gstack <pid>
>>>>>>>>>
>>>>>>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there.
>>>>>>>>>
>>>>>>>>> Dale
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>> | From: "Dario Trussardi" <[hidden email]>
>>>>>>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]>
>>>>>>>>> | Sent: Friday, March 30, 2012 7:06:30 AM
>>>>>>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment
>>>>>>>>> |
>>>>>>>>> | Ciao,
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | i have a server based on Ubuntu and Lighttpd.
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1.
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | One work with :
>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9030
>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9031
>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9032
>>>>>>>>> |
>>>>>>>>> | and other with:
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9020
>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9021
>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9022
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | All work fine for more time and the lighttpd redirect the request to
>>>>>>>>> | the specific environment.
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | This is one example of lighttpd configuration:
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | $HTTP["host"] =~ "dariospizza.com$" {
>>>>>>>>> |
>>>>>>>>> | url.redirect = (
>>>>>>>>> |
>>>>>>>>> | "^/$" => " http://www.dariospizza.com/dariospizza "
>>>>>>>>> |
>>>>>>>>> | )
>>>>>>>>> |
>>>>>>>>> | $HTTP["url"] =~ "^/dariospizza" {
>>>>>>>>> |
>>>>>>>>> | fastcgi.server = (
>>>>>>>>> |
>>>>>>>>> | "/dariospizza" =>
>>>>>>>>> |
>>>>>>>>> | (
>>>>>>>>> |
>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>>>>>> | "disable","mode" => "responder" ),
>>>>>>>>> |
>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>>>>>> | "disable","mode" => "responder" ),
>>>>>>>>> |
>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>>>>>> | "disable","mode" => "responder" )
>>>>>>>>> |
>>>>>>>>> | )
>>>>>>>>> |
>>>>>>>>> | )
>>>>>>>>> |
>>>>>>>>> | }
>>>>>>>>> |
>>>>>>>>> | $HTTP["url"] =~ "^/Dariospizza/" {
>>>>>>>>> |
>>>>>>>>> | server.document-root = "/mnt/ISTDataBase/dbbase"
>>>>>>>>> |
>>>>>>>>> | }
>>>>>>>>> |
>>>>>>>>> | $HTTP["url"] =~ "^/filesdbbase" {
>>>>>>>>> |
>>>>>>>>> | fastcgi.server = (
>>>>>>>>> |
>>>>>>>>> | "/filesdbbase" =>
>>>>>>>>> |
>>>>>>>>> | (
>>>>>>>>> |
>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" =>
>>>>>>>>> | "disable","mode" => "responder" ),
>>>>>>>>> |
>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" =>
>>>>>>>>> | "disable","mode" => "responder" ),
>>>>>>>>> |
>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" =>
>>>>>>>>> | "disable","mode" => "responder" )
>>>>>>>>> |
>>>>>>>>> | )
>>>>>>>>> |
>>>>>>>>> | )
>>>>>>>>> |
>>>>>>>>> | }
>>>>>>>>> |
>>>>>>>>> | }
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | But sometime the system becomes unstable .
>>>>>>>>> |
>>>>>>>>> | The request for FastCGI 9020 are sent to the Glass environment that
>>>>>>>>> | manages 9030 port.
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | I don't understund because it and what is the original problem.
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | For solve it i need to stoop and restart the two Glass environment.
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | Any considerations, idea are welcome.
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | Thanks,
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> | Dario
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>> |
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

123