Web request problem on server with two Glass environment
Locked
123
123
Re: Sending Email from Seaside/GLASS
 
|
> The directy usage of the gmail mail server is not possible at the moment. For this someone would need to add SSL support to gemstone. Gemstone 3.0 has an foreign function interface that makes it possible but nobody did it so far.
> I think you have two options. Either use a static SSL connector [1] that wraps any connection into SSL. Or the most complicated and most reliable thing would be to have a local mail server configured that acts as a relay between your machine and google. > > [1] http://www.stunnel.org/ > > Norbert How about an interface to Amazon's Simple Email Service? Has anyone done any work in that area for Gemstone? Larry On Apr 9, 2012, at 6:58 AM, Norbert Hartl wrote: > > Am 09.04.2012 um 11:30 schrieb Lawrence Kellogg: > >> >> On Apr 9, 2012, at 5:11 AM, Norbert Hartl wrote: >> >>> >>> Am 08.04.2012 um 21:31 schrieb Lawrence Kellogg: >>> >>>> Hello, >>>> I'm finally getting around to trying to send email from Seaside under GLASS. I have the following method. >>>> >>>> sendEmailVerificationMessage >>>> | msg | >>>> SendMail mailhost: 'smtp.gmail.com'. >>>> msg := WAEmailMessage >>>> from: (WAEmailAddress address: '[hidden email]' username: 'PracticeMusic Help') >>>> to: (WAEmailAddress address: self user emailAddress username: self user screenName) >>>> subject: 'Email Address Verification'. >>>> msg body: 'Test email message'. >>>> GRGemStonePlatform current seasideDeliverEmailMessage: msg >>>> >>>> >>>> I went through and defined all of the smptp parameters for GMail based on this page: http://email.about.com/od/accessinggmail/f/Gmail_SMTP_Settings.htm >>>> >>>> When I try to call my method, the image just hangs, and this is written to the terminal window, over and over: >>>> >>>> Waiting for server to write... >>>> Waiting for server to write... >>>> Waiting for server to write... >>>> Waiting for server to write... >>>> Waiting for server to write... >>>> Waiting for server to write… >>>> >>>> >>>> Any idea what I'm doing wrong? >>>> >>> You need TLS/SSL to send mails over gmail. It is on the mentioned web page. You cannot do this in gemstone at the moment. >>> >> >> Hello Norbert, >> Yes, I noticed the requirement for TLS/SSL, but I was hoping it was covered, magically. ;-) Thanks for letting me know. >> Does this mean there s no way to send email from Gemstone, at the moment? >> >> I need a feature where users can request their own passwords, or a link to a page to reset their passwords. >> > |
Re: Sending Email from Seaside/GLASS
|
|
In reply to this post by Larry Kellogg
Ciao,
in my Gemstone 2.x environment i load SMTP sopport from : http://seaside.gemstone.com/ss/smtp It work fine for me. Regards, Dario > > On Apr 9, 2012, at 5:11 AM, Norbert Hartl wrote: > >> >> Am 08.04.2012 um 21:31 schrieb Lawrence Kellogg: >> >>> Hello, >>> I'm finally getting around to trying to send email from Seaside under GLASS. I have the following method. >>> >>> sendEmailVerificationMessage >>> | msg | >>> SendMail mailhost: 'smtp.gmail.com'. >>> msg := WAEmailMessage >>> from: (WAEmailAddress address: '[hidden email]' username: 'PracticeMusic Help') >>> to: (WAEmailAddress address: self user emailAddress username: self user screenName) >>> subject: 'Email Address Verification'. >>> msg body: 'Test email message'. >>> GRGemStonePlatform current seasideDeliverEmailMessage: msg >>> >>> >>> I went through and defined all of the smptp parameters for GMail based on this page: http://email.about.com/od/accessinggmail/f/Gmail_SMTP_Settings.htm >>> >>> When I try to call my method, the image just hangs, and this is written to the terminal window, over and over: >>> >>> Waiting for server to write... >>> Waiting for server to write... >>> Waiting for server to write... >>> Waiting for server to write... >>> Waiting for server to write... >>> Waiting for server to write… >>> >>> >>> Any idea what I'm doing wrong? >>> >> You need TLS/SSL to send mails over gmail. It is on the mentioned web page. You cannot do this in gemstone at the moment. >> > > Hello Norbert, > Yes, I noticed the requirement for TLS/SSL, but I was hoping it was covered, magically. ;-) Thanks for letting me know. > Does this mean there s no way to send email from Gemstone, at the moment? > > I need a feature where users can request their own passwords, or a link to a page to reset their passwords. > > Regards, > > Larry > www.practicemusic.com > > > > >> Norbert >> > |
Re: Sending Email from Seaside/GLASS
|
|
In reply to this post by Larry Kellogg
Am 11.04.2012 um 07:04 schrieb Lawrence Kellogg: >> The directy usage of the gmail mail server is not possible at the moment. For this someone would need to add SSL support to gemstone. Gemstone 3.0 has an foreign function interface that makes it possible but nobody did it so far. >> I think you have two options. Either use a static SSL connector [1] that wraps any connection into SSL. Or the most complicated and most reliable thing would be to have a local mail server configured that acts as a relay between your machine and google. >> >> [1] http://www.stunnel.org/ >> >> Norbert > > > How about an interface to Amazon's Simple Email Service? Has anyone done any work in that area for Gemstone? > > Norbert > > On Apr 9, 2012, at 6:58 AM, Norbert Hartl wrote: > >> >> Am 09.04.2012 um 11:30 schrieb Lawrence Kellogg: >> >>> >>> On Apr 9, 2012, at 5:11 AM, Norbert Hartl wrote: >>> >>>> >>>> Am 08.04.2012 um 21:31 schrieb Lawrence Kellogg: >>>> >>>>> Hello, >>>>> I'm finally getting around to trying to send email from Seaside under GLASS. I have the following method. >>>>> >>>>> sendEmailVerificationMessage >>>>> | msg | >>>>> SendMail mailhost: 'smtp.gmail.com'. >>>>> msg := WAEmailMessage >>>>> from: (WAEmailAddress address: '[hidden email]' username: 'PracticeMusic Help') >>>>> to: (WAEmailAddress address: self user emailAddress username: self user screenName) >>>>> subject: 'Email Address Verification'. >>>>> msg body: 'Test email message'. >>>>> GRGemStonePlatform current seasideDeliverEmailMessage: msg >>>>> >>>>> >>>>> I went through and defined all of the smptp parameters for GMail based on this page: http://email.about.com/od/accessinggmail/f/Gmail_SMTP_Settings.htm >>>>> >>>>> When I try to call my method, the image just hangs, and this is written to the terminal window, over and over: >>>>> >>>>> Waiting for server to write... >>>>> Waiting for server to write... >>>>> Waiting for server to write... >>>>> Waiting for server to write... >>>>> Waiting for server to write... >>>>> Waiting for server to write… >>>>> >>>>> >>>>> Any idea what I'm doing wrong? >>>>> >>>> You need TLS/SSL to send mails over gmail. It is on the mentioned web page. You cannot do this in gemstone at the moment. >>>> >>> >>> Hello Norbert, >>> Yes, I noticed the requirement for TLS/SSL, but I was hoping it was covered, magically. ;-) Thanks for letting me know. >>> Does this mean there s no way to send email from Gemstone, at the moment? >>> >>> I need a feature where users can request their own passwords, or a link to a page to reset their passwords. >>> >> > |
Re: Sending Email from Seaside/GLASS
|
|
In reply to this post by dario trussardi
On Apr 11, 2012, at 3:27 AM, Dario Trussardi wrote: > Ciao, > > in my Gemstone 2.x environment i load SMTP sopport from : http://seaside.gemstone.com/ss/smtp > > It work fine for me. Do you set up an mail server on your system? I was trying to send through GMail. Could you tell me a little bit more about how you got this to work? Larry > > Regards, > > Dario > >> >> On Apr 9, 2012, at 5:11 AM, Norbert Hartl wrote: >> >>> >>> Am 08.04.2012 um 21:31 schrieb Lawrence Kellogg: >>> >>>> Hello, >>>> I'm finally getting around to trying to send email from Seaside under GLASS. I have the following method. >>>> >>>> sendEmailVerificationMessage >>>> | msg | >>>> SendMail mailhost: 'smtp.gmail.com'. >>>> msg := WAEmailMessage >>>> from: (WAEmailAddress address: '[hidden email]' username: 'PracticeMusic Help') >>>> to: (WAEmailAddress address: self user emailAddress username: self user screenName) >>>> subject: 'Email Address Verification'. >>>> msg body: 'Test email message'. >>>> GRGemStonePlatform current seasideDeliverEmailMessage: msg >>>> >>>> >>>> I went through and defined all of the smptp parameters for GMail based on this page: http://email.about.com/od/accessinggmail/f/Gmail_SMTP_Settings.htm >>>> >>>> When I try to call my method, the image just hangs, and this is written to the terminal window, over and over: >>>> >>>> Waiting for server to write... >>>> Waiting for server to write... >>>> Waiting for server to write... >>>> Waiting for server to write... >>>> Waiting for server to write... >>>> Waiting for server to write… >>>> >>>> >>>> Any idea what I'm doing wrong? >>>> >>> You need TLS/SSL to send mails over gmail. It is on the mentioned web page. You cannot do this in gemstone at the moment. >>> >> >> Hello Norbert, >> Yes, I noticed the requirement for TLS/SSL, but I was hoping it was covered, magically. ;-) Thanks for letting me know. >> Does this mean there s no way to send email from Gemstone, at the moment? >> >> I need a feature where users can request their own passwords, or a link to a page to reset their passwords. >> >> Regards, >> >> Larry >> www.practicemusic.com >> >> >> >> >>> Norbert >>> >> > |
Re: Sending Email from Seaside/GLASS
|
|
Am 11.04.2012 um 13:23 schrieb Lawrence Kellogg:
I don't install mail servers on my machines. This is a tedious task to do. It is just feasible if having mail capabilties is really important for your setup. I'm fortunate to have a well configured mail server. I started a long time ago with some other hackers to maintain a bigger server for stuff belonging to friends, family and trials. But gmail is the same thing as our server. It is just that our machine has virtual servers and I install gemstone on those. Therefor I don't need use SSL to send emails (the mail server is just a virtual instance on the same machine). On the amazon instances I use pharo as deployment platform. For this I fixed the TLS support of the wonderful Zodiac SMTP client. I use our mail server but connect it via TLS and SMTP Auth to gain permission for mail relaying. So you can see until now I circumvented to problem :) Norbert
|
Re: Sending Email from Seaside/GLASS
|
|
[…]
> For this I fixed the TLS support of the wonderful Zodiac SMTP client. I use our mail server but connect it via TLS and SMTP Auth to gain permission for mail relaying. > So you can see until now I circumvented to problem :) What do you think about integrating this into seaside-email? best -tobias |
Re: Sending Email from Seaside/GLASS
|
|
In reply to this post by Larry Kellogg
Ciao,
I do it some time ago, and i don't remember what i do. ( but i think it's not complicate > I think have load the sudo apt-get install mailx on my server based on ubuntu. ) Now when i open a ssh on the server it report : 'You have mail'. From Gemstone i call : SendMail mailhost: 'out.aliceposta.it ' " my smtp"
SendMail mailfrom: '[hidden email]' .
SendMail sendFrom: '[hidden email]'
to: emailTo asString
subject: 'Conferma ordine'
text: text. I hope this help. Dario
|
Re: Web request problem on server with two Glass environment
|
|
In reply to this post by Larry Kellogg
Larry,
Off the top of my head: - As which user is Gemstone running? Did you also include to run the scripts as that user in the monit configuration file? - If monit is running, what is the output of 'sudo monit summary' ? - what is the output of 'sudo monit start fastcgi_9001' ? We'll get there... Johan On 10 Apr 2012, at 17:50, Lawrence Kellogg wrote: > > On Apr 10, 2012, at 8:20 AM, Johan Brichau wrote: > >> If there's an error connecting to the monit daemon, then it's not running normally. >> >> There should be a start/stop script in /etc/init.d/ >> >> sudo /etc/init.d/monit start > > I am getting monit to start but it just says failed to start > > $ sudo /etc/init.d/monit start > Starting monit: [ OK ] > > and the log output > > [UTC Apr 10 15:19:44] info : 'system_ip-10-189-103-87.ec2.internal' Monit started > [UTC Apr 10 15:19:44] error : 'fastcgi_9001' process is not running > [UTC Apr 10 15:19:44] info : 'fastcgi_9001' trying to restart > [UTC Apr 10 15:19:44] info : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems > [UTC Apr 10 15:20:14] error : 'fastcgi_9001' failed to start > [UTC Apr 10 15:21:14] error : 'fastcgi_9001' process is not running > [UTC Apr 10 15:21:14] info : 'fastcgi_9001' trying to restart > [UTC Apr 10 15:21:14] info : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems > [UTC Apr 10 15:21:44] error : 'fastcgi_9001' failed to start > > my startAllGems script is just > > runSeasideGems30 start FastCGI 9001 > > and works fine when I run it from the command line. > > my stopAllGems script is just > > runSeasideGems30 stop > > I'm stuck….. > > Larry > > > > >> >> After 4-5 minutes, monit actually starts and only then you can get results from 'sudo monit summary' >> >> If that really fails, look for errors in /var/log/messages (and family) >> >> On 10 Apr 2012, at 11:38, Lawrence Kellogg wrote: >> >>> >>> On Apr 10, 2012, at 2:30 AM, Johan Brichau wrote: >>> >>>> Larry, >>>> >>>> What is the output of the following command on your system? >>>> >>>> sudo monit summary >>> >>> I have also seen these messages... >>> >>> $sudo monit >>> monit daemon at 7971 awakened >>> >>> $ sudo monit summary >>> monit: error connecting to the monit daemon >>> >>> >>> Larry >>> >>>> >>>> >>>> On 09 Apr 2012, at 11:52, Lawrence Kellogg wrote: >>>> >>>>> >>>>> On Apr 8, 2012, at 7:39 PM, Lawrence Kellogg wrote: >>>>> >>>>>> >>>>>> On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote: >>>>>> >>>>>>> Larry, >>>>>>> >>>>>>> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/ >>>>>>> If you have any questions, I'm happy to answer them. >>>>>>> >>>>>>> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem. >>>>>>> >>>>>>> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid >>>>>>> start program = "<start gem 9001>" as uid sites and gid sites >>>>>>> stop program = "<stop gem 9001>" as uid sites and gid sites >>>>>>> # Empty FastCGI request >>>>>>> if failed port 9001 >>>>>>> # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09) >>>>>>> # padding 8 bytes (0x08), followed by 8xNULLs padding >>>>>>> send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00" >>>>>>> # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A) >>>>>>> expect "\0x01\0x0A" >>>>>>> timeout 20 seconds >>>>>>> then restart >>>>>>> >>>>> >>>>> Johan, >>>>> I used the configuration you specified above and just replaced these two lines: >>>>> >>>>> start program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001" >>>>> stop program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 stop" >>>>> >>>>> but it does not seem to work. I can see that monit is running. If I kill -9 my FastCGI Gem, the system does not stop and restart all of the Gems. Also, monit does not seem to start everything up initially. What am I missing? >>>>> >>>>> Regards, >>>>> >>>>> Larry >>>>> >>>>> >>>>>> >>>>>> Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Larry >>>>>> >>>>>>> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote: >>>>>>> >>>>>>>> >>>>>>>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote: >>>>>>>> >>>>>>>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request. >>>>>>>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone. >>>>>>>> >>>>>>>> >>>>>>>> Hello Johan, >>>>>>>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible? >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Larry >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0. >>>>>>>>> >>>>>>>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this. >>>>>>>>> >>>>>>>>> that's all I know so far... >>>>>>>>> >>>>>>>>> Johan >>>>>>>>> >>>>>>>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote: >>>>>>>>> >>>>>>>>>> Dario, >>>>>>>>>> >>>>>>>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences. >>>>>>>>>> >>>>>>>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes. >>>>>>>>>> >>>>>>>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points: >>>>>>>>>> >>>>>>>>>> kill -USR1 <pid> >>>>>>>>>> >>>>>>>>>> to get the smalltalk stack dumped to the gem log and: >>>>>>>>>> >>>>>>>>>> $GEMSTONE/bin/gstack <pid> >>>>>>>>>> >>>>>>>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there. >>>>>>>>>> >>>>>>>>>> Dale >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>> | From: "Dario Trussardi" <[hidden email]> >>>>>>>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]> >>>>>>>>>> | Sent: Friday, March 30, 2012 7:06:30 AM >>>>>>>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment >>>>>>>>>> | >>>>>>>>>> | Ciao, >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | i have a server based on Ubuntu and Lighttpd. >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1. >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | One work with : >>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9030 >>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9031 >>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9032 >>>>>>>>>> | >>>>>>>>>> | and other with: >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9020 >>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9021 >>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9022 >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | All work fine for more time and the lighttpd redirect the request to >>>>>>>>>> | the specific environment. >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | This is one example of lighttpd configuration: >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | $HTTP["host"] =~ "dariospizza.com$" { >>>>>>>>>> | >>>>>>>>>> | url.redirect = ( >>>>>>>>>> | >>>>>>>>>> | "^/$" => " http://www.dariospizza.com/dariospizza " >>>>>>>>>> | >>>>>>>>>> | ) >>>>>>>>>> | >>>>>>>>>> | $HTTP["url"] =~ "^/dariospizza" { >>>>>>>>>> | >>>>>>>>>> | fastcgi.server = ( >>>>>>>>>> | >>>>>>>>>> | "/dariospizza" => >>>>>>>>>> | >>>>>>>>>> | ( >>>>>>>>>> | >>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" => >>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>> | >>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" => >>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>> | >>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" => >>>>>>>>>> | "disable","mode" => "responder" ) >>>>>>>>>> | >>>>>>>>>> | ) >>>>>>>>>> | >>>>>>>>>> | ) >>>>>>>>>> | >>>>>>>>>> | } >>>>>>>>>> | >>>>>>>>>> | $HTTP["url"] =~ "^/Dariospizza/" { >>>>>>>>>> | >>>>>>>>>> | server.document-root = "/mnt/ISTDataBase/dbbase" >>>>>>>>>> | >>>>>>>>>> | } >>>>>>>>>> | >>>>>>>>>> | $HTTP["url"] =~ "^/filesdbbase" { >>>>>>>>>> | >>>>>>>>>> | fastcgi.server = ( >>>>>>>>>> | >>>>>>>>>> | "/filesdbbase" => >>>>>>>>>> | >>>>>>>>>> | ( >>>>>>>>>> | >>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" => >>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>> | >>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" => >>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>> | >>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" => >>>>>>>>>> | "disable","mode" => "responder" ) >>>>>>>>>> | >>>>>>>>>> | ) >>>>>>>>>> | >>>>>>>>>> | ) >>>>>>>>>> | >>>>>>>>>> | } >>>>>>>>>> | >>>>>>>>>> | } >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | But sometime the system becomes unstable . >>>>>>>>>> | >>>>>>>>>> | The request for FastCGI 9020 are sent to the Glass environment that >>>>>>>>>> | manages 9030 port. >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | I don't understund because it and what is the original problem. >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | For solve it i need to stoop and restart the two Glass environment. >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | Any considerations, idea are welcome. >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | Thanks, >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | Dario >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > |
Re: Sending Email from Seaside/GLASS
|
|
In reply to this post by Tobias Pape
Am 11.04.2012 um 14:42 schrieb Tobias Pape: > […] >> For this I fixed the TLS support of the wonderful Zodiac SMTP client. I use our mail server but connect it via TLS and SMTP Auth to gain permission for mail relaying. >> So you can see until now I circumvented to problem :) > > What do you think about integrating this into seaside-email? > Good idea. The first thing I would like to do is to move the TLS/SSL dependent code of the Zodiac SMTP client into Zodiac itself because it has wider usage than SMTP. I took a quick peek at the seaside code. Authentication is already there. The easy part to get it going would be to make the smtp client pluggable. The only thing left then is to integrate a switch that decides between no ssl/ssl/tls. There is no clear way how to do it but it shouldn't be to hard to find a way. Norbert |
Re: Sending Email from Seaside/GLASS
|
|
In reply to this post by Larry Kellogg
On 12-04-11 04:23 AM, Lawrence Kellogg wrote:
> Do you set up an mail server on your system? I was trying to send through GMail. Could you tell me a little bit more about how you got this to work? > > Larry One option: http://wiki.debian.org/GmailAndExim4 But it mentions that gmail imposes a limit of 100 messages/account/day if used as a SMTP relay |
Re: Web request problem on server with two Glass environment
|
|
In reply to this post by Johan Brichau-2
On Apr 11, 2012, at 11:36 AM, Johan Brichau wrote: > Larry, > > Off the top of my head: > > - As which user is Gemstone running? Did you also include to run the scripts as that user in the monit configuration file? 500 1648 1 0 Apr08 ? 00:02:18 /opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/sys/stoned seaside 500 1649 1648 0 Apr08 ? 00:02:12 /opt/gemstone/product/sys/shrpcmonitor 'seaside@10.189.103.87' setlocationst 500 1655 1648 0 Apr08 ? 00:00:10 /opt/gemstone/product/sys/pgsvrmain seaside@10.189.103.87 0 1 -1 500 1657 1648 0 Apr08 ? 00:00:22 /opt/gemstone/product/sys/pgsvrmain TCP 60430 90 500 1665 1648 0 Apr08 ? 00:09:52 /opt/gemstone/product/sys/gem pagemanager 'seaside' -T 5000 500 1709 1648 0 Apr08 ? 00:00:13 /opt/gemstone/product/sys/gem symbolgem 'seaside' -T 10000 500 1710 1648 0 Apr08 ? 00:00:06 /opt/gemstone/product/sys/gem admingcgem 'seaside' -T 5000 500 1714 1648 0 Apr08 ? 00:01:30 /opt/gemstone/product/sys/gem reclaimgcgem 'seaside' 0 0 -T 5000 500 1729 1 0 Apr08 ? 00:00:00 /opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/sys/netldid gs64ldi -g -n -a root 1762 1 0 Apr08 ? 00:00:00 nginx: master process nginx nginx 1763 1762 0 Apr08 ? 00:00:01 nginx: worker process The kid of 500. I don't know what that means though. The Gems also run as 500 500 14315 1 1 20:51 pts/0 00:00:00 /opt/gemstone/product/bin/topaz -l -T50000 500 14317 1 82 20:51 pts/0 00:00:03 /opt/gemstone/product/bin/topaz -l -T200000 500 14319 1 3 20:51 pts/0 00:00:00 /opt/gemstone/product/bin/topaz -l -T45000 Perhaps this is the problem. I tried this: start program = "/bin/bash -c '/opt/gemstone/product/seaside/bin/startAllGems&> /tmp/startAllGems.log' " stop program = " /bin/bash -c '/opt/gemstone/product/seaside/bin/stopAllGems&> /tmp/stopAllGems.log ' " because I read here: http://mmonit.com/wiki/Monit/FAQ that bin/bash should be used to excute scripts. after I start monit, my log file shows this: $ cat startAllGems.log Starting Gems.... Starting FastCGI gem on port 9001 Starting maintenance gem Starting service gem but I don't see any Gems running and my service is not reachable. > - If monit is running, what is the output of 'sudo monit summary' ? monit is running root 14369 1 0 20:55 ? 00:00:00 monit but sudo monit summary gives an error $ sudo monit summary monit: error connecting to the monit daemon > - what is the output of 'sudo monit start fastcgi_9001' ? > $ sudo monit start fastcgi_9001 monit: Cannot connect to the monit daemon. Did you start it with http support? > We'll get there… Thanks, I know we'll figure it out. Larry > > Johan > > On 10 Apr 2012, at 17:50, Lawrence Kellogg wrote: > >> >> On Apr 10, 2012, at 8:20 AM, Johan Brichau wrote: >> >>> If there's an error connecting to the monit daemon, then it's not running normally. >>> >>> There should be a start/stop script in /etc/init.d/ >>> >>> sudo /etc/init.d/monit start >> >> I am getting monit to start but it just says failed to start >> >> $ sudo /etc/init.d/monit start >> Starting monit: [ OK ] >> >> and the log output >> >> [UTC Apr 10 15:19:44] info : 'system_ip-10-189-103-87.ec2.internal' Monit started >> [UTC Apr 10 15:19:44] error : 'fastcgi_9001' process is not running >> [UTC Apr 10 15:19:44] info : 'fastcgi_9001' trying to restart >> [UTC Apr 10 15:19:44] info : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems >> [UTC Apr 10 15:20:14] error : 'fastcgi_9001' failed to start >> [UTC Apr 10 15:21:14] error : 'fastcgi_9001' process is not running >> [UTC Apr 10 15:21:14] info : 'fastcgi_9001' trying to restart >> [UTC Apr 10 15:21:14] info : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems >> [UTC Apr 10 15:21:44] error : 'fastcgi_9001' failed to start >> >> my startAllGems script is just >> >> runSeasideGems30 start FastCGI 9001 >> >> and works fine when I run it from the command line. >> >> my stopAllGems script is just >> >> runSeasideGems30 stop >> >> I'm stuck….. >> >> Larry >> >> >> >> >>> >>> After 4-5 minutes, monit actually starts and only then you can get results from 'sudo monit summary' >>> >>> If that really fails, look for errors in /var/log/messages (and family) >>> >>> On 10 Apr 2012, at 11:38, Lawrence Kellogg wrote: >>> >>>> >>>> On Apr 10, 2012, at 2:30 AM, Johan Brichau wrote: >>>> >>>>> Larry, >>>>> >>>>> What is the output of the following command on your system? >>>>> >>>>> sudo monit summary >>>> >>>> I have also seen these messages... >>>> >>>> $sudo monit >>>> monit daemon at 7971 awakened >>>> >>>> $ sudo monit summary >>>> monit: error connecting to the monit daemon >>>> >>>> >>>> Larry >>>> >>>>> >>>>> >>>>> On 09 Apr 2012, at 11:52, Lawrence Kellogg wrote: >>>>> >>>>>> >>>>>> On Apr 8, 2012, at 7:39 PM, Lawrence Kellogg wrote: >>>>>> >>>>>>> >>>>>>> On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote: >>>>>>> >>>>>>>> Larry, >>>>>>>> >>>>>>>> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/ >>>>>>>> If you have any questions, I'm happy to answer them. >>>>>>>> >>>>>>>> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem. >>>>>>>> >>>>>>>> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid >>>>>>>> start program = "<start gem 9001>" as uid sites and gid sites >>>>>>>> stop program = "<stop gem 9001>" as uid sites and gid sites >>>>>>>> # Empty FastCGI request >>>>>>>> if failed port 9001 >>>>>>>> # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09) >>>>>>>> # padding 8 bytes (0x08), followed by 8xNULLs padding >>>>>>>> send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00" >>>>>>>> # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A) >>>>>>>> expect "\0x01\0x0A" >>>>>>>> timeout 20 seconds >>>>>>>> then restart >>>>>>>> >>>>>> >>>>>> Johan, >>>>>> I used the configuration you specified above and just replaced these two lines: >>>>>> >>>>>> start program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001" >>>>>> stop program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 stop" >>>>>> >>>>>> but it does not seem to work. I can see that monit is running. If I kill -9 my FastCGI Gem, the system does not stop and restart all of the Gems. Also, monit does not seem to start everything up initially. What am I missing? >>>>>> >>>>>> Regards, >>>>>> >>>>>> Larry >>>>>> >>>>>> >>>>>>> >>>>>>> Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Larry >>>>>>> >>>>>>>> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote: >>>>>>>>> >>>>>>>>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request. >>>>>>>>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone. >>>>>>>>> >>>>>>>>> >>>>>>>>> Hello Johan, >>>>>>>>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Larry >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0. >>>>>>>>>> >>>>>>>>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this. >>>>>>>>>> >>>>>>>>>> that's all I know so far... >>>>>>>>>> >>>>>>>>>> Johan >>>>>>>>>> >>>>>>>>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote: >>>>>>>>>> >>>>>>>>>>> Dario, >>>>>>>>>>> >>>>>>>>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences. >>>>>>>>>>> >>>>>>>>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes. >>>>>>>>>>> >>>>>>>>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points: >>>>>>>>>>> >>>>>>>>>>> kill -USR1 <pid> >>>>>>>>>>> >>>>>>>>>>> to get the smalltalk stack dumped to the gem log and: >>>>>>>>>>> >>>>>>>>>>> $GEMSTONE/bin/gstack <pid> >>>>>>>>>>> >>>>>>>>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there. >>>>>>>>>>> >>>>>>>>>>> Dale >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> | From: "Dario Trussardi" <[hidden email]> >>>>>>>>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]> >>>>>>>>>>> | Sent: Friday, March 30, 2012 7:06:30 AM >>>>>>>>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment >>>>>>>>>>> | >>>>>>>>>>> | Ciao, >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | i have a server based on Ubuntu and Lighttpd. >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1. >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | One work with : >>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9030 >>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9031 >>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9032 >>>>>>>>>>> | >>>>>>>>>>> | and other with: >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9020 >>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9021 >>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9022 >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | All work fine for more time and the lighttpd redirect the request to >>>>>>>>>>> | the specific environment. >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | This is one example of lighttpd configuration: >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | $HTTP["host"] =~ "dariospizza.com$" { >>>>>>>>>>> | >>>>>>>>>>> | url.redirect = ( >>>>>>>>>>> | >>>>>>>>>>> | "^/$" => " http://www.dariospizza.com/dariospizza " >>>>>>>>>>> | >>>>>>>>>>> | ) >>>>>>>>>>> | >>>>>>>>>>> | $HTTP["url"] =~ "^/dariospizza" { >>>>>>>>>>> | >>>>>>>>>>> | fastcgi.server = ( >>>>>>>>>>> | >>>>>>>>>>> | "/dariospizza" => >>>>>>>>>>> | >>>>>>>>>>> | ( >>>>>>>>>>> | >>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" => >>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>> | >>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" => >>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>> | >>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" => >>>>>>>>>>> | "disable","mode" => "responder" ) >>>>>>>>>>> | >>>>>>>>>>> | ) >>>>>>>>>>> | >>>>>>>>>>> | ) >>>>>>>>>>> | >>>>>>>>>>> | } >>>>>>>>>>> | >>>>>>>>>>> | $HTTP["url"] =~ "^/Dariospizza/" { >>>>>>>>>>> | >>>>>>>>>>> | server.document-root = "/mnt/ISTDataBase/dbbase" >>>>>>>>>>> | >>>>>>>>>>> | } >>>>>>>>>>> | >>>>>>>>>>> | $HTTP["url"] =~ "^/filesdbbase" { >>>>>>>>>>> | >>>>>>>>>>> | fastcgi.server = ( >>>>>>>>>>> | >>>>>>>>>>> | "/filesdbbase" => >>>>>>>>>>> | >>>>>>>>>>> | ( >>>>>>>>>>> | >>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" => >>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>> | >>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" => >>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>> | >>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" => >>>>>>>>>>> | "disable","mode" => "responder" ) >>>>>>>>>>> | >>>>>>>>>>> | ) >>>>>>>>>>> | >>>>>>>>>>> | ) >>>>>>>>>>> | >>>>>>>>>>> | } >>>>>>>>>>> | >>>>>>>>>>> | } >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | But sometime the system becomes unstable . >>>>>>>>>>> | >>>>>>>>>>> | The request for FastCGI 9020 are sent to the Glass environment that >>>>>>>>>>> | manages 9030 port. >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | I don't understund because it and what is the original problem. >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | For solve it i need to stoop and restart the two Glass environment. >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | Any considerations, idea are welcome. >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | Thanks, >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | Dario >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>>> | >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > |
Re: Sending Email from Seaside/GLASS
|
|
In reply to this post by NorbertHartl
Am 2012-04-11 um 17:57 schrieb Norbert Hartl: > > Am 11.04.2012 um 14:42 schrieb Tobias Pape: > >> […] >>> For this I fixed the TLS support of the wonderful Zodiac SMTP client. I use our mail server but connect it via TLS and SMTP Auth to gain permission for mail relaying. >>> So you can see until now I circumvented to problem :) >> >> What do you think about integrating this into seaside-email? >> > Good idea. The first thing I would like to do is to move the TLS/SSL dependent code of the Zodiac SMTP client into Zodiac itself because it has wider usage than SMTP. I took a quick peek at the seaside code. Authentication is already there. The easy part to get it going would be to make the smtp client pluggable. The only thing left then is to integrate a switch that decides between no ssl/ssl/tls. There is no clear way how to do it but it shouldn't be to hard to find a way. The SMTP-stuff is already platform-dependent via grease. It should be possible to make that stuff pluggable and add a TLS/no-TLS option to the seaside-config best -tobias |
Re: Sending Email from Seaside/GLASS
|
|
Am 12.04.2012 um 11:34 schrieb Tobias Pape: > > Am 2012-04-11 um 17:57 schrieb Norbert Hartl: > >> >> Am 11.04.2012 um 14:42 schrieb Tobias Pape: >> >>> […] >>>> For this I fixed the TLS support of the wonderful Zodiac SMTP client. I use our mail server but connect it via TLS and SMTP Auth to gain permission for mail relaying. >>>> So you can see until now I circumvented to problem :) >>> >>> What do you think about integrating this into seaside-email? >>> >> Good idea. The first thing I would like to do is to move the TLS/SSL dependent code of the Zodiac SMTP client into Zodiac itself because it has wider usage than SMTP. I took a quick peek at the seaside code. Authentication is already there. The easy part to get it going would be to make the smtp client pluggable. The only thing left then is to integrate a switch that decides between no ssl/ssl/tls. There is no clear way how to do it but it shouldn't be to hard to find a way. > > The SMTP-stuff is already platform-dependent via grease. > It should be possible to make that stuff pluggable and add > a TLS/no-TLS option to the seaside-config > Norbert |
Re: Sending Email from Seaside/GLASS
|
|
Am 2012-04-12 um 12:00 schrieb Norbert Hartl: > > Am 12.04.2012 um 11:34 schrieb Tobias Pape: > >> >> Am 2012-04-11 um 17:57 schrieb Norbert Hartl: >> >>> >>> Am 11.04.2012 um 14:42 schrieb Tobias Pape: >>> >>>> […] >>>>> For this I fixed the TLS support of the wonderful Zodiac SMTP client. I use our mail server but connect it via TLS and SMTP Auth to gain permission for mail relaying. >>>>> So you can see until now I circumvented to problem :) >>>> >>>> What do you think about integrating this into seaside-email? >>>> >>> Good idea. The first thing I would like to do is to move the TLS/SSL dependent code of the Zodiac SMTP client into Zodiac itself because it has wider usage than SMTP. I took a quick peek at the seaside code. Authentication is already there. The easy part to get it going would be to make the smtp client pluggable. The only thing left then is to integrate a switch that decides between no ssl/ssl/tls. There is no clear way how to do it but it shouldn't be to hard to find a way. >> >> The SMTP-stuff is already platform-dependent via grease. >> It should be possible to make that stuff pluggable and add >> a TLS/no-TLS option to the seaside-config >> > Exactly what I was saying ;) nearly. I was talking from the Seaside-email configuration/Grease impl perspective ;) |
Re: Web request problem on server with two Glass environment
|
|
In reply to this post by Larry Kellogg
Johan,
Any thoughts? I really should use monit because that Object does not exist error comes up every once in a while and makes the Gem unresponsive, which means my service becomes unreachable. A bummer. Thanks, Larry On Apr 11, 2012, at 4:59 PM, Lawrence Kellogg wrote: > > On Apr 11, 2012, at 11:36 AM, Johan Brichau wrote: > >> Larry, >> >> Off the top of my head: >> >> - As which user is Gemstone running? Did you also include to run the scripts as that user in the monit configuration file? > > > 500 1648 1 0 Apr08 ? 00:02:18 /opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/sys/stoned seaside > 500 1649 1648 0 Apr08 ? 00:02:12 /opt/gemstone/product/sys/shrpcmonitor 'seaside@10.189.103.87' setlocationst > 500 1655 1648 0 Apr08 ? 00:00:10 /opt/gemstone/product/sys/pgsvrmain seaside@10.189.103.87 0 1 -1 > 500 1657 1648 0 Apr08 ? 00:00:22 /opt/gemstone/product/sys/pgsvrmain TCP 60430 90 > 500 1665 1648 0 Apr08 ? 00:09:52 /opt/gemstone/product/sys/gem pagemanager 'seaside' -T 5000 > 500 1709 1648 0 Apr08 ? 00:00:13 /opt/gemstone/product/sys/gem symbolgem 'seaside' -T 10000 > 500 1710 1648 0 Apr08 ? 00:00:06 /opt/gemstone/product/sys/gem admingcgem 'seaside' -T 5000 > 500 1714 1648 0 Apr08 ? 00:01:30 /opt/gemstone/product/sys/gem reclaimgcgem 'seaside' 0 0 -T 5000 > 500 1729 1 0 Apr08 ? 00:00:00 /opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/sys/netldid gs64ldi -g -n -a > root 1762 1 0 Apr08 ? 00:00:00 nginx: master process nginx > nginx 1763 1762 0 Apr08 ? 00:00:01 nginx: worker process > > The kid of 500. I don't know what that means though. > > The Gems also run as 500 > > 500 14315 1 1 20:51 pts/0 00:00:00 /opt/gemstone/product/bin/topaz -l -T50000 > 500 14317 1 82 20:51 pts/0 00:00:03 /opt/gemstone/product/bin/topaz -l -T200000 > 500 14319 1 3 20:51 pts/0 00:00:00 /opt/gemstone/product/bin/topaz -l -T45000 > > Perhaps this is the problem. > > I tried this: > > start program = "/bin/bash -c '/opt/gemstone/product/seaside/bin/startAllGems&> /tmp/startAllGems.log' " > stop program = " /bin/bash -c '/opt/gemstone/product/seaside/bin/stopAllGems&> /tmp/stopAllGems.log ' " > > because I read here: http://mmonit.com/wiki/Monit/FAQ > > that bin/bash should be used to excute scripts. > > after I start monit, my log file shows this: > > $ cat startAllGems.log > Starting Gems.... > Starting FastCGI gem on port 9001 > Starting maintenance gem > Starting service gem > > but I don't see any Gems running and my service is not reachable. > >> - If monit is running, what is the output of 'sudo monit summary' ? > > > monit is running > > root 14369 1 0 20:55 ? 00:00:00 monit > > but sudo monit summary gives an error > > $ sudo monit summary > monit: error connecting to the monit daemon > > >> - what is the output of 'sudo monit start fastcgi_9001' ? >> > > $ sudo monit start fastcgi_9001 > monit: Cannot connect to the monit daemon. Did you start it with http support? > > >> We'll get there… > > Thanks, I know we'll figure it out. > > Larry > > >> >> Johan >> >> On 10 Apr 2012, at 17:50, Lawrence Kellogg wrote: >> >>> >>> On Apr 10, 2012, at 8:20 AM, Johan Brichau wrote: >>> >>>> If there's an error connecting to the monit daemon, then it's not running normally. >>>> >>>> There should be a start/stop script in /etc/init.d/ >>>> >>>> sudo /etc/init.d/monit start >>> >>> I am getting monit to start but it just says failed to start >>> >>> $ sudo /etc/init.d/monit start >>> Starting monit: [ OK ] >>> >>> and the log output >>> >>> [UTC Apr 10 15:19:44] info : 'system_ip-10-189-103-87.ec2.internal' Monit started >>> [UTC Apr 10 15:19:44] error : 'fastcgi_9001' process is not running >>> [UTC Apr 10 15:19:44] info : 'fastcgi_9001' trying to restart >>> [UTC Apr 10 15:19:44] info : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems >>> [UTC Apr 10 15:20:14] error : 'fastcgi_9001' failed to start >>> [UTC Apr 10 15:21:14] error : 'fastcgi_9001' process is not running >>> [UTC Apr 10 15:21:14] info : 'fastcgi_9001' trying to restart >>> [UTC Apr 10 15:21:14] info : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems >>> [UTC Apr 10 15:21:44] error : 'fastcgi_9001' failed to start >>> >>> my startAllGems script is just >>> >>> runSeasideGems30 start FastCGI 9001 >>> >>> and works fine when I run it from the command line. >>> >>> my stopAllGems script is just >>> >>> runSeasideGems30 stop >>> >>> I'm stuck….. >>> >>> Larry >>> >>> >>> >>> >>>> >>>> After 4-5 minutes, monit actually starts and only then you can get results from 'sudo monit summary' >>>> >>>> If that really fails, look for errors in /var/log/messages (and family) >>>> >>>> On 10 Apr 2012, at 11:38, Lawrence Kellogg wrote: >>>> >>>>> >>>>> On Apr 10, 2012, at 2:30 AM, Johan Brichau wrote: >>>>> >>>>>> Larry, >>>>>> >>>>>> What is the output of the following command on your system? >>>>>> >>>>>> sudo monit summary >>>>> >>>>> I have also seen these messages... >>>>> >>>>> $sudo monit >>>>> monit daemon at 7971 awakened >>>>> >>>>> $ sudo monit summary >>>>> monit: error connecting to the monit daemon >>>>> >>>>> >>>>> Larry >>>>> >>>>>> >>>>>> >>>>>> On 09 Apr 2012, at 11:52, Lawrence Kellogg wrote: >>>>>> >>>>>>> >>>>>>> On Apr 8, 2012, at 7:39 PM, Lawrence Kellogg wrote: >>>>>>> >>>>>>>> >>>>>>>> On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote: >>>>>>>> >>>>>>>>> Larry, >>>>>>>>> >>>>>>>>> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/ >>>>>>>>> If you have any questions, I'm happy to answer them. >>>>>>>>> >>>>>>>>> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem. >>>>>>>>> >>>>>>>>> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid >>>>>>>>> start program = "<start gem 9001>" as uid sites and gid sites >>>>>>>>> stop program = "<stop gem 9001>" as uid sites and gid sites >>>>>>>>> # Empty FastCGI request >>>>>>>>> if failed port 9001 >>>>>>>>> # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09) >>>>>>>>> # padding 8 bytes (0x08), followed by 8xNULLs padding >>>>>>>>> send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00" >>>>>>>>> # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A) >>>>>>>>> expect "\0x01\0x0A" >>>>>>>>> timeout 20 seconds >>>>>>>>> then restart >>>>>>>>> >>>>>>> >>>>>>> Johan, >>>>>>> I used the configuration you specified above and just replaced these two lines: >>>>>>> >>>>>>> start program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001" >>>>>>> stop program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 stop" >>>>>>> >>>>>>> but it does not seem to work. I can see that monit is running. If I kill -9 my FastCGI Gem, the system does not stop and restart all of the Gems. Also, monit does not seem to start everything up initially. What am I missing? >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Larry >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Larry >>>>>>>> >>>>>>>>> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote: >>>>>>>>>> >>>>>>>>>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request. >>>>>>>>>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hello Johan, >>>>>>>>>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Larry >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0. >>>>>>>>>>> >>>>>>>>>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this. >>>>>>>>>>> >>>>>>>>>>> that's all I know so far... >>>>>>>>>>> >>>>>>>>>>> Johan >>>>>>>>>>> >>>>>>>>>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote: >>>>>>>>>>> >>>>>>>>>>>> Dario, >>>>>>>>>>>> >>>>>>>>>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences. >>>>>>>>>>>> >>>>>>>>>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes. >>>>>>>>>>>> >>>>>>>>>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points: >>>>>>>>>>>> >>>>>>>>>>>> kill -USR1 <pid> >>>>>>>>>>>> >>>>>>>>>>>> to get the smalltalk stack dumped to the gem log and: >>>>>>>>>>>> >>>>>>>>>>>> $GEMSTONE/bin/gstack <pid> >>>>>>>>>>>> >>>>>>>>>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there. >>>>>>>>>>>> >>>>>>>>>>>> Dale >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> | From: "Dario Trussardi" <[hidden email]> >>>>>>>>>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]> >>>>>>>>>>>> | Sent: Friday, March 30, 2012 7:06:30 AM >>>>>>>>>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment >>>>>>>>>>>> | >>>>>>>>>>>> | Ciao, >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | i have a server based on Ubuntu and Lighttpd. >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1. >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | One work with : >>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9030 >>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9031 >>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9032 >>>>>>>>>>>> | >>>>>>>>>>>> | and other with: >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9020 >>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9021 >>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9022 >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | All work fine for more time and the lighttpd redirect the request to >>>>>>>>>>>> | the specific environment. >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | This is one example of lighttpd configuration: >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | $HTTP["host"] =~ "dariospizza.com$" { >>>>>>>>>>>> | >>>>>>>>>>>> | url.redirect = ( >>>>>>>>>>>> | >>>>>>>>>>>> | "^/$" => " http://www.dariospizza.com/dariospizza " >>>>>>>>>>>> | >>>>>>>>>>>> | ) >>>>>>>>>>>> | >>>>>>>>>>>> | $HTTP["url"] =~ "^/dariospizza" { >>>>>>>>>>>> | >>>>>>>>>>>> | fastcgi.server = ( >>>>>>>>>>>> | >>>>>>>>>>>> | "/dariospizza" => >>>>>>>>>>>> | >>>>>>>>>>>> | ( >>>>>>>>>>>> | >>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" => >>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>> | >>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" => >>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>> | >>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" => >>>>>>>>>>>> | "disable","mode" => "responder" ) >>>>>>>>>>>> | >>>>>>>>>>>> | ) >>>>>>>>>>>> | >>>>>>>>>>>> | ) >>>>>>>>>>>> | >>>>>>>>>>>> | } >>>>>>>>>>>> | >>>>>>>>>>>> | $HTTP["url"] =~ "^/Dariospizza/" { >>>>>>>>>>>> | >>>>>>>>>>>> | server.document-root = "/mnt/ISTDataBase/dbbase" >>>>>>>>>>>> | >>>>>>>>>>>> | } >>>>>>>>>>>> | >>>>>>>>>>>> | $HTTP["url"] =~ "^/filesdbbase" { >>>>>>>>>>>> | >>>>>>>>>>>> | fastcgi.server = ( >>>>>>>>>>>> | >>>>>>>>>>>> | "/filesdbbase" => >>>>>>>>>>>> | >>>>>>>>>>>> | ( >>>>>>>>>>>> | >>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" => >>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>> | >>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" => >>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>> | >>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" => >>>>>>>>>>>> | "disable","mode" => "responder" ) >>>>>>>>>>>> | >>>>>>>>>>>> | ) >>>>>>>>>>>> | >>>>>>>>>>>> | ) >>>>>>>>>>>> | >>>>>>>>>>>> | } >>>>>>>>>>>> | >>>>>>>>>>>> | } >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | But sometime the system becomes unstable . >>>>>>>>>>>> | >>>>>>>>>>>> | The request for FastCGI 9020 are sent to the Glass environment that >>>>>>>>>>>> | manages 9030 port. >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | I don't understund because it and what is the original problem. >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | For solve it i need to stoop and restart the two Glass environment. >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | Any considerations, idea are welcome. >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | Thanks, >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | Dario >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>>> | >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > |
Re: Web request problem on server with two Glass environment
|
|
Larry,
I think the monit daemon was not configured correctly. Check the monit configuration file (/etc/monitrc but it depends on your distribution's installation) that it contains the following line: set httpd port 2812 and allow localhost From this file, you also need to include all directives for checking the fastcgi gems (amongst others) And you also need to execute the start/stop command as a specific user. In monit you can do the following by explicitly giving a uid/gid start program = "..." as uid 500 and gid 500 It's also strange that uid 500 is not a named user, actually... Johan On 13 Apr 2012, at 13:30, Lawrence Kellogg wrote: > Johan, > Any thoughts? I really should use monit because that Object does not exist error comes up every once in a while > and makes the Gem unresponsive, which means my service becomes unreachable. A bummer. > > Thanks, > > Larry > > > On Apr 11, 2012, at 4:59 PM, Lawrence Kellogg wrote: > >> >> On Apr 11, 2012, at 11:36 AM, Johan Brichau wrote: >> >>> Larry, >>> >>> Off the top of my head: >>> >>> - As which user is Gemstone running? Did you also include to run the scripts as that user in the monit configuration file? >> >> >> 500 1648 1 0 Apr08 ? 00:02:18 /opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/sys/stoned seaside >> 500 1649 1648 0 Apr08 ? 00:02:12 /opt/gemstone/product/sys/shrpcmonitor 'seaside@10.189.103.87' setlocationst >> 500 1655 1648 0 Apr08 ? 00:00:10 /opt/gemstone/product/sys/pgsvrmain seaside@10.189.103.87 0 1 -1 >> 500 1657 1648 0 Apr08 ? 00:00:22 /opt/gemstone/product/sys/pgsvrmain TCP 60430 90 >> 500 1665 1648 0 Apr08 ? 00:09:52 /opt/gemstone/product/sys/gem pagemanager 'seaside' -T 5000 >> 500 1709 1648 0 Apr08 ? 00:00:13 /opt/gemstone/product/sys/gem symbolgem 'seaside' -T 10000 >> 500 1710 1648 0 Apr08 ? 00:00:06 /opt/gemstone/product/sys/gem admingcgem 'seaside' -T 5000 >> 500 1714 1648 0 Apr08 ? 00:01:30 /opt/gemstone/product/sys/gem reclaimgcgem 'seaside' 0 0 -T 5000 >> 500 1729 1 0 Apr08 ? 00:00:00 /opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/sys/netldid gs64ldi -g -n -a >> root 1762 1 0 Apr08 ? 00:00:00 nginx: master process nginx >> nginx 1763 1762 0 Apr08 ? 00:00:01 nginx: worker process >> >> The kid of 500. I don't know what that means though. >> >> The Gems also run as 500 >> >> 500 14315 1 1 20:51 pts/0 00:00:00 /opt/gemstone/product/bin/topaz -l -T50000 >> 500 14317 1 82 20:51 pts/0 00:00:03 /opt/gemstone/product/bin/topaz -l -T200000 >> 500 14319 1 3 20:51 pts/0 00:00:00 /opt/gemstone/product/bin/topaz -l -T45000 >> >> Perhaps this is the problem. >> >> I tried this: >> >> start program = "/bin/bash -c '/opt/gemstone/product/seaside/bin/startAllGems&> /tmp/startAllGems.log' " >> stop program = " /bin/bash -c '/opt/gemstone/product/seaside/bin/stopAllGems&> /tmp/stopAllGems.log ' " >> >> because I read here: http://mmonit.com/wiki/Monit/FAQ >> >> that bin/bash should be used to excute scripts. >> >> after I start monit, my log file shows this: >> >> $ cat startAllGems.log >> Starting Gems.... >> Starting FastCGI gem on port 9001 >> Starting maintenance gem >> Starting service gem >> >> but I don't see any Gems running and my service is not reachable. >> >>> - If monit is running, what is the output of 'sudo monit summary' ? >> >> >> monit is running >> >> root 14369 1 0 20:55 ? 00:00:00 monit >> >> but sudo monit summary gives an error >> >> $ sudo monit summary >> monit: error connecting to the monit daemon >> >> >>> - what is the output of 'sudo monit start fastcgi_9001' ? >>> >> >> $ sudo monit start fastcgi_9001 >> monit: Cannot connect to the monit daemon. Did you start it with http support? >> >> >>> We'll get there… >> >> Thanks, I know we'll figure it out. >> >> Larry >> >> >>> >>> Johan >>> >>> On 10 Apr 2012, at 17:50, Lawrence Kellogg wrote: >>> >>>> >>>> On Apr 10, 2012, at 8:20 AM, Johan Brichau wrote: >>>> >>>>> If there's an error connecting to the monit daemon, then it's not running normally. >>>>> >>>>> There should be a start/stop script in /etc/init.d/ >>>>> >>>>> sudo /etc/init.d/monit start >>>> >>>> I am getting monit to start but it just says failed to start >>>> >>>> $ sudo /etc/init.d/monit start >>>> Starting monit: [ OK ] >>>> >>>> and the log output >>>> >>>> [UTC Apr 10 15:19:44] info : 'system_ip-10-189-103-87.ec2.internal' Monit started >>>> [UTC Apr 10 15:19:44] error : 'fastcgi_9001' process is not running >>>> [UTC Apr 10 15:19:44] info : 'fastcgi_9001' trying to restart >>>> [UTC Apr 10 15:19:44] info : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems >>>> [UTC Apr 10 15:20:14] error : 'fastcgi_9001' failed to start >>>> [UTC Apr 10 15:21:14] error : 'fastcgi_9001' process is not running >>>> [UTC Apr 10 15:21:14] info : 'fastcgi_9001' trying to restart >>>> [UTC Apr 10 15:21:14] info : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems >>>> [UTC Apr 10 15:21:44] error : 'fastcgi_9001' failed to start >>>> >>>> my startAllGems script is just >>>> >>>> runSeasideGems30 start FastCGI 9001 >>>> >>>> and works fine when I run it from the command line. >>>> >>>> my stopAllGems script is just >>>> >>>> runSeasideGems30 stop >>>> >>>> I'm stuck….. >>>> >>>> Larry >>>> >>>> >>>> >>>> >>>>> >>>>> After 4-5 minutes, monit actually starts and only then you can get results from 'sudo monit summary' >>>>> >>>>> If that really fails, look for errors in /var/log/messages (and family) >>>>> >>>>> On 10 Apr 2012, at 11:38, Lawrence Kellogg wrote: >>>>> >>>>>> >>>>>> On Apr 10, 2012, at 2:30 AM, Johan Brichau wrote: >>>>>> >>>>>>> Larry, >>>>>>> >>>>>>> What is the output of the following command on your system? >>>>>>> >>>>>>> sudo monit summary >>>>>> >>>>>> I have also seen these messages... >>>>>> >>>>>> $sudo monit >>>>>> monit daemon at 7971 awakened >>>>>> >>>>>> $ sudo monit summary >>>>>> monit: error connecting to the monit daemon >>>>>> >>>>>> >>>>>> Larry >>>>>> >>>>>>> >>>>>>> >>>>>>> On 09 Apr 2012, at 11:52, Lawrence Kellogg wrote: >>>>>>> >>>>>>>> >>>>>>>> On Apr 8, 2012, at 7:39 PM, Lawrence Kellogg wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote: >>>>>>>>> >>>>>>>>>> Larry, >>>>>>>>>> >>>>>>>>>> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/ >>>>>>>>>> If you have any questions, I'm happy to answer them. >>>>>>>>>> >>>>>>>>>> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem. >>>>>>>>>> >>>>>>>>>> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid >>>>>>>>>> start program = "<start gem 9001>" as uid sites and gid sites >>>>>>>>>> stop program = "<stop gem 9001>" as uid sites and gid sites >>>>>>>>>> # Empty FastCGI request >>>>>>>>>> if failed port 9001 >>>>>>>>>> # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09) >>>>>>>>>> # padding 8 bytes (0x08), followed by 8xNULLs padding >>>>>>>>>> send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00" >>>>>>>>>> # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A) >>>>>>>>>> expect "\0x01\0x0A" >>>>>>>>>> timeout 20 seconds >>>>>>>>>> then restart >>>>>>>>>> >>>>>>>> >>>>>>>> Johan, >>>>>>>> I used the configuration you specified above and just replaced these two lines: >>>>>>>> >>>>>>>> start program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001" >>>>>>>> stop program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 stop" >>>>>>>> >>>>>>>> but it does not seem to work. I can see that monit is running. If I kill -9 my FastCGI Gem, the system does not stop and restart all of the Gems. Also, monit does not seem to start everything up initially. What am I missing? >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Larry >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Larry >>>>>>>>> >>>>>>>>>> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote: >>>>>>>>>>> >>>>>>>>>>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request. >>>>>>>>>>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hello Johan, >>>>>>>>>>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Larry >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0. >>>>>>>>>>>> >>>>>>>>>>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this. >>>>>>>>>>>> >>>>>>>>>>>> that's all I know so far... >>>>>>>>>>>> >>>>>>>>>>>> Johan >>>>>>>>>>>> >>>>>>>>>>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Dario, >>>>>>>>>>>>> >>>>>>>>>>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences. >>>>>>>>>>>>> >>>>>>>>>>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes. >>>>>>>>>>>>> >>>>>>>>>>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points: >>>>>>>>>>>>> >>>>>>>>>>>>> kill -USR1 <pid> >>>>>>>>>>>>> >>>>>>>>>>>>> to get the smalltalk stack dumped to the gem log and: >>>>>>>>>>>>> >>>>>>>>>>>>> $GEMSTONE/bin/gstack <pid> >>>>>>>>>>>>> >>>>>>>>>>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there. >>>>>>>>>>>>> >>>>>>>>>>>>> Dale >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> | From: "Dario Trussardi" <[hidden email]> >>>>>>>>>>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]> >>>>>>>>>>>>> | Sent: Friday, March 30, 2012 7:06:30 AM >>>>>>>>>>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment >>>>>>>>>>>>> | >>>>>>>>>>>>> | Ciao, >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | i have a server based on Ubuntu and Lighttpd. >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1. >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | One work with : >>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9030 >>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9031 >>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9032 >>>>>>>>>>>>> | >>>>>>>>>>>>> | and other with: >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9020 >>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9021 >>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9022 >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | All work fine for more time and the lighttpd redirect the request to >>>>>>>>>>>>> | the specific environment. >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | This is one example of lighttpd configuration: >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | $HTTP["host"] =~ "dariospizza.com$" { >>>>>>>>>>>>> | >>>>>>>>>>>>> | url.redirect = ( >>>>>>>>>>>>> | >>>>>>>>>>>>> | "^/$" => " http://www.dariospizza.com/dariospizza " >>>>>>>>>>>>> | >>>>>>>>>>>>> | ) >>>>>>>>>>>>> | >>>>>>>>>>>>> | $HTTP["url"] =~ "^/dariospizza" { >>>>>>>>>>>>> | >>>>>>>>>>>>> | fastcgi.server = ( >>>>>>>>>>>>> | >>>>>>>>>>>>> | "/dariospizza" => >>>>>>>>>>>>> | >>>>>>>>>>>>> | ( >>>>>>>>>>>>> | >>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" => >>>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>>> | >>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" => >>>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>>> | >>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" => >>>>>>>>>>>>> | "disable","mode" => "responder" ) >>>>>>>>>>>>> | >>>>>>>>>>>>> | ) >>>>>>>>>>>>> | >>>>>>>>>>>>> | ) >>>>>>>>>>>>> | >>>>>>>>>>>>> | } >>>>>>>>>>>>> | >>>>>>>>>>>>> | $HTTP["url"] =~ "^/Dariospizza/" { >>>>>>>>>>>>> | >>>>>>>>>>>>> | server.document-root = "/mnt/ISTDataBase/dbbase" >>>>>>>>>>>>> | >>>>>>>>>>>>> | } >>>>>>>>>>>>> | >>>>>>>>>>>>> | $HTTP["url"] =~ "^/filesdbbase" { >>>>>>>>>>>>> | >>>>>>>>>>>>> | fastcgi.server = ( >>>>>>>>>>>>> | >>>>>>>>>>>>> | "/filesdbbase" => >>>>>>>>>>>>> | >>>>>>>>>>>>> | ( >>>>>>>>>>>>> | >>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" => >>>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>>> | >>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" => >>>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>>> | >>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" => >>>>>>>>>>>>> | "disable","mode" => "responder" ) >>>>>>>>>>>>> | >>>>>>>>>>>>> | ) >>>>>>>>>>>>> | >>>>>>>>>>>>> | ) >>>>>>>>>>>>> | >>>>>>>>>>>>> | } >>>>>>>>>>>>> | >>>>>>>>>>>>> | } >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | But sometime the system becomes unstable . >>>>>>>>>>>>> | >>>>>>>>>>>>> | The request for FastCGI 9020 are sent to the Glass environment that >>>>>>>>>>>>> | manages 9030 port. >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | I don't understund because it and what is the original problem. >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | For solve it i need to stoop and restart the two Glass environment. >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | Any considerations, idea are welcome. >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | Thanks, >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | Dario >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>>> | >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > |
Re: Web request problem on server with two Glass environment
|
|
Johan,
The struggle continues. ;-) On Apr 16, 2012, at 2:07 PM, Johan Brichau wrote: > Larry, > > I think the monit daemon was not configured correctly. > Check the monit configuration file (/etc/monitrc but it depends on your distribution's installation) that it contains the following line: > > set httpd port 2812 and allow localhost > I put this line in the monit.conf file, it has been commented out, that is located in /etc and restarted monit. The logs mention something about starting httpd, but otherwise, there is not much difference from before. > From this file, you also need to include all directives for checking the fastcgi gems (amongst others) > > And you also need to execute the start/stop command as a specific user. In monit you can do the following by explicitly giving a uid/gid > start program = "..." as uid 500 and gid 500 I tried with 500 but monit says that 500 does not exist in the system and will not let me do it. I used root instead but it still fails to start. Should my start/stop programs be scripts? Should I include that /bash line before the commands? Like this: >>> start program = "/bin/bash -c '/opt/gemstone/product/seaside/bin/startAllGems&> /tmp/startAllGems.log' " >>> stop program = " /bin/bash -c '/opt/gemstone/product/seaside/bin/stopAllGems&> /tmp/stopAllGems.log ' " Every variation seems to fail for me. I tried /opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001 and /opt/gemstone/product/seaside/bin/runSeasideGems30 stop FastCGI 9001 which works on the command lines but not from monit. > > It's also strange that uid 500 is not a named user, actually… > Yes, it is strange. Maybe some Unix expert will explain to us what is going on, because I have no idea. Larry > Johan > > On 13 Apr 2012, at 13:30, Lawrence Kellogg wrote: > >> Johan, >> Any thoughts? I really should use monit because that Object does not exist error comes up every once in a while >> and makes the Gem unresponsive, which means my service becomes unreachable. A bummer. >> >> Thanks, >> >> Larry >> >> >> On Apr 11, 2012, at 4:59 PM, Lawrence Kellogg wrote: >> >>> >>> On Apr 11, 2012, at 11:36 AM, Johan Brichau wrote: >>> >>>> Larry, >>>> >>>> Off the top of my head: >>>> >>>> - As which user is Gemstone running? Did you also include to run the scripts as that user in the monit configuration file? >>> >>> >>> 500 1648 1 0 Apr08 ? 00:02:18 /opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/sys/stoned seaside >>> 500 1649 1648 0 Apr08 ? 00:02:12 /opt/gemstone/product/sys/shrpcmonitor 'seaside@10.189.103.87' setlocationst >>> 500 1655 1648 0 Apr08 ? 00:00:10 /opt/gemstone/product/sys/pgsvrmain seaside@10.189.103.87 0 1 -1 >>> 500 1657 1648 0 Apr08 ? 00:00:22 /opt/gemstone/product/sys/pgsvrmain TCP 60430 90 >>> 500 1665 1648 0 Apr08 ? 00:09:52 /opt/gemstone/product/sys/gem pagemanager 'seaside' -T 5000 >>> 500 1709 1648 0 Apr08 ? 00:00:13 /opt/gemstone/product/sys/gem symbolgem 'seaside' -T 10000 >>> 500 1710 1648 0 Apr08 ? 00:00:06 /opt/gemstone/product/sys/gem admingcgem 'seaside' -T 5000 >>> 500 1714 1648 0 Apr08 ? 00:01:30 /opt/gemstone/product/sys/gem reclaimgcgem 'seaside' 0 0 -T 5000 >>> 500 1729 1 0 Apr08 ? 00:00:00 /opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/sys/netldid gs64ldi -g -n -a >>> root 1762 1 0 Apr08 ? 00:00:00 nginx: master process nginx >>> nginx 1763 1762 0 Apr08 ? 00:00:01 nginx: worker process >>> >>> The kid of 500. I don't know what that means though. >>> >>> The Gems also run as 500 >>> >>> 500 14315 1 1 20:51 pts/0 00:00:00 /opt/gemstone/product/bin/topaz -l -T50000 >>> 500 14317 1 82 20:51 pts/0 00:00:03 /opt/gemstone/product/bin/topaz -l -T200000 >>> 500 14319 1 3 20:51 pts/0 00:00:00 /opt/gemstone/product/bin/topaz -l -T45000 >>> >>> Perhaps this is the problem. >>> >>> I tried this: >>> >>> start program = "/bin/bash -c '/opt/gemstone/product/seaside/bin/startAllGems&> /tmp/startAllGems.log' " >>> stop program = " /bin/bash -c '/opt/gemstone/product/seaside/bin/stopAllGems&> /tmp/stopAllGems.log ' " >>> >>> because I read here: http://mmonit.com/wiki/Monit/FAQ >>> >>> that bin/bash should be used to excute scripts. >>> >>> after I start monit, my log file shows this: >>> >>> $ cat startAllGems.log >>> Starting Gems.... >>> Starting FastCGI gem on port 9001 >>> Starting maintenance gem >>> Starting service gem >>> >>> but I don't see any Gems running and my service is not reachable. >>> >>>> - If monit is running, what is the output of 'sudo monit summary' ? >>> >>> >>> monit is running >>> >>> root 14369 1 0 20:55 ? 00:00:00 monit >>> >>> but sudo monit summary gives an error >>> >>> $ sudo monit summary >>> monit: error connecting to the monit daemon >>> >>> >>>> - what is the output of 'sudo monit start fastcgi_9001' ? >>>> >>> >>> $ sudo monit start fastcgi_9001 >>> monit: Cannot connect to the monit daemon. Did you start it with http support? >>> >>> >>>> We'll get there… >>> >>> Thanks, I know we'll figure it out. >>> >>> Larry >>> >>> >>>> >>>> Johan >>>> >>>> On 10 Apr 2012, at 17:50, Lawrence Kellogg wrote: >>>> >>>>> >>>>> On Apr 10, 2012, at 8:20 AM, Johan Brichau wrote: >>>>> >>>>>> If there's an error connecting to the monit daemon, then it's not running normally. >>>>>> >>>>>> There should be a start/stop script in /etc/init.d/ >>>>>> >>>>>> sudo /etc/init.d/monit start >>>>> >>>>> I am getting monit to start but it just says failed to start >>>>> >>>>> $ sudo /etc/init.d/monit start >>>>> Starting monit: [ OK ] >>>>> >>>>> and the log output >>>>> >>>>> [UTC Apr 10 15:19:44] info : 'system_ip-10-189-103-87.ec2.internal' Monit started >>>>> [UTC Apr 10 15:19:44] error : 'fastcgi_9001' process is not running >>>>> [UTC Apr 10 15:19:44] info : 'fastcgi_9001' trying to restart >>>>> [UTC Apr 10 15:19:44] info : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems >>>>> [UTC Apr 10 15:20:14] error : 'fastcgi_9001' failed to start >>>>> [UTC Apr 10 15:21:14] error : 'fastcgi_9001' process is not running >>>>> [UTC Apr 10 15:21:14] info : 'fastcgi_9001' trying to restart >>>>> [UTC Apr 10 15:21:14] info : 'fastcgi_9001' start: /opt/gemstone/product/seaside/bin/startAllGems >>>>> [UTC Apr 10 15:21:44] error : 'fastcgi_9001' failed to start >>>>> >>>>> my startAllGems script is just >>>>> >>>>> runSeasideGems30 start FastCGI 9001 >>>>> >>>>> and works fine when I run it from the command line. >>>>> >>>>> my stopAllGems script is just >>>>> >>>>> runSeasideGems30 stop >>>>> >>>>> I'm stuck….. >>>>> >>>>> Larry >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> After 4-5 minutes, monit actually starts and only then you can get results from 'sudo monit summary' >>>>>> >>>>>> If that really fails, look for errors in /var/log/messages (and family) >>>>>> >>>>>> On 10 Apr 2012, at 11:38, Lawrence Kellogg wrote: >>>>>> >>>>>>> >>>>>>> On Apr 10, 2012, at 2:30 AM, Johan Brichau wrote: >>>>>>> >>>>>>>> Larry, >>>>>>>> >>>>>>>> What is the output of the following command on your system? >>>>>>>> >>>>>>>> sudo monit summary >>>>>>> >>>>>>> I have also seen these messages... >>>>>>> >>>>>>> $sudo monit >>>>>>> monit daemon at 7971 awakened >>>>>>> >>>>>>> $ sudo monit summary >>>>>>> monit: error connecting to the monit daemon >>>>>>> >>>>>>> >>>>>>> Larry >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 09 Apr 2012, at 11:52, Lawrence Kellogg wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On Apr 8, 2012, at 7:39 PM, Lawrence Kellogg wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Apr 8, 2012, at 3:35 PM, Johan Brichau wrote: >>>>>>>>>> >>>>>>>>>>> Larry, >>>>>>>>>>> >>>>>>>>>>> Monit is relatively easy to set up. I think the best thing to do is stroll through http://mmonit.com/monit/ >>>>>>>>>>> If you have any questions, I'm happy to answer them. >>>>>>>>>>> >>>>>>>>>>> Below is a snippet to be included in the monit configuration files, for each of the seaside fastcgi gems you have running. It configures monit to check the process by pid _and_ send a fastcgi packet to the port when it checks the process. I just removed the part to start/stop a gem because I made some specific scripts for our purpose, but you should be able to put a command there that uses the default GLASS scripts to start/stop a gem. >>>>>>>>>>> >>>>>>>>>>> check process fastcgi_9001 with pidfile /opt/gemstone/product/seaside/data/FastCGI_server-9001.pid >>>>>>>>>>> start program = "<start gem 9001>" as uid sites and gid sites >>>>>>>>>>> stop program = "<stop gem 9001>" as uid sites and gid sites >>>>>>>>>>> # Empty FastCGI request >>>>>>>>>>> if failed port 9001 >>>>>>>>>>> # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09) >>>>>>>>>>> # padding 8 bytes (0x08), followed by 8xNULLs padding >>>>>>>>>>> send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00" >>>>>>>>>>> # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A) >>>>>>>>>>> expect "\0x01\0x0A" >>>>>>>>>>> timeout 20 seconds >>>>>>>>>>> then restart >>>>>>>>>>> >>>>>>>>> >>>>>>>>> Johan, >>>>>>>>> I used the configuration you specified above and just replaced these two lines: >>>>>>>>> >>>>>>>>> start program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001" >>>>>>>>> stop program = "/opt/gemstone/product/seaside/bin/runSeasideGems30 stop" >>>>>>>>> >>>>>>>>> but it does not seem to work. I can see that monit is running. If I kill -9 my FastCGI Gem, the system does not stop and restart all of the Gems. Also, monit does not seem to start everything up initially. What am I missing? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Larry >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks! I guess I can use this to replace my supervise process. I changed the scripts to not write to pid files with the thought that the Gem would just crash and get restarted, but then I ran into this weird case where the gem is running, but cannot respond to requests. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Larry >>>>>>>>>> >>>>>>>>>>> On 08 Apr 2012, at 16:10, Lawrence Kellogg wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Apr 8, 2012, at 4:48 AM, Johan Brichau wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Indeed, sometimes a fastcgi gem process can still be alive but will no longer accept a fastcgi request. >>>>>>>>>>>>> Therefore, we run a monit process that sends a fastcgi packet every x minutes to the gem. If it's not responding correctly, monit will restart the gem. I can recommend that to everyone. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hello Johan, >>>>>>>>>>>> I am interested in running such a monit process as you have described. Is it hard to set up? What do I have to do? Does monit kill a running Gem if it is no longer responding to requests? I use supervise to restart a fastcgi Gem if it crashes but I have this case where an "object does not exist" error is stopping fastcgi traffic, but the Gem is still up and running. I would like to have the monit process take care of this case. Is it possible? >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> Larry >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I have not yet gotten to the bottom of this problem yet because it's something that pops up relatively rare. However, I found that the unresponsiveness is probably due to the gatesemaphore being 'depleted' (i.e. 10 processes have decreased the semaphore but never increased it again). This is because I have been tracing the state of the semaphore over time, where I notice that the gatesemaphore's value can be less than 10 (which is its initial value) on an idle server. The gem would be blocked once the gatesemaphore hits 0. >>>>>>>>>>>>> >>>>>>>>>>>>> I have a suspicion that it has something to do with concurrency conflicts because I notice a correlation with the number of concurrency conflict messages in the log and the decreasing value of the gatesemaphore. But it's not really consistent and I have not seen any paths in the code that might lead to this. >>>>>>>>>>>>> >>>>>>>>>>>>> that's all I know so far... >>>>>>>>>>>>> >>>>>>>>>>>>> Johan >>>>>>>>>>>>> >>>>>>>>>>>>> On 30 Mar 2012, at 18:14, Dale Henrichs wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Dario, >>>>>>>>>>>>>> >>>>>>>>>>>>>> It is not unheard of that gems become "hung" in a state in which they haven't died, but they are no longer responding to FASTcgi requests .... I don't think we've characterized the issues, but perhaps some of the other folks on this list (Norbert?, Johan?) can share their experiences. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I would like to plug all the holes, but if you are getting non-responsive gems, then there are still some holes. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If you have a gem in the unresponsive state, we'd like to get a couple of data points: >>>>>>>>>>>>>> >>>>>>>>>>>>>> kill -USR1 <pid> >>>>>>>>>>>>>> >>>>>>>>>>>>>> to get the smalltalk stack dumped to the gem log and: >>>>>>>>>>>>>> >>>>>>>>>>>>>> $GEMSTONE/bin/gstack <pid> >>>>>>>>>>>>>> >>>>>>>>>>>>>> to get a C stack sample dumped to stdout ... both of these can give us a pretty good idea of what the gem is doing and we can go from there. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dale >>>>>>>>>>>>>> >>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> | From: "Dario Trussardi" <[hidden email]> >>>>>>>>>>>>>> | To: "beta discussion Gemstone Seaside" <[hidden email]> >>>>>>>>>>>>>> | Sent: Friday, March 30, 2012 7:06:30 AM >>>>>>>>>>>>>> | Subject: [GS/SS Beta] Web request problem on server with two Glass environment >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | Ciao, >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | i have a server based on Ubuntu and Lighttpd. >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | On it i run 2 GLASS environment based on GemStone64Bit2.4.4.1. >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | One work with : >>>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9030 >>>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9031 >>>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9032 >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | and other with: >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9020 >>>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9021 >>>>>>>>>>>>>> | Starting WAFastCGIAdaptor gem on port 9022 >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | All work fine for more time and the lighttpd redirect the request to >>>>>>>>>>>>>> | the specific environment. >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | This is one example of lighttpd configuration: >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | $HTTP["host"] =~ "dariospizza.com$" { >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | url.redirect = ( >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | "^/$" => " http://www.dariospizza.com/dariospizza " >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ) >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | $HTTP["url"] =~ "^/dariospizza" { >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | fastcgi.server = ( >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | "/dariospizza" => >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ( >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" => >>>>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" => >>>>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" => >>>>>>>>>>>>>> | "disable","mode" => "responder" ) >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ) >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ) >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | } >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | $HTTP["url"] =~ "^/Dariospizza/" { >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | server.document-root = "/mnt/ISTDataBase/dbbase" >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | } >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | $HTTP["url"] =~ "^/filesdbbase" { >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | fastcgi.server = ( >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | "/filesdbbase" => >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ( >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9020,"check-local" => >>>>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9021,"check-local" => >>>>>>>>>>>>>> | "disable","mode" => "responder" ), >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ("host" => "192.168.1.100","port" => 9022,"check-local" => >>>>>>>>>>>>>> | "disable","mode" => "responder" ) >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ) >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | ) >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | } >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | } >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | But sometime the system becomes unstable . >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | The request for FastCGI 9020 are sent to the Glass environment that >>>>>>>>>>>>>> | manages 9030 port. >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | I don't understund because it and what is the original problem. >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | For solve it i need to stoop and restart the two Glass environment. >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | Any considerations, idea are welcome. >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | Thanks, >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | Dario >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>>> | >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > |
Re: Web request problem on server with two Glass environment
|
|
On 17 Apr 2012, at 01:29, Lawrence Kellogg wrote: >> set httpd port 2812 and allow localhost >> > > I put this line in the monit.conf file, it has been commented out, that is located in /etc and restarted monit. The logs mention something about starting httpd, but > otherwise, there is not much difference from before. Aha! I'm pretty sure that is why you were not getting any output from 'sudo monit summary'. Can you tell me what it says now this has been changed? For the start/stop scripts: I suspect that the environment where monit launches the command does not include the environment variables needed by Gemstone. Therefore, make sure that the scripts that are launched by monit to start/stop are as follows: #!/bin/bash #Import Gemstone env variables source /opt/gemstone/product/seaside/defSeaside #start the fastcgi gem on port $1 /opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI $1 Let me know how it goes Johan |
Re: Web request problem on server with two Glass environment
|
|
On Apr 17, 2012, at 3:59 AM, Johan Brichau wrote: > > On 17 Apr 2012, at 01:29, Lawrence Kellogg wrote: > >>> set httpd port 2812 and allow localhost >>> >> >> I put this line in the monit.conf file, it has been commented out, that is located in /etc and restarted monit. The logs mention something about starting httpd, but >> otherwise, there is not much difference from before. > > Aha! I'm pretty sure that is why you were not getting any output from 'sudo monit summary'. > Can you tell me what it says now this has been changed? > > For the start/stop scripts: I suspect that the environment where monit launches the command does not include the environment variables needed by Gemstone. Therefore, make sure that the scripts that are launched by monit to start/stop are as follows: > > #!/bin/bash > > #Import Gemstone env variables > source /opt/gemstone/product/seaside/defSeaside > > #start the fastcgi gem on port $1 > /opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI $1 > > Eureka! Yes, I believe that the main problem was that the Gemstone environment variables were not defined, but I also believe there is a problem with the pid file hanging around that prevented the start command from working correctly. I was under the mistaken impression that monit would run the stop command and then start command if there was some failure , but it only runs the start command, as far as I can tell. So, I put a stop command before the start command and now everything seems to work correctly. Now, I can go see a movie and not worry so much that my service has crashed. > /opt/gemstone/product/seaside/bin/runSeasideGems30 stop FastCGI 9001 > /opt/gemstone/product/seaside/bin/runSeasideGems30 start FastCGI 9001 Here is the summary output: sudo monit summary The Monit daemon 5.2.5 uptime: 7m Process 'fastcgi_9001' running Thanks so much for your help! Larry > Let me know how it goes > Johan |
Re: Web request problem on server with two Glass environment
|
|
Larry,
Great news ;-) Don't forget to monitor the other fastcgi gems as well (I'm assuming you are running 3 fastcgi gems) Similarly, the maintenance gem is important and I also monitor the size of the extent using monit. On 17 Apr 2012, at 13:50, Lawrence Kellogg wrote: > sudo monit summary > The Monit daemon 5.2.5 uptime: 7m > > Process 'fastcgi_9001' running |
«
Return to GLASS
|
1 view|%1 views
| Free forum by Nabble | Edit this page |