Mysterious gem death
Locked
 
|
Hi everyone,
I've been struggling with this problem for weeks. My seaside gems die and I find NO LOGS at all to see what's going on :( This happens when I evaluate some big reports. The amount of objects involved is not really that big, but the logic is a little complex. I'm suspecting that maybe a stack overflow is happening but for any reason it's not being logged? Could that be possible? I already configured the stack limit to the maximum, because I had stack limit problems before. I also raised the timeout of the seaside request in the apache config (it's set to one hour and the gems die within seconds). At some point I had temporal memory issues, but I raised the tempobj_cache_size and now I don't get memory problems in my logs. I'm planning to execute these reports using topaz scritps instead of seaside requests, but if it's a problem with the stack limit or even my design I don't think that could help... Do you know any cases where these "mysterious gem death" might occur, with no logs at all? Thanks! Carla. |
|
|
Carla,
Normally the log file is deleted if a gem exits with no error...so gem logs created by running GemTools won't stay around unless the gem exits with an error condition. On the other hand in the runSeaside scripts, the log file is created by dumping stdout to a file, so the log file shouldn't just disappear.....By default the log files are put into /opt/gemstone, but depending upon how you have configured/installed things the default location can change ... So I am curious how you are running seaside in the first place so that we might be able to track down the location of the log files and then find out what's happening... For example can you see the log files from normal runs? Anyway, I think we can track this one down ... Dale On Mar 23, 2011, at 8:33 AM, Carla F. Griggio wrote: > Hi everyone, > > I've been struggling with this problem for weeks. My seaside gems die and I find NO LOGS at all to see what's going on :( > This happens when I evaluate some big reports. The amount of objects involved is not really that big, but the logic is a little complex. I'm suspecting that maybe a stack overflow is happening but for any reason it's not being logged? Could that be possible? > > I already configured the stack limit to the maximum, because I had stack limit problems before. > I also raised the timeout of the seaside request in the apache config (it's set to one hour and the gems die within seconds). > At some point I had temporal memory issues, but I raised the tempobj_cache_size and now I don't get memory problems in my logs. > I'm planning to execute these reports using topaz scritps instead of seaside requests, but if it's a problem with the stack limit or even my design I don't think that could help... > > Do you know any cases where these "mysterious gem death" might occur, with no logs at all? > > Thanks! > Carla. > > |
|
|
Hi Dale,
Sorry, I meant something else. I have the seaside gems log files, but when the gems die they don't log anything special about anything that might have happened to make them die. When a seaside gem dies, I'm used to find in the logs that it has occurred a memory overflow, some lock or write conflict, a stack overflow, etc., but in this specific case my seaside requests ends with an Internal Server Error and I can't find any logs that help me understand what happened. Cheers, Carla On Wed, Mar 23, 2011 at 2:05 PM, Dale Henrichs <[hidden email]> wrote: Carla, |
|
|
Carla,
Okay, so this is odd that the gem is disappearing without a trace ... I _would_ expect some sort of logging in the log file fo rmost of the errors ... is there anything at all in the log file ... exit message or logout or is the log just chopped off at process termination? Dale On Mar 23, 2011, at 11:19 AM, Carla F. Griggio wrote: Hi Dale, Sorry, I meant something else. I have the seaside gems log files, but when the gems die they don't log anything special about anything that might have happened to make them die. When a seaside gem dies, I'm used to find in the logs that it has occurred a memory overflow, some lock or write conflict, a stack overflow, etc., but in this specific case my seaside requests ends with an Internal Server Error and I can't find any logs that help me understand what happened. Cheers, Carla On Wed, Mar 23, 2011 at 2:05 PM, Dale Henrichs <[hidden email]<mailto:[hidden email]>> wrote: Carla, Normally the log file is deleted if a gem exits with no error...so gem logs created by running GemTools won't stay around unless the gem exits with an error condition. On the other hand in the runSeaside scripts, the log file is created by dumping stdout to a file, so the log file shouldn't just disappear.....By default the log files are put into /opt/gemstone, but depending upon how you have configured/installed things the default location can change ... So I am curious how you are running seaside in the first place so that we might be able to track down the location of the log files and then find out what's happening... For example can you see the log files from normal runs? Anyway, I think we can track this one down ... Dale On Mar 23, 2011, at 8:33 AM, Carla F. Griggio wrote: > Hi everyone, > > I've been struggling with this problem for weeks. My seaside gems die and I find NO LOGS at all to see what's going on :( > This happens when I evaluate some big reports. The amount of objects involved is not really that big, but the logic is a little complex. I'm suspecting that maybe a stack overflow is happening but for any reason it's not being logged? Could that be possible? > > I already configured the stack limit to the maximum, because I had stack limit problems before. > I also raised the timeout of the seaside request in the apache config (it's set to one hour and the gems die within seconds). > At some point I had temporal memory issues, but I raised the tempobj_cache_size and now I don't get memory problems in my logs. > I'm planning to execute these reports using topaz scritps instead of seaside requests, but if it's a problem with the stack limit or even my design I don't think that could help... > > Do you know any cases where these "mysterious gem death" might occur, with no logs at all? > > Thanks! > Carla. > > |
|
|
In reply to this post by Carla F. Griggio
Literally dies or stops responding to requests?
On Wed, Mar 23, 2011 at 2:19 PM, Carla F. Griggio <[hidden email]> wrote: > Hi Dale, > > Sorry, I meant something else. I have the seaside gems log files, but when > the gems die they don't log anything special about anything that might have > happened to make them die. > When a seaside gem dies, I'm used to find in the logs that it has occurred a > memory overflow, some lock or write conflict, a stack overflow, etc., but in > this specific case my seaside requests ends with an Internal Server Error > and I can't find any logs that help me understand what happened. > > Cheers, > Carla > > > > On Wed, Mar 23, 2011 at 2:05 PM, Dale Henrichs <[hidden email]> wrote: >> >> Carla, >> >> Normally the log file is deleted if a gem exits with no error...so gem >> logs created by running GemTools won't stay around unless the gem exits with >> an error condition. >> >> On the other hand in the runSeaside scripts, the log file is created by >> dumping stdout to a file, so the log file shouldn't just disappear.....By >> default the log files are put into /opt/gemstone, but depending upon how you >> have configured/installed things the default location can change ... >> >> So I am curious how you are running seaside in the first place so that we >> might be able to track down the location of the log files and then find out >> what's happening... >> >> For example can you see the log files from normal runs? >> >> Anyway, I think we can track this one down ... >> >> Dale >> >> On Mar 23, 2011, at 8:33 AM, Carla F. Griggio wrote: >> >> > Hi everyone, >> > >> > I've been struggling with this problem for weeks. My seaside gems die >> > and I find NO LOGS at all to see what's going on :( >> > This happens when I evaluate some big reports. The amount of objects >> > involved is not really that big, but the logic is a little complex. I'm >> > suspecting that maybe a stack overflow is happening but for any reason it's >> > not being logged? Could that be possible? >> > >> > I already configured the stack limit to the maximum, because I had stack >> > limit problems before. >> > I also raised the timeout of the seaside request in the apache config >> > (it's set to one hour and the gems die within seconds). >> > At some point I had temporal memory issues, but I raised the >> > tempobj_cache_size and now I don't get memory problems in my logs. >> > I'm planning to execute these reports using topaz scritps instead of >> > seaside requests, but if it's a problem with the stack limit or even my >> > design I don't think that could help... >> > >> > Do you know any cases where these "mysterious gem death" might occur, >> > with no logs at all? >> > >> > Thanks! >> > Carla. >> > >> > >> > > |
|
|
Hi!
Well, I had forgot that I had a proccess running that checks if the gems are up, and if they are down it starts them again, so I was asuming that everytime I got the Internal Server Error the gem had died but in a matter of seconds it was up again because of that process... but now that you mention it, I checked and that's not what's happening. The gem doesn't die, but Seaside request is stopped and I get Internal Server Error. If I refresh, it responds normally. I'll give you the apache log in a couple of minutes. On Wed, Mar 23, 2011 at 3:38 PM, Sean Allen <[hidden email]> wrote: Literally dies or stops responding to requests? |
|
|
This log is from /opt/gemstone/apache/logs/glass-vhost-8081-error.log
[Wed Mar 23 13:27:08 2011] [error] [client 127.0.0.1] FastCGI: incomplete headers (0 bytes) received from server "/opt/gemstone/apache/htdocs/glass1/seaside" [Wed Mar 23 13:29:09 2011] [error] [client 127.0.0.1] (111)Connection refused: FastCGI: failed to connect to server "/opt/gemstone/apache/htdocs/glass1/seaside": connect() failed, referer: http://10.50.6.200/seaside/SueldosEducacionGlass?_s=ERIE30uQRU_WEiZe&_k=YxG2Zllr [Wed Mar 23 13:29:09 2011] [error] [client 127.0.0.1] FastCGI: incomplete headers (0 bytes) received from server "/opt/gemstone/apache/htdocs/glass1/seaside", referer: http://10.50.6.200/seaside/SueldosEducacionGlass?_s=ERIE30uQRU_WEiZe&_k=YxG2Zllr [Wed Mar 23 13:38:50 2011] [error] [client 127.0.0.1] (111)Connection refused: FastCGI: failed to connect to server "/opt/gemstone/apache/htdocs/glass1/seaside": connect() failed [Wed Mar 23 13:38:50 2011] [error] [client 127.0.0.1] FastCGI: incomplete headers (0 bytes) received from server "/opt/gemstone/apache/htdocs/glass1/seaside" On Wed, Mar 23, 2011 at 5:26 PM, Carla F. Griggio <[hidden email]> wrote: Hi! |
|
|
And this is from glass-error.log
[Wed Mar 23 16:08:07 2011] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network [Wed Mar 23 16:08:07 2011] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network [Wed Mar 23 16:11:03 2011] [error] [client 168.83.88.150] proxy: error reading status line from remote server localhost [Wed Mar 23 16:11:03 2011] [error] [client 168.83.88.150] proxy: Error reading from remote server returned by /seaside/SueldosEducacionGlass [Wed Mar 23 16:13:54 2011] [info] [client 168.83.86.123] (104)Connection reset by peer: core_output_filter: writing data to the network [Wed Mar 23 16:13:54 2011] [info] [client 168.83.86.123] (32)Broken pipe: core_output_filter: writing data to the network On Wed, Mar 23, 2011 at 5:31 PM, Carla F. Griggio <[hidden email]> wrote: This log is from /opt/gemstone/apache/logs/glass-vhost-8081-error.log |
|
|
Are you using fast cgi or swazoo?
On Wed, Mar 23, 2011 at 4:45 PM, Carla F. Griggio <[hidden email]> wrote: > And this is from glass-error.log > > [Wed Mar 23 16:08:07 2011] [info] [client 127.0.0.1] (32)Broken pipe: > core_output_filter: writing data to the network > [Wed Mar 23 16:08:07 2011] [info] [client 127.0.0.1] (32)Broken pipe: > core_output_filter: writing data to the network > [Wed Mar 23 16:11:03 2011] [error] [client 168.83.88.150] proxy: error > reading status line from remote server localhost > [Wed Mar 23 16:11:03 2011] [error] [client 168.83.88.150] proxy: Error > reading from remote server returned by /seaside/SueldosEducacionGlass > [Wed Mar 23 16:13:54 2011] [info] [client 168.83.86.123] (104)Connection > reset by peer: core_output_filter: writing data to the network > [Wed Mar 23 16:13:54 2011] [info] [client 168.83.86.123] (32)Broken pipe: > core_output_filter: writing data to the network > > > On Wed, Mar 23, 2011 at 5:31 PM, Carla F. Griggio <[hidden email]> > wrote: >> >> This log is from /opt/gemstone/apache/logs/glass-vhost-8081-error.log >> >> [Wed Mar 23 13:27:08 2011] [error] [client 127.0.0.1] FastCGI: incomplete >> headers (0 bytes) received from server >> "/opt/gemstone/apache/htdocs/glass1/seaside" >> [Wed Mar 23 13:29:09 2011] [error] [client 127.0.0.1] (111)Connection >> refused: FastCGI: failed to connect to server >> "/opt/gemstone/apache/htdocs/glass1/seaside": connect() failed, referer: >> http://10.50.6.200/seaside/SueldosEducacionGlass?_s=ERIE30uQRU_WEiZe&_k=YxG2Zllr >> [Wed Mar 23 13:29:09 2011] [error] [client 127.0.0.1] FastCGI: incomplete >> headers (0 bytes) received from server >> "/opt/gemstone/apache/htdocs/glass1/seaside", referer: >> http://10.50.6.200/seaside/SueldosEducacionGlass?_s=ERIE30uQRU_WEiZe&_k=YxG2Zllr >> [Wed Mar 23 13:38:50 2011] [error] [client 127.0.0.1] (111)Connection >> refused: FastCGI: failed to connect to server >> "/opt/gemstone/apache/htdocs/glass1/seaside": connect() failed >> [Wed Mar 23 13:38:50 2011] [error] [client 127.0.0.1] FastCGI: incomplete >> headers (0 bytes) received from server >> "/opt/gemstone/apache/htdocs/glass1/seaside" >> >> >> On Wed, Mar 23, 2011 at 5:26 PM, Carla F. Griggio >> <[hidden email]> wrote: >>> >>> Hi! >>> Well, I had forgot that I had a proccess running that checks if the gems >>> are up, and if they are down it starts them again, so I was asuming that >>> everytime I got the Internal Server Error the gem had died but in a matter >>> of seconds it was up again because of that process... but now that you >>> mention it, I checked and that's not what's happening. The gem doesn't die, >>> but Seaside request is stopped and I get Internal Server Error. If I >>> refresh, it responds normally. >>> >>> I'll give you the apache log in a couple of minutes. >>> >>> >>> >>> On Wed, Mar 23, 2011 at 3:38 PM, Sean Allen <[hidden email]> >>> wrote: >>>> >>>> Literally dies or stops responding to requests? >>>> >>>> On Wed, Mar 23, 2011 at 2:19 PM, Carla F. Griggio >>>> <[hidden email]> wrote: >>>> > Hi Dale, >>>> > >>>> > Sorry, I meant something else. I have the seaside gems log files, but >>>> > when >>>> > the gems die they don't log anything special about anything that might >>>> > have >>>> > happened to make them die. >>>> > When a seaside gem dies, I'm used to find in the logs that it has >>>> > occurred a >>>> > memory overflow, some lock or write conflict, a stack overflow, etc., >>>> > but in >>>> > this specific case my seaside requests ends with an Internal Server >>>> > Error >>>> > and I can't find any logs that help me understand what happened. >>>> > >>>> > Cheers, >>>> > Carla >>>> > >>>> > >>>> > >>>> > On Wed, Mar 23, 2011 at 2:05 PM, Dale Henrichs <[hidden email]> >>>> > wrote: >>>> >> >>>> >> Carla, >>>> >> >>>> >> Normally the log file is deleted if a gem exits with no error...so >>>> >> gem >>>> >> logs created by running GemTools won't stay around unless the gem >>>> >> exits with >>>> >> an error condition. >>>> >> >>>> >> On the other hand in the runSeaside scripts, the log file is created >>>> >> by >>>> >> dumping stdout to a file, so the log file shouldn't just >>>> >> disappear.....By >>>> >> default the log files are put into /opt/gemstone, but depending upon >>>> >> how you >>>> >> have configured/installed things the default location can change ... >>>> >> >>>> >> So I am curious how you are running seaside in the first place so >>>> >> that we >>>> >> might be able to track down the location of the log files and then >>>> >> find out >>>> >> what's happening... >>>> >> >>>> >> For example can you see the log files from normal runs? >>>> >> >>>> >> Anyway, I think we can track this one down ... >>>> >> >>>> >> Dale >>>> >> >>>> >> On Mar 23, 2011, at 8:33 AM, Carla F. Griggio wrote: >>>> >> >>>> >> > Hi everyone, >>>> >> > >>>> >> > I've been struggling with this problem for weeks. My seaside gems >>>> >> > die >>>> >> > and I find NO LOGS at all to see what's going on :( >>>> >> > This happens when I evaluate some big reports. The amount of >>>> >> > objects >>>> >> > involved is not really that big, but the logic is a little complex. >>>> >> > I'm >>>> >> > suspecting that maybe a stack overflow is happening but for any >>>> >> > reason it's >>>> >> > not being logged? Could that be possible? >>>> >> > >>>> >> > I already configured the stack limit to the maximum, because I had >>>> >> > stack >>>> >> > limit problems before. >>>> >> > I also raised the timeout of the seaside request in the apache >>>> >> > config >>>> >> > (it's set to one hour and the gems die within seconds). >>>> >> > At some point I had temporal memory issues, but I raised the >>>> >> > tempobj_cache_size and now I don't get memory problems in my logs. >>>> >> > I'm planning to execute these reports using topaz scritps instead >>>> >> > of >>>> >> > seaside requests, but if it's a problem with the stack limit or >>>> >> > even my >>>> >> > design I don't think that could help... >>>> >> > >>>> >> > Do you know any cases where these "mysterious gem death" might >>>> >> > occur, >>>> >> > with no logs at all? >>>> >> > >>>> >> > Thanks! >>>> >> > Carla. >>>> >> > >>>> >> > >>>> >> >>>> > >>>> > >>> >> > > |
|
|
Fast CGI
On Wed, Mar 23, 2011 at 6:15 PM, Sean Allen <[hidden email]> wrote: Are you using fast cgi or swazoo? |
|
|
In reply to this post by Carla F. Griggio
So it seems that the processes are continuing to run, so there should be errors recorded somewhere ....I would like to see the gem logs...also have you looked for debug continuations (using the Debug button in GemTools) or looked for errors in the Object Log (using the oobect log component)?
Dale On Mar 23, 2011, at 1:45 PM, Carla F. Griggio wrote: And this is from glass-error.log [Wed Mar 23 16:08:07 2011] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network [Wed Mar 23 16:08:07 2011] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network [Wed Mar 23 16:11:03 2011] [error] [client 168.83.88.150] proxy: error reading status line from remote server localhost [Wed Mar 23 16:11:03 2011] [error] [client 168.83.88.150] proxy: Error reading from remote server returned by /seaside/SueldosEducacionGlass [Wed Mar 23 16:13:54 2011] [info] [client 168.83.86.123] (104)Connection reset by peer: core_output_filter: writing data to the network [Wed Mar 23 16:13:54 2011] [info] [client 168.83.86.123] (32)Broken pipe: core_output_filter: writing data to the network On Wed, Mar 23, 2011 at 5:31 PM, Carla F. Griggio <[hidden email]<mailto:[hidden email]>> wrote: This log is from /opt/gemstone/apache/logs/glass-vhost-8081-error.log [Wed Mar 23 13:27:08 2011] [error] [client 127.0.0.1] FastCGI: incomplete headers (0 bytes) received from server "/opt/gemstone/apache/htdocs/glass1/seaside" [Wed Mar 23 13:29:09 2011] [error] [client 127.0.0.1] (111)Connection refused: FastCGI: failed to connect to server "/opt/gemstone/apache/htdocs/glass1/seaside": connect() failed, referer: http://10.50.6.200/seaside/SueldosEducacionGlass?_s=ERIE30uQRU_WEiZe&_k=YxG2Zllr [Wed Mar 23 13:29:09 2011] [error] [client 127.0.0.1] FastCGI: incomplete headers (0 bytes) received from server "/opt/gemstone/apache/htdocs/glass1/seaside", referer: http://10.50.6.200/seaside/SueldosEducacionGlass?_s=ERIE30uQRU_WEiZe&_k=YxG2Zllr [Wed Mar 23 13:38:50 2011] [error] [client 127.0.0.1] (111)Connection refused: FastCGI: failed to connect to server "/opt/gemstone/apache/htdocs/glass1/seaside": connect() failed [Wed Mar 23 13:38:50 2011] [error] [client 127.0.0.1] FastCGI: incomplete headers (0 bytes) received from server "/opt/gemstone/apache/htdocs/glass1/seaside" On Wed, Mar 23, 2011 at 5:26 PM, Carla F. Griggio <[hidden email]<mailto:[hidden email]>> wrote: Hi! Well, I had forgot that I had a proccess running that checks if the gems are up, and if they are down it starts them again, so I was asuming that everytime I got the Internal Server Error the gem had died but in a matter of seconds it was up again because of that process... but now that you mention it, I checked and that's not what's happening. The gem doesn't die, but Seaside request is stopped and I get Internal Server Error. If I refresh, it responds normally. I'll give you the apache log in a couple of minutes. On Wed, Mar 23, 2011 at 3:38 PM, Sean Allen <[hidden email]<mailto:[hidden email]>> wrote: Literally dies or stops responding to requests? On Wed, Mar 23, 2011 at 2:19 PM, Carla F. Griggio <[hidden email]<mailto:[hidden email]>> wrote: > Hi Dale, > > Sorry, I meant something else. I have the seaside gems log files, but when > the gems die they don't log anything special about anything that might have > happened to make them die. > When a seaside gem dies, I'm used to find in the logs that it has occurred a > memory overflow, some lock or write conflict, a stack overflow, etc., but in > this specific case my seaside requests ends with an Internal Server Error > and I can't find any logs that help me understand what happened. > > Cheers, > Carla > > > > On Wed, Mar 23, 2011 at 2:05 PM, Dale Henrichs <[hidden email]<mailto:[hidden email]>> wrote: >> >> Carla, >> >> Normally the log file is deleted if a gem exits with no error...so gem >> logs created by running GemTools won't stay around unless the gem exits with >> an error condition. >> >> On the other hand in the runSeaside scripts, the log file is created by >> dumping stdout to a file, so the log file shouldn't just disappear.....By >> default the log files are put into /opt/gemstone, but depending upon how you >> have configured/installed things the default location can change ... >> >> So I am curious how you are running seaside in the first place so that we >> might be able to track down the location of the log files and then find out >> what's happening... >> >> For example can you see the log files from normal runs? >> >> Anyway, I think we can track this one down ... >> >> Dale >> >> On Mar 23, 2011, at 8:33 AM, Carla F. Griggio wrote: >> >> > Hi everyone, >> > >> > I've been struggling with this problem for weeks. My seaside gems die >> > and I find NO LOGS at all to see what's going on :( >> > This happens when I evaluate some big reports. The amount of objects >> > involved is not really that big, but the logic is a little complex. I'm >> > suspecting that maybe a stack overflow is happening but for any reason it's >> > not being logged? Could that be possible? >> > >> > I already configured the stack limit to the maximum, because I had stack >> > limit problems before. >> > I also raised the timeout of the seaside request in the apache config >> > (it's set to one hour and the gems die within seconds). >> > At some point I had temporal memory issues, but I raised the >> > tempobj_cache_size and now I don't get memory problems in my logs. >> > I'm planning to execute these reports using topaz scritps instead of >> > seaside requests, but if it's a problem with the stack limit or even my >> > design I don't think that could help... >> > >> > Do you know any cases where these "mysterious gem death" might occur, >> > with no logs at all? >> > >> > Thanks! >> > Carla. >> > >> > >> > > |
|
|
I didn't find debug continuations, but I checked the gemnetobject logs at /home/glass and found this (after a long memory usage report):
GemStone: Error Fatal VM temporary object memory is full , perm space overflow Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20 Arg 1: 20 [Info]: Logging out at 03/17/11 16:35:50 ART ***************************************************** ****** GemStone Abnormal Shutdown at 03/17/11 16:35:51 ART ***************************************************** ----------------------------------------------------- GemStone: Error Fatal VM temporary object memory is full , perm space overflow Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20 Arg 1: 20 I solved the problem by executing these reports with a topaz script, and I assign more TEMPOBJ_CACHE_SIZE to the gem. Now I know yet another place where I can find logs :P Thanks! Carla. On Wed, Mar 23, 2011 at 6:56 PM, Dale Henrichs <[hidden email]> wrote: So it seems that the processes are continuing to run, so there should be errors recorded somewhere ....I would like to see the gem logs...also have you looked for debug continuations (using the Debug button in GemTools) or looked for errors in the Object Log (using the oobect log component)? |
«
Return to GLASS
|
1 view|%1 views
| Free forum by Nabble | Edit this page |