HIGH CPU - not frozen, but s...l....o....w seaside image

> After a while, it went down to about 45% and has slowed to a whopping
> but steady 40%. So, I'm just going to wait until it settles down.
>
> In the meantime, I'd like to find out what was going on. This is a
> remote, headless seaside image (latest production) and 3.9 Final.
>
> Any advice is much appreciated!
>
> thanks,
> brad
>
>
>
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

> 2007/3/16, Brad Fuller <[hidden email]>:
>> I have an image that seemed to not respond, and was running about 99%
>> (from top).  I tried ATL+. to see if that would help from VNC, but isn't
>> that just for UI issues?
>
> No
>
>> What are good ways to found out what is
>> happening to an image that is taking up all CPU time?
>
> Process explorer or gdb.
>
>> After a while, it went down to about 45% and has slowed to a whopping
>> but steady 40%. So, I'm just going to wait until it settles down.
>>
>> In the meantime, I'd like to find out what was going on. This is a
>> remote, headless seaside image (latest production) and 3.9 Final.
>>
>> Any advice is much appreciated!
>>
>> thanks,
>> brad
>>
>>
>>
>> _______________________________________________
>> Seaside mailing list
>> [hidden email]
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

> I have almost the same problem...
>
> I am running Squeak3.9 with Seaside2.7a1-mb.205. I use squeak as a  
> transparent persistant object database and save the image every  
> 12hours. Everything works fine.
>
> BUT after some days, sometimes two days, sometimes one week, the  
> image finish to hang strangely : everything works fine in the  
> seaside app, you can use the application normally perhaps a little  
> slower than usual, but it could be my network. But any attempt to  
> save the image, or update the code via monticello, or use a  
> SmartRefStream to save a part of the image using a Seaside app just  
> makes my browser load until timeout (with WAVersionUploader for  
> example). If I check the process using top, I just see Squeak at  
> 99.9% CPU. If I use the WAScreenshot app to look at the UI, there  
> is no debugger, no error, nothing out of the ordinary...
>
> This is very bad because when it ends like this, I have no choice  
> to kill squeak and loose all the data since the last save of the  
> image. And when I restart with the last save, everything is ok ant  
> squeak is quick and very responsive. Until the next times it hangs...
>
> Every 12 hours I 'clean' the image before saving it with this code :
> WARegistry clearAllHandlers.
> MCFileBasedRepository flushAllCaches.
> Smalltalk garbageCollect.
>
> I run this code and the save of the image inside an on:do: Error  
> block with log of the stack in case of an Error, but I get nothing.
>
> I don't know what to add to be sure the image will stay healthy.  
> Should I save then kill  and restart the image each night ?
>
> How can I track or inspect or debug the image in such a state to  
> find out what is happening?
>
> Thanks in advance for any suggestion, I am a little lost with no  
> clue there.
>
> Florian
>
> Philippe Marschall wrote:
>> 2007/3/16, Brad Fuller <[hidden email]>:
>>> I have an image that seemed to not respond, and was running about  
>>> 99%
>>> (from top).  I tried ATL+. to see if that would help from VNC,  
>>> but isn't
>>> that just for UI issues?
>> No
>>> What are good ways to found out what is
>>> happening to an image that is taking up all CPU time?
>> Process explorer or gdb.
>>> After a while, it went down to about 45% and has slowed to a  
>>> whopping
>>> but steady 40%. So, I'm just going to wait until it settles down.
>>>
>>> In the meantime, I'd like to find out what was going on. This is a
>>> remote, headless seaside image (latest production) and 3.9 Final.
>>>
>>> Any advice is much appreciated!
>>>
>>> thanks,
>>> brad
>>>
>>>
>>>
>>> _______________________________________________
>>> Seaside mailing list
>>> [hidden email]
>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>
>> _______________________________________________
>> Seaside mailing list
>> [hidden email]
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> I guess, the UI is frozen and alt-. does not work anymore (or you run
> headless)...
>
> Then, I suggest to printAllStacks(), printCurrentStack() (or similar),
> and do a backtrace with gdb when the image is in a bad state. This
> should give you more datapoints.
>
> Please keep us up to date or ask for more help if necessary.
>
> Adrian
>
> Btw., any unusual high memory consumption of such an image?
>
>
> On Mar 19, 2007, at 13:29 , Florian Minjat wrote:
>
>> I have almost the same problem...
>>
>> I am running Squeak3.9 with Seaside2.7a1-mb.205. I use squeak as a
>> transparent persistant object database and save the image every
>> 12hours. Everything works fine.
>>
>> BUT after some days, sometimes two days, sometimes one week, the image
>> finish to hang strangely : everything works fine in the seaside app,
>> you can use the application normally perhaps a little slower than
>> usual, but it could be my network. But any attempt to save the image,
>> or update the code via monticello, or use a SmartRefStream to save a
>> part of the image using a Seaside app just makes my browser load until
>> timeout (with WAVersionUploader for example). If I check the process
>> using top, I just see Squeak at 99.9% CPU. If I use the WAScreenshot
>> app to look at the UI, there is no debugger, no error, nothing out of
>> the ordinary...
>>
>> This is very bad because when it ends like this, I have no choice to
>> kill squeak and loose all the data since the last save of the image.
>> And when I restart with the last save, everything is ok ant squeak is
>> quick and very responsive. Until the next times it hangs...
>>
>> Every 12 hours I 'clean' the image before saving it with this code :
>>     WARegistry clearAllHandlers.
>>     MCFileBasedRepository flushAllCaches.
>>     Smalltalk garbageCollect.
>>
>> I run this code and the save of the image inside an on:do: Error block
>> with log of the stack in case of an Error, but I get nothing.
>>
>> I don't know what to add to be sure the image will stay healthy.
>> Should I save then kill  and restart the image each night ?
>>
>> How can I track or inspect or debug the image in such a state to find
>> out what is happening?
>>
>> Thanks in advance for any suggestion, I am a little lost with no clue
>> there.
>>
>> Florian
>>
>> Philippe Marschall wrote:
>>> 2007/3/16, Brad Fuller <[hidden email]>:
>>>> I have an image that seemed to not respond, and was running about 99%
>>>> (from top).  I tried ATL+. to see if that would help from VNC, but
>>>> isn't
>>>> that just for UI issues?
>>> No
>>>> What are good ways to found out what is
>>>> happening to an image that is taking up all CPU time?
>>> Process explorer or gdb.
>>>> After a while, it went down to about 45% and has slowed to a whopping
>>>> but steady 40%. So, I'm just going to wait until it settles down.
>>>>
>>>> In the meantime, I'd like to find out what was going on. This is a
>>>> remote, headless seaside image (latest production) and 3.9 Final.
>>>>
>>>> Any advice is much appreciated!
>>>>
>>>> thanks,
>>>> brad
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Seaside mailing list
>>>> [hidden email]
>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>>
>>> _______________________________________________
>>> Seaside mailing list
>>> [hidden email]
>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>> _______________________________________________
>> Seaside mailing list
>> [hidden email]
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

> Florian Minjat writes:
>> I didn't manage to get information from my image last time. So I just
>> killed it and launched it with nohup to be sure I can debug it  
>> next times.
>
> If you can debug with gdb, get the C backtrace too.
>
> Bryce
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> You can of course invoke call (int) printAllStacks()
> that should print all the stacks for all the smalltalk processes, which
> is great for giving clues.
>
> On Mar 21, 2007, at 12:08 PM, <[hidden email]>
> <[hidden email]> wrote:
>
>> Florian Minjat writes:
>>> I didn't manage to get information from my image last time. So I just
>>> killed it and launched it with nohup to be sure I can debug it next
>>> times.
>>
>> If you can debug with gdb, get the C backtrace too.
>>
>> Bryce
>> _______________________________________________
>> Seaside mailing list
>> [hidden email]
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>

> That's what I did each times.
> But the first time, with the frozen image without nohup, I didn't  
> get anything with gdb more than this :
> (gdb) call printAllStacks()
> $1 = 2
> (gdb) call printCallStack()
> $2 = -1551081468
> With no output.
> I tried with the advice of Adrian to redirect stdout and stderr  
> inside gdb (http://developer.apple.com/technotes/tn/tn2032.html),  
> but with no success.
>
> So I killed it and relauched with nohup, and when it froze again I  
> got the output I wanted in the nohup.out file.
>
> But the problem is : why did the headless image froze when it got a  
> call to InputSensor>cursorPoint ? There should be a safety in this  
> method to answer 0@0 when the image is headless for example.
> Another problem is that is worked a little (50-70 calls to  
> InputSensor>cursorPoint) and then the image was to slow (100% cpu)  
> to do it. So this behavior could be triggered somewhere else in the  
> image and is annoying to debug...
>
> Florian
>
>
> John M McIntosh wrote:
>> You can of course invoke call (int) printAllStacks()
>> that should print all the stacks for all the smalltalk processes,  
>> which is great for giving clues.
>> On Mar 21, 2007, at 12:08 PM, <[hidden email]>  
>> <[hidden email]> wrote:
>>> Florian Minjat writes:
>>>> I didn't manage to get information from my image last time. So I  
>>>> just
>>>> killed it and launched it with nohup to be sure I can debug it  
>>>> next times.
>>>
>>> If you can debug with gdb, get the C backtrace too.
>>>
>>> Bryce
>>> _______________________________________________
>>> Seaside mailing list
>>> [hidden email]
>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>> --
>> =====================================================================
>> ======
>> John M. McIntosh <[hidden email]>
>> Corporate Smalltalk Consulting Ltd.  http://
>> www.smalltalkconsulting.com
>> =====================================================================
>> ======
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> That's what I did each times.
> But the first time, with the frozen image without nohup, I didn't  
> get anything with gdb more than this :
> (gdb) call printAllStacks()
> $1 = 2
> (gdb) call printCallStack()
> $2 = -1551081468
> With no output.
> I tried with the advice of Adrian to redirect stdout and stderr  
> inside gdb (http://developer.apple.com/technotes/tn/tn2032.html),  
> but with no success.
>
> So I killed it and relauched with nohup, and when it froze again I  
> got the output I wanted in the nohup.out file.
>
> But the problem is : why did the headless image froze when it got a  
> call to InputSensor>cursorPoint ? There should be a safety in this  
> method to answer 0@0 when the image is headless for example.
> Another problem is that is worked a little (50-70 calls to  
> InputSensor>cursorPoint) and then the image was to slow (100% cpu)  
> to do it. So this behavior could be triggered somewhere else in the  
> image and is annoying to debug...
>
> Florian

> I'm wondering here if it printed the diagnostic messages to your console
> log? If you are on a mac look at your console.log and system.log via the
> Console application. The printAllStacks() should have sent output to stdout
>
> "Note that if you are debugging an already running program that you have
> attached to in GDB (as opposed to launching the app from within GDB),
> stdout and stderr will not be hooked up to the Terminal (they will point
> most likely to the Console or to Project Builder's Run tab, depending
> upon how you have launched your app)."
>
> So yes perhaps output to Console?
>
>
> On Mar 22, 2007, at 1:44 AM, Florian Minjat wrote:
>
>> That's what I did each times.
>> But the first time, with the frozen image without nohup, I didn't get
>> anything with gdb more than this :
>> (gdb) call printAllStacks()
>> $1 = 2
>> (gdb) call printCallStack()
>> $2 = -1551081468
>> With no output.
>> I tried with the advice of Adrian to redirect stdout and stderr inside
>> gdb (http://developer.apple.com/technotes/tn/tn2032.html), but with no
>> success.
>>
>> So I killed it and relauched with nohup, and when it froze again I got
>> the output I wanted in the nohup.out file.
>>
>> But the problem is : why did the headless image froze when it got a
>> call to InputSensor>cursorPoint ? There should be a safety in this
>> method to answer 0@0 when the image is headless for example.
>> Another problem is that is worked a little (50-70 calls to
>> InputSensor>cursorPoint) and then the image was to slow (100% cpu) to
>> do it. So this behavior could be triggered somewhere else in the image
>> and is annoying to debug...
>>
>> Florian
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>

> Florian Minjat writes:
>  > That's what I did each times.
>  > But the first time, with the frozen image without nohup, I didn't get
>  > anything with gdb more than this :
>  > (gdb) call printAllStacks()
>  > $1 = 2
>  > (gdb) call printCallStack()
>  > $2 = -1551081468
>  > With no output.
>  > I tried with the advice of Adrian to redirect stdout and stderr inside
>  > gdb (http://developer.apple.com/technotes/tn/tn2032.html), but with no
>  > success.
>
> The reason I suggested getting the C backtrace was to see if it
> was failing in the GC or elsewhere. Failing in the GC could be caused
> by many things as the GC does not react well to a corrupt heap.
>
> I've seen Squeak lock up consuming 100% CPU due to the 2 Gig memory
> problem due to using ints rather than unsigned ints for address
> calculation.
>
> It would be helpful if you could find out if your problem is related
> to that, or something new.
>
> Bryce
>

> From the output (with the C backtrace) I've got, I think my problem  
> was that the call to InputSensor>cursorPoint during a data save on  
> a file using SmartRefStream. The image was headless and it stuck  
> trying to find the cursor position for the progress bar. I got rid  
> of the ui stuff in SmartRefStream. I will wait to see if it hangs  
> again.
>
> Florian
>
> [hidden email] wrote:
>> Florian Minjat writes:
>>  > That's what I did each times.
>>  > But the first time, with the frozen image without nohup, I  
>> didn't get  > anything with gdb more than this :
>>  > (gdb) call printAllStacks()
>>  > $1 = 2
>>  > (gdb) call printCallStack()
>>  > $2 = -1551081468
>>  > With no output.
>>  > I tried with the advice of Adrian to redirect stdout and stderr  
>> inside  > gdb (http://developer.apple.com/technotes/tn/ 
>> tn2032.html), but with no  > success.
>> The reason I suggested getting the C backtrace was to see if it
>> was failing in the GC or elsewhere. Failing in the GC could be caused
>> by many things as the GC does not react well to a corrupt heap.  
>> I've seen Squeak lock up consuming 100% CPU due to the 2 Gig memory
>> problem due to using ints rather than unsigned ints for address
>> calculation. It would be helpful if you could find out if your  
>> problem is related
>> to that, or something new.
>> Bryce
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside