Re: [Pharo-project] Protecting against stack overflow ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] Protecting against stack overflow ?

Eliot Miranda-2



2012/12/9 Sven Van Caekenberghe <[hidden email]>
Hi,

Would it be possible to build some kind of protection against stack overflow ?

note that there already is some mechanism.  The low space mechanism is supposed to protect against stack overflow but with today's memories and the difficulty of testing it, this mechanism often a) takes way too long to kick in, and b) either leaves the system in a very sluggish state or simply fails to kick in and the system crashes with an out-of-memory error.

Part of the problem is in designing a mechanism which is in keeping with the reflective nature of the system.  What I mean is that we now have two different VMs, the Interpreter and the STack/Cog VMs.  First one wants a check which is cheap.  That means a check which isn't run on every send.  In the interpreter a natural time to check for recursion would be on e.g. fullGC, checking for repetition in the current process's context chain.  The the Stack/Cog VMs the natural time is on evacuating a stack page when the stack zone is full.  But these are two different mechanisms, both of which are deep in the VM, neither of which can be conveniently intercepted from Smalltalk, unlike e.g. doesNotUnderstand:.

Personally I like the idea of reviving the low space mechanism and maintaining it properly.  For example, if the low space mechanism causes the low space semaphore to fire often, because the limit was set only a little way above the current memory size, and not, as it is now, a little below the absolute limit, then when it fires, the low space process could inspect the current process's stack (e.g. via accelerated via a primitive for performance) and see how deep it is, and possibly whether there is recursion (e.g. a bag of context methods would soon reveal high-frequency methods, and then those contexts could be inspected for repetition, but KISS says just have a per-process limit on the maximum depth of stack).
 
This is actually the most common situation for loop/cycle that either results in an unusable image or a crash.
Interrupting with CMD-. is not always possible.
A possible solution could be to just limit the stack size and throw an exception.

For example, in LispWorks, it goes like this:

CL-USER 5 > (defun foo () (cons (random 100) (foo)))
FOO

CL-USER 6 > (foo)

Stack overflow (stack size 48128).
  1 (continue) Extend stack by 50%.
  2 (abort) Return to level 0.
  3 Return to top loop level 0.

Type :b for backtrace or :c <option number> to proceed.
Type :bug-form "<subject>" for a bug report template or :? for other options.

CL-USER 7 : 1 > :bq

ERROR <- RUNTIME:BAD-ARGS-OR-STACK <- LENGTH <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- EVAL <- CAPI::CAPI-TOP-LEVEL-FUNCTION <- CAPI::INTERACTIVE-PANE-TOP-LOOP <- MP::PROCESS-SG-FUNCTION

Obviously, this is possible, the question is, can it be done easily and without paying an unbearable performance price ?

Sven

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill







--
best,
Eliot



Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] Protecting against stack overflow ?

Bob Arning-2
What I found useful and simple in a particular case was a high-priority process on a rather short timer.  Each time it woke up it checked the end of object memory. When that was above an arbitrary number (400M in this case) it opened a debugger on the offending process - known in advance in this case, but not too hard to figure out at the time. The overhead for running this is very small and did what I needed at the time.

Cheers,
Bob

On 12/10/12 8:33 PM, Eliot Miranda wrote:



2012/12/9 Sven Van Caekenberghe <[hidden email]>
Hi,

Would it be possible to build some kind of protection against stack overflow ?

note that there already is some mechanism.  The low space mechanism is supposed to protect against stack overflow but with today's memories and the difficulty of testing it, this mechanism often a) takes way too long to kick in, and b) either leaves the system in a very sluggish state or simply fails to kick in and the system crashes with an out-of-memory error.

Part of the problem is in designing a mechanism which is in keeping with the reflective nature of the system.  What I mean is that we now have two different VMs, the Interpreter and the STack/Cog VMs.  First one wants a check which is cheap.  That means a check which isn't run on every send.  In the interpreter a natural time to check for recursion would be on e.g. fullGC, checking for repetition in the current process's context chain.  The the Stack/Cog VMs the natural time is on evacuating a stack page when the stack zone is full.  But these are two different mechanisms, both of which are deep in the VM, neither of which can be conveniently intercepted from Smalltalk, unlike e.g. doesNotUnderstand:.

Personally I like the idea of reviving the low space mechanism and maintaining it properly.  For example, if the low space mechanism causes the low space semaphore to fire often, because the limit was set only a little way above the current memory size, and not, as it is now, a little below the absolute limit, then when it fires, the low space process could inspect the current process's stack (e.g. via accelerated via a primitive for performance) and see how deep it is, and possibly whether there is recursion (e.g. a bag of context methods would soon reveal high-frequency methods, and then those contexts could be inspected for repetition, but KISS says just have a per-process limit on the maximum depth of stack).
 
This is actually the most common situation for loop/cycle that either results in an unusable image or a crash.
Interrupting with CMD-. is not always possible.
A possible solution could be to just limit the stack size and throw an exception.

For example, in LispWorks, it goes like this:

CL-USER 5 > (defun foo () (cons (random 100) (foo)))
FOO

CL-USER 6 > (foo)

Stack overflow (stack size 48128).
  1 (continue) Extend stack by 50%.
  2 (abort) Return to level 0.
  3 Return to top loop level 0.

Type :b for backtrace or :c <option number> to proceed.
Type :bug-form "<subject>" for a bug report template or :? for other options.

CL-USER 7 : 1 > :bq

ERROR <- RUNTIME:BAD-ARGS-OR-STACK <- LENGTH <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO <- FOO
<- FOO <- FOO <- EVAL <- CAPI::CAPI-TOP-LEVEL-FUNCTION <- CAPI::INTERACTIVE-PANE-TOP-LOOP <- MP::PROCESS-SG-FUNCTION

Obviously, this is possible, the question is, can it be done easily and without paying an unbearable performance price ?

Sven

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill







--
best,
Eliot




    



Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] Protecting against stack overflow ?

timrowledge
In reply to this post by Eliot Miranda-2

On 10-12-2012, at 5:33 PM, Eliot Miranda <[hidden email]> wrote:

>
> Personally I like the idea of reviving the low space mechanism and maintaining it properly.

I hope this isn't implying that the low space mechanism is currently not used! That would be bad.

Beware of details when running low on space (I know Eliot is likely to remember since he is old enough to recall us running on 1Mb RAM machines with no virtual memory) - last time I had issues with the Squeak lowspace trigger it was complicated by
a) Dan I stealing memory for some GC purpose *without going via appropriate checking*
b) A hard assumption that you would get more memory from the OS and thus a) would be survivable.
T'ain't always so. Do It RIght.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
You never really learn to swear until you learn to drive in Silicon Valley