BUG/REGRESSION while debugging Generator >> #nextPut:
Locked
12
Tackling Context>>#runUntilErrorReturnFro
Re: Tackling Context>>#runUntilErrorReturnFro
Re: Tackling Context>>#runUntilErrorReturnFro
Re: Tackling Context>>#runUntilErrorReturnFro
12
BUG/REGRESSION while debugging Generator >> #nextPut:
 
|
Hi all,
I discovered a bug when stepping over a call to Generator >> #nextPut:. !! Save your image before trying the following !!
Generator
on: [:stream | stream
nextPut: #foo]
The first is annoying for newcomers, but I see this behavior is reasonable and without the effect of the second, it wouldn't be a big problem.
The second, however, looks like a bigger issue to me. Actually, I think the most serious aspect is that in Squeak 5.1, the same procedure does not crash your image, but you are forwarded to the emergency debugger and can terminate the process:
  Also, this is not the first scenario
in the last time where I got an infinite debugger chain. See [BUG(s)] in Context control (#jump, #runUntilErrorOrReturnFrom:) for
a similar issue. And there were even more situations which I could not yet reproduce exactly.
I'm afraid that the recent changes to the debuggers might have weakened its ability to detect recursive errors. Can someone else tell about these problems?
I suppose we ignore a large number of recursive errors as in MorphicDebugger >> #openOn:context:label:contents:fullView:, the uiBlock is often triggered as a separate UI message which will be executed after the recursion flag has been cleared
by the base class.
Would be great if someone could have a look at it or share more information :)
Best,
Christoph
Carpe Squeak!
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
By request, screenshots from a clean image ...
⤴ Press over Press cmd-dot ⤵
The screenshots from 5.1 were made in a clean 5.1 image.
Best, Christoph Von: Thiede, Christoph
Gesendet: Freitag, 13. Dezember 2019 19:48:00 An: Squeak Dev Betreff: BUG/REGRESSION while debugging Generator >> #nextPut: Hi all,
I discovered a bug when stepping over a call to Generator >> #nextPut:. !! Save your image before trying the following !!
Generator
on: [:stream | stream
nextPut: #foo]
The first is annoying for newcomers, but I see this behavior is reasonable and without the effect of the second, it wouldn't be a big problem.
The second, however, looks like a bigger issue to me. Actually, I think the most serious aspect is that in Squeak 5.1, the same procedure does not crash your image, but you are forwarded to the emergency debugger and can terminate the process:
  Also, this is not the first scenario
in the last time where I got an infinite debugger chain. See [BUG(s)] in Context control (#jump, #runUntilErrorOrReturnFrom:) for
a similar issue. And there were even more situations which I could not yet reproduce exactly.
I'm afraid that the recent changes to the debuggers might have weakened its ability to detect recursive errors. Can someone else tell about these problems?
I suppose we ignore a large number of recursive errors as in MorphicDebugger >> #openOn:context:label:contents:fullView:, the uiBlock is often triggered as a separate UI message which will be executed after the recursion flag has been cleared
by the base class.
Would be great if someone could have a look at it or share more information :)
Best,
Christoph
Carpe Squeak!
|
pastedImage.png (33K)
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
Hi all. I am just investigating this issue. However, looking at the tests for Generator, I would :-) suggest :-) to re-phrase this example: Generator on: [:g | g yield: #foo]. -or- Generator on: [:generator | generator yield: #foo]. In any case, countless debuggers show up on "step over". "Step into" works fine.  Squeak 5.3beta #19273 Image 6521 (32 bit) VM: 201911282316 (cog.spur) Windows 10 Best, Marcel
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Squeak 5.3beta #19276 Image 68021 (64 bit) VM 201911282316 Win10 (1903).
@Dave As mentioned, in Squeak 5.1, we get an emergency debugger instead of infinite debuggers. I think this point is clearly a regression. See Morphic-ct.1610 for an approach to prevent countless debuggers - I'm afraid it does not fix this problem, but it prevents others.
What code are you exactly referring to when you talk about the interrupt handler? Via debugging, I found out that the BlockCannotReturn errors are already raised before interrupting. The debugger chain is only blocking itself. You can test this by putting a simple #inform: before the Debugger opening.
Maybe the bug is related to the other context simulation bugs I reported a few weeks ago? I could not explain why the error does not occur if you step into ... Via debugging the debugger, I found out that the bug is raised in the debugged process itself, not in the debugger process.
Best, Christoph
Von: Squeak-dev <[hidden email]> im Auftrag von Taeumel, Marcel
Gesendet: Montag, 16. Dezember 2019 11:49:55 An: John Pfersich via Squeak-dev Betreff: Re: [squeak-dev] BUG/REGRESSION while debugging Generator >> #nextPut:
Hi all.
I am just investigating this issue. However, looking at the tests for Generator, I would :-) suggest :-) to re-phrase this example:
Generator on: [:g | g yield: #foo].
-or-
Generator on: [:generator | generator yield: #foo].
In any case, countless debuggers show up on "step over". "Step into" works fine.
 Squeak 5.3beta #19273
Image 6521 (32 bit)
VM: 201911282316 (cog.spur)
Windows 10
Best,
Marcel
Carpe Squeak!
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
In reply to this post by Christoph Thiede
Hi Christophe, On Fri, Dec 13, 2019 at 10:48 AM Thiede, Christoph <[hidden email]> wrote:
Apologies. I can now reproduce the bug. I was stupidly stepping over the block evaluation, not the send of nextPut:.
So the issue seems to me to be the interaction between the block established in runUntilErrorOrReturnFrom: to catch unhandled errors (this one: Context contextOn: UnhandledError do: [:ex | error ifNil: [ error := ex exception. topContext := thisContext. ex resumeUnchecked: here jump] ifNotNil: [ex pass] ] ) and the swapSender: in Generator>>nextPut:. So after the swapSender: the Generator's continue variable refers to this BlockClosure>>on:do:. BlockClosure>>ensure: pair introduced by runUntilErrorOrReturnFrom:. What I don't understand was how the evaluateOnBehalfOf: changes interact with that. But what I can say is that the Debugger is leaving the Generator (or any code that uses swapSender:) in an invalid state because a) once the swapSender: occurs the protect blocks (the on:do:,ensure: pair) are no longer on the stack. What the swapSender should have been morphed into doing was preserving the protect blocks under the current context, or rather that swapSender should have been persuaded to swap the senders of the protect blocks. Now when the debugger continues stepping it never gets to the protect blocks because they have been swapped out of the way, and execution proceeds all the way down to the fork in Generator>>reset. You can observe this by - stepping through "Generator on: [:stream | stream nextPut: #foo]" until at the send of #nextPut: - inspecting the Debugger (via the Morphic tool handle menu "inspect model" - debugging "self doStep" to step the Debugger through the send - when you get to runUntilErrorOrReturnFrom: *ONLY STEP UP TO THE jump* *DO NOT STEP OVER THE jump* - at the jump, step into, then do step into, and you'll find the debugger now in Generator>>nextPut:, with the stack still including the protect pair. But when you step over the swapSender: then, of course, they've gone missing, and the stack points back to the fork in reset with nothing on the stack to stop execution (because there's no ensure: block which is what runUntilErrorOrReturnFrom: inserted to try and catch any attempt to return). So to get to the point of the swapSender: you should see this: Context>>jump Context>>runUntilErrorOrReturnFrom: [] in Process>>complete: BlockClosure>>ensure: Process>>evaluate:onBehalfOf: Process>>complete: Process>>completeStep: [] in MorphicDebugger(Debugger)>>doStep BlockClosure>>on:do: MorphicDebugger(Debugger)>>handleLabelUpdatesIn:whenExecuting: MorphicDebugger(Debugger)>>doStep and the stack from the Context about to jump is Generator>>nextPut: BlockClosure>>on:do: BlockClosure>>ensure: [] in UndefinedObject>>DoIt Generator>>fork [] in Generator>>reset Generator>>reset Generator>>initializeOn: Generator class>>on: UndefinedObject>>DoIt ... and the Generator's "continue" stack immediately before the swapSender: in Generator>>nextPut: is Generator>>reset Generator>>initializeOn: Generator class>>on: UndefinedObject>>DoIt What we'd like to see after the swapSender: is the protect blocks moved to the continue stack: Generator>>nextPut: BlockClosure>>on:do: BlockClosure>>ensure: Generator>>reset Generator>>initializeOn: Generator class>>on: UndefinedObject>>DoIt and the new "continue" stack looking like this: Generator>>fork [] in Generator>>reset Generator>>reset Generator>>initializeOn: Generator class>>on: UndefinedObject>>DoIt ... What I don't know is how to fix this in the general case. This seems hard. The debugger is executing Smalltalk (i.e. *not* stepping bytecodes) in the context of runUntilErrorOrReturnFrom: and a swapSender: could happen at an arbitrarily deep point in that execution and we want the protect blocks in runUntilErrorOrReturnFrom: to protect execution after a swapSender:. I have no idea how to do that. Now what's more likely is that I'm not seeing what's obviously wrong about the introduction of evaluate:onBehalfOf:. Why does this make any difference? It seems to me that the issue is with runUntilErrorOrReturnFrom: and swapSender:, not with evaluate:onBehalfOf:. What am I missing? > Best, > Christoph _,,,^..^,,,_ best, Eliot |
Re: BUG/REGRESSION while debugging Generator >> #nextPut:
|
|
Hi Eliot,
you're genius, thanks a lot for your answer! :-)
First, for convenience only, here is a snippet to reproduce the bug even simpler:
First, what exact behavior do we expect when stepping over some code that includes a sender swap, speaking in general? The current implementation wants to keep the process running until we return from the selected message - which is the desired behavior in
most cases. Just in this special case, our problem is that the selected message will never return.
If I understand your goal definition correctly, the debugger would halt each time a sender swap occurs while the over button is pressed? I fear this could be inconvenient if you step over any method from a higher abstraction level that uses a generator in any deep
implementation details. So I would like to refine the goal rather as "when pressing over, the process should be resumed until the suspended context is in the stack of the original context (i. e. where the over button was pressed)". For illustration, neglecting
the whole optimization, Process >> #complete: should basically equal the pseudocode "self runUntil: [:ctxt | aContext = ctxt or: [aContext hasSender: ctxt]]". Would you agree so far?
Second, my approach was to modify #swapSender: to manually identify and transport these "essential" sender contexts. The
attached changeset is really WIP, so it only works for the simplest examples, but it might give you an impression of my approach. It does not yet work properly for the following code, if you step over #contents, for example:
If you like the idea, I will try to fix that later.
I know that this approach is in O(n) whereas the current implementation has a constant complexity only, but to me, this appears inherent and inevitable complexity.
However, we would need this modification only for debugging. This leads me to my next concern:
From an architectural point of view, I would like to put this modification into a decorator class (DebuggedContext) that is only installed
from #runUntilErrorOrReturnFrom:.
But it appears the VM does not like this, you cannot even do:
(computation has been terminated), unless you are
debugging. I'd guess that primitive 160 always allocates new memory for the receiver, which would be invisible for the image, but visible to the VM. But I don't have any clue of the actual implementation! Any chance to get around this limitation?
In general, how would you think about this approach? Do you have any better ideas? Looking forward to your opinion :-)
Best,
Christoph
Von: Squeak-dev <[hidden email]> im Auftrag von Eliot Miranda <[hidden email]>
Gesendet: Dienstag, 24. Dezember 2019 03:57:12 An: The general-purpose Squeak developers list Betreff: Re: [squeak-dev] BUG/REGRESSION while debugging Generator >> #nextPut: Hi Christophe,
On Fri, Dec 13, 2019 at 10:48 AM Thiede, Christoph <[hidden email]> wrote:
Apologies. I can now reproduce the bug. I was stupidly stepping over the block evaluation, not the send of nextPut:.
So the issue seems to me to be the interaction between the block established in runUntilErrorOrReturnFrom: to catch unhandled errors (this one:
Context contextOn: UnhandledError do: [:ex |
error ifNil: [
error := ex exception.
topContext := thisContext.
ex resumeUnchecked: here jump]
ifNotNil: [ex pass]
]
)
and the swapSender: in Generator>>nextPut:. So after the swapSender: the Generator's continue variable refers to this BlockClosure>>on:do:. BlockClosure>>ensure: pair introduced by runUntilErrorOrReturnFrom:. What I don't understand was how the evaluateOnBehalfOf:
changes interact with that. But what I can say is that the Debugger is leaving the Generator (or any code that uses swapSender:) in an invalid state because
a) once the swapSender: occurs the protect blocks (the on:do:,ensure: pair) are no longer on the stack. What the swapSender should have been morphed into doing was preserving the protect blocks under the current context, or rather that swapSender should
have been persuaded to swap the senders of the protect blocks. Now when the debugger continues stepping it never gets to the protect blocks because they have been swapped out of the way, and execution proceeds all the way down to the fork in Generator>>reset.
You can observe this by
- stepping through "Generator on: [:stream | stream nextPut: #foo]" until at the send of #nextPut:
- inspecting the Debugger (via the Morphic tool handle menu "inspect model"
- debugging "self doStep" to step the Debugger through the send
- when you get to runUntilErrorOrReturnFrom: *ONLY STEP UP TO THE jump* *DO NOT STEP OVER THE jump*
- at the jump, step into, then do step into, and you'll find the debugger now in Generator>>nextPut:, with the stack still including the protect pair. But when you step over the swapSender: then, of course, they've gone missing, and the stack points back
to the fork in reset with nothing on the stack to stop execution (because there's no ensure: block which is what runUntilErrorOrReturnFrom: inserted to try and catch any attempt to return).
So to get to the point of the swapSender: you should see this:
Context>>jump
Context>>runUntilErrorOrReturnFrom:
[] in Process>>complete:
BlockClosure>>ensure:
Process>>evaluate:onBehalfOf:
Process>>complete:
Process>>completeStep:
[] in MorphicDebugger(Debugger)>>doStep
BlockClosure>>on:do:
MorphicDebugger(Debugger)>>handleLabelUpdatesIn:whenExecuting:
MorphicDebugger(Debugger)>>doStep
and the stack from the Context about to jump is
Generator>>nextPut:
BlockClosure>>on:do:
BlockClosure>>ensure:
[] in UndefinedObject>>DoIt
Generator>>fork
[] in Generator>>reset
Generator>>reset
Generator>>initializeOn:
Generator class>>on:
UndefinedObject>>DoIt ...
and the Generator's "continue" stack immediately before the swapSender: in Generator>>nextPut: is
Generator>>reset
Generator>>initializeOn:
Generator class>>on:
UndefinedObject>>DoIt
What we'd like to see after the swapSender: is the protect blocks moved to the continue stack:
Generator>>nextPut:
BlockClosure>>on:do:
BlockClosure>>ensure:
Generator>>reset
Generator>>initializeOn:
Generator class>>on:
UndefinedObject>>DoIt
and the new "continue" stack looking like this:
Generator>>fork
[] in Generator>>reset
Generator>>reset
Generator>>initializeOn:
Generator class>>on:
UndefinedObject>>DoIt ...
What I don't know is how to fix this in the general case. This seems hard. The debugger is executing Smalltalk (i.e. *not* stepping bytecodes) in the context of runUntilErrorOrReturnFrom: and a swapSender: could happen at an arbitrarily deep point in
that execution and we want the protect blocks in runUntilErrorOrReturnFrom: to protect execution after a swapSender:. I have no idea how to do that.
Now what's more likely is that I'm not seeing what's obviously wrong about the introduction of evaluate:onBehalfOf:. Why does this make any difference? It seems to me that the issue is with runUntilErrorOrReturnFrom: and swapSender:, not with evaluate:onBehalfOf:. What am I missing? > Best, > Christoph _,,,^..^,,,_
best, Eliot
Carpe Squeak!
|
Tackling Context>>#runUntilErrorReturnFro
|
|
Re: Tackling Context>>#runUntilErrorReturnFro
|
|
The formatting of this message was unacceptable. I'm attaching it in a reformated style ...
Christoph
Von: Squeak-dev <[hidden email]> im Auftrag von Thiede, Christoph
Gesendet: Montag, 8. März 2021 20:21 Uhr An: [hidden email] Betreff: [squeak-dev] Tackling Context>>#runUntilErrorReturnFrom: (was: BUG/REGRESSION while debugging Generator >> #nextPut:) Hi all, hi Eliot, hi Marcel,
it's been a long time, but this issue still exists and I have been making a lot of thoughts about it while investigating this and other issues. Marcel and I will soon release a fix for the infinite debugger chains in general, but this issue is a separate
one.
<short recap>
Under certain circumstances, running a single step in a debugger (mostly via the Over button) abandons the current UI process and proceeds the debugged process in its place. This is caused by an
insane and extremely powerful hack in Context>>#runUntilErrorReturnFrom:, which hacks the context to be simulated into the currently running process. This can speed up debugging by around factor 1000. To ensure that the execution returns to the debugging
process, two guard contexts are installed on top of the respective context.
Unfortunately, if the debugged process contains any piece of context metaprogramming, i.e. by implementing coroutine logic with Context>>#swapSender:, or by performing #jumps, this can uninstall the guard contexts or make them unreachable, eventually
causing the debugged process to successfully hijack the debugging process and never return control back to it.
</short recap>
After all, I think there is no viable alternative to explicitly informing the debugger about such acts of context metaprogramming. For this reason, I have set up a working and unpolished prototype in my image, which, in a nutshell, applies the following
changes to Context:
1. In #jump and #swapSender:, insert a send {self informDebuggerAboutContextSwitchTo: coroutine} right before installing the new sender.
2. #informDebuggerAboutContextSwitchTo: searches the sender stack for an UnhandledError handler that was installed by #runUntilErrorOrReturnFrom:. If it finds one, it checks whether this context would still be available after the context switch. (If yes,
nothing is done, otherwise stepping over complex messages such as {self systemNavigation allCallsOn: #foo} would become really messy.) If no, an UnhandledError is signaled to abort the #runUntilErrorOrReturnFrom: execution prematurely.
You can find the details in the attached changeset, but IMO implementation details are less relevant for the current discussion than a general understanding of the problem and the solution approach. Here are some questions for you:
1. Would you agree that is the right approach to the problem because there are no alternatives?
2. Now it's getting indeed a bit more technical: At the moment, #findNextRunUntilErrorOrReturnFromCalleeContextUpTo: in my implementation (horrible name, I know ...) manually traverses the chain of #nextHandlerContexts to search for an UnhandledError handler
of interest. However, I don't really like this approach because a) it creates a
high coupling to the #runUntilErrorOrReturnFrom: implementation (well, on the other hand, both methods resist in the same class ...) and b) it might introduce a
noticeable performance drop (premature benchmarks have suggested an overhead of 5%-25% depending on the task).
I need to build better measures, but a question in general: Are we willing to accept this performance drop in order to regain correctness? (In my opinion, we should be willing, it is only used for debugging and "still fast enough for our neurons"
:-))
An alternative to manually scanning all handler contexts could be to introduce a new exception for this (maybe ContextSwitchNotification) and let the VM do all the work. Or am I overrating "The Great VM" in this regard and things won't be able to become
faster than #handleSignal: at all? For such a low-level, performance-critical decision, I think design questions should be secondary. Do you have some thoughts and opinions about this
I am looking forward to your feedback! Let's get this problem solved, too, definitively before the next release! :-)
Best, Christoph
Carpe Squeak!
|
Re: Tackling Context>>#runUntilErrorReturnFro
|
|
Re: Tackling Context>>#runUntilErrorReturnFro
|
| Free forum by Nabble | Edit this page |