Please try out | Fixes for debugger invocation during code simulation

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Please try out | Fixes for debugger invocation during code simulation

marcel.taeumel
Hi all!

Christoph put in a lot of effort to figure out the cause of infinite debugger chains under various circumstances. He is curious how far "debugging the debugger" can work and how possible improvements to code simulation might look like.

We basically figured out that the debugger that pops up from an unhandled error (or warning) must never try debugging a simulated process but always the "genuine process", which is the one that is actually running and which may do the code simulation for another process.

Please find attached a change set with that fix. Tests are in "DebuggerTests" and "ProcessTest". The most interesting methods are ProcessorScheduler >> #debugContext:title:full:contents: and Process >> #debugWithTitle:full:contents:. Please take a look at their comments.

We will merge this into Trunk in a some days, given that there are no objections. :-)

Related discussions:

Best,
Marcel and Christoph

P.S.: Debugging a dialog invocation looks nicer now. Try it out.




use-genuine-process-to-fix-simulation.2.cs (37K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please try out | Fixes for debugger invocation during code simulation

Christoph Thiede

Thanks a lot for the collaboration, Marcel! :-)


We basically figured out that the debugger that pops up from an unhandled error (or warning) must never try debugging a simulated process but always the "genuine process", which is the one that is actually running and which may do the code simulation for another process.


A small summary of this bug for all interested readers who are not yet familiar with the debugging machinery: The actual issue causing all the debugger chains was a mechanism called process-faithful debugging. In a nutshell, it makes sure that when you debug an expression such as "Process activeProcess", the expression answers the correct process instance even when being debugged (that is, it must answer the process being debugged instead of the debugger process which is responsible for the debugging). Process-faithful debugging works by redirecting all sends to ProcessorScheduler >> #activeProcess to a so-called #effectiveProcess. However, when some error occurred in the debugging process during a debugging step (that is, a simulation error such as [1] [2] occurred), a second debugger was opened - but because #activeProcess was just being subjected to redirection, this second debugger did not interrupt the debugging process but the process being debugged instead. As a consequence, the faulty debugging process kept running and was given the opportunity to do any other harm. If you now take a look into how Object >> #at: or Object >> #doesNotUnderstand: are implemented (that is, by unlimited recursion), you will see that it could fire one debugger after the next one this way, without limitation, causing the infinite debugger chains that broke your images.

The solution, in its simplest form, was pretty straightforward: When opening a debugger from an error (see StandardToolSet >> #handleError: and others), make sure not to honor process-faithful debugging but always open the debugger on the actually running process, which we decided to call genuineProcess for sake of differentiation from #activeProcess. The rest of the changeset focuses on some additional refactoring of the overall debugging API on Process and ProcessorScheduler, striving to make it more intuitive for application developers to choose the right process and context for debugging.

If you want to learn even more details, the first thread provided by Marcel should be quite interesting (though also quite long). Otherwise, we'll be here for your questions. :-)

What an interesting project this has been! I'm already looking forward to the next one! :-)

Best,
Christoph


Von: Squeak-dev <[hidden email]> im Auftrag von Taeumel, Marcel
Gesendet: Donnerstag, 11. März 2021 18:01:55
An: squeak-dev
Betreff: [squeak-dev] Please try out | Fixes for debugger invocation during code simulation
 
Hi all!

Christoph put in a lot of effort to figure out the cause of infinite debugger chains under various circumstances. He is curious how far "debugging the debugger" can work and how possible improvements to code simulation might look like.

We basically figured out that the debugger that pops up from an unhandled error (or warning) must never try debugging a simulated process but always the "genuine process", which is the one that is actually running and which may do the code simulation for another process.

Please find attached a change set with that fix. Tests are in "DebuggerTests" and "ProcessTest". The most interesting methods are ProcessorScheduler >> #debugContext:title:full:contents: and Process >> #debugWithTitle:full:contents:. Please take a look at their comments.

We will merge this into Trunk in a some days, given that there are no objections. :-)

Related discussions:

Best,
Marcel and Christoph

P.S.: Debugging a dialog invocation looks nicer now. Try it out.



Carpe Squeak!
Reply | Threaded
Open this post in threaded view
|

Re: Please try out | Fixes for debugger invocation during code simulation

Nicolas Cellier
Great explanation and great findings!


Le jeu. 11 mars 2021 à 19:52, Thiede, Christoph <[hidden email]> a écrit :

Thanks a lot for the collaboration, Marcel! :-)


We basically figured out that the debugger that pops up from an unhandled error (or warning) must never try debugging a simulated process but always the "genuine process", which is the one that is actually running and which may do the code simulation for another process.


A small summary of this bug for all interested readers who are not yet familiar with the debugging machinery: The actual issue causing all the debugger chains was a mechanism called process-faithful debugging. In a nutshell, it makes sure that when you debug an expression such as "Process activeProcess", the expression answers the correct process instance even when being debugged (that is, it must answer the process being debugged instead of the debugger process which is responsible for the debugging). Process-faithful debugging works by redirecting all sends to ProcessorScheduler >> #activeProcess to a so-called #effectiveProcess. However, when some error occurred in the debugging process during a debugging step (that is, a simulation error such as [1] [2] occurred), a second debugger was opened - but because #activeProcess was just being subjected to redirection, this second debugger did not interrupt the debugging process but the process being debugged instead. As a consequence, the faulty debugging process kept running and was given the opportunity to do any other harm. If you now take a look into how Object >> #at: or Object >> #doesNotUnderstand: are implemented (that is, by unlimited recursion), you will see that it could fire one debugger after the next one this way, without limitation, causing the infinite debugger chains that broke your images.

The solution, in its simplest form, was pretty straightforward: When opening a debugger from an error (see StandardToolSet >> #handleError: and others), make sure not to honor process-faithful debugging but always open the debugger on the actually running process, which we decided to call genuineProcess for sake of differentiation from #activeProcess. The rest of the changeset focuses on some additional refactoring of the overall debugging API on Process and ProcessorScheduler, striving to make it more intuitive for application developers to choose the right process and context for debugging.

If you want to learn even more details, the first thread provided by Marcel should be quite interesting (though also quite long). Otherwise, we'll be here for your questions. :-)

What an interesting project this has been! I'm already looking forward to the next one! :-)

Best,
Christoph


Von: Squeak-dev <[hidden email]> im Auftrag von Taeumel, Marcel
Gesendet: Donnerstag, 11. März 2021 18:01:55
An: squeak-dev
Betreff: [squeak-dev] Please try out | Fixes for debugger invocation during code simulation
 
Hi all!

Christoph put in a lot of effort to figure out the cause of infinite debugger chains under various circumstances. He is curious how far "debugging the debugger" can work and how possible improvements to code simulation might look like.

We basically figured out that the debugger that pops up from an unhandled error (or warning) must never try debugging a simulated process but always the "genuine process", which is the one that is actually running and which may do the code simulation for another process.

Please find attached a change set with that fix. Tests are in "DebuggerTests" and "ProcessTest". The most interesting methods are ProcessorScheduler >> #debugContext:title:full:contents: and Process >> #debugWithTitle:full:contents:. Please take a look at their comments.

We will merge this into Trunk in a some days, given that there are no objections. :-)

Related discussions:

Best,
Marcel and Christoph

P.S.: Debugging a dialog invocation looks nicer now. Try it out.





image.png (18K) Download Attachment
image.png (18K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Please try out | Fixes for debugger invocation during code simulation

David T. Lewis
+1

Thank you for doing this and for explaining the issues so well.

Dave


On Fri, Mar 12, 2021 at 03:13:55AM +0100, Nicolas Cellier wrote:

> Great explanation and great findings!
>
>
> Le jeu. 11 mars 2021 ?? 19:52, Thiede, Christoph <
> [hidden email]> a ??crit :
>
> > Thanks a lot for the collaboration, Marcel! :-)
> >
> >
> > > We basically figured out that the debugger that pops up from an
> > unhandled error (or warning) must never try debugging a simulated process
> > but always the "genuine process", which is the one that is actually running
> > and which may do the code simulation for another process.
> >
> > *A small summary of this bug* for all interested readers who are not
> > yet familiar with the debugging machinery: The actual issue causing all the
> > debugger chains was a mechanism called *process-faithful debugging.* In a
> > nutshell, it makes sure that when you * debug an expression such as
> > "Process activeProcess",* the expression answers the correct process
> > instance even when being debugged (that is, it must answer the *process **being
> > debugged* instead of the *debugger process* which is responsible for the
> > debugging). Process-faithful debugging works by *redirecting all sends to
> > ProcessorScheduler **>> **#activeProcess to a so-called #effectiveProcess*
> > *.* However, when some error occurred in the debugging process during
> > a debugging step (that is, *a simulation error* such as [1] [2]
> > occurred), a second debugger was opened - but because #activeProcess was
> > just being subjected to redirection, this second debugger did not interrupt
> > the *debugging process* but the *process being debugged* instead. As a
> > consequence, *the faulty debugging process kept running* and was given
> > the opportunity to do any other harm. If you now take a look into how
> > Object >> #at: or Object >> #doesNotUnderstand: are implemented (that
> > is, by *unlimited recursion*), you will see that *it could fire one
> > debugger after the next one* this way, without limitation, causing the
> > infinite debugger chains that broke your images.
> >
> > *The solution,* in its simplest form, was pretty straightforward: *When
> > opening a debugger from an error* (see StandardToolSet >> #handleError:
> > and others), make sure *not to honor process-faithful debugging* but
> > always open the debugger on the actually running process, which we decided
> > to call *genuineProcess* for sake of differentiation from #activeProcess.
> > The rest of the changeset focuses on some *additional refactoring of the
> > overall debugging API* on Process and ProcessorScheduler, striving to
> > make it more intuitive for application developers to choose the right
> > process and context for debugging.
> >
> > If you want to learn even more details, the first thread provided by
> > Marcel should be quite interesting (though also quite long). Otherwise,
> > we'll be here for your questions. :-)
> >
> > What an interesting project this has been! I'm already looking forward to
> > the next one! :-)
> >
> > Best,
> > Christoph
> >
> > [1] http://forum.world.st/The-Trunk-Kernel-ct-1357-mcz-td5124373.html
> > [2]
> > http://forum.world.st/BUG-REGRESSION-while-debugging-Generator-nextPut-tp5108125.html
> >
> > ------------------------------
> > *Von:* Squeak-dev <[hidden email]> im
> > Auftrag von Taeumel, Marcel
> > *Gesendet:* Donnerstag, 11. M??rz 2021 18:01:55
> > *An:* squeak-dev
> > *Betreff:* [squeak-dev] Please try out | Fixes for debugger invocation
> > during code simulation
> >
> > Hi all!
> >
> > Christoph put in a lot of effort to figure out the cause of infinite
> > debugger chains under various circumstances. He is curious how far
> > "debugging the debugger" can work and how possible improvements to code
> > simulation might look like.
> >
> > We basically figured out that the debugger that pops up from an unhandled
> > error (or warning) must never try debugging a simulated process but always
> > the "genuine process", which is the one that is actually running and which
> > may do the code simulation for another process.
> >
> > Please find attached a change set with that fix. Tests are in
> > "DebuggerTests" and "ProcessTest". *The most interesting methods are
> > ProcessorScheduler >> #debugContext:title:full:contents: and Process >>
> > #debugWithTitle:full:contents:. *Please take a look at their comments.
> >
> > We will merge this into Trunk in a some days, given that there are no
> > objections. :-)
> >
> > Related discussions:
> > http://forum.world.st/I-broke-the-debugger-tp5110752p5110814.html
> > http://forum.world.st/The-Inbox-Morphic-ct-1610-mcz-tp5108228.html
> >
> > Best,
> > Marcel and Christoph
> >
> > P.S.: Debugging a dialog invocation looks nicer now. Try it out.
> >
> >
> >




>


Reply | Threaded
Open this post in threaded view
|

Re: Please try out | Fixes for debugger invocation during code simulation

Jaromir Matas
In reply to this post by Christoph Thiede
Hi Marcel, Christoph,

trying to follow this wonderful job, thanks indeed :) A question: How is the
genuine process supposed to answer #isTerminated, isActive etc.? At the
moment it answers true to isTerminated if the effective process variable is
set. As a consequence the genuine process is not shown on the Process
Browser in such a case.

<http://forum.world.st/file/t372955/active_genuine_effective.png>

The effective process rightly answers true to isActiveProcess but what about
the genuine one?

Generally there's not much difference between a genuine and a terminated
process. Both have suspendedContext = nil. I guess the only difference is
that the genuine process is not terminated, and Processor is the place
keeping this information.

Should there be two active processes? The genuine and the effective? And
extend the isActiveProcess like

        self == Processor activeProcess or: [self effectiveProcess == Processor
activeProcess]

Thanks again, I hope I'm not confused again :)




-----
^[^ Jaromir
--
Sent from: http://forum.world.st/Squeak-Dev-f45488.html

^[^ Jaromir
Reply | Threaded
Open this post in threaded view
|

Re: Please try out | Fixes for debugger invocation during code simulation

marcel.taeumel
Hi Jaromir.

How is the genuine process supposed to answer #isTerminated, isActive etc.?

So, you are wondering about the result of "genuineProcess isTerminated" being sent from a simulated process? Since it is an instance variable, this construction would not easily be possible. Meaning, it is not intended to access the genuine process from outside "Processor".

Should there be two active processes? The genuine and the effective?

Well, at best, code simulation would be oblivious to the objects involved. So, that question would not arise. Yet, thinking about nested code simulation, one might argue that n processes are "active". :-) From a general perspective, however, only 1 process is running at a time. And most of that time, I would argue, it is some process being the genuine process, not an effective one. ;-) 

As a consequence the genuine process is not shown on the Process Browser in such a case. [... ] And extend the isActiveProcess like [...]

No further changes to Process >> #isActiveProcess are needed for our current understanding of code simulation. :-) Given that debugging the debugger is tricky, debugging a process browser belongs to a similar category.

Best,
Marcel

Am 13.03.2021 21:30:15 schrieb Jaromir Matas <[hidden email]>:

Hi Marcel, Christoph,

trying to follow this wonderful job, thanks indeed :) A question: How is the
genuine process supposed to answer #isTerminated, isActive etc.? At the
moment it answers true to isTerminated if the effective process variable is
set. As a consequence the genuine process is not shown on the Process
Browser in such a case.



The effective process rightly answers true to isActiveProcess but what about
the genuine one?

Generally there's not much difference between a genuine and a terminated
process. Both have suspendedContext = nil. I guess the only difference is
that the genuine process is not terminated, and Processor is the place
keeping this information.

Should there be two active processes? The genuine and the effective? And
extend the isActiveProcess like

self == Processor activeProcess or: [self effectiveProcess == Processor
activeProcess]

Thanks again, I hope I'm not confused again :)




-----
^[^ Jaromir
--
Sent from: http://forum.world.st/Squeak-Dev-f45488.html



Reply | Threaded
Open this post in threaded view
|

Re: Please try out | Fixes for debugger invocation during code simulation

Jaromir Matas
Hi Marcel,

> So, you are wondering about the result of "genuineProcess isTerminated"
> being sent from a simulated process?

No, I'm talking about a real situation while debugging e.g. the Christoph's
example:
1. open Process Browser
2. open an explorer window on the UI process
3. do-it [self error: 'e1'] ensure: [^2]
4. now the UI process has become the debugged process and a new UI has been
spawned
5. open an explorer window on the new UI process
6. hit Abandon on the debugger window and see what happened on the enclosed
printscreen:

<http://forum.world.st/file/t372955/Active_effective_genuine_2.png>

- the original UI process became the UI process again but because of the bug
Christoph has reported this original UI process is now still the effective
process and the genuine process is still running and responding true to self
#isTerminated e.g. in the explorer window etc. This causes the genuine
process to disappear from the Process Browser window which I find very
inconvenient and most likely wrong - well, we're no longer debugging yet the
genuine process is not showing which makes it more difficult to realize what
happened.

This is why I ask whether the genuine process shouldn't be considered
something but terminated :) Process Browser filters terminated processes
out.

> From a general perspective, however, only 1 process is running at a time.
> And most of that time, I would argue, it is some process being the genuine
> process, not an effective one. ;-) 

I guess, however with simulation and jumps this may become a bit blurred...
I've got to think about it.

BTW, if you repeat step 3 will get you to the Unwind error and if you hit
Proceed you end up with the two UI processes alternating every worldCycle,
now of course both without the effective variable set and both again visible
in the Process Browser.

The only point I'm making is there are real pathological situations where
you might want to see the genuine process while another process acts as the
active process and considering it terminated is very confusing.

Thanks and best regards,



-----
^[^ Jaromir
--
Sent from: http://forum.world.st/Squeak-Dev-f45488.html

^[^ Jaromir
Reply | Threaded
Open this post in threaded view
|

Re: Please try out | Fixes for debugger invocation during code simulation

Jaromir Matas
Marcel, Christoph, one more question if I may: which of the two UI processes
described in steps 1-6 is actually really running/VM executing their code?
I'm unable to figure out... The genuine one reported as terminated or the
active one but being only effective? Please, your help will be much
appreciated. Many Thanks, J



-----
^[^ Jaromir
--
Sent from: http://forum.world.st/Squeak-Dev-f45488.html

^[^ Jaromir
Reply | Threaded
Open this post in threaded view
|

Re: Please try out | Fixes for debugger invocation during code simulation

marcel.taeumel
Hi Jaromir,

The only point I'm making is there are real pathological situations where
> you might want to see the genuine process while another process acts as the
> active process and considering it terminated is very confusing.

So, you see no problems in merging this change into Trunk?

(For new discussions, you can always open up a new thread on the mailing list.)

Best,
Marcel

Am 15.03.2021 00:21:52 schrieb Jaromir Matas <[hidden email]>:

Marcel, Christoph, one more question if I may: which of the two UI processes
described in steps 1-6 is actually really running/VM executing their code?
I'm unable to figure out... The genuine one reported as terminated or the
active one but being only effective? Please, your help will be much
appreciated. Many Thanks, J



-----
^[^ Jaromir
--
Sent from: http://forum.world.st/Squeak-Dev-f45488.html



Reply | Threaded
Open this post in threaded view
|

Re: Please try out | Fixes for debugger invocation during code simulation

Jaromir Matas
Hi Marcel,

>> The only point I'm making is there are real pathological situations where
>> you might want to see the genuine process while another process acts as
>> the
>> active process and considering it terminated is very confusing.


> So, you see no problems in merging this change into Trunk?

Of course I'm not objecting merging the change, it's great :) Sorry for
mixing this up with the "genuineProcess isTerminated = true", I haven't
realized it was an issue until just recently. It is totally independent and
was there even before, with Processor activeProcess behaving the same.
Thanks again,




-----
^[^ Jaromir
--
Sent from: http://forum.world.st/Squeak-Dev-f45488.html

^[^ Jaromir
Reply | Threaded
Open this post in threaded view
|

Re: Please try out | Fixes for debugger invocation during code simulation

Christoph Thiede
Hi Jaromir,

sorry for the late reply! I very much appreciate your feedback on these
things. :-)

Just for clarification, under normal circumstances, #isTerminated seems to
work as expected. If you debug the UI process (e.g., by evaluating "self
halt"), both the process being debugged and the debugging process answer
#false when sent #isTerminated.

The only exception from this rule appears to occur when the Processor >>
#evaluate:onBehalfOf: logic does not get unwound correctly so the
effectiveProcess is not reset, which is a consequence of the bug I described
in [1]. Unfortunately, there is even another way to reproduce this problem
by corrupting the stack of the simulated process and then debugging it using
#runUntilErrorOrReturnFrom:, for instance by stepping over a method that
does "thisContext privSender: nil" - see also [2] about this concern. In
this situation, a "cannot return" error is shown and after terminating it,
the debugged process keeps answering false when sent #isTerminated. I think
this is also related to the thread you started in [3]. Let's continue
discussing these issues in the relevant threads - I'll add my two cents
there soon!

However, nothing of these issues is a regression of the changeset proposed
in this thread. :-) We only renamed a few things (activeProcess to
genuineProcess) and refactored the accessors, but the conceptional logic of
process-faithful debugging has not been changed.

Still, missing resets of the effectiveProcess are indeed a general problem,
and at least in the event of a #cannotReturn:, there is no chance to unwind
#evaluate:onBehalfOf: properly. While we should fix the actual simulation
issues, I wonder whether we can and whether we should find a way to heal
processes again after a missing reset. A simple heuristic could be setting
the effectiveProcess to nil in every process that does not have an
#evaluate:onBehalfOf: context on its stack. Hypothetically, we could install
a workaround for this check in #isTerminated. Thinking more radically, we
could also replace the effectiveProcess variable with a stateless scoping
exception analogously to CurrentReadOnlySourceFiles, but this would affect
performance. Also, I'm not sure whether a process is automatically
invalidates as soon as the #evaluate:onBehalfOf: disappears from its stack
during simulation - just think of coroutines as used by the Generator,
probably they would be a legal operation as long as #evaluate:onBehalfOf:
will appear on the stack again at the end of the simulation. Questions over
questions ... Maybe you have some other good ideas here. :-)

Best,
Christoph

[1]
http://forum.world.st/Bug-in-Process-gt-gt-terminate-Returning-from-unwind-contexts-td5127570.html
[2]
http://forum.world.st/BUG-REGRESSION-while-debugging-Generator-gt-gt-nextPut-td5108125.html
[3]
http://forum.world.st/Refactoring-terminate-to-get-rid-of-cannot-return-errors-etc-td5127732.html



-----
Carpe Squeak!
--
Sent from: http://forum.world.st/Squeak-Dev-f45488.html

Carpe Squeak!