The undocumented behavior of Process class>>copyStack

Recently, I was asked to explain why Process>> walkbackOn… fails periodically when running inside a forked block.

Very simplified, it boils down to:

[ Process copyStack walkbackOn: String new asStream maxLevels: 50 stopSelector: nil filtered: false arguments: true ] forkAt: 3.

Periodical errors are tricky and need systematical analysis as well as some gut feelings.

I won’t go into details, but fast forward things a little. First step, compare what’s the difference between a “good” process and a “bad” process. To make things simple, I decided to keep the contents of the Process in an Array.

process := Process copyStack.

good := process asArray.

process walkbackOn: …

process := Process copyStack.

bad := process asArray.

process walkbackOn: … FAILS!

Compare contents of “good” and “bad”. They are equal. Not what I expected. The bug must be obviously somewhere in the code that walks the stack. Trying to understand this, I saw that the size of “process” and “bad” was different. In other words, if I evaluated “process asArray”, I would also get different contents. Somehow the “process” object changed!

Obviously, the bug is not in the walk stack method, but due to the process being walked somehow getting mutated. So there is either some ugly bug in the VM or something is rotten with the #copyStack method. I was guessing the second.

Some time ago Leandro Caniglia wrote an excellent explanation of the purpose of the #copyStack and the primitive behind it. May be I was missing something. Some more experimenting.

proc1 := CurrentProcess.

proc2 := Process copyStack.

proc1 == proc2  … true !!!

proc2 == CurrentProcess … true !!!

Wait a minute! What’s going in here! Wasn’t it supposed to make a copy of the current process native stack? Apparently not! The documentation on the primitive is limited. Apparently it should have been called #fillCurrentProcessFromNativeStack … and return CurrentProcess for convenience.

It becomes clear what’s going wrong. The walkback method may be interrupted by a VM interrupt. Because it’s not the U/I process, the current process yields to have the U/I process respond to the user. This causes a call to #copyStack. Example call stack when issue happens:

Process class>>copyStack

ProcessScheduler>>copyStack

ProcessScheduler>>suspendActive

ProcessScheduler>>yield

ProcessScheduler>>resume:

Semaphore>>signal

Process class>>osEventInterruptGui

Process class>>osEventInterrupt

Object>>perform:

Object>>vmInterrupt:

---- Interrupt ----

Character>>=

IndexedCollection>>includes:

Character>>isVowel

Process>>printWalkbackArgument:

Process>>walkbackListTo:maxLevels:filtered:arguments:

Process>>walkbackOn:maxLevels:stopSelector:filtered:arguments:

a Smalltalk expression (#DoIt)

---- Irrelevant caller code ----

The return value from #copyStack is not an immutable object! Another call to #copyStack from the same process as where it was first returned will overwrite it! Solution is to create a copy of the value returned by #copyStack. Be sure that this copy is done with disabled interrupts, otherwise the same issue as above.

The lesson of the day; be careful what you do with the result from the #copyStack method. Unexpected things may happen.

-- Todor

Hi Leandro,

No. It’s not enough. The invocation of #objectShallowCopy may get interrupted (the same problem described in the original post) and we’ll have an inconsistent Process object. For example, the instvars “contents” will be from the first version of the object and “startPosition” and “endPosition” from another version of the object. Obviously, not a valid OrderedCollection (and Process) object. You will need to guard the copy process from interruptions.

I would do it as:

Process>>safeCopy

                [

                ^self class new

                                startPosition: self startPosition;

                                endPosition: self endPosition;

                                contents: self contents;

                                topFrame: self topFrame;

                                frameBias: self frameBias;

                                priority: self priority;

                                sendFrame: self sendFrame;

                                runable: self runable;

                                isUserIF: self isUserIF;

                                debugger: self debugger;

                                name: self name;

                                interruptFrame: self interruptFrame;

                                exceptionEnvironment: self exceptionEnvironment;

                                terminationBlock: self terminationBlock;

                                protectionBlock: self protectionBlock;

                                yourself.

                ] evaluateWithoutInterrupts.

Basically, I would create a new object and copy all the instance vars. I don’t see a need to create a (deep) copy of the “contents” array. To my understanding, the VM does not modify this array. The copy stack primitive simply creates a new array (needs to be verified) and replaces the old contents. Debugger may however modify the contents of a debugged process, but I am not expert on that. The important thing is that the copy operation itself does not get interrupted.

-- Todor

From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] On Behalf Of Leandro Caniglia
Sent: Wednesday, 07. May, 2014 20:19
To: [hidden email]
Subject: Re: The undocumented behavior of Process class>>copyStack

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

Hi Bruce,

No. It’s no VA code. It was just “pseudo code” to demonstrate that you only need to copy the instance variables. So, the #startPosition, #endPosition etc. should be considered accessor method to the corresponding instance variables.

-Todor

From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] On Behalf Of Bruce W
Sent: Thursday, 08. May, 2014 14:23
To: [hidden email]
Subject: Re: The undocumented behavior of Process class>>copyStack

Hi Todor,

Is this for VisualAge?

Our VSE "Process" class does not have  #startPosition or #endPosition instance methods.

Regards,
- bruce



From:        Todor Todorov <[hidden email]>
To:        [hidden email]
Date:        05/08/2014 06:14 AM
Subject:        Re: The undocumented behavior of Process class>>copyStack
Sent by:        Using Visual Smalltalk for Windows/Enterprise <[hidden email]>

Hi Leandro,
 
No. It’s not enough. The invocation of #objectShallowCopy may get interrupted (the same problem described in the original post) and we’ll have an inconsistent Process object. For example, the instvars “contents” will be from the first version of the object and “startPosition” and “endPosition” from another version of the object. Obviously, not a valid OrderedCollection (and Process) object. You will need to guard the copy process from interruptions.
 
I would do it as:
Process>>safeCopy
                [
                ^self class new
                                startPosition: self startPosition;
                                endPosition: self endPosition;
                                contents: self contents;
                                topFrame: self topFrame;
                                frameBias: self frameBias;
                                priority: self priority;
                                sendFrame: self sendFrame;
                                runable: self runable;
                                isUserIF: self isUserIF;
                                debugger: self debugger;
                                name: self name;
                                interruptFrame: self interruptFrame;
                                exceptionEnvironment: self exceptionEnvironment;
                                terminationBlock: self terminationBlock;
                                protectionBlock: self protectionBlock;
                                yourself.
                ] evaluateWithoutInterrupts.
 
Basically, I would create a new object and copy all the instance vars. I don’t see a need to create a (deep) copy of the “contents” array. To my understanding, the VM does not modify this array. The copy stack primitive simply creates a new array (needs to be verified) and replaces the old contents. Debugger may however modify the contents of a debugged process, but I am not expert on that. The important thing is that the copy operation itself does not get interrupted.
 
-- Todor
 
From: Using Visual Smalltalk for Windows/Enterprise [[hidden email]] On Behalf Of Leandro Caniglia
Sent: Wednesday, 07. May, 2014 20:19
To: [hidden email]
Subject: Re: The undocumented behavior of Process class>>copyStack
 
Hi Todor,
A question. Was #copy enough to solve the problem? I'm asking because in the document you mentioned there is a section about copying the stack. I've pasted below the relevant part.
Regards,
Leandro

There is a point worth mentioning here: the default implementation of #shallowCopy inherited from IndexedCollection does not suffice because it only copies indexed entries but not the other named instance variables. Thus, we have to override #shallowCopy with:

Process >> #shallowCopy
   ^self objectShallowCopy postShallowCopy

where #objectShallowCopy is Object >> #shallowCopy and #postShallowCopy is:

Process >> #postShallowCopy
   contents := contents shallowCopy

Since Process >> #copy just sends #shallowCopy this is enough to have the #copy we were looking for.
 
On Wed, May 7, 2014 at 11:07 AM, Todor Todorov <[hidden email]> wrote:
Recently, I was asked to explain why Process>> walkbackOn… fails periodically when running inside a forked block.
 
Very simplified, it boils down to:
[ Process copyStack walkbackOn: String new asStream maxLevels: 50 stopSelector: nil filtered: false arguments: true ] forkAt: 3.
 
Periodical errors are tricky and need systematical analysis as well as some gut feelings.
 
I won’t go into details, but fast forward things a little. First step, compare what’s the difference between a “good” process and a “bad” process. To make things simple, I decided to keep the contents of the Process in an Array.
 
process := Process copyStack.
good := process asArray.
process walkbackOn: …
 
process := Process copyStack.
bad := process asArray.
process walkbackOn: … FAILS!
 
Compare contents of “good” and “bad”. They are equal. Not what I expected. The bug must be obviously somewhere in the code that walks the stack. Trying to understand this, I saw that the size of “process” and “bad” was different. In other words, if I evaluated “process asArray”, I would also get different contents. Somehow the “process” object changed!
 
 
Obviously, the bug is not in the walk stack method, but due to the process being walked somehow getting mutated. So there is either some ugly bug in the VM or something is rotten with the #copyStack method. I was guessing the second.
 
Some time ago Leandro Caniglia wrote an excellent explanation of the purpose of the #copyStack and the primitive behind it. May be I was missing something. Some more experimenting.
 
proc1 := CurrentProcess.
proc2 := Process copyStack.
proc1 == proc2  … true !!!
proc2 == CurrentProcess … true !!!
 
 
Wait a minute! What’s going in here! Wasn’t it supposed to make a copy of the current process native stack? Apparently not! The documentation on the primitive is limited. Apparently it should have been called #fillCurrentProcessFromNativeStack … and return CurrentProcess for convenience.
 
It becomes clear what’s going wrong. The walkback method may be interrupted by a VM interrupt. Because it’s not the U/I process, the current process yields to have the U/I process respond to the user. This causes a call to #copyStack. Example call stack when issue happens:
Process class>>copyStack
ProcessScheduler>>copyStack
ProcessScheduler>>suspendActive
ProcessScheduler>>yield
ProcessScheduler>>resume:
Semaphore>>signal
Process class>>osEventInterruptGui
Process class>>osEventInterrupt
Object>>perform:
Object>>vmInterrupt:
---- Interrupt ----
Character>>=
IndexedCollection>>includes:
Character>>isVowel
Process>>printWalkbackArgument:
Process>>walkbackListTo:maxLevels:filtered:arguments:
Process>>walkbackOn:maxLevels:stopSelector:filtered:arguments:
a Smalltalk expression (#DoIt)
---- Irrelevant caller code ----
 
 
The return value from #copyStack is not an immutable object! Another call to #copyStack from the same process as where it was first returned will overwrite it! Solution is to create a copy of the value returned by #copyStack. Be sure that this copy is done with disabled interrupts, otherwise the same issue as above.
 
 
The lesson of the day; be careful what you do with the result from the #copyStack method. Unexpected things may happen.
 
 
-- Todor
 
*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***
 
*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

www.labware.com
Results Count