#fork and deterministic resumption of the resulting process

>Hi -
>
>In my never-ending quest for questionable behavior in multi-threaded
>situations just today I ran into a pattern which is dangerously common
>in our code. It basically goes like this:
>
>MyClass>>startWorkerProcess
> "worker is an instance variable"
> worker:= [self runWorkerProcess] fork.
>
>MyClass>>runWorkerProcess
> "Run the worker process"
> [Processor activeProcess == worker] whileTrue:[
> "...do the work..."
> ].
>
>MyClass>>stopWorkerProcess
> "Stop the worker process"
> worker := nil. "let it terminate itself"
>
>Those of you who can immediately tell what the problem is should get a
>medal for an outstanding knack of analyzing concurrency problems ;-)
>
>For the rest of us, the problem is that #fork in the above is not
>deterministic in the way that there is no guarantee whether the "worker"
>variable will be assigned when we enter the worker loop. It *would* be
>deterministic if the priority were below or above the current process'
>priority but when it's the same it can be affected by environmental
>effects (external signals, delay processing etc) leading to some very
>obscure runtime problems (in the above, the process would just not start).
>
>To fix this problem I have changed BlockContext>>fork and
>BlockContext>>forkAt: to read, e.g.,
>
>BlockContext>>fork
>   "Create and schedule a Process running the code in the receiver."
>   ^self forkAt: Processor activePriority
>
>BlockContext>>forkAt: priority
>   "Create and schedule a Process running the code in the receiver
>   at the given priority. Answer the newly created process."
>   | forkedProcess helperProcess |
>   forkedProcess := self newProcess.
>   forkedProcess priority: priority.
>   priority = Processor activePriority ifTrue:[
>     helperProcess := [forkedProcess resume] newProcess.
>     helperProcess priority: priority-1.
>     helperProcess resume.
>   ] ifFalse:[
>     forkedProcess resume
>   ].
>   ^forkedProcess
>
>This will make sure that #fork has (for the purpose of resumption) the
>same semantics as forking at a lower priority has.
>
>What do people think about this?
>
>Cheers,
>   - Andreas
>

>> I ran into a similar problem in VAST that I got around by extending Block and
>> Process.  I extended Block with #forkReady, #forkReadyAt: and some others and
>> Process with #ready.  The #forkReady extension to Block basically does what it
>> says, it creates a new fork or process in the ready state by does not start it
>> immediately.  The new process will wait for its normal turn to start.  The
>> #ready extension to Process makes the process ready without running it
>> immediately and is used by #forkReady.
>
>I bet that your #forkReady has the same race condition problem that
>Andreas is trying to solve in Squeak's #fork.  :-(
>
>Paolo
>

>
>> I ran into a similar problem in VAST that I got around by extending  
>> Block and
>> Process.  I extended Block with #forkReady, #forkReadyAt: and some  
>> others and
>> Process with #ready.  The #forkReady extension to Block basically  
>> does what it
>> says, it creates a new fork or process in the ready state by does  
>> not start it
>> immediately.  The new process will wait for its normal turn to  
>> start.  The
>> #ready extension to Process makes the process ready without running  
>> it
>> immediately and is used by #forkReady.
>
> I bet that your #forkReady has the same race condition problem that  
> Andreas is trying to solve in Squeak's #fork.  :-(

>
> On Feb 5, 2008, at 1:12 PM, Paolo Bonzini wrote:
>
>>
>>> I ran into a similar problem in VAST that I got around by  
>>> extending Block and
>>> Process.  I extended Block with #forkReady, #forkReadyAt: and some  
>>> others and
>>> Process with #ready.  The #forkReady extension to Block basically  
>>> does what it
>>> says, it creates a new fork or process in the ready state by does  
>>> not start it
>>> immediately.  The new process will wait for its normal turn to  
>>> start.  The
>>> #ready extension to Process makes the process ready without  
>>> running it
>>> immediately and is used by #forkReady.
>>
>> I bet that your #forkReady has the same race condition problem that  
>> Andreas is trying to solve in Squeak's #fork.  :-(
>
> I think you misunderstand (unless I misunderstand)... it sounds like  
> the Process created by #forkReady doesn't start until explicitly  
> resumed:

> Michael van der Gulik wrote:
> > Andreas's original code was buggy and his proposed fix was incorrect.
>
> If I would need any more proof that people don't get what I'm trying to
> fix, this would do it ;-) Seriously, you *really* don't get what I'm
> trying to address with the fix.
>
> Anyway, I don't care. Croquet has that fix and it's your choice to
> ignore it and deal with the consequences. And with that I'm out of this
> thread for real and apologize for the pointless waste of bandwidth.

> Paolo Bonzini wrote:
>>> That's part of the reason why I won't pursue these changes here.  
>>> To me these changes are just as important as the ones that I  
>>> posted for Delay and Semaphore. However, unless one understands  
>>> the kinds of problems that are caused by the current code it is  
>>> pointless to argue that fixing them is important - I'm sure that  
>>> unless people had been bitten by Delay and Semaphore we would have  
>>> the same kinds of debates with all sorts of well-meant advise on  
>>> how you "ought" to write your code ;-)
>> It's not that I don't think it's important.  I think the *bugs* are  
>> important to fix, but that the root cause just *cannot* be fixed.
>
> This completely depends on your definition of "root cause" and  
> "cannot". For me, it's the fact that fork will behave in 99.99% of  
> the cases in one way and in 0.01% in a different way. That kind of  
> non-determinism is probably the root cause for many lingering bugs  
> in our system and it *can* be eliminated.
>
>> It's just that:
>> 1) the many people who made the same mistake maybe were just  
>> cutting'n'pasting buggy code;
>
> That is of course a possibility but unless you think the majority of  
> people recognized the bug in the code snippet I posted, I fail to  
> see how this makes a difference.
>
>> 2) especially, the fix is not 100% safe unless I'm mistaken.
>
> What do you mean by "100% safe"? It is 100% deterministic (which is  
> what I care about); I'm not sure what you mean when you use the term  
> "safe" here.

> On 06/02/2008, nicolas cellier <[hidden email]> wrote:
> > Michael van der Gulik a écrit :
> > > Er... I can't tell if you're joking or if you're serious.
> > >
> > > I always write my code under the assumption that some day, in the
> > > future, some bright spark will write a better VM that will run
> > > separate Smalltalk Processes on separate OS processes/threads by
> > > whatever means, thus better utilising multi-cored or multiple CPUs.
> > >
> > > When that day comes, my code will run faster and your code will break.
> > >
> >
> > I see, sustainable development. Your writing code for our children.
> >
>
> Oh, come on, how you can be so blind?
> You need a ground, based on which you can build your application.
> You need some API with clear rules for play.
> And these rules defined by shared principles and common terms, easily
> understandable by people.
> Newcomer who will start doing things should not spend time harvesting,
> how #fork works on particular squeak version.
>
> So, when i say, #fork, it should fork today, and should fork tomorrow,
> and should fork after 20 years. Because it's a generic rule, and API
> should behave as generic rule dictates: do whatever it take to make
> two processes run in parallel.
> But if you start doing reverse: let API influence rules, then what you
> will get in result?
> 10 APIs and each defining own semantics of fork?