WAProcess>>critical:ifError problem

Hi,

I'm having problems with the following method in WAProcess, on a
Dolphin 5 Smalltalk:

critical: aBlock ifError: errorBlock
   |value|
   mutex critical:[
        responseSem := Semaphore new.
        process := [
                        [value := aBlock on: Error do: errorBlock]
                        ensure: [responseSem signal]
                ] fork.
        responseSem wait
   ].
   process := nil.
   ^ value


After evaluating aBlock value gets its contents (a WAResponse).
But when the mutex ends the critical evaluation, value is nil.

Perhaps it has something with the Continuations that happens inside
the process, or something similar.

What can be causing this?


This message is cross posted  to Seaside list, and Dolphin newsgroup
intentionally.

Best regards,

--
Esteban.

Hi,

Without having found a solution to this problem, evaluating the
following code on a workspace, answers the expected value.

|value mutex block errorBlock |
block := [Processor sleep: 1000. 'Ey! I am not nil!'].
errorBlock := [Transcript show: 'Error!';cr].
mutex := Semaphore forMutualExclusion.
mutex critical:[
     responseSem := Semaphore new.
     process := [
             [value := block on: Error do: errorBlock]
             ensure: [responseSem signal]
         ] fork.
     responseSem wait
].
process := nil.
^ value "display-it"


Perhaps the problem in WAProcess>>critical:ifError: is because
aBlock does some continuation stuff inside.

Can that be a reason?

Best regards,

--
Esteban.



Esteban A. Maringolo escribió:

Esteban,

> Perhaps the problem in WAProcess>>critical:ifError: is because
> aBlock does some continuation stuff inside.
>
> Can that be a reason?

Seems quite likely to me.  It might even be what's supposed to happen.
Apologies in advance if I'm missing the point, or if the following is already
obvious to you, but, for instance, if you set up some continuations stuff:

    c := nil.
    [[:cc | c := cc] callCC. Time now] value. "display it"

(The example's much clearer if you do "display it" rather than just evaluating
the expression).  That sets c to a continuation. Now evaluate it:

    c value. "evaluate it"

We end up back in the original "display it", so the continuation is working.
Now set up a simplified version of your example:

    block := [c value. 'Aha!'].
    value := nil.
    [value := block value] fork.
    value.

When you evaluate that lot, the forked process is hijacked and replaced with
the original "display it" again, so the forked code never executes past the
expression
        c value
in the block and so never does return 'Aha!' or assign that to value.

Of course, that's all on the assumption that the real code is doing the
equivalent of
        c value
somewhere, which may not be true.  But if not then I don't see what that
complicated fork-and-wait logic is for.

    -- chris

Hi Chris,

Chris Uppal escribió:
> Esteban,

>>Perhaps the problem in WAProcess>>critical:ifError: is because
>>aBlock does some continuation stuff inside.
>>
>>Can that be a reason?

> Seems quite likely to me.  It might even be what's supposed to happen.

To me too.

> Apologies in advance if I'm missing the point, or if the following is already
> obvious to you, but, for instance, if you set up some continuations stuff:

No apology is neccesary, I'm still assimilating this continuation stuff.

>     c := nil.
>     [[:cc | c := cc] callCC. Time now] value. "display it"
>     c value. "evaluate it"
> We end up back in the original "display it", so the continuation is working.

Well... the strange is not that, because continuations have that
expected behaviour.
The strange, to me,  is that inside of the evaluated block, value
got it's value (which in this is case is a WAResponse, kind of
HTTPResponse), but once it lefts the ensured evaluation, value is nil.

responseSem := Semaphore new.
process := [
        [value := aBlock on: Error do: errorBlock]
                ensure: [responseSem signal]
        ] fork.
responseSem wait.


Perhaps it has something to do with how the process copies, or how
it's, but I'm guessing to much beyond my understanding.

Andy, Blair?

One thing to note is this works as it is on VW and Squeak, and I
have the problem with both D5 and Dx6.

Best regards,

--
Esteban.

Following with this, which is stopping me to continue the port, some
other questions arise...

What is the difference between a aMutex and a Semaphore
forMutualExclusion?

Both provide a #critical: execution, and Mutex uses a semaphore for
it's implementation.

Which is the difference in Dolphin? Some further read to exercise or
learn about this?

Best regards,

--
Esteban.


Esteban A. Maringolo (yo) escribió:
> Chris Uppal escribió:
>> Esteban,

>>> Perhaps the problem in WAProcess>>critical:ifError: is because
>>> aBlock does some continuation stuff inside.
>>>
>>> Can that be a reason?

>> Seems quite likely to me.  It might even be what's supposed to happen.

>What is the difference between a aMutex and a Semaphore
>forMutualExclusion?

I could be wrong, but I think the difference is that the same process can
enter a Mutex critical section multiple times, while a Semaphore
forMutualExclusion can be entered only once, even by the process which had
locked the critical section.

For instance:

m := Mutex new.
m critical: [
    Transcript show: '1'; cr.
    m critical: [Transcript show: '2'; cr.]]

will show '1 2' while

m := Semaphore forMutualExclusion.
m critical: [
    Transcript show: '1'; cr.
    m critical: [Transcript show: '2'; cr.]]

will show '1' and lock.

I don't know if there are more differences, my guess is there might be.

regards

Martin Rubi wrote:
>>What is the difference between a aMutex and a Semaphore
>>forMutualExclusion?
>
>
> I could be wrong, but I think the difference is that the same process can
> enter a Mutex critical section multiple times, while a Semaphore
> forMutualExclusion can be entered only once, even by the process which had
> locked the critical section.

That is my understanding too.

Have a good one,

Bill


--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Esteban,

> Perhaps it has something to do with how the process copies, or how
> it's, but I'm guessing to much beyond my understanding.

I think you are going to have to create a small example that reproduces this.
It clearly depends on exactly /what/ "continuations stuff" happens in the
block.  I spent a few minutes on it, but was unable to find a case that
displayed the behaviour.  E.g the following does work:

    c := nil.
    block := [[:cc | c := cc. c value: 'Aha!!'] callCC].

    sem := Semaphore new.
    value := nil.
    [
        [value := block value]
            ensure:  [sem signal]
    ] fork.
    sem wait.

So it seems that /making/ a copy does not cause problems (which is not
surprising).  Using a copy (jumping back to its state) does not inherently
cause problems, at least it works in the simple case above.  And I wasn't able
to think of a more complicated pattern of "jumps" that still "should" assign to
'value'.

BTW, code executed from workspaces isn't quite the same as "real" methods, so
it's probably easier to try to create a very small class that displays the
problem and only then (if at all) to try to translate that into workspace code.

    -- chris

Hi,

I've found something, that may have relation to how the process are
copied.

Esteban A. Maringolo escribió:
> Hi Chris,
> Chris Uppal escribió:

>>> Perhaps the problem in WAProcess>>critical:ifError: is because
>>> aBlock does some continuation stuff inside.
>>> Can that be a reason?
>> Seems quite likely to me.  It might even be what's supposed to happen.
> To me too.

> The strange, to me,  is that inside of the evaluated block, value got
> it's value (which in this is case is a WAResponse, kind of
> HTTPResponse), but once it lefts the ensured evaluation, value is nil.

> responseSem := Semaphore new.
> process := [
>     [value := aBlock on: Error do: errorBlock]
>         ensure: [responseSem signal]
>     ] fork.
> responseSem wait.

> Perhaps it has something to do with how the process copies, or how it's,
> but I'm guessing to much beyond my understanding.
> Andy, Blair?

I've found that the ensured block was executed twice, it is a
semaphore signaled twice too. Removing the ensuring everything works
ok. What is the relation of this to continuations... I can't tell,
but my guess is that has something with ProcessCopier.

So... replacing:

WAProcessMonitor>>critical: aBlock ifError: errorBlock
| value |
value := nil.
mutex critical: [ "mutex is a mutually exclusive semaphore"
   responseSem := Semaphore new.
   process := [
     [value := aBlock on: Error do: errorBlock]
        ensure: [responseSem signal]
   ] fork.
   responseSem wait.
].
process := nil.
^value

By:

WAProcessMonitor>>critical: aBlock ifError: errorBlock
| value |
value := nil.
mutex critical: [
   responseSem := Semaphore new.
   process := [
        value := aBlock on: Error do: errorBlock.
        responseSem signal
        ] fork.
   responseSem wait.
].
process := nil.
^value

Everything starts working magically. Of course, no ensuring is done,
but by now it wasn't necessary, in the future this has to be solved.

Best regards,

--
Esteban.

Chris Uppal escribió:
> Esteban,
>>Perhaps it has something to do with how the process copies, or how
>>it's, but I'm guessing to much beyond my understanding.

> I think you are going to have to create a small example that reproduces this.

Yes, I'll make a test case to reproduce the behavior, someday...

Sometimes it worked, and sometimes it doesn't, definitely it has to
be something related with what happens inside of the protected block.

 > Using a copy (jumping back to its state) does not inherently
> cause problems, at least it works in the simple case above.  And I wasn't able
> to think of a more complicated pattern of "jumps" that still "should" assign to
> 'value'.

> BTW, code executed from workspaces isn't quite the same as "real" methods, so
> it's probably easier to try to create a very small class that displays the
> problem and only then (if at all) to try to translate that into workspace code.

Ha!, Process bug hunting, does it qualify as an olympic sport? :-)

Best regards,

--
Esteban.

Esteban,

> I've found that the ensured block was executed twice, it is a
> semaphore signaled twice too. Removing the ensuring everything works
> ok. What is the relation of this to continuations... I can't tell,
> but my guess is that has something with ProcessCopier.

Ahh.... Now it's starting to come a little bit clearer; not yet perfectly
clear, but getting closer...

Given this:

    c := nil.
    block := [[:cc | c := cc. c value: 'Aha!!'] callCC].

    sem := Semaphore new.
    value := nil.
    [
        [value := block value]
            ensure:  [Transcript display: 'ooerr...'; cr. sem signal]
    ] fork.
    sem wait.

(Which is just as before except that I've added some tracing to the #ensure:)

When block is evaluated, the #callCC makes a copy of the process's
#exceptionEnvironment, which includes the cleanup code in the #ensure:.

Next when we send #value to the Continuation in 'c', a new Process is created
from the Continuation (with a copy of the original #exceptionEnvironment) and
the executing process (the one that was #fork-ed) is terminated.  Terminating
it, as always, runs any #ensure: blocks.  So the Semaphore is signalled.

But the new process is a copy of the original one, and has the /same/ #ensure
block. So as it dies (having assigned to value and then run out of things to
do) its #ensure block is (as always) executed.  So the semaphore is signalled
again.

If you now evaluate:
    c value: 22.
 (just as an experiment), then 22 will be assigned to value, and the #ensure
block will be executed a third time.

Lastly, if you set c to nil, and then wait for a little while for the GC to
reclaim it, the Continuation's finalise will run, and that too will execute the
cleanup block.

I haven't been able to make 'value' come out as nil with this, but if I had
executed
    c value.
in the last paragraph but one, then value would have been set to nil.  Is it
possible that that is what you are seeing ?

I'm not sure what the behaviour of #ensure: (and similar) should be when
continuations are used.  In one way it seems perfectly natural that the block
should be executed as many times as the Continuation is restarted.  In another
way that's might be thought a bit surprising, as you'd normally want it to
execute exactly once, but then /when/ should the block be executed ?

It doesn't seem right to me that the ensure blocks of the Process that is being
hijacked should run -- I think the intended semantics is that the data stored
in the Continuation is used to /replace/ the current Process, overwriting it.
If so, then the hijacked Process's #ensure blocks should just be lost.  I tried
to hack the system to make that happen by redefining
Continuation>>continue:with: as:

    continue: aProcess with: anObject
           Processor activeProcess exceptionEnvironment: nil.
           aProcess continueWith: anObject

Which seems to remove that particular problem.  I don't really know what I'm
doing here, though, so it's almost certainly not the correct fix, but it would
be interesting to see if it (as a temporary hack) solves your problem.

Also, I'm not at all sure that #ensure blocks should execute as part of the
Continuation's finalisation, I can't see a good logical justification for that.
I /think/ that Continuation>>stack: should remove its stored Process's
finalisability and only restore it again in #continue:with: -- but I haven't
attempted to test that "fix".

    -- chris

Esteban,

> Ha!, Process bug hunting, does it qualify as an olympic sport? :-)

<grin/>

You might be surprised.  According to the BBC news website, the last time the
Olympics were run in London, you could win medals for painting or writing
poetry...

    -- chris

I wrote:

> Next when we send #value to the Continuation in 'c', a new Process is
> created from the Continuation (with a copy of the original
> #exceptionEnvironment) and the executing process (the one that was
> #fork-ed) is terminated.  Terminating it, as always, runs any #ensure:
> blocks.  So the Semaphore is signalled.

So the waiting Process can now execute.  If it wakes up quickly, it will read
'value' before the new Process has a chance to write it.  If it takes a long
time to wake up (like me today ;-) then it'll see the correct value.

I think that completes the picture.  Whee!!  ;-)

    -- chris

Hi Chris,

Chris Uppal escribió:
> I wrote:

>>Next when we send #value to the Continuation in 'c', a new Process is
>>created from the Continuation (with a copy of the original
>>#exceptionEnvironment) and the executing process (the one that was
>>#fork-ed) is terminated.  Terminating it, as always, runs any #ensure:
>>blocks.  So the Semaphore is signalled.

> So the waiting Process can now execute.  If it wakes up quickly, it will read
> 'value' before the new Process has a chance to write it.  If it takes a long
> time to wake up (like me today ;-) then it'll see the correct value.

> I think that completes the picture.  Whee!!  ;-)

This explains why when debugging and restarting the method
evaluation it worked (sometimes).

Thanks for this, and all the dedication to test and reproduce the
situation.

I can't test your suggested hack to Continuation until I get home,
meanwhile we wait for an authoritative answer from Andy or Blair,
and if the hack (bugfix?) is acceptable, put it into the mainstream
Dolphin x6 first release.

Best regards,

--
Esteban.

"Esteban A. Maringolo" <[hidden email]> wrote in
message news:4373865c$[hidden email]...
>... I can't test your suggested hack to Continuation until I get home,
>meanwhile we wait for an authoritative answer from Andy or Blair, and if
>the hack (bugfix?) is acceptable, put it into the mainstream Dolphin x6
>first release.
>

I'm not really sure what the correct behaviour is here - it goes beyond my
knowledge of how continuations are supposed to work, and I'm afraid I don't
have the time right now to learn more.

All I can say is that nilling out the exceptionEnvironment of the process
only stops the ensure blocks from running because it prevents the process
from being unwound when it is terminated. #ensure: handlers are not linked
into the exception environment chain, but are implemented by special marked
blocks in the stack. Nilling out the exception environment removes the
system's handler at the bottom of the stack which catches the termination
signal, preventing thee process from terminating correctly. This could well
have undesirable effects (or on the other hand it might not).

So my view is that it is not appropriate to include this modification in
X6.0 because:
a) we don't really know what the right behaviour is (are VW/Squeak behaving
correctly, or just in the same way)
b) the full consequences of preventing proper process termination from
taking place need to be considered.
c) there is no time left before the release

Regards

Blair

Esteban,

perhaps it is not to late to show my ignorance... but:

the only place Browse for Containing text WAP is the #ShlwapiLibrary
class, but I have the impression it is not the target of this
discussion., and obviously everyone else knows about it...

So: What is that famous "WAProcess"?

Thanks,
Janos

Hi Janos,

Janos escribió:
> Esteban,
> perhaps it is not to late to show my ignorance... but:
>
> the only place Browse for Containing text WAP is the #ShlwapiLibrary
> class, but I have the impression it is not the target of this
> discussion., and obviously everyone else knows about it...
>
> So: What is that famous "WAProcess"?

In fact, it was a typo, the actual name of the class is
WAProcessMonitor, and is a class of the Seaside Web Framework, which
is already available for Squeak and VisualWorks. I'm porting it to
Dolphin 6, but it is a work in progress yet.

Regards,

--
Esteban.