Process-specific state broken and uncomplete

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Process-specific state broken and uncomplete

Igor Stasenko
Hello,

i just found that Squeak images Process class contains unused ivars:
island env

in Pharo image, env ivar is used for holding a process-specific state,
which is convenient, so one could use:
Processor activeProcess environmentAt: put:...

but in Squeak image there is even no such methods.

There are also some caveats with such thing:
during process termination, Processor activeProcess could point to
different process,
and so, accessing to process environment may lead to error(s).

To avoid issues like these, the process termination action (stack
unwinding) should be always performed only for active process.
One of a trick i think, is to replace the suspendedContext with own
context, and then activate given process in order to unwind its stack
etc.

Another way is to remember environment somewhere else during process
termination, and use different method to access it i.e. instead of:
Processor activeProcess environmentAt:
use
self processEnvironmentAt:

where #processEnvironmentAt: is implemented in Object class, and can
handle termination gracefully.

But i think replacing suspendedContext is more generic.
What you think?

--
Best regards,
Igor Stasenko AKA sig.

Reply | Threaded
Open this post in threaded view
|

Re: Process-specific state broken and uncomplete

Levente Uzonyi-2
On Mon, 8 Nov 2010, Igor Stasenko wrote:

> Hello,
>
> i just found that Squeak images Process class contains unused ivars:
> island env

IIRC those were added by Matthew Fulmer to support ProcessSpecific _and_
Cobalt. The idea is that the two can be used together this way, because
the definition of Process doesn't have to be changed.

>
> in Pharo image, env ivar is used for holding a process-specific state,
> which is convenient, so one could use:
> Processor activeProcess environmentAt: put:...
>
> but in Squeak image there is even no such methods.

So they integrated the full ProcessSpecific package. Is it a problem that
you have to load it into Squeak?


Levente

>
> There are also some caveats with such thing:
> during process termination, Processor activeProcess could point to
> different process,
> and so, accessing to process environment may lead to error(s).
>
> To avoid issues like these, the process termination action (stack
> unwinding) should be always performed only for active process.
> One of a trick i think, is to replace the suspendedContext with own
> context, and then activate given process in order to unwind its stack
> etc.
>
> Another way is to remember environment somewhere else during process
> termination, and use different method to access it i.e. instead of:
> Processor activeProcess environmentAt:
> use
> self processEnvironmentAt:
>
> where #processEnvironmentAt: is implemented in Object class, and can
> handle termination gracefully.
>
> But i think replacing suspendedContext is more generic.
> What you think?
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>
>

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Process-specific state broken and uncomplete

Igor Stasenko
On 9 November 2010 03:31, Levente Uzonyi <[hidden email]> wrote:

> On Mon, 8 Nov 2010, Igor Stasenko wrote:
>
>> Hello,
>>
>> i just found that Squeak images Process class contains unused ivars:
>> island env
>
> IIRC those were added by Matthew Fulmer to support ProcessSpecific _and_
> Cobalt. The idea is that the two can be used together this way, because the
> definition of Process doesn't have to be changed.
>
Sure.

>>
>> in Pharo image, env ivar is used for holding a process-specific state,
>> which is convenient, so one could use:
>> Processor activeProcess environmentAt: put:...
>>
>> but in Squeak image there is even no such methods.
>
> So they integrated the full ProcessSpecific package. Is it a problem that
> you have to load it into Squeak?
>
>
So, its a separate package, except that ivars already there..
Strange choice. Maybe it worth mentioning then in Process comment that
these vars are there for purpose?
Otherwise someone may attempt to remove them :)

> Levente
>
>>
>> There are also some caveats with such thing:
>> during process termination, Processor activeProcess could point to
>> different process,
>> and so, accessing to process environment may lead to error(s).
>>
>> To avoid issues like these, the process termination action (stack
>> unwinding) should be always performed only for active process.
>> One of a trick i think, is to replace the suspendedContext with own
>> context, and then activate given process in order to unwind its stack
>> etc.
>>
>> Another way is to remember environment somewhere else during process
>> termination, and use different method to access it i.e. instead of:
>> Processor activeProcess environmentAt:
>> use
>> self processEnvironmentAt:
>>
>> where #processEnvironmentAt: is implemented in Object class, and can
>> handle termination gracefully.
>>
>> But i think replacing suspendedContext is more generic.
>> What you think?
>>
>> --
>> Best regards,
>> Igor Stasenko AKA sig.
>>
>>
>
>



--
Best regards,
Igor Stasenko AKA sig.

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Process-specific state broken and uncomplete

Levente Uzonyi-2
On Tue, 9 Nov 2010, Igor Stasenko wrote:

> On 9 November 2010 03:31, Levente Uzonyi <[hidden email]> wrote:

snip

>>
>> So they integrated the full ProcessSpecific package. Is it a problem that
>> you have to load it into Squeak?
>>
>>
> So, its a separate package, except that ivars already there..

Yes, it's on SqueakSource.

> Strange choice. Maybe it worth mentioning then in Process comment that
> these vars are there for purpose?
> Otherwise someone may attempt to remove them :)

Good idea.


Levente

snip

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Process-specific state broken and uncomplete

Igor Stasenko
In reply to this post by Igor Stasenko
On 9 November 2010 05:59, Levente Uzonyi <[hidden email]> wrote:

> On Tue, 9 Nov 2010, Igor Stasenko wrote:
>
> snip
>
>> Suppose you have a domain object which provides following ways to
>> access the session:
>>
>> session: forkedSession
>>   Processor activeProcess environmentAt: #session put: forkedSession
>>
>> session
>>   ^ Processor activeProcess environmentAt: #session
>>
>>
>> now, here what i'd like to write:
>>
>> | session |
>>
>> session := MySession new .
>> session initialize blabla.
>>
>> "good fork"
>> good := [
>>  self session: session.
>>  [ self process stuff ] ensure: [ session close ]
>> ] newProcess.
>>
>> good resume.
>> 1 minute asDelay wait.
>> good terminate.
>> self assert: (session isClosed).
>>
>> In the above case, everything is ok: session object is passed in 'session'
>> temp
>> and visible inside ensure block directly, so, no matter what may
>> happen (process termination),
>> it is guaranteed that session will be closed at the end.
>>
>> "bad fork"
>>
>> session := MySession new .
>> session initialize blabla.
>>
>> bad := [
>>  self session: session.
>>  [ self process stuff ] ensure: [ self session close ]
>> ] newProcess.
>>
>> bad resume.
>> 1 minute asDelay wait.
>> bad terminate.
>> self assert: (session isClosed).
>>
>>
>> In this case, a session is accessed indirectly, through process specific
>> state.
>> And test might fail or not, depending if process was already
>> terminated before sending #terminate,
>> or its still running.
>>
>> The 'bad fork' illustrates that one should not attempt to access a
>> process specific state in ensure blocks,
>> because during process termination, stack unwinding is performed by
>> other process, which could have different session object,
>> or don't have it at all.
>>
>> This is simple, of course, to implement session finalization without
>> using process spefic state. But now consider that session has not only
>> #close method, but allows doing something more userful, like:
>>
>> self session backupStuff.
>> [   .. do something.. ] ensure: [ self session restoreStuff ].
>>
>> And again, same problem: unless you have a direct reference to session
>> object, you can't do it.
>> Not with using process-specific state, because it is flawed.
>>
>> In other words, anything which may want to access to a
>> process-specific state, like:
>>
>> [ ... ... ] ensure: [ (Processor activeProcess environmentAt: #foo)
>> doSomething ]
>>
>> have no guarantees to work correctly in all cases.
>>
>> This defeats a purpose of process-specific state, because you can't
>> rely on it for building a reliable stuff.
>
> Not really. You can store the sessions in a session store which is
> responsible for retrieving the session:
>
> [
>        sessionStore useSession: sessionKey during: [ :session |
>                Processor activeProcess environmentAt: #session put: session.
>                self doSomething.
>                Processor activeProcess environmentAt: #session put: nil ] ]
> fork.
>
> SessionStore >> useSession: sessionKey during: aBlock
>
>        | session |
>        [
>                session := self sessionFor: sessionKey.
>                aBlock value: session ]
>                ensure: [
>                        session ifNotNil: [
>                                self releaseSession: session ] ]
>
> In #doSomething you can access the session via the process. We're using this
> pattern in PostgresV3's connection pool. It works pretty well.
>
No, you can't. I feel sorry for you :)

A #useSession:during: using ensure, and of course it will work
correctly, because you not accessing session via
process-specific storage. This is illustrated by my "good fork" example.

now replace #doSomething with this:

[ 1 year asDelay wait ] ensure: [
  self assert:  (Processor activeProcess environmentAt: #session) notNil ]

and then terminate a process which running your fork.

In attachment you'll find a test, which shows the problem (i using
tests from Pharo image).


>
> Levente
>
>
> snip
>
>



--
Best regards,
Igor Stasenko AKA sig.

ProcessSpecificTest-testLocalVariableInEnsureBlock.st (692 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Process-specific state broken and uncomplete

Levente Uzonyi-2
On Tue, 9 Nov 2010, Igor Stasenko wrote:

snip

>
> No, you can't. I feel sorry for you :)
>
> A #useSession:during: using ensure, and of course it will work
> correctly, because you not accessing session via
> process-specific storage. This is illustrated by my "good fork" example.

Erm, no. In both of your examples the session wasn't aquired by the forked
process. It was available via a variable. If you aquire the session _in_
the forked process _from_ the session store, then the #ensure: block in
the session store will be able to retrieve it.

>
> now replace #doSomething with this:
>
> [ 1 year asDelay wait ] ensure: [
>  self assert:  (Processor activeProcess environmentAt: #session) notNil ]
>
> and then terminate a process which running your fork.

Replaced (in theory) and terminated. What will happen? This assertion in
this #ensure: block will fail, but the session will be retrieved properly
by the session store's unwind block.

So you can use process local variables, but you shouldn't access them
during unwinding (#ensure:, #ifCurtailed:, whatever). This is possible as
I showed it in my example.

>
> In attachment you'll find a test, which shows the problem (i using
> tests from Pharo image).

The test fails, because you're accessing a process local variable in an
unwind block, which you shouldn't do.


Levente

>
>
>>
>> Levente
>>
>>
>> snip
>>
>>
>
>
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Process-specific state broken and uncomplete

Igor Stasenko
On 9 November 2010 06:42, Levente Uzonyi <[hidden email]> wrote:

> On Tue, 9 Nov 2010, Igor Stasenko wrote:
>
> snip
>
>>
>> No, you can't. I feel sorry for you :)
>>
>> A #useSession:during: using ensure, and of course it will work
>> correctly, because you not accessing session via
>> process-specific storage. This is illustrated by my "good fork" example.
>
> Erm, no. In both of your examples the session wasn't aquired by the forked
> process. It was available via a variable. If you aquire the session _in_ the
> forked process _from_ the session store, then the #ensure: block in the
> session store will be able to retrieve it.
>

this doesn't changes anything. The point is that once you accessing
_any_ process-specific state in ensure
block, you're in trouble. And of course, if you capturing such state
before entering ensure block, you are safe.
But this not always an option.

>>
>> now replace #doSomething with this:
>>
>> [ 1 year asDelay wait ] ensure: [
>>  self assert:  (Processor activeProcess environmentAt: #session) notNil ]
>>
>> and then terminate a process which running your fork.
>
> Replaced (in theory) and terminated. What will happen? This assertion in
> this #ensure: block will fail, but the session will be retrieved properly by
> the session store's unwind block.
>
> So you can use process local variables, but you shouldn't access them during
> unwinding (#ensure:, #ifCurtailed:, whatever). This is possible as I showed
> it in my example.
>
i didn't said this is not possible.

>>
>> In attachment you'll find a test, which shows the problem (i using
>> tests from Pharo image).
>
> The test fails, because you're accessing a process local variable in an
> unwind block, which you shouldn't do.
>

Why i shouldn't?
Give me the reasons why. ("because it works that way" is not a good answer ;)
It sounds like you feel completely happy with current status quo, and
don't want to address that problem.

I don't know how else to convince you that this is wrong.
Suppose you wrote a library, which using a process-local state, buried
somewhere deeply.
Users of your library might even be not aware of it, because they will
be using a higher-level api.
A developer who writes own code, sure thing, thinks that he is free to
use #ensure: #ifCurtailed:
whereever he needs to, isnt?
But with stuff like that, you putting him in the position, that he
should be aware of such subtle nuances, and be very careful about
using your api inside ensure blocks.

This is really beyond my understanding. Why we should not provide a
consistent API instead?

>
> Levente
>



--
Best regards,
Igor Stasenko AKA sig.

Reply | Threaded
Open this post in threaded view
|

Re: Process-specific state broken and uncomplete

Philippe Marschall-2-3
In reply to this post by Igor Stasenko
On 08.11.2010 06:39, Igor Stasenko wrote:

> Hello,
>
> i just found that Squeak images Process class contains unused ivars:
> island env
>
> in Pharo image, env ivar is used for holding a process-specific state,
> which is convenient, so one could use:
> Processor activeProcess environmentAt: put:...
>
> but in Squeak image there is even no such methods.
>
> There are also some caveats with such thing:
> during process termination, Processor activeProcess could point to
> different process,
> and so, accessing to process environment may lead to error(s).
>
> To avoid issues like these, the process termination action (stack
> unwinding) should be always performed only for active process.
> One of a trick i think, is to replace the suspendedContext with own
> context, and then activate given process in order to unwind its stack
> etc.
>
> Another way is to remember environment somewhere else during process
> termination, and use different method to access it i.e. instead of:
> Processor activeProcess environmentAt:
> use
> self processEnvironmentAt:
>
> where #processEnvironmentAt: is implemented in Object class, and can
> handle termination gracefully.
>
> But i think replacing suspendedContext is more generic.
> What you think?

Not so convinced. Fast thread locals are important and a Symbol lookup
in an IdentityDictionary is pretty fast. And polluting Object with stuff
that doesn't belong there isn't a very convincing alternative to me.

Cheers
PHilippe


Reply | Threaded
Open this post in threaded view
|

Re: Process-specific state broken and uncomplete

Igor Stasenko
On 9 November 2010 08:06, Philippe Marschall <[hidden email]> wrote:

> On 08.11.2010 06:39, Igor Stasenko wrote:
>> Hello,
>>
>> i just found that Squeak images Process class contains unused ivars:
>> island env
>>
>> in Pharo image, env ivar is used for holding a process-specific state,
>> which is convenient, so one could use:
>> Processor activeProcess environmentAt: put:...
>>
>> but in Squeak image there is even no such methods.
>>
>> There are also some caveats with such thing:
>> during process termination, Processor activeProcess could point to
>> different process,
>> and so, accessing to process environment may lead to error(s).
>>
>> To avoid issues like these, the process termination action (stack
>> unwinding) should be always performed only for active process.
>> One of a trick i think, is to replace the suspendedContext with own
>> context, and then activate given process in order to unwind its stack
>> etc.
>>
>> Another way is to remember environment somewhere else during process
>> termination, and use different method to access it i.e. instead of:
>> Processor activeProcess environmentAt:
>> use
>> self processEnvironmentAt:
>>
>> where #processEnvironmentAt: is implemented in Object class, and can
>> handle termination gracefully.
>>
>> But i think replacing suspendedContext is more generic.
>> What you think?
>
> Not so convinced. Fast thread locals are important and a Symbol lookup
> in an IdentityDictionary is pretty fast. And polluting Object with stuff
> that doesn't belong there isn't a very convincing alternative to me.
>
does that means that you prefer to replace suspendedContext?

> Cheers
> PHilippe
>
>
>



--
Best regards,
Igor Stasenko AKA sig.

Reply | Threaded
Open this post in threaded view
|

Re: Process-specific state broken and uncomplete

Philippe Marschall-2-3
On 09.11.2010 07:17, Igor Stasenko wrote:

> On 9 November 2010 08:06, Philippe Marschall <[hidden email]> wrote:
>> On 08.11.2010 06:39, Igor Stasenko wrote:
>>> Hello,
>>>
>>> i just found that Squeak images Process class contains unused ivars:
>>> island env
>>>
>>> in Pharo image, env ivar is used for holding a process-specific state,
>>> which is convenient, so one could use:
>>> Processor activeProcess environmentAt: put:...
>>>
>>> but in Squeak image there is even no such methods.
>>>
>>> There are also some caveats with such thing:
>>> during process termination, Processor activeProcess could point to
>>> different process,
>>> and so, accessing to process environment may lead to error(s).
>>>
>>> To avoid issues like these, the process termination action (stack
>>> unwinding) should be always performed only for active process.
>>> One of a trick i think, is to replace the suspendedContext with own
>>> context, and then activate given process in order to unwind its stack
>>> etc.
>>>
>>> Another way is to remember environment somewhere else during process
>>> termination, and use different method to access it i.e. instead of:
>>> Processor activeProcess environmentAt:
>>> use
>>> self processEnvironmentAt:
>>>
>>> where #processEnvironmentAt: is implemented in Object class, and can
>>> handle termination gracefully.
>>>
>>> But i think replacing suspendedContext is more generic.
>>> What you think?
>>
>> Not so convinced. Fast thread locals are important and a Symbol lookup
>> in an IdentityDictionary is pretty fast. And polluting Object with stuff
>> that doesn't belong there isn't a very convincing alternative to me.
>>
> does that means that you prefer to replace suspendedContext?

I prefer to keep env. I can live with the current limitations. Process
termination is just so tricky (as other languages found out as well)
that it's IMHO better to simply avoid it (like finalization) instead of
coming up with some clever hacks to fix it.

VW for example has three or four methods to terminate a process.

Cheers
Philippe