Global session object?

I wanna have this:

SmalltalkImage>>snapshot: save andQuit: quit
...
resuming ifTrue: [ session := self newSession ].
...

(where session is additional instance variable of SmalltalkImage).

SmalltalkImage>>newSession
"Just answer unique object, which never can be identical to any
previous session object,
this is all what we need for detecting session change.
A session object don't needs to carry any state (we have plenty of
other objects in image which can do this for us). it just needs to be
unique"

  ^ Object new.

SmalltalkImage>>session
 ^ session

then, i can write in own code:

Smalltalk session == mySession ifFalse: [  self initForNewSession.
mySession := Smalltalk session ]

So, basically, this little addition allows you to detect whether
session changed or not between
two different calls to your code (given that you remembered the
previous session somewhere).
Currently we don't have such feature, and looking how some existing
code deals with session management,
i see how it can be simplified.
If you want to do something similar today, you will need to register
in startup list.. which IMO stinks ;)

i don't like using startup lists, since you never know , what is the
right order of resource initialization,
and what inter-dependencies they may form, and changing dependencies
over time will require changing startup order, again and again. Not
fun.
This technique allows you to lazily re-initialize any of your
object(s) due to session change, once the control flow hits your code,
but not before.
(so you don't have to do all accounting at image startup time, you
doing it only when it required/requested, which means shorter image
startup time).

The notion of session is highly important for external resource management.

I using it in NativeBoost from very beginning, and wanna propose to
use it globally.


--
Best regards,
Igor Stasenko.

Hello,

Yes, this sounds good.

        Marcus

On Oct 8, 2012, at 5:11 AM, Igor Stasenko <[hidden email]> wrote:

--
Marcus Denker -- http://marcusdenker.de

Yes, session management is cross-cutting concern, so it makes sense to
have it in the system, and not just as a NB feature.

Object new, is a cool workaround for the unique ID!

Fernando

On Mon, Oct 8, 2012 at 8:33 AM, Marcus Denker <[hidden email]> wrote:

Just from the deleted wiki page on our google code project site:


Squeak startup and shutdown lists are iterated too often, and startup is ignored by the network classes.

= Introduction =
== Squeak/Pharo Problems ==
Squeak's startup and shutdown lists are not used as such.  Each snapshot forces a shutdown and then a startup.  This is not only inefficient, it is unfair to things like open sockets, database connections, etc.  Platform windows will not be well served: imagine shutting down the entire IDE for a snapshot; something needs to be done.

== Dolphin's Solution ==
Dolphin solves the problem through session managers that control the startup and shutdown sequences; they polymorphically adapt to manage deployed GUI applications, the IDE, console applications, etc.  Events triggered off of a singleton session manager sub-instance assist in external resource management.

Dolphin does not free external resources on a snapshot.  Resources are freed on shutdown (by registering for the shutdown events) and pointers/handles are cleared on *startup*.  Having worked with it for years, I have gone from considering it strange to believing it is the correct design.


== First Steps Toward a Fix ==
It became clear that my life would be easier given a singleton SessionManager that triggers #sessionStarted and #sessionStopped.  The open question was whether Squeak's startup and shutdown lists with the quitting and resuming flags would be enough to reproduce Dolphin's behavior, which is to trigger #sessionStarted and #sessionStopped when *and only when* the image starts and stops, respectively.  Somewhat surprisingly, it appears to be.  I am left wondering why people have tolerated the need to litter images with #initializeNetwork sends; that should be done once every time the image starts.


== Code ==
See

  http://code.google.com/p/pharo/issues/detail?id=1200


On 2012-10-08, at 10:41, Fernando Olivero <[hidden email]> wrote:

On 08.10.2012 11:31, Camillo Bruni wrote:
> Just from the deleted wiki page on our google code project site:
>
>
> Squeak startup and shutdown lists are iterated too often, and startup is ignored by the network classes.
>
> = Introduction =
> == Squeak/Pharo Problems ==
> Squeak's startup and shutdown lists are not used as such.  Each snapshot forces a shutdown and then a startup.  This is not only inefficient, it is unfair to things like open sockets, database connections, etc.  Platform windows will not be well served: imagine shutting down the entire IDE for a snapshot; something needs to be done.
This is false, using startUp:/shutDown: serves exactly this purpose.
>
> Dolphin does not free external resources on a snapshot.  Resources are freed on shutdown (by registering for the shutdown events) and pointers/handles are cleared on *startup*.  Having worked with it for years, I have gone from considering it strange to believing it is the correct design.
>
>
> == First Steps Toward a Fix ==
> It became clear that my life would be easier given a singleton SessionManager that triggers #sessionStarted and #sessionStopped.  The open question was whether Squeak's startup and shutdown lists with the quitting and resuming flags would be enough to reproduce Dolphin's behavior, which is to trigger #sessionStarted and #sessionStopped when *and only when* the image starts and stops, respectively.  Somewhat surprisingly, it appears to be.  I am left wondering why people have tolerated the need to litter images with #initializeNetwork sends; that should be done once every time the image starts.
And this is true, you can achieve the same using startUp:/shutDown: as
you can in Dolphin.
Whether that is the best interface, or a manager/announcer would be
better is a different question.

Ironically, the session object Igor describes would necessitate the same
kind of intrusive checks as #initializeNetwork.

IMO, lazily checking whether the image has been restarted whenever you
want to do something stinks.
Registering to be notified when image starts/stops is a lot better.

Cheers,
Henry

On 08.10.2012 13:03, Henrik Sperre Johansen wrote:
>
>
> IMO, lazily checking whether the image has been restarted whenever you
> want to do something stinks.
> Registering to be notified when image starts/stops is a lot better.
Not to mention, less error-prone.
Miss one session check that the current session is the right one
somewhere... and everything might blow up if that is the first piece of
functionality used.

Cheers,
Henry

On 8 October 2012 13:03, Henrik Sperre Johansen
<[hidden email]> wrote:
Then you should do things once at startup.

> IMO, lazily checking whether the image has been restarted whenever you want
> to do something stinks.
> Registering to be notified when image starts/stops is a lot better.
>
I want both. Doing everything at startup time is not necessary ,
especially when you want faster startup time.
And regarding stinks..
how do you think, a following pseudo-code is less stinky:

startup

 MyClassOne allInstancesDo: [:i | i resetExternalHandle ].
 MyOtherClass allInstancesDo: [:i | i clearExternalHandle ].
....
 YetAnOtherClass allInstancesDo: [:i | i reinitExternalHandles ].


(a tip: allInstances  scanning whole heap)


> Cheers,
> Henry

--
Best regards,
Igor Stasenko.

On 8 October 2012 13:06, Henrik Sperre Johansen
<[hidden email]> wrote: True.
I did not said that my method should replace old. Sometimes lazy
initialization is not what you want.
But having no way to perform lazy initialization is even worse.


> Cheers,
> Henry
>



--
Best regards,
Igor Stasenko.

I like your idea, but just so this does not get overlooked:

See also [Stack]Interpreter>>getThisSessionID which is used in e.g.
OSProcessPlugin>>primitiveGetSession for external resource management.
You could move the primitive into the interpreter if you wanted to make
it mandatory.

This is the same session identifier that is used in the file and socket
plugins, so it might have some advantages over using a newly allocated
object that would not get allocated until after the plugins are already
initialized. But it does need VM support, so your Object new approach is
better in that respect.

Dave

On Mon, Oct 08, 2012 at 05:11:34AM +0200, Igor Stasenko wrote:

On 08.10.2012 13:25, Igor Stasenko wrote: Ah, I just objected to the notion that startup lists inherently stinks,
and took the diametrical view for the sake of argument :)
Yes, lazy initialization has its uses too, but some guidelines/advice*
on when to what those uses are, and when you'd use startUp: registration
instead, is needed I think.

Notice; in most cases where such cleanup at startup is needed (sockets,
window handles, etc.), additional cleanup is needed when the object goes
out of scope during regular use, and so a registry usually exist.
Thus, your startup code isn't usually
X allInstancesDo: #something
but
X weakRegistry allEntriesDo: #someting.

which stinks a whole lot less.

Cheers,
Henry


*So in my view, to use lazy init over startUp registration, it would
need to:
- Concern instance data.
- Not need weak cleanup.
- Be containable to a single accessor

On 8 October 2012 13:39, David T. Lewis <[hidden email]> wrote:
yes, i saw that.
The issue with sessionID, generated by VM, that there is a chance of collision:

        globalSessionID := 0.
        [globalSessionID = 0]
                whileTrue: [globalSessionID := self
                                                cCode: 'time(NULL) + ioMSecs()'
                                                inSmalltalk: [(Random new next * SmallInteger maxVal) asInteger]].

while 'Object new' can never collide, no matter what happens :)

Also, according to VM sources, this thing is obsolete, or at least not
used as it supposedly should be used,
like in primitive literal:

Interpreter>>primitiveExternalCall
        "Call an external primitive. The external primitive methods
        contain as first literal an array consisting of:
        * The module name (String | Symbol)
        * The function name (String | Symbol)
        * The session ID (SmallInteger) [OBSOLETE]
        * The function index (Integer) in the externalPrimitiveTable

(i don't remember where, i also seen more code, when hacking HydraVM,
with commented
sections of code which dealt with session id, but then turned into NOP).

Another question, why session id was not exposed to language by VM
(your primitive in OSProcess plugin does that, but since plugin is
optional, the question remains the same).

The proposed change is trivial to introduce and use, but what
important, it don't requires altering VM.
Altering VM in order to fix/improve session management, will
immediately bring new issue: how to play nicely when running images
which expect such feature to be present, while users (for no good
reason, of course ;) using old/obsolete VMs to run image.

This is yet another thing which demonstrates that the less code you
put in VM, the less you have to maintain and care less about
compatibility issues.
That's why i think that VM should be stupid, contain no complex logic
and excessive features, only ones
which having good reason to stay there (either no way how to do same
thing in language, or gives a nice performance advantage).

> Dave

--
Best regards,
Igor Stasenko.

On 8 October 2012 13:41, Henrik Sperre Johansen
<[hidden email]> wrote:
Yes.

> Thus, your startup code isn't usually
> X allInstancesDo: #something
> but
> X weakRegistry allEntriesDo: #someting.
>
> which stinks a whole lot less.
>
in perfect world, yes :)

But world is imperfect..
For example, look here:

FT2Handle>>shutDown: quitting
        "we must not save handles (which are pointers) in the image"
        self clearRegistry.
        FreeTypeFace allInstances do:[:i |
                "destroy any faces that are still being referenced"
                i isValid
                        ifTrue:[i destroyHandle]].
        FT2Handle allSubInstances do: [:h | h beNull]. "if some handle was
not registered"

because of the above, when i pass freetype faces handles to Cairo,
which caching a lot of stuff inside (obviously for performance reasons),
when you save an image, and then try to use Cairo after that (like
rendering text with same font)
an image is crashing.
Which means that "we must not save handles (which are pointers) in the
image" now blocking
me from having nice interoperability with Cairo..

Needless to say that destroying handles will force their recreation
after snapshot, since image
continues running and will keep using freetype fonts.

This is actually the reason why i want to introduce session object(s),
so that i can fix freetype code
to play more nicely with cairo library.




--
Best regards,
Igor Stasenko.

On 8 October 2012 13:41, Henrik Sperre Johansen
[hidden email] wrote:

On 08.10.2012 13:25, Igor Stasenko wrote:

On 8 October 2012 13:06, Henrik Sperre Johansen
[hidden email] wrote:

On 08.10.2012 13:03, Henrik Sperre Johansen wrote:

IMO, lazily checking whether the image has been restarted whenever you
want to do something stinks.
Registering to be notified when image starts/stops is a lot better.

Not to mention, less error-prone.
Miss one session check that the current session is the right one
somewhere... and everything might blow up if that is the first piece of
functionality used.

True.
I did not said that my method should replace old. Sometimes lazy
initialization is not what you want.
But having no way to perform lazy initialization is even worse.

Ah, I just objected to the notion that startup lists inherently stinks, and
took the diametrical view for the sake of argument :)
Yes, lazy initialization has its uses too, but some guidelines/advice* on
when to what those uses are, and when you'd use startUp: registration
instead, is needed I think.

Notice; in most cases where such cleanup at startup is needed (sockets,
window handles, etc.), additional cleanup is needed when the object goes out
of scope during regular use, and so a registry usually exist.

Yes.

Thus, your startup code isn't usually
X allInstancesDo: #something
but
X weakRegistry allEntriesDo: #someting.

which stinks a whole lot less.

in perfect world, yes :)

But world is imperfect..
For example, look here:

FT2Handle>>shutDown: quitting
	"we must not save handles (which are pointers) in the image"
	self clearRegistry.
	FreeTypeFace allInstances do:[:i |
		"destroy any faces that are still being referenced"
		i isValid
			ifTrue:[i destroyHandle]].
	FT2Handle allSubInstances do: [:h | h beNull].	"if some handle was
not registered"

because of the above, when i pass freetype faces handles to Cairo,
which caching a lot of stuff inside (obviously for performance reasons),
when you save an image, and then try to use Cairo after that (like
rendering text with same font)
an image is crashing.
Which means that "we must not save handles (which are pointers) in the
image" now blocking
me from having nice interoperability with Cairo..

Needless to say that destroying handles will force their recreation
after snapshot, since image
continues running and will keep using freetype fonts.

This is actually the reason why i want to introduce session object(s),
so that i can fix freetype code
to play more nicely with cairo library.

I agree with this part from the wiki entry:
"Dolphin does not free external resources on a snapshot. Resources are freed on shutdown (by registering for the shutdown events) and pointers/handles are cleared on *startup*. Having worked with it for years, I have gone from considering it strange to believing it is the correct design."

Rewriting the shutdown to a startUp: method (conditional on the resuming parameter) shouldn't be too much work.

A short analysis of the code reveals:
1) FT2Face/FT2MemoryFaceData aren't instatiated.
2) Both FT2Face and FreeTypeExternalMemory instances are both registered in the FT2Handle registry. (if used in a way which instantiates handle)
3) FT2Library recreates a new instance based for each call to current, which may be problematic, but current return values are never kept, just used to check that FT is present.

So really, cleaning the entries found in registry is all that's actually needed at startup.

Cheers,
Henry

On Mon, Oct 08, 2012 at 04:12:13PM +0200, Igor Stasenko wrote:
>
> Another question, why session id was not exposed to language by VM
> (your primitive in OSProcess plugin does that, but since plugin is
> optional, the question remains the same).

I don't know. Maybe nobody other than me ever wanted to use it ;)

The session identifier is exposed through the InterpreterProxy, so
it is part of the defined interface that is available to plugins.
FilePlugin (and SocketPlugin IIRC) uses this identifier in its data
structure to represent a file, so it provides a check to ensure that
a file reference was created in the current session.

I used this in OSPP because I wanted a separate plugin (not an extension
of FilePlugin or SocketPlugin) that could interact with those same data
structures. I did it that way because I wanted OSPP to be separate
from the VM and the standard plugins, so anybody could build it without
help from a VM guru. And I added the primitive so that I could do as
much of the work in the image as possible. I guess we think alike in
that regard ;)

I agree.

Dave

On Oct 8, 2012, at 4:12 PM, Igor Stasenko wrote:

how can they collide? I do not see it and it should be obvious. I want to learn :)

Go for it igor. But put a nice comment explaining the Object new

On 8 October 2012 21:39, Stéphane Ducasse <[hidden email]> wrote:
They will collide eventually because id is 32-bit number.
So, the max what you can have is 2^32 unique ids , after that you
collide with existing ids inevitably.
But given that  'time(NULL) + ioMSecs()' is far from having good
random distribution,
the collision probability is far more probable than 1/(2^32).

When such collision happens, some objects in your system (which retain
previously generated session id) will think that session is not
changed,
and all bad consequences will come shortly after that :)

Still, those objects usually do not live long enough to have collision.
And so, existing approach actually is good enough.

But still, that's only a probability, while 'Object new' is certainty
that you can never collide, and so
it is simply better.



--
Best regards,
Igor Stasenko.

On 8 October 2012 17:12, Henrik Sperre Johansen
<[hidden email]> wrote: I will put some more reasons for the sake of discussion & learning.

1. during startup, you have to be very careful about dependencies
between services,
the errors like using uninitialized service(s) makes image
unrecoverable (because VM crashing).
For example , putting 'Transcript show: ' before freetype initialized
may kill an image,
beyond the point of recovery. The hardest part in it, that since
everything is late bound, it sometimes really hard to put
initialization in right order. And loading any new code in image will
also contribute to chaos
of interdependencies, unless all developers know that they should not
use facility A before facility B is properly initialized. And the only
way to do it is to add own checks and one more #startup method,
which 'enabling' your service after all dependencies initialized.
But the problem is that the longer chain of initialization gets, the
more chances that something will go wrong (as well as getting lost in
order of dependencies).

2. garbage collection. Yes, you can walk over weak registry to
invalidate all handles before they having
any chance to be used. Unless during startup, you losing sole
reference to object in registry and then VM triggers GC _before_
entering your startup code. As result , finalization code will attempt
to free non-existing resources.. Which in most cases also leads to VM
crash.

So, i would not agree that using lazy-initialization & check before
use is more error prone than startup :)



--
Best regards,
Igor Stasenko.

On 09.10.2012 04:30, Igor Stasenko wrote: WTF?
Dependencies amongst services is one thing, but process interaction
during startup/shutdown should *not* be a source of problems.
(I assume show: triggers an invalidateRect:, which pokes the UI thread
to do an update. Otherwise, show: somehow triggers a UI update directly,
which is a bug)
I mean, if the snapshot isn't the sole running process/ running at
highest priority, all manner of strange things could potentially happen,
both during/after shutdown and before/during startup.
When it comes to inter-dependent services; there's rarely dependencies
between those dealing with cleanup of external resources.
Startup services dependent on these, and eachother, sure.
Again, is startUp: and the startup list the right/best abstractions?
Perhaps not. Perhaps a DSL to declare dependencies would be nice, but
then you're quickly moving into metacello-style complexity instead. How would this ever happen in normal uses?
Does the snapshot primitive not run a GC before saving the image state?
In which case, you'd have to explicitly dereference things that is
supposed to be cleaned up as a later part of startup, and then trigger a
GC...
Maybe it's just me, but I'd characterize that as an edge case that can
be classified under PEBKAC rather than the process itself being error-prone.

Cheers,
Henry

On 9 October 2012 13:21, Henrik Sperre Johansen
<[hidden email]> wrote:
Remember lately people was reporting problems because they were doing
image snapshot
in background process?
I guess we should use fork during snapshot to ensure there's no chance
for scheduler to interfere.
But this is just one thing..
The problem is that the bugs like that will keep popping out over and
over again, which will require
fixes over and over again. While i would prefer having code which is
resilient to such stupid bugs,
and don't needs to be modified that often.

and that's why i use session-change detection in my code.
Then dependents of my service don't needs to be hard-coupled with it,
and can use it at any moment.

it does, but that doesn't matters. I talking about what happens at boot time,
during startup phase: some code runs _before_ reaching your startup
code, which invalidates
all external handles. And that code may:
 - lose the last reference to your object(s)
 - trigger GC

unless you make your invalidation code to run first, things like that
could happen.
(but then someone else would also like to run his own code first, and
we start over again ;) ).

> In which case, you'd have to explicitly dereference things that is supposed
> to be cleaned up as a later part of startup, and then trigger a GC...
> Maybe it's just me, but I'd characterize that as an edge case that can be
> classified under PEBKAC rather than the process itself being error-prone.
>
What is PEBKAC?

That's the problem: exactly because it is an edge case.
Because since everyone assumes that "this cannot normally happen",
when it happens,
you are helpless, because you would never think that problem in that
place and will look into
something unrelated, means that bugs of such kinds will be very hard
to reproduce and fix.

Answering your question, how this could happen.. a simplest example
which comes into my mind is
semaphores:
imagine that you have process which waits on semaphore. And it could
happen that process stack will hold a sole reference to your object,
which represents an external resource.

Now, it could also happen that semaphore which blocking process
belongs to service which will be initialized first during image
startup phase.
And initialization of this service can simply drop the reference to
that semaphore in order to replace it with fresh one..
(and this is not extremely rare , if you look into image, many
services doing like that).

And then GC triggers, and the old semaphore and the process which was
waiting on it will be collected,
and as result, losing last strong reference to the object which
represents your external resource, registered in weak registry.
The next thing which happens is running finalization process which
will try to finalize non-existing external resource.

> Cheers,
> Henry


--
Best regards,
Igor Stasenko.

Igor Stasenko wrote

1. during startup, you have to be very careful about dependencies
between services,
the errors like using uninitialized service(s) makes image
unrecoverable (because VM crashing).
For example , putting 'Transcript show: ' before freetype initialized
may kill an image,
beyond the point of recovery. The hardest part in it, that since
everything is late bound, it sometimes really hard to put
initialization in right order. And loading any new code in image will
also contribute to chaos
of interdependencies, unless all developers know that they should not
use facility A before facility B is properly initialized. And the only
way to do it is to add own checks and one more #startup method,
which 'enabling' your service after all dependencies initialized.
But the problem is that the longer chain of initialization gets, the
more chances that something will go wrong (as well as getting lost in
order of dependencies).

Just shooting from the hip, isn't that the same problem that more or less all unix os-es have on start up, what parts of systems are alive at particular moment and which ones are not. Instead of hard wiring all dependencies, maybe introduce different run levels, and roughly divide registered start up actions based on the run level?