Why do we process the shutdown list if not snapshotting?

Hi, there!

Are there any reasons to process the shutdown list if we just quit without saving/snapshotting? At the moment, we do so...

See SmalltalkImage >> snapshot:andQuit:withExitCode:embedded:

Best,
Marcel

Hi Marcel,

    IMO yes.  Anything that needs t flush and close (eg a database connexion?) would want this.  What's /not/ needed is compressing all the bitmap D.  So shutdown could do with an argument that reflects if saving or quitting

_,,,^..^,,,_ (phone)

Hi Eliot,

we can distinguish between "resuming" and "quitting." For the latter, however, we cannot know whether that quitting makes a snapshot or not.

I think we need another argument.

Best,
Marcel

On Sun, 17 Apr 2016, marcel.taeumel wrote:

> Hi Eliot,
>
> we can distinguish between "resuming" and "quitting." For the latter,
> however, we cannot know whether that quitting makes a snapshot or not.
>
> I think we need another argument.

That would be nice. I hate when RFB nags me when I just want to quit the
image.

Levente

> That would be nice. I hate when RFB nags me when I just want to quit the
> image.

You can use the "flatline your image" desktop hot-key for that, Command+Shift+_.

Hi Marcel,

    I don't understand.  When one comes to quit the system either one saves and quits or one quits.  In the example of forms, if one saves and quits compression is necessary, but if one merely quits it isn't.  The argument needs to distinguish between these two cases.

_,,,^..^,,,_ (phone)

On 18.04.2016, at 07:23, Eliot Miranda <[hidden email]> wrote:

> Hi Marcel,
>
>    I don't understand.  When one comes to quit the system either one saves and quits or one quits.  In the example of forms, if one saves and quits compression is necessary, but if one merely quits it isn't.  The argument needs to distinguish between these two cases.

There's still saving and not quitting...

Hi there,

there are three different scenarios for shutdown:

1. Save and resume
2. Save and quit
3. Quit only

For general clean-up routines, 1 and 2 should clean-up, while 3 does not have to. So, one variable might be sufficient. :-)

Any other ways of using shutdown?

Best,
Marcel

Oh, "save and resume" would not delete a temp folder from disk while "save and quit" would do so. Hmm... we need both variables.

Best,
Marcel

No, you need three variables. Your scenarios are correct, I think, but 1 and 2 need to differentiate. If you're saving and quitting you want to compress forms and remove tmpdirs and such, but if you're saving and keeping the image open you want to skip the tmpdir removal, not close open sockets, maybe only set a variable for startup to re-init that stuff...

marcel.taeumel wrote

Hi there,

there are three different scenarios for shutdown:

1. Save and resume
2. Save and quit
3. Quit only

For general clean-up routines, 1 and 2 should clean-up, while 3 does not have to. So, one variable might be sufficient. :-)

Any other ways of using shutdown?

Best,
Marcel

Hi Tim, Tobias & Marcel,

> On Apr 18, 2016, at 12:47 AM, timfelgentreff <[hidden email]> wrote:
>
> No, you need three variables. Your scenarios are correct, I think, but 1 and
> 2 need to differentiate. If you're saving and quitting you want to compress
> forms and remove tmpdirs and such, but if you're saving and keeping the
> image open you want to skip the tmpdir removal, not close open sockets,
> maybe only set a variable for startup to re-init that stuff...

Ok so perhaps there are three states, but that doesn't imply three variables.  One would suffice if a symbol were used: #quit #save #saveAndQuit.

But is it feasible to ignore #save and expect startUp: to deal with the deferred clean up?  #save is not, and usefully /should not/ be a shut down.  To exit the system, any save must be followed by either a #quit or a #saveAndQuit.  So IMO #save should not shutDown:.  Here are two motivating examples

1. Use of #save to produce a checkpoint, or produce an image to debug some issue at a later date.  In this case the image should be as close to the running image as possible.  Not shutting down in #save is required here.

2. Cleaning up C pointers.  It is common enough for applications to allocate external memory and refer to it through pointer objects (IIRC, Alien & ExternalAddress in Squeak et al, CPointer in VW).  These /cannot/ be released on #save, because doing so would force shutting down of external code that is using that memory.  So instead very early on startUp: all such referents are visited (eg via allInstances, or by enumerating a weak set) and their references are nilled (since the memory is released implicitly on #quit).

So IMO #save should /not/ run either the shutdown list or the startup list in the saving image, and should only run the startup list in the new resuming image.

This may require sophistication on behalf of the programmer, but the start up and shut down lists are indeed nontrivial things to program; they're often maintaining critical infrastructure such as the delay system, and total ordering is extremely important.  Good comments can help.

If you look at VW, you'll see use of pragmas to provide better plug fabulist in terms of providing a priority for ordering, and IIRC for prerequisites.  We could consider a similar approach.


_,,,^..^,,,_ (phone)
Eliot

Hi

2016-04-18 8:40 GMT+02:00 marcel.taeumel <[hidden email]>:

Hi there,

there are three different scenarios for shutdown:

1. Save and resume
2. Save and quit
3. Quit only

For general clean-up routines, 1 and 2 should clean-up, while 3 does not
have to. So, one variable might be sufficient. :-)

Any other ways of using shutdown?

But good practice is to close DB connections when app is closing for example

Hi Denis,

On Apr 18, 2016, at 1:07 AM, Denis Kudriashov <[hidden email]> wrote:

Hi

2016-04-18 8:40 GMT+02:00 marcel.taeumel <[hidden email]>:

Hi there,

there are three different scenarios for shutdown:

1. Save and resume
2. Save and quit
3. Quit only

For general clean-up routines, 1 and 2 should clean-up, while 3 does not
have to. So, one variable might be sufficient. :-)

Any other ways of using shutdown?

But good practice is to close DB connections when app is closing for example

Agreed. Bug do you agree that #save is not a time to close, whereas #saveAndQuit and #quit are?

I think this is a place where the decision might best rest with the image user. We could present them (at save time) with a list of open OS resources and ask them what to do with each.
I did work on an application in VW that did not close db connections and it attempted (usually successfully) to transparently re-open them at startup. Also VA has code to recreate the OS widgets on startup.
But this can only happen if specific code/effort is put into it for each kind of resource. And presumably each image, even if it did handle some situations, would not handle all possible ones, so the user, assuming they know what their image capabilities are, would be in the best position to decide.

Florin

> But good practice is to close DB connections when app is closing for example
>
>
> Agreed. Bug do you agree that #save is not a time to close, whereas
> #saveAndQuit and #quit are?

Wait, I don't, for the exact reason Denis mentions above.  When I save
the image, regardless whether its gonna be shut down, Magma will close
down, so that the state of the saved image is not with open files,
sockets, etc.

On a save-and-resume, it then goes ahead and Re-Opens everythiing.
Sure, this forces it to go through its shutdown/startup routine on a
save but... so what?  Couldn't your external memory pointers re-init
themselves similarly?  What is the goal here, to improve performance
of saving the image?

I think the goal here is to allow applications to keep their resources open if snapshotting does not entail quitting. Whatever the reasons may be.

At the moment, we collected several opinions and use cases here, which suggest to provide two boolean variables, maybe "snapshotting" and "quitting", in the shutdown-interface. Then we would have three different flavors:

#shutDown
#shutDown:
#shutDown:and:

Or something like that. Classes can start using the new one if they want to.

Best,
Marcel

On Mon, 18 Apr 2016, Chris Muller wrote:

I guess it's to avoid the unnecessary downtime.

Levente

> On 18-04-2016, at 8:59 AM, marcel.taeumel <[hidden email]> wrote:
>
> I think the goal here is to allow applications to keep their resources open
> if snapshotting does not entail quitting. Whatever the reasons may be.

A generally laudable aim in my view. We should try to make it so that the minimum possible is done when saving a snapshot - I’d even stop hibernating bitmaps personally - but that does entail making the startup code able to do almost all the cleanup and restart. The problem with that is the amount of startup work is already ridiculous.

We could - as a gedankenexperiment - imagine doing nothing at all before the snapshot primitive. What would we need to do on a fresh startup to cope with whatever mess that left in the saved image? Is it noticeably different to what we already do? Can that be reduced?

I can see some things that would almost certainly want to be handled even on a save-and-continue; a database might very well want to do a synchronise, for example. But do sockets and file handles and stuff need to do anything?
I can see value in doing a gc before the save. I think. Maybe not? A gc and deal with dead weak pointers? That would potentially close down any dormant resources, which is something we’d be wanting sorted anyway.

If we’re quitting and saving then it makes sense to do all the important cleanups - close files, sockets, databases, shutdown connections to gpio deamons before the save, since that means the saved image will be cleaner and starting it up will go faster. If we’re quitting and not saving, maybe there are things that no longer need doing amongst the list.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Never do at  compile-time what you can put off till run-time

If you are willing to save a snapshot in a state in which resources
were not cleaned up -- and you're okay with that -- then one of two
things must be true, either:

  - You never intend to actaully consume that snapshot or,
  - you have code which is unnecessarily cleaning up resources when it
doesn't need to, so just remove that code

Is your use-case that you want to do a "quick save" which doesn't
clean up, and then, eventually, a "save and quit" which does clean up?

This assumes the image will never crash?  It assumes you would never
need to re-load the quick-saved image?  If so, then you should either
not save at all, or remove the clean up code.

I think the reasons do matter.  If we can't articulate WHY what we're
doing is a laudable aim, then we're just adding more and more API for
obscure things which no one will ever use.  This does not move us
toward a simpler system that anyone could fully understand.


On Mon, Apr 18, 2016 at 10:59 AM, marcel.taeumel <[hidden email]> wrote: