Saving the image in the background

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Saving the image in the background

Damien Cassou-3
Hi,

I would like to know if the following code is safe:

[SmalltalkImage current snapshot: true andQuit: false]
     forkAt: Processor userBackgroundPriority

Is there any problem if the main process modifies something in the same
time ?


Thank you

Reply | Threaded
Open this post in threaded view
|

RE: Saving the image in the background

Peter Crowther-2
> From: Damien Cassou
> I would like to know if the following code is safe:
>
> [SmalltalkImage current snapshot: true andQuit: false]
>      forkAt: Processor userBackgroundPriority
>
> Is there any problem if the main process modifies something
> in the same time ?

In general it is not safe.  Code before the snapshot puts a number of
objects into a state suitable for recovery after the image restarts, and
code after the snapshot restores them.  You also are forking at
userBackgroundPriority, which means that the system's tidy-up before a
snapshot and recovery after a snapshot could be pre-empted by your own
processes, so you might get errors in the main process from a
freshly-started image that has only partly recovered when restarted.

Your specific case may not trigger any of the problems - follow the code
through the various hooks to see what is affected.

                - Peter

Reply | Threaded
Open this post in threaded view
|

Re: Saving the image in the background

Damien Cassou-3
Peter Crowther wrote:

>> From: Damien Cassou
>> I would like to know if the following code is safe:
>>
>> [SmalltalkImage current snapshot: true andQuit: false]
>>      forkAt: Processor userBackgroundPriority
>>
>> Is there any problem if the main process modifies something
>> in the same time ?
>
> In general it is not safe.  Code before the snapshot puts a number of
> objects into a state suitable for recovery after the image restarts, and
> code after the snapshot restores them.  You also are forking at
> userBackgroundPriority, which means that the system's tidy-up before a
> snapshot and recovery after a snapshot could be pre-empted by your own
> processes, so you might get errors in the main process from a
> freshly-started image that has only partly recovered when restarted.
>
> Your specific case may not trigger any of the problems - follow the code
> through the various hooks to see what is affected.

Is it better if I do not fork at all ?

Reply | Threaded
Open this post in threaded view
|

Re: Saving the image in the background

Philippe Marschall
In reply to this post by Damien Cassou-3
2006/7/30, Damien Cassou <[hidden email]>:
> Hi,
>
> I would like to know if the following code is safe:
>
> [SmalltalkImage current snapshot: true andQuit: false]
>      forkAt: Processor userBackgroundPriority
>
> Is there any problem if the main process modifies something in the same
> time ?

Doesn't thay stall the image anyway? IIRC it makes a full GC with blocks.

If you have OSProcess you can fork the image (search the mailinglist,
Avi once asked about this). I don't know if this now works on Mac OS.
Also pay attention that if you have two forked images at the same
time, that they don't write into the same image file (if you want to
use it for Pier). Otherwise you might end up with a corrupted image
file.

Philippe

Reply | Threaded
Open this post in threaded view
|

Re: Saving the image in the background

David T. Lewis
On Sun, Jul 30, 2006 at 03:04:58PM +0200, Philippe Marschall wrote:
>
> If you have OSProcess you can fork the image (search the mailinglist,
> Avi once asked about this). I don't know if this now works on Mac OS.
> Also pay attention that if you have two forked images at the same
> time, that they don't write into the same image file (if you want to
> use it for Pier). Otherwise you might end up with a corrupted image
> file.

This is done by evaluating "UnixProcess saveImageInBackgroundNicely".
The image will be saved by a copy of your Squeak image running in a
separate OS process at low priority, with little or no impact on your
main Squeak process.

It should also work on OS X if you are using the unix VM for OS X that
supplies the OSPP plugin (I would appreciate if someone can confirm
this). It might also work on the regular Mac VM if you compile your
own OSPP plugin, but I'm not sure of this.

Dave


Reply | Threaded
Open this post in threaded view
|

Re: Saving the image in the background

timrowledge
In reply to this post by Damien Cassou-3

On 30-Jul-06, at 6:02 AM, Damien Cassou wrote:

> Peter Crowther wrote:
>>> From: Damien Cassou
>>> I would like to know if the following code is safe:
>>>
>>> [SmalltalkImage current snapshot: true andQuit: false]
>>>      forkAt: Processor userBackgroundPriority
>>>
>>> Is there any problem if the main process modifies something in  
>>> the same time ?
>> In general it is not safe.  Code before the snapshot puts a number of
>> objects into a state suitable for recovery after the image  
>> restarts, and
>> code after the snapshot restores them.  You also are forking at
>> userBackgroundPriority, which means that the system's tidy-up  
>> before a
>> snapshot and recovery after a snapshot could be pre-empted by your  
>> own
>> processes, so you might get errors in the main process from a
>> freshly-started image that has only partly recovered when restarted.
>> Your specific case may not trigger any of the problems - follow  
>> the code
>> through the various hooks to see what is affected.
>
> Is it better if I do not fork at all ?

I strongly urge you to not try saving an image in a background  
process, except just possibly in a very high priority one. As Peter  
mentions any other priority can be pre-empted and so all that careful  
go-to-sleep code (take a look at the fairly scary list of jobs to be  
done before the actual snapshot) could be completely messed up. That  
could lead to both the active image and the saved snapshot being  
unusable, which is unlikely to please you!

Many years ago somebody at a large US automotive manufacturer put a  
snapshot primitive call into a background process. The resulting  
debugging work at PPS cost my team 5-6 man-months of effort and a lot  
of aggravation. Bad idea.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
"Oh bother" said Pooh, as he reached for the reset button



Reply | Threaded
Open this post in threaded view
|

Re: Saving the image in the background

Klaus D. Witzel
Tim,

let me understand the difference between the "regular" process priority  
when a save/+-/quit is done from a menu and when it is done at  
userBackgroundPriority. In both cases the shutDown list is executed and  
sensors and processes (and other facilities) are made aware that they must  
become quiet now because it must be possible for them to come to live on  
another platform.

If there where any problem(s) at userBackgroundPriority, wouldn't they be  
the same at the UIProcess's userSchedulingPriority?

Thank you in advance.

/Klaus

P.S. I also cannot understand the original issue, the snapshot primitive  
does not run faster or slower in background: garbage collection (part of  
snapshot) and snapshot itself are atomic.

On Sun, 30 Jul 2006 18:09:15 +0200, tim Rowledge <[hidden email]> wrote:

> On 30-Jul-06, at 6:02 AM, Damien Cassou wrote:
>> Peter Crowther wrote:
>>>> From: Damien Cassou
>>>> I would like to know if the following code is safe:
>>>>
>>>> [SmalltalkImage current snapshot: true andQuit: false]
>>>>      forkAt: Processor userBackgroundPriority
>>>>
>>>> Is there any problem if the main process modifies something in the  
>>>> same time ?
>>> In general it is not safe.  Code before the snapshot puts a number of
>>> objects into a state suitable for recovery after the image restarts,  
>>> and
>>> code after the snapshot restores them.  You also are forking at
>>> userBackgroundPriority, which means that the system's tidy-up before a
>>> snapshot and recovery after a snapshot could be pre-empted by your own
>>> processes, so you might get errors in the main process from a
>>> freshly-started image that has only partly recovered when restarted.
>>> Your specific case may not trigger any of the problems - follow the  
>>> code
>>> through the various hooks to see what is affected.
>>
>> Is it better if I do not fork at all ?
>
> I strongly urge you to not try saving an image in a background process,  
> except just possibly in a very high priority one. As Peter mentions any  
> other priority can be pre-empted and so all that careful go-to-sleep  
> code (take a look at the fairly scary list of jobs to be done before the  
> actual snapshot) could be completely messed up. That could lead to both  
> the active image and the saved snapshot being unusable, which is  
> unlikely to please you!
>
> Many years ago somebody at a large US automotive manufacturer put a  
> snapshot primitive call into a background process. The resulting  
> debugging work at PPS cost my team 5-6 man-months of effort and a lot of  
> aggravation. Bad idea.
>
>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> "Oh bother" said Pooh, as he reached for the reset button
>
>
>
>



Reply | Threaded
Open this post in threaded view
|

Re: Saving the image in the background

johnmci
In reply to this post by David T. Lewis
With the mac carbon 3 .8.12 series of VMs you can call any unix DLL  
you have handy. Thus if you
have a Unix OSX version of the OSPP you can just use that, no need to  
compile a *.bundle.


On 30-Jul-06, at 8:25 AM, David T. Lewis wrote:

> On Sun, Jul 30, 2006 at 03:04:58PM +0200, Philippe Marschall wrote:
>>
>> If you have OSProcess you can fork the image (search the mailinglist,
>> Avi once asked about this). I don't know if this now works on Mac OS.
>> Also pay attention that if you have two forked images at the same
>> time, that they don't write into the same image file (if you want to
>> use it for Pier). Otherwise you might end up with a corrupted image
>> file.
>
> This is done by evaluating "UnixProcess saveImageInBackgroundNicely".
> The image will be saved by a copy of your Squeak image running in a
> separate OS process at low priority, with little or no impact on your
> main Squeak process.
>
> It should also work on OS X if you are using the unix VM for OS X that
> supplies the OSPP plugin (I would appreciate if someone can confirm
> this). It might also work on the regular Mac VM if you compile your
> own OSPP plugin, but I'm not sure of this.
>
> Dave
>
>

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===



Reply | Threaded
Open this post in threaded view
|

Re: Saving the image in the background

johnmci
In reply to this post by timrowledge

On 30-Jul-06, at 9:09 AM, tim Rowledge wrote:

> I strongly urge you to not try saving an image in a background  
> process, except just possibly in a very high priority one. As Peter  
> mentions any other priority can be pre-empted and so all that  
> careful go-to-sleep code (take a look at the fairly scary list of  
> jobs to be done before the actual snapshot) could be completely  
> messed up. That could lead to both the active image and the saved  
> snapshot being unusable, which is unlikely to please you!
>
> Many years ago somebody at a large US automotive manufacturer put a  
> snapshot primitive call into a background process. The resulting  
> debugging work at PPS cost my team 5-6 man-months of effort and a  
> lot of aggravation. Bad idea.

Tim, I think the code suggestion from many months ago forks a  
background unix process that does the snapshot. It of course is an  
identical copy of the parent at the time of the fork maintained by  
the magic of virtual memory. The original parent process just keeps  
running and memory updates don't alter what the child process has.  
On a mulitple CPU that tosses the repsonsiblity of saving to another  
process at the cost of memory.


--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===



Reply | Threaded
Open this post in threaded view
|

Re: Saving the image in the background

Gerardo Richarte
In reply to this post by Damien Cassou-3
Damien dijo:
> I would like to know if the following code is safe:
>
> [SmalltalkImage current snapshot: true andQuit: false]
>      forkAt: Processor userBackgroundPriority
>
> Is there any problem if the main process modifies something in the same
> time ?
>
>  
    The problems have already explained, and I've been following this
thread with great interest, because a "similar" problem we have is to
find a way to save the image doing everything from Squeak (suppose you
want, for example, to "save" the image over the net, using FTP,
implementing the FTP protocol in Squeak: the image will be changing
while you do the networking stuff)

    The solution we came up to, and until somebody finds a better way
(please do!), is to make a copy of the image in a String or ByteArray
(from a primitive). This will effectively duplicate the size of the
image, but so far, is the only solution we found.

    This trick may apply to your case, depending on what you really want
to put in background, but as Klaus said, if you want to put the garbage
collection in background, you cannot do it by forking a Squeak process,
as this is a natively atomic operation, and the same goes for storing
the bits in a file. This two can only be forked natively.

    Any better ideas? please?! :-)

    gera

Reply | Threaded
Open this post in threaded view
|

Re: Saving the image in the background

timrowledge
In reply to this post by Klaus D. Witzel

On 30-Jul-06, at 10:28 AM, Klaus D. Witzel wrote:

> Tim,
>
> let me understand the difference between the "regular" process  
> priority when a save/+-/quit is done from a menu and when it is  
> done at userBackgroundPriority. In both cases the shutDown list is  
> executed and sensors and processes (and other facilities) are made  
> aware that they must become quiet now because it must be possible  
> for them to come to live on another platform.

Correct.
>
> If there where any problem(s) at userBackgroundPriority, wouldn't  
> they be the same at the UIProcess's userSchedulingPriority?

Same problem but higher likelihood of them happening, I think. You  
might consider some critical section protection to avoid interruptions.

The problem at PPS I mentioned happened because the customer had a  
background process that ran the snapshot primitive rather than the  
entire snapshot sequence. Thus the snapshot file on disc had a lot of  
objects in indeterminate state. Not a happy thing for a payroll  
processing program.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: CAO: Compare Apples to Oranges



Reply | Threaded
Open this post in threaded view
|

Re: Saving the image in the background

Damien Cassou-3
In reply to this post by Damien Cassou-3
Damien Cassou wrote:
> I would like to know if the following code is safe:
>
> [SmalltalkImage current snapshot: true andQuit: false]
>     forkAt: Processor userBackgroundPriority
>
> Is there any problem if the main process modifies something in the same
> time ?

I'm sorry not having been clear enough. I just wanted to save the image
when a user modify the model. To avoid a long delay for him, I wanted to
use a fork.

But it seems this was not a good idea.

My solution is to remove the fork and let the user wait some seconds :-)

Thank you for all your explanations

Reply | Threaded
Open this post in threaded view
|

Re: Saving the image in the background

Avi  Bryant

On Jul 31, 2006, at 11:01 AM, Damien Cassou wrote:

> Damien Cassou wrote:
>> I would like to know if the following code is safe:
>> [SmalltalkImage current snapshot: true andQuit: false]
>>     forkAt: Processor userBackgroundPriority
>> Is there any problem if the main process modifies something in the  
>> same time ?
>
> I'm sorry not having been clear enough. I just wanted to save the  
> image when a user modify the model. To avoid a long delay for him,  
> I wanted to use a fork.
>
> But it seems this was not a good idea.
>
> My solution is to remove the fork and let the user wait some  
> seconds :-)

Forking a separate Smalltalk process is a bad idea, but forking the  
whole unix process works just fine.  You just need to modify the  
snapshot prim to do this.  There was a long thread about this  
earlier, here's a post with some code:

http://lists.squeakfoundation.org/pipermail/squeak-dev/2005-October/ 
095896.html

I used a VM with that prim for months without any ill effects.

Avi