Smalltalk › Pharo › Pharo Smalltalk Developers

What is correct way to save image automatically in background?

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

12 messages Options

Denis Kudriashov

What is correct way to save image automatically in background?

In my app I need update pharo image with some resources (new package, icons, configs...) remotelly.
After updating I need save image session. For that app show user some special screen, save image and return to ussual app state.

My app has process that do system changes when needed and "commit" it as I wrote.

I do something like

[self blockUI.
self doUpdate.
SmalltalkImage current snapshot: true andQuit: false.
self unblockUI.
] fork

And sometimes after image restart I have image crash due to some world cicle activity and freetype plugin failure.

I try little [SmalltalkImage current snapshot: true andQuit: false] fork code from workspace. And situation same - sometimes image cant restart.

I never see that when I save image manually.

What is correct way to save image for my purpose?

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Lukas Renggli

Re: What is correct way to save image automatically in background?

> What is correct way to save image for my purpose?

Don't fork.

Lukas

--
Lukas Renggli
www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Denis Kudriashov

Re: What is correct way to save image automatically in background?

But How I can do that without fork? I track changes in another process and when I do update I save image in this process.
How I can initiate image saving from ui process?

2010/7/23 Lukas Renggli <[hidden email]>

> What is correct way to save image for my purpose?

Don't fork.

Lukas

--
Lukas Renggli
www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Lukas Renggli

Re: What is correct way to save image automatically in background?

> But How I can do that without fork? I track changes in another process and
> when I do update I save image in this process.
> How I can initiate image saving from ui process?

I don't understand what you are saying.

The thing is that using #fork to save the image has no effect (other
than wrecking the image), because #snapshot:andQuit: is eventually
calling a primitive that locks the complete VM anyway.

If you want to fork while saving the image you need to use an OS-level
fork, not a Smalltalk-level fork. You can do this with OSProcess for
example (check the Seaside mailing-list archive).

Lukas

--
Lukas Renggli
www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Denis Kudriashov

Re: What is correct way to save image automatically in background?

Sorry for not correct thread title.

I dont need save image in background while all system continue work.

I just want save image from my smalltalk process.
And when I do this I sometimes cant restart image (it crash due to some world cycle activity and freetype plugin failure).
But when I save image manually I never see that situation.

Maybe somebody know why this happen? And how I must save image from some smalltalk process correctly?

2010/7/23 Lukas Renggli <[hidden email]>

> But How I can do that without fork? I track changes in another process and
> when I do update I save image in this process.
> How I can initiate image saving from ui process?

I don't understand what you are saying.

The thing is that using #fork to save the image has no effect (other
than wrecking the image), because #snapshot:andQuit: is eventually
calling a primitive that locks the complete VM anyway.

If you want to fork while saving the image you need to use an OS-level
fork, not a Smalltalk-level fork. You can do this with OSProcess for
example (check the Seaside mailing-list archive).

Lukas

--
Lukas Renggli
www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

David T. Lewis

Re: What is correct way to save image automatically in background?

In reply to this post by Lukas Renggli

On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:

> > But How I can do that without fork? I track changes in another process and
> > when I do update I save image in this process.
> > How I can initiate image saving from ui process?
>
> I don't understand what you are saying.
>
> The thing is that using #fork to save the image has no effect (other
> than wrecking the image), because #snapshot:andQuit: is eventually
> calling a primitive that locks the complete VM anyway.
>
> If you want to fork while saving the image you need to use an OS-level
> fork, not a Smalltalk-level fork. You can do this with OSProcess for
> example (check the Seaside mailing-list archive).

"UnixProcess saveImageInBackgroundNicely"

Explained in the method comment:

UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag
"When Squeak is used as a server it is sometimes desirable to periodically
save image snapshots. This method forks a headless Squeak to perform a
snapshot without impacting the server Squeak. Very little additional memory
is required to do this because Unix copy-on-write memory management allows
the two Squeak images to share object memory while the save is performed.
The saved image is given a time stamped name, and the image name of
the main server Squeak remains unchanged. If niceFlag is true, the
background OS process runs at lowered scheduling priority."

Dave

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Lukas Renggli

Re: What is correct way to save image automatically in background?

David, is what is said about memory management really true? I thought
that saving the image triggered a garbage collect what forces to copy
the whole memory anyway.

Lukas

On 23 July 2010 14:00, David T. Lewis <[hidden email]> wrote:

> On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:
>> > But How I can do that without fork? I track changes in another process and
>> > when I do update I save image in this process.
>> > How I can initiate image saving from ui process?
>>
>> I don't understand what you are saying.
>>
>> The thing is that using #fork to save the image has no effect (other
>> than wrecking the image), because #snapshot:andQuit: is eventually
>> calling a primitive that locks the complete VM anyway.
>>
>> If you want to fork while saving the image you need to use an OS-level
>> fork, not a Smalltalk-level fork. You can do this with OSProcess for
>> example (check the Seaside mailing-list archive).
>
> "UnixProcess saveImageInBackgroundNicely"
>
> Explained in the method comment:
>
> UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag
> "When Squeak is used as a server it is sometimes desirable to periodically
> save image snapshots. This method forks a headless Squeak to perform a
> snapshot without impacting the server Squeak. Very little additional memory
> is required to do this because Unix copy-on-write memory management allows
> the two Squeak images to share object memory while the save is performed.
> The saved image is given a time stamped name, and the image name of
> the main server Squeak remains unchanged. If niceFlag is true, the
> background OS process runs at lowered scheduling priority."
>
> Dave
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

--
Lukas Renggli
www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

David T. Lewis

Re: What is correct way to save image automatically in background?

Yes, it really is true. That said, I would expect the practical impact
to vary depending on how busy the main image is.

If you are running a Squeak/Pharo image that is not busy, such as your
own personal desktop image or a Seaside image that is not currently
receiving requests, then the additional system memory used by the
forked background image is extremely small (it's pretty amazing
actually, try watching it on a system monitor such as xosview).
However, I would expect that if you were to do this with a busy
Seaside image, then that image would be doing a lot of updates to
memory and possibly garbage collection, so that will force the OS
to use more memory.

Unfortunately this means that the busier your image is, the less
benefit you get from a memory usage point of view. But even at that,
you still get the advantage of doing the image save in a background
OS process running at lower OS scheduling priority.

I do not actually run any Seaside server images, so someone would
need to try this on a real system to find out how much benefit it
delivers in practice. I would be interested to know the results.

Dave

On Fri, Jul 23, 2010 at 02:12:14PM +0200, Lukas Renggli wrote:

> David, is what is said about memory management really true? I thought
> that saving the image triggered a garbage collect what forces to copy
> the whole memory anyway.
>
> Lukas
>
> On 23 July 2010 14:00, David T. Lewis <[hidden email]> wrote:
> > On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:
> >> > But How I can do that without fork? I track changes in another process and
> >> > when I do update I save image in this process.
> >> > How I can initiate image saving from ui process?
> >>
> >> I don't understand what you are saying.
> >>
> >> The thing is that using #fork to save the image has no effect (other
> >> than wrecking the image), because #snapshot:andQuit: is eventually
> >> calling a primitive that locks the complete VM anyway.
> >>
> >> If you want to fork while saving the image you need to use an OS-level
> >> fork, not a Smalltalk-level fork. You can do this with OSProcess for
> >> example (check the Seaside mailing-list archive).
> >
> > "UnixProcess saveImageInBackgroundNicely"
> >
> > Explained in the method comment:
> >
> > UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag
> > ? ? ? ?"When Squeak is used as a server it is sometimes desirable to periodically
> > ? ? ? ?save image snapshots. This method forks a headless Squeak to perform a
> > ? ? ? ?snapshot without impacting the server Squeak. Very little additional memory
> > ? ? ? ?is required to do this because Unix copy-on-write memory management allows
> > ? ? ? ?the two Squeak images to share object memory while the save is performed.
> > ? ? ? ?The saved image is given a time stamped name, and the image name of
> > ? ? ? ?the main server Squeak remains unchanged. If niceFlag is true, the
> > ? ? ? ?background OS process runs at lowered scheduling priority."
> >
> > Dave
> >
> >
> > _______________________________________________
> > Pharo-project mailing list
> > [hidden email]
> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
> >
>
>
>
> --
> Lukas Renggli
> www.lukas-renggli.ch
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Lukas Renggli

Re: What is correct way to save image automatically in background?

Cool, thanks for the update.

On Friday, July 23, 2010, David T. Lewis <[hidden email]> wrote:

> Yes, it really is true. That said, I would expect the practical impact
> to vary depending on how busy the main image is.
>
> If you are running a Squeak/Pharo image that is not busy, such as your
> own personal desktop image or a Seaside image that is not currently
> receiving requests, then the additional system memory used by the
> forked background image is extremely small (it's pretty amazing
> actually, try watching it on a system monitor such as xosview).
> However, I would expect that if you were to do this with a busy
> Seaside image, then that image would be doing a lot of updates to
> memory and possibly garbage collection, so that will force the OS
> to use more memory.
>
> Unfortunately this means that the busier your image is, the less
> benefit you get from a memory usage point of view. But even at that,
> you still get the advantage of doing the image save in a background
> OS process running at lower OS scheduling priority.
>
> I do not actually run any Seaside server images, so someone would
> need to try this on a real system to find out how much benefit it
> delivers in practice. I would be interested to know the results.
>
> Dave
>
>
> On Fri, Jul 23, 2010 at 02:12:14PM +0200, Lukas Renggli wrote:
>> David, is what is said about memory management really true? I thought
>> that saving the image triggered a garbage collect what forces to copy
>> the whole memory anyway.
>>
>> Lukas
>>
>> On 23 July 2010 14:00, David T. Lewis <[hidden email]> wrote:
>> > On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:
>> >> > But How I can do that without fork? I track changes in another process and
>> >> > when I do update I save image in this process.
>> >> > How I can initiate image saving from ui process?
>> >>
>> >> I don't understand what you are saying.
>> >>
>> >> The thing is that using #fork to save the image has no effect (other
>> >> than wrecking the image), because #snapshot:andQuit: is eventually
>> >> calling a primitive that locks the complete VM anyway.
>> >>
>> >> If you want to fork while saving the image you need to use an OS-level
>> >> fork, not a Smalltalk-level fork. You can do this with OSProcess for
>> >> example (check the Seaside mailing-list archive).
>> >
>> > "UnixProcess saveImageInBackgroundNicely"
>> >
>> > Explained in the method comment:
>> >
>> > UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag
>> > ? ? ? ?"When Squeak is used as a server it is sometimes desirable to periodically
>> > ? ? ? ?save image snapshots. This method forks a headless Squeak to perform a
>> > ? ? ? ?snapshot without impacting the server Squeak. Very little additional memory
>> > ? ? ? ?is required to do this because Unix copy-on-write memory management allows
>> > ? ? ? ?the two Squeak images to share object memory while the save is performed.
>> > ? ? ? ?The saved image is given a time stamped name, and the image name of
>> > ? ? ? ?the main server Squeak remains unchanged. If niceFlag is true, the
>> > ? ? ? ?background OS process runs at lowered scheduling priority."
>> >
>> > Dave
>> >
>> >
>> > _______________________________________________
>> > Pharo-project mailing list
>> > [hidden email]
>> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>> >
>>
>>
>>
>> --
>> Lukas Renggli
>> www.lukas-renggli.ch
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

--
Lukas Renggli
www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Yanni Chiu

Re: What is correct way to save image automatically in background?

In reply to this post by Lukas Renggli

Here's my understanding of how it works. Garbage collection is completed
before the Unix fork system call is made. At the moment of the fork, the
original server VM process and the new image saving VM process share the
exact same pages in the Unix OS - due to "Unix copy-on-write memory
management".

Immediately after the fork, the server VM process continues processing
requests. Meanwhile, the image saving VM process proceeds to save the
image. As each process continues execution, they will "dirty" their
memory pages. Each dirty page means that that page is no longer shared
between the two processes - leading to increased OS memory resources
being allocated to the two processes.

If the server VM process is very active, then a lot of dirty pages will
be created, before the image saving VM process can complete. So, running
the image saving process at a lower priority may actually be detrimental.

One thing that occurs to me now, is whether or not the garbage
collection is completed before the fork. If not, and it's done first
thing by the image saving process, then it will be as you describe - the
whole memory gets copied anyway.

--
Yanni

Lukas Renggli wrote:

> David, is what is said about memory management really true? I thought
> that saving the image triggered a garbage collect what forces to copy
> the whole memory anyway.
>
> Lukas
>
> On 23 July 2010 14:00, David T. Lewis <[hidden email]> wrote:
>> On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:
>>>> But How I can do that without fork? I track changes in another process and
>>>> when I do update I save image in this process.
>>>> How I can initiate image saving from ui process?
>>> I don't understand what you are saying.
>>>
>>> The thing is that using #fork to save the image has no effect (other
>>> than wrecking the image), because #snapshot:andQuit: is eventually
>>> calling a primitive that locks the complete VM anyway.
>>>
>>> If you want to fork while saving the image you need to use an OS-level
>>> fork, not a Smalltalk-level fork. You can do this with OSProcess for
>>> example (check the Seaside mailing-list archive).
>> "UnixProcess saveImageInBackgroundNicely"
>>
>> Explained in the method comment:
>>
>> UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag
>> "When Squeak is used as a server it is sometimes desirable to periodically
>> save image snapshots. This method forks a headless Squeak to perform a
>> snapshot without impacting the server Squeak. Very little additional memory
>> is required to do this because Unix copy-on-write memory management allows
>> the two Squeak images to share object memory while the save is performed.
>> The saved image is given a time stamped name, and the image name of
>> the main server Squeak remains unchanged. If niceFlag is true, the
>> background OS process runs at lowered scheduling priority."
>>
>> Dave

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

NorbertHartl

Re: What is correct way to save image automatically in background?

In reply to this post by David T. Lewis

On 23.07.2010, at 17:11, David T. Lewis wrote:

Yes, it really is true. That said, I would expect the practical impact
to vary depending on how busy the main image is.

If you are running a Squeak/Pharo image that is not busy, such as your
own personal desktop image or a Seaside image that is not currently
receiving requests, then the additional system memory used by the
forked background image is extremely small (it's pretty amazing
actually, try watching it on a system monitor such as xosview).
However, I would expect that if you were to do this with a busy
Seaside image, then that image would be doing a lot of updates to
memory and possibly garbage collection, so that will force the OS
to use more memory.

What you are saying depends on the environment underneath. In theory it is pretty easy. The fork doubles the process memory when forking. Systems like linux have something called "memory overcommit". They don't allocate the reserved memory unless something is written. I think that is what you meant by saying "... is extremely small...".

Why I am saying this? Because especially for seaside images the situation could be different. It is very common nowadays to use OS virtualization to install things like seaside images. Virtualizers disable "memory overcommitment" because otherwise one virtual instance could have side effects on other instances. So you are back with doubling process memory. You need to be careful if you have a small virtual machine or if you try to fork from several images at the same time.

hope this adds anything,

Norbert

Unfortunately this means that the busier your image is, the less
benefit you get from a memory usage point of view. But even at that,
you still get the advantage of doing the image save in a background
OS process running at lower OS scheduling priority.

I do not actually run any Seaside server images, so someone would
need to try this on a real system to find out how much benefit it
delivers in practice. I would be interested to know the results.

Dave

On Fri, Jul 23, 2010 at 02:12:14PM +0200, Lukas Renggli wrote:
David, is what is said about memory management really true? I thought
that saving the image triggered a garbage collect what forces to copy
the whole memory anyway.

Lukas

On 23 July 2010 14:00, David T. Lewis <[hidden email]> wrote:
On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:
But How I can do that without fork? I track changes in another process and
when I do update I save image in this process.
How I can initiate image saving from ui process?

I don't understand what you are saying.

The thing is that using #fork to save the image has no effect (other
than wrecking the image), because #snapshot:andQuit: is eventually
calling a primitive that locks the complete VM anyway.

If you want to fork while saving the image you need to use an OS-level
fork, not a Smalltalk-level fork. You can do this with OSProcess for
example (check the Seaside mailing-list archive).

"UnixProcess saveImageInBackgroundNicely"

Explained in the method comment:

UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag
? ? ? ?"When Squeak is used as a server it is sometimes desirable to periodically
? ? ? ?save image snapshots. This method forks a headless Squeak to perform a
? ? ? ?snapshot without impacting the server Squeak. Very little additional memory
? ? ? ?is required to do this because Unix copy-on-write memory management allows
? ? ? ?the two Squeak images to share object memory while the save is performed.
? ? ? ?The saved image is given a time stamped name, and the image name of
? ? ? ?the main server Squeak remains unchanged. If niceFlag is true, the
? ? ? ?background OS process runs at lowered scheduling priority."

Dave

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

--
Lukas Renggli
www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

johnmci

Re: What is correct way to save image automatically in background?

In reply to this post by Yanni Chiu

On 2010-07-23, at 9:03 AM, Yanni Chiu wrote:

> Here's my understanding of how it works. Garbage collection is completed before the Unix fork system call is made. At the moment of the fork, the original server VM process and the new image saving VM process share the exact same pages in the Unix OS - due to "Unix copy-on-write memory management".
>
> Immediately after the fork, the server VM process continues processing requests. Meanwhile, the image saving VM process proceeds to save the image. As each process continues execution, they will "dirty" their memory pages. Each dirty page means that that page is no longer shared between the two processes - leading to increased OS memory resources being allocated to the two processes.
>
> If the server VM process is very active, then a lot of dirty pages will be created, before the image saving VM process can complete. So, running the image saving process at a lower priority may actually be detrimental.
>
> One thing that occurs to me now, is whether or not the garbage collection is completed before the fork. If not, and it's done first thing by the image saving process, then it will be as you describe - the whole memory gets copied anyway.
>
> --
> Yanni

Let me clarify a few things.

First the Copy On Write only makes a copy of the original page if the page is going to be written to. Now *usually* 99% of the write activity is in young/new space which starts with zero bytes of memory after the Full GC is triggered. So as the saving task is writing out say the 0-100MB of memory, the running VM is busy allocating and writing to objects at 100+MB so there will be no effect on pages 0-100MB. Of course there are some updates into the 0-100MB area but that is rare as compared to write activity in new space.

So if the process doing the write runs at a lower priority it shouldn't have an impact on the process running the VM.

The danger is that a full GC should occur which would cause a compaction, but a server image should not be doing full GC more than a few times per hour.... Maybe never...

--
===========================================================================
John M. McIntosh <[hidden email]> Twitter: squeaker68882
Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com
===========================================================================

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

smime.p7s (3K) Download Attachment