VM freeze and crash when saving

Hi,
   I have experienced some problems with my production image and VM.
   I use a unixshell server (Linux 2.6.16.13-xenU #1 SMP Sat May 13
14:18:58 EDT 2006 i686 GNU/Linux).
   My VM is the last one I got using an apt-get install squeakvm :
$ squeakvm -version
3.9-7 #1 Sun Dec 31 00:39:01 CET 2006 gcc 3.3.5
Squeak3.9alpha of 4 July 2005 [latest update: #7021]
Linux sarge 2.6.8-3-686 #1 Tue Dec 5 21:26:38 UTC 2006 i686 GNU/Linux

   My image is built with a Squeak3.9-final-7067 base with some
up-to-date packages : PreferenceBrowser, Vassili's Regex, Seaside.

   The crash : I use an internal image database. My application has
two database updates per days. So just before and after the update, I
save the image, and I save the database using a SmartRefStream to
create a timestamped file. The application si based on seaside.
   And regularly the image freeze completely (ie seaside website don't
even respond), or partially (I can browse the site). But i both cases,
trying to save of the image or save of the bdd just make the image use
100% of the cpu. No save is possible. An example using top :
28291 squeak      25   0  327m  54m  812 R 99.0 42.4 144:26.61 squeak

   So yesterday I tried to debug the vm using gdb and calling
printAllStacks. Here is the resulting crash :

(gdb) call printAllStacks()
Program received signal SIGSEGV, Segmentation fault.
0x08062c68 in printAllStacks ()
    at /home/piumarta/squeak/platforms/unix/src/vm/interp.c:20722
20722   in /home/piumarta/squeak/platforms/unix/src/vm/interp.c

   And you can found attached the output of the vm before the crash.
   I can't understand the causes of the crash. It could be linked to
the memory over 1Go access...
   And as this problem is recurrent of my production environement, I'd
like to have some ideas on how correct this. Like building my own VM
using some options ? Or downgrade to the 3.8 VM ?

Florian

Process
-1512342376 UserInputEvent>transformedBy:
-1512342468 MorphicEventDispatcher>dispatchDefault:with:
-1512342560 MorphicEventDispatcher>dispatchEvent:with:
-1512342668 Morph>processEvent:using:
-1512342912 PasteUpMorph>processEvent:using:
-1512343036 Morph>processEvent:
-1512343128 HandMorph>sendEvent:focus:clear:
-1512343220 HandMorph>sendMouseEvent:
-1512343312 HandMorph>handleEvent:
-1512343432 MouseOverHandler>processMouseOver:
-1512343616 HandMorph>processEvents
-1512343524 [] in WorldState>doOneCycleNowFor:
-1512343708 SequenceableCollection>do:
-1512343800 WorldState>handsDo:
-1512343892 WorldState>doOneCycleNowFor:
-1512343984 WorldState>doOneCycleFor:
-1512344076 PasteUpMorph>doOneCycle
-1532192696 [] in >spawnNewProcess
-1532192512 [] in BlockContext>newProcess



Process
-1526267788 >finalizationProcess
-1526267972 [] in >restartFinalizationProcess
-1526267880 [] in BlockContext>newProcess














Process
-1537801344 >timerInterruptWatcher
-1537801004 [] in >startTimerInterruptWatcher
-1537800912 [] in BlockContext>newProcess


































































Process
-1519028260 InputSensor>userInterruptWatcher
-1519028444 [] in InputSensor>installInterruptWatcher
-1519028352 [] in BlockContext>newProcess

Process
-1512344352 Delay>wait
-1512344260 [] in EventSensor>eventTickler
-1512344444 BlockContext>on:do:
-1519027768 EventSensor>eventTickler
-1519028044 [] in EventSensor>installEventTickler
-1519027860 [] in BlockContext>newProcess



Process
-1519027212 SystemDictionary>lowSpaceWatcher
-1519027120 [] in SystemDictionary>installLowSpaceWatcher
-1519027028 [] in BlockContext>newProcess




































































































Segmentation fault

-1512342376 UserInputEvent>transformedBy:
-1512342468 MorphicEventDispatcher>dispatchDefault:with:
-1512342560 MorphicEventDispatcher>dispatchEvent:with:
-1512342668 Morph>processEvent:using:
-1512342912 PasteUpMorph>processEvent:using:
-1512343036 Morph>processEvent:
-1512343128 HandMorph>sendEvent:focus:clear:
-1512343220 HandMorph>sendMouseEvent:
-1512343312 HandMorph>handleEvent:
-1512343432 MouseOverHandler>processMouseOver:
-1512343616 HandMorph>processEvents
-1512343524 [] in WorldState>doOneCycleNowFor:
-1512343708 SequenceableCollection>do:
-1512343800 WorldState>handsDo:
-1512343892 WorldState>doOneCycleNowFor:
-1512343984 WorldState>doOneCycleFor:
-1512344076 PasteUpMorph>doOneCycle
-1532192696 [] in >spawnNewProcess
-1532192512 [] in BlockContext>newProcess

On Thu, Apr 05, 2007 at 11:03:25AM +0200, Florian Minjat wrote:
Florian,

I cannot help with the actual problem you are having, but I can
suggest a workaround to prevent the production image getting hung.
If you do your saves in a forked background image, the production
image should keep running even if the background image hangs. You
will still need to monitor the system and kill the hung images
when it happens, but at least the production image could keep
running. Actually, you could have the production image keep track
of the forked image, and terminate it after a minute or so if it
has not yet exited; that way you would not need to monitor it
manually.

See UnixProcess class>>saveImageInBackground:nice: for an example
of how to do it. This is in OSProcess (on SqueakMap).

HTH,
Dave

David T. Lewis wrote:

Thanks for the tip !
I will look at this example to see if I can do the same thing with my
image. It could be a great workaround.

Florian

David T. Lewis wrote:
I tried to do a 'UnixProcess saveBackgroundImageNicely' on my prod
server, but nothing append. It returned an instance of UnixProcess
with pid of -1 which is weird.
Am I missing something ?

Florian

On Thu, Apr 05, 2007 at 05:24:39PM +0200, Florian Minjat wrote:
Florian,

I am away from my Squeak computer now so I can't check any
actual code, but it probably means that the actual unix fork()
failed for some reason and gave an error return value of -1
rather than the pid of the forked process. Look at the unix
man page for fork (man 2 fork) for an explanation. Perhaps
there is some memory usage restriction on your unix account
that prevents you from starting a second copy of Squeak, but
I'm just guessing.

I can't remember the method name, but if you look in class
UnixOSProcessAccessor you will probably find a method for
retrieving the most recent value of errno. If you call this
immediately after the fork, you can probably see the errno
value that would explain the problem (but be warned that
the value of errno can up updated by any system call, so
it's possible you will get a misleading result). If the
method is #primErrno, then you would call it with
"OSProcess accessor primErrno" (but I don't remember
what I actually called it).

Most likely this will just turn out to be a restriction of
your ISP, sorry.

Dave

David T. Lewis wrote:
Hi,
I am sorry but I could not find any method like *errno* in the
OSProcess-Unix package :/.
And I really don't see why I couldn't create a fork on my virtual
server account...

Florian losing faith in squeak capacity in production environnement...

David T. Lewis wrote:
My problem could also come from the fast that I didn't have the
external plugin...
I tried to use VMMaker to create it :
I got the plateform dir using svn ( http://wiki.squeak.org/squeak/2106 ).
I drag and droped the UnixOsProcessPluginInterpreterGetThisSession
from the left frame of VMMaker to the external pluguin frame.
I did 'Generate external plugins'.
I make a build source, went into it, ran
../platforms/unix/config/configure (
http://wiki.squeak.org/squeak/2173 ), and got a error :
'checking sanity of generated src directory... bad
missing file: /home/squeak/vmmaker/src/plugins.int'
What did I miss ?

Florian

Florian Minjat wrote:
Auto reply : the plugin must be drag and dropped in the 'internal'
section.
configure works, but make return an error :

make[1]: *** No rule to make target `UnixOSProcessPlugin.a'.  Stop.
make: *** [UnixOSProcessPlugin/UnixOSProcessPlugin.a] Error 2

There I am stuck again.

Florian

On Fri, Apr 06, 2007 at 01:50:15PM +0200, Florian Minjat wrote:
I apologize, my memory was wrong. There is no such method for getting
the errno value. Please disregard that suggestion.

What I can tell you for sure is that the return value of -1 from
the fork primitive in OSPP is the actual return value from the
Unix fork() system call. If you look at the man page for fork() on the
system you are using, it will explain what can cause this. Since
you are running on an hosting service with various resource limits,
I am guessing that there is some resource limit that is preventing
you from starting another Squeak VM on that system.

Dave

On Fri, Apr 06, 2007 at 02:32:23PM +0200, Florian Minjat wrote:
Check where VMMaker put the sources. If it was /home/squeak/vmmaker/src32,
then make a line to /home/squeak/vmmaker/src, like this:
$ cd /home/squeak/vmmaker
$ ln -s src32 src

Then try configure and make once more.

But by the way, I don't think you really need to build the plugin.
It's already included in most of the Unix VMs. If you can inspect
"OSProcess thisOSProcess" and see a UnixProcess with pid and other
variables filled in, then you already have the plugin.

Dave

David T. Lewis wrote:
It was ok for the sources.
I tried differents things, all with differents errors...
But I also tried "OSProcess thisOSProcess" and I got a process with
the right pid. So no plugin required and wind waving since one hour...

I tried to see the man page of fork (man fork, man 2 fork, ...), but I
don't have any.

Florian

Florian Minjat wrote:
> It was ok for the sources.
> I tried differents things, all with differents errors...
> But I also tried "OSProcess thisOSProcess" and I got a process with the
> right pid. So no plugin required and wind waving since one hour...
>
> I tried to see the man page of fork (man fork, man 2 fork, ...), but I
> don't have any.
>
> Florian

Ok I don't understand a thing sometimes.
After all this trouble to make this work, I tried again to create a
forked image.
And it worked !
No idea what changed. Because I saved my image under another name when
I got vmmaker and all the stuff to build the plugin. So I though it
could come from the installation of vmmaker.
But when I rolled back to my previous image, it worked too...

So now I use a forked vm to save my image and another forked vm to
save my bdd.

Hope this will stay stable.
Thanks a lot for the help.

Florian

On Fri, Apr 06, 2007 at 04:25:09PM +0200, Florian Minjat wrote:
I think that the problem with #forkSqueak will come back again.
It was probably caused by some restriction on the ISP server.
For example, when your production image gets very large, you may
find that the #forkSqueak no longer works (I am only guessing
of course). For this reason, I suggest you have some kind of
log file to keep track of whether the backups succeed. You
could have the background Squeak (the one doing the actual
save) write something to a log file when it is done, or you
can have the foreground Squeak check the status of the
background Squeak (#succeeded or #isComplete) to see if it
ran successfully. Also you know that if the background
Squeak has a pid of -1, then the fork must have failed, so
can watch for that specific problem.

> Thanks a lot for the help.

Sure!

Dave

David T. Lewis wrote:
You was right. The fork worked four times and then always returned -1.
The image is not so large (50Mo). But as my virtual server does not
have that much ram (128Mo), the os doesn't like the allocation of a
new one...
So this method is not secured at all.
I need to find something else...

Florian

Hi Florian,

Sorry that I was right :(

Dave

On Wed, Apr 18, 2007 at 04:16:11PM +0200, Florian Minjat wrote: