sweep failed to find exact end of memory

Hi,

On my machine doing a:

f1 :=  Form extent: 800 @ 600  * 4 depth:  32.
f2 :=  Form extent: 800 @ 600  * 4 depth:  32.
f3 :=  Form extent: 800 @ 600  * 4 depth:  32.
f4 :=  Form extent: 800 @ 600  * 4 depth:  32.

results into an:

"sweep failed to find exact end of memory" - vm crash

both under the squeak 3.9-8 and 3.7-3 unix vms.

my system:

ubuntu i386  1GB RAM 1GB Swap

Any ideas, why this happens?


- Jens -

The same happens under a different  machine with a different image,
here is a stacktrace:

sweep failed to find exact end of memory

-2121753020 Behavior>new:
-2121753164 Form>setExtent:depth:
-2121753256 >extent:depth:
2142442516 UndefinedObject>?
2142437672 Compiler>evaluate:in:to:notifying:ifFail:logged:
2142437764 [] in ParagraphEditor>evaluateSelection
2142437488 BlockContext>on:do:
2142437396 ParagraphEditor>evaluateSelection
2142437304 ParagraphEditor>doIt
2142437580 [] in ParagraphEditor>doIt:
2142437212 Controller>terminateAndInitializeAround:
2142437120 ParagraphEditor>doIt:
2142436872 ParagraphEditor>dispatchOnCharacter:with:
2142436768 TextMorphEditor>dispatchOnCharacter:with:
2142436676 ParagraphEditor>readKeyboard
2142436584 TextMorphEditor>readKeyboard
2142436160 [] in TextMorph>keyStroke:
2142436068 TextMorph>handleInteraction:fromEvent:
2142435976 TextMorphForEditView>handleInteraction:fromEvent:
2142435832 TextMorph>keyStroke:
2142435740 TextMorphForEditView>keyStroke:
2142435648 TextMorph>handleKeystroke:
2142435280 KeyboardEvent>sentTo:
2142435188 Morph>handleEvent:
2142435096 Morph>handleFocusEvent:
2142435372 [] in HandMorph>sendFocusEvent:to:clear:
2142435464 [] in PasteUpMorph>becomeActiveDuring:
2142435004 BlockContext>on:do:
2142434912 PasteUpMorph>becomeActiveDuring:
2142434728 HandMorph>sendFocusEvent:to:clear:
2142434636 HandMorph>sendEvent:focus:clear:
2142434504 HandMorph>sendKeyboardEvent:
2142434412 HandMorph>handleEvent:
2142434192 HandMorph>processEvents
2142434284 [] in WorldState>doOneCycleNowFor:
2142434100 SequenceableCollection>do:
2142434008 WorldState>handsDo:
2142433916 WorldState>doOneCycleNowFor:
2142433824 WorldState>doOneCycleFor:
2142433732 PasteUpMorph>doOneCycle
2013725548 [] in >spawnNewProcess
2013725732 [] in BlockContext>newProcess


Jens Lincke schrieb:

On 2/21/07, Jens Lincke <[hidden email]> wrote:

Does that change in sign near 2147483647 seem ominous to anyone else?
Perhaps the VM allocated more memory than it could manage.

Cheers!

--Tom Phoenix

Squeak 3.9-7, Gentoo i686, 512 MB RAM, 800 MB Swap
8 form without crash...

-- Pavel

On 2/21/07, Jens Lincke <[hidden email]> wrote:

Hi,

try more swap,

I disabled my swap the second machine, so that I only have 500MB real
ram left
an the vm did not crash.

I enabled the swap and the vm crashed again... so I think it is

an  issue of signed / unsigned int when memory over 1GB of RAM is allocated.

- Jens -

Pavel Krivanek schrieb:

On 21-Feb-07, at 10:37 AM, Jens Lincke wrote:

It's an old issue; look up assorted reports of problems with > 2Gb  
pointers. The VM code is mostly written under the assumption that  
singed integers suffice for pointers and we really need to fix that.  
It's an issue of available time.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
New: It comes in different colors from the previous version.

On 2/21/07, tim Rowledge <[hidden email]> wrote:

> It's an old issue; look up assorted reports of problems with > 2Gb
> pointers. The VM code is mostly written under the assumption that
> singed integers suffice for pointers and we really need to fix that.
> It's an issue of available time.

Do you have an estimate, hopefully under 1 million euros, of what it
would take to fix this?

Avi

On 21-Feb-07, at 10:58 AM, Avi Bryant wrote:

> On 2/21/07, tim Rowledge <[hidden email]> wrote:
>
>> It's an old issue; look up assorted reports of problems with > 2Gb
>> pointers. The VM code is mostly written under the assumption that
>> singed integers suffice for pointers and we really need to fix that.
>> It's an issue of available time.
>
> Do you have an estimate, hopefully under 1 million euros, of what it
> would take to fix this?

That's a distressingly difficult question to answer. I bet there are  
some nasty hacks one could do that would 'fix' it on the basis of  
forcing the C compiler to do signed-unsigned conversions but... I  
never trust a C compiler. Aside from Slang code there are of course  
the primitives and platform code that would be indirectly affected,  
if only at the level of "I have change this decl to stop an oodleplex  
of compiler warnings obscuring a real problem". There's also a  
boatload of VMMaker patches needing proper integration anyway and it  
would stupid not to do them as well.

I'd love to say, "oh a week" but long experience suggests that the  
truth is normally "double the digits and up the unit" so probably  
'two months' is higher in the truthiness scale.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: BH: Branch and Hang

Tim!  Do you support Bounty Projects!  :)

I believe we still have an open discussion about File Based Image Segments
too, which is another funded bounty project.
http://weeklysqueak.wordpress.com/2006/12/19/storing-imagesegments-outside-t
he-image/ .

Here is recent conversation:
http://www.nabble.com/ImageSegments-as-persistence-t3254689.html 
 
Ron Teitelbaum

Yes, I can confirm that.
With the current swap 800MB I was able to allocate 16 forms without
problems but when I used 40GB swap, it crashed for 4 forms too.

-- Pavel

On 2/21/07, Jens Lincke <[hidden email]> wrote:

On 2/21/07, tim Rowledge <[hidden email]> wrote:
Avoiding hacks and warnings and integrating VMMaker patches are all
laudable things, but do you even have two months to spare, funded or
no?  What would it take to get a temporary fix to this bug, however
ugly?  I've seen it myself and can be a bit of a showstopper if you
like to work with large images.  We could certainly pay for a week of
someone's time to do this if that was an option.

Avi

Hi Avi,

I think that you should set the memory limit for the Squeak VM for
now. It seems that it avoids this crashes. 40GB swap and 700MB memory
limit - no crash. Not perfect but better than random crashes.

Cheers,
-- Pavel

On 2/21/07, Avi Bryant <[hidden email]> wrote:

On 2/21/07, Pavel Krivanek <[hidden email]> wrote:
> Hi Avi,
>
> I think that you should set the memory limit for the Squeak VM for
> now. It seems that it avoids this crashes. 40GB swap and 700MB memory
> limit - no crash. Not perfect but better than random crashes.

Try a 700MB memory limit with a 400MB image, and I bet you still get a crash.

Avi

On Feb 21, 2007, at 21:00 , Avi Bryant wrote:

A hack to fix exactly this one reproducible failure might be  
relatively simple, because it looks debuggable. But we fixed a lot of  
places already, and there are always more lurking it seems.

> I've seen it myself and can be a bit of a showstopper if you
> like to work with large images.  We could certainly pay for a week of
> someone's time to do this if that was an option.

Well, unless you really fix it, by pretty much rewriting the whole  
code generator to use actual pointers for oops and patching the  
places in the interpreter that require signed ints and fixing all the  
plugins, you can't be anywhere near certain that the code actually  
does what you intend it to do. The ground is laid in the 64 bit work,  
but this is not tested extensively either, and probably buggy.

- Bert -

On Feb 21, 2007, at 18:32 , Jens Lincke wrote:

> "sweep failed to find exact end of memory" - vm crash


I have a hunch that declaring endOfMemoryLocal as unsigned in  
ObjectMemory>>sweepPhase might solve this particular problem.

Jens, could you try this?

- Bert -

It's not so much the amount of memory as where it's allocated. A 10Mb  
allocation that involves >2Gb values will still go kaboom (There was  
supposed to be an earth-shattering Kaboom, where was the earth-
shattering kaboom?)

Some OSs allocate from low adresses; some from high. Some do weird  
things.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: CEQ: Corrupt and Erase Queue

On 21-Feb-07, at 12:00 PM, Avi Bryant wrote:

>
> Avoiding hacks and warnings and integrating VMMaker patches are all
> laudable things, but do you even have two months to spare, funded or
> no?

Well, it looks pretty likely I'll have some time after mid-march for  
at least a while. The issue then will be whether I do more sitting in  
front of a computer or do I go and make some furniture.

> What would it take to get a temporary fix to this bug, however
> ugly?  I've seen it myself and can be a bit of a showstopper if you
> like to work with large images.  We could certainly pay for a week of
> someone's time to do this if that was an option.

Sounds like a worthwhile discussion to have.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Trojan:  Storage device for replicating codes...

Bert Freudenberg schrieb: I changed

sqInt sweepPhase(void) {
register struct foo * foo = &fum;
    sqInt endOfMemoryLocal;
    ...

to

sqInt sweepPhase(void) {
register struct foo * foo = &fum;
    usqInt endOfMemoryLocal;

and the "sweep failed to find exact end of memory" was gone, but now I get

Segmentation fault

-2143533612 BlockContext>ensure:
-2143534184 Semaphore>critical:
-2143534276 >recycleBuffer:

Segmentation fault

that means, the vm gets a segfault even in the segfault handler...

- Jens -

On Feb 21, 2007, at 22:15 , Jens Lincke wrote:

So the "sweep phase" error is gone :) And that's a different bug ...  
really, I think we'll see a lot more unless this is approached  
systematically ...

- Bert -

On 2/21/07, tim Rowledge <[hidden email]> wrote:
> It's not so much the amount of memory as where it's allocated. A 10Mb
> allocation that involves >2Gb values will still go kaboom (There was
> supposed to be an earth-shattering Kaboom, where was the earth-
> shattering kaboom?)
>
> Some OSs allocate from low adresses; some from high. Some do weird
> things.

Right - I see this a lot more under Linux than under OS X, for example.

Avi