gcc -Wall -pedantic (was: sweep failed to find exact end of memory)

 
On Mon, May 07, 2007 at 08:12:29PM -0700, Craig Latta wrote:
>  
>
>      Those are part of Spoon. The first of those is only called in
> response to interactive human decision. I'm interested in problems with
> the second, though (but it only touches method trailer bytes, I'm not
> terribly concerned about it).

If you adopt the changes I did for the base VMMaker, then use the
methods in ObjectMemory category "oop comparison" to do the comparisons,
i.e. use #oop:isLessThan: rather than #<. This will do the type casting
without upsetting the slang inliner.

BTW did any of you try the slang browser at http://wiki.squeak.org/squeak/5916?
It's a lot more convenient than generating a new interp.c every time you
change a method.

Dave

 
On Wed, Apr 25, 2007 at 06:59:49AM -0400, David T. Lewis wrote:
> The attached zip contains six change sets and an update for sqMemoryAccess.h.
> The changes are intended to resolve problems with oop variable declarations
> and comparison operations that may occur on platforms that assign object
> memory to high virtual memory address values.

Can anyone confirm whether these changes are producing the intended
results, i.e. a system that used to crash with >2GB oop issues, and
no longer crashes after applying the changes? I don't have any Squeak
platform that exhibits the problem, so so I can't confirm whether or
not the problem is actually resolved.

Thanks,

Dave

 
Ok, well I did turn unsigned/signed warnings on and did wander thru  
the entire set of messge looking for something wrong, but did not  
find anything.
Right now I need to alter the mac VM source code to handle sqInt  
values versus integer, then compile up a test harness that will  
allocate memory over
or saddling the 2GB boundary.  Maybe start this later today or  
tomorrow if time permits...

On May 27, 2007, at 6:22 AM, David T. Lewis wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

 
Ok, I build a new VM using these changes then poked about  and  
wondered about the following compares as below.

Also did some tests, by altering mmap I ran the VM at the 3GB  
boundary, that seemed to work ok.

I then attempted to run it at the 1.5 GB boundary using a memory size of

641,732,608  at the 1.5GB boundary

The basic image (a 3.5 image) started at 20MB. I then allocated 450  
mb of memory and did macro-benchmarks

| suck |
suck := OrderedCollection new.
        suck add: (ByteArray new: 1024*1024*450).
100 timesRepeat: [
        Smalltalk macroBenchmarks.
        suck add: (ByteArray new: 1024*1024*1).
        Transcript show: Smalltalk garbageCollectMost;cr.
        Transcript show: Smalltalk garbageCollect;cr].


However when young space approached, when over, inbetween, whatever  
the 2GB boundary the VM core dumped.
I was unable to printAllStacks(). I'll look a bit further and put  
some more debugging in to see if I can determine why.


- questionable? signed/unsigned compares at least from the compiler  
viewpoint.

primitivePerformAt: lookupClass
primitiveDoPrimitiveWithArgs

                self success: (self stackPointerIndex + arraySize) < cntxSize].

primitiveObjectAtPut
primitiveObjectAt

                self success: index <= ((self literalCountOf: thisReceiver) +  
LiteralStart).

postGCAction
                        (self sizeOfFree: freeBlock) > shrinkThreshold
       
                Ok here if the size of the free > 2GB that is an issue
                Also shrinkThreshold is signed, but should it be?

incrementalGC
               
                (((self sizeOfFree: freeBlock) < growHeadroom)

I think you need to check usage of sizeOfFree:  it could be > 2GB,  
very unlikely, but one never knows




On May 27, 2007, at 6:22 AM, David T. Lewis wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

 
ok, in debugging vm I get  failure in

   static inline sqInt longAtput(sqInt oop, sqInt val) { return  
longAtPointerput(pointerForOop(oop), val); }

oop is 0xe02e2c98
val is 310,038

called via inCompMove

        } else {
                newFreeChunk = newOop + (sizeBitsOf(newOop));
                /* begin setSizeOfFree:to: */
                longAtput(newFreeChunk, (bytesFreed & AllButTypeMask) |  
HeaderTypeFree);
        }


newOop is 0x7ffb4558
newFreeChunk is -533844840  or  0xe02e2c98
Well that has a wrong feel to it...
Let me attached an image of the debugger vars in case someone wants  
to puzzle out the issue.


On May 27, 2007, at 6:22 AM, David T. Lewis wrote:
--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

 

On 8-Jun-07, at 2:07 PM, John M McIntosh wrote:

>
> - questionable? signed/unsigned compares at least from the compiler  
> viewpoint.
>
> primitivePerformAt: lookupClass
> primitiveDoPrimitiveWithArgs
>
> self success: (self stackPointerIndex + arraySize) < cntxSize].

That shouldn't be a problem; stackPointerIndex is pretty much limited  
to 32 words, arraySize is practically limited to quite small but  
could possibly get large in insane code and cntxSize is 40 (-ish?) words
>
> primitiveObjectAtPut
> primitiveObjectAt
>
> self success: index <= ((self literalCountOf: thisReceiver) +  
> LiteralStart).
again, operating inside methods/context/etc so fairly limited value  
range. literal count is limited to 256 I think, LiteralStart is 6 or  
thereabouts.

>
> postGCAction
> (self sizeOfFree: freeBlock) > shrinkThreshold
Looks like sizeOfFree returns a signed int. shrinkThreshold should  
only be a +ve number anyway and should be checked somewhere in the  
setting of its value. Guess sizeOfFree ought to be changed.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful random insult:- Can't find log base two of 65536 without a  
calculator.

 
Further testing at the 2gb boundary shows a variety of different  
errors from run to run. I'd say the incremental GC logic
trash memory somehow as it goes over the 2gb boundary.

On May 27, 2007, at 6:22 AM, David T. Lewis wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

 
I stuck some compares in the pointer accessors and got


                        for (w = firstWord; w <= lastWord; w += BytesPerWord) {
                                longAtput(target, longAt(w));
                                target += BytesPerWord;
                        }


in incCompMove

w is 0x86401004
firstWord is 0x7ffffff4
lastWord is 0x7ffffffc


mmm Oh wait w being an integer of value -2042621948  is well always  
less than lastWord.

Perhaps other suspects?

        for (i = firstField; i <= lastField; i += BytesPerWord) {
        for (i = BaseHeaderSize; i <= lastField; i += BytesPerWord) {
        for (i = ((lastPointerOf(oop)) + BytesPerWord); i <= (sz -  
BaseHeaderSize); i += BytesPerWord) {


On May 27, 2007, at 6:22 AM, David T. Lewis wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

 
On Fri, Jun 08, 2007 at 02:07:35PM -0700, John M McIntosh wrote:
Hi John,

Thanks for the feedback. It's good that you've got a fairly repeatable
failure scenario. When I did those changes, I was working on a 64 bit
machine, and the Squeak heap gets allocated at e.g. 0x2b39e0c5a000, so
I would not have been seeing the 2GB boundary on that machine. I guess
I should go back to a 32 bit box and reproduce the problem there.

I might not get a chance to look at this further for a couple of weeks
though. Right now my pet project is to see if I can get Areithfa Ffenestri
working on unix.

Dave

 
David, in order to test this I changed

    debug = mmap( 2147483648U-512*1024*1024, gMaxHeapSize+pageSize

then asked for 600MB of squeak memory.
mmap allocated memory for me on the 2GB-512MB boundary, so I then had  
88 MB of space over the 2GB boundary. I could work towards.

I'd think if you look at sqUnixMemory.c and  fiddle a bit with

       if (MAP_FAILED == (heap= mmap(0, heapLimit, MAP_PROT,  
MAP_FLAGS, devZero, 0)))

you might be able to change the 0 to say.. .

0x7FFFFFFFF0000000

Then ask for 600MB of memory for the image.

That would set the end of memory at 0x8000000015800000
344MB over the negative sign boundary.

Check heap then, it *might* say 0x7FFFFFFFF0000000 if it works as  
expected.
   I was able to use this to set the start at 1gb, 1.5gb, 2gb, 3gb on  
the mac in 32bit mode.

On Jun 9, 2007, at 8:08 AM, David T. Lewis wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

 
Attached is a change set that includes changes to work with size of  
free block as usqInt and the fix for the incCompMove: do loop.

With this fix it appears the VM will run below and cross the 2GB  
boundary in 32bit, and run at the 3GB boundary in 32bit mode.


On Jun 9, 2007, at 8:08 AM, David T. Lewis wrote:

> Thanks for the feedback. It's good that you've got a fairly repeatable
> failure scenario.

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

 
BTW, this is a really great catch John. We have seen various lockups in
this precise loop and the subtlety of the lockup condition is beautiful.
  I mean you really have to appreciate that the condition here is that
"lastWord + BytesPerWord <= lastWord" for the situation to go wrong.

Cheers,
   - Andreas

John M McIntosh wrote: