Only ever grows? (was: Re: [squeak-dev] how/where is memory returned to the OS?)

> > Unfortunately there is no easy fix , which will fit all possible use scenarios.
> > As you can see from comments in code, this feature was disabled on purpose
> > for cases when your application works extensively with external
> > resources/libraries,
> > which using malloc() a lot.
>
> You mean FFI?  Someone took the trouble to write this for the manpage.
>  My gut thinks FFI has been around longer than the releasing of memory
> feature -- not sure.
>
>        -mmap size[mk]
>               requests that a variable heap of at most  size  bytes  be  allo???
>               cated.   (The  suffixes  are  as  described  for  the  '-memory'
>               option.)  squeak will initially allocate a heap  that  is  large
>               enough  to  hold the image, with a small amount of headroom.  If
>               at any time Squeak requires more memory for its image then addi???
>               tional space will be allocated dynamically.  Likewise, when mem???
>               ory is no longer needed it will deallocated and returned to  the
>               system.   The size argument places an upper limit on how big the
>               heap can grow in this fashion.  squeak uses a  dynamic  heap  by
>               default  with  the maximum size set to 75% of the available vir???
>               tual memory or 1 gigabyte, whichever is smaller.
>
> Eliot, is the useMmap variable you referred to set by this argument?
> I tried launching my image with -mmap 700M but it still didn't seem to
> want to deallocate.
>
> Same with the Classic VM.

> Eliot is referring to Ian Piumarta's explanation in sqUnixMemory.c here:
>
>     /* Note:
>      *
>      *   The code allows memory to be overallocated; i.e., the initial
>      *   block is reserved via mmap() and then the unused portion
>      *   munmap()ped from the top end.  This is INHERENTLY DANGEROUS since
>      *   malloc() may randomly map new memory in the block we "reserved"
>      *   and subsequently unmap()ped.  Enabling this causes crashes in
>      *   Croquet, which makes heavy use of the FFI and thus calls malloc()
>      *   all over the place.
>      *
>      *   For this reason, overallocateMemory is DISABLED by default.
>      *
>      *   The upshot of all this is that Squeak will claim (and hold on to)
>      *   ALL of the available virtual memory (or at least 75% of it) when
>      *   it starts up.  If you can't live with that, use the -memory
>      *   option to allocate a fixed size heap.
>      */
>
> See also the man page for sbrk for some related cautions about mixing malloc
> with lower level allocation/deallocation calls.

>> Eliot is referring to Ian Piumarta's explanation in sqUnixMemory.c here:
>>
>>     /* Note:
>>      *
>>      *   The code allows memory to be overallocated; i.e., the initial
>>      *   block is reserved via mmap() and then the unused portion
>>      *   munmap()ped from the top end.  This is INHERENTLY DANGEROUS since
>>      *   malloc() may randomly map new memory in the block we "reserved"
>>      *   and subsequently unmap()ped.  Enabling this causes crashes in
>>      *   Croquet, which makes heavy use of the FFI and thus calls malloc()
>>      *   all over the place.
>>      *
>>      *   For this reason, overallocateMemory is DISABLED by default.
>>      *
>>      *   The upshot of all this is that Squeak will claim (and hold on to)
>>      *   ALL of the available virtual memory (or at least 75% of it) when
>>      *   it starts up.  If you can't live with that, use the -memory
>>      *   option to allocate a fixed size heap.
>>      */
>>
>> See also the man page for sbrk for some related cautions about mixing malloc
>> with lower level allocation/deallocation calls.
>
> If the VM starts wth a block initially allocated for the
> object-memory, and extended as needed for subsequent allocation
> (String new: 4000000), but THEN reduced once not needed -- why would a
> *subsequent* malloc call grabbing memory from the part that was
> deallocated cause a crash?  At first I thought the crash would occur
> after a subsequent extension of object-memory into memory that was
> malloc'd for another purpose (plugin) but..  that could happen on the
> _first_ time it is extended, couldn't it?
>
> IOW:
>
> I start out using 40MB for object memory.
>
>     0----------------------40MB
>
> Plugins make some malloc calls, which allocate somewhere just above..
>
>     0----------------------40MB-----+5MB(malloc'd)
>
> Now I need more allocation for object memory, so I'm trampling what
> the plugin malloc'd?
>
> Sorry, I unfortunately do not know C at all, thanks for helping me
> understand the situation.
>
>> Chris, may I ask why you see this as a problem? Is there some practical
>> situation in which you are running out of real system memory as a consequence
>> of the memory mapping strategy?
>
> I have a Squeak-based server where 99% of requests are small, but once
> a big request comes in, that server image jumps from 40MB to 400MB.
> The extra memory was only needed for 10 seconds but unfortunately sits
> there unused forever-after.  So it's almost like a memory-leak where
> the only way to recover it is to bounce the server.
>
> What would be worse is if the over-allocated image then affects GC
> performance.  I'm still trying to figure out whether this is the case
> -- it seems like not thank goodness.  Do you know for sure the answer
> to that?  IOW should full GC's be just as fast after the big request
> is long-gone (reclaimed) as they were before the big request?
>