I have an image which needed to allocate 500MB of memory for a large
operation. Now that operation is done and all objects have been reclaimed. According to VM I am now only using 200MB. (Smalltalk vmParameterAt: 2) max: (Smalltalk vmParameter at: 1) "221923374" However, when I look at my VM process in "top" or the System Monitor, it still says its >500MB. How does this work? |
On Fri, Oct 19, 2012 at 10:46:45AM -0500, Chris Muller wrote:
> I have an image which needed to allocate 500MB of memory for a large > operation. Now that operation is done and all objects have been > reclaimed. According to VM I am now only using 200MB. > > (Smalltalk vmParameterAt: 2) max: (Smalltalk vmParameter at: 1) "221923374" > > However, when I look at my VM process in "top" or the System Monitor, > it still says its >500MB. > > How does this work? Address space gets allocated as needed to meet your >500MB requirement, and that address space stays mapped to your VM process, hence displayed by "top". Actual memory pages are freed automagically by the virtual memory system. The sizes reported by the VM parameters reflect the size of the object memory, and is a better indication of the actual amount of memory being actively used. Dave |
In reply to this post by Chris Muller-3
Hi Chris,
On Fri, Oct 19, 2012 at 8:46 AM, Chris Muller <[hidden email]> wrote: I have an image which needed to allocate 500MB of memory for a large Depends on the platform. On Mac OS X memory is not returned. On Windows memory is allocated and freed via VirtualAlloc and VirtualFree. On Unix memory is allocated and returned via mmap and munmap by default. What OS are you on? Looks like the unix code could be ported to the Mac easily.
best, Eliot |
>> I have an image which needed to allocate 500MB of memory for a large
>> operation. Now that operation is done and all objects have been >> reclaimed. According to VM I am now only using 200MB. >> >> (Smalltalk vmParameterAt: 2) max: (Smalltalk vmParameter at: 1) >> "221923374" >> >> However, when I look at my VM process in "top" or the System Monitor, >> it still says its >500MB. >> >> How does this work? > > Depends on the platform. On Mac OS X memory is not returned. On Windows > memory is allocated and freed via VirtualAlloc and VirtualFree. On Unix > memory is allocated and returned via mmap and munmap by default. What OS > are you on? Looks like the unix code could be ported to the Mac easily. Linux chrisT520 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux So, after several hours top still shows >500MB. Will it ever go back down to the level reported by vm-parameters [1,2 or 3]? |
On Fri, Oct 19, 2012 at 11:31 AM, Chris Muller <[hidden email]> wrote: --
I'm not sure. It all depends on the useMmap variable in the linux vm. If it is non-zero then a cursory reading of the source says that yes, the memory should be returned.
best, Eliot |
On 19 October 2012 21:42, Eliot Miranda <[hidden email]> wrote:
> > > On Fri, Oct 19, 2012 at 11:31 AM, Chris Muller <[hidden email]> wrote: >> >> >> I have an image which needed to allocate 500MB of memory for a large >> >> operation. Now that operation is done and all objects have been >> >> reclaimed. According to VM I am now only using 200MB. >> >> >> >> (Smalltalk vmParameterAt: 2) max: (Smalltalk vmParameter at: 1) >> >> "221923374" >> >> >> >> However, when I look at my VM process in "top" or the System Monitor, >> >> it still says its >500MB. >> >> >> >> How does this work? >> > >> > Depends on the platform. On Mac OS X memory is not returned. On Windows >> > memory is allocated and freed via VirtualAlloc and VirtualFree. On Unix >> > memory is allocated and returned via mmap and munmap by default. What >> > OS >> > are you on? Looks like the unix code could be ported to the Mac easily. >> >> Linux chrisT520 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 >> UTC 2012 x86_64 x86_64 x86_64 GNU/Linux >> >> So, after several hours top still shows >500MB. Will it ever go back >> down to the level reported by vm-parameters [1,2 or 3]? > > > I'm not sure. It all depends on the useMmap variable in the linux vm. If > it is non-zero then a cursory reading of the source says that yes, the > memory should be returned. I may be mistaken. But what i have seen, there is no way release address space back to OS. It is not because platform code do not releasing the memory, but it is because VM never does that. At least , last time i looked at that code, it was like that. Also, can i remind you (It been a while since i wrote about that.. ), what i told before that we need a better memory manager: the existing one is less than adequate for modern systems. The model, which it employs (reserving huge amount of address space, for instance) does not works well for HydraVM.. because if you reserve 512M address space per interpreter instance, you cannot have more than 4-8 (depending on system) at once.. > -- > best, > Eliot -- Best regards, Igor Stasenko. |
To be more precise:
the function which responsible for that is sqShrinkMemoryBy() on windows it is disabled (see #define DO_NOT_SHRINK ) in unix sources (sqUnixMemory.c), it saying merely the same: * The code allows memory to be overallocated; i.e., the initial * block is reserved via mmap() and then the unused portion * munmap()ped from the top end. This is INHERENTLY DANGEROUS since * malloc() may randomly map new memory in the block we "reserved" * and subsequently unmap()ped. Enabling this causes crashes in * Croquet, which makes heavy use of the FFI and thus calls malloc() * all over the place. * * For this reason, overallocateMemory is DISABLED by default. * * The upshot of all this is that Squeak will claim (and hold on to) * ALL of the available virtual memory (or at least 75% of it) when * it starts up. If you can't live with that, use the -memory * option to allocate a fixed size heap. so, it never munmap()'s the mapped memory. that's why it showing 512M in use, while in fact your image might need much less than that :) -- Best regards, Igor Stasenko. |
In reply to this post by Eliot Miranda-2
>> So, after several hours top still shows >500MB. Will it ever go back
>> down to the level reported by vm-parameters [1,2 or 3]? > > I'm not sure. It all depends on the useMmap variable in the linux vm. If > it is non-zero then a cursory reading of the source says that yes, the > memory should be returned. It appears Igor is right. Even if I just do: Y:=String new: 550000000. and then: Y:=nil nothing I do other than rebooting the image seems to recover that memory for other processes. But rebooting the image is not an option for a 24x7 server. I hope this can be fixed because it really neuters Squeak's ability to outshine other systems by its flexibility. |
On 21 October 2012 21:01, Chris Muller <[hidden email]> wrote:
>>> So, after several hours top still shows >500MB. Will it ever go back >>> down to the level reported by vm-parameters [1,2 or 3]? >> >> I'm not sure. It all depends on the useMmap variable in the linux vm. If >> it is non-zero then a cursory reading of the source says that yes, the >> memory should be returned. > > It appears Igor is right. Even if I just do: > > Y:=String new: 550000000. > > and then: > > Y:=nil > > nothing I do other than rebooting the image seems to recover that > memory for other processes. But rebooting the image is not an option > for a 24x7 server. > > I hope this can be fixed because it really neuters Squeak's ability to > outshine other systems by its flexibility. > As you can see from comments in code, this feature was disabled on purpose for cases when your application works extensively with external resources/libraries, which using malloc() a lot. IMO, the right solution to this would be to employ a memory segments model (which i talked before), but that will mean a lot of refactorings in VM. But would allow much better coexistance between conventional C heap and object memory segments, which can be allocated/deallocated dynamically. The downside of it, is of course, that object memory address space will be no longer continuous (so you have to say bye to simple and inexpensive checks like oop < youngMemoryStart etc). But the upside is to be able to have various kinds of memory segments, like 'non-movable segment' and 'read-only segments' etc.. To my thinking this is the right way to go. -- Best regards, Igor Stasenko. |
Free forum by Nabble | Edit this page |