Smalltalk › Squeak › Squeak - Dev

how/where is memory returned to the OS?

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

9 messages Options

Chris Muller-3

how/where is memory returned to the OS?

I have an image which needed to allocate 500MB of memory for a large
operation. Now that operation is done and all objects have been
reclaimed. According to VM I am now only using 200MB.

(Smalltalk vmParameterAt: 2) max: (Smalltalk vmParameter at: 1) "221923374"

However, when I look at my VM process in "top" or the System Monitor,
it still says its >500MB.

How does this work?

David T. Lewis

Re: how/where is memory returned to the OS?

On Fri, Oct 19, 2012 at 10:46:45AM -0500, Chris Muller wrote:

> I have an image which needed to allocate 500MB of memory for a large
> operation. Now that operation is done and all objects have been
> reclaimed. According to VM I am now only using 200MB.
>
> (Smalltalk vmParameterAt: 2) max: (Smalltalk vmParameter at: 1) "221923374"
>
> However, when I look at my VM process in "top" or the System Monitor,
> it still says its >500MB.
>
> How does this work?

Address space gets allocated as needed to meet your >500MB requirement,
and that address space stays mapped to your VM process, hence displayed
by "top". Actual memory pages are freed automagically by the virtual
memory system. The sizes reported by the VM parameters reflect the size
of the object memory, and is a better indication of the actual amount
of memory being actively used.

Dave

Eliot Miranda-2

Re: how/where is memory returned to the OS?

In reply to this post by Chris Muller-3

Hi Chris,

On Fri, Oct 19, 2012 at 8:46 AM, Chris Muller <[hidden email]> wrote:

I have an image which needed to allocate 500MB of memory for a large
operation. Now that operation is done and all objects have been
reclaimed. According to VM I am now only using 200MB.

(Smalltalk vmParameterAt: 2) max: (Smalltalk vmParameter at: 1) "221923374"

However, when I look at my VM process in "top" or the System Monitor,
it still says its >500MB.

How does this work?

Depends on the platform. On Mac OS X memory is not returned. On Windows memory is allocated and freed via VirtualAlloc and VirtualFree. On Unix memory is allocated and returned via mmap and munmap by default. What OS are you on? Looks like the unix code could be ported to the Mac easily.

--
best,

Eliot

Chris Muller-4

Re: how/where is memory returned to the OS?

>> I have an image which needed to allocate 500MB of memory for a large
>> operation. Now that operation is done and all objects have been
>> reclaimed. According to VM I am now only using 200MB.
>>
>> (Smalltalk vmParameterAt: 2) max: (Smalltalk vmParameter at: 1)
>> "221923374"
>>
>> However, when I look at my VM process in "top" or the System Monitor,
>> it still says its >500MB.
>>
>> How does this work?
>
> Depends on the platform. On Mac OS X memory is not returned. On Windows
> memory is allocated and freed via VirtualAlloc and VirtualFree. On Unix
> memory is allocated and returned via mmap and munmap by default. What OS
> are you on? Looks like the unix code could be ported to the Mac easily.

Linux chrisT520 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

So, after several hours top still shows >500MB. Will it ever go back
down to the level reported by vm-parameters [1,2 or 3]?

Eliot Miranda-2

Re: how/where is memory returned to the OS?

On Fri, Oct 19, 2012 at 11:31 AM, Chris Muller <[hidden email]> wrote:

>> I have an image which needed to allocate 500MB of memory for a large
>> operation. Now that operation is done and all objects have been
>> reclaimed. According to VM I am now only using 200MB.
>>
>> (Smalltalk vmParameterAt: 2) max: (Smalltalk vmParameter at: 1)
>> "221923374"
>>
>> However, when I look at my VM process in "top" or the System Monitor,
>> it still says its >500MB.
>>
>> How does this work?
>
> Depends on the platform. On Mac OS X memory is not returned. On Windows
> memory is allocated and freed via VirtualAlloc and VirtualFree. On Unix
> memory is allocated and returned via mmap and munmap by default. What OS
> are you on? Looks like the unix code could be ported to the Mac easily.

Linux chrisT520 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

So, after several hours top still shows >500MB. Will it ever go back
down to the level reported by vm-parameters [1,2 or 3]?

I'm not sure. It all depends on the useMmap variable in the linux vm. If it is non-zero then a cursory reading of the source says that yes, the memory should be returned.

--
best,

Eliot

Igor Stasenko

Re: how/where is memory returned to the OS?

On 19 October 2012 21:42, Eliot Miranda <[hidden email]> wrote:

>
>
> On Fri, Oct 19, 2012 at 11:31 AM, Chris Muller <[hidden email]> wrote:
>>
>> >> I have an image which needed to allocate 500MB of memory for a large
>> >> operation. Now that operation is done and all objects have been
>> >> reclaimed. According to VM I am now only using 200MB.
>> >>
>> >> (Smalltalk vmParameterAt: 2) max: (Smalltalk vmParameter at: 1)
>> >> "221923374"
>> >>
>> >> However, when I look at my VM process in "top" or the System Monitor,
>> >> it still says its >500MB.
>> >>
>> >> How does this work?
>> >
>> > Depends on the platform. On Mac OS X memory is not returned. On Windows
>> > memory is allocated and freed via VirtualAlloc and VirtualFree. On Unix
>> > memory is allocated and returned via mmap and munmap by default. What
>> > OS
>> > are you on? Looks like the unix code could be ported to the Mac easily.
>>
>> Linux chrisT520 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09
>> UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
>>
>> So, after several hours top still shows >500MB. Will it ever go back
>> down to the level reported by vm-parameters [1,2 or 3]?
>
>
> I'm not sure. It all depends on the useMmap variable in the linux vm. If
> it is non-zero then a cursory reading of the source says that yes, the
> memory should be returned.

I may be mistaken. But what i have seen, there is no way release
address space back to OS.
It is not because platform code do not releasing the memory, but it is
because VM never does that.

At least , last time i looked at that code, it was like that.

Also, can i remind you (It been a while since i wrote about that.. ),
what i told before that we need a better memory manager:
the existing one is less than adequate for modern systems.
The model, which it employs (reserving huge amount of address space,
for instance) does not works
well for HydraVM..
because if you reserve 512M address space per interpreter instance,
you cannot have more than 4-8 (depending on system) at once..

> --
> best,
> Eliot

--
Best regards,
Igor Stasenko.

Igor Stasenko

Re: how/where is memory returned to the OS?

To be more precise:
the function which responsible for that is sqShrinkMemoryBy()

on windows it is disabled
(see
#define DO_NOT_SHRINK
)

in unix sources (sqUnixMemory.c), it saying merely the same:

* The code allows memory to be overallocated; i.e., the initial
* block is reserved via mmap() and then the unused portion
* munmap()ped from the top end. This is INHERENTLY DANGEROUS since
* malloc() may randomly map new memory in the block we "reserved"
* and subsequently unmap()ped. Enabling this causes crashes in
* Croquet, which makes heavy use of the FFI and thus calls malloc()
* all over the place.
*
* For this reason, overallocateMemory is DISABLED by default.
*
* The upshot of all this is that Squeak will claim (and hold on to)
* ALL of the available virtual memory (or at least 75% of it) when
* it starts up. If you can't live with that, use the -memory
* option to allocate a fixed size heap.

so, it never munmap()'s the mapped memory.
that's why it showing 512M in use, while in fact your image might need
much less than that :)

--
Best regards,
Igor Stasenko.

Chris Muller-3

Re: how/where is memory returned to the OS?

In reply to this post by Eliot Miranda-2

>> So, after several hours top still shows >500MB. Will it ever go back
>> down to the level reported by vm-parameters [1,2 or 3]?
>
> I'm not sure. It all depends on the useMmap variable in the linux vm. If
> it is non-zero then a cursory reading of the source says that yes, the
> memory should be returned.

It appears Igor is right. Even if I just do:

Y:=String new: 550000000.

and then:

Y:=nil

nothing I do other than rebooting the image seems to recover that
memory for other processes. But rebooting the image is not an option
for a 24x7 server.

I hope this can be fixed because it really neuters Squeak's ability to
outshine other systems by its flexibility.

Igor Stasenko

Re: how/where is memory returned to the OS?

On 21 October 2012 21:01, Chris Muller <[hidden email]> wrote:

>>> So, after several hours top still shows >500MB. Will it ever go back
>>> down to the level reported by vm-parameters [1,2 or 3]?
>>
>> I'm not sure. It all depends on the useMmap variable in the linux vm. If
>> it is non-zero then a cursory reading of the source says that yes, the
>> memory should be returned.
>
> It appears Igor is right. Even if I just do:
>
> Y:=String new: 550000000.
>
> and then:
>
> Y:=nil
>
> nothing I do other than rebooting the image seems to recover that
> memory for other processes. But rebooting the image is not an option
> for a 24x7 server.
>
> I hope this can be fixed because it really neuters Squeak's ability to
> outshine other systems by its flexibility.
>

Unfortunately there is no easy fix , which will fit all possible use scenarios.
As you can see from comments in code, this feature was disabled on purpose
for cases when your application works extensively with external
resources/libraries,
which using malloc() a lot.

IMO, the right solution to this would be to employ a memory segments
model (which i talked before),
but that will mean a lot of refactorings in VM. But would allow much
better coexistance between
conventional C heap and object memory segments, which can be
allocated/deallocated dynamically.
The downside of it, is of course, that object memory address space
will be no longer continuous
(so you have to say bye to simple and inexpensive checks like
oop < youngMemoryStart etc).
But the upside is to be able to have various kinds of memory segments,
like 'non-movable segment'
and 'read-only segments' etc..

To my thinking this is the right way to go.

--
Best regards,
Igor Stasenko.