What is meant by 'shrink' in the VM parameters?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

What is meant by 'shrink' in the VM parameters?

Ken Causey-3
 
I'm wondering what 'shrink' means in the following VM parameter
comments:

"24 memory threshold above which shrinking object memory (rw)"

and

"32 number of shrink memory requests (read-only)"

My first thought is that a shrink represents Squeak release memory back
to the operating system once it is no longer needed.

As I've mentioned before we are trying to reduce the RAM footprint of
the www.squeak.org process and are watching various related figures.

We can see the value in VM parameter 32 increase, but while the value
for parameter 3 (end of memory) goes down as you would expect, the
operating system's view of the memory usage does not change.

Ken

signature.asc (197 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] What is meant by 'shrink' in the VM parameters?

johnmci
 
It *would* give the memory back to the operating system.
But we don't anymore again read the following

/* Note:
*
*   The code allows memory to be overallocated; i.e., the initial
*   block is reserved via mmap() and then the unused portion
*   munmap()ped from the top end.  This is INHERENTLY DANGEROUS since
*   malloc() may randomly map new memory in the block we "reserved"
*   and subsequently unmap()ped.  Enabling this causes crashes in
*   Croquet, which makes heavy use of the FFI and thus calls malloc()
*   all over the place.
*  
*   For this reason, overallocateMemory is DISABLED by default.
*  
*   The upshot of all this is that Squeak will claim (and hold on to)
*   ALL of the available virtual memory (or at least 75% of it) when
*   it starts up.  If you can't live with that, use the -memory
*   option to allocate a fixed size heap.
*/


On 2010-10-22, at 6:24 AM, Ken Causey wrote:

> I'm wondering what 'shrink' means in the following VM parameter
> comments:
>
> "24 memory threshold above which shrinking object memory (rw)"
>
> and
>
> "32 number of shrink memory requests (read-only)"
>
> My first thought is that a shrink represents Squeak release memory back
> to the operating system once it is no longer needed.
>
> As I've mentioned before we are trying to reduce the RAM footprint of
> the www.squeak.org process and are watching various related figures.
>
> We can see the value in VM parameter 32 increase, but while the value
> for parameter 3 (end of memory) goes down as you would expect, the
> operating system's view of the memory usage does not change.
>
> Ken
>
--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================





smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] What is meant by 'shrink' in the VM parameters?

johnmci
In reply to this post by Ken Causey-3
 
Further to this you should consider the setting for setGCBiasToGrow
and what are the numbers out of

        3 end of memory (read-only)
        31 number of grow memory requests (read-only)
        32 number of shrink memory requests (read-only)

I'll suggest you chart them by time to understand what the spikes are?
does it go from 60MB to 180MB then back to 60MB in a second? and pound #31 up by 20?


On 2010-10-22, at 6:24 AM, Ken Causey wrote:

> I'm wondering what 'shrink' means in the following VM parameter
> comments:
>
> "24 memory threshold above which shrinking object memory (rw)"
>
> and
>
> "32 number of shrink memory requests (read-only)"
>
> My first thought is that a shrink represents Squeak release memory back
> to the operating system once it is no longer needed.
>
> As I've mentioned before we are trying to reduce the RAM footprint of
> the www.squeak.org process and are watching various related figures.
>
> We can see the value in VM parameter 32 increase, but while the value
> for parameter 3 (end of memory) goes down as you would expect, the
> operating system's view of the memory usage does not change.
>
> Ken
>
--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================





smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] What is meant by 'shrink' in the VM parameters?

Ken Causey-3
In reply to this post by johnmci
 
On Fri, 2010-10-22 at 09:23 -0700, John M McIntosh wrote:

> It *would* give the memory back to the operating system.
> But we don't anymore again read the following
>
> /* Note:
> *
> *   The code allows memory to be overallocated; i.e., the initial
> *   block is reserved via mmap() and then the unused portion
> *   munmap()ped from the top end.  This is INHERENTLY DANGEROUS since
> *   malloc() may randomly map new memory in the block we "reserved"
> *   and subsequently unmap()ped.  Enabling this causes crashes in
> *   Croquet, which makes heavy use of the FFI and thus calls malloc()
> *   all over the place.
> *  
> *   For this reason, overallocateMemory is DISABLED by default.
> *  
> *   The upshot of all this is that Squeak will claim (and hold on to)
> *   ALL of the available virtual memory (or at least 75% of it) when
> *   it starts up.  If you can't live with that, use the -memory
> *   option to allocate a fixed size heap.
> */
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
OK, I should have replied to this aspect before, sorry about that.

I'm sure I'm not telling you anything here I just want to be clear we
are using the same terminology or that you at least understand what I'm
saying:

Note that under Linux there are two (really more than two, but two
major) memory figures.  One is Virtual Set Size (VSZ) and the other is
Resident Set Size (RSS).  It's nothing new that with Squeak on Linux if
you don't specify a -memory option it mmaps a gigabyte and this is
reflected in the VSZ, in my experience this value never changes, I'm not
sure what would happen if more than a gigabyte was actually addressed.
I'm not at all concerned about the VSZ value because this does not count
against true allocated RAM.  So I may be wrong but I believe what you
are referring to in VSZ and is irrelevant to us.

My problem instead is with RSS which represents an actual cost in used
RAM.  What I am expecting to see is that the RSS value I see is in some
way representative of the end of memory (vm parameter 3) figure.  This
is true initially, but as I've said before, when the end of memory drops
due to a GC, the RSS does not.

Setting the value of -memory is not going to help here.   That's simply
going to change the VSZ and set a lower ceiling, a limit on the RSS.
The reality is that the way the website is emulated there are short
periods of time in which it needs one or two hundred megabytes of
memory, but these times are short lived and this is evidence by the end
of memory figure dropping back to the normal base level.

Ken

signature.asc (197 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] What is meant by 'shrink' in the VM parameters?

johnmci
 
Ok, well I think you are into forensic here.  So let me make some suggestions.

Have a timer based task that wakes up and looks for the  allocated memory > what 200MB?
When that happens do as a one time shot a count of all the instance allocation sizes, then
look at dumping all the process stacks so you understand what everything is doing at the time that it wanted & was using 200MB.

I note the dreadful 1994 defaults for stack dumps are too low, you might need to look at that code that
creates the stack dump to print more data.  For Sophie we changed things from dumping 4000 bytes to providing upwards of 400Kbytes to help diagnose things.

If you find that you've a ByteArray that's 150MB then maybe you could override new: for ByteArray and place some
diagnostics for dumping the caller's stack so you can determine why it's wanting 150MB for a few 100 ms.


On 2010-10-22, at 12:37 PM, Ken Causey wrote:

> Note that under Linux there are two (really more than two, but two
> major) memory figures.  One is Virtual Set Size (VSZ) and the other is
> Resident Set Size (RSS).  It's nothing new that with Squeak on Linux if
> you don't specify a -memory option it mmaps a gigabyte and this is
> reflected in the VSZ, in my experience this value never changes, I'm not
> sure what would happen if more than a gigabyte was actually addressed.
> I'm not at all concerned about the VSZ value because this does not count
> against true allocated RAM.  So I may be wrong but I believe what you
> are referring to in VSZ and is irrelevant to us.

Yes the VSZ would include the GB mmap, this value is a promise(lie) about what the app could at anytime want.

>
> My problem instead is with RSS which represents an actual cost in used
> RAM.  What I am expecting to see is that the RSS value I see is in some
> way representative of the end of memory (vm parameter 3) figure.  This
> is true initially, but as I've said before, when the end of memory drops
> due to a GC, the RSS does not.

If you take a 50MB image and expand it for 100ms to 200MB then RSS will show 200MB,
If the image vm parameter 3 goes back to 60MB this will NOT be reflected in the RSS value because the
operating system memory manager gave you the 200MB, this value will decay back to 60MB+  over time
but that decay is function of some very smart algorithms in the operating system which depend on how much
RAM other process are using, how much CPU timing Squeak is using. As you know Squeak chews CPU so the
operating system might pull idle pages from Squeak at a slower rate.

Frankly here you need to do a "vm_stat 1"  to understand if the operating system is under any paging issues.
If the RSS is high and the vm parm 3 low, then I'd fully expect the operating system has zero paging going on
so there is NO effort on it's part to steal pages from the Squeak process because that's expensive and Squeak has
shown it wanted the pages so why pull them back?

It's simplistic to consider a high RSS as bad, I'd rather be concerned about seeing a page_in/out rate in vm_stat showing 30+ then I'd bitch...
Note that at 30 pages/second that's like 1000 ms for hard disk rotation/access/seek,  bad bad, but I guess in new school thought it's only 300ms.

I'll bet page_in/out rate is zero..


>
> Setting the value of -memory is not going to help here.   That's simply
> going to change the VSZ and set a lower ceiling, a limit on the RSS.
> The reality is that the way the website is emulated there are short
> periods of time in which it needs one or two hundred megabytes of
> memory, but these times are short lived and this is evidence by the end
> of memory figure dropping back to the normal base level.
>
> Ken

--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================





smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] What is meant by 'shrink' in the VM parameters?

Ken Causey-3
 
Thanks for the info John.  We'll take this and watch things for a while.

You are quite probably right about my reading too much into the RSS
value.  I've felt like on the occasions when the RSS does go down that
it is because it is moving into swap.  What has triggered my concern is
when free memory is down to single digit megabytes and swap is half
used.  Looking at the process list the biggest offender that seems to
stick out is the www.squeak.org process.  And so I'm working on it
perhaps worrying about issues of no real concern.  I still need a bit of
convincing though but you have provided information that will help me
decide.

Thank you,

Ken

On Fri, 2010-10-22 at 13:11 -0700, John M McIntosh wrote:

> Ok, well I think you are into forensic here.  So let me make some suggestions.
>
> Have a timer based task that wakes up and looks for the  allocated memory > what 200MB?
> When that happens do as a one time shot a count of all the instance allocation sizes, then
> look at dumping all the process stacks so you understand what everything is doing at the time that it wanted & was using 200MB.
>
> I note the dreadful 1994 defaults for stack dumps are too low, you might need to look at that code that
> creates the stack dump to print more data.  For Sophie we changed things from dumping 4000 bytes to providing upwards of 400Kbytes to help diagnose things.
>
> If you find that you've a ByteArray that's 150MB then maybe you could override new: for ByteArray and place some
> diagnostics for dumping the caller's stack so you can determine why it's wanting 150MB for a few 100 ms.
>
>
> On 2010-10-22, at 12:37 PM, Ken Causey wrote:
>
> > Note that under Linux there are two (really more than two, but two
> > major) memory figures.  One is Virtual Set Size (VSZ) and the other is
> > Resident Set Size (RSS).  It's nothing new that with Squeak on Linux if
> > you don't specify a -memory option it mmaps a gigabyte and this is
> > reflected in the VSZ, in my experience this value never changes, I'm not
> > sure what would happen if more than a gigabyte was actually addressed.
> > I'm not at all concerned about the VSZ value because this does not count
> > against true allocated RAM.  So I may be wrong but I believe what you
> > are referring to in VSZ and is irrelevant to us.
>
> Yes the VSZ would include the GB mmap, this value is a promise(lie) about what the app could at anytime want.
>
> >
> > My problem instead is with RSS which represents an actual cost in used
> > RAM.  What I am expecting to see is that the RSS value I see is in some
> > way representative of the end of memory (vm parameter 3) figure.  This
> > is true initially, but as I've said before, when the end of memory drops
> > due to a GC, the RSS does not.
>
> If you take a 50MB image and expand it for 100ms to 200MB then RSS will show 200MB,
> If the image vm parameter 3 goes back to 60MB this will NOT be reflected in the RSS value because the
> operating system memory manager gave you the 200MB, this value will decay back to 60MB+  over time
> but that decay is function of some very smart algorithms in the operating system which depend on how much
> RAM other process are using, how much CPU timing Squeak is using. As you know Squeak chews CPU so the
> operating system might pull idle pages from Squeak at a slower rate.
>
> Frankly here you need to do a "vm_stat 1"  to understand if the operating system is under any paging issues.
> If the RSS is high and the vm parm 3 low, then I'd fully expect the operating system has zero paging going on
> so there is NO effort on it's part to steal pages from the Squeak process because that's expensive and Squeak has
> shown it wanted the pages so why pull them back?
>
> It's simplistic to consider a high RSS as bad, I'd rather be concerned about seeing a page_in/out rate in vm_stat showing 30+ then I'd bitch...
> Note that at 30 pages/second that's like 1000 ms for hard disk rotation/access/seek,  bad bad, but I guess in new school thought it's only 300ms.
>
> I'll bet page_in/out rate is zero..
>
>
> >
> > Setting the value of -memory is not going to help here.   That's simply
> > going to change the VSZ and set a lower ceiling, a limit on the RSS.
> > The reality is that the way the website is emulated there are short
> > periods of time in which it needs one or two hundred megabytes of
> > memory, but these times are short lived and this is evidence by the end
> > of memory figure dropping back to the normal base level.
> >
> > Ken
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>
>


signature.asc (197 bytes) Download Attachment