What is meant by 'shrink' in the VM parameters?

> I'm wondering what 'shrink' means in the following VM parameter
> comments:
>
> "24 memory threshold above which shrinking object memory (rw)"
>
> and
>
> "32 number of shrink memory requests (read-only)"
>
> My first thought is that a shrink represents Squeak release memory back
> to the operating system once it is no longer needed.
>
> As I've mentioned before we are trying to reduce the RAM footprint of
> the www.squeak.org process and are watching various related figures.
>
> We can see the value in VM parameter 32 increase, but while the value
> for parameter 3 (end of memory) goes down as you would expect, the
> operating system's view of the memory usage does not change.
>
> Ken
>

> I'm wondering what 'shrink' means in the following VM parameter
> comments:
>
> "24 memory threshold above which shrinking object memory (rw)"
>
> and
>
> "32 number of shrink memory requests (read-only)"
>
> My first thought is that a shrink represents Squeak release memory back
> to the operating system once it is no longer needed.
>
> As I've mentioned before we are trying to reduce the RAM footprint of
> the www.squeak.org process and are watching various related figures.
>
> We can see the value in VM parameter 32 increase, but while the value
> for parameter 3 (end of memory) goes down as you would expect, the
> operating system's view of the memory usage does not change.
>
> Ken
>

> It *would* give the memory back to the operating system.
> But we don't anymore again read the following
>
> /* Note:
> *
> *   The code allows memory to be overallocated; i.e., the initial
> *   block is reserved via mmap() and then the unused portion
> *   munmap()ped from the top end.  This is INHERENTLY DANGEROUS since
> *   malloc() may randomly map new memory in the block we "reserved"
> *   and subsequently unmap()ped.  Enabling this causes crashes in
> *   Croquet, which makes heavy use of the FFI and thus calls malloc()
> *   all over the place.
> *  
> *   For this reason, overallocateMemory is DISABLED by default.
> *  
> *   The upshot of all this is that Squeak will claim (and hold on to)
> *   ALL of the available virtual memory (or at least 75% of it) when
> *   it starts up.  If you can't live with that, use the -memory
> *   option to allocate a fixed size heap.
> */
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================

>
> On 2010-10-22, at 6:24 AM, Ken Causey wrote:
>
> > I'm wondering what 'shrink' means in the following VM parameter
> > comments:
> >
> > "24 memory threshold above which shrinking object memory (rw)"
> >
> > and
> >
> > "32 number of shrink memory requests (read-only)"
> >
> > My first thought is that a shrink represents Squeak release memory back
> > to the operating system once it is no longer needed.
> >
> > As I've mentioned before we are trying to reduce the RAM footprint of
> > the www.squeak.org process and are watching various related figures.
> >
> > We can see the value in VM parameter 32 increase, but while the value
> > for parameter 3 (end of memory) goes down as you would expect, the
> > operating system's view of the memory usage does not change.
> >
> > Ken
> >
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>
>

> Ok, well I think you are into forensic here.  So let me make some suggestions.
>
> Have a timer based task that wakes up and looks for the  allocated memory > what 200MB?
> When that happens do as a one time shot a count of all the instance allocation sizes, then
> look at dumping all the process stacks so you understand what everything is doing at the time that it wanted & was using 200MB.
>
> I note the dreadful 1994 defaults for stack dumps are too low, you might need to look at that code that
> creates the stack dump to print more data.  For Sophie we changed things from dumping 4000 bytes to providing upwards of 400Kbytes to help diagnose things.
>
> If you find that you've a ByteArray that's 150MB then maybe you could override new: for ByteArray and place some
> diagnostics for dumping the caller's stack so you can determine why it's wanting 150MB for a few 100 ms.
>
>
> On 2010-10-22, at 12:37 PM, Ken Causey wrote:
>
> > Note that under Linux there are two (really more than two, but two
> > major) memory figures.  One is Virtual Set Size (VSZ) and the other is
> > Resident Set Size (RSS).  It's nothing new that with Squeak on Linux if
> > you don't specify a -memory option it mmaps a gigabyte and this is
> > reflected in the VSZ, in my experience this value never changes, I'm not
> > sure what would happen if more than a gigabyte was actually addressed.
> > I'm not at all concerned about the VSZ value because this does not count
> > against true allocated RAM.  So I may be wrong but I believe what you
> > are referring to in VSZ and is irrelevant to us.
>
> Yes the VSZ would include the GB mmap, this value is a promise(lie) about what the app could at anytime want.
>
> >
> > My problem instead is with RSS which represents an actual cost in used
> > RAM.  What I am expecting to see is that the RSS value I see is in some
> > way representative of the end of memory (vm parameter 3) figure.  This
> > is true initially, but as I've said before, when the end of memory drops
> > due to a GC, the RSS does not.
>
> If you take a 50MB image and expand it for 100ms to 200MB then RSS will show 200MB,
> If the image vm parameter 3 goes back to 60MB this will NOT be reflected in the RSS value because the
> operating system memory manager gave you the 200MB, this value will decay back to 60MB+  over time
> but that decay is function of some very smart algorithms in the operating system which depend on how much
> RAM other process are using, how much CPU timing Squeak is using. As you know Squeak chews CPU so the
> operating system might pull idle pages from Squeak at a slower rate.
>
> Frankly here you need to do a "vm_stat 1"  to understand if the operating system is under any paging issues.
> If the RSS is high and the vm parm 3 low, then I'd fully expect the operating system has zero paging going on
> so there is NO effort on it's part to steal pages from the Squeak process because that's expensive and Squeak has
> shown it wanted the pages so why pull them back?
>
> It's simplistic to consider a high RSS as bad, I'd rather be concerned about seeing a page_in/out rate in vm_stat showing 30+ then I'd bitch...
> Note that at 30 pages/second that's like 1000 ms for hard disk rotation/access/seek,  bad bad, but I guess in new school thought it's only 300ms.
>
> I'll bet page_in/out rate is zero..
>
>
> >
> > Setting the value of -memory is not going to help here.   That's simply
> > going to change the VSZ and set a lower ceiling, a limit on the RSS.
> > The reality is that the way the website is emulated there are short
> > periods of time in which it needs one or two hundred megabytes of
> > memory, but these times are short lived and this is evidence by the end
> > of memory figure dropping back to the normal base level.
> >
> > Ken
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>
>

> On Fri, Oct 22, 2010 at 1:31 PM, Ken Causey <[hidden email]> wrote:
> > Thanks for the info John.  We'll take this and watch things for a while.
> >
> > You are quite probably right about my reading too much into the RSS
> > value.  I've felt like on the occasions when the RSS does go down that
> > it is because it is moving into swap.  What has triggered my concern is
> > when free memory is down to single digit megabytes and swap is half
> > used.
>
> Hi Ken,
>
> Would you remind us what the original problem was? If I understand
> your recent messages on the subject, it seems that a) Squeak isn't
> using large amounts of memory and b) the memory used isn't creeping up
> over time. Is it just that the RSS is significantly larger than what
> the VM reports it's using for the object memory, and you're wondering
> why the kernel leaving stuff paged in?
>
> Colin