linux memory fragmentation

Hi,

I am noticing quite high i/o wait on our server running gemstone (with multiple stones).
Another thing I noticed was the high usage of the swap although real memory was still free.

It turns out that the memory fragmentation on our box is quite high. A lot of small memory chunks were available but not larger ones.
It appears that Linux prefers to hand out larger chunks from the swap instead of the fragmented pages in RAM.

Is it possible that Gemstone always requests small chunks of memory subsequently, leading to the fragmentation?
Is there a way to configure the Gemstone processes to allocate a larger chunk immediately?

Johan

Johan,

[paraphrasing after a discussion with our expert]

GemStone probably should allocate larger chunks of memory, but for now that aspect of the product is not configurable.

Linux ends up using swap space even though you have "plenty of real memory" under pressure from the file buffers, etc. We recommend that you size swap space to be 2x real memory to minimize swap space fragmentation.

If you've already allocated that amount of swap space, then I think we'd like to see some statmon output for each of the stones, so we can try to determine what else might be going on ...

Dale

----- Original Message -----
| From: "Johan Brichau" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Tuesday, August 2, 2011 3:07:11 AM
| Subject: [GS/SS Beta] linux memory fragmentation
|
| Hi,
|
| I am noticing quite high i/o wait on our server running gemstone
| (with multiple stones).
| Another thing I noticed was the high usage of the swap although real
| memory was still free.
|
| It turns out that the memory fragmentation on our box is quite high.
| A lot of small memory chunks were available but not larger ones.
| It appears that Linux prefers to hand out larger chunks from the swap
| instead of the fragmented pages in RAM.
|
| Is it possible that Gemstone always requests small chunks of memory
| subsequently, leading to the fragmentation?
| Is there a way to configure the Gemstone processes to allocate a
| larger chunk immediately?
|
| Johan

What would be the consequences of running without swap, from GemStone's
point of view?  I know you can configure a linux system without swap and
8GB of unbuffered ECC ram is $90.  The free license automatically limits
the shared page cache size to 2GB.   Wouldn't GemStone's internal
controls take care of the out-of-shared-page-cache condition before
running out of physical RAM if you had enough in your server?


My intuition is that if the server with 8GB RAM had no swap space and
only the free version of GemStone then you'd never have a problem.  But
I have no idea and of course I could set up a test environment and see
what happens but was curious if the consequences were already known.

Thanks

Paul


On 08/02/2011 12:38 PM, Dale Henrichs wrote:

Paul,

You are correct that in theory if you can get your entire system to run in memory you would be very happy. Whether 8GB is enough or not depends upon how many topaz processes you are running as servers and how you have tuned the OS.

I think it has to do with the way that Linux likes to work ... in theory you only need RAM for your application, but Unix systems have had a long history of pushing memory pages it thinks aren't needed to disk (swap space).

In the late 80's I routinely ran the Smalltalk images (Tektronix Smalltalk at the time) allocating twice as much memory for the process as was needed, because the OS only kept half of what you asked for in memory anyway:(

A quick scan through this article[1] (from 2007) yields a couple of interesting quotes:

  ...It is possible to run a Linux system without a swap space, and the system will
  run well if you have a large amount of memory -- but if you run out of physical
  memory then the system will crash...

  ...A rule of thumb is as follows:
    1) for a desktop system, use a swap space of double system memory, as it will
       allow you to run a large number of applications (many of which may will be
       idle and easily swapped), making more RAM available for the active
       applications;
    2) for a server, have a smaller amount of swap available (say half of physical
       memory) so that you have some flexibility for swapping when needed, but
       monitor the amount of swap space used and upgrade your RAM if necessary;
    3) for older desktop machines (with say only 128MB), use as much swap space
       as you can spare, even up to 1GB.

  ...The Linux 2.6 kernel added a new kernel parameter called swappiness to let
  administrators tweak the way Linux swaps. It is a number from 0 to 100. In
  essence, higher values lead to more pages being swapped, and lower values lead to
  more applications being kept in memory, even if they are idle....

The "It is possible" in the first quote leads me to believe that if you want to run your system entirely in memory, then buying more RAM is not enough...you'll have to tune your OS parameters as well.

Since GemStone has a number of different linux processes any of which may go idle during their normal course of operation, I would say that Rule 1 should be followed unless you undertake the task of tuning the system to ensure that the GemStone system is not swapped out.

The final quote implies that Linux is disposed to push idle memory pages out to swap space even if there is plenty of RAM available ...

Dale

[1] http://www.linux.com/news/software/applications/8208-all-about-linux-swap-space

----- Original Message -----
| From: "Paul DeBruicker" <[hidden email]>
| To: [hidden email]
| Sent: Tuesday, August 2, 2011 9:48:45 AM
| Subject: Re: [GS/SS Beta] linux memory fragmentation
|
| What would be the consequences of running without swap, from
| GemStone's
| point of view?  I know you can configure a linux system without swap
| and
| 8GB of unbuffered ECC ram is $90.  The free license automatically
| limits
| the shared page cache size to 2GB.   Wouldn't GemStone's internal
| controls take care of the out-of-shared-page-cache condition before
| running out of physical RAM if you had enough in your server?
|
|
| My intuition is that if the server with 8GB RAM had no swap space and
| only the free version of GemStone then you'd never have a problem.
|  But
| I have no idea and of course I could set up a test environment and
| see
| what happens but was curious if the consequences were already known.
|
| Thanks
|
| Paul
|
|
| On 08/02/2011 12:38 PM, Dale Henrichs wrote:
| > Johan,
| >
| > [paraphrasing after a discussion with our expert]
| >
| > GemStone probably should allocate larger chunks of memory, but for
| > now that aspect of the product is not configurable.
| >
| > Linux ends up using swap space even though you have "plenty of real
| > memory" under pressure from the file buffers, etc. We recommend
| > that you size swap space to be 2x real memory to minimize swap
| > space fragmentation.
| >
| > If you've already allocated that amount of swap space, then I think
| > we'd like to see some statmon output for each of the stones, so we
| > can try to determine what else might be going on ...
| >
| > Dale
| >
| > ----- Original Message -----
| > | From: "Johan Brichau"<[hidden email]>
| > | To: "GemStone Seaside beta discussion"<[hidden email]>
| > | Sent: Tuesday, August 2, 2011 3:07:11 AM
| > | Subject: [GS/SS Beta] linux memory fragmentation
| > |
| > | Hi,
| > |
| > | I am noticing quite high i/o wait on our server running gemstone
| > | (with multiple stones).
| > | Another thing I noticed was the high usage of the swap although
| > | real
| > | memory was still free.
| > |
| > | It turns out that the memory fragmentation on our box is quite
| > | high.
| > | A lot of small memory chunks were available but not larger ones.
| > | It appears that Linux prefers to hand out larger chunks from the
| > | swap
| > | instead of the fragmented pages in RAM.
| > |
| > | Is it possible that Gemstone always requests small chunks of
| > | memory
| > | subsequently, leading to the fragmentation?
| > | Is there a way to configure the Gemstone processes to allocate a
| > | larger chunk immediately?
| > |
| > | Johan
|
|

Dale,

Thanks for this expert response!

We probably need to allocate more swapspace indeed, but probably also more RAM:

There is only 500k swapspace allocated (which seems to be a standard linux server setup nowadays). This swap was almost immediately full but vmstat never showed a lot of swap-in/out happening, which lead me to believe that all was fine. As you say, Linux has this tendency to swap out stuff.

Also, the SPC statistics are fine, showing only high page read/writes during MFC.

But my guess is that we are also pushing it by running 10 stones on that box, each with 3 fastcgi gems and maintenance gems popping in as cron jobs.
I already asked for another box with more memory but I will ask to increase the swap on that box in the meantime.

thanks!
Johan

On 02 Aug 2011, at 18:38, Dale Henrichs wrote:

Hi Paul,

On 02 Aug 2011, at 18:48, Paul DeBruicker wrote:

> The free license automatically limits the shared page cache size to 2GB.   Wouldn't GemStone's internal controls take care of the out-of-shared-page-cache condition before running out of physical RAM if you had enough in your server?

That's true. In fact, statmonitor allows you to monitor that aspect of Gemstone quite well.
I am only seeing cache misses and page read spikes when doing an MFC, so I have the impression I am running as much in memory as possible.

The difference is that I'm running multiple stones on a single box. I was expecting memory problems (just by calculating) but never saw them happening until now. The reason might be a higher degree of usage of each of the stones (including larger sets of data in each stone).

In contrast, I had a gemstone running on a 256k slice host and there I saw the swap trashing happening quite clearly. Here, the memory fragmentation is still a strange phenomenon to me. Well… you learn something new every day.

Johan

On 02 Aug 2011, at 19:47, Johan Brichau wrote:

> But my guess is that we are also pushing it by running 10 stones on that box, each with 3 fastcgi gems and maintenance gems popping in as cron jobs.

I forgot to mention, but the box has 8Gb of RAM. Each stone has 500Mb SPC and the gems have a temp object space of max 50Mb.