HostAllocateFromReservedMemory: mmap() failed on squeaksource3 build

As you might know I provide the build for squeaksource3 on my jenkins server

https://ci.hartl.name/job/squeaksource3-gs24/

Tobias complaint about failing builds due to out of memory conditions. Then I had a closer look but I'm not sure how to interpret. The build on jenkins is a two phase build. First phase loads the code with

---
Gofer new
squeaksource: 'MetacelloRepository';
package: 'ConfigurationOfSqueakSource';
load.

MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
 ((Smalltalk at: #ConfigurationOfSqueakSource) project version: #bleedingEdge) load: #( 'All') ].
---

Second phase loads test code and executes tests with

---
((Smalltalk at: #ConfigurationOfSqueakSource) project version: #bleedingEdge) load: 'Tests'].

HDTestReport runPackages: #('SqueakSource-Tests').
---

This was failing due to a temporary object memory full. I tweaked the setting to be

---
MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
   ((Smalltalk at: #ConfigurationOfSqueakSource) project version: #bleedingEdge) load: 'Tests'].
%
commit
doit
MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
   HDTestReport runPackages: #('SqueakSource-Tests')].
---

To have as much memory for processing as possible. Running tests however eats up 1,5GB memory within approx. 5 seconds. It is 1,5GB because that is the space left on the machine. After that I get the error message in the subject because the machine runs out of memory. The size of the shared page cache is 100MB and temp object memory is 50MB. What can drive GemStone to use that much of memory? And how do I fix the issue for squeaksource3?

thanks,

Norbert

Norbert,

Before going too far, you can increase the amount of temp-obj space that a gem uses. I assume that you are using `topaz` so the first thing to do is to bump up the size of the temp-obj cache when you start the gem with the following command:

  topaz -T100000

which should give you a 100M cache ... then see if you are still running out of memory.

I'm confused that you say that:

  Running tests however eats up 1,5GB memory within
  approx. 5 seconds

The temp-obj space is pegged as a maximum value and the temp-obj space does not consume available memory. There is a C heap that the gems use that can grow until available memory is exhausted, but you get a different error message.

So running out of temp-obj space means that you consumed the 50M that you allocated ...

So the first steps are to increase the size of the temp-obj cache to see if you can get a successful run in.

If you are curious why you are running out of memory, there should be information in the log file showing the stack and the contents of memory at the moment that you ran out of temp-obj space.

If you'd like to have a little control (i.e., interactively check things out) then I've got a post[1], that describes some techniques for debugging your out of memory condition.

Dale

[1] http://gemstonesoup.wordpress.com/2008/11/19/gemstone-101-managing-out-of-memory-situations/

----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Wednesday, February 15, 2012 5:39:22 AM
| Subject: [GS/SS Beta] HostAllocateFromReservedMemory: mmap() failed on squeaksource3 build
|
| As you might know I provide the build for squeaksource3 on my jenkins
| server
|
| https://ci.hartl.name/job/squeaksource3-gs24/
|
| Tobias complaint about failing builds due to out of memory
| conditions. Then I had a closer look but I'm not sure how to
| interpret. The build on jenkins is a two phase build. First phase
| loads the code with
|
| ---
| Gofer new
| squeaksource: 'MetacelloRepository';
| package: 'ConfigurationOfSqueakSource';
| load.
|
| MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
|  ((Smalltalk at: #ConfigurationOfSqueakSource) project version:
|  #bleedingEdge) load: #( 'All') ].
| ---
|
| Second phase loads test code and executes tests with
|
| ---
| ((Smalltalk at: #ConfigurationOfSqueakSource) project version:
| #bleedingEdge) load: 'Tests'].
|
| HDTestReport runPackages: #('SqueakSource-Tests').
| ---
|
| This was failing due to a temporary object memory full. I tweaked the
| setting to be
|
| ---
| MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
|    ((Smalltalk at: #ConfigurationOfSqueakSource) project version:
|    #bleedingEdge) load: 'Tests'].
| %
| commit
| doit
| MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
|    HDTestReport runPackages: #('SqueakSource-Tests')].
| ---
|
| To have as much memory for processing as possible. Running tests
| however eats up 1,5GB memory within approx. 5 seconds. It is 1,5GB
| because that is the space left on the machine. After that I get the
| error message in the subject because the machine runs out of memory.
| The size of the shared page cache is 100MB and temp object memory is
| 50MB. What can drive GemStone to use that much of memory? And how do
| I fix the issue for squeaksource3?
|
| thanks,
|
| Norbert
|
|

Am 15.02.2012 um 18:40 schrieb Dale Henrichs:

> Norbert,
>
> Before going too far, you can increase the amount of temp-obj space that a gem uses. I assume that you are using `topaz` so the first thing to do is to bump up the size of the temp-obj cache when you start the gem with the following command:
>
>  topaz -T100000
>
> which should give you a 100M cache ... then see if you are still running out of memory.
>
But then I would get a "temporary object memory full", right? I use a commitOnAlmostOutOfMemory handler. So under normal circumstances I shouldn't get complaints about the temporary object memory

> I'm confused that you say that:
>
>  Running tests however eats up 1,5GB memory within
>  approx. 5 seconds
>
To be clear, I mean that

> ps -eo pid,rss,cmd | grep topaz

PID     RSS      CMD
19907 1648584 /opt/gemstone/product/bin/topaz -l -T200000

and it takes 5-10 seconds going from the usual 70-90MB I see for a topaz process to the 1,6GB you can see above.

> The temp-obj space is pegged as a maximum value and the temp-obj space does not consume available memory. There is a C heap that the gems use that can grow until available memory is exhausted, but you get a different error message.
>
Ok, as the gems are not using shared memory directly but copying the needed objects into their heap it might outgrow, right?

> So running out of temp-obj space means that you consumed the 50M that you allocated ...
>
I'm runnnig out of free machine memory.

> So the first steps are to increase the size of the temp-obj cache to see if you can get a successful run in.
>
Same thing.

> If you are curious why you are running out of memory, there should be information in the log file showing the stack and the contents of memory at the moment that you ran out of temp-obj space.
>
I'll need to search for the gem logfile first because it is not written into hudson workspace as it seems.

> If you'd like to have a little control (i.e., interactively check things out) then I've got a post[1], that describes some techniques for debugging your out of memory condition.
>
I can "fix it" somehow. I discovered that I run out of memory because the tests are running twice. The first time they run the topaz eats up to 850MB. The memory is not freed until the second run and running the tests a second time outgrows the available memory on the machine. I don't have much time to analyze what's going on. Maybe Tobias can give more information what happens in the test what might cause it.

Norbert

Norbert,

Sorry, I misunderstood the problem that you were having temp-obj cache issues, sorry ... Nonetheless, using a commitOnAlmostOutOfMemory handler does not guarantee that you won't run out of temp-obj cache ... The commitOnAlmostOutOfMemory only works if there are a significant number of dirty persistent objects populating vm memory ... If the temporary objects are not rooted in a persistent root, then committing won't help the situation.

Now on to the virtual memory issue...

The best way to get a handle on virtual memory issues to run statmonitor during your test run  and then ship us the file for analysis:

  statmonitor <stonename> -A -i 1 -u 0 -z

With the statmon file, we should be able to get a handle on exactly what is causing you to run out of virtual memory ...

As a rule of thumb a vm will use somewhere around 2 times the TOC allocation and `ps` includes shared page cache pages in RSS. The back of the envelope says that your vm grow only by max of 400M, but yours is growing by a Gig, so we can't use the "rule of thumb" and look at the actual allocations to understand what might be happening....

Dale

----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Thursday, February 16, 2012 3:48:46 AM
| Subject: Re: [GS/SS Beta] HostAllocateFromReservedMemory: mmap() failed on squeaksource3 build
|
|
| Am 15.02.2012 um 18:40 schrieb Dale Henrichs:
|
| > Norbert,
| >
| > Before going too far, you can increase the amount of temp-obj space
| > that a gem uses. I assume that you are using `topaz` so the first
| > thing to do is to bump up the size of the temp-obj cache when you
| > start the gem with the following command:
| >
| >  topaz -T100000
| >
| > which should give you a 100M cache ... then see if you are still
| > running out of memory.
| >
| But then I would get a "temporary object memory full", right? I use a
| commitOnAlmostOutOfMemory handler. So under normal circumstances I
| shouldn't get complaints about the temporary object memory
|
| > I'm confused that you say that:
| >
| >  Running tests however eats up 1,5GB memory within
| >  approx. 5 seconds
| >
| To be clear, I mean that
|
| > ps -eo pid,rss,cmd | grep topaz
|
| PID     RSS      CMD
| 19907 1648584 /opt/gemstone/product/bin/topaz -l -T200000
|
| and it takes 5-10 seconds going from the usual 70-90MB I see for a
| topaz process to the 1,6GB you can see above.
|
| > The temp-obj space is pegged as a maximum value and the temp-obj
| > space does not consume available memory. There is a C heap that
| > the gems use that can grow until available memory is exhausted,
| > but you get a different error message.
| >
| Ok, as the gems are not using shared memory directly but copying the
| needed objects into their heap it might outgrow, right?
|
| > So running out of temp-obj space means that you consumed the 50M
| > that you allocated ...
| >
| I'm runnnig out of free machine memory.
|
| > So the first steps are to increase the size of the temp-obj cache
| > to see if you can get a successful run in.
| >
| Same thing.
|
| > If you are curious why you are running out of memory, there should
| > be information in the log file showing the stack and the contents
| > of memory at the moment that you ran out of temp-obj space.
| >
| I'll need to search for the gem logfile first because it is not
| written into hudson workspace as it seems.
|
| > If you'd like to have a little control (i.e., interactively check
| > things out) then I've got a post[1], that describes some
| > techniques for debugging your out of memory condition.
| >
| I can "fix it" somehow. I discovered that I run out of memory because
| the tests are running twice. The first time they run the topaz eats
| up to 850MB. The memory is not freed until the second run and
| running the tests a second time outgrows the available memory on the
| machine. I don't have much time to analyze what's going on. Maybe
| Tobias can give more information what happens in the test what might
| cause it.
|
| Norbert
|
| > Dale
| >
| > [1]
| > http://gemstonesoup.wordpress.com/2008/11/19/gemstone-101-managing-out-of-memory-situations/
| >
| > ----- Original Message -----
| > | From: "Norbert Hartl" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Wednesday, February 15, 2012 5:39:22 AM
| > | Subject: [GS/SS Beta] HostAllocateFromReservedMemory: mmap()
| > | failed on squeaksource3 build
| > |
| > | As you might know I provide the build for squeaksource3 on my
| > | jenkins
| > | server
| > |
| > | https://ci.hartl.name/job/squeaksource3-gs24/
| > |
| > | Tobias complaint about failing builds due to out of memory
| > | conditions. Then I had a closer look but I'm not sure how to
| > | interpret. The build on jenkins is a two phase build. First phase
| > | loads the code with
| > |
| > | ---
| > | Gofer new
| > | squeaksource: 'MetacelloRepository';
| > | package: 'ConfigurationOfSqueakSource';
| > | load.
| > |
| > | MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
| > |  ((Smalltalk at: #ConfigurationOfSqueakSource) project version:
| > |  #bleedingEdge) load: #( 'All') ].
| > | ---
| > |
| > | Second phase loads test code and executes tests with
| > |
| > | ---
| > | ((Smalltalk at: #ConfigurationOfSqueakSource) project version:
| > | #bleedingEdge) load: 'Tests'].
| > |
| > | HDTestReport runPackages: #('SqueakSource-Tests').
| > | ---
| > |
| > | This was failing due to a temporary object memory full. I tweaked
| > | the
| > | setting to be
| > |
| > | ---
| > | MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
| > |    ((Smalltalk at: #ConfigurationOfSqueakSource) project version:
| > |    #bleedingEdge) load: 'Tests'].
| > | %
| > | commit
| > | doit
| > | MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
| > |    HDTestReport runPackages: #('SqueakSource-Tests')].
| > | ---
| > |
| > | To have as much memory for processing as possible. Running tests
| > | however eats up 1,5GB memory within approx. 5 seconds. It is
| > | 1,5GB
| > | because that is the space left on the machine. After that I get
| > | the
| > | error message in the subject because the machine runs out of
| > | memory.
| > | The size of the shared page cache is 100MB and temp object memory
| > | is
| > | 50MB. What can drive GemStone to use that much of memory? And how
| > | do
| > | I fix the issue for squeaksource3?
| > |
| > | thanks,
| > |
| > | Norbert
| > |
| > |
|
|