Garbage collection policies?

Hi folks,

we just improved the importer of Moose, it's now quite fast - but  
profiling shows that we spend 80% of the time in garbage collection.  
So I thought about giving the GC the hint that we are in a growth-
only phase.

The scenario is this - when loading a 20MB sized .mse file, first we  
read the complete file into one single literal array and then we  
traverse this array and create our business objects, which is about  
200MB of objects which we plan to keep, and finally we throw the  
initial array away.

Is there a way to communicate this to the GC to make it smarter?

Grüsse,
        Adrian K

Adrian Kuhn <[hidden email]> wrote:

| Hi folks,

| we just improved the importer of Moose, it's now quite fast - but
| profiling shows that we spend 80% of the time in garbage collection.
| So I thought about giving the GC the hint that we are in a growth-
| only phase.

| The scenario is this - when loading a 20MB sized .mse file, first we
| read the complete file into one single literal array and then we
| traverse this array and create our business objects, which is about
| 200MB of objects which we plan to keep, and finally we throw the
| initial array away.

| Is there a way to communicate this to the GC to make it smarter?

Try raising your growthregimeUpperBound to something above 200Mb.  See launcher->settings->memory policy...

| Grüsse,
| Adrian K
---
Eliot Miranda                 ,,,^..^,,,                mailto:[hidden email]
VisualWorks Engineering, Cincom  Smalltalk: scene not herd  Tel +1 408 216 4581
3350 Scott Blvd, Bldg 36 Suite B, Santa Clara, CA 95054 USA Fax +1 408 216 4500

We had the same problem. Simply loading LargeMemoryPolicy from the
public store database corrected the problem. Also read this post:
http://www.cincomsmalltalk.com/blog/blogView?showComments=true&entry=3229957285

Runar Jordahl

Hello Adrian,

Runar Jordahl schrieb am 23.09.2006 17:47:
> We had the same problem. Simply loading LargeMemoryPolicy from the
> public store database corrected the problem. Also read this post:
> http://www.cincomsmalltalk.com/blog/blogView?showComments=true&entry=3229957285

If that should not be sufficient for your application:

vwMemoryMgmt.pdf in the doc/TechNotes directory of the VisualWorks
distribution is an essential read if you have to tune the MemoryPolicy.

If you set growthRegimeUpperBound to more than 512 MB, you also have to
enlarge memoryUpperBound. It should be larger than
growthRegimeUpperBound by at least one or two times the value of
preferredGrowthIncrement or by 10%, whichever is bigger.

If you increase preferredGrowthIncrement (we found 10 MB to be a good
value for an application using several hundred MB of RAM), make sure to
set freeMemoryUpperBound to a value higher than
preferredGrowthIncrement. That's not documented, but we ran into
problems when freeMemoryUpperBound was smaller than
preferredGrowthIncrement ("out of memory" crashes of the Object Engine,
probably due to boundary conditions when OldSpace segments were returned
to the OS).

In applications creating lots of temporary objects, e.g. when processing
Strings or doing Float calculations, it is a good idea to enlarge some
of the startup sizes of ObjectMemory to prevent temporary objects from
being tenured into OldSpace. This reduces incremental garbage collection
activity. I usually set Eden and SurvivorSpace to at least 10 times the
default size, and also enlarge StackSpace.

If you need to tune the MemoryPolicy beyond that (e.g. tuning the
incremental garbage collector by modifying the MemoryPolicy), the
necessary modifications depend on the application's behavior. Luckily,
you don't have to go that far for most applications. If you are not very
familiar with the details of memory management in VW, it's actually
cheaper to hire John McIntosh for a week or two than trying to do it
yourself (and you'll also learn a lot for your money). At least, that's
what we did a couple of years ago, and it was one of the best ideas we
had at the time. ;-)

My favorite advice concerning GC: The best way to tune garbage
collection is not to create garbage. :-)

Joachim Geidel