cleaning/shrinking the stone?

Hi,
I need some advice here... this is the problem: I have a gemstone running who is near reach the 4G (it is 3.7G right now), and I want to reduce the amount of space used, because I'm sure the real objects are not so many to use that bunch of memory: if I passive them, I get no more than 15M of data... and If I restore the passivated objects in a fresh glass instalation, the occupied space is around 850M... so I think all the remaining space is discardable, but I don't know what are those objects nor how to discard them.
Any help?

Cheers,
Esteban

Esteban,

do you mean you are looking at the file size? The file does not shrink if you delete objects it will only grow. That means that there is free space inside the file. If you really want to shrink the file than you need to backup, copy a new extent and do a full backup restore. Than the new extent will only grow to the size of the actual repository.

Norbert

On 23.03.2010, at 14:24, Esteban Lorenzano wrote:

> Hi,
> I need some advice here... this is the problem: I have a gemstone running who is near reach the 4G (it is 3.7G right now), and I want to reduce the amount of space used, because I'm sure the real objects are not so many to use that bunch of memory: if I passive them, I get no more than 15M of data... and If I restore the passivated objects in a fresh glass instalation, the occupied space is around 850M... so I think all the remaining space is discardable, but I don't know what are those objects nor how to discard them.
> Any help?
>
> Cheers,
> Esteban

The Systems Administator guide gives a couple commands you can use to
see how large the extent is and how much free space there is.

The backup and full restore will as Norbert says, reclaim the free space.

On Tue, Mar 23, 2010 at 10:04 AM, Norbert Hartl <[hidden email]> wrote:

The first thing to do is to produce a "File Size Report" (Admin...>DoIt...>File Size Report) and it will look like this:

Extent #1
-----------
   Filename = !TCP@localhost#dbf!/export/foos2/users/dhenrich/2.0/h_gss64bit/fast50/gs/product/seaside/data/extent0.dbf

   File size =       204.00 Megabytes
   Space available = 86.72 Megabytes

Totals
------
   Repository size = 204.00 Megabytes
   Free Space =      86.72 Megabytes

If you have lots of freespace and want to shrink the size of the extent file, then do a backup and restore into a virgin extent (one copied from $GEMSTONE/bin/extent0.dbf).

So the key piece of information is the free space. If the freespace is large, then you are okay. If the freespace is small, then you've probably got cruft in your repository beyond your 15M object model.

I assume that you've been running MFC's regularly (if not do that first).

Danie and Otto have shared some of their cleanup scripts that get rid of things like class histories, method version histories, Object Log, etc. I'll try to track it down in a bit (unless they beat me to the punch:)

If you're still not happy with the gains, you might take a look at the technique that James describes for determining what objects are in your data base ... it will give you instance counts by class:

  http://programminggems.wordpress.com/2009/12/15/scanbackup-2/

If you are going this route, it would be useful to see an instance count report of the initial "bloated" extent, so you can see the kinds of gains you get from Danie/Otto's techniques.

Dale

----- "Esteban Lorenzano" <[hidden email]> wrote:

| Hi,
| I need some advice here... this is the problem: I have a gemstone
| running who is near reach the 4G (it is 3.7G right now), and I want to
| reduce the amount of space used, because I'm sure the real objects are
| not so many to use that bunch of memory: if I passive them, I get no
| more than 15M of data... and If I restore the passivated objects in a
| fresh glass instalation, the occupied space is around 850M... so I
| think all the remaining space is discardable, but I don't know what
| are those objects nor how to discard them.
| Any help?
|
| Cheers,
| Esteban

I've collected doits suggested by Otto, Danie and Gerhard. I've classified them into SAFE, PROBABLY SAFE?, PROBABLY NOT SAFE. There should be no problems running the "SAFE" doits. The others I'm not completely convinced are healthy ... to be far some of these expressions were used to clean up an image in preparation for bootstrapping fresh code, so they weren't intended to be safe:) Also the #nilDefault methods don't exist:

"SAFE"
 ObjectLogEntry emptyLog.
 MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].
 MCDefinition clearInstances.
 MCMethodDefinition cachedDefinitions  removeKeys:
    (MCMethodDefinition cachedDefinitions keys).
 MCMethodDefinition shutDown.
 MethodVersionHistory uniqueInstance cleanUp.

"PROBABLY SAFE?"
 MADescriptionBuilder default flush.
 MADescriptionBuilder nilDefault.

"PROBABLY NOT SAFE"
 MCPackageManager nilRegistry.
 MCPackageManager allSubclasses do: [:sc | sc nilRegistry ].
 MCWorkingCopy nilRegistry.
 MCRepositoryGroup nilDefault.
 MCCacheRepository nilDefault.
 WADispatcher nilDefault.
 WALibrary nilDefault.
 WALibrary allSubclasses do: [:sc | sc nilDefault ].
 WAFileHandler nilDefault.
 PackageOrganizer nilDefault.

Another thing to keep in mind when cleaning up extents. The MFC does not have an immediate effect of free space (and in the short term may actually reduce free space as the live objects are moved off of pages containing dead objects). the stone goes through a whole sequence of operations to reclaim dead objects. For more info about garbage collection see:

  http://programminggems.wordpress.com/2010/02/02/video-17/
  http://programminggems.wordpress.com/2010/02/03/video-18/ 
  http://programminggems.wordpress.com/2010/02/05/video-20/

Dale
 
----- "Esteban Lorenzano" <[hidden email]> wrote:

| Hi,
| I need some advice here... this is the problem: I have a gemstone
| running who is near reach the 4G (it is 3.7G right now), and I want to
| reduce the amount of space used, because I'm sure the real objects are
| not so many to use that bunch of memory: if I passive them, I get no
| more than 15M of data... and If I restore the passivated objects in a
| fresh glass instalation, the occupied space is around 850M... so I
| think all the remaining space is discardable, but I don't know what
| are those objects nor how to discard them.
| Any help?
|
| Cheers,
| Esteban

Here's a log of File Size Reports and MFC's that I've run on a small development repository. Long story short, started with 88M free ended with 117M free:

STARTING POINT
------------------------------------------
Extent #1
-----------
   Filename = !TCP@localhost#dbf!/export/foos2/users/dhenrich/2.0/h_gss64bit/fast50/gs/product/seaside/data/extent0.dbf

   File size =       204.00 Megabytes
   Space available = 87.98 Megabytes

Totals
------
   Repository size = 204.00 Megabytes
   Free Space =      87.98 Megabytes
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MFC #1
------------------------------------------
---Starting mark for collect. (23 March 2010 10:12:26 am)
        AbortingError 3020: Successful completion of markForCollection.
    <917781> live objects found.
    <174557> possible dead objects, occupying approximately <15710130> bytes, may be reclaimed.
---Finished mark for collect. (23 March 2010 10:12:28 am)

Extent #1
-----------
   Filename = !TCP@localhost#dbf!/export/foos2/users/dhenrich/2.0/h_gss64bit/fast50/gs/product/seaside/data/extent0.dbf

   File size =       204.00 Megabytes
   Space available = 47.95 Megabytes

Totals
------
   Repository size = 204.00 Megabytes
   Free Space =      47.95 Megabytes
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
run "safe" cleanups
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MFC #2
------------------------------------------
---Starting mark for collect. (23 March 2010 10:16:41 am)
        AbortingError 3020: Successful completion of markForCollection.
    <738076> live objects found.
    <179757> possible dead objects, occupying approximately <16178130> bytes, may be reclaimed.
---Finished mark for collect. (23 March 2010 10:16:43 am)
Extent #1
-----------
   Filename = !TCP@localhost#dbf!/export/foos2/users/dhenrich/2.0/h_gss64bit/fast50/gs/product/seaside/data/extent0.dbf

   File size =       204.00 Megabytes
   Space available = 71.56 Megabytes

Totals
------
   Repository size = 204.00 Megabytes
   Free Space =      71.56 Megabytes
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
An hour or so later

Extent #1
-----------
   Filename = !TCP@localhost#dbf!/export/foos2/users/dhenrich/2.0/h_gss64bit/fast50/gs/product/seaside/data/extent0.dbf

   File size =       204.00 Megabytes
   Space available = 116.94 Megabytes

Totals
------
   Repository size = 204.00 Megabytes
   Free Space =      116.94 Megabytes
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MFC #3
------------------------------------------
---Starting mark for collect. (23 March 2010 11:13:17 am)
        AbortingError 3020: Successful completion of markForCollection.
    <710299> live objects found.
    <2248> possible dead objects, occupying approximately <202320> bytes, may be reclaimed.
---Finished mark for collect. (23 March 2010 11:13:18 am)

Dale
----- "Dale Henrichs" <[hidden email]> wrote:

| I've collected doits suggested by Otto, Danie and Gerhard. I've
| classified them into SAFE, PROBABLY SAFE?, PROBABLY NOT SAFE. There
| should be no problems running the "SAFE" doits. The others I'm not
| completely convinced are healthy ... to be far some of these
| expressions were used to clean up an image in preparation for
| bootstrapping fresh code, so they weren't intended to be safe:) Also
| the #nilDefault methods don't exist:
|
| "SAFE"
|  ObjectLogEntry emptyLog.
|  MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].
|  MCDefinition clearInstances.
|  MCMethodDefinition cachedDefinitions  removeKeys:
|     (MCMethodDefinition cachedDefinitions keys).
|  MCMethodDefinition shutDown.
|  MethodVersionHistory uniqueInstance cleanUp.
|
| "PROBABLY SAFE?"
|  MADescriptionBuilder default flush.
|  MADescriptionBuilder nilDefault.
|
| "PROBABLY NOT SAFE"
|  MCPackageManager nilRegistry.
|  MCPackageManager allSubclasses do: [:sc | sc nilRegistry ].
|  MCWorkingCopy nilRegistry.
|  MCRepositoryGroup nilDefault.
|  MCCacheRepository nilDefault.
|  WADispatcher nilDefault.
|  WALibrary nilDefault.
|  WALibrary allSubclasses do: [:sc | sc nilDefault ].
|  WAFileHandler nilDefault.
|  PackageOrganizer nilDefault.
|
| Another thing to keep in mind when cleaning up extents. The MFC does
| not have an immediate effect of free space (and in the short term may
| actually reduce free space as the live objects are moved off of pages
| containing dead objects). the stone goes through a whole sequence of
| operations to reclaim dead objects. For more info about garbage
| collection see:
|
|   http://programminggems.wordpress.com/2010/02/02/video-17/
|   http://programminggems.wordpress.com/2010/02/03/video-18/ 
|   http://programminggems.wordpress.com/2010/02/05/video-20/
|
| Dale
|  
| ----- "Esteban Lorenzano" <[hidden email]> wrote:
|
| | Hi,
| | I need some advice here... this is the problem: I have a gemstone
| | running who is near reach the 4G (it is 3.7G right now), and I want
| to
| | reduce the amount of space used, because I'm sure the real objects
| are
| | not so many to use that bunch of memory: if I passive them, I get
| no
| | more than 15M of data... and If I restore the passivated objects in
| a
| | fresh glass instalation, the occupied space is around 850M... so I
| | think all the remaining space is discardable, but I don't know what
| | are those objects nor how to discard them.
| | Any help?
| |
| | Cheers,
| | Esteban

Ok, making a full backup/restore solves the problem (it backs the dbf to 850M)... but I still would like to know why it grows so much.
Maybe I'm doing something wrong? or not doing something?
Thanks to all for the help!!!

Cheers,
Esteban  

El 23/03/2010, a las 3:16p.m., Dale Henrichs escribió:

Esteban,

How often do you run your MFC?

Dale
----- "Esteban Lorenzano" <[hidden email]> wrote:

| Ok, making a full backup/restore solves the problem (it backs the dbf
| to 850M)... but I still would like to know why it grows so much.
| Maybe I'm doing something wrong? or not doing something?
| Thanks to all for the help!!!
|
| Cheers,
| Esteban  
|
| El 23/03/2010, a las 3:16p.m., Dale Henrichs escribió:
|
| > Here's a log of File Size Reports and MFC's that I've run on a small
| development repository. Long story short, started with 88M free ended
| with 117M free:
| >
| > STARTING POINT
| > ------------------------------------------
| > Extent #1
| > -----------
| >   Filename =
| !TCP@localhost#dbf!/export/foos2/users/dhenrich/2.0/h_gss64bit/fast50/gs/product/seaside/data/extent0.dbf
| >
| >   File size =       204.00 Megabytes
| >   Space available = 87.98 Megabytes
| >
| > Totals
| > ------
| >   Repository size = 204.00 Megabytes
| >   Free Space =      87.98 Megabytes
| >
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| > MFC #1
| > ------------------------------------------
| > ---Starting mark for collect. (23 March 2010 10:12:26 am)
| > AbortingError 3020: Successful completion of markForCollection.
| >    <917781> live objects found.
| >    <174557> possible dead objects, occupying approximately
| <15710130> bytes, may be reclaimed.
| > ---Finished mark for collect. (23 March 2010 10:12:28 am)
| >
| > Extent #1
| > -----------
| >   Filename =
| !TCP@localhost#dbf!/export/foos2/users/dhenrich/2.0/h_gss64bit/fast50/gs/product/seaside/data/extent0.dbf
| >
| >   File size =       204.00 Megabytes
| >   Space available = 47.95 Megabytes
| >
| > Totals
| > ------
| >   Repository size = 204.00 Megabytes
| >   Free Space =      47.95 Megabytes
| >
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| > run "safe" cleanups
| >
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| > MFC #2
| > ------------------------------------------
| > ---Starting mark for collect. (23 March 2010 10:16:41 am)
| > AbortingError 3020: Successful completion of markForCollection.
| >    <738076> live objects found.
| >    <179757> possible dead objects, occupying approximately
| <16178130> bytes, may be reclaimed.
| > ---Finished mark for collect. (23 March 2010 10:16:43 am)
| > Extent #1
| > -----------
| >   Filename =
| !TCP@localhost#dbf!/export/foos2/users/dhenrich/2.0/h_gss64bit/fast50/gs/product/seaside/data/extent0.dbf
| >
| >   File size =       204.00 Megabytes
| >   Space available = 71.56 Megabytes
| >
| > Totals
| > ------
| >   Repository size = 204.00 Megabytes
| >   Free Space =      71.56 Megabytes
| >
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| > An hour or so later
| >
| > Extent #1
| > -----------
| >   Filename =
| !TCP@localhost#dbf!/export/foos2/users/dhenrich/2.0/h_gss64bit/fast50/gs/product/seaside/data/extent0.dbf
| >
| >   File size =       204.00 Megabytes
| >   Space available = 116.94 Megabytes
| >
| > Totals
| > ------
| >   Repository size = 204.00 Megabytes
| >   Free Space =      116.94 Megabytes
| >
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| > MFC #3
| > ------------------------------------------
| > ---Starting mark for collect. (23 March 2010 11:13:17 am)
| > AbortingError 3020: Successful completion of markForCollection.
| >    <710299> live objects found.
| >    <2248> possible dead objects, occupying approximately <202320>
| bytes, may be reclaimed.
| > ---Finished mark for collect. (23 March 2010 11:13:18 am)
| >
| > Dale
| > ----- "Dale Henrichs" <[hidden email]> wrote:
| >
| > | I've collected doits suggested by Otto, Danie and Gerhard. I've
| > | classified them into SAFE, PROBABLY SAFE?, PROBABLY NOT SAFE.
| There
| > | should be no problems running the "SAFE" doits. The others I'm
| not
| > | completely convinced are healthy ... to be far some of these
| > | expressions were used to clean up an image in preparation for
| > | bootstrapping fresh code, so they weren't intended to be safe:)
| Also
| > | the #nilDefault methods don't exist:
| > |
| > | "SAFE"
| > |  ObjectLogEntry emptyLog.
| > |  MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache
| ].
| > |  MCDefinition clearInstances.
| > |  MCMethodDefinition cachedDefinitions  removeKeys:
| > |     (MCMethodDefinition cachedDefinitions keys).
| > |  MCMethodDefinition shutDown.
| > |  MethodVersionHistory uniqueInstance cleanUp.
| > |
| > | "PROBABLY SAFE?"
| > |  MADescriptionBuilder default flush.
| > |  MADescriptionBuilder nilDefault.
| > |
| > | "PROBABLY NOT SAFE"
| > |  MCPackageManager nilRegistry.
| > |  MCPackageManager allSubclasses do: [:sc | sc nilRegistry ].
| > |  MCWorkingCopy nilRegistry.
| > |  MCRepositoryGroup nilDefault.
| > |  MCCacheRepository nilDefault.
| > |  WADispatcher nilDefault.
| > |  WALibrary nilDefault.
| > |  WALibrary allSubclasses do: [:sc | sc nilDefault ].
| > |  WAFileHandler nilDefault.
| > |  PackageOrganizer nilDefault.
| > |
| > | Another thing to keep in mind when cleaning up extents. The MFC
| does
| > | not have an immediate effect of free space (and in the short term
| may
| > | actually reduce free space as the live objects are moved off of
| pages
| > | containing dead objects). the stone goes through a whole sequence
| of
| > | operations to reclaim dead objects. For more info about garbage
| > | collection see:
| > |
| > |   http://programminggems.wordpress.com/2010/02/02/video-17/
| > |   http://programminggems.wordpress.com/2010/02/03/video-18/ 
| > |   http://programminggems.wordpress.com/2010/02/05/video-20/
| > |
| > | Dale
| > |  
| > | ----- "Esteban Lorenzano" <[hidden email]> wrote:
| > |
| > | | Hi,
| > | | I need some advice here... this is the problem: I have a
| gemstone
| > | | running who is near reach the 4G (it is 3.7G right now), and I
| want
| > | to
| > | | reduce the amount of space used, because I'm sure the real
| objects
| > | are
| > | | not so many to use that bunch of memory: if I passive them, I
| get
| > | no
| > | | more than 15M of data... and If I restore the passivated objects
| in
| > | a
| > | | fresh glass instalation, the occupied space is around 850M... so
| I
| > | | think all the remaining space is discardable, but I don't know
| what
| > | | are those objects nor how to discard them.
| > | | Any help?
| > | |
| > | | Cheers,
| > | | Esteban