Solving some Garbage Collection issues

Hi Dale,

I just now added them to the configs and will let you know in a couple of days if it worked ;-)

On 10 Nov 2011, at 18:16, Dale Henrichs wrote:

> Johan,
>
> I was curious if you have set
>
>  GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90
>
> in your config files ... I'm running with this setting for SS3 and I am not seeing unreasonable growth (I see a steady state number of objects voted down). Either the GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE is helping (even with only 90% flushed) or the character of the data structures involved are different or we are looking at a different issue ...
>
> Dale
>
> ----- Original Message -----
> | From: "Johan Brichau" <[hidden email]>
> | To: "GemStone Seaside beta discussion" <[hidden email]>
> | Sent: Tuesday, November 8, 2011 2:21:43 AM
> | Subject: Re: [GS/SS Beta] Solving some Garbage Collection issues
> |
> | Hi Carla,
> |
> | We are experiencing similar repository growth on a continuous basis.
> |
> | In a steadily used GLASS installation, we see a weekly build-up of
> | roughly 2 to 3 gigabytes of data that is continuously being
> | collected by the MFC but never reclaimed. This starts to put a lot
> | more stress on the MFC cycles (we do them nightly only).
> |
> | The only solution we found (and which was suggested by Dale on this
> | list) is that we restart all seaside gems (and the maintenance gem
> | if you have one) without restarting the stone. The MFC/reclaim cycle
> | that follows this restart is guaranteed to clean up all of the dead
> | objects. Somehow, the seaside gems are preventing the reclaim.
> |
> | We execute this restart on a weekly basis. Sometimes, I even do a
> | backup/restore of the stone. According to the manual, this compacts
> | the object table. I am not sure if this backup/restore really helps,
> | but sometimes I got the impression that MFC operations became
> | quicker after that.
> |
> | I am also not very comfortable with this behavior and, from time to
> | time, I try to investigate the cause but I have been unsuccessful so
> | far.
> | Dale also suggested the following in a previous mail on this list. I
> | did not try this yet.
> |
> | > You might try the following in your system.conf:
> | >
> | >  GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90
> | >
> | > #=========================================================================
> | > # GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE: Percent of pom generation area
> | > #   to be cleared when voting on possible dead objects.
> | > #   If value is > 0 and < 100, subspaces of pom generation older
> | > #   than 5 minutes are cleared; the number of subspaces cleared is
> | > #   the specified percentage of spaces in use rounded down to an
> | > #   integral number of spaces.
> | > #   If value == 100, all subspaces of pom generation are cleared
> | > without
> | > #   regard to their age.
> | > #
> | > # If this value is not specified, or the specified value is out of
> | > range,
> | > # the default is used.
> | >
> | > You won't get a complete flush of POM objects on vote but
> | > eventually you'll flush out the older references... If you are
> | > experiencing this problem.
> |
> |
> | cheers
> | Johan
> |
> | On 07 Nov 2011, at 16:40, Carla F. Griggio wrote:
> |
> | > Hi everyone! (Specially Dale :P)
> | >
> | > I've been having some garbage collection issues: I think my
> | > repository should have 1GB top of stored objects, and it's always
> | > around 2.5GB (sometimes it even gets annoyingly large, like 7GB or
> | > so). The thing is that although some Mark For Collection is done
> | > from time to time, the reclaim of dead objects always fails due to
> | > the Seaside Gems running and holding on to the repository.
> | > So last friday during Smalltalks 2011 here in Argentina I grabbed
> | > Dale and he explained what to do about it.
> | >
> | > So, Dale, you told me to follow this list:
> | >
> | > 1) Run the "SAFE" list from this post to clean the image:
> | > http://forum.world.st/cleaning-shrinking-the-stone-td1679041.html#a1679436
> | > 2) Shut Down
> | > 3) File Size Report (I think this should actually be the first
> | > step, maybe?)
> | > 4) Start Stone
> | > 5) MFC
> | > 6) Do SystemRepository reclaimAll
> | > 7) Check again the File Size Report
> | > 8) If the file size is not what I expect, try to find references
> | > holding to what I supposed should be dead objects, and get rid of
> | > them. Then repeat from step 5?
> | >
> | > This should be a script to be executed by a cron process during
> | > moments I know the system is not being used (for example: saturday
> | > nights).
> | >
> | > While trying to do this, I found some unexpected problems... So
> | > here's the list I actually followed and what happened:
> | >
> | > 1) File Size Report:
> | > File size =       10502.00 Megabytes
> | > Space available = 8968.34 Megabytes
> | >
> | > Note: It's funny that today the repository wasn't following the
> | > pattern I described in the beggining... If everyday I had this
> | > file size report results I wouldn't really have a problem :P But
> | > well... I still continued through the list.
> | >
> | > 2) Clean Image
> | > ObjectLogEntry emptyLog.
> | >  MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache
> | >  ].
> | >  MCDefinition clearInstances.
> | >  MCMethodDefinition cachedDefinitions  removeKeys:
> | >     (MCMethodDefinition cachedDefinitions keys).
> | >  MCMethodDefinition shutDown.
> | >  MethodVersionHistory uniqueInstance cleanUp.
> | >
> | > 3) Here I should have done a Shut Down of Gemstone and then start
> | > only the stone, but for some reason I couldnt. Although I stoped
> | > Gemstone, some processes were still running (admingc, reclaim,
> | > shared page monitor, etc) and apparently one of them was holding
> | > the repository, so I couldn't start the stone.
> | > I tried rebooting the whole system, and then do everything again,
> | > but I got the same results.
> | > What I ended up doing was rebooting, and with the whole gemstone
> | > system running OK, only shut down the seaside gems and mantainance
> | > gem.
> | >
> | > 4) MFC
> | > 5) SystemRepository reclaimAll
> | >
> | > 6) File Size Report:
> | >
> | > File size =       10502.00 Megabytes
> | > Space available = 8321.94 Megabytes
> | >
> | > Wow, you can see I then had less space available!
> | >
> | > But I waited a little while longer and then I got these results:
> | >
> | > File size =       10502.00 Megabytes
> | > Space available = 9109.39 Megabytes
> | >
> | > :) That's better :) But I still think the size could be a little
> | > smaller.
> | >
> | > The admingcgem log said this:
> | >
> | > Starting doSweepWsUnion at 11/07/11 11:56:52 ART
> | >   Starting values: WSU size=0  PD size=1584716
> | >   Finished first GarRemoveNotDead at 11/07/11 11:56:52 ART. Removed
> | >   0 objects from possibleDead
> | > [Info]: WsUnion during first sweep size = 0
> | > [Info]: Finished second GarRemoveNotDead at 11/07/11 11:56:52 ART.
> | >  Removed 0 objs from possibleDead
> | > [Info]: SweepWsUnion ending possibleDead size = 1584716
> | >   Finished sweeping possible dead at 11/07/11 11:56:52 ART.
> | >   possible dead size=1584716  notDead size=0
> | >
> | > And the reclaimgcgem log said this:
> | >
> | > 11/07/11 11:57:55 ART
> | >    1 reclaims  130 pagesProcessed  130 pagesReclaimed  10
> | >    allValidPages  8 singleObjPages
> | >    20243 processedObjs  14940 liveObjs  13 shadowObjs 5290 deadObjs
> | >     155.7 avgObjsPerPage
> | > [Info]: Parameter changes noticed on '11/07/11 12:07:23 ART':
> | >   reclaimMinPages = 1 Pages.
> | >
> | >
> | > Dale, you explained something to me about the relation between
> | > possible dead size and notDead size. What was that? If notDead
> | > size = 0 it means that there's nothing more to be reclaimed? Or
> | > should possible dead size be = 0?
> | >
> | > Gracias!
> | >
> | > Carla.
> | >
> |
> |

Hi Dale, Carla,

It's now been a week since I set the parameter below and I have not seen any unreasonable growth since.

GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90

Thanks for the info!
Johan

On 10 Nov 2011, at 21:27, Johan Brichau wrote:

> Hi Dale,
>
> I just now added them to the configs and will let you know in a couple of days if it worked ;-)
>
>
> On 10 Nov 2011, at 18:16, Dale Henrichs wrote:
>
>> Johan,
>>
>> I was curious if you have set
>>
>> GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90
>>
>> in your config files ... I'm running with this setting for SS3 and I am not seeing unreasonable growth (I see a steady state number of objects voted down). Either the GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE is helping (even with only 90% flushed) or the character of the data structures involved are different or we are looking at a different issue ...
>>
>> Dale
>>
>> ----- Original Message -----
>> | From: "Johan Brichau" <[hidden email]>
>> | To: "GemStone Seaside beta discussion" <[hidden email]>
>> | Sent: Tuesday, November 8, 2011 2:21:43 AM
>> | Subject: Re: [GS/SS Beta] Solving some Garbage Collection issues
>> |
>> | Hi Carla,
>> |
>> | We are experiencing similar repository growth on a continuous basis.
>> |
>> | In a steadily used GLASS installation, we see a weekly build-up of
>> | roughly 2 to 3 gigabytes of data that is continuously being
>> | collected by the MFC but never reclaimed. This starts to put a lot
>> | more stress on the MFC cycles (we do them nightly only).
>> |
>> | The only solution we found (and which was suggested by Dale on this
>> | list) is that we restart all seaside gems (and the maintenance gem
>> | if you have one) without restarting the stone. The MFC/reclaim cycle
>> | that follows this restart is guaranteed to clean up all of the dead
>> | objects. Somehow, the seaside gems are preventing the reclaim.
>> |
>> | We execute this restart on a weekly basis. Sometimes, I even do a
>> | backup/restore of the stone. According to the manual, this compacts
>> | the object table. I am not sure if this backup/restore really helps,
>> | but sometimes I got the impression that MFC operations became
>> | quicker after that.
>> |
>> | I am also not very comfortable with this behavior and, from time to
>> | time, I try to investigate the cause but I have been unsuccessful so
>> | far.
>> | Dale also suggested the following in a previous mail on this list. I
>> | did not try this yet.
>> |
>> | > You might try the following in your system.conf:
>> | >
>> | >  GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90
>> | >
>> | > #=========================================================================
>> | > # GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE: Percent of pom generation area
>> | > #   to be cleared when voting on possible dead objects.
>> | > #   If value is > 0 and < 100, subspaces of pom generation older
>> | > #   than 5 minutes are cleared; the number of subspaces cleared is
>> | > #   the specified percentage of spaces in use rounded down to an
>> | > #   integral number of spaces.
>> | > #   If value == 100, all subspaces of pom generation are cleared
>> | > without
>> | > #   regard to their age.
>> | > #
>> | > # If this value is not specified, or the specified value is out of
>> | > range,
>> | > # the default is used.
>> | >
>> | > You won't get a complete flush of POM objects on vote but
>> | > eventually you'll flush out the older references... If you are
>> | > experiencing this problem.
>> |
>> |
>> | cheers
>> | Johan
>> |
>> | On 07 Nov 2011, at 16:40, Carla F. Griggio wrote:
>> |
>> | > Hi everyone! (Specially Dale :P)
>> | >
>> | > I've been having some garbage collection issues: I think my
>> | > repository should have 1GB top of stored objects, and it's always
>> | > around 2.5GB (sometimes it even gets annoyingly large, like 7GB or
>> | > so). The thing is that although some Mark For Collection is done
>> | > from time to time, the reclaim of dead objects always fails due to
>> | > the Seaside Gems running and holding on to the repository.
>> | > So last friday during Smalltalks 2011 here in Argentina I grabbed
>> | > Dale and he explained what to do about it.
>> | >
>> | > So, Dale, you told me to follow this list:
>> | >
>> | > 1) Run the "SAFE" list from this post to clean the image:
>> | > http://forum.world.st/cleaning-shrinking-the-stone-td1679041.html#a1679436
>> | > 2) Shut Down
>> | > 3) File Size Report (I think this should actually be the first
>> | > step, maybe?)
>> | > 4) Start Stone
>> | > 5) MFC
>> | > 6) Do SystemRepository reclaimAll
>> | > 7) Check again the File Size Report
>> | > 8) If the file size is not what I expect, try to find references
>> | > holding to what I supposed should be dead objects, and get rid of
>> | > them. Then repeat from step 5?
>> | >
>> | > This should be a script to be executed by a cron process during
>> | > moments I know the system is not being used (for example: saturday
>> | > nights).
>> | >
>> | > While trying to do this, I found some unexpected problems... So
>> | > here's the list I actually followed and what happened:
>> | >
>> | > 1) File Size Report:
>> | > File size =       10502.00 Megabytes
>> | > Space available = 8968.34 Megabytes
>> | >
>> | > Note: It's funny that today the repository wasn't following the
>> | > pattern I described in the beggining... If everyday I had this
>> | > file size report results I wouldn't really have a problem :P But
>> | > well... I still continued through the list.
>> | >
>> | > 2) Clean Image
>> | > ObjectLogEntry emptyLog.
>> | >  MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache
>> | >  ].
>> | >  MCDefinition clearInstances.
>> | >  MCMethodDefinition cachedDefinitions  removeKeys:
>> | >     (MCMethodDefinition cachedDefinitions keys).
>> | >  MCMethodDefinition shutDown.
>> | >  MethodVersionHistory uniqueInstance cleanUp.
>> | >
>> | > 3) Here I should have done a Shut Down of Gemstone and then start
>> | > only the stone, but for some reason I couldnt. Although I stoped
>> | > Gemstone, some processes were still running (admingc, reclaim,
>> | > shared page monitor, etc) and apparently one of them was holding
>> | > the repository, so I couldn't start the stone.
>> | > I tried rebooting the whole system, and then do everything again,
>> | > but I got the same results.
>> | > What I ended up doing was rebooting, and with the whole gemstone
>> | > system running OK, only shut down the seaside gems and mantainance
>> | > gem.
>> | >
>> | > 4) MFC
>> | > 5) SystemRepository reclaimAll
>> | >
>> | > 6) File Size Report:
>> | >
>> | > File size =       10502.00 Megabytes
>> | > Space available = 8321.94 Megabytes
>> | >
>> | > Wow, you can see I then had less space available!
>> | >
>> | > But I waited a little while longer and then I got these results:
>> | >
>> | > File size =       10502.00 Megabytes
>> | > Space available = 9109.39 Megabytes
>> | >
>> | > :) That's better :) But I still think the size could be a little
>> | > smaller.
>> | >
>> | > The admingcgem log said this:
>> | >
>> | > Starting doSweepWsUnion at 11/07/11 11:56:52 ART
>> | >   Starting values: WSU size=0  PD size=1584716
>> | >   Finished first GarRemoveNotDead at 11/07/11 11:56:52 ART. Removed
>> | >   0 objects from possibleDead
>> | > [Info]: WsUnion during first sweep size = 0
>> | > [Info]: Finished second GarRemoveNotDead at 11/07/11 11:56:52 ART.
>> | >  Removed 0 objs from possibleDead
>> | > [Info]: SweepWsUnion ending possibleDead size = 1584716
>> | >   Finished sweeping possible dead at 11/07/11 11:56:52 ART.
>> | >   possible dead size=1584716  notDead size=0
>> | >
>> | > And the reclaimgcgem log said this:
>> | >
>> | > 11/07/11 11:57:55 ART
>> | >    1 reclaims  130 pagesProcessed  130 pagesReclaimed  10
>> | >    allValidPages  8 singleObjPages
>> | >    20243 processedObjs  14940 liveObjs  13 shadowObjs 5290 deadObjs
>> | >     155.7 avgObjsPerPage
>> | > [Info]: Parameter changes noticed on '11/07/11 12:07:23 ART':
>> | >   reclaimMinPages = 1 Pages.
>> | >
>> | >
>> | > Dale, you explained something to me about the relation between
>> | > possible dead size and notDead size. What was that? If notDead
>> | > size = 0 it means that there's nothing more to be reclaimed? Or
>> | > should possible dead size be = 0?
>> | >
>> | > Gracias!
>> | >
>> | > Carla.
>> | >
>> |
>> |
>