[Glass] gc settings

> Otto,
>
> I talked this over with one of the guys here and he suggests that the trick
> is to not create shadow pages faster than the gc gem can handle them ... the
> gcgems are pretty aggreessive by default ...
>
> He suggests that you  commit frequently and the  monitor the
> PagesNeedsReclaiming stat in your migration loop and when it gets above a
> certain threshold (a couple thousand pages?) stop migrating and go into an
> "idle loop" where you abort (to avoid CR backlog and allow pages to run
> through their normal life cycle) and check for the stat to drop below
> another threshold (250 pages? once it's below your threshold get going again
> ...
>
> regarding the epoch gc. the become operation does not itself generate a lot
> of dead if you dow a become between a persistent object and a temp object
> and commit, the temp object will be written as the persist object and no
> garbage will be produced ... you will however create shadowpages as the new
> versions of the objects are copied to new pages (page reclaim is why the
> gcgems get revved up) ... now if you are creating short-lived persist
> objects as some other part of your  migration process then an epoch will
> help keep those dead boys in check ...
>
> Dale
>
>
> On Fri, Feb 14, 2014 at 4:07 AM, Otto Behrens <[hidden email]> wrote:
>>
>> Hi,
>>
>> Do you have a set of GcUser settings that we can use for aggressive
>> garbage collection during a migration process? We would like to do the
>> garbage collection (epoch and reclaim) while the migration process is
>> generating a lot of dead stuff. We don't mind if the migration takes a
>> bit longer. We do mind if the migration generates a lot of free space
>> in the extent.
>>
>> Thanks
>> Otto
>> _______________________________________________
>> Glass mailing list
>> [hidden email]
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>

> Yeah a couple thousand was a very conservative figure :) ... depends upon
> how much extra space you can afford in your repo ... at 4M (16k) pages,
> that's 64G which might be bigger than a lot of GLASS data bases (not all,
> though:)
>
> Dale
>
>
> On Fri, Feb 14, 2014 at 2:32 PM, Jon Paynter <[hidden email]> wrote:
>>
>> Another option is to allocate extra time for extra aggressive cleanup of
>> your shadow pages.  Then let the migration run all out for as long as it
>> needs.  When its done, kick everyone out of the stone and spin up a bunch of
>> reclaim gems.  Like Dale said, they quite fast.
>>
>> Im not sure where the "couple thousand" number came from -- but we will
>> only pause a migration for cleanup if the shadowpages get above 4million.
>>
>>
>> On Fri, Feb 14, 2014 at 1:35 PM, Otto Behrens <[hidden email]> wrote:
>>>
>>> Thanks a lot. Trying it out.
>>>
>>> On Fri, Feb 14, 2014 at 9:44 PM, Dale Henrichs
>>> <[hidden email]> wrote:
>>> > Otto,
>>> >
>>> > I talked this over with one of the guys here and he suggests that the
>>> > trick
>>> > is to not create shadow pages faster than the gc gem can handle them
>>> > ... the
>>> > gcgems are pretty aggreessive by default ...
>>> >
>>> > He suggests that you  commit frequently and the  monitor the
>>> > PagesNeedsReclaiming stat in your migration loop and when it gets above
>>> > a
>>> > certain threshold (a couple thousand pages?) stop migrating and go into
>>> > an
>>> > "idle loop" where you abort (to avoid CR backlog and allow pages to run
>>> > through their normal life cycle) and check for the stat to drop below
>>> > another threshold (250 pages? once it's below your threshold get going
>>> > again
>>> > ...
>>> >
>>> > regarding the epoch gc. the become operation does not itself generate a
>>> > lot
>>> > of dead if you dow a become between a persistent object and a temp
>>> > object
>>> > and commit, the temp object will be written as the persist object and
>>> > no
>>> > garbage will be produced ... you will however create shadowpages as the
>>> > new
>>> > versions of the objects are copied to new pages (page reclaim is why
>>> > the
>>> > gcgems get revved up) ... now if you are creating short-lived persist
>>> > objects as some other part of your  migration process then an epoch
>>> > will
>>> > help keep those dead boys in check ...
>>> >
>>> > Dale
>>> >
>>> >
>>> > On Fri, Feb 14, 2014 at 4:07 AM, Otto Behrens <[hidden email]>
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> Do you have a set of GcUser settings that we can use for aggressive
>>> >> garbage collection during a migration process? We would like to do the
>>> >> garbage collection (epoch and reclaim) while the migration process is
>>> >> generating a lot of dead stuff. We don't mind if the migration takes a
>>> >> bit longer. We do mind if the migration generates a lot of free space
>>> >> in the extent.
>>> >>
>>> >> Thanks
>>> >> Otto
>>> >> _______________________________________________
>>> >> Glass mailing list
>>> >> [hidden email]
>>> >> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>> >
>>> >
>>> _______________________________________________
>>> Glass mailing list
>>> [hidden email]
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>>
>>
>> _______________________________________________
>> Glass mailing list
>> [hidden email]
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>
>
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>