[Glass] taming growth during migration

Thanks.

We have only the session doing the migrations logged in.

What I also noticed was that the repository grows close to the maximum
quite quickly and then leaves about 185M free on the disk where the
extent is. The process continues for quite a while at that point,
doing a number of migrations until it eventually goes over the free
space threshold. The reclaim gem is quite busy also.

Perhaps this is a support call?

Cheers
Otto

On Thu, Feb 5, 2015 at 7:20 PM, Bill Erickson
<[hidden email]> wrote:
> Otto,
>
> I just dealt with this same problem from another customer in Tech Support.
>
> One thing that is essential is that you keep your commit record backlog down
> as much as possible.  Make sure you don't have any sessions during this time
> period that are in transaction and not periodically committing/aborting, as
> this can cause serious DB growth using space to hang onto the commit
> records.  These also fill up the cache, flushing out needed data pages and
> slowing down other sessions as they will need to re-load needed pages back
> into the cache, which tends to exacerbate the problem.
>
> Regards,
> ------------------------------------------------------------------------
> Bill Erickson
> GemTalk Systems Engineering
> 15220 NW Greenbrier Parkway #240, Beaverton OR 97006
> ------------------------------------------------------------------------
>
> On Thu, Feb 5, 2015 at 5:55 AM, Otto Behrens via Glass
> <[hidden email]> wrote:
>>
>> Hi,
>>
>> We're migrating a large number of objects in our systems in this release.
>> Our problem is that while this migration is running, the repository grows
>> too much and runs out of space.
>>
>> We can just give it more space, but I feel it is a better solution to get
>> the garbage collector to clean up faster. So, if I'm not smoking my socks,
>> this boils down to tuning parameters so that the Gc keeps up with the
>> process that generates the garbage.
>>
>> We did the following which helped a lot: we commit in batches while
>> looping over objects and after each commit, we monitor PagesNeedReclaimSize
>> and wait for it to drop below 250 before we continue with the next batch.
>>
>> We played with the batch sizes. Too few objects in a batch increases the
>> commit rate so high it runs out of space very quickly. Too many creates a
>> larger number of objects to Gc in a epoch / reclaim. I found a batch size
>> that sortof works.
>>
>> What I can figure out from the stats (using VSD) is that the
>> PagesNeedReclaimSize stays low.
>>
>> After a restore, loaded new code, just before we start the migration
>> process:
>>
>> SystemRepository fileSizeReport
>>    File size =       10334.00 Megabytes
>>    Space available = 55.67 Megabytes
>>
>> Just after the stone stops allowings commits because of the free space
>> threshold (and the topaz session exits):
>>
>> SystemRepository fileSizeReport
>>    Repository size = 14000.00 Megabytes
>>    Free Space =      2732.08 Megabytes
>>
>> So, it appears as if there is enough free space to work with, but it does
>> not become available.
>>
>> I see this in the reclaimgcgem log, and wondered if I should lower /
>> increase this parameter:
>> "Committing the transaction. Reason: #objsMovedPerCommitThreshold
>> exceeded"
>>
>> If you have any ideas on this, or if you need logs / statmonitor files or
>> something else, please let me know. Still investigating...
>>
>> We are running GS 3.1.0.5.
>>
>> Cheers
>> Otto
>>
>> _______________________________________________
>> Glass mailing list
>> [hidden email]
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>

Hey Otto,

Depends on how deep you want to go.. ;-)

If I see a question on the GLASS list that I can throw out a quick response for (like yours), I will.  But for anything requiring looking at logs or detailed analysis, I have to switch over to my official "Tech Support" hat and direct you there.

Regards,
Bill Erickson

------------------------------------------------------------------------
Bill Erickson
GemTalk Systems Engineering
15220 NW Greenbrier Parkway #240, Beaverton OR 97006
------------------------------------------------------------------------

On Thu, Feb 5, 2015 at 9:32 AM, Otto Behrens <[hidden email]> wrote:

Thanks.

We have only the session doing the migrations logged in.

What I also noticed was that the repository grows close to the maximum
quite quickly and then leaves about 185M free on the disk where the
extent is. The process continues for quite a while at that point,
doing a number of migrations until it eventually goes over the free
space threshold. The reclaim gem is quite busy also.

Perhaps this is a support call?

Cheers
Otto

On Thu, Feb 5, 2015 at 7:20 PM, Bill Erickson
<[hidden email]> wrote:
> Otto,
>
> I just dealt with this same problem from another customer in Tech Support.
>
> One thing that is essential is that you keep your commit record backlog down
> as much as possible.  Make sure you don't have any sessions during this time
> period that are in transaction and not periodically committing/aborting, as
> this can cause serious DB growth using space to hang onto the commit
> records.  These also fill up the cache, flushing out needed data pages and
> slowing down other sessions as they will need to re-load needed pages back
> into the cache, which tends to exacerbate the problem.
>
> Regards,
> ------------------------------------------------------------------------
> Bill Erickson
> GemTalk Systems Engineering
> 15220 NW Greenbrier Parkway #240, Beaverton OR 97006
> ------------------------------------------------------------------------
>
> On Thu, Feb 5, 2015 at 5:55 AM, Otto Behrens via Glass
> <[hidden email]> wrote:
>>
>> Hi,
>>
>> We're migrating a large number of objects in our systems in this release.
>> Our problem is that while this migration is running, the repository grows
>> too much and runs out of space.
>>
>> We can just give it more space, but I feel it is a better solution to get
>> the garbage collector to clean up faster. So, if I'm not smoking my socks,
>> this boils down to tuning parameters so that the Gc keeps up with the
>> process that generates the garbage.
>>
>> We did the following which helped a lot: we commit in batches while
>> looping over objects and after each commit, we monitor PagesNeedReclaimSize
>> and wait for it to drop below 250 before we continue with the next batch.
>>
>> We played with the batch sizes. Too few objects in a batch increases the
>> commit rate so high it runs out of space very quickly. Too many creates a
>> larger number of objects to Gc in a epoch / reclaim. I found a batch size
>> that sortof works.
>>
>> What I can figure out from the stats (using VSD) is that the
>> PagesNeedReclaimSize stays low.
>>
>> After a restore, loaded new code, just before we start the migration
>> process:
>>
>> SystemRepository fileSizeReport
>>    File size =       10334.00 Megabytes
>>    Space available = 55.67 Megabytes
>>
>> Just after the stone stops allowings commits because of the free space
>> threshold (and the topaz session exits):
>>
>> SystemRepository fileSizeReport
>>    Repository size = 14000.00 Megabytes
>>    Free Space =      2732.08 Megabytes
>>
>> So, it appears as if there is enough free space to work with, but it does
>> not become available.
>>
>> I see this in the reclaimgcgem log, and wondered if I should lower /
>> increase this parameter:
>> "Committing the transaction. Reason: #objsMovedPerCommitThreshold
>> exceeded"
>>
>> If you have any ideas on this, or if you need logs / statmonitor files or
>> something else, please let me know. Still investigating...
>>
>> We are running GS 3.1.0.5.
>>
>> Cheers
>> Otto
>>
>> _______________________________________________
>> Glass mailing list
>> [hidden email]
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>