[Glass] Backups, GC and removing translogs

Hi guys,

I want to know what do you think about the following scenario. I want to start making some cleaning for my gemstones. After reading several emails etc I came to the following steps that I would like to run DAILY:

1) Stop all seaside gems (including maintainance etc)

2) Perform MFC and reclaim:

[

System beginTransaction.

SystemRepository markForCollection.

SystemRepository reclaimAll.

System commitTransaction.

] on: Halt,  Warning do: [:ex | ex resume]

This step is not mandatory for me since I do run GLASS maintainance VM which does a MFC every hour...but...it might be a good idea anyway. 

3) Perform backup :

SystemRepository startNewLog.

SystemRepository fullBackupCompressedTo: '/opt/somewhere/backup-dateXXX.dbf.gz'.

4) Clean:

Now...from what I understand, this script: https://github.com/noha/stone-creator/blob/master/bin/delete-old-tranlogs.sh  does:

LAST_NEEDED_TRANLOG=`$COPYDBF_EXE -i $FILE 2>&1 | grep tranlog | cut -d ' ' -f 13`

I have just checked copydbf and indeed, seems to be answer the same as #oldestLogFileIdForRecovery. So it seems safe to use this tool, correct? 

However...from what I understand..the critical part there is the $FILE which should be the last backup (in my example the latest in /opt/somewhere/). Is this correct? 

Mariano,

I think this sounds fine overall ... just a few minor comments in-lined

On Mon, Aug 11, 2014 at 1:33 PM, Mariano Martinez Peck <[hidden email]> wrote:

Hi guys,

I want to know what do you think about the following scenario. I want to start making some cleaning for my gemstones. After reading several emails etc I came to the following steps that I would like to run DAILY:

1) Stop all seaside gems (including maintainance etc)

2) Perform MFC and reclaim:

[

System beginTransaction.

SystemRepository markForCollection.

SystemRepository reclaimAll.

System commitTransaction.

] on: Halt,  Warning do: [:ex | ex resume]

This step is not mandatory for me since I do run GLASS maintainance VM which does a MFC every hour...but...it might be a good idea anyway. 

If you are going to run an mfc once a day, then I would suggest that you can probably get away with eliminating the hourly mfc and run on epoch gc instead. The mfc  introduces excessive disk i/o as  every page (with live objects) is dragged through the SPC and at the end, you will need to pull your "standard" working set of pages in off of disk..

3) Perform backup :

SystemRepository startNewLog.

SystemRepository fullBackupCompressedTo: '/opt/somewhere/backup-dateXXX.dbf.gz'.

It's a good idea to validate your backup after the backup is completed. Take a look at the backup script that I use up on SS3[1] for an example using copydbf. This at least avoids issues with file level corruption. I believe that some folks do a restore from backup of the newly created backup file along with some validation which covers any logical corruption. Whether or not you go this far really depends upon how critical the backups are ...

For super critical backups you will want to arrange for off-site storage of backups and tranlogs.

In the backup script you will see that I force the creation of a new tranlog right before the backup ...

Frankly I forget why I put that trick in there:), but the idea is that you will avoid having monster tranlogs lying around ...

[1] https://github.com/glassdb/webEditionHome/blob/master/bin/startGemStoneBackup.sh#L65-83

4) Clean:

Now...from what I understand, this script: https://github.com/noha/stone-creator/blob/master/bin/delete-old-tranlogs.sh  does:

LAST_NEEDED_TRANLOG=`$COPYDBF_EXE -i $FILE 2>&1 | grep tranlog | cut -d ' ' -f 13`

I have just checked copydbf and indeed, seems to be answer the same as #oldestLogFileIdForRecovery. So it seems safe to use this tool, correct? 

I don't know offhand, but if Norbert put it in his script, then I trust it:)

 

However...from what I understand..the critical part there is the $FILE which should be the last backup (in my example the latest in /opt/somewhere/). Is this correct? 

If you are keeping several backups around you need to keep the tranlogs that apply to the oldest backup that you are keeping... Using the latest backup file will indeed give you the list of old and new tranlogs needed for the latest, but there may be a tranlog or two that are still needed for the oldest backup that you are keeping.  

On Mon, Aug 11, 2014 at 6:15 PM, Dale Henrichs <[hidden email]> wrote:

Mariano,

I think this sounds fine overall ... just a few minor comments in-lined

Hi Dale, thanks, I answer below. 

 

On Mon, Aug 11, 2014 at 1:33 PM, Mariano Martinez Peck <[hidden email]> wrote:

Hi guys,

I want to know what do you think about the following scenario. I want to start making some cleaning for my gemstones. After reading several emails etc I came to the following steps that I would like to run DAILY:

1) Stop all seaside gems (including maintainance etc)

2) Perform MFC and reclaim:

[

System beginTransaction.

SystemRepository markForCollection.

SystemRepository reclaimAll.

System commitTransaction.

] on: Halt,  Warning do: [:ex | ex resume]

This step is not mandatory for me since I do run GLASS maintainance VM which does a MFC every hour...but...it might be a good idea anyway. 

If you are going to run an mfc once a day, then I would suggest that you can probably get away with eliminating the hourly mfc and run on epoch gc instead. The mfc  introduces excessive disk i/o as  every page (with live objects) is dragged through the SPC and at the end, you will need to pull your "standard" working set of pages in off of disk..

Yes, I was thinking the same. Once I have this GC+backup+clean stuff working, I will disable hourly MFC from GLASS'  maintainance VM. 

 

3) Perform backup :

SystemRepository startNewLog.

SystemRepository fullBackupCompressedTo: '/opt/somewhere/backup-dateXXX.dbf.gz'.

It's a good idea to validate your backup after the backup is completed. Take a look at the backup script that I use up on SS3[1] for an example using copydbf. This at least avoids issues with file level corruption. I believe that some folks do a restore from backup of the newly created backup file along with some validation which covers any logical corruption. Whether or not you go this far really depends upon how critical the backups are ...

This is a good idea. Thanks. I will try to add this soon. 

 

For super critical backups you will want to arrange for off-site storage of backups and tranlogs.

I, I was thinking about that too. Mostly...tranlogs. Because right now, with my scheme, if I have a disk failure, I loose the last 24 hours edits because translogs are only written in same disk and only backup at night. But I guess this is a second step for me. 

 

In the backup script you will see that I force the creation of a new tranlog right before the backup ...

Yes, but I am already doing that, right? (see lines above). I mean, I send #startNewLog before #fullBackupCompressedTo: 

 

Frankly I forget why I put that trick in there:), but the idea is that you will avoid having monster tranlogs lying around ...

[1] https://github.com/glassdb/webEditionHome/blob/master/bin/startGemStoneBackup.sh#L65-83

4) Clean:

Now...from what I understand, this script: https://github.com/noha/stone-creator/blob/master/bin/delete-old-tranlogs.sh  does:

LAST_NEEDED_TRANLOG=`$COPYDBF_EXE -i $FILE 2>&1 | grep tranlog | cut -d ' ' -f 13`

I have just checked copydbf and indeed, seems to be answer the same as #oldestLogFileIdForRecovery. So it seems safe to use this tool, correct? 

I don't know offhand, but if Norbert put it in his script, then I trust it:)

 

However...from what I understand..the critical part there is the $FILE which should be the last backup (in my example the latest in /opt/somewhere/). Is this correct? 

If you are keeping several backups around you need to keep the tranlogs that apply to the oldest backup that you are keeping... Using the latest backup file will indeed give you the list of old and new tranlogs needed for the latest, but there may be a tranlog or two that are still needed for the oldest backup that you are keeping.  

Here is my main question now. Originally, I thought: I want to a daily backup of the last 7 days (or whatever number). And then, I would need all the tranlogs needed for the oldest day, let's say the 7th. So, originally I thought my backup script would move those 7 backup files and all needed tranlogs to my backup disk. And always keep that: 7 backups and needed tranlogs for last day. 

But now I wonder....I would be moving the tranlogs at the same time I move the backups (once a day, at night). Hence, I think it doesn't make sense. Why? Because why would I want an older backup + tranlogs, if I already have the next days backup as well. Is there any advantage of keeping the tranlogs that I am not seeing?  Maybe to take an older backup and do cherry picking of the transactions? 

 

Of course, a different thing would be if I directly write the tranlogs in my backup disk as well. Because there I could have even the last 24 hours edits.