Backup procedure
Locked
12
12
 
|
Ciao,
i need to setup procedure for backup my devKit glass environment. For now i use: This procedure do: runs a
GemStone full backup (compressed) and validates the resulting backup file
It work fine and startNewLog before do the backup. Now i need to complete the backup. I think: 1) save the new backup into another system into local network ( named backupSystem ) 2) remove the old backup from the server and from the backupSystem 3) remove the old tranlog from the server 4) remove other .... logs from the server Someone has already addressed this issue? The goal is to have a system which automatically manages all the step and is ready for restoring in the break case. Thanks for considerations.. Dario _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
Hi Dario, Yes, I have to do that and another couple of things (like disabling monit so that it won't start my gems while they are down for GC) Here is the full script. While it won't work out of the box for you, I think you can really get an idea. Dale, you may want to take something from it too (in fact, this script was a fork of yours, but I added quite a few things). I am not a bash expert, so it's far from perfect. Cheers, #!/bin/bash # Actually I only need this for the gemstone.secret check..because the rest gets from the site source source /opt/gemstone/product/seaside/defSeaside if [ -s $GEMSTONE/seaside/etc/gemstone.secret ]; then . $GEMSTONE/seaside/etc/gemstone.secret else echo 'Missing password file $GEMSTONE/seaside/etc/gemstone.secret' exit 1 fi SiteSubfolder="$1" # Requires a Sites subfolder password as a parameter if [ "a$1" = "a" ]; then echo 'Missing argument <Sites subfolder>' exit 1 fi source $MYAPP_SITES_DIRECTORY/$SiteSubfolder/gemstone/env date=`date +%Y%m%d_%H%M%S` backupfile=${GEMSTONE_DATADIR}/backups/${GEMSTONE_NAME}_backup_${date}.dbf.gz su -m $GEMSTONE_USER -c "touch $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log" su -m $GEMSTONE_USER -c "touch $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log" echo "`date` ------------------------ Starting backup of ${GEMSTONE_NAME} ------------------------" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log if [ ! -d "${GEMSTONE_DATADIR}/backups" ]; then su -m $GEMSTONE_USER -c "mkdir -p '${GEMSTONE_DATADIR}/backups'" fi echo "`date` Adding backup to object log" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 " begin run (ObjectLogEntry trace: 'BACKUP: begin ' object: '${backupfile}') addToLog. % commit logout quit " echo "`date` Stopping monit" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log monit unmonitor -g ${GEMSTONE_NAME} >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1 echo "`date` Stopping seaside gems" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log sh $APPLICATION_DIR/scripts/stopSeasideGems.sh echo "`date` Performing #markForCollection and #reclaimAll" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 " begin run [ [ MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ]. MCDefinition clearInstances. MCMethodDefinition cachedDefinitions removeKeys: (MCMethodDefinition cachedDefinitions keys). MCMethodDefinition shutDown. MethodVersionHistory uniqueInstance cleanUp. SystemRepository markForCollection. SystemRepository reclaimAll. ] on: Halt, Warning do: [:ex | ex resume] ] on: Error do: [:ex | ]. % commit logout quit " echo "`date` Start new tranlog and perform backup" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 " run | id | id := SystemRepository startNewLog. [ id < 0 ] whileTrue: [ System sleep: 1. id := SystemRepository startNewLog ]. SystemRepository fullBackupCompressedTo: '${backupfile}' % logout quit " echo "`date` Start again Seaside Gems" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log sh $APPLICATION_DIR/scripts/startSeasideGems.sh sleep 10 echo "`date` Starting monit" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log monit monitor -g ${GEMSTONE_NAME} >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1 echo "`date` Writing object log" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 " begin run (ObjectLogEntry trace: 'BACKUP: completed ' object: '${backupfile}') addToLog. % commit logout quit " if [ $? -eq 0 ] then echo "`date` Successful backup ... starting validation of backup" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log echo "------------------------------------------------------------" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log $GEMSTONE/bin/copydbf ${backupfile} /dev/null >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1 if [ $? -eq 0 ] then echo "`date` Successful validation " >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log echo "------------------------------------------------------------" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log echo "------------------------------------------------------------" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log echo "`date` Delete old backups" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 " run | autoGeneratedBackups backupsToKeep | autoGeneratedBackups := (FACompatibilityUtils current directoryFromPath: '${GEMSTONE_DATADIR}/backups/') faEntries select: [ :anEntry | ((anEntry faBasename beginsWith: '${GEMSTONE_NAME}_backup_') and: [ anEntry faBasename endsWith: '.dbf.gz' ]) and: [anEntry faBasename size = '${GEMSTONE_NAME}_backup_${date}.dbf.gz' size ] ]. (FileStream oldFileNamed: '$GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log') setToEnd; nextPutAll: ('There are ', ((FACompatibilityUtils current directoryFromPath: '${GEMSTONE_DATADIR}/backups/') faEntries) size asString, ' entries in backup folder'); crlf; nextPutAll: ('There are ', autoGeneratedBackups size asString, ' autogenerated backups'); crlf; flush. autoGeneratedBackups := autoGeneratedBackups asSortedCollection: [:entryA :entryB | entryA faBasename < entryB faBasename]. backupsToKeep := autoGeneratedBackups last: (7 min: autoGeneratedBackups size). (autoGeneratedBackups reject: [ :each | backupsToKeep includes: each ]) do: [:anEntry | (FileStream oldFileNamed: '$GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log') setToEnd; nextPutAll: ('File to be deleted: ', anEntry faBasename); crlf; flush. anEntry faParentDirectory deleteFileNamed: anEntry faBasename. ]. System performOnServer: 'ln -sf ''', backupsToKeep first fullName , ''' ''', backupsToKeep last faParentDirectory pathName, '/OldestBackup'''. % commit logout quit " echo "`date` Delete old tranlogs file and only keep the needed ones for the oldest backup `readlink ${GEMSTONE_DATADIR}/backups/OldestBackup` " >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log sh $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/delete-old-tranlogs.sh -d ${GEMSTONE_DATADIR} -f ${GEMSTONE_DATADIR}/backups/OldestBackup -g /opt/gemstone/product -r >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1 echo "`date` ---------------------- Backup finihed -----------------------" exit 0 else reason="Failed validatino" echo "`date` Failed validation" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log fi else reason="Failed backup" echo "`date` Failed backup" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log fi $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 " run 'For details about the failure, scan backward in log file to previous topaz session or copydbf sessions' % logout quit run (ObjectLogEntry fatal: 'BACKUP: failed ' object: '${reason}. See $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log for details') addToLog. % commit logout quit " echo "------------------------------------------------------------" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log exit 101 #failure On Mon, Jun 8, 2015 at 11:46 AM, Dario Trussardi via Glass <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
Thanks Mariano, I've made a note of your script[1]
Dale [1] https://github.com/GsDevKit/gsDevKitHome/issues/80 On 06/08/2015 08:00 AM, Mariano
Martinez Peck via Glass wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
In reply to this post by GLASS mailing list
On Mon, Jun 8, 2015 at 2:04 PM, Johan Brichau <[hidden email]> wrote:
Hi johan, Thanks for asking. Yes, that was one of my concerns. Good to know a stop/start would work too.
Indeed, and I was already thinking about that. At some point, I remembered having a problem in which the GC would not run because they were waiting for the "vote" of some seaside gems (at least that's what I thought). I THINK this is related to GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE = 90; thingy. Right? So..if I make sure my seaside gems do have such flag, I am safe now? I am using 3.1.0.6 and latest GLASS. Thanks Johan for asking!
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
Mariano,
Yes, the GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE = 90 setting is related and it cleans the temp obj memory, which is holding onto the objects. In fact, I never recycle the seaside fastcgi gems but I sometimes have to recycle the servicevm gem once I notice that several Gb are not being reclaimed after an MFC. It always helps. I have not figured out why the servicevm is behaving different in our case because the flag is set for all gems. I never experienced a voting issue though, except if I accidentally launched another MFC concurrently. I don’t know but maybe a backup also blocks the voting process for an MFC? I think the manual mentions the conditions of the voting though. cheers Johan
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
On Mon, Jun 8, 2015 at 2:27 PM, Johan Brichau via Glass <[hidden email]> wrote:
You mean you do not even ever stop/start seaside gems??? mmmmmm
Did you check if one of the gems is fired linked while the others don't? if so, then you may be getting the flags of the gem.exe rather than the one of the seaside gems (I had this issue in the past)
mmmm in my case I THINK that the one gem that was opened and rejecting the MFC because of voting was actually gemtools...not seaside gems. So...I guess I am stopping the gems while MFC just to be super sure it will be fine and the MFC takes little time so far. ...
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
Nope. Only when they crash ;)
I checked. Does not seem to be the case. It’s a mystery (for now :)
Ha yes. I think gemtools also causes that. But mind that leaving a gemtools open without interaction for a long time while seaside is serving requests will also grow your transaction backlog. So I never leave gemtools connected for a long time. cheers, Johan _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
Yeah, the support code surrounding GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE
has evolved over time ...
Back in 2.4.x when the original problem with seaside gems was discovered, I think that there might have been a few bugs in the implementation of GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE that lead to the initial recommendation of cycling gems (that way you get 100% GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE). Also the max value of GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE was limited to 90%, so there was quite a bit of room for some references to be kepts around. Here's the comment about GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE from the 2.4.4.1 $GEMSTONE/data/system.conf file: #========================================================================= # GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE: Percent of pom generation area # to be thrown away when voting on possible dead objects. # Only subspaces of pom generation older than 5 minutes are thrown away. # The most recently used subspace is never thrown away. # # If this value is not specified, or the specified value is out of range, # the default is used. # # Runtime equivalent: #GemPomGenPruneOnVote # Default: 50 # min: 0 max: 90 # GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE = 50; Overtime the implementation has been changed to the point where you are allowed to specify a GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE of 100% so the need to cycle gems can be completely eliminated. Here's the comment from the $GEMSTONE/data/system.conf file for 2.4.6 (3.3 is identical): #========================================================================= # GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE: Percent of pom generation area # to be cleared when voting on possible dead objects. # If value is > 0 and < 100, subspaces of pom generation older # than 5 minutes are cleared; the number of subspaces cleared is # the specified percentage of spaces in use rounded down to an # integral number of spaces. # If value == 100, all subspaces of pom generation are cleared without # regard to their age. # # If this value is not specified, or the specified value is out of range, # the default is used. # # Runtime equivalent: #GemPomGenPruneOnVote # Default: 50 # min: 0 max: 100 # GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE = 50; So for the semantics that applies to your version take a look at the $GEMSTONE/data/system.conf in your product tree ... With regards to GemTools, tODE and topaz, Johan is correct you don't want to stay logged into a production system for very long or you may will see excessive extent growth ... the stone has to keep around ALL of the meta data (Commit Record Backlog) for all of the commits between you last abort and the current transaction ... for systems with high commit rates the commit record backlog can climb pretty quickly ... Dale On 06/08/2015 11:35 AM, Johan Brichau
via Glass wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
On Mon, Jun 8, 2015 at 4:27 PM, Dale Henrichs via Glass <[hidden email]> wrote:
Hi Dale, Thanks for telling us. Now, if you were to guess, would you prefer 1) to stop/start the seaside gems and let a value of 90% or so... or 2) let 100% and do not even start/stop gems? Thanks in advance,
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
On 06/08/2015 12:41 PM, Mariano
Martinez Peck wrote:
If you're using a version of GemStone that supports the 100% option then I'd be inclined to go that route ... starting and stopping gems just requires extra machinery and can interrupt users ... Now if I happen to have a gem or two that are doing batch processing and the potential for referencing a bunch of persistent objects I might think twice and consider the cost of restarting the (presumably long rnning) batch process and refilling the cache for those gems versus the amount of repository growth I might encounter ... With the ability to flush all persistent objects you're basically stopping/restarting with respect to POM ... Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
On Mon, Jun 8, 2015 at 7:48 PM, Dale Henrichs <[hidden email]> wrote:
Ok, it makes sense your analysis. Thanks. BTW....if I set 100% and I do not stop/start seaside gems, I am still fine not restarting seaside maintenance vm AND i am also find with the backup code (besides MFC)? I mean, won't the running seaside gems affect at all to the backup? I guess not, but just want to confirm. _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
In reply to this post by GLASS mailing list
Ciao Mariano, thanks for sharing your work. I have some questions: see below > Hi Dario, > > Yes, I have to do that and another couple of things (like disabling monit so that it won't start my gems while they are down for GC) > > Here is the full script. While it won't work out of the box for you, I think you can really get an idea. > > Dale, you may want to take something from it too (in fact, this script was a fork of yours, but I added quite a few things). > > I am not a bash expert, so it's far from perfect. > > Cheers, > > > > #!/bin/bash > > # Actually I only need this for the gemstone.secret check..because the rest gets from the site source > source /opt/gemstone/product/seaside/defSeaside > > if [ -s $GEMSTONE/seaside/etc/gemstone.secret ]; then > . $GEMSTONE/seaside/etc/gemstone.secret > else > echo 'Missing password file $GEMSTONE/seaside/etc/gemstone.secret' > exit 1 > fi > > SiteSubfolder="$1" > > # Requires a Sites subfolder password as a parameter > if [ "a$1" = "a" ]; then > echo 'Missing argument <Sites subfolder>' > exit 1 > fi > > source $MYAPP_SITES_DIRECTORY/$SiteSubfolder/gemstone/env > > > date=`date +%Y%m%d_%H%M%S` > backupfile=${GEMSTONE_DATADIR}/backups/${GEMSTONE_NAME}_backup_${date}.dbf.gz > > su -m $GEMSTONE_USER -c "touch $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log" > su -m $GEMSTONE_USER -c "touch $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log" > > > echo "`date` ------------------------ Starting backup of ${GEMSTONE_NAME} ------------------------" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > > > if [ ! -d "${GEMSTONE_DATADIR}/backups" ]; then > su -m $GEMSTONE_USER -c "mkdir -p '${GEMSTONE_DATADIR}/backups'" > fi > > > > echo "`date` Adding backup to object log" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 " > begin > run > (ObjectLogEntry trace: 'BACKUP: begin ' object: '${backupfile}') addToLog. > % > commit > logout > quit > " > > > > echo "`date` Stopping monit" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > monit unmonitor -g ${GEMSTONE_NAME} >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1 > > echo "`date` Stopping seaside gems" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > sh $APPLICATION_DIR/scripts/stopSeasideGems.sh > > echo "`date` Performing #markForCollection and #reclaimAll" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 " > begin > run > [ > [ I don't know the next commands : > MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ]. > MCDefinition clearInstances. > MCMethodDefinition cachedDefinitions removeKeys: > (MCMethodDefinition cachedDefinitions keys). > MCMethodDefinition shutDown. > MethodVersionHistory uniqueInstance cleanUp. But you run markForCollection every time ? I think to run markForCollection only at the end of a cycle ( week - month ) when i remove link of hold data > SystemRepository markForCollection. > SystemRepository reclaimAll. > > ] on: Halt, Warning do: [:ex | ex resume] > ] on: Error do: [:ex | ]. > % > commit > logout > quit > " > > > > echo "`date` Start new tranlog and perform backup" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 " > run > | id | > id := SystemRepository startNewLog. > [ id < 0 ] whileTrue: [ > System sleep: 1. > id := SystemRepository startNewLog ]. > SystemRepository fullBackupCompressedTo: '${backupfile}' > % > logout > quit > " > > echo "`date` Start again Seaside Gems" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > sh $APPLICATION_DIR/scripts/startSeasideGems.sh > sleep 10 > > echo "`date` Starting monit" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > monit monitor -g ${GEMSTONE_NAME} >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1 > > > > > echo "`date` Writing object log" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 " > begin > run > (ObjectLogEntry trace: 'BACKUP: completed ' object: '${backupfile}') addToLog. > % > commit > > logout > quit > " > > > if [ $? -eq 0 ] > then > echo "`date` Successful backup ... starting validation of backup" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > echo "------------------------------------------------------------" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > $GEMSTONE/bin/copydbf ${backupfile} /dev/null >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1 > if [ $? -eq 0 ] > then > echo "`date` Successful validation " >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > echo "------------------------------------------------------------" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > echo "------------------------------------------------------------" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > > > echo "`date` Delete old backups" >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/runTopazStringScript.sh $1 " > > run > | autoGeneratedBackups backupsToKeep | > autoGeneratedBackups := (FACompatibilityUtils current directoryFromPath: '${GEMSTONE_DATADIR}/backups/') faEntries > select: [ :anEntry | ((anEntry faBasename beginsWith: '${GEMSTONE_NAME}_backup_') > and: [ anEntry faBasename endsWith: '.dbf.gz' ]) > and: [anEntry faBasename size = '${GEMSTONE_NAME}_backup_${date}.dbf.gz' size ] > ]. > (FileStream oldFileNamed: '$GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log') > setToEnd; > nextPutAll: ('There are ', ((FACompatibilityUtils current directoryFromPath: '${GEMSTONE_DATADIR}/backups/') faEntries) size asString, ' entries in backup folder'); crlf; > nextPutAll: ('There are ', autoGeneratedBackups size asString, ' autogenerated backups'); crlf; > flush. > autoGeneratedBackups := autoGeneratedBackups asSortedCollection: [:entryA :entryB | entryA faBasename < entryB faBasename]. > backupsToKeep := autoGeneratedBackups last: (7 min: autoGeneratedBackups size). > (autoGeneratedBackups reject: [ :each | backupsToKeep includes: each ]) do: [:anEntry | > (FileStream oldFileNamed: '$GEMSTONE_LOGDIR/${GEMSTONE_NAME}_old_backups.log') > setToEnd; nextPutAll: ('File to be deleted: ', anEntry faBasename); crlf; flush. > anEntry faParentDirectory deleteFileNamed: anEntry faBasename. > ]. > System performOnServer: 'ln -sf ''', backupsToKeep first fullName , ''' ''', backupsToKeep last faParentDirectory pathName, '/OldestBackup'''. > % > commit > logout > quit > " > I don't understund what the system do in this steps. > > echo "`date` Delete old tranlogs file and only keep the needed ones for the oldest backup `readlink > ${GEMSTONE_DATADIR}/backups/OldestBackup` " >> $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log > sh $MYAPP_GEMSTONE_ADDITIONS_SCRIPTS/delete-old-tranlogs.sh -d ${GEMSTONE_DATADIR} -f Can you sharing the delete-old-tranlogs.sh code? > ${GEMSTONE_DATADIR}/backups/OldestBackup -g /opt/gemstone/product -r >> > $GEMSTONE_LOGDIR/${GEMSTONE_NAME}_backup.log 2>&1 > > echo "`date` ---------------------- Backup finihed -----------------------" > Thanks for any considerations. Dario _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
On Tue, Jun 9, 2015 at 12:10 PM, Dario Trussardi via Glass <[hidden email]> wrote:
Hi Dario, My answers below.
Yes, this is up to you. YOU decide when to run it. In my case, I run it every night as part of the cleaning and backup script. Currently, my repositories are not that big so MFC does not take very long and I do not have that many users at the time the cron runs (3am or so). So I prefer to run it everyday and keep my system clean and small. If you do not want to run MFC before the backup, then do not do it. You may want to backup everyday yet run MFC once a week or month. BTW...be careful with " I think to run markForCollection ". If you are using the glass scripts "runSeasideGems30" that will be running a maintenance vm which will be running MFC every hour. To disable that, what I am doing is as part of my load code I do: (Smalltalk at: #WAGemStoneMaintenanceTask) removeTaskNamed: 'Mark For Collect'. Note that you must do that as part of your post-load code or similar because if you happen to load a new version of seaside packages at some point, it might fire again the class side #initialize and hence rebuild the tasks.
It's just because my bash skills are a bit limited and I prefer to script in smalltalk :) Basically, what it does, is to take the directory of backups ('${GEMSTONE_DATADIR}/backups/'), take all those files which are on the pattern ${GEMSTONE_NAME}_backup_YYYYMMDD_HHMMSS.dbf.gz, then sorts them, then keep only the newest 7 and delete the rest.
This is part of Norbert Hartl code but I do not remember if I modified it too: https://github.com/noha/stone-creator/blob/master/bin/delete-old-tranlogs.sh My version (maybe I did not changed it) is: #!/bin/bash while getopts ":d:g:f:r" opt; do case $opt in d) DATA_DIR=$OPTARG ;; f) FILE=$OPTARG ;; g) export GEMSTONE=$OPTARG ;; r) REMOVE_TRANLOGS=1 ;; \?) echo "Invalid option: -$OPTARG" >&2 ;; esac done function help { echo "usage: $0 -d [directory] -g [gemstonedir] -r" echo " -d [directory] data directory of stone (containing extent0.dbf) -f [file] file to take information about needed tranlog from (extent or backup) -g [gemstonedir] directory of gemstone installation (default: /opt/gemstone/product) -r really remove tranlogs. Without this switch they are only shown " exit } if [ "$DATA_DIR" == "" ]; then help; fi if [ ! -f "$FILE" ]; then echo "did not find $FILE" exit fi if [ "$GEMSTONE" == "" ]; then COPYDBF_EXE=`which copydbf` else COPYDBF_EXE="$GEMSTONE/bin/copydbf" fi if [ ! -x "$COPYDBF_EXE" ]; then echo "cannot find copydbf binary. please use -e argument or adjust your path variable" exit fi LAST_NEEDED_TRANLOG=`$COPYDBF_EXE -i $FILE 2>&1 | grep tranlog | cut -d ' ' -f 13` if [ "$LAST_NEEDED_TRANLOG" == "" ]; then echo "could not find information about last needed tranlog" exit fi if [[ ! "$LAST_NEEDED_TRANLOG" =~ tranlog[0-9][0-9]*.dbf$ ]]; then echo "parse error in output from copydby. Result is not a tranlog filename" exit fi SORTED_LIST_OF_FILES=`ls -1 $DATA_DIR/tranlog* | xargs -n1 basename | sort -k2 -tg -n` LINES_IN_LIST=`echo "$SORTED_LIST_OF_FILES" | wc -l` if [ "$LINES_IN_LIST" -eq 1 ]; then echo "only one tranlog file found. nothing to delete" exit fi LINE_OF_LAST_NEEDED_TRANLOG=`echo "$SORTED_LIST_OF_FILES" | grep -n $LAST_NEEDED_TRANLOG | cut -d ':' -f 1` if [ "$LINE_OF_LAST_NEEDED_TRANLOG" == "" ]; then echo "could not find last needed tranlog in list of available tranlogs." exit fi if [ "$LINE_OF_LAST_NEEDED_TRANLOG" -lt 2 ]; then echo "nothing to do" exit fi LINE_OF_LAST_OBSOLETE_FILE=$(($LINE_OF_LAST_NEEDED_TRANLOG -1 )) for obsolete in `echo "$SORTED_LIST_OF_FILES" | head -n $LINE_OF_LAST_OBSOLETE_FILE`; do if [ ! -z "$REMOVE_TRANLOGS" ]; then echo "removing $obsolete" rm "$DATA_DIR/$obsolete" else echo "file $obsolete is obsolete" fi done
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
In reply to this post by GLASS mailing list
On 06/09/2015 07:13 AM, Mariano Martinez Peck wrote: > > > Ok, it makes sense your analysis. Thanks. > BTW....if I set 100% and I do not stop/start seaside gems, I am still > fine not restarting seaside maintenance vm AND i am also find with the > backup code (besides MFC)? I mean, won't the running seaside gems > affect at all to the backup? I guess not, but just want to confirm. > > Keep in mind that even with 100% setting, references from temp objects to persistent objects will still count as references to the object and will vote down the gc ... clearing POM means that the cached values (in POM) won't count as references... When the backup runs, it backs up the state of the system from a given checkpoint (point in time), any changes by running gems that may have been made while the backups were being written are saved in the tranlogs so it is definitely safe to run backups while other gems are running. Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
On Tue, Jun 9, 2015 at 1:50 PM, Dale Henrichs <[hidden email]> wrote:
Thanks Dale. BTW... I have just changed the MFC part of the script I pasted at the beginning of the thread. I used to do myself the #markForCollection as you can see. However, I think the code implemented by WAGemStoneMaintenanceTask class >> #maintenanceTaskMarkForCollect is much better, because it prints some results into the ObjectLog. So instead of doing "SystemRepository markForCollection." I am now doing: "WAGemStoneMaintenanceTask maintenanceTaskMarkForCollect performTask: 0.". Cheers, _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
Ciao Mariano,
How i can understand if Mark For Collect is running automatically every hour? Thanks, Dario
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
On Fri, Jul 24, 2015 at 10:51 AM, Mariano Martinez Peck <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
On 07/24/2015 06:52 AM, Mariano
Martinez Peck via Glass wrote:
This is a good list from Mariano. I would add, that you should inspect the result of: WAGemStoneMaintenanceTask tasks and if it looks like the following: . -> anOrderedCollection( Mark For Collect, Seaside Session Expiration) (class)@ -> OrderedCollection (oop)@ -> 257463553 (committed)@ -> true (size)@ -> 2 1@ -> Mark For Collect 2@ -> Seaside Session Expiration (Specifically including the `Mark for Collect` task) AND you are running the maintenance VM, then you are running an MFC ... Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
In reply to this post by GLASS mailing list
Ciao,
I works with gsDevKit 3.1.0.6 repository. It support the GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE set to 100 % ? If i right understand when i have gem without batch processing i don't have problem with the GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE parameter. When i have gem with batch processing ( other some minutes ) i found the problem about the GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE parameter. But I understand that the wisest solution would be to wait for batch processing to finish, and only after go - make starting the garbage collections mechanism. It's right ? If yes, how i can manage it ? Another questions: i have a ubuntu server with 8GB of memory and the SHR_PAGE_CACHE_SIZE_KB set to 2097152. Now the system is load with some date and i think it used all the SHR_PAGE_CACHE. But when do the login to the system the shell report: memory usage: 9% The memory usage don't consider the kernel.shm* parameter? The ipcs -lm report:
------ Limiti della memoria condivisa --------
numero massimo di segmenti = 4096
dimensione max seg (kbyte) = 6291456
max total shared memory (kbytes) = 6291456
dimensione min seg (byte) = 1 Thanks for any considerations, Dario
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
On 10/20/2015 09:37 AM, Trussardi Dario
Romano via Glass wrote:
Ciao,I don't know off hand ... take a look in $GEMSTONE/data/system.conf ... there are comments for every conf setting possible that fully describe the ranges, default values, etc. Every active Gem could end up voting downa dead object because it has kept a reference in it's head ... I mention the long running batch process because it could have a "valid" reference to the "dead object" obtained in a transaction before the object became "dead" ... Well I think the wise answer is to not worry too much about when a dead object actually disappears from your repository ... if you have gems that will be running for weeks at a time, then set the GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE to the max value allowed for the GemStone version that you are running and then let "nature take it's course" it may take ore time than you think for the object to go away but it will go away eventually. If you are under repository size pressure then the guaranteed method to avoid voting down dead objects is to restart ALL of the gems in the system before running the MFC ... but I think this is an extreme solution that should only be used when you are under critical repository size pressures ... Why do you think that the full SHR_PAGE_CACHE is used? Are you looking at statmonitor output with vsd? I think it depends upon what command you get to see the memory usage and on linux, I don't think there is a good way to get memory usage information when shared memory is involved, some bits of memory can get paged out and so on .... I would say that you should run statmonitor and use vsd to look at the results ... once you've started the statmonitor, use vsd (an X application) to look at the stat file and I recommend the "Template > New Template Chart > CacheMix" to start with ... there are 100's of stats, and I just don't have the bandwidth to coach folks on the interpretation of the results, but you can check "Show Statistics Info" which will give you documentation to read about each stat when selected in the chart ... Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
«
Return to GLASS
|
1 view|%1 views
| Free forum by Nabble | Edit this page |