Translog space problem
Locked
 
|
Hi, we had a problem on our production server which
caused a system crash. We went to the logs and we found that there was no space
to open a new transaction log.
The solution was to commit all the transaction logs
to the extent using topaz, then we moved all the transaction logs to
another directory to make some space and we restarted gemstone.
Now it seems to be stable but we don´t know if it
was a good solution, maybe we have to change gemstone configuration to
automatically manage these situations.
These are some lines from
seaside.log:
AioWait thread tranlog write ERROR op assuming
ECANCELED after aio_cancel , recordId 33.1021823 ioCount 124517 errno 125,
Operation canceled
AioWait thread tranlog write ERROR op assuming
ECANCELED after aio_cancel , recordId 33.1021996 ioCount 124518 errno 125,
Operation canceled
AioWait thread tranlog write ERROR op assuming
ECANCELED after aio_cancel , recordId 33.1022037 ioCount 124519 errno 125,
Operation canceled
AioWait thread tranlog write ERROR op assuming
ECANCELED after aio_cancel , recordId 33.1022121 ioCount 124520 errno 125,
Operation canceled
AioWait thread tranlog write ERROR op assuming
ECANCELED after aio_cancel , recordId 33.1022232 ioCount 124521 errno 125,
Operation canceled
AioWait thread tranlog write ERROR op assuming
ECANCELED after aio_cancel , recordId 33.1022233 ioCount 124522 errno 125,
Operation canceled
AioWait thread tranlog write ERROR op assuming
ECANCELED after aio_cancel , recordId 33.1022252 ioCount 124523 errno 125,
Operation canceled
AioWait thread tranlog write ERROR op assuming
ECANCELED after aio_cancel , recordId 33.1022253 ioCount 124524 errno 125,
Operation canceled
AioWait thread tranlog write ERROR op assuming
ECANCELED after aio_cancel , recordId 33.1022427 ioCount 124525 errno 125,
Operation canceled
AioWait thread tranlog write ERROR op incomplete
write , recordId 33.1463179 ioCount 3520 errno 28, No space left on
device
--- 09/01/10 08:26:12 UTC ---
Log flush error: File = /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf DBF Op: Write; DBF Record: 0; Error: write() failure; System Codes: errno=28,ENOSPC, There is no space left on the device (or, in fcntl(), there are no more record locks) Could not initialize new
transaction log file. There is probably no
space on the disk. filename = /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf GemStone is unable to initialize new transaction log file fileName = /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf error = LogOpenLog: GemStone is unable to find a log file, fileId 34 . reason = No valid logs were found. Could not initialize new
transaction log file. There is probably no
space on the disk. filename = /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf GemStone is unable to initialize new transaction log file fileName = /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf error = LogOpenLog: GemStone is unable to find a log file, fileId 34 . reason = No valid logs were found. All available tranlog space is
full. Either add a new tranlog
directory or archive and delete existing tranlogs to make more room. Sessions may be suspended until tranlog space becomes available. Log flush error:
File = /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf DBF Op: Write; DBF Record: 0; Error: write() failure; System Codes: errno=28,ENOSPC, There is no space left on the device (or, in fcntl(), there are no more record locks) Thanks for your help
Sebastian Van Lacke
|
|
|
Sebastian,
Just keeping the last 3-4 tranlog files will give you enough to recover from a crash. This is what we do where I work, and to my knowledge theres never been any issues in almost 10yrs of operation.
When the tranlog space fills up the stone stalls or pauses - Then I just remove all but the last 2-3 files and everything continues on its way.
If I remember correctly, once the stone process performs a checkpoint the tranlogs are no longer needed.
- Somebody please correct me if im wrong here...
Jon.
On Wed, Sep 1, 2010 at 8:12 AM, Sebastian Van Lacke <[hidden email]> wrote:
|
|
|
In reply to this post by Sebastia Van Lacke
Sebastian,
There are two considerations for keeping tranlogs around: 1. crash recovery 2. restore from backup For crash recovery ... where the stone process crashes for some reason. You will need the set of tranlogs that were created since the last checkpoint. You can evaluate the following to find out the oldest tranlog that is needed to recover from the last checkpoint: SystemRepository oldestLogFileIdForRecovery. So for crash recovery it makes sense to keep the oldest tranlog needed plus all of the newer tranlogs. You could run a cron job once a week or month that moved all tranlogs that are _older_ than the oldest tranlog needed to be in a good position to recover from a stone crash. For restore from backup... where the extent file is corrupted on disk, you will need to restore from a backup into a fresh extent. In this case you need all of the tranlogs from the point of the backup to the present. So when you archive move your tranlogs from the primary disk, you will want to keep the tranlogs around with your backup file. When you run a backup, I a checkpoint is performed, so you'll need the tranlog that was active when you started the backup. Depending upon the amount of disk space needed for the tranlogs during a restoreFromCurrentLogs you may want to look at the family of *ArchiveLog* methods in the Repository class for additional recovery options. Finally, given that the checkpoint interval is probably 15 minutes or so and that your tranlogs are not growing at incredible rates, you could simplify the archival scripts and choose to archive tranlogs that are a week old. The backup and restore code is smart enough to ignore old tranlogs. Dale Sebastian Van Lacke wrote: > Hi, we had a problem on our production server which caused a system > crash. We went to the logs and we found that there was no space to open > a new transaction log. > The solution was to commit all the transaction logs to the extent using > topaz, then we moved all the transaction logs to another directory to > make some space and we restarted gemstone. > Now it seems to be stable but we don´t know if it was a good solution, > maybe we have to change gemstone configuration to automatically manage > these situations. > These are some lines from seaside.log: > > > > AioWait thread tranlog write ERROR op assuming ECANCELED after > aio_cancel , recordId 33.1021823 ioCount 124517 errno 125, Operation > canceled > > AioWait thread tranlog write ERROR op assuming ECANCELED after > aio_cancel , recordId 33.1021996 ioCount 124518 errno 125, Operation > canceled > > AioWait thread tranlog write ERROR op assuming ECANCELED after > aio_cancel , recordId 33.1022037 ioCount 124519 errno 125, Operation > canceled > > AioWait thread tranlog write ERROR op assuming ECANCELED after > aio_cancel , recordId 33.1022121 ioCount 124520 errno 125, Operation > canceled > > AioWait thread tranlog write ERROR op assuming ECANCELED after > aio_cancel , recordId 33.1022232 ioCount 124521 errno 125, Operation > canceled > > AioWait thread tranlog write ERROR op assuming ECANCELED after > aio_cancel , recordId 33.1022233 ioCount 124522 errno 125, Operation > canceled > > AioWait thread tranlog write ERROR op assuming ECANCELED after > aio_cancel , recordId 33.1022252 ioCount 124523 errno 125, Operation > canceled > > AioWait thread tranlog write ERROR op assuming ECANCELED after > aio_cancel , recordId 33.1022253 ioCount 124524 errno 125, Operation > canceled > > AioWait thread tranlog write ERROR op assuming ECANCELED after > aio_cancel , recordId 33.1022427 ioCount 124525 errno 125, Operation > canceled > > > > AioWait thread tranlog write ERROR op incomplete write , recordId > 33.1463179 ioCount 3520 errno 28, No space left on device > > --- 09/01/10 08:26:12 UTC --- > Log flush error: > File = > /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf > DBF Op: Write; DBF Record: 0; > Error: write() failure; System Codes: errno=28,ENOSPC, There is no space > left on the device (or, in fcntl(), there are no more record locks) > > Could not initialize new transaction log file. There is probably no > space on the disk. > filename = > /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf > GemStone is unable to initialize new transaction log file > fileName = > /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf > error = LogOpenLog: GemStone is unable to find a log file, > fileId 34 . > reason = No valid logs were found. > > Could not initialize new transaction log file. There is probably no > space on the disk. > filename = > /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf > GemStone is unable to initialize new transaction log file > fileName = > /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf > error = LogOpenLog: GemStone is unable to find a log file, > fileId 34 . > reason = No valid logs were found. > > All available tranlog space is full. Either add a new tranlog > directory or archive and delete existing tranlogs to make more room. > Sessions may be suspended until tranlog space becomes available. > > Log flush error: > File = > /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf > DBF Op: Write; DBF Record: 0; > Error: write() failure; System Codes: errno=28,ENOSPC, There is no space > left on the device (or, in fcntl(), there are no more record locks) > > Thanks for your help > > Sebastian Van Lacke |
Re: Translog space problem
|
|
ive been meaning to make a script that does translog cleanup
ill post it up when i have it done On 10-09-01 10:23 AM, Dale Henrichs wrote: > Sebastian, > > There are two considerations for keeping tranlogs around: > > 1. crash recovery > 2. restore from backup > > For crash recovery ... where the stone process crashes for some > reason. You will need the set of tranlogs that were created since the > last checkpoint. You can evaluate the following to find out the oldest > tranlog that is needed to recover from the last checkpoint: > > SystemRepository oldestLogFileIdForRecovery. > > So for crash recovery it makes sense to keep the oldest tranlog needed > plus all of the newer tranlogs. You could run a cron job once a week > or month that moved all tranlogs that are _older_ than the oldest > tranlog needed to be in a good position to recover from a stone crash. > > For restore from backup... where the extent file is corrupted on disk, > you will need to restore from a backup into a fresh extent. In this > case you need all of the tranlogs from the point of the backup to the > present. So when you archive move your tranlogs from the primary disk, > you will want to keep the tranlogs around with your backup file. When > you run a backup, I a checkpoint is performed, so you'll need the > tranlog that was active when you started the backup. > > Depending upon the amount of disk space needed for the tranlogs during > a restoreFromCurrentLogs you may want to look at the family of > *ArchiveLog* methods in the Repository class for additional recovery > options. > > Finally, given that the checkpoint interval is probably 15 minutes or > so and that your tranlogs are not growing at incredible rates, you > could simplify the archival scripts and choose to archive tranlogs > that are a week old. The backup and restore code is smart enough to > ignore old tranlogs. > > Dale > > Sebastian Van Lacke wrote: >> Hi, we had a problem on our production server which caused a system >> crash. We went to the logs and we found that there was no space to >> open a new transaction log. >> The solution was to commit all the transaction logs to the extent >> using topaz, then we moved all the transaction logs to another >> directory to make some space and we restarted gemstone. >> Now it seems to be stable but we don´t know if it was a good >> solution, maybe we have to change gemstone configuration to >> automatically manage these situations. >> These are some lines from seaside.log: >> >> >> >> AioWait thread tranlog write ERROR op assuming ECANCELED after >> aio_cancel , recordId 33.1021823 ioCount 124517 errno 125, Operation >> canceled >> >> AioWait thread tranlog write ERROR op assuming ECANCELED after >> aio_cancel , recordId 33.1021996 ioCount 124518 errno 125, Operation >> canceled >> >> AioWait thread tranlog write ERROR op assuming ECANCELED after >> aio_cancel , recordId 33.1022037 ioCount 124519 errno 125, Operation >> canceled >> >> AioWait thread tranlog write ERROR op assuming ECANCELED after >> aio_cancel , recordId 33.1022121 ioCount 124520 errno 125, Operation >> canceled >> >> AioWait thread tranlog write ERROR op assuming ECANCELED after >> aio_cancel , recordId 33.1022232 ioCount 124521 errno 125, Operation >> canceled >> >> AioWait thread tranlog write ERROR op assuming ECANCELED after >> aio_cancel , recordId 33.1022233 ioCount 124522 errno 125, Operation >> canceled >> >> AioWait thread tranlog write ERROR op assuming ECANCELED after >> aio_cancel , recordId 33.1022252 ioCount 124523 errno 125, Operation >> canceled >> >> AioWait thread tranlog write ERROR op assuming ECANCELED after >> aio_cancel , recordId 33.1022253 ioCount 124524 errno 125, Operation >> canceled >> >> AioWait thread tranlog write ERROR op assuming ECANCELED after >> aio_cancel , recordId 33.1022427 ioCount 124525 errno 125, Operation >> canceled >> >> >> >> AioWait thread tranlog write ERROR op incomplete write , recordId >> 33.1463179 ioCount 3520 errno 28, No space left on device >> >> --- 09/01/10 08:26:12 UTC --- >> Log flush error: >> File = >> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >> DBF Op: Write; DBF Record: 0; >> Error: write() failure; System Codes: errno=28,ENOSPC, There is no >> space left on the device (or, in fcntl(), there are no more record >> locks) >> >> Could not initialize new transaction log file. There is probably no >> space on the disk. >> filename = >> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >> GemStone is unable to initialize new transaction log file >> fileName = >> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >> error = LogOpenLog: GemStone is unable to find a log >> file, fileId 34 . >> reason = No valid logs were found. >> >> Could not initialize new transaction log file. There is probably no >> space on the disk. >> filename = >> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >> GemStone is unable to initialize new transaction log file >> fileName = >> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >> error = LogOpenLog: GemStone is unable to find a log >> file, fileId 34 . >> reason = No valid logs were found. >> >> All available tranlog space is full. Either add a new tranlog >> directory or archive and delete existing tranlogs to make more room. >> Sessions may be suspended until tranlog space becomes available. >> >> Log flush error: >> File = >> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >> DBF Op: Write; DBF Record: 0; >> Error: write() failure; System Codes: errno=28,ENOSPC, There is no >> space left on the device (or, in fcntl(), there are no more record >> locks) >> >> Thanks for your help >> >> Sebastian Van Lacke > |
Re: Translog space problem
|
|
In reply to this post by Sebastia Van Lacke
Thanks for your help!
Clayton we will appreciate that script to automatize the translog clenaning. Dale, I ommited a detail. There was 4Gb of free space on disk, nevertheless Gemstone couldn´t create a new translog file. Do you know why this happens? Sebastian |
|
|
In reply to this post by Clayton Cottingham-3
Can you github them so people can easily work together on improving
them over time? On Wed, Sep 1, 2010 at 2:28 PM, Clayton Cottingham <[hidden email]> wrote: > ive been meaning to make a script that does translog cleanup > > ill post it up when i have it done > > On 10-09-01 10:23 AM, Dale Henrichs wrote: >> Sebastian, >> >> There are two considerations for keeping tranlogs around: >> >> 1. crash recovery >> 2. restore from backup >> >> For crash recovery ... where the stone process crashes for some >> reason. You will need the set of tranlogs that were created since the >> last checkpoint. You can evaluate the following to find out the oldest >> tranlog that is needed to recover from the last checkpoint: >> >> SystemRepository oldestLogFileIdForRecovery. >> >> So for crash recovery it makes sense to keep the oldest tranlog needed >> plus all of the newer tranlogs. You could run a cron job once a week >> or month that moved all tranlogs that are _older_ than the oldest >> tranlog needed to be in a good position to recover from a stone crash. >> >> For restore from backup... where the extent file is corrupted on disk, >> you will need to restore from a backup into a fresh extent. In this >> case you need all of the tranlogs from the point of the backup to the >> present. So when you archive move your tranlogs from the primary disk, >> you will want to keep the tranlogs around with your backup file. When >> you run a backup, I a checkpoint is performed, so you'll need the >> tranlog that was active when you started the backup. >> >> Depending upon the amount of disk space needed for the tranlogs during >> a restoreFromCurrentLogs you may want to look at the family of >> *ArchiveLog* methods in the Repository class for additional recovery >> options. >> >> Finally, given that the checkpoint interval is probably 15 minutes or >> so and that your tranlogs are not growing at incredible rates, you >> could simplify the archival scripts and choose to archive tranlogs >> that are a week old. The backup and restore code is smart enough to >> ignore old tranlogs. >> >> Dale >> >> Sebastian Van Lacke wrote: >>> Hi, we had a problem on our production server which caused a system >>> crash. We went to the logs and we found that there was no space to >>> open a new transaction log. >>> The solution was to commit all the transaction logs to the extent >>> using topaz, then we moved all the transaction logs to another >>> directory to make some space and we restarted gemstone. >>> Now it seems to be stable but we don´t know if it was a good >>> solution, maybe we have to change gemstone configuration to >>> automatically manage these situations. >>> These are some lines from seaside.log: >>> >>> >>> >>> AioWait thread tranlog write ERROR op assuming ECANCELED after >>> aio_cancel , recordId 33.1021823 ioCount 124517 errno 125, Operation >>> canceled >>> >>> AioWait thread tranlog write ERROR op assuming ECANCELED after >>> aio_cancel , recordId 33.1021996 ioCount 124518 errno 125, Operation >>> canceled >>> >>> AioWait thread tranlog write ERROR op assuming ECANCELED after >>> aio_cancel , recordId 33.1022037 ioCount 124519 errno 125, Operation >>> canceled >>> >>> AioWait thread tranlog write ERROR op assuming ECANCELED after >>> aio_cancel , recordId 33.1022121 ioCount 124520 errno 125, Operation >>> canceled >>> >>> AioWait thread tranlog write ERROR op assuming ECANCELED after >>> aio_cancel , recordId 33.1022232 ioCount 124521 errno 125, Operation >>> canceled >>> >>> AioWait thread tranlog write ERROR op assuming ECANCELED after >>> aio_cancel , recordId 33.1022233 ioCount 124522 errno 125, Operation >>> canceled >>> >>> AioWait thread tranlog write ERROR op assuming ECANCELED after >>> aio_cancel , recordId 33.1022252 ioCount 124523 errno 125, Operation >>> canceled >>> >>> AioWait thread tranlog write ERROR op assuming ECANCELED after >>> aio_cancel , recordId 33.1022253 ioCount 124524 errno 125, Operation >>> canceled >>> >>> AioWait thread tranlog write ERROR op assuming ECANCELED after >>> aio_cancel , recordId 33.1022427 ioCount 124525 errno 125, Operation >>> canceled >>> >>> >>> >>> AioWait thread tranlog write ERROR op incomplete write , recordId >>> 33.1463179 ioCount 3520 errno 28, No space left on device >>> >>> --- 09/01/10 08:26:12 UTC --- >>> Log flush error: >>> File = >>> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >>> DBF Op: Write; DBF Record: 0; >>> Error: write() failure; System Codes: errno=28,ENOSPC, There is no >>> space left on the device (or, in fcntl(), there are no more record >>> locks) >>> >>> Could not initialize new transaction log file. There is probably no >>> space on the disk. >>> filename = >>> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >>> GemStone is unable to initialize new transaction log file >>> fileName = >>> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >>> error = LogOpenLog: GemStone is unable to find a log >>> file, fileId 34 . >>> reason = No valid logs were found. >>> >>> Could not initialize new transaction log file. There is probably no >>> space on the disk. >>> filename = >>> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >>> GemStone is unable to initialize new transaction log file >>> fileName = >>> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >>> error = LogOpenLog: GemStone is unable to find a log >>> file, fileId 34 . >>> reason = No valid logs were found. >>> >>> All available tranlog space is full. Either add a new tranlog >>> directory or archive and delete existing tranlogs to make more room. >>> Sessions may be suspended until tranlog space becomes available. >>> >>> Log flush error: >>> File = >>> /opt/gemstone/GemStone64Bit2.3.1-x86_64.Linux/seaside/data/tranlog34.dbf >>> DBF Op: Write; DBF Record: 0; >>> Error: write() failure; System Codes: errno=28,ENOSPC, There is no >>> space left on the device (or, in fcntl(), there are no more record >>> locks) >>> >>> Thanks for your help >>> >>> Sebastian Van Lacke >> > |
|
|
In reply to this post by Sebastia Van Lacke
Sebastian Van Lacke wrote:
> Thanks for your help! > > Clayton we will appreciate that script to automatize the translog clenaning. > > Dale, I ommited a detail. There was 4Gb of free space on disk, nevertheless > Gemstone couldn´t create a new translog file. Do you know why this happens? > > Sebastian > I'm not exactly sure, but reviewing the error messages in the stone log it looks like the _likely_ cause is not enough disk space, but there may have been some other disk related resource that was saturated ... were there a lot of log files? Dale |
«
Return to GLASS
|
1 view|%1 views
| Free forum by Nabble | Edit this page |