Restore 3.1.0.6 backup

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Restore 3.1.0.6 backup

GLASS mailing list
Ciao,

i have a deployment system based on stone 3.1.0.6 and gsDevKitHome.

The system was installed in early 2015.

Now i do some test to restored the database backup ( from this deployment system ) on another system  (  called emergency PC )

This emergency PC is based on GsDevKit_home git ebc8e7b commit.

On it i create the the stone with the command:  createStone base_3106  3.1.0.6

A this point all works fine from tode command.   ( ws  for example )

Now from topaz session i successfully gave the commands:

1) SystemRepository restoreFromBackup: '/opt/GsDkB/GsDevKit_home/server/stones/base_3106/backups/backup_20181023.dbf.gz'

2) SystemRepository commitRestore

But after when i login from tode and i do the: ws   command 
the system sometimes answer with the error :

a InternalError occurred ( error 2261 ) , The object with object ID aTDSessionDescription is corrupt.  

Reason 'store past end'

in this case after 40 seconds the tode shell report:

GciSessionNotLoggedInError: Session no longer logged in.
Check the gem log for error messages (look at the most recent gem log file: `ls $GS_HOME/server/stones/base_3106/logs/gemnetobject*.log`.
If your stone is running -and healthy (`cat $GS_HOME/server/stones/base_3106/logs/base_3106.log`), you can try again.

sometimes answer the error:

a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'
In this case :
after four Abort    button click  the system open the workspace 

and the tode shell  report : AUTOCOMMIT DISABLEDAbortered ......


and the  tode 1 >     is red

Some considerations ?

Thanks,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Restore 3.1.0.6 backup

GLASS mailing list



On 10/24/2018 09:11 AM, Trussardi Dario Romano via Glass wrote:
Ciao,

i have a deployment system based on stone 3.1.0.6 and gsDevKitHome.

The system was installed in early 2015.

Now i do some test to restored the database backup ( from this deployment system ) on another system  (  called emergency PC )

This emergency PC is based on GsDevKit_home git ebc8e7b commit.

On it i create the the stone with the command:  createStone base_3106  3.1.0.6

A this point all works fine from tode command.   ( ws  for example )

Now from topaz session i successfully gave the commands:

1) SystemRepository restoreFromBackup: '/opt/GsDkB/GsDevKit_home/server/stones/base_3106/backups/backup_20181023.dbf.gz'

2) SystemRepository commitRestore

But after when i login from tode and i do the: ws   command 
the system sometimes answer with the error :

a InternalError occurred ( error 2261 ) , The object with object ID aTDSessionDescription is corrupt.  

Reason 'store past end'

in this case after 40 seconds the tode shell report:

GciSessionNotLoggedInError: Session no longer logged in.
Check the gem log for error messages (look at the most recent gem log file: `ls $GS_HOME/server/stones/base_3106/logs/gemnetobject*.log`.
If your stone is running -and healthy (`cat $GS_HOME/server/stones/base_3106/logs/base_3106.log`), you can try again.

sometimes answer the error:

a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'
In this case :
after four Abort    button click  the system open the workspace 

and the tode shell  report : AUTOCOMMIT DISABLEDAbortered ......


and the  tode 1 >     is red

Some considerations ?
Dario,

Am I right in thinking that you are doing the restore from backup into a stone on another system? If so you are likely missing some of the disk structure that tODE expects to find. but first I would like to see a full stack from the error that you are getting, because I would like to understand the context in which the "corrupt" object errors are occurring I will spend some time today and try to reproduce the problem locally so that I can characterize the problem more thoroughly ...

Dale


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Restore 3.1.0.6 backup

GLASS mailing list

Dario,

In the tODE shell try running the following:

script --script=rebuildSys
If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to $GS_HOME/shared/repos then the missing repo directory errors might go away ---

If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...

Dale

On 10/30/2018 09:50 AM, Dale Henrichs wrote:



On 10/24/2018 09:11 AM, Trussardi Dario Romano via Glass wrote:
Ciao,

i have a deployment system based on stone 3.1.0.6 and gsDevKitHome.

The system was installed in early 2015.

Now i do some test to restored the database backup ( from this deployment system ) on another system  (  called emergency PC )

This emergency PC is based on GsDevKit_home git ebc8e7b commit.

On it i create the the stone with the command:  createStone base_3106  3.1.0.6

A this point all works fine from tode command.   ( ws  for example )

Now from topaz session i successfully gave the commands:

1) SystemRepository restoreFromBackup: '/opt/GsDkB/GsDevKit_home/server/stones/base_3106/backups/backup_20181023.dbf.gz'

2) SystemRepository commitRestore

But after when i login from tode and i do the: ws   command 
the system sometimes answer with the error :

a InternalError occurred ( error 2261 ) , The object with object ID aTDSessionDescription is corrupt.  

Reason 'store past end'

in this case after 40 seconds the tode shell report:

GciSessionNotLoggedInError: Session no longer logged in.
Check the gem log for error messages (look at the most recent gem log file: `ls $GS_HOME/server/stones/base_3106/logs/gemnetobject*.log`.
If your stone is running -and healthy (`cat $GS_HOME/server/stones/base_3106/logs/base_3106.log`), you can try again.

sometimes answer the error:

a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'
In this case :
after four Abort    button click  the system open the workspace 

and the tode shell  report : AUTOCOMMIT DISABLEDAbortered ......


and the  tode 1 >     is red

Some considerations ?
Dario,

Am I right in thinking that you are doing the restore from backup into a stone on another system? If so you are likely missing some of the disk structure that tODE expects to find. but first I would like to see a full stack from the error that you are getting, because I would like to understand the context in which the "corrupt" object errors are occurring I will spend some time today and try to reproduce the problem locally so that I can characterize the problem more thoroughly ...

Dale



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Restore 3.1.0.6 backup

GLASS mailing list
In reply to this post by GLASS mailing list
Ciao Dale,


On 10/24/2018 09:11 AM, Trussardi Dario Romano via Glass wrote:
Ciao,

i have a deployment system based on stone 3.1.0.6 and gsDevKitHome.

The system was installed in early 2015.

Now i do some test to restored the database backup ( from this deployment system ) on another system  (  called emergency PC )

This emergency PC is based on GsDevKit_home git ebc8e7b commit.

On it i create the the stone with the command:  createStone base_3106  3.1.0.6

A this point all works fine from tode command.   ( ws  for example )

Now from topaz session i successfully gave the commands:

1) SystemRepository restoreFromBackup: '/opt/GsDkB/GsDevKit_home/server/stones/base_3106/backups/backup_20181023.dbf.gz'

2) SystemRepository commitRestore

But after when i login from tode and i do the: ws   command 
the system sometimes answer with the error :

a InternalError occurred ( error 2261 ) , The object with object ID aTDSessionDescription is corrupt.  

Reason 'store past end'

If in the dialog  that reports the error i click  the Debug button the system it's like in the screenshot



But after is instabled and blocked for 40 seconds and ....

in this case after 40 seconds the tode shell report:

GciSessionNotLoggedInError: Session no longer logged in.
Check the gem log for error messages (look at the most recent gem log file: `ls $GS_HOME/server/stones/base_3106/logs/gemnetobject*.log`.
If your stone is running -and healthy (`cat $GS_HOME/server/stones/base_3106/logs/base_3106.log`), you can try again.

But the base_3106.log haven't new data 

sometimes answer the error:

a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'
In this case :
after four Abort    button click  the system open the workspace 

and the tode shell  report : AUTOCOMMIT DISABLEDAbortered ......


and the  tode 1 >     is red

Some considerations ?
Dario,

Am I right in thinking that you are doing the restore from backup into a stone on another system?

Yes

If so you are likely missing some of the disk structure that tODE expects to find. but first I would like to see a full stack from the error that you are getting, because I would like to understand the context in which the "corrupt" object errors are occurring I will spend some time today and try to reproduce the problem locally so that I can characterize the problem more thoroughly ...

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Restore 3.1.0.6 backup

GLASS mailing list
In reply to this post by GLASS mailing list
Dale,

Dario,

In the tODE shell try running the following:

script --script=rebuildSys
I do this command  in the system where i restored the data ( Where the system is "unstable" after the SystemRepository restoreFromBackup: ..... ) 

If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

The system answer the following error:
a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

I press the Abort for some time.

After 60 seconds the Ubuntu system open a window :

The application  gem has closed unexpectedly "


The system is running with 86% of memory (of  8GB )  and  Swap at 81% ( of 8GB )

The tode shell report : GciSessionNotLoggedInError: Session no longer logged in
The recent gem log file is:


I need to restart the system ?

Now when i do the tode shell ws  command  i have:

a error dialog and do some click on the Abort  button the tode shell report after 50 seconds:
 
GciSessionNotLoggedInError: Session no longer logged in.
Dario


When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to $GS_HOME/shared/repos then the missing repo directory errors might go away ---

If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

gemnetobject7877.log (31K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Restore 3.1.0.6 backup

GLASS mailing list

Okay you're getting a SigSegv during scavenge and that is what is killing the gem ... I'm having the gem log file analyzed to see what else we can learn.

Dale


On 10/30/2018 12:51 PM, Trussardi Dario Romano via Glass wrote:
Dale,

Dario,

In the tODE shell try running the following:

script --script=rebuildSys
I do this command  in the system where i restored the data ( Where the system is "unstable" after the SystemRepository restoreFromBackup: ..... ) 

If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

The system answer the following error:
a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

I press the Abort for some time.

After 60 seconds the Ubuntu system open a window :

The application  gem has closed unexpectedly "


The system is running with 86% of memory (of  8GB )  and  Swap at 81% ( of 8GB )

The tode shell report : GciSessionNotLoggedInError: Session no longer logged in
The recent gem log file is:



I need to restart the system ?

Now when i do the tode shell ws  command  i have:

a error dialog and do some click on the Abort  button the tode shell report after 50 seconds:
 
GciSessionNotLoggedInError: Session no longer logged in.
Dario


When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to $GS_HOME/shared/repos then the missing repo directory errors might go away ---

If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Restore 3.1.0.6 backup

GLASS mailing list

Dario,

You may have run into a memory manager bug that has been fixed in later versions of GemStone ... you might try using a larger TOC to avoid the bug:

GEM_TEMPOBJ_CACHE_SIZE=200000;

let me know if that works,

Dale


On 10/30/2018 02:11 PM, Dale Henrichs wrote:

Okay you're getting a SigSegv during scavenge and that is what is killing the gem ... I'm having the gem log file analyzed to see what else we can learn.

Dale


On 10/30/2018 12:51 PM, Trussardi Dario Romano via Glass wrote:
Dale,

Dario,

In the tODE shell try running the following:

script --script=rebuildSys
I do this command  in the system where i restored the data ( Where the system is "unstable" after the SystemRepository restoreFromBackup: ..... ) 

If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

The system answer the following error:
a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

I press the Abort for some time.

After 60 seconds the Ubuntu system open a window :

The application  gem has closed unexpectedly "


The system is running with 86% of memory (of  8GB )  and  Swap at 81% ( of 8GB )

The tode shell report : GciSessionNotLoggedInError: Session no longer logged in
The recent gem log file is:



I need to restart the system ?

Now when i do the tode shell ws  command  i have:

a error dialog and do some click on the Abort  button the tode shell report after 50 seconds:
 
GciSessionNotLoggedInError: Session no longer logged in.
Dario


When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to $GS_HOME/shared/repos then the missing repo directory errors might go away ---

If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Restore 3.1.0.6 backup

GLASS mailing list
Ciao Dale,

sorry but I had a problem on a server.

 I was able to verify the restore on an old system and everything worked. 

 Do not ask me the details ( now ) but I think everything ( the origin GLASS where i do the backup -  and the Target GLASS where i restore  the data )

 refers to the old gsDevKitHome environment.

Dario,

You may have run into a memory manager bug that has been fixed in later versions of GemStone ... you might try using a larger TOC to avoid the bug:

GEM_TEMPOBJ_CACHE_SIZE=200000;
In which file ( path ) should I set this parameter?

Keep in mind that the problems about restore  I had on a system where I have the GsDevKit_home environment

Then:  the origin GLASS where i do the backup is based on gsDevKitHome

the target where i do the restore is based on gsDevKit_Home

let me know if that works,

Dale


On 10/30/2018 02:11 PM, Dale Henrichs wrote:

Okay you're getting a SigSegv during scavenge and that is what is killing the gem ... I'm having the gem log file analyzed to see what else we can learn.

Dale


On 10/30/2018 12:51 PM, Trussardi Dario Romano via Glass wrote:
Dale,

Dario,

In the tODE shell try running the following:

script --script=rebuildSys
I do this command  in the system where i restored the data ( Where the system is "unstable" after the SystemRepository restoreFromBackup: ..... ) 

If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

The system answer the following error:
a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

I press the Abort for some time.

After 60 seconds the Ubuntu system open a window :

The application  gem has closed unexpectedly "


The system is running with 86% of memory (of  8GB )  and  Swap at 81% ( of 8GB )

The tode shell report : GciSessionNotLoggedInError: Session no longer logged in
The recent gem log file is:



I need to restart the system ?

Now when i do the tode shell ws  command  i have:

a error dialog and do some click on the Abort  button the tode shell report after 50 seconds:
 
GciSessionNotLoggedInError: Session no longer logged in.
Dario


When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to

What do you mean?

If I have Seaside and other git repository, how should I behave?

$GS_HOME/shared/repos then the missing repo directory errors might go away ---

In the origin GLASS where i do the backup,  is based on gsDevKitHome,
 but i don't have any git entry about my repository.

If I remember correctly I managed ( at begin 2015 ) everything with GemTools and the Monticello Package.

This morning when, via remote tode, I gave the project list command I got the following result.

Thanks,

Dario


If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Restore 3.1.0.6 backup

GLASS mailing list



On 10/31/2018 11:30 AM, Trussardi Dario Romano via Glass wrote:
Ciao Dale,

sorry but I had a problem on a server.

 I was able to verify the restore on an old system and everything worked. 

 Do not ask me the details ( now ) but I think everything ( the origin GLASS where i do the backup -  and the Target GLASS where i restore  the data )

 refers to the old gsDevKitHome environment.
So you have resolved the "corrupt object errors"? Hardware issues would be a a likely cause of corrupt obj errors and sigsegvs and "garbage collection errors" and if you've "solved" your problem by running on different hardware then it isn't likely (tho still possible) that you are running into the bugs that have been fixed in later versions of GemStone ...

Dario,

You may have run into a memory manager bug that has been fixed in later versions of GemStone ... you might try using a larger TOC to avoid the bug:

GEM_TEMPOBJ_CACHE_SIZE=200000;
In which file ( path ) should I set this parameter?
This would be set in the system.conf file ($GS_HOME/server/stones/<stone-name>/extents/system.conf) or the gem.conf file ($GS_HOME/server/stones/<stone-name>/gem.conf)

Keep in mind that the problems about restore  I had on a system where I have the GsDevKit_home environment

Then:  the origin GLASS where i do the backup is based on gsDevKitHome

the target where i do the restore is based on gsDevKit_Home
Ah now I understand a bit more ... the `script --script=rebuildSys` script should work in the new environment, but the version of tODE installed for gsDevKitHome may be too old (I don't recall at the moment).

First we want to get rid of the "corrupt object errors" ... when those are cleared up I then expect that if there are additional incompatibilities between tODE for gsDevKitHome and tODE in GsDevKit_home, they should show up as regular Smalltalk errors (instead of SigSegV) and we can work through the process of getting your app moved to the new version of tODE ...

let me know if that works,

Dale


On 10/30/2018 02:11 PM, Dale Henrichs wrote:

Okay you're getting a SigSegv during scavenge and that is what is killing the gem ... I'm having the gem log file analyzed to see what else we can learn.

Dale


On 10/30/2018 12:51 PM, Trussardi Dario Romano via Glass wrote:
Dale,

Dario,

In the tODE shell try running the following:

script --script=rebuildSys
I do this command  in the system where i restored the data ( Where the system is "unstable" after the SystemRepository restoreFromBackup: ..... ) 

If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

The system answer the following error:
a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

I press the Abort for some time.

After 60 seconds the Ubuntu system open a window :

The application  gem has closed unexpectedly "


The system is running with 86% of memory (of  8GB )  and  Swap at 81% ( of 8GB )

The tode shell report : GciSessionNotLoggedInError: Session no longer logged in
The recent gem log file is:



I need to restart the system ?

Now when i do the tode shell ws  command  i have:

a error dialog and do some click on the Abort  button the tode shell report after 50 seconds:
 
GciSessionNotLoggedInError: Session no longer logged in.
Dario


When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to

What do you mean?
The tode_rogue.dbf.gz backup file came from my laptop (rogue) and I did the restore of the backup file in a stone that is running on my desktop machine (foos). Both machines are running in GsDevKit_home, so they do not exactly match your situation. The important thing is that immediately after the restore tODE's internal state (which does depend upon directory structure) is out of synch an you can see at the bottom of the console the "odd ls results":


Then when I run `script --script=rebuildSys`, I get the "repository related errors" ... which I `Abort`ed


After aborting several of the errors, the `script` command finishes and is finally correct:


A look at the project list:


shows that the projects with `rogue` in the path are not being found on disk on foos ... to repair the "red" projects, you need to use the Project>unlock menu item followed by the Project>unregister menu item (assuming that you've got the git repos copied over to the new machine's $GS_HOME/shared/repos directory and have checked out the same SHA):

When you're done, you will have gotten rid of the redness:


but the project appear to be "unloaded" so as a final step you need to reload each of the projects (repairing the unlock) ...If you load the tODE project first a bunch of the dependent projects will be reloaded:

I would make a new backup at this point in case something fails partway through ... then reload any other projects that were originally loaded (most of the projects in this project list weren't loaded in the first place ...

If I have Seaside and other git repository, how should I behave?
So if you have Seaside and other projects, you make sure that the git repo is copied over and that the correct version of the project is loaded ... if you have Monticello projects (non-git projects) they will not show up as red and will not need to be reloaded .....

If you have errors at any step, I will need to take a look at the specific error to figure out what might be happening ... in the end we will probably want/to update to a newer version of tODE ... but I will have to set up a test gsdevkithome so that I can try experiments myself ...

$GS_HOME/shared/repos then the missing repo directory errors might go away ---

In the origin GLASS where i do the backup,  is based on gsDevKitHome,
 but i don't have any git entry about my repository.

If I remember correctly I managed ( at begin 2015 ) everything with GemTools and the Monticello Package.

This morning when, via remote tode, I gave the project list command I got the following result.

Yes this looks consistent with my instructions from above ... so make a new backup (do `man bu` to see the man page for the backup-related commands) and then `bu backup <backup-name>` to create a backup ... `bu list` list backup files, `bu restore <backup-file>` restores from backup ...

Dale
Thanks,

Dario


If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Restore 3.1.0.6 backup

GLASS mailing list
Dale, 

On 10/31/2018 11:30 AM, Trussardi Dario Romano via Glass wrote:
Ciao Dale,

sorry but I had a problem on a server.

it was not the server in question.

 I was able to verify the restore on an old system and everything worked. 

I have hard disk where is replicate the operating system and  the gsDevKitHome environment and the 3.1.0.6 database.
After startup the emergency PC with this device i do  ( from tode shell ) the restore operation without problem. 

I used the same original  gestionale.dbf.gz   backup file.


 Do not ask me the details ( now ) but I think everything ( the origin GLASS where i do the backup -  and the Target GLASS where i restore  the data )

 refers to the old gsDevKitHome environment.
So you have resolved the "corrupt object errors"? Hardware issues would be a a likely cause of corrupt obj errors and sigsegvs and "garbage collection errors" and if you've "solved" your problem by running on different hardware then it isn't likely (tho still possible) that you are running into the bugs that have been fixed in later versions of GemStone ...

OK, good to know.

But in the emergency system with GsDevKit_home environment and relative 3.1.0.6 database

 where i restored the same original gestionale.dbf.gz   backup file

the problem is still that  ( see my first email dated 24/10 )

You may have run into a memory manager bug that has been fixed in later versions of GemStone ... you might try using a larger TOC to avoid the bug:

GEM_TEMPOBJ_CACHE_SIZE=200000;
In which file ( path ) should I set this parameter?
This would be set in the system.conf file ($GS_HOME/server/stones/<stone-name>/extents/system.conf) or the gem.conf file ($GS_HOME/server/stones/<stone-name>/gem.conf)

I changed both.


Keep in mind that the problems about restore  I had on a system where I have the GsDevKit_home environment

Then:  the origin GLASS where i do the backup is based on gsDevKitHome

the target where i do the restore is based on gsDevKit_Home
Ah now I understand a bit more ... the `script --script=rebuildSys` script should work in the new environment, but the version of tODE installed for gsDevKitHome may be too old (I don't recall at the moment).

Now the tODE where i do the  script --script=rebuildSys  is based on gsDevKit_Home  

But the system answer the same error:
a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

I press the Abort for some time.

After 40 seconds the tode shell report : GciSessionNotLoggedInError: Session no longer logged in
The recent gem log file is:


Question: 

is it possible that the problem is related to the DataCurator password?

The restore  database has a different password than the 'swordfish' 

I change to the new restore password into: 

1)  tODE entry description password
2)   the ../server/base_3106/product/seaside/etc/ gemstone.secret

  but I have some doubts.

Also because when i run:   stopStone base_3106 the system report:

=================
   GsDevKit script: stopStone base_3106
              path: /opt/GsDKb/GsDevKit_home/bin/stopStone
=================


Error running shell command: '/opt/GsDKb/GsDevKit_home/server/bin/gs/stopGemstone'
with args:
STDOUT: '--- 02/11/18 14:59:41.276 CET ---
stopstone[Info]: GemStone version ''3.1.0.6''
stopstone[Info]: initiating ''base_3106'' shutdown...
-----------------------------------------------------
GemStone: Error         Fatal
Login failed:  the GemStone userId/password combination is invalid
or expired.
Error Category: 231169 [GemStone] Number: 4051  Arg Count: 0 Context : 20 exception : 20
'
STDERR: =================
   GsDevKit script: stopGemstone
              path: /opt/GsDKb/GsDevKit_home/server/bin/gs/stopGemstone
=================
stopstone[Error]: Invalid username(DataCurator) or password.
Error on or near line 22 :: stopGemstone  :: stopGemstone
Error: Shell command: '/opt/GsDKb/GsDevKit_home/server/bin/gs/stopGemstone' failed.
GsDevKitStopstoneCommandLineHandler class(Object)>>error:
GsDevKitStopstoneCommandLineHandler class(GsDevKitAbstractCommandLineHandler class)>>runShellCommand:args:noError:
GsDevKitStopstoneCommandLineHandler(GsDevKitAbstractCommandLineHandler)>>runShellCommand:args:
GsDevKitStopstoneCommandLineHandler>>activate
GsDevKitStopstoneCommandLineHandler class(CommandLineHandler class)>>activateWith:
PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: in Block: [ aCommandLinehandler activateWith: commandLine ]
BlockClosure>>on:do:
PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand:
PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand
PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument:
PharoCommandLineHandler(BasicCommandLineHandler)>>activate in Block: [ self handleArgument: (self arguments ifEmpty: [ ...etc...
BlockClosure>>on:do:
PharoCommandLineHandler(BasicCommandLineHandler)>>activate
PharoCommandLineHandler>>activate
PharoCommandLineHandler class(CommandLineHandler class)>>activateWith:
PharoCommandLineHandler class>>activateWith: in Block: [ super activateWith: aCommandLine ]
WorldState>>runStepMethodsIn:
WorldMorph>>runStepMethods
WorldState>>doOneCycleNowFor:
WorldState>>doOneCycleFor:
WorldMorph>>doOneCycle
MorphicUIManager>>spawnNewProcess in Block: [ ...
BlockClosure>>newProcess in Block: [ ...
Error on or near line 99 :: devKitCommandLine stopstone base_3106 :: devKitCommandLine stopstone base_3106
Error on or near line 74 :: stopStone base_3106 :: stopStone base_3106

Thanks,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

gemnetobject4067.log (34K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Restore 3.1.0.6 backup

GLASS mailing list
In reply to this post by GLASS mailing list

Dario,

At this point I would like to understand whether or not you have an outstanding issue with backups ... if you do have outstanding issues lets start again and pursue only one question at a time ...

Dale

On 10/31/18 11:30 AM, Trussardi Dario Romano via Glass wrote:
Ciao Dale,

sorry but I had a problem on a server.

 I was able to verify the restore on an old system and everything worked. 

 Do not ask me the details ( now ) but I think everything ( the origin GLASS where i do the backup -  and the Target GLASS where i restore  the data )

 refers to the old gsDevKitHome environment.

Dario,

You may have run into a memory manager bug that has been fixed in later versions of GemStone ... you might try using a larger TOC to avoid the bug:

GEM_TEMPOBJ_CACHE_SIZE=200000;
In which file ( path ) should I set this parameter?

Keep in mind that the problems about restore  I had on a system where I have the GsDevKit_home environment

Then:  the origin GLASS where i do the backup is based on gsDevKitHome

the target where i do the restore is based on gsDevKit_Home

let me know if that works,

Dale


On 10/30/2018 02:11 PM, Dale Henrichs wrote:

Okay you're getting a SigSegv during scavenge and that is what is killing the gem ... I'm having the gem log file analyzed to see what else we can learn.

Dale


On 10/30/2018 12:51 PM, Trussardi Dario Romano via Glass wrote:
Dale,

Dario,

In the tODE shell try running the following:

script --script=rebuildSys
I do this command  in the system where i restored the data ( Where the system is "unstable" after the SystemRepository restoreFromBackup: ..... ) 

If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

The system answer the following error:
a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

I press the Abort for some time.

After 60 seconds the Ubuntu system open a window :

The application  gem has closed unexpectedly "


The system is running with 86% of memory (of  8GB )  and  Swap at 81% ( of 8GB )

The tode shell report : GciSessionNotLoggedInError: Session no longer logged in
The recent gem log file is:



I need to restart the system ?

Now when i do the tode shell ws  command  i have:

a error dialog and do some click on the Abort  button the tode shell report after 50 seconds:
 
GciSessionNotLoggedInError: Session no longer logged in.
Dario


When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to

What do you mean?

If I have Seaside and other git repository, how should I behave?

$GS_HOME/shared/repos then the missing repo directory errors might go away ---

In the origin GLASS where i do the backup,  is based on gsDevKitHome,
 but i don't have any git entry about my repository.

If I remember correctly I managed ( at begin 2015 ) everything with GemTools and the Monticello Package.

This morning when, via remote tode, I gave the project list command I got the following result.

Thanks,

Dario


If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Restore 3.1.0.6 backup

GLASS mailing list
Ciao,

Dario,

At this point I would like to understand whether or not you have an outstanding issue with backups ..

1) The deployment system is based on gsDevKitHome ( where i do the backup )

2) I have another hard disk with operating system and the same gsDevKitHome environment.

In this system the restore works fine.

In case of problems on the deployment server, can in any case guarantee the restart from the last backup.

And this is what matters.



3) When i do the restore  ( of the same backup file ) on a system based on GsDevKit_home environment ( with the same stone version 3.1.0.6 )

i found some problematic.  

Now if you believe in any case to stay on the problem and resolve the issue in any case,

i see to resume the matter the next few days.

What do you think ?

4) I'm wondering how i can take data from a stone version to a later stone version.

Do i need saving the data in SIXX format and restoring it to the new version of the stone?

. if you do have outstanding issues lets start again and pursue only one question at a time ... 


Thanks,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass