Restore 3.1.0.6 backup

Ciao, Check the gem log for error messages (look at the most recent gem log file: `ls $GS_HOME/server/stones/base_3106/logs/gemnetobject*.log`.
If your stone is running -and healthy (`cat $GS_HOME/server/stones/base_3106/logs/base_3106.log`), you can try again.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Ciao,

i have a deployment system based on stone 3.1.0.6 and gsDevKitHome.

The system was installed in early 2015.

Now i do some test to restored the database backup ( from this deployment system ) on another system  (  called emergency PC )

This emergency PC is based on GsDevKit_home git ebc8e7b commit.

On it i create the the stone with the command:  createStone base_3106  3.1.0.6

A this point all works fine from tode command.   ( ws  for example )

Now from topaz session i successfully gave the commands:

1) SystemRepository restoreFromBackup: '/opt/GsDkB/GsDevKit_home/server/stones/base_3106/backups/backup_20181023.dbf.gz'

2) SystemRepository commitRestore

But after when i login from tode and i do the: ws   command 

the system sometimes answer with the error :

a InternalError occurred ( error 2261 ) , The object with object ID aTDSessionDescription is corrupt.  

Reason 'store past end'

in this case after 40 seconds the tode shell report:

GciSessionNotLoggedInError: Session no longer logged in.

Check the gem log for error messages (look at the most recent gem log file: `ls $GS_HOME/server/stones/base_3106/logs/gemnetobject*.log`.
If your stone is running -and healthy (`cat $GS_HOME/server/stones/base_3106/logs/base_3106.log`), you can try again.

sometimes answer the error:

a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

In this case :

after four Abort    button click  the system open the workspace 

and the tode shell  report : AUTOCOMMIT DISABLEDAbortered ......

and the  tode 1 >     is red

Some considerations ?

Dario,

Am I right in thinking that you are doing the restore from backup into a stone on another system? If so you are likely missing some of the disk structure that tODE expects to find. but first I would like to see a full stack from the error that you are getting, because I would like to understand the context in which the "corrupt" object errors are occurring I will spend some time today and try to reproduce the problem locally so that I can characterize the problem more thoroughly ...

Dale


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Dario,

In the tODE shell try running the following:

script --script=rebuildSys

If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to $GS_HOME/shared/repos then the missing repo directory errors might go away ---

If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...

Dale

On 10/24/2018 09:11 AM, Trussardi Dario Romano via Glass wrote:

Ciao,

i have a deployment system based on stone 3.1.0.6 and gsDevKitHome.

The system was installed in early 2015.

Now i do some test to restored the database backup ( from this deployment system ) on another system  (  called emergency PC )

This emergency PC is based on GsDevKit_home git ebc8e7b commit.

On it i create the the stone with the command:  createStone base_3106  3.1.0.6

A this point all works fine from tode command.   ( ws  for example )

Now from topaz session i successfully gave the commands:

1) SystemRepository restoreFromBackup: '/opt/GsDkB/GsDevKit_home/server/stones/base_3106/backups/backup_20181023.dbf.gz'

2) SystemRepository commitRestore

But after when i login from tode and i do the: ws   command 

the system sometimes answer with the error :

a InternalError occurred ( error 2261 ) , The object with object ID aTDSessionDescription is corrupt.  

Reason 'store past end'

in this case after 40 seconds the tode shell report:

GciSessionNotLoggedInError: Session no longer logged in.

Check the gem log for error messages (look at the most recent gem log file: `ls $GS_HOME/server/stones/base_3106/logs/gemnetobject*.log`.
If your stone is running -and healthy (`cat $GS_HOME/server/stones/base_3106/logs/base_3106.log`), you can try again.

sometimes answer the error:

a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

In this case :

after four Abort    button click  the system open the workspace 

and the tode shell  report : AUTOCOMMIT DISABLEDAbortered ......

and the  tode 1 >     is red

Some considerations ?

Dario,

Am I right in thinking that you are doing the restore from backup into a stone on another system? If so you are likely missing some of the disk structure that tODE expects to find. but first I would like to see a full stack from the error that you are getting, because I would like to understand the context in which the "corrupt" object errors are occurring I will spend some time today and try to reproduce the problem locally so that I can characterize the problem more thoroughly ...

Dale

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Okay you're getting a SigSegv during scavenge and that is what is killing the gem ... I'm having the gem log file analyzed to see what else we can learn.

Dale

Dale,

Dario,

In the tODE shell try running the following:

script --script=rebuildSys

I do this command  in the system where i restored the data ( Where the system is "unstable" after the SystemRepository restoreFromBackup: ..... ) 

If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

The system answer the following error:

a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

I press the Abort for some time.

After 60 seconds the Ubuntu system open a window :

" The application  gem has closed unexpectedly "

The system is running with 86% of memory (of  8GB )  and  Swap at 81% ( of 8GB )

The tode shell report : GciSessionNotLoggedInError: Session no longer logged in

The recent gem log file is:

I need to restart the system ?

Now when i do the tode shell ws  command  i have:

a error dialog and do some click on the Abort  button the tode shell report after 50 seconds:

 

GciSessionNotLoggedInError: Session no longer logged in.

Dario

When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to $GS_HOME/shared/repos then the missing repo directory errors might go away ---

If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Dario,

You may have run into a memory manager bug that has been fixed in later versions of GemStone ... you might try using a larger TOC to avoid the bug:

GEM_TEMPOBJ_CACHE_SIZE=200000;

let me know if that works,

Dale

Okay you're getting a SigSegv during scavenge and that is what is killing the gem ... I'm having the gem log file analyzed to see what else we can learn.

Dale

On 10/30/2018 12:51 PM, Trussardi Dario Romano via Glass wrote:

Dale,

Dario,

In the tODE shell try running the following:

script --script=rebuildSys

I do this command  in the system where i restored the data ( Where the system is "unstable" after the SystemRepository restoreFromBackup: ..... ) 

If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

The system answer the following error:

a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

I press the Abort for some time.

After 60 seconds the Ubuntu system open a window :

" The application  gem has closed unexpectedly "

The system is running with 86% of memory (of  8GB )  and  Swap at 81% ( of 8GB )

The tode shell report : GciSessionNotLoggedInError: Session no longer logged in

The recent gem log file is:

I need to restart the system ?

Now when i do the tode shell ws  command  i have:

a error dialog and do some click on the Abort  button the tode shell report after 50 seconds:

 

GciSessionNotLoggedInError: Session no longer logged in.

Dario

When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to $GS_HOME/shared/repos then the missing repo directory errors might go away ---

If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Ciao Dale,

sorry but I had a problem on a server.

 I was able to verify the restore on an old system and everything worked. 

 Do not ask me the details ( now ) but I think everything ( the origin GLASS where i do the backup -  and the Target GLASS where i restore  the data )

 refers to the old gsDevKitHome environment.

So you have resolved the "corrupt object errors"? Hardware issues would be a a likely cause of corrupt obj errors and sigsegvs and "garbage collection errors" and if you've "solved" your problem by running on different hardware then it isn't likely (tho still possible) that you are running into the bugs that have been fixed in later versions of GemStone ...

Dario,

You may have run into a memory manager bug that has been fixed in later versions of GemStone ... you might try using a larger TOC to avoid the bug:

GEM_TEMPOBJ_CACHE_SIZE=200000;

In which file ( path ) should I set this parameter?

This would be set in the system.conf file ($GS_HOME/server/stones/<stone-name>/extents/system.conf) or the gem.conf file ($GS_HOME/server/stones/<stone-name>/gem.conf)

Keep in mind that the problems about restore  I had on a system where I have the GsDevKit_home environment

Then:  the origin GLASS where i do the backup is based on gsDevKitHome

the target where i do the restore is based on gsDevKit_Home

Ah now I understand a bit more ... the `script --script=rebuildSys` script should work in the new environment, but the version of tODE installed for gsDevKitHome may be too old (I don't recall at the moment).

First we want to get rid of the "corrupt object errors" ... when those are cleared up I then expect that if there are additional incompatibilities between tODE for gsDevKitHome and tODE in GsDevKit_home, they should show up as regular Smalltalk errors (instead of SigSegV) and we can work through the process of getting your app moved to the new version of tODE ...

let me know if that works,

Dale

On 10/30/2018 02:11 PM, Dale Henrichs wrote:

Okay you're getting a SigSegv during scavenge and that is what is killing the gem ... I'm having the gem log file analyzed to see what else we can learn.

Dale

On 10/30/2018 12:51 PM, Trussardi Dario Romano via Glass wrote:

Dale,

Dario,

In the tODE shell try running the following:

script --script=rebuildSys

I do this command  in the system where i restored the data ( Where the system is "unstable" after the SystemRepository restoreFromBackup: ..... ) 

If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

The system answer the following error:

a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

I press the Abort for some time.

After 60 seconds the Ubuntu system open a window :

" The application  gem has closed unexpectedly "

The system is running with 86% of memory (of  8GB )  and  Swap at 81% ( of 8GB )

The tode shell report : GciSessionNotLoggedInError: Session no longer logged in

The recent gem log file is:

I need to restart the system ?

Now when i do the tode shell ws  command  i have:

a error dialog and do some click on the Abort  button the tode shell report after 50 seconds:

 

GciSessionNotLoggedInError: Session no longer logged in.

Dario

When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to

What do you mean?

The tode_rogue.dbf.gz backup file came from my laptop (rogue) and I did the restore of the backup file in a stone that is running on my desktop machine (foos). Both machines are running in GsDevKit_home, so they do not exactly match your situation. The important thing is that immediately after the restore tODE's internal state (which does depend upon directory structure) is out of synch an you can see at the bottom of the console the "odd ls results":


Then when I run `script --script=rebuildSys`, I get the "repository related errors" ... which I `Abort`ed


After aborting several of the errors, the `script` command finishes and is finally correct:


A look at the project list:


shows that the projects with `rogue` in the path are not being found on disk on foos ... to repair the "red" projects, you need to use the Project>unlock menu item followed by the Project>unregister menu item (assuming that you've got the git repos copied over to the new machine's $GS_HOME/shared/repos directory and have checked out the same SHA):

When you're done, you will have gotten rid of the redness:


but the project appear to be "unloaded" so as a final step you need to reload each of the projects (repairing the unlock) ...If you load the tODE project first a bunch of the dependent projects will be reloaded:

I would make a new backup at this point in case something fails partway through ... then reload any other projects that were originally loaded (most of the projects in this project list weren't loaded in the first place ...

If I have Seaside and other git repository, how should I behave?

So if you have Seaside and other projects, you make sure that the git repo is copied over and that the correct version of the project is loaded ... if you have Monticello projects (non-git projects) they will not show up as red and will not need to be reloaded .....

If you have errors at any step, I will need to take a look at the specific error to figure out what might be happening ... in the end we will probably want/to update to a newer version of tODE ... but I will have to set up a test gsdevkithome so that I can try experiments myself ...

$GS_HOME/shared/repos then the missing repo directory errors might go away ---

In the origin GLASS where i do the backup,  is based on gsDevKitHome,

 but i don't have any git entry about my repository.

If I remember correctly I managed ( at begin 2015 ) everything with GemTools and the Monticello Package.

This morning when, via remote tode, I gave the project list command I got the following result.

Yes this looks consistent with my instructions from above ... so make a new backup (do `man bu` to see the man page for the backup-related commands) and then `bu backup <backup-name>` to create a backup ... `bu list` list backup files, `bu restore <backup-file>` restores from backup ...

Dale

Thanks,

Dario

If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

=================

   GsDevKit script: stopStone base_3106

              path: /opt/GsDKb/GsDevKit_home/bin/stopStone

=================

Error running shell command: '/opt/GsDKb/GsDevKit_home/server/bin/gs/stopGemstone'

with args:

STDOUT: '--- 02/11/18 14:59:41.276 CET ---

stopstone[Info]: GemStone version ''3.1.0.6''

stopstone[Info]: initiating ''base_3106'' shutdown...

-----------------------------------------------------

GemStone: Error         Fatal

Login failed:  the GemStone userId/password combination is invalid

or expired.

Error Category: 231169 [GemStone] Number: 4051  Arg Count: 0 Context : 20 exception : 20

'

STDERR: =================

   GsDevKit script: stopGemstone

              path: /opt/GsDKb/GsDevKit_home/server/bin/gs/stopGemstone

=================

stopstone[Error]: Invalid username(DataCurator) or password.

Error on or near line 22 :: stopGemstone  :: stopGemstone

Error: Shell command: '/opt/GsDKb/GsDevKit_home/server/bin/gs/stopGemstone' failed.

GsDevKitStopstoneCommandLineHandler class(Object)>>error:

GsDevKitStopstoneCommandLineHandler class(GsDevKitAbstractCommandLineHandler class)>>runShellCommand:args:noError:

GsDevKitStopstoneCommandLineHandler(GsDevKitAbstractCommandLineHandler)>>runShellCommand:args:

GsDevKitStopstoneCommandLineHandler>>activate

GsDevKitStopstoneCommandLineHandler class(CommandLineHandler class)>>activateWith:

PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: in Block: [ aCommandLinehandler activateWith: commandLine ]

BlockClosure>>on:do:

PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand:

PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand

PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument:

PharoCommandLineHandler(BasicCommandLineHandler)>>activate in Block: [ self handleArgument: (self arguments ifEmpty: [ ...etc...

BlockClosure>>on:do:

PharoCommandLineHandler(BasicCommandLineHandler)>>activate

PharoCommandLineHandler>>activate

PharoCommandLineHandler class(CommandLineHandler class)>>activateWith:

PharoCommandLineHandler class>>activateWith: in Block: [ super activateWith: aCommandLine ]

WorldState>>runStepMethodsIn:

WorldMorph>>runStepMethods

WorldState>>doOneCycleNowFor:

WorldState>>doOneCycleFor:

WorldMorph>>doOneCycle

MorphicUIManager>>spawnNewProcess in Block: [ ...

BlockClosure>>newProcess in Block: [ ...

Error on or near line 99 :: devKitCommandLine stopstone base_3106 :: devKitCommandLine stopstone base_3106

Error on or near line 74 :: stopStone base_3106 :: stopStone base_3106

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Dario,

At this point I would like to understand whether or not you have an outstanding issue with backups ... if you do have outstanding issues lets start again and pursue only one question at a time ...

Dale

Ciao Dale,

sorry but I had a problem on a server.

 I was able to verify the restore on an old system and everything worked. 

 Do not ask me the details ( now ) but I think everything ( the origin GLASS where i do the backup -  and the Target GLASS where i restore  the data )

 refers to the old gsDevKitHome environment.

Dario,

You may have run into a memory manager bug that has been fixed in later versions of GemStone ... you might try using a larger TOC to avoid the bug:

GEM_TEMPOBJ_CACHE_SIZE=200000;

In which file ( path ) should I set this parameter?

Keep in mind that the problems about restore  I had on a system where I have the GsDevKit_home environment

Then:  the origin GLASS where i do the backup is based on gsDevKitHome

the target where i do the restore is based on gsDevKit_Home

let me know if that works,

Dale

On 10/30/2018 02:11 PM, Dale Henrichs wrote:

Okay you're getting a SigSegv during scavenge and that is what is killing the gem ... I'm having the gem log file analyzed to see what else we can learn.

Dale

On 10/30/2018 12:51 PM, Trussardi Dario Romano via Glass wrote:

Dale,

Dario,

In the tODE shell try running the following:

script --script=rebuildSys

I do this command  in the system where i restored the data ( Where the system is "unstable" after the SystemRepository restoreFromBackup: ..... ) 

If you get some walkbacks about missing directories go ahead press the terminate button ... most of the repair should be complete ... the repository paths will have to be repaired separately.

The system answer the following error:

a InternalError occurred ( error 2261 ) , The object with object ID remoteNil is corrupt.

Reason 'store past end'

I press the Abort for some time.

After 60 seconds the Ubuntu system open a window :

" The application  gem has closed unexpectedly "

The system is running with 86% of memory (of  8GB )  and  Swap at 81% ( of 8GB )

The tode shell report : GciSessionNotLoggedInError: Session no longer logged in

The recent gem log file is:

I need to restart the system ?

Now when i do the tode shell ws  command  i have:

a error dialog and do some click on the Abort  button the tode shell report after 50 seconds:

 

GciSessionNotLoggedInError: Session no longer logged in.

Dario

When I did a restore to a different machine I did notice that doing an `ls` in the tODE shell gave me an "odd" result, but I didn't see the issues that you were reporting ... rebuildSys should connect the tODE directory structure to the current GsDevKit_home directory structure ... if you copy over your git repositories to

What do you mean?

If I have Seaside and other git repository, how should I behave?

$GS_HOME/shared/repos then the missing repo directory errors might go away ---

In the origin GLASS where i do the backup,  is based on gsDevKitHome,

 but i don't have any git entry about my repository.

If I remember correctly I managed ( at begin 2015 ) everything with GemTools and the Monticello Package.

This morning when, via remote tode, I gave the project list command I got the following result.

Thanks,

Dario

If they don't I should be able to suggest additional things to try ... at the end of the day, we should be able to build up a repair script that properly repairs the tODE structures when you are restoring to completely different machine ... a use case that I had not planned for ... but a use case that I think is important ...

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass