Smalltalk › Gemtalk › GLASS

Gemstone status: cache frozen

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

8 messages Options

GLASS mailing list

Gemstone status: cache frozen

Hi,

i have a devKit glass environment run on Ubuntu server with GemStone_daemontools support.

Tonight due to bad weather, the system go down ( but i don't know the detail: how many time is down ........ ...)

The /gemstone_status command this morning report:

Status   Version    Owner    Pid   Port   Started     Type       Name
------- --------- --------- ----- ----- ------------ ------      ----
frozen 3.1.0.6   scandella 1132 46543 giu 04 20:49 cache       gestionale~9af495ccd1149d82

/etc/service/gs_maintenance: down 19 seconds, normally up
/etc/service/gs_seaside-9060: down 19 seconds, normally up
/etc/service/gs_seaside-9061: down 19 seconds, normally up
/etc/service/gs_seaside-9062: down 19 seconds, normally up
/etc/service/gs_seaside-9063: down 19 seconds, normally up
/etc/service/gs_seaside-9064: down 19 seconds, normally up
/etc/service/gs_seaside-9065: down 19 seconds, normally up
/etc/service/gs_statmon-1: down 19 seconds, normally up
/etc/service/gs_statmon-60: down 19 seconds, normally up

When i do the command : ./gemstone_start the system answer:

This script restarts all Gemstone processes.
Shall I continue? (Y/N)
n
Not continuing
scandella@scandella:/srv/gitrepository/GemStone_daemontools_setup/bin$ ./gemstone_start
This script restarts all Gemstone processes.
Shall I continue? (Y/N)
y

startstone[Error]: Stone process (id=3831) has died.
startstone[Error]: Examine '/opt/oodb/gsDevKitHome/gemstone/stones/gestionale/logs/gestionale.log' for more information. Excerpt follows:

A this point i need to kill the gemstone cache task.

After when i restart the system ( ./gemstone_start ) al works fine.

I don't understand because the cache is go into frozen stataus.

Some considerations ?

Thanks,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Gemstone status: cache frozen

A “frozen” state means that there is a lock file but the PID associated with the lock file is not responding. Typically this is because the process did not exit cleanly so did not delete the lock file. Typically using the ‘-c’ (for cleanup locks) argument on gslist will remove the frozen status.

James

On Jun 7, 2015, at 10:49 PM, Dario Trussardi via Glass <[hidden email]> wrote:

Hi,

i have a devKit glass environment run on Ubuntu server with GemStone_daemontools support.

Tonight due to bad weather, the system go down ( but i don't know the detail: how many time is down ........ ...)

The /gemstone_status command this morning report:

Status   Version    Owner    Pid   Port   Started     Type       Name
------- --------- --------- ----- ----- ------------ ------      ----
frozen 3.1.0.6   scandella 1132 46543 giu 04 20:49 cache       gestionale~9af495ccd1149d82

/etc/service/gs_maintenance: down 19 seconds, normally up
/etc/service/gs_seaside-9060: down 19 seconds, normally up
/etc/service/gs_seaside-9061: down 19 seconds, normally up
/etc/service/gs_seaside-9062: down 19 seconds, normally up
/etc/service/gs_seaside-9063: down 19 seconds, normally up
/etc/service/gs_seaside-9064: down 19 seconds, normally up
/etc/service/gs_seaside-9065: down 19 seconds, normally up
/etc/service/gs_statmon-1: down 19 seconds, normally up
/etc/service/gs_statmon-60: down 19 seconds, normally up

When i do the command : ./gemstone_start the system answer:

This script restarts all Gemstone processes.
Shall I continue? (Y/N)
n
Not continuing
scandella@scandella:/srv/gitrepository/GemStone_daemontools_setup/bin$ ./gemstone_start
This script restarts all Gemstone processes.
Shall I continue? (Y/N)
y

startstone[Error]: Stone process (id=3831) has died.
startstone[Error]: Examine '/opt/oodb/gsDevKitHome/gemstone/stones/gestionale/logs/gestionale.log' for more information. Excerpt follows:

A this point i need to kill the gemstone cache task.
After when i restart the system ( ./gemstone_start ) al works fine.

I don't understand because the cache is go into frozen stataus.

Some considerations ?

Thanks,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Gemstone status: cache frozen

In reply to this post by GLASS mailing list

Hi,

Hi,

i have a devKit glass environment run on Ubuntu server with GemStone_daemontools support.

Tonight due to bad weather, the system go down ( but i don't know the detail: how many time is down ........ ...)

I think the system is not down because the cache frozen result start: giu 04 20:49

But because and when the cache go into frozen status?

Because other service

OK 3.1.0.6 scandella 28923 40500 apr 23 12:24 Stone gestionale

OK 3.1.0.6 scandella 1430 50377 mar 18 08:51 Netldi gs64ldi

are down?

Thanks for considerations,

Dario

The /gemstone_status command this morning report:

Status   Version    Owner    Pid   Port   Started     Type       Name
------- --------- --------- ----- ----- ------------ ------      ----
frozen 3.1.0.6   scandella 1132 46543 giu 04 20:49 cache       gestionale~9af495ccd1149d82

/etc/service/gs_maintenance: down 19 seconds, normally up
/etc/service/gs_seaside-9060: down 19 seconds, normally up
/etc/service/gs_seaside-9061: down 19 seconds, normally up
/etc/service/gs_seaside-9062: down 19 seconds, normally up
/etc/service/gs_seaside-9063: down 19 seconds, normally up
/etc/service/gs_seaside-9064: down 19 seconds, normally up
/etc/service/gs_seaside-9065: down 19 seconds, normally up
/etc/service/gs_statmon-1: down 19 seconds, normally up
/etc/service/gs_statmon-60: down 19 seconds, normally up

When i do the command : ./gemstone_start the system answer:

This script restarts all Gemstone processes.
Shall I continue? (Y/N)
n
Not continuing
scandella@scandella:/srv/gitrepository/GemStone_daemontools_setup/bin$ ./gemstone_start
This script restarts all Gemstone processes.
Shall I continue? (Y/N)
y

startstone[Error]: Stone process (id=3831) has died.
startstone[Error]: Examine '/opt/oodb/gsDevKitHome/gemstone/stones/gestionale/logs/gestionale.log' for more information. Excerpt follows:

A this point i need to kill the gemstone cache task.
After when i restart the system ( ./gemstone_start ) al works fine.

I don't understand because the cache is go into frozen stataus.

Some considerations ?

Thanks,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Gemstone status: cache frozen

So what do you get from ‘gslist -cvl’?

On Jun 8, 2015, at 1:38 AM, Dario Trussardi via Glass <[hidden email]> wrote:

Hi,

Hi,

i have a devKit glass environment run on Ubuntu server with GemStone_daemontools support.

Tonight due to bad weather, the system go down ( but i don't know the detail: how many time is down ........ ...)

I think the system is not down because the cache frozen result start:   giu 04 20:49

But because and when the cache go into frozen status?

Because other service

OK 3.1.0.6 scandella 28923 40500 apr 23 12:24 Stone gestionale
OK 3.1.0.6 scandella 1430 50377 mar 18 08:51 Netldi gs64ldi

are down?

Thanks for considerations,

Dario

The /gemstone_status command this morning report:

Status   Version    Owner    Pid   Port   Started     Type       Name
------- --------- --------- ----- ----- ------------ ------      ----
frozen 3.1.0.6   scandella 1132 46543 giu 04 20:49 cache       gestionale~9af495ccd1149d82

/etc/service/gs_maintenance: down 19 seconds, normally up
/etc/service/gs_seaside-9060: down 19 seconds, normally up
/etc/service/gs_seaside-9061: down 19 seconds, normally up
/etc/service/gs_seaside-9062: down 19 seconds, normally up
/etc/service/gs_seaside-9063: down 19 seconds, normally up
/etc/service/gs_seaside-9064: down 19 seconds, normally up
/etc/service/gs_seaside-9065: down 19 seconds, normally up
/etc/service/gs_statmon-1: down 19 seconds, normally up
/etc/service/gs_statmon-60: down 19 seconds, normally up

When i do the command : ./gemstone_start the system answer:

This script restarts all Gemstone processes.
Shall I continue? (Y/N)
n
Not continuing
scandella@scandella:/srv/gitrepository/GemStone_daemontools_setup/bin$ ./gemstone_start
This script restarts all Gemstone processes.
Shall I continue? (Y/N)
y

startstone[Error]: Stone process (id=3831) has died.
startstone[Error]: Examine '/opt/oodb/gsDevKitHome/gemstone/stones/gestionale/logs/gestionale.log' for more information. Excerpt follows:

A this point i need to kill the gemstone cache task.
After when i restart the system ( ./gemstone_start ) al works fine.

I don't understand because the cache is go into frozen stataus.

Some considerations ?

Thanks,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Gemstone status: cache frozen

James,

So what do you get from ‘gslist -cvl’?

now the ./gemstone_status report :

Status Version Owner Pid Port Started Type Name
------- --------- --------- ----- ----- ------------ ------ ----
OK 3.1.0.6 scandella 4073 53125 giu 08 06:01 Stone gestionale
OK 3.1.0.6 scandella 4075 50387 giu 08 06:01 cache gestionale~9af495ccd1149d82
OK 3.1.0.6 scandella 4058 50377 giu 08 06:01 Netldi gs64ldi

/etc/service/gs_maintenance: up (pid 4140) 35397 seconds
/etc/service/gs_seaside-9060: up (pid 4146) 35397 seconds
/etc/service/gs_seaside-9061: up (pid 4148) 35397 seconds
/etc/service/gs_seaside-9062: up (pid 4150) 35397 seconds
/etc/service/gs_seaside-9063: up (pid 4152) 35397 seconds
/etc/service/gs_seaside-9064: up (pid 4159) 35397 seconds
/etc/service/gs_seaside-9065: up (pid 4168) 35397 seconds
/etc/service/gs_statmon-1: up (pid 4115) 35397 seconds
/etc/service/gs_statmon-60: up (pid 4116) 35397 seconds

Tomorrow the /gemstone_status command report:

Status   Version    Owner    Pid   Port   Started     Type       Name
------- --------- --------- ----- ----- ------------ ------      ----
frozen  3.1.0.6   scandella  1132 46543 giu 04 20:49 cache       gestionale~9af495ccd1149d82

/etc/service/gs_maintenance: down 19 seconds, normally up
/etc/service/gs_seaside-9060: down 19 seconds, normally up
/etc/service/gs_seaside-9061: down 19 seconds, normally up
/etc/service/gs_seaside-9062: down 19 seconds, normally up
/etc/service/gs_seaside-9063: down 19 seconds, normally up
/etc/service/gs_seaside-9064: down 19 seconds, normally up
/etc/service/gs_seaside-9065: down 19 seconds, normally up
/etc/service/gs_statmon-1: down 19 seconds, normally up
/etc/service/gs_statmon-60: down 19 seconds, normally up

As you can see the cache result start: giu 04 20:49.

I infer from this that the system has never completely turned off ( i have a UPS to power the system )

But there are no other active services ( Stone - Netldi )

I don't understund because this inconsistent status.

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Gemstone status: cache frozen

Dario,

Okay it looks like your shrpcmonitor proces is hung. Please pass along the contents of th *1132pcmon.log (the shrpcomonitory log file) as it may have some clues ... The stone long would also be interesting

If you are wanting to get restarted again without waiting for analysis (which is reasonable:), it should be safe to `kill` the 1132 process (don't use -9 unless the process won't go away with a standard kill) ... If you ever use -9 to kill the shrpcmonitor you must manually clean up the shared memory using `ipcs -a` and `ipcrm`... in this case its worth making sure that the shared memory is cleaned up regardless of how you kill the shrpcmonitor.

Once you've killed the frozen shrpcmonitor and cleaned up shared memory, you can restart your stone and things should come up again ... be aware that you might have to/want to restore from logs if the stone crashed before completing a transaction ... If you do a vanilla `startstone` with no args the startstone will refuse to start if the system wasn't cleanly shutdown and the messages should point you in the right direction for recovery options ...

Dale

On 06/08/2015 07:12 AM, Dario Trussardi via Glass wrote:

James,

So what do you get from ‘gslist -cvl’?

now the    ./gemstone_status report :

Status Version Owner Pid Port Started Type Name

------- --------- --------- ----- ----- ------------ ------ ----

OK 3.1.0.6 scandella 4073 53125 giu 08 06:01 Stone gestionale

OK 3.1.0.6 scandella 4075 50387 giu 08 06:01 cache gestionale~9af495ccd1149d82

OK 3.1.0.6 scandella 4058 50377 giu 08 06:01 Netldi gs64ldi

/etc/service/gs_maintenance: up (pid 4140) 35397 seconds

/etc/service/gs_seaside-9060: up (pid 4146) 35397 seconds

/etc/service/gs_seaside-9061: up (pid 4148) 35397 seconds

/etc/service/gs_seaside-9062: up (pid 4150) 35397 seconds

/etc/service/gs_seaside-9063: up (pid 4152) 35397 seconds

/etc/service/gs_seaside-9064: up (pid 4159) 35397 seconds

/etc/service/gs_seaside-9065: up (pid 4168) 35397 seconds

/etc/service/gs_statmon-1: up (pid 4115) 35397 seconds

/etc/service/gs_statmon-60: up (pid 4116) 35397 seconds

Tomorrow the /gemstone_status command report:

Status   Version    Owner    Pid   Port   Started     Type       Name

------- --------- --------- ----- ----- ------------ ------      ----

frozen  3.1.0.6   scandella  1132 46543 giu 04 20:49 cache       gestionale~9af495ccd1149d82

/etc/service/gs_maintenance: down 19 seconds, normally up

/etc/service/gs_seaside-9060: down 19 seconds, normally up

/etc/service/gs_seaside-9061: down 19 seconds, normally up

/etc/service/gs_seaside-9062: down 19 seconds, normally up

/etc/service/gs_seaside-9063: down 19 seconds, normally up

/etc/service/gs_seaside-9064: down 19 seconds, normally up

/etc/service/gs_seaside-9065: down 19 seconds, normally up

/etc/service/gs_statmon-1: down 19 seconds, normally up

/etc/service/gs_statmon-60: down 19 seconds, normally up

As you can see the cache result start: giu 04 20:49.

I infer from this that the system has never completely turned off ( i have a UPS to power the system )

But there are no other active services ( Stone - Netldi )

I don't understund because this inconsistent status.

Dario
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Gemstone status: cache frozen

Dale, James,

Dario,

Okay it looks like your shrpcmonitor proces is hung. Please pass along the contents of th *1132pcmon.log (the shrpcomonitory log file) as it may have some clues ... The stone long would also be interesting

In the rush to restart, i lose the logs file.

What I would like to understand is why the system went into this state.

Because the Stone and Netldi was not report by gslist and cache was frozen ?

What can brought the system in this condition?

At first time i thought the system is shutdown and restart , but i wrong, the frozen cache result start June 04 20:49

This mean the system is not shutdown and restart in the night of June 08, when the system go into inconsistent status.

Thanks for any considerations,

Dario

If you are wanting to get restarted again without waiting for analysis (which is reasonable:), it should be safe to `kill` the 1132 process (don't use -9 unless the process won't go away with a standard kill) ... If you ever use -9 to kill the shrpcmonitor you must manually clean up the shared memory using `ipcs -a` and `ipcrm`... in this case its worth making sure that the shared memory is cleaned up regardless of how you kill the shrpcmonitor.

Once you've killed the frozen shrpcmonitor and cleaned up shared memory, you can restart your stone and things should come up again ... be aware that you might have to/want to restore from logs if the stone crashed before completing a transaction ... If you do a vanilla `startstone` with no args the startstone will refuse to start if the system wasn't cleanly shutdown and the messages should point you in the right direction for recovery options ...

Dale

On 06/08/2015 07:12 AM, Dario Trussardi via Glass wrote:
James,

So what do you get from ‘gslist -cvl’?

now the    ./gemstone_status report :

Status Version Owner Pid Port Started Type Name

------- --------- --------- ----- ----- ------------ ------ ----

OK 3.1.0.6 scandella 4073 53125 giu 08 06:01 Stone gestionale

OK 3.1.0.6 scandella 4075 50387 giu 08 06:01 cache gestionale~9af495ccd1149d82

OK 3.1.0.6 scandella 4058 50377 giu 08 06:01 Netldi gs64ldi

/etc/service/gs_maintenance: up (pid 4140) 35397 seconds

/etc/service/gs_seaside-9060: up (pid 4146) 35397 seconds

/etc/service/gs_seaside-9061: up (pid 4148) 35397 seconds

/etc/service/gs_seaside-9062: up (pid 4150) 35397 seconds

/etc/service/gs_seaside-9063: up (pid 4152) 35397 seconds

/etc/service/gs_seaside-9064: up (pid 4159) 35397 seconds

/etc/service/gs_seaside-9065: up (pid 4168) 35397 seconds

/etc/service/gs_statmon-1: up (pid 4115) 35397 seconds

/etc/service/gs_statmon-60: up (pid 4116) 35397 seconds

Tomorrow the /gemstone_status command report:

Status   Version    Owner    Pid   Port   Started     Type       Name

------- --------- --------- ----- ----- ------------ ------      ----

frozen  3.1.0.6   scandella  1132 46543 giu 04 20:49 cache       gestionale~9af495ccd1149d82

/etc/service/gs_maintenance: down 19 seconds, normally up

/etc/service/gs_seaside-9060: down 19 seconds, normally up

/etc/service/gs_seaside-9061: down 19 seconds, normally up

/etc/service/gs_seaside-9062: down 19 seconds, normally up

/etc/service/gs_seaside-9063: down 19 seconds, normally up

/etc/service/gs_seaside-9064: down 19 seconds, normally up

/etc/service/gs_seaside-9065: down 19 seconds, normally up

/etc/service/gs_statmon-1: down 19 seconds, normally up

/etc/service/gs_statmon-60: down 19 seconds, normally up

As you can see the cache result start: giu 04 20:49.

I infer from this that the system has never completely turned off ( i have a UPS to power the system )

But there are no other active services ( Stone - Netldi )

I don't understund because this inconsistent status.

Dario
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Gemstone status: cache frozen

Dario.

Without any logs it is not possible to guess what might have happened ... with the `gslist` only listing the shpcmonitor process, it does mean that the stone itself was no longer running. Whether the stone process stopped because of a problem with the shrpcmonitor or had crashed itself is only knowable if we can see the logs.

Dale

On 06/09/2015 03:14 AM, Dario Trussardi via Glass wrote:

Dale, James,
Dario,

Okay it looks like your shrpcmonitor proces is hung. Please pass along the contents of th *1132pcmon.log (the shrpcomonitory log file) as it may have some clues ... The stone long would also be interesting

In the rush to restart, i lose the logs file.

What I would like to understand is why the system went into this state.

Because the Stone and Netldi was not report by gslist and cache was frozen ?

What can brought the system in this condition?

At first time i thought the system is shutdown and restart , but i wrong, the frozen cache result start June 04 20:49

This mean the system is not shutdown and restart in the night of June 08, when the system go into inconsistent status.

Thanks for any considerations,

Dario
If you are wanting to get restarted again without waiting for analysis (which is reasonable:), it should be safe to `kill` the 1132 process (don't use -9 unless the process won't go away with a standard kill) ... If you ever use -9 to kill the shrpcmonitor you must manually clean up the shared memory using `ipcs -a` and `ipcrm`... in this case its worth making sure that the shared memory is cleaned up regardless of how you kill the shrpcmonitor.

Once you've killed the frozen shrpcmonitor and cleaned up shared memory, you can restart your stone and things should come up again ... be aware that you might have to/want to restore from logs if the stone crashed before completing a transaction ... If you do a vanilla `startstone` with no args the startstone will refuse to start if the system wasn't cleanly shutdown and the messages should point you in the right direction for recovery options ...

Dale

On 06/08/2015 07:12 AM, Dario Trussardi via Glass wrote:
James,

So what do you get from ‘gslist -cvl’?

now the    ./gemstone_status report :

Status Version Owner Pid Port Started Type Name

------- --------- --------- ----- ----- ------------ ------ ----

OK 3.1.0.6 scandella 4073 53125 giu 08 06:01 Stone gestionale

OK 3.1.0.6 scandella 4075 50387 giu 08 06:01 cache gestionale~9af495ccd1149d82

OK 3.1.0.6 scandella 4058 50377 giu 08 06:01 Netldi gs64ldi

/etc/service/gs_maintenance: up (pid 4140) 35397 seconds

/etc/service/gs_seaside-9060: up (pid 4146) 35397 seconds

/etc/service/gs_seaside-9061: up (pid 4148) 35397 seconds

/etc/service/gs_seaside-9062: up (pid 4150) 35397 seconds

/etc/service/gs_seaside-9063: up (pid 4152) 35397 seconds

/etc/service/gs_seaside-9064: up (pid 4159) 35397 seconds

/etc/service/gs_seaside-9065: up (pid 4168) 35397 seconds

/etc/service/gs_statmon-1: up (pid 4115) 35397 seconds

/etc/service/gs_statmon-60: up (pid 4116) 35397 seconds

Tomorrow the /gemstone_status command report:

Status   Version    Owner    Pid   Port   Started     Type       Name

------- --------- --------- ----- ----- ------------ ------      ----

frozen  3.1.0.6   scandella  1132 46543 giu 04 20:49 cache       gestionale~9af495ccd1149d82

/etc/service/gs_maintenance: down 19 seconds, normally up

/etc/service/gs_seaside-9060: down 19 seconds, normally up

/etc/service/gs_seaside-9061: down 19 seconds, normally up

/etc/service/gs_seaside-9062: down 19 seconds, normally up

/etc/service/gs_seaside-9063: down 19 seconds, normally up

/etc/service/gs_seaside-9064: down 19 seconds, normally up

/etc/service/gs_seaside-9065: down 19 seconds, normally up

/etc/service/gs_statmon-1: down 19 seconds, normally up

/etc/service/gs_statmon-60: down 19 seconds, normally up

As you can see the cache result start: giu 04 20:49.

I infer from this that the system has never completely turned off ( i have a UPS to power the system )

But there are no other active services ( Stone - Netldi )

I don't understund because this inconsistent status.

Dario
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass