Stone startup delayed by reading

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Stone startup delayed by reading

GLASS mailing list

When I started my stone during the last days i noticed a much longer startup time of my stone. looking through my log-files I noticed the following lines. What happened here ?


...
Totals
------
Repository Size = 110320 Mbytes = 7060480 pages
Free Space = 55955 Mbytes = 3581166 pages
---------------------------------------------------
Started AIO page server, processId 3707
Stone startup delayed by reading 535481 outstanding commit records.
535000 remaining to be read
534500 remaining to be read
534000 remaining to be read
...

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Stone startup delayed by reading

GLASS mailing list
When I have a slow startup is usually after a crash or some stone abnormal shutdown. Basically, at startup it starts in recovery mode, checking/scanning the tranlog files. That's why it takes longer. After it is up, further starts are normal again. 

You can check what I mention in the stone log. 

Cheers,

On Wed, Jun 22, 2016 at 5:03 AM, Marten Feldtmann via Glass <[hidden email]> wrote:

When I started my stone during the last days i noticed a much longer startup time of my stone. looking through my log-files I noticed the following lines. What happened here ?


...
Totals
------
Repository Size = 110320 Mbytes = 7060480 pages
Free Space = 55955 Mbytes = 3581166 pages
---------------------------------------------------
Started AIO page server, processId 3707
Stone startup delayed by reading 535481 outstanding commit records.
535000 remaining to be read
534500 remaining to be read
534000 remaining to be read
...

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass




--

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Stone startup delayed by reading

Richard Sargent
Administrator
GLASS mailing list wrote
When I have a slow startup is usually after a crash or some stone abnormal
shutdown. Basically, at startup it starts in recovery mode,
checking/scanning the tranlog files. That's why it takes longer. After it
is up, further starts are normal again.
Are you asking for it to work differently than it does? I hope not!
After all, if we don't run the recovery process, what state will the database be in? Unknown. Unknown and potentially dangerous.


It's best to run it on a stable machine which isn't inclined to crash and on which you can control the shutdown.


You can check what I mention in the stone log.

Cheers,

On Wed, Jun 22, 2016 at 5:03 AM, Marten Feldtmann via Glass <
[hidden email]> wrote:

> When I started my stone during the last days i noticed a much longer
> startup time of my stone. looking through my log-files I noticed the
> following lines. What happened here ?
>
>
> ...
> Totals
>  ------
>  Repository Size = 110320 Mbytes = 7060480 pages
>  Free Space = 55955 Mbytes = 3581166 pages
>  ---------------------------------------------------
>  Started AIO page server, processId 3707
>  Stone startup delayed by reading 535481 outstanding commit records.
>  535000 remaining to be read
>  534500 remaining to be read
>  534000 remaining to be read
> ...
>
>
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>


--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Stone startup delayed by reading

GLASS mailing list
In reply to this post by GLASS mailing list
No its totally correct with the behaviour. I stopped 3.3 and restarted it (and again and again) and did not get rid of this problem. That was my problem.

Now I did a backup und restore and all is fine now.

I had no crash or something like that.


Von Samsung Mobile gesendet


-------- Ursprüngliche Nachricht --------
Von: Richard Sargent via Glass
Datum:22.06.2016 17:44 (GMT+01:00)
Betreff: Re: [Glass] Stone startup delayed by reading

GLASS mailing list wrote
> When I have a slow startup is usually after a crash or some stone abnormal
> shutdown. Basically, at startup it starts in recovery mode,
> checking/scanning the tranlog files. That's why it takes longer. After it
> is up, further starts are normal again.

Are you asking for it to work differently than it does? I hope not!
After all, if we don't run the recovery process, what state will the
database be in? Unknown. Unknown and potentially dangerous.


It's best to run it on a stable machine which isn't inclined to crash and on
which you can control the shutdown.



> You can check what I mention in the stone log.
>
> Cheers,
>
> On Wed, Jun 22, 2016 at 5:03 AM, Marten Feldtmann via Glass <

> glass@.gemtalksystems

>> wrote:
>
>> When I started my stone during the last days i noticed a much longer
>> startup time of my stone. looking through my log-files I noticed the
>> following lines. What happened here ?
>>
>>
>> ...
>> Totals
>>  ------
>>  Repository Size = 110320 Mbytes = 7060480 pages
>>  Free Space = 55955 Mbytes = 3581166 pages
>>  ---------------------------------------------------
>>  Started AIO page server, processId 3707
>>  Stone startup delayed by reading 535481 outstanding commit records.
>>  535000 remaining to be read
>>  534500 remaining to be read
>>  534000 remaining to be read
>> ...
>>
>>
>> _______________________________________________
>> Glass mailing list
>>

> Glass@.gemtalksystems

>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>
> _______________________________________________
> Glass mailing list

> Glass@.gemtalksystems

> http://lists.gemtalksystems.com/mailman/listinfo/glass





--
View this message in context: http://forum.world.st/Stone-startup-delayed-by-reading-tp4902363p4902503.html
Sent from the GLASS mailing list archive at Nabble.com.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Stone startup delayed by reading

GLASS mailing list
In reply to this post by Richard Sargent


On Wed, Jun 22, 2016 at 12:44 PM, Richard Sargent via Glass <[hidden email]> wrote:
GLASS mailing list wrote
> When I have a slow startup is usually after a crash or some stone abnormal
> shutdown. Basically, at startup it starts in recovery mode,
> checking/scanning the tranlog files. That's why it takes longer. After it
> is up, further starts are normal again.

Are you asking for it to work differently than it does? I hope not!

No!!! I was just explaining Marten a possible reason (that I faced a couple of time) of why the stone would take longer to start/
 
After all, if we don't run the recovery process, what state will the
database be in? Unknown. Unknown and potentially dangerous.


Exactly. Totally agree. 
 

It's best to run it on a stable machine which isn't inclined to crash and on
which you can control the shutdown.



> You can check what I mention in the stone log.
>
> Cheers,
>
> On Wed, Jun 22, 2016 at 5:03 AM, Marten Feldtmann via Glass <

> glass@.gemtalksystems

>> wrote:
>
>> When I started my stone during the last days i noticed a much longer
>> startup time of my stone. looking through my log-files I noticed the
>> following lines. What happened here ?
>>
>>
>> ...
>> Totals
>>  ------
>>  Repository Size = 110320 Mbytes = 7060480 pages
>>  Free Space = 55955 Mbytes = 3581166 pages
>>  ---------------------------------------------------
>>  Started AIO page server, processId 3707
>>  Stone startup delayed by reading 535481 outstanding commit records.
>>  535000 remaining to be read
>>  534500 remaining to be read
>>  534000 remaining to be read
>> ...
>>
>>
>> _______________________________________________
>> Glass mailing list
>>

> Glass@.gemtalksystems

>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>
> _______________________________________________
> Glass mailing list

> Glass@.gemtalksystems

> http://lists.gemtalksystems.com/mailman/listinfo/glass





--
View this message in context: http://forum.world.st/Stone-startup-delayed-by-reading-tp4902363p4902503.html
Sent from the GLASS mailing list archive at Nabble.com.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass



--

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Stone startup delayed by reading

GLASS mailing list
In reply to this post by GLASS mailing list
Marten,

This situation occurs ("Stone startup delayed by reading 535481 outstanding commit records.") when you stop a stone that has an outstanding commit record backlog.

The most common reason for a commit record backlog is to have a development session (topaz, tODE, Jade)  logged in while your system is actively committing. When you shut down a stone, the commit records are written to disk (most of which are probably already on disk) and the stone exits ... when the stone starts up, it has to process all of the outstanding commit records before going into normal operation.

Soooo, if you are seeing this over multiple start/stop stone sessions, then you are apparently creating a _new_ commit record backlog before stopping the stone ...

So I think that you should try to track down the session that is causing the commit record backlog ...

Before we get too far into this, I want to rule out the most common cause: a development session sitting idle while a production/dev system is doing commits.

If you've got this situation, you should stop leaving your development session logged in during heavy commits and or do a `SystemRepository reclaimAll` in your development evironment before stopping the stone and after shutting down all of your headless vms ... be sure to do a commit or abort in the your dev environment before running reclaimAll ..

If you are not leaving a dev environment idle, then your headless servers need to be responding to SigAbort signals and that is done by code that looks something like the following:

System transactionMode: #manualBegin.

Exception
  installStaticException:
    [:ex :cat :num :args |
      "Run the abort in a lowPriority process, since we must acquire the
       transactionMutex."
      [
        GRPlatform current transactionMutex
          critical: [
            GRPlatform current doAbortTransaction ].
        System enableSignaledAbortError.
      ] forkAt: Processor lowestPriority.
    ]
  category: GemStoneError
  number: 6009
  subtype: nil.
System enableSignaledAbortError.
"This thread is needed to handle the SigAbort exception, when the primary
 thread is blocked on an accept. Assuming default 60 second
 STN_GEM_ABORT_TIMEOUT, wake up at 30 second intervals."
[
  [ true ] whileTrue: [ (Delay forSeconds: 30) wait ].
] forkAt: Processor lowestPriority.


If you are not leaving dev environments idle and you have code like the above in your headless vms ... then we need to hunt down the session that is creating a commit record backlog ...

Dale


On 06/22/2016 01:03 AM, Marten Feldtmann via Glass wrote:

When I started my stone during the last days i noticed a much longer startup time of my stone. looking through my log-files I noticed the following lines. What happened here ?


...
Totals
 ------
 Repository Size = 110320 Mbytes = 7060480 pages
 Free Space = 55955 Mbytes = 3581166 pages
 ---------------------------------------------------
 Started AIO page server, processId 3707
 Stone startup delayed by reading 535481 outstanding commit records.
 535000 remaining to be read
 534500 remaining to be read
 534000 remaining to be read
...


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass