[Glass] GS 3.1.0.5: Fatal Internal Error, 5015, 15621, IO Error (RepRead Error)

My development repository occasionally gets into a state where the session
dies and GemTools tells me "Please report to your GemStone Administrator:
Fatal Internal Error, 5015, 15621, IO Error" when running certain
Smalltalk expressions or doing a search for strings in source. Same in Topaz.

This persists across restarting the stone, I need to rebuild the extent
from scratch.

The only strange thing I did is "WAFastCGIAdaptor startOn: 8383" which I
then needed to interrupt by pressing ALT-. Thereafter any Smalltalk, even
just evaluating '1' would cause a stacktrace which I hope to reproduce
here when it happens again.

Logfile produced by using GEM_HALT_ON_ERROR = 4126 attached.



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

My development repository occasionally gets into a state where the session
dies and GemTools tells me "Please report to your GemStone Administrator:
Fatal Internal Error, 5015, 15621, IO Error" when running certain
Smalltalk expressions or doing a search for strings in source. Same in Topaz.

This persists across restarting the stone, I need to rebuild the extent
from scratch.

The only strange thing I did is "WAFastCGIAdaptor startOn: 8383" which I
then needed to interrupt by pressing ALT-. Thereafter any Smalltalk, even
just evaluating '1' would cause a stacktrace which I hope to reproduce
here when it happens again.

Logfile produced by using GEM_HALT_ON_ERROR = 4126 attached.



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

These fatal errors are still going on. so far it affects only development
and staging repositories, so we can work around it, but we're growing
increasingly nervous that it might hit production as well.

To recap: repositories are becoming randomly corrupted into a state where
any meaningfull Smalltalk evaluation causes topaz to dump errors like:

RepRead Error hdr.pageId 0 != pageId
14635-----------------------------------------------------
GemStone: Error         Fatal

Please report to your GemStone Administrator:  Fatal Internal Error,

5015, 14635, IO Error
Error Category: 231169 [Internal Fatal] Number: 4126  Arg Count: 3 Context
: 20 exception : 20
Arg 1: 40122
Arg 2: 117082
Arg 3: 20

These bugs seem to be random. Rebuilding a clean development extent will
fail one time and pass the next even though there are no changes to the
code that is being filed in on top a virgin exten0.dbf in the two
situations. Sometimes reboots inbetween seem to be needed for the problem
to go away.

Any advice for getting to the bottom of this?

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

> Pieter,
>
> start with a fresh extent0.seaside.dbf and
> before starting stone , add
>    STN_TRAN_FULL_LOGGING=true;
>    STN_TRAN_LOG_DEBUG_LEVEL=3;
> to the stone config file .
> Then after repository gets corrupted, run pageaudit and
> if errors are similar , send us the extent and the tranlogs
> for analysis (if possible).  Otherwise we will want you to
> run some tranlog searches for us ...

Will do.

But note that we are using /dev/null as STN_TRAN_LOG_DIRECTORIES, so there
will not be any tranlogs for us to send/search through. I sometimes wonder
if using /dev/null in itself may be behind this bug?

Do you recommend taking pains that STN_TRAN_LOG_DEBUG_LEVEL is *not* set
on production repositories?

It really seems this bug only occurs when tranlogging to /dev/null.

It's now two weeks later after switching tranlogs to real directories, and
this bug still has not resurfaced. Used to happen a handful of times a
week.

But on development test-suites etc. run about 3 times faster when
tranlogging to /dev/null.

Is there other way to get debug information on this while still
tranlogging to /dev/null?

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

It really seems this bug only occurs when tranlogging to /dev/null.

It's now two weeks later after switching tranlogs to real directories, and
this bug still has not resurfaced. Used to happen a handful of times a
week.

But on development test-suites etc. run about 3 times faster when
tranlogging to /dev/null.

Is there other way to get debug information on this while still
tranlogging to /dev/null?

Dale,

We're going to upgrade to 3.1.0.6 or 3.2.1 first, and get back to you if
the bug persists.

But note that this bug only ever occured to stones that had never been
shut down at all. In other words, to a pristine extent0 that was fired up,
used to restore or had our code loaded to it, and had not yet the first
shutdown of its lifetime.

Once a stone was broken, not even stopstone could log in to it.
Conversely, any stone that could be shutdown and restarted would
thereafter be totally immune to this bug, in my experience.

So I'm not sure if the shutdown code is involved here at all.