[Glass] Nested transactions and 'CorruptObj error', or: how to abort in GLASS?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[Glass] Nested transactions and 'CorruptObj error', or: how to abort in GLASS?

Pieter Nagel-3
TLDR: how is a GLASS app supposed to handling aborting?

Our biggest headache porting from GS64 2.4 to 3.1.0.4 relates to aborting.
 Specifically, we have code that does a batch of operations, bulks up all
error messages, and if there were any errors, wants to abort all side
effects of the batch and show an error message.

Currently, under 2.4, "System rollbackDirtyList: System dirtyListId" is
used to abort. This was done (apparently on the recommendation of this
mailing list) as a work-around for the fact that "System abortTransaction"
breaks horribly - it seems to cause the Seaside machinery and adaptors to
lose state about the very request that is being handled. But "System
rollbackDirtyList" no longer exists in 3.1.

The only "blessed" infrastructure for aborting that seems to be provided
in GLASS on 2.4 is SafelyPerformBlockRequiringAbort. But the block that
one can "safely perform" is called outside of the Seaside request handling
stack, so one can't do any interesting processing like call:'ing a new
component that shows an error message. Also, it is hard-wired to retry the
same request again, where in our case that would just lead to re-doing the
same batch of operations that would just lead to the same errors and the
same need to abort again.

GS 3.1 introduces nested transactions, which on the surface would seem to
be the ideal mechanism to handle these kinds of requirements.

Ideally, the GLASS infrastructure would enter a nested transaction at a
suitable boundary before running end-user code, so an application can just
abort its own subtransaction if it wants to.

We first went the route of just wrapping our batch process in a block that
enters a nested transaction, aborts on exceptions (and re-raises the
exception, or commits otherwise. But this causes horrible Heisenbugs,
where at best the current session becomes unusable, and at worst the stone
file is corrupt.

The problems we're having seem similar to the fixed OBJ_ERR_CORRUPT_OBJ
bugs mentioned in the 3.1.0.5 release notes (even though not explicitly
saying that they are related to nested transaction), so I'm first going to
take that for a spin.

But in the meantime: is our intended use of subtransactions within the
scope of "sanctioned behaviour" on GLASS?


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Nested transactions and 'CorruptObj error', or: how to abort in GLASS?

Pieter Nagel-3
Still no luck with nested transactions on 3.1.0.5:

---
[1] AbstractException >> _signalFromPrimitive: (envId 0)
        handleInCextensionBool: nil
        num: nil
        res: nil
        .t1: nil
        .t2: a InternalError occurred (error 2261), The object with object ID
aGsMethodLookupCache( 284640->aGsNativeCode) is corrupt. Reason: 'attempt
to commit a hidden object'
        receiver: a InternalError occurred (error 2261), The object with object
ID aGsMethodLookupCache( 284640->aGsNativeCode) is corrupt. Reason:
'attempt to commit a hidden object'
[2] System class >> _primitiveCommit: (envId 0)
        commitMode: 0
        receiver: System
[3] System class >> __commit: (envId 0)
        commitMode: 0
        receiver: System
[4] [] in  System class >> _localCommit: (envId 0)
        actualMode: 0
        self: System
        receiver: System
[5] ExecBlock >> onException:do: (envId 0)
        anException: Error
        handlerBlock: anExecBlock1
        receiver: System
[6] System class >> _localCommit: (envId 0)
        commitMode: 0
        commitResult: nil
        actualMode: 0
        self: System
        .t1: anExecBlock0
        .t2: Error
        .t3: anExecBlock1
        receiver: System
[7] TransactionBoundaryDefaultPolicy >> commit: (envId 0)
        commitMode: 0
        res: nil
        .t1: System
        .t2: 0
        receiver: aSessionMethodTransactionBoundaryPolicy
[8] System class >> _commit: (envId 0)
        commitMode: 0
        coordinator: aSessionMethodTransactionBoundaryPolicy
        .t1: aSessionMethodTransactionBoundaryPolicy
        .t2: 0
        receiver: System
[9] System class >> commitTransaction (envId 0)
        receiver: System
[10] OBGemStonePlatform class >> doAutoCommit (envId 0)
        result: nil
        .t1: System
        receiver: OBGemStonePlatform
[11] JadeServer >> evaluate:inContext: (envId 0)
        aString: 'WonkaTestRunner runDomainTests.
'
        anObject: nil
        result: true
        .t1: OBGemStonePlatform
        receiver: aJadeServer
[12] JadeServer >> printIt:in: (envId 0)
        aString: 'WonkaTestRunner runDomainTests.
'
        aContext: nil
        receiver: aJadeServer
[13] GsNMethod class >> _gsReturnToC (envId 0)
        receiver: nil
---

This happens as part of the auto-commit that happens right at the end of
"print it"'ing the code that runs our entire unit test, so it is hard to
point a finger at which one of the many tests called caused the initial
corruption (except to say this does not happen when not using nested
transaction).

I've seen the above error many times under 3.1.0.4 as well.


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Nested transactions and 'CorruptObj error', or: how to abort in GLASS?

Dale Henrichs-3
Pieter,

We'd like to find out more information about the failure that you are seeing ... specifically we'd like to start by getting a C stack at the point of the error. If you could add:

  GEM_HALT_ON_ERROR = 2261;

to your system.conf (or gem.conf) and then restart your seaside gems and keep an eye on your gem log file ... 

When you hit the error, you should get a c and smalltalk stacks dumped to the log file so send us a copy of the log file and go from there ...

Basically, you shouldn't be able to reference a GsMethodLookupCache from persistent state, so we're interesting in finding out how it's sneaking in ...

Dale


On Wed, Mar 19, 2014 at 5:08 AM, Pieter Nagel <[hidden email]> wrote:
Still no luck with nested transactions on 3.1.0.5:

---
[1] AbstractException >> _signalFromPrimitive: (envId 0)
        handleInCextensionBool: nil
        num: nil
        res: nil
        .t1: nil
        .t2: a InternalError occurred (error 2261), The object with object ID
aGsMethodLookupCache( 284640->aGsNativeCode) is corrupt. Reason: 'attempt
to commit a hidden object'
        receiver: a InternalError occurred (error 2261), The object with object
ID aGsMethodLookupCache( 284640->aGsNativeCode) is corrupt. Reason:
'attempt to commit a hidden object'
[2] System class >> _primitiveCommit: (envId 0)
        commitMode: 0
        receiver: System
[3] System class >> __commit: (envId 0)
        commitMode: 0
        receiver: System
[4] [] in  System class >> _localCommit: (envId 0)
        actualMode: 0
        self: System
        receiver: System
[5] ExecBlock >> onException:do: (envId 0)
        anException: Error
        handlerBlock: anExecBlock1
        receiver: System
[6] System class >> _localCommit: (envId 0)
        commitMode: 0
        commitResult: nil
        actualMode: 0
        self: System
        .t1: anExecBlock0
        .t2: Error
        .t3: anExecBlock1
        receiver: System
[7] TransactionBoundaryDefaultPolicy >> commit: (envId 0)
        commitMode: 0
        res: nil
        .t1: System
        .t2: 0
        receiver: aSessionMethodTransactionBoundaryPolicy
[8] System class >> _commit: (envId 0)
        commitMode: 0
        coordinator: aSessionMethodTransactionBoundaryPolicy
        .t1: aSessionMethodTransactionBoundaryPolicy
        .t2: 0
        receiver: System
[9] System class >> commitTransaction (envId 0)
        receiver: System
[10] OBGemStonePlatform class >> doAutoCommit (envId 0)
        result: nil
        .t1: System
        receiver: OBGemStonePlatform
[11] JadeServer >> evaluate:inContext: (envId 0)
        aString: 'WonkaTestRunner runDomainTests.
'
        anObject: nil
        result: true
        .t1: OBGemStonePlatform
        receiver: aJadeServer
[12] JadeServer >> printIt:in: (envId 0)
        aString: 'WonkaTestRunner runDomainTests.
'
        aContext: nil
        receiver: aJadeServer
[13] GsNMethod class >> _gsReturnToC (envId 0)
        receiver: nil
---

This happens as part of the auto-commit that happens right at the end of
"print it"'ing the code that runs our entire unit test, so it is hard to
point a finger at which one of the many tests called caused the initial
corruption (except to say this does not happen when not using nested
transaction).

I've seen the above error many times under 3.1.0.4 as well.


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Nested transactions and 'CorruptObj error', or: how to abort in GLASS?

Dale Henrichs-3
In reply to this post by Pieter Nagel-3
I've addressed the corruptObj issues in a separate mail, so I'll just comment below ...


On Wed, Mar 19, 2014 at 4:11 AM, Pieter Nagel <[hidden email]> wrote:
TLDR: how is a GLASS app supposed to handling aborting?

Our biggest headache porting from GS64 2.4 to 3.1.0.4 relates to aborting.
 Specifically, we have code that does a batch of operations, bulks up all
error messages, and if there were any errors, wants to abort all side
effects of the batch and show an error message.

Yes doing an abort "out-of-school" in the seaside stack is not really a good idea ...  commits  aren't a real good idea, so it is best to avoid transactions altogether ...

Currently, under 2.4, "System rollbackDirtyList: System dirtyListId" is
used to abort. This was done (apparently on the recommendation of this
mailing list) as a work-around for the fact that "System abortTransaction"
breaks horribly - it seems to cause the Seaside machinery and adaptors to
lose state about the very request that is being handled. But "System
rollbackDirtyList" no longer exists in 3.1.

It was a bit of a sticky wicket to properly track all of the dirty object transitions using rollbackDirtyList: and I do think that if you must manage persistent state in the Seaside stack that nested transactions should be a better way to go ... when I get out from under 3.2, I'd like to start looking into some generic support ... if possible ...


The only "blessed" infrastructure for aborting that seems to be provided
in GLASS on 2.4 is SafelyPerformBlockRequiringAbort. But the block that
one can "safely perform" is called outside of the Seaside request handling
stack, so one can't do any interesting processing like call:'ing a new
component that shows an error message. Also, it is hard-wired to retry the
same request again, where in our case that would just lead to re-doing the
same batch of operations that would just lead to the same errors and the
same need to abort again.

GS 3.1 introduces nested transactions, which on the surface would seem to
be the ideal mechanism to handle these kinds of requirements.

Ideally, the GLASS infrastructure would enter a nested transaction at a
suitable boundary before running end-user code, so an application can just
abort its own subtransaction if it wants to.

Still have to be real careful about the possible transitions between session state and user state so I'd be inclined to not offer it as a standard capability ... avoiding all transactions is best ... but in the use cases where a user is willing to manage things a bit closer, I think it can find some uses ...
 

We first went the route of just wrapping our batch process in a block that
enters a nested transaction, aborts on exceptions (and re-raises the
exception, or commits otherwise. But this causes horrible Heisenbugs,
where at best the current session becomes unusable, and at worst the stone
file is corrupt.

I'd be interested in getting more details about where you are putting the transaction boundaries (the ones that "work" and the ones that "don't work" ... the silent unusable are the most interesting ... 

The problems we're having seem similar to the fixed OBJ_ERR_CORRUPT_OBJ
bugs mentioned in the 3.1.0.5 release notes (even though not explicitly
saying that they are related to nested transaction), so I'm first going to
take that for a spin.

Just an FYI, but the corrupt_obj error messages do not always refer to persistent object corruption .. very often this type of error message  is used when incorrect VM state is detected and the error is intended to render the session useless so that you aren't able to persist corrupt objects ...

But in the meantime: is our intended use of subtransactions within the
scope of "sanctioned behaviour" on GLASS?

I'll answer with a "cautious yes"  ... the trick is to find the proper nesting for the transactions ... I can imagine scenarios where seaside is building up some persistent state and temp state where the temp state is hooked into the persistent state at a later point in the stack ... an abort at the wrong spot could easily create a discontinuity between the persistent state and temp state which would lead to nastiness ... Ideally the subtransaction would wrap ONLY your user data with no seaside session state involved at all ... that way an abort will only rollback your data and leave the session state alone ...

Dale


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Nested transactions and 'CorruptObj error', or: how to abort in GLASS?

Pieter Nagel-3
Dale,

I'll get back to you on the C stacktrace when I'm back on site next
week, if the client is willing to sink more time into this (there's talk
of getting around the problem by ditching transactions and following the
"look before you leap" idiom, so the code can be semantically equivalent
under Pharo, which would have value to us in and of itself).

However:

On Wed, 2014-03-19 at 13:47 -0700, Dale Henrichs wrote:

> Yes doing an abort "out-of-school" in the seaside stack is not really
a good idea ...  commits  aren't a real good idea, so it is best to
avoid transactions altogether ...

and

>  ... I'd be inclined to not offer [transactions] as a standard
capability ... avoiding all transactions is best...

I find this state most perplexing.

The whole point of running a Seaside application in GemStone, as opposed
to, say, Pharo, is that GemStone offers persistence. And transactions
are a large part of persistence, so saying "it is best to avoid
transactions altogether" negates a large part of the value proposition
that GemStone offers in the first place.

Frankly, I would expect GemStone to *promote* the use of transactions,
to further leverage what differentiates you from other options. If there
are tricky implementations details, I'd expect you to rather steer
people towards using transactions at the right level than pinning a
"caveat emptor" disclaimer on them.

--
Pieter Nagel






_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Nested transactions and 'CorruptObj error', or: how to abort in GLASS?

Johan Brichau-3
Pieter,

The GLASS setup makes the use of transactions transparant. That means you can run a Seaside application just as if you would be relying on image persistence.
imho, that makes for a great out-of-the-box experience. Of course, that also means that when you want to control the transactions, you need to understand how the transparant persistency works.

Once you have more specific requirements, it is not hard to find out where Seaside in GLASS is wrapping the transactions and work with them.
That is exactly what we do using the DALi framework [1,2] for Yesplan. It was designed to abstract the differences between how we can do transactions with OODBs like GOODS in Pharo and the 'transaction-per-request' approach in GLASS.

Some of the things you describe fit with how we work with DALi. Things like aborting a transaction and popping up an error message or 'lightweight' nested transactions (i.e. tracking changes to our own objects). However, I think I will need to write up more elaborate documentation about it to make it more usable for outsiders. Also, it does not fit all cases you describe and you need to build the application with the DALI framework in mind, so I have never been sure if it was really applicable to others, but it would seem so....

Johan

[1] http://www.esug.org/wiki/pier/Conferences/2011/Schedule-And-Talks/Dali
[2] http://ss3.gemstone.com/ss/DALi.html/Overview

On 20 Mar 2014, at 14:26, Pieter Nagel <[hidden email]> wrote:

> I find this state most perplexing.
>
> The whole point of running a Seaside application in GemStone, as opposed
> to, say, Pharo, is that GemStone offers persistence. And transactions
> are a large part of persistence, so saying "it is best to avoid
> transactions altogether" negates a large part of the value proposition
> that GemStone offers in the first place.
>
> Frankly, I would expect GemStone to *promote* the use of transactions,
> to further leverage what differentiates you from other options. If there
> are tricky implementations details, I'd expect you to rather steer
> people towards using transactions at the right level than pinning a
> "caveat emptor" disclaimer on them.

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Nested transactions and 'CorruptObj error', or: how to abort in GLASS?

Pieter Nagel-3
In reply to this post by Pieter Nagel-3
I'm happy to report that our problems with aborting under Seaside in GS3
are now solved, thanks to advice that I received off-list from Michael
Veigl.

For the sake of others with this question who may search the archive, I
reproduce his email below, with his permission:

--- snip ---

Hello Pieter,

We are in the same situation that out of a couple of subsequent actions
one might
fail and every change done before must be rolled back to avoid
inconsistencies.
Maybe our approach can help you, it is similar to yours but does not use
native
nested transactions.
As you know aborting from within a seaside application is not recommended,
however
committing "carefully" should not do harm. Thus we split up execution of
our domain
code into two steps. You could as well say, we wrap our code into some
transaction
handler.
So the steps are these:

1. Do a commit
We assume that a commit prior to any changes generated by our domain logic
will not
lead to inconsistencies within continuations and other seaside specific data.
Such a commit however should be checked for conflicts already at this
time. Our code
fragment looks like this:
        System _validateTransaction <= 0 ifFalse: [WARetryHttpRequest
signal: 'Transaction
would not commit, retry'].
        System commitTransaction.
        System inTransaction ifFalse: [System beginTransaction]

2. Perform your domain specific code and check the result
        a. In case of an error do an abort:
                System inTransaction ifTrue: [System abortTransaction].
                System beginTransaction
        The abort is safe because seaside state has been saved before
        b. If everything goes well just continue, the seaside framework
will commit later on

It is worth mentioning that you have to guarantee the session being in a
transaction
when leaving your domain code, otherwise seaside's own commit would fail.
In 2.a.
#beginTransaction could be executed conditionally depending on the session
being in
transaction or not.

I hope you get the idea and it works out for you.

Good luck,
Michael Veigl

--- snip ---

The only thing I changed is, I do not call _validateTransaction before
commitTransaction. This reads to me like a potential race condition: what
if between the validate and commit something happens that will make the
commit fail?

Instead, I call commitTransaction and then do the WARetryHttpRequest in
case of failure, since commitTransaction will then either commit or fail
atomically.

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Nested transactions and 'CorruptObj error', or: how to abort in GLASS?

Pieter Nagel-3
In reply to this post by Dale Henrichs-3
> Pieter,
>
> We'd like to find out more information about the failure that you are
> seeing ... specifically we'd like to start by getting a C stack at the
> point of the error. If you could add:

Regrettably I am unable to produce the C stack for you, since the client
is not willing to spend more time on this and we already found another
solution not involving nested transactions.

Thanks for the help.

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Nested transactions and 'CorruptObj error', or: how to abort in GLASS?

Dale Henrichs-3
In reply to this post by Pieter Nagel-3
Pieter,

I'm glad you have found a satisfactory solution! 

Thanks Michael!

Dale


On Tue, Mar 25, 2014 at 7:14 AM, Pieter Nagel <[hidden email]> wrote:
I'm happy to report that our problems with aborting under Seaside in GS3
are now solved, thanks to advice that I received off-list from Michael
Veigl.

For the sake of others with this question who may search the archive, I
reproduce his email below, with his permission:

--- snip ---

Hello Pieter,

We are in the same situation that out of a couple of subsequent actions
one might
fail and every change done before must be rolled back to avoid
inconsistencies.
Maybe our approach can help you, it is similar to yours but does not use
native
nested transactions.
As you know aborting from within a seaside application is not recommended,
however
committing "carefully" should not do harm. Thus we split up execution of
our domain
code into two steps. You could as well say, we wrap our code into some
transaction
handler.
So the steps are these:

1. Do a commit
We assume that a commit prior to any changes generated by our domain logic
will not
lead to inconsistencies within continuations and other seaside specific data.
Such a commit however should be checked for conflicts already at this
time. Our code
fragment looks like this:
        System _validateTransaction <= 0 ifFalse: [WARetryHttpRequest
signal: 'Transaction
would not commit, retry'].
        System commitTransaction.
        System inTransaction ifFalse: [System beginTransaction]

2. Perform your domain specific code and check the result
        a. In case of an error do an abort:
                System inTransaction ifTrue: [System abortTransaction].
                System beginTransaction
        The abort is safe because seaside state has been saved before
        b. If everything goes well just continue, the seaside framework
will commit later on

It is worth mentioning that you have to guarantee the session being in a
transaction
when leaving your domain code, otherwise seaside's own commit would fail.
In 2.a.
#beginTransaction could be executed conditionally depending on the session
being in
transaction or not.

I hope you get the idea and it works out for you.

Good luck,
Michael Veigl

--- snip ---

The only thing I changed is, I do not call _validateTransaction before
commitTransaction. This reads to me like a potential race condition: what
if between the validate and commit something happens that will make the
commit fail?

Instead, I call commitTransaction and then do the WARetryHttpRequest in
case of failure, since commitTransaction will then either commit or fail
atomically.

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass