Transaction Logs so large ....

One of the most surprising things I noticed with my application is the tremendous storage need of my transaction logs.

Whenever I test my application (with around 200 clients) I get one transaction log file within 4 minutes - each with a size of 1 GB. That would lead to 120GB transaction logs within one days with 8 hours working time.

And of course I do full transaction log .... the database size after doing a backup is 3.4 GB. Normally the clients do not do so much work - but they send lots of events ...

I looked at the inventory and I did not find any strange things ... any ideas how to get an idea why such an amount of space is needed for the transaction logs ....

How do classes like Set, Arrays and changes to them results in transaction logs changes ??? I prefer to use Arrays at this moment ...

Ok, I have to make this perhaps a little bit clearer:

In my test cases I have 200 to 400 clients (written in Python) playing their role as client simulators. They do their requests via http/rest against Gemstone/s against 8 responding gem/zinc tasks (running with different configurations). They produce in total (10-20) events per seconds against the database. These events are changing or adding data - no deletion at this point.

In terms of speed Gemstone has no problem answering the requests (even with only one answering tasks it would work).

The current database file is around 64 GB of size.

The size of the uncompressed backup file is around 4 GB.

As I mentioned in my older postings: this system produces 1GB transaction log files within 4-5 minutes.Therefore we have around 100GB transaction data each day ... wow.

This seems to indicate, that not the extents files will be a problem, but the management of the transaction files is far more difficult to handle (and I was not aware of this :-))).

Question to the more experienced Gemstone/S developers: is this ok (a general pattern) ?

Marten Feldtmann via Glass <[hidden email]> hat am 22. Oktober 2016 um 22:08 geschrieben:

One of the most surprising things I noticed with my application is the tremendous storage need of my transaction logs.

Whenever I test my application (with around 200 clients) I get one transaction log file within 4 minutes - each with a size of 1 GB. That would lead to 120GB transaction logs within one days with 8 hours working time.

And of course I do full transaction log .... the database size after doing a backup is 3.4 GB. Normally the clients do not do so much work - but they send lots of events ...

I looked at the inventory and I did not find any strange things ... any ideas how to get an idea why such an amount of space is needed for the transaction logs ....

How do classes like Set, Arrays and changes to them results in transaction logs changes ??? I prefer to use Arrays at this moment ...

_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass

Tranlog files at first have transaction sequence not yet represented in the random-access database extent files. Sequential files are faster to update and changes can be replayed to extents of different ages (like from restore or cloning). Tranlog files that only contain transactions older than the last Checkpoint Commit can be archived. A checkpoint commit is the point where extents now contain the tranlog changes. You do not need to keep full tranlog files older than the checkpoint commit record in the extents backup that you may wish restore. GS DBAs write shell scripts to manage these things, knowing what tranlog files are no longer useful to keep.

If you already know this stuff then your question would be related to checkpoint commits not happening in a timely manner. How that can happen and how to improve that is a much deeper discussion, but is usually remedied by changing gems to not stay with old views of the database. That can mean more frequent aborts for example, and starting a transaction a short time before making changes to commit. Just one gem staying in-transaction for a long time can hold up checkpoint commits. A developer logged into (and in-transaction) a very active database can hold up checkpoints causing a backlog of transactions not yet applied to extent files.

Paul Baumann

For information purposes: We have no Seaside in this system ... "only" zinc tasks serving rest requests.

Marten

Otto Behrens <[hidden email]> hat am 24. Oktober 2016 um 11:20 geschrieben:


I suspect it is the persistence of the seaside continuations, with all
the temps and things that go with it. We have the same problem; our
number of users are picking up.

I believe that there is a way to make the continuations transient,
which we have not tried.

We load balance across gemstone sessions, which may influence what you do.

On Mon, Oct 24, 2016 at 10:48 AM, Marten Feldtmann via Glass
<[hidden email]> wrote:

Ok, I have to make this perhaps a little bit clearer:

In my test cases I have 200 to 400 clients (written in Python) playing their
role as client simulators. They do their requests via http/rest against
Gemstone/s against 8 responding gem/zinc tasks (running with different
configurations). They produce in total (10-20) events per seconds against
the database. These events are changing or adding data - no deletion at this
point.

In terms of speed Gemstone has no problem answering the requests (even with
only one answering tasks it would work).

The current database file is around 64 GB of size.

The size of the uncompressed backup file is around 4 GB.

As I mentioned in my older postings: this system produces 1GB transaction
log files within 4-5 minutes.Therefore we have around 100GB transaction data
each day ... wow.

This seems to indicate, that not the extents files will be a problem, but
the management of the transaction files is far more difficult to handle (and
I was not aware of this :-))).

Question to the more experienced Gemstone/S developers: is this ok (a
general pattern) ?


Marten Feldtmann via Glass <[hidden email]> hat am 22.
Oktober 2016 um 22:08 geschrieben:


One of the most surprising things I noticed with my application is the
tremendous storage need of my transaction logs.

Whenever I test my application (with around 200 clients) I get one
transaction log file within 4 minutes - each with a size of 1 GB. That would
lead to 120GB transaction logs within one days with 8 hours working time.


And of course I do full transaction log .... the database size after doing a
backup is 3.4 GB. Normally the clients do not do so much work - but they
send lots of events ...

I looked at the inventory and I did not find any strange things ... any
ideas how to get an idea why such an amount of space is needed for the
transaction logs ....

How do classes like Set, Arrays and changes to them results in transaction
logs changes ??? I prefer to use Arrays at this moment ...




_______________________________________________ Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

>

> "more commits than what a gem can react to"

Difficulties like that can happen with continueTransaction, not aborts nor beginTransaction. That said, continueTransaction is usually what a gem is coded to try first when it receives a SigAbort.

Write code so that idle gems abort/begin every so often (like 15 seconds) without having to be signaled. Aborts are low cost. Do an abort when the request comes in (before processing it). Busy gems that are well done should never be in transaction long enough to get a SigAbort.

SigAborts only (normally) go to the gems holding the oldest transaction and time is normally allowed for them to respond. Other gems may be holding a transaction view started a millisecond after those and they would not (normally) be signaled. A trick is to write code to proactively signal more gems than normally would be so that a 45 second allowed response time is able to gain more than a millisecond before the next oldest get signaled. Again though, good code doesn't need to be signaled.

Do you get many commit conflicts? What is your strategy for handling them? Load balancing efforts can sometimes cause more work resolving conflicts. A common strategy is to have many gems feed work to an RCQueue that a single "manager" gem processes without conflicts. Of course then changes are not immediately visible to the requesting gem and more otherwise only-transient objects become persistent.

Paul Baumann

Concerning transactions: Well, all Topaz scripts (starting the Zinc server) are started in #manualBegin transactionMode. If not in that state the handling of SigAbort handlers is more difficult.

Other than that: after an incoming request a BeginTransaction is done, the work is done and a commit or abort is then done ... then the gem is again out of an transaction - and then it should be no candidate for SigAbort handler (I'm not sure of that last statement).

Conflicts - yes initially lots of. After that we introduce retries on the server side and now the conflicts are not very often - though the external programmers should always consider this ( ... and its not done).

Other conflicts can not be solved in this way and need a different API behaviour (normally implemented with RcQueue and worker topaz processes on the server side). The offered API then returns a TASK id and the programmer has to watch for its task ... long-term logic tasks can be programmed in a conflict-free way then on the server side.

For logic conflicts we have a revision counter in the objects.

(We use an API-HTTP-REST-RPC oriented way of programming the Gemstone/S database when interacting via C#, Python or Javascript with the database).

Paul Baumann <[hidden email]> hat am 24. Oktober 2016 um 14:37 geschrieben:

> "more commits than what a gem can react to"

Difficulties like that can happen with continueTransaction, not aborts nor beginTransaction. That said, continueTransaction is usually what a gem is coded to try first when it receives a SigAbort.

Write code so that idle gems abort/begin every so often (like 15 seconds) without having to be signaled. Aborts are low cost. Do an abort when the request comes in (before processing it). Busy gems that are well done should never be in transaction long enough to get a SigAbort.

SigAborts only (normally) go to the gems holding the oldest transaction and time is normally allowed for them to respond. Other gems may be holding a transaction view started a millisecond after those and they would not (normally) be signaled. A trick is to write code to proactively signal more gems than normally would be so that a 45 second allowed response time is able to gain more than a millisecond before the next oldest get signaled. Again though, good code doesn't need to be signaled.

Do you get many commit conflicts? What is your strategy for handling them? Load balancing efforts can sometimes cause more work resolving conflicts. A common strategy is to have many gems feed work to an RCQueue that a single "manager" gem processes without conflicts. Of course then changes are not immediately visible to the requesting gem and more otherwise only-transient objects become persistent.

Paul Baumann

On Oct 24, 2016 7:39 AM, "Otto Behrens" <[hidden email]> wrote:

> If you already know this stuff then your question would be related to
> checkpoint commits not happening in a timely manner. How that can happen and
> how to improve that is a much deeper discussion, but is usually remedied by
> changing gems to not stay with old views of the database. That can mean more
> frequent aborts for example, and starting a transaction a short time before
> making changes to commit. Just one gem staying in-transaction for a long
> time can hold up checkpoint commits. A developer logged into (and
> in-transaction) a very active database can hold up checkpoints causing a
> backlog of transactions not yet applied to extent files.

I understand that if there are gems holding onto old views with other
gems committing a lot, it may happen that more garbage than usual will
end up in the extent and the gems will take longer to commit because
they have more work to determine the write set union etc. I suppose if
things start to overflow one could create bigger and bigger
transaction logs because a gem would be committing temporary objects
to the repository.

If one install proper SigAbort handlers for gems then the gems should
be catching them and aborting, unless there are way more commits than
what the gems can react to via sig aborts? Is it in this case that the
condition may occur?

In our application, we start a number of gems that service seaside
requests. These gems (as well as the other gems we run in the
background), all have sig abort handlers. The load balancing also
helps to ensure that the seaside based application distributes work to
idle gems. So in my view, gems should be cooperating well.

In spite of this, we still see 10GB of tranlogs created in a few hours
under low load. (The db is around 23GB).

I suppose the only way to really analyse what's going on is to run
statmonitor and understand the output. From my previous attempts
though, it appears as if the Seaside infrastructure is really heavy on
transaction logs.

Manual begin mode can still refer to an old view of data, it is not the same as transactionless mode where the view is allowed to automatically adjust to show newer changes. You still would benefit from periodic aborts to avoid the signals to abort. Transactionless mode does not get SigAbort.

Paul Baumann

That's very interesting.

This means, that I should introduce my own subclass of ZnServer and its #serveConnectionOn: (or the #noteAcceptWaitTimedOut) should be rewritten to make an empty transaction (with #beginTransaction and #abortTransaction) on a periodical base (15 seconds ?).

Marten

Paul Baumann <[hidden email]> hat am 24. Oktober 2016 um 15:41 geschrieben:

Manual begin mode can still refer to an old view of data, it is not the same as transactionless mode where the view is allowed to automatically adjust to show newer changes. You still would benefit from periodic aborts to avoid the signals to abort. Transactionless mode does not get SigAbort.

Paul Baumann

On Oct 24, 2016 9:13 AM, "Marten Feldtmann" <[hidden email]> wrote:

Concerning transactions: Well, all Topaz scripts (starting the Zinc server) are started in #manualBegin transactionMode. If not in that state the handling of SigAbort handlers is more difficult.

Other than that: after an incoming request a BeginTransaction is done, the work is done and a commit or abort is then done ... then the gem is again out of an transaction - and then it should be no candidate for SigAbort handler (I'm not sure of that last statement).

Conflicts - yes initially lots of. After that we introduce retries on the server side and now the conflicts are not very often - though the external programmers should always consider this ( ... and its not done).

Other conflicts can not be solved in this way and need a different API behaviour (normally implemented with RcQueue and worker topaz processes on the server side). The offered API then returns a TASK id and the programmer has to watch for its task ... long-term logic tasks can be programmed in a conflict-free way then on the server side.

For logic conflicts we have a revision counter in the objects.

(We use an API-HTTP-REST-RPC oriented way of programming the Gemstone/S database when interacting via C#, Python or Javascript with the database).

Paul Baumann <[hidden email]> hat am 24. Oktober 2016 um 14:37 geschrieben:

> "more commits than what a gem can react to"

Difficulties like that can happen with continueTransaction, not aborts nor beginTransaction. That said, continueTransaction is usually what a gem is coded to try first when it receives a SigAbort.

Write code so that idle gems abort/begin every so often (like 15 seconds) without having to be signaled. Aborts are low cost. Do an abort when the request comes in (before processing it). Busy gems that are well done should never be in transaction long enough to get a SigAbort.

SigAborts only (normally) go to the gems holding the oldest transaction and time is normally allowed for them to respond. Other gems may be holding a transaction view started a millisecond after those and they would not (normally) be signaled. A trick is to write code to proactively signal more gems than normally would be so that a 45 second allowed response time is able to gain more than a millisecond before the next oldest get signaled. Again though, good code doesn't need to be signaled.

Do you get many commit conflicts? What is your strategy for handling them? Load balancing efforts can sometimes cause more work resolving conflicts. A common strategy is to have many gems feed work to an RCQueue that a single "manager" gem processes without conflicts. Of course then changes are not immediately visible to the requesting gem and more otherwise only-transient objects become persistent.

Paul Baumann

On Oct 24, 2016 7:39 AM, "Otto Behrens" <[hidden email]> wrote:

> If you already know this stuff then your question would be related to
> checkpoint commits not happening in a timely manner. How that can happen and
> how to improve that is a much deeper discussion, but is usually remedied by
> changing gems to not stay with old views of the database. That can mean more
> frequent aborts for example, and starting a transaction a short time before
> making changes to commit. Just one gem staying in-transaction for a long
> time can hold up checkpoint commits. A developer logged into (and
> in-transaction) a very active database can hold up checkpoints causing a
> backlog of transactions not yet applied to extent files.

I understand that if there are gems holding onto old views with other
gems committing a lot, it may happen that more garbage than usual will
end up in the extent and the gems will take longer to commit because
they have more work to determine the write set union etc. I suppose if
things start to overflow one could create bigger and bigger
transaction logs because a gem would be committing temporary objects
to the repository.

If one install proper SigAbort handlers for gems then the gems should
be catching them and aborting, unless there are way more commits than
what the gems can react to via sig aborts? Is it in this case that the
condition may occur?

In our application, we start a number of gems that service seaside
requests. These gems (as well as the other gems we run in the
background), all have sig abort handlers. The load balancing also
helps to ensure that the seaside based application distributes work to
idle gems. So in my view, gems should be cooperating well.

In spite of this, we still see 10GB of tranlogs created in a few hours
under low load. (The db is around 23GB).

I suppose the only way to really analyse what's going on is to run
statmonitor and understand the output. From my previous attempts
though, it appears as if the Seaside infrastructure is really heavy on
transaction logs.

I recall using several approaches, I'm not sure which I stayed with. It was nine years ago, and I've also addressed this for GBS clients. A server subclass was one, because there was no better hook. I recall there being a facility that could be registered for transaction changes. I recall noting an expiration time some seconds from current. I recall Norm had added (or perhaps optimized) a way to determine view age at my request. I recall the subclass being to add an instvar and because #continueTransaction needed special handling (like it did not notify as the others). Those were ways to know view age in a general way, and hopefully obsolete way for more recent releases. I recall a "busy while" global being updated and checked to defer aborts by a background process (which might have been GBS specific code). Sorry, I'd have to see the code to help with specifics. What you've said sounds familiar though.

Active management of view age relieved serious performance risks (to company operations) that had previously existed with reliance on reactionary SigAbort. Avoid long-held database views. We went from serious backlog alarms triggered several times a day to eventually none at all and quick recovery for any commit record backlog.

Paul Baumann

Of course Dale. I had brought the discussion to other areas. It had sounded like more tranlog files were kept than necessary. I had not touched at all on how to reduce how quickly they grow.

I thought that Marten was asking about large transaction logs and I didn't see that he had trouble with large extents as he is using a Sigabort Handler ...

Dale

Of course Dale. I had brought the discussion to other areas. It had sounded like more tranlog files were kept than necessary. I had not touched at all on how to reduce how quickly they grow.

On Oct 24, 2016 1:54 PM, "Dale Henrichs via Glass" <[hidden email]> wrote:

On 10/24/2016 04:22 AM, Paul Baumann via Glass wrote:

Tranlog files at first have transaction sequence not yet represented in the random-access database extent files. Sequential files are faster to update and changes can be replayed to extents of different ages (like from restore or cloning). Tranlog files that only contain transactions older than the last Checkpoint Commit can be archived. A checkpoint commit is the point where extents now contain the tranlog changes. You do not need to keep full tranlog files older than the checkpoint commit record in the extents backup that you may wish restore. GS DBAs write shell scripts to manage these things, knowing what tranlog files are no longer useful to keep.

If you already know this stuff then your question would be related to checkpoint commits not happening in a timely manner. How that can happen and how to improve that is a much deeper discussion, but is usually remedied by changing gems to not stay with old views of the database. That can mean more frequent aborts for example, and starting a transaction a short time before making changes to commit. Just one gem staying in-transaction for a long time can hold up checkpoint commits. A developer logged into (and in-transaction) a very active database can hold up checkpoints causing a backlog of transactions not yet applied to extent files.

Paul, a CR backlog dos not cause tranlogs to grow ... CR backlog shows up as extent growth not tranlog file growth...

Dale
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass