Strange behaviour ... too many data and commit failed not due to memory, but conflicts

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Strange behaviour ... too many data and commit failed not due to memory, but conflicts

GLASS mailing list

Today I noticed a behaviour I found strange. In one of our project we had to import address data and my general test case was around100000 addresses and the import into Gemstone/S went without problems (using ONE transaction) - it was a single task (no other applications running agsint the database). It never failed. It simply worked.

Now I talked with a customer and he told me about projects with 300.000 addresses and additional columns (etc). Much more data ...

Ok, I build a test case data set and to my surprise: Gemstone/S was NEVER able to import that data set in ONE transaction. The commit ALWAYS failed. The commit failed but not due to memory problems, but due to conflict problems ! Again no other task was running against this database (all the other REST topaz scripts had nothing to do).

Then I wanted to know, why the commit always failed and I managed to get this output:

failureRead-Write Conflicts...

Write-Write Conflicts...    9749490177

Write-Dependency Conflicts...

Write-ReadLock Conflicts...

Write-WriteLock Conflicts...

Rc-Write-Write Conflicts...

Synchronized-Commit Conflicts...

When I import the same data - but doing a commit after e.g. 50000 addresses, the import is again stable as usual. The test cases uses lots of RcCounter instances (and those values are changed with each address) - but other like that I see no unusual stuff.


I know, I know - no real solid information, but this behaviour seemed strange to me ...


But perhaps someone has a hint ????



Marten


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts

GLASS mailing list
Marten,

I think 9749490177 is the oop of the object with the conflict. You should look up the object (e.g. Object _objectForOop:974949017). Whatever that object is, it will be informative for diagnosing the problem.


In general, it's a good guideline to keep your transactions down to a reasonable size when possible. Of course, there is merit in an "all or nothing" approach with a single transaction.


On Wed, Feb 22, 2017 at 1:23 PM, Marten Feldtmann via Glass <[hidden email]> wrote:

Today I noticed a behaviour I found strange. In one of our project we had to import address data and my general test case was around100000 addresses and the import into Gemstone/S went without problems (using ONE transaction) - it was a single task (no other applications running agsint the database). It never failed. It simply worked.

Now I talked with a customer and he told me about projects with 300.000 addresses and additional columns (etc). Much more data ...

Ok, I build a test case data set and to my surprise: Gemstone/S was NEVER able to import that data set in ONE transaction. The commit ALWAYS failed. The commit failed but not due to memory problems, but due to conflict problems ! Again no other task was running against this database (all the other REST topaz scripts had nothing to do).

Then I wanted to know, why the commit always failed and I managed to get this output:

failureRead-Write Conflicts...

Write-Write Conflicts...    9749490177

Write-Dependency Conflicts...

Write-ReadLock Conflicts...

Write-WriteLock Conflicts...

Rc-Write-Write Conflicts...

Synchronized-Commit Conflicts...

When I import the same data - but doing a commit after e.g. 50000 addresses, the import is again stable as usual. The test cases uses lots of RcCounter instances (and those values are changed with each address) - but other like that I see no unusual stuff.


I know, I know - no real solid information, but this behaviour seemed strange to me ...


But perhaps someone has a hint ????



Marten


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts

GLASS mailing list
In reply to this post by GLASS mailing list

Well, you are right that without details it is nearly impossible to guess ... [as Richard mentions, `Obect _objectForOop: 9749490177` is your best clue]

You don't mention if there is any other activity in the system?

A commit conflict implies that there is at least one other gem doing work of some kind so presumably the single commit took long enough for another gem to modify an object in common (the object with oop 9749490177) ...

So if the object itself doesn't give you a big enough clue, you should think about the activity that might be going on in other gems ...

Dale


On 02/22/2017 01:23 PM, Marten Feldtmann via Glass wrote:

Today I noticed a behaviour I found strange. In one of our project we had to import address data and my general test case was around100000 addresses and the import into Gemstone/S went without problems (using ONE transaction) - it was a single task (no other applications running agsint the database). It never failed. It simply worked.

Now I talked with a customer and he told me about projects with 300.000 addresses and additional columns (etc). Much more data ...

Ok, I build a test case data set and to my surprise: Gemstone/S was NEVER able to import that data set in ONE transaction. The commit ALWAYS failed. The commit failed but not due to memory problems, but due to conflict problems ! Again no other task was running against this database (all the other REST topaz scripts had nothing to do).

Then I wanted to know, why the commit always failed and I managed to get this output:

failureRead-Write Conflicts...

Write-Write Conflicts...    9749490177

Write-Dependency Conflicts...

Write-ReadLock Conflicts...

Write-WriteLock Conflicts...

Rc-Write-Write Conflicts...

Synchronized-Commit Conflicts...

When I import the same data - but doing a commit after e.g. 50000 addresses, the import is again stable as usual. The test cases uses lots of RcCounter instances (and those values are changed with each address) - but other like that I see no unusual stuff.


I know, I know - no real solid information, but this behaviour seemed strange to me ...


But perhaps someone has a hint ????



Marten



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts

GLASS mailing list
In reply to this post by GLASS mailing list
Did you try inspecting (Object _objectForOpp: 9749490177) to see which was the conflict?

On Wed, Feb 22, 2017 at 6:23 PM, Marten Feldtmann via Glass <[hidden email]> wrote:

Today I noticed a behaviour I found strange. In one of our project we had to import address data and my general test case was around100000 addresses and the import into Gemstone/S went without problems (using ONE transaction) - it was a single task (no other applications running agsint the database). It never failed. It simply worked.

Now I talked with a customer and he told me about projects with 300.000 addresses and additional columns (etc). Much more data ...

Ok, I build a test case data set and to my surprise: Gemstone/S was NEVER able to import that data set in ONE transaction. The commit ALWAYS failed. The commit failed but not due to memory problems, but due to conflict problems ! Again no other task was running against this database (all the other REST topaz scripts had nothing to do).

Then I wanted to know, why the commit always failed and I managed to get this output:

failureRead-Write Conflicts...

Write-Write Conflicts...    9749490177

Write-Dependency Conflicts...

Write-ReadLock Conflicts...

Write-WriteLock Conflicts...

Rc-Write-Write Conflicts...

Synchronized-Commit Conflicts...

When I import the same data - but doing a commit after e.g. 50000 addresses, the import is again stable as usual. The test cases uses lots of RcCounter instances (and those values are changed with each address) - but other like that I see no unusual stuff.


I know, I know - no real solid information, but this behaviour seemed strange to me ...


But perhaps someone has a hint ????



Marten


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass




--

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts

GLASS mailing list


On Wed, Feb 22, 2017 at 7:09 PM, Mariano Martinez Peck <[hidden email]> wrote:
Did you try inspecting (Object _objectForOpp: 9749490177) to see which was the conflict?


Sorry, I was offline and the email was sent automatically (didn't see Dale and Richard answers)
 
On Wed, Feb 22, 2017 at 6:23 PM, Marten Feldtmann via Glass <[hidden email]> wrote:

Today I noticed a behaviour I found strange. In one of our project we had to import address data and my general test case was around100000 addresses and the import into Gemstone/S went without problems (using ONE transaction) - it was a single task (no other applications running agsint the database). It never failed. It simply worked.

Now I talked with a customer and he told me about projects with 300.000 addresses and additional columns (etc). Much more data ...

Ok, I build a test case data set and to my surprise: Gemstone/S was NEVER able to import that data set in ONE transaction. The commit ALWAYS failed. The commit failed but not due to memory problems, but due to conflict problems ! Again no other task was running against this database (all the other REST topaz scripts had nothing to do).

Then I wanted to know, why the commit always failed and I managed to get this output:

failureRead-Write Conflicts...

Write-Write Conflicts...    9749490177

Write-Dependency Conflicts...

Write-ReadLock Conflicts...

Write-WriteLock Conflicts...

Rc-Write-Write Conflicts...

Synchronized-Commit Conflicts...

When I import the same data - but doing a commit after e.g. 50000 addresses, the import is again stable as usual. The test cases uses lots of RcCounter instances (and those values are changed with each address) - but other like that I see no unusual stuff.


I know, I know - no real solid information, but this behaviour seemed strange to me ...


But perhaps someone has a hint ????



Marten


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass




--



--

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts

Richard Sargent
Administrator
In reply to this post by GLASS mailing list
GLASS mailing list wrote
<!DOCTYPE html>

   
Today I noticed a behaviour I found strange. In one of our project we had to import address data and my general test case was around100000 addresses and the import into Gemstone/S went without problems (using ONE transaction) - it was a single task (no other applications running agsint the database). It never failed. It simply worked. Now I talked with a customer and he told me about projects with 300.000 addresses and additional columns (etc). Much more data ...
Marten,

Independent of the write conflict, there is the question of designing for this problem space. You appear to have an open-ended problem, both in terms of number of rows but also in terms of number of columns. You should build a solution that supports this characteristic.

If there is a practical limit to the number of columns, you can focus on how many rows to process in a batch. Otherwise, you would want to design around the number of cells. You want batches large enough that a significant amount of work is done in each, but not so large that any kind of error wastes too much time retrying the batch. Keeping the length of time down protects against commit record backlogs as well as write conflicts. It cannot eliminate the possibility, but it does reduce the chances as well as making recovery more effective.


Ok, I build a test case data set and to my surprise: Gemstone/S was NEVER able to import that data set in ONE transaction. The commit ALWAYS failed. The commit failed but not due to memory problems, but due to conflict problems ! Again no other task was running against this database (all the other REST topaz scripts had nothing to do). Then I wanted to know, why the commit always failed and I managed to get this output: failureRead-Write Conflicts... Write-Write Conflicts...    9749490177 Write-Dependency Conflicts... Write-ReadLock Conflicts... Write-WriteLock Conflicts... Rc-Write-Write Conflicts... Synchronized-Commit Conflicts... When I import the same data - but doing a commit after e.g. 50000 addresses, the import is again stable as usual. The test cases uses lots of RcCounter instances (and those values are changed with each address) - but other like that I see no unusual stuff. I know, I know - no real solid information, but this behaviour seemed strange to me ... But perhaps someone has a hint ???? Marten
 

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts

GLASS mailing list

Thanks for all answers ... and the solution was a very simple one ... and it was very clear, when inspecting the object responsible for the conflict (I never looked in that area before). More or less: own logic fault.

To summarize: The UI developer implemented a background AJAX call, which is regularly independently send to Gemstone/S. That means (as an example), that the address import call has a duration of about 2 minutes and within that call two other calls were sent to the database - within the same "session". These simple calls changed the "lastActivity"-attribute within the session objects - and the long term importing task never had a chance to finish its work without problems (because after importing all with success it wanted to update the lastActivity attribute of its own session - also) - poor task :-).

That seems to be a starting point for writing my own RcDateAndTime class :-))))

Therefore: own fault and a classical example for concurrency problems and a nice starting point to think about how to make it better.

Just to let you know ...

Marten

Richard Sargent via Glass <[hidden email]> hat am 23. Februar 2017 um 00:49 geschrieben:

GLASS mailing list wrote

<!DOCTYPE html>

Today I noticed a behaviour I found strange. In one of our project we had
to import address data and my general test case was around100000 addresses
and the import into Gemstone/S went without problems (using ONE
transaction) - it was a single task (no other applications running agsint
the database). It never failed. It simply worked. Now I talked with a
customer and he told me about projects with 300.000 addresses and
additional columns (etc). Much more data ...

Marten,

Independent of the write conflict, there is the question of designing for
this problem space. You appear to have an open-ended problem, both in terms
of number of rows but also in terms of number of columns. You should build a
solution that supports this characteristic.

If there is a practical limit to the number of columns, you can focus on how
many rows to process in a batch. Otherwise, you would want to design around
the number of cells. You want batches large enough that a significant amount
of work is done in each, but not so large that any kind of error wastes too
much time retrying the batch. Keeping the length of time down protects
against commit record backlogs as well as write conflicts. It cannot
eliminate the possibility, but it does reduce the chances as well as making
recovery more effective.

Ok, I build a test case data set and to my surprise: Gemstone/S was NEVER
able to import that data set in ONE transaction. The commit ALWAYS failed.
The commit failed but not due to memory problems, but due to conflict
problems ! Again no other task was running against this database (all the
other REST topaz scripts had nothing to do). Then I wanted to know, why
the commit always failed and I managed to get this output:
failureRead-Write Conflicts... Write-Write Conflicts...   
9749490177 Write-Dependency Conflicts... Write-ReadLock Conflicts...
Write-WriteLock Conflicts... Rc-Write-Write Conflicts...
Synchronized-Commit Conflicts... When I import the same data - but doing a
commit after e.g. 50000 addresses, the import is again stable as usual.
The test cases uses lots of RcCounter instances (and those values are
changed with each address) - but other like that I see no unusual stuff. I
know, I know - no real solid information, but this behaviour seemed
strange to me ... But perhaps someone has a hint ???? Marten

_______________________________________________
Glass mailing list

Glass@.gemtalksystems

http://lists.gemtalksystems.com/mailman/listinfo/glass

--
View this message in context: http://forum.world.st/Strange-behaviour-too-many-data-and-commit-failed-not-due-to-memory-but-conflicts-tp4935505p4935524.html
Sent from the GLASS mailing list archive at Nabble.com.

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Loading...