Strange behaviour ... too many data and commit failed not due to memory, but conflicts
Locked
Strange behaviour ... too many data and commit failed not due to memory, but conflicts
 
|
Today I noticed a behaviour I found strange. In one of our project we had to import address data and my general test case was around100000 addresses and the import into Gemstone/S went without problems (using ONE transaction) - it was a single task (no other applications running agsint the database). It never failed. It simply worked. Now I talked with a customer and he told me about projects with 300.000 addresses and additional columns (etc). Much more data ... Ok, I build a test case data set and to my surprise: Gemstone/S was NEVER able to import that data set in ONE transaction. The commit ALWAYS failed. The commit failed but not due to memory problems, but due to conflict problems ! Again no other task was running against this database (all the other REST topaz scripts had nothing to do). Then I wanted to know, why the commit always failed and I managed to get this output: failureRead-Write Conflicts... When I import the same data - but doing a commit after e.g. 50000 addresses, the import is again stable as usual. The test cases uses lots of RcCounter instances (and those values are changed with each address) - but other like that I see no unusual stuff. I know, I know - no real solid information, but this behaviour seemed strange to me ... But perhaps someone has a hint ???? Marten _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts
|
|
Marten, I think 9749490177 is the oop of the object with the conflict. You should look up the object (e.g. Object _objectForOop:974949017). Whatever that object is, it will be informative for diagnosing the problem.On Wed, Feb 22, 2017 at 1:23 PM, Marten Feldtmann via Glass <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts
|
|
In reply to this post by GLASS mailing list
Well, you are right that without details it is nearly impossible
to guess ... [as Richard mentions, `Obect _objectForOop:
9749490177` is your best clue] You don't mention if there is any other activity in the system? A commit conflict implies that there is at least one other gem
doing work of some kind so presumably the single commit took long
enough for another gem to modify an object in common (the object
with oop 9749490177) ... So if the object itself doesn't give you a big enough clue, you should think about the activity that might be going on in other gems ... Dale On 02/22/2017 01:23 PM, Marten
Feldtmann via Glass wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts
|
|
In reply to this post by GLASS mailing list
Did you try inspecting (Object _objectForOpp: 9749490177) to see which was the conflict? On Wed, Feb 22, 2017 at 6:23 PM, Marten Feldtmann via Glass <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts
|
|
On Wed, Feb 22, 2017 at 7:09 PM, Mariano Martinez Peck <[hidden email]> wrote:
Sorry, I was offline and the email was sent automatically (didn't see Dale and Richard answers)
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts
Administrator
|
In reply to this post by GLASS mailing list
Marten, Independent of the write conflict, there is the question of designing for this problem space. You appear to have an open-ended problem, both in terms of number of rows but also in terms of number of columns. You should build a solution that supports this characteristic. If there is a practical limit to the number of columns, you can focus on how many rows to process in a batch. Otherwise, you would want to design around the number of cells. You want batches large enough that a significant amount of work is done in each, but not so large that any kind of error wastes too much time retrying the batch. Keeping the length of time down protects against commit record backlogs as well as write conflicts. It cannot eliminate the possibility, but it does reduce the chances as well as making recovery more effective.
|
Re: Strange behaviour ... too many data and commit failed not due to memory, but conflicts
|
|
Thanks for all answers ... and the solution was a very simple one ... and it was very clear, when inspecting the object responsible for the conflict (I never looked in that area before). More or less: own logic fault. To summarize: The UI developer implemented a background AJAX call, which is regularly independently send to Gemstone/S. That means (as an example), that the address import call has a duration of about 2 minutes and within that call two other calls were sent to the database - within the same "session". These simple calls changed the "lastActivity"-attribute within the session objects - and the long term importing task never had a chance to finish its work without problems (because after importing all with success it wanted to update the lastActivity attribute of its own session - also) - poor task :-). That seems to be a starting point for writing my own RcDateAndTime class :-)))) Therefore: own fault and a classical example for concurrency problems and a nice starting point to think about how to make it better. Just to let you know ... Marten
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
| Free forum by Nabble | Edit this page |