GlorpSession refresh: creates new instances when there are references to existing ones

Hi there,

This is happening on VAST and I have no idea whether it also happens on other Dialects. I just wonder if there is any explanation for it. The effect causes us quite a few problems in a multi-user system where objects can be updated outside our session and we need to read teh updated values.

Let me start with a snippet of code:

|objects sess obj|
obj := (objects := sess read: MyClass where: [:ea| ea id = 53160]) first.
Transcript cr; show: 'Before: ', (MyClass allInstances select: [:ea| ea id= 53160]) size asString.
obj count: obj count +1.
sess commitUnitOfWork; beginUnitOfWork.
sess refresh: obj.
Transcript cr; show: 'After: ', (MyClass allInstances select: [:ea| ea id= 53160]) size asString.
Transcript cr; show: obj count asString.
Transcript cr.

The interesting part here is that the number of instances before and after the #refresh: is 2. Which is as expected (given my shallow knowledge of Glorp): one is the copy kept in the cache/undoMap and the other one is the instance my Application is supposed to work with.
If I repeatedly run this snippet, the number of instances remains 2....

...UNTIL I open an inspector on MyClass allInstances select: [:ea| ea id= 53160]. As soon as I rerun the snippet, the number rises to 3 and remains at 3 for every consecutive run. UNTIL I open a new inspector on the same expression again. You can play that game over and over again: every time I open an inspector on the currently loaded instances and do the refresh: I end up with one more instance.

The most problematic thing here is, however, that the variable #count is some old value in all but the latest instance. So each object holding onto an "old" copy of the row/object sees old and invalid data.

What I expected was that either the number of instances does not change at all (the logic variant) and everybody looking at any of these instances sees the latest and correct value of count. The less logical thing that I would find acceptable is that there may be more an more instances showing up, but they all have the correct value of count (however that might magically happen).

Unfortunately, both expectations were wrong. The situation is that what you see as count depends on how old the instance you hold in your hands is, because as son as you hold an instance in your hand #refresh: will create a new instance. This is of course really bad.

So either I am doing it all wrong or there is something broken in the VAST port of Glorp. I would really appreciate if somebody on VW or Pharo could test this and tell us what they see. And it would even be better if somebody can tell me what's wrong about my expectations.

What I want to achieve is the following:

I read:where: an object
I open a GUI on this object (holding a reference onto that object).
I save the object by sending commitUnitOfWork to the GlorpSession
The database does some calculation in a Trigger or so (this is the scenario in which we found this out, but this step doesn't really matter)
I refresh: the object
I see the latest value of #count in my GUI


How do people handle this?


Joachim

Hi again


my last post was probably not clear enough as to what the problem really is. I refined my test script a little to demonstrate the effect a little better.

So here is a slightly modified version of the script from may last post:


|objects sess obj col|
sess := KontolinoSession instance dbSession.
obj := (objects := sess read: Buchungsbeleg where: [:ea| ea id = 53160]) first.
Transcript cr; show: 'Before: ', (Buchungsbeleg allInstances select: [:ea| ea id= 53160]) size asString.
obj belegnummer: (obj belegnummer asNumber +1) asString.
sess commitUnitOfWork; beginUnitOfWork.
sess refresh: obj.
col := Buchungsbeleg allInstances select: [:ea| ea id= 53160].
Transcript cr; show: 'After: ', col size asString; cr.
col do: [:b| Transcript show: b hash asString, ': ', b belegnummer asString, ' ; '].

col inspect.  "<-- this line makes a big difference!!!"

Transcript cr; show: obj belegnummer asString.
Transcript cr.



The output of a few runs of this script looks like this when the "col inspect" is commented out


Before: 2
After: 2
13434: 35 ; 11523: 35 ;
35

Before: 2
After: 2
13434: 36 ; 10785: 36 ;
36

Before: 2
After: 2
13434: 37 ; 16835: 37 ;
37

Before: 2
After: 2
13434: 38 ; 15950: 38 ;
38

As you can see, this is really fine and what is to be expected.
But if I remove the comment on "col inspect" and have the inspector open on each run, the result looks very different:



Before: 2
After: 2
13434: 39 ; 10653: 39 ;
39

Before: 2
After: 3
13434: 40 ; 10653: 39 ; 17521: 40 ;
40

Before: 3
After: 4
13434: 41 ; 10653: 39 ; 17521: 40 ; 19129: 41 ;
41

Before: 4
After: 5
13434: 42 ; 10653: 39 ; 17521: 40 ; 19129: 41 ; 5939: 42 ;
42


You see, not all existing instances are updated any more.

In fact, we see situations in which #refresh: seems not to refresh our business objects with the current values from the DB. My theory is that some of the "old versions" are referenced somewhere in the application and therefore display invalid, old data.


Am I missing something obvious or is this a bug?

Joachim

David,


there is nothing to be sorry for. I am glad somebody helps looking intot this. The real issue is that in our application, some objects aren't updated with the actual values from the DB.

I tried your suggestion adding "System globalGarbageCollect" (I am on VAST) to the end of my Script. Here is the output:


Before: 2
After: 2
22153: 22 ; 11760: 22 ;
22

Before: 2
After: 3
22153: 23 ; 11760: 22 ; 13436: 23 ;
23

Before: 3
After: 4
22153: 24 ; 11760: 22 ; 13436: 23 ; 15230: 24 ;
24

Before: 4
After: 5
22153: 25 ; 11760: 22 ; 13436: 23 ; 15230: 24 ; 2477: 25 ;
25

Unfortunately, VAST has no allInstancesWrakly: or similar (that I am aware of), so I cannot test your other suggestion. But I guess the fact that not opeining an inspector does not increase the number of instances proves that teh references to the objects is the key here.


So I agree your argument sounds logical. The guess would be these are old copies that Glorp holds internally for rollback and such (undoMap). But these should never get into my hands as a user of Glorp. So these should not cause any problems "above" Glorp.

My biggest problem for now is that a) I don't understand why the above happens and b) I am not really sure this is really related to my bugs in the program where objects do not always get updated with the right values in a #refresh: operation.


Thanks for lending me an eye and brain. I'll have to llok further.

Joachim





Am Freitag, 9. Februar 2018 19:07:08 UTC+1 schrieb Wallen, David:

> You see, not all existing instances are updated any more.

 

I’m not so sure I understand what the problem is, so forgive my ignorance, but your second set of results is where the con inspect is enabled, right?

If so, then it looks to a naïve person (like myself) as if the

            Buchungsbeleg allInstances select: [:ea| ea id= 53160].

is counting a bunch of instances from past runs, instances which will probably be GC’d soon since they are no longer referenced. If you put

            ObjectMemory collectAllGarbage

in there at the end of each run, does that change your results?

 

Oh, btw, try using

            (Buchungsbeleg allInstancesWeakly: true)

to avoid returning yet another set of references to those guys, allowing them to live through another GC or so.

 

Otoh, sorry if I’m just not seeing the real issue.

Dave

 

 

 

From: <a href="javascript:" target="_blank" gdf-obfuscated-mailto="ZLJYkfSYBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">glorp...@... [mailto:<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ZLJYkfSYBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">glorp...@googlegroups.com] On Behalf Of jtuchel
Sent: Friday, February 09, 2018 5:55 AM
To: glorp-group
Subject: Re: GlorpSession refresh: creates new instances when there are references to existing ones

 

Hi again


my last post was probably not clear enough as to what the problem really is. I refined my test script a little to demonstrate the effect a little better.

So here is a slightly modified version of the script from may last post:


|objects sess obj col|
sess := KontolinoSession instance dbSession.
obj := (objects := sess read: Buchungsbeleg where: [:ea| ea id = 53160]) first.
Transcript cr; show: 'Before: ', (Buchungsbeleg allInstances select: [:ea| ea id= 53160]) size asString.
obj belegnummer: (obj belegnummer asNumber +1) asString.
sess commitUnitOfWork; beginUnitOfWork.
sess refresh: obj.
col := Buchungsbeleg allInstances select: [:ea| ea id= 53160].
Transcript cr; show: 'After: ', col size asString; cr.
col do: [:b| Transcript show: b hash asString, ': ', b belegnummer asString, ' ; '].

col inspect.  "<-- this line makes a big difference!!!"

Transcript cr; show: obj belegnummer asString.
Transcript cr.



The output of a few runs of this script looks like this when the "col inspect" is commented out


Before: 2
After: 2
13434: 35 ; 11523: 35 ;
35

Before: 2
After: 2
13434: 36 ; 10785: 36 ;
36

Before: 2
After: 2
13434: 37 ; 16835: 37 ;
37

Before: 2
After: 2
13434: 38 ; 15950: 38 ;
38

As you can see, this is really fine and what is to be expected.
But if I remove the comment on "col inspect" and have the inspector open on each run, the result looks very different:



Before: 2
After: 2
13434: 39 ; 10653: 39 ;
39

Before: 2
After: 3
13434: 40 ; 10653: 39 ; 17521: 40 ;
40

Before: 3
After: 4
13434: 41 ; 10653: 39 ; 17521: 40 ; 19129: 41 ;
41

Before: 4
After: 5
13434: 42 ; 10653: 39 ; 17521: 40 ; 19129: 41 ; 5939: 42 ;
42


You see, not all existing instances are updated any more.

In fact, we see situations in which #refresh: seems not to refresh our business objects with the current values from the DB. My theory is that some of the "old versions" are referenced somewhere in the application and therefore display invalid, old data.


Am I missing something obvious or is this a bug?

Joachim

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="ZLJYkfSYBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">glorp-group...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="ZLJYkfSYBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;"> glorp...@....
Visit this group at <a href="https://groups.google.com/group/glorp-group" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/glorp-group&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/glorp-group&#39;;return true;">https://groups.google.com/group/glorp-group.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

Thanks to the comments I received from David and Thomas Brodt who contacted me privately, I found out I am using Glorp all wrong....

So after a lot of searching and playing and Transcript show: and stuff I found out my problem is not remotely related to #refresh: at all (some of you may not be surprised ;-) ).
To make a long story short: refresh: does refresh and all is fine and these additional instances however strange they might look are not a problem for application code.

What causes trouble is my use of Transactions or Unit(s)OfWork in Glorp. It turns out that when I do a #rollbackUnitOfWork, an object that has just been correctly refreshed, gets rolled back to its initial state, which, from the perspective of the Smalltalk image, is not what is currently there in the database. It is what was read from the database at the first read within the current session. This is, of course, a clear case of „works as intended“.

So obviously, I am using Glorp the wrong way when it comes to Transaction handling.

I’ll try to give a short intro to what we do.

We start off by sending a beginUnitOfWork to the Glory session and reading our „root object“. When the user works with the objects, they get read by their proxies, nothing special here. When the user clicks „OK“ in a screen, we do both: a commitUnitOfWork and beginUnitOfWork and just carry on doing our page rendering and stuff. There are „cancel“ buttons that do a rollbackUnitOfWork;beginUnitOfWork in some places, but to be honest, in a web context these make little sense, because nobody clicks them or at least there is no penalty for not clicking them. Cancel Buttons are essentially useless in web apps.

Our Web App has a menu on the left side of the page where you can start a new „dialog“ any time you want, be it in the middle of changing objects or after a commit. So we added a #rollbackUnitOfWork to the handler for all menu items, just to make sure no changes that are not yet committed don’t survive from the current screen. This has worked for a few years, because we use a framework that only stores valid values into the business objects when they save / commit.

With more and more Ajax creeping into our dialogs, there are now changes to the business objects in-between opening a screen and saving/committing (or rolling back). That is why we introduced the safety net of this additional rollback.

Our business model is so interconnected that it is hard to work with memento objects (how many objects do have to be provided as working copies to make business checks work correctly?).
On the other hand, doing a full reread of all objects every time you commit or rollback takes up to a few seconds, so we tried to live with the (obviously dangerous) combination of long-lived units of work (in the image, not on the database side).

Is there a way of doing a „rollback these objects to the state that’s in the database now“ that also resets what Glorp thinks of as the initial value? Maybe something like „proxy these objects“?

What is other people’s strategy for transaction handling in a non-modal application when re-loading everything all the time is not a desirable option?

I hope for comments, ideas, impulses.

Joachim

I dug a bit deeper and found that what happens is that when #rollbackUnitOfWork does its cleanup, the object that was kept in the Transaction's undoMap is not the "latest" one.

What I considered the latest one is one that was refreshed in a previous operation (right after the last commit). So #refresh: doesn't seem to update the copy that's kept in the Transacton's undoMap.

Eo give a little example that explains a bit better:

Say we have a Car class and a Wheel class. Let's assume that counting the wheels of a car is an expensive operation, so we implement a Database Trigger that updates a car's wheelCount variable. This Trigger fires whenever a wheel is INSERTED or DELETED or a car is updated. In order to keep the wheelCount variable in sync within our application we do this:

addWheel: aWheel
 self wheels add: aWheel.
 self dbSession commitUnitoOfWork; beginUnitOfWork.
 self dbSession refresh: self.
  

So let's assume the wheelCount was 2 when we read the car from the database and we run this operation once. The car has a wheelCount of 3 now - both in the DB and in the application.

If I now do a rollbackUnitOfWork, the car will be rolled back in Transaction>>abort to a wheelCount of 2 (remember, the committed value is 3 in the databse), because that is the value in the Transction's undoMap. The car is now officially out of sync with the databse.  I guess this is due to the fact that refresh: doesn't change the undoMap (must do research on this).

I wonder if this is good or bad? Am I expecting somethig wrong? Is it okay that the undoMap contains an old value for wheelCount? I am not sure. In the spirit of repeatable reads it is, right? OTOH, refresh: is a brute force attack on Glorp's book keeping anyways, so can't I expect refresh: to also update the undoMap?


So, to conclude, I am puzzled. Am I misusing Glorp and expecting something wrong or did I find a bug? Not sure about it.

Any thoughts?

Joachim

Is it not possible to refresh before starting unit of work? Something like:

Madhu,

seems you made my day!
My first tests look more than promising.

Now that you mention it: the solution sounds logical, how come I made such an interstellar journey around this problem...?

I need to make quite a few changes to our workflow management to make this work, but it's absolutely worth it.

Thanks Madhu. If we happen to meet (maybe in Cagliari) please remind me I owe you a drink.


Joachim







Am Sonntag, 11. Februar 2018 08:53:46 UTC+1 schrieb Madhu:

Is it not possible to refresh before starting unit of work? Something like:

self dbSession commitUnitOfWork.
self dbSession refresh: self.

self dbSession beginUnitOfWork.

Regards,

Madhu.

On Sun, Feb 11, 2018 at 8:02 AM, jtuchel <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="UL5dNqUUBQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jtu...@...> wrote:

I dug a bit deeper and found that what happens is that when #rollbackUnitOfWork does its cleanup, the object that was kept in the Transaction's undoMap is not the "latest" one.

What I considered the latest one is one that was refreshed in a previous operation (right after the last commit). So #refresh: doesn't seem to update the copy that's kept in the Transacton's undoMap.

Eo give a little example that explains a bit better:

Say we have a Car class and a Wheel class. Let's assume that counting the wheels of a car is an expensive operation, so we implement a Database Trigger that updates a car's wheelCount variable. This Trigger fires whenever a wheel is INSERTED or DELETED or a car is updated. In order to keep the wheelCount variable in sync within our application we do this:

addWheel: aWheel
 self wheels add: aWheel.
 self dbSession commitUnitoOfWork; beginUnitOfWork.
 self dbSession refresh: self.
  

So let's assume the wheelCount was 2 when we read the car from the database and we run this operation once. The car has a wheelCount of 3 now - both in the DB and in the application.

If I now do a rollbackUnitOfWork, the car will be rolled back in Transaction>>abort to a wheelCount of 2 (remember, the committed value is 3 in the databse), because that is the value in the Transction's undoMap. The car is now officially out of sync with the databse.  I guess this is due to the fact that refresh: doesn't change the undoMap (must do research on this).

I wonder if this is good or bad? Am I expecting somethig wrong? Is it okay that the undoMap contains an old value for wheelCount? I am not sure. In the spirit of repeatable reads it is, right? OTOH, refresh: is a brute force attack on Glorp's book keeping anyways, so can't I expect refresh: to also update the undoMap?


So, to conclude, I am puzzled. Am I misusing Glorp and expecting something wrong or did I find a bug? Not sure about it.

Any thoughts?

Joachim

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="UL5dNqUUBQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">glorp-group...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="UL5dNqUUBQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">glorp...@....
Visit this group at <a href="https://groups.google.com/group/glorp-group" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/glorp-group&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/glorp-group&#39;;return true;">https://groups.google.com/group/glorp-group.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

Just an update,


the correction Madhu suggested is in production now and works very well.


looking back, it's not easy to understand why I didn't see that the problem is not #refresh: but the later rollbacks. But seems like some bugs just want to be found the hard way...

Thanks to all who took the time to read and offer thoughts. It is sooo important to have people to discuss problems with.


Joachim


Am Sonntag, 11. Februar 2018 19:07:26 UTC+1 schrieb jtuchel:

Madhu,

seems you made my day!
My first tests look more than promising.

Now that you mention it: the solution sounds logical, how come I made such an interstellar journey around this problem...?

I need to make quite a few changes to our workflow management to make this work, but it's absolutely worth it.

Thanks Madhu. If we happen to meet (maybe in Cagliari) please remind me I owe you a drink.


Joachim







Am Sonntag, 11. Februar 2018 08:53:46 UTC+1 schrieb Madhu:

Is it not possible to refresh before starting unit of work? Something like:

self dbSession commitUnitOfWork.
self dbSession refresh: self.

self dbSession beginUnitOfWork.

Regards,

Madhu.

On Sun, Feb 11, 2018 at 8:02 AM, jtuchel <[hidden email]> wrote:

I dug a bit deeper and found that what happens is that when #rollbackUnitOfWork does its cleanup, the object that was kept in the Transaction's undoMap is not the "latest" one.

What I considered the latest one is one that was refreshed in a previous operation (right after the last commit). So #refresh: doesn't seem to update the copy that's kept in the Transacton's undoMap.

Eo give a little example that explains a bit better:

Say we have a Car class and a Wheel class. Let's assume that counting the wheels of a car is an expensive operation, so we implement a Database Trigger that updates a car's wheelCount variable. This Trigger fires whenever a wheel is INSERTED or DELETED or a car is updated. In order to keep the wheelCount variable in sync within our application we do this:

addWheel: aWheel
 self wheels add: aWheel.
 self dbSession commitUnitoOfWork; beginUnitOfWork.
 self dbSession refresh: self.
  

So let's assume the wheelCount was 2 when we read the car from the database and we run this operation once. The car has a wheelCount of 3 now - both in the DB and in the application.

If I now do a rollbackUnitOfWork, the car will be rolled back in Transaction>>abort to a wheelCount of 2 (remember, the committed value is 3 in the databse), because that is the value in the Transction's undoMap. The car is now officially out of sync with the databse.  I guess this is due to the fact that refresh: doesn't change the undoMap (must do research on this).

I wonder if this is good or bad? Am I expecting somethig wrong? Is it okay that the undoMap contains an old value for wheelCount? I am not sure. In the spirit of repeatable reads it is, right? OTOH, refresh: is a brute force attack on Glorp's book keeping anyways, so can't I expect refresh: to also update the undoMap?


So, to conclude, I am puzzled. Am I misusing Glorp and expecting something wrong or did I find a bug? Not sure about it.

Any thoughts?

Joachim

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to glorp-group...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/glorp-group" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/glorp-group&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/glorp-group&#39;;return true;">https://groups.google.com/group/glorp-group.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.