Best practices for using obects across Transaction/UOW borders

As you may have read in my previous question about commits for Transactions with lots of objects loeded from the database, we are having issues with big numbers of objects in the caches.

This leads me to an old question I always had when it comes to effectively use objects across Transaction borders.

Example:

1. We load a list of objects into memory in one dialog. Here we commit some change to some object (like the time somebody took a look at this list).
2. The user double clicks one of these objects and opens a dialog to make some changes to this object and commits these changes and closes the dialog
3. The initialt list of objects is updated and the user can choose to work on the details of another object

I guess this is not a very weird and strange scenario, and yet it is complicated from the perspective of an ORM. How do we make sure none of the Transactions (UOW) gets out of sync with the database?

What is the best practice for such a scenario.

1. Do we use long UOWs by using #commitUnitOfWorkAndContinue?
   
2. Is it best to use commitUnitOfWork and make sure the UOW in the dialog re-reads its object in a new UOW in order to track the changes? Simply handing the object to the dialog after a commitUOW for the logging of last access would detach the object from the database (the new Transaction doesn't know this object because it is not in the new Transaction's undoMap)
3. Do people use #inTransactionDo: and take the full responsibility of all the bookkeeping?

We chose to use the ...AndContinue variants, because we thought this is the best compromise between speed and comfort. But it seems this approach has its limits when the number of objects gets large (see the other post).

We also thought that approach #2 might be dangerous, because something like

self session commitUnitOfWork.

self showDetailsDialogFor: self selectedObject.

has the danger of detaching the selected object or some objects it references from the database. It would then probably mean it is much better to just call other dialogs with the id of the "root object" to reload for editing, just to make sure the dialog reads the object and all its referenced objects in its own Transaction...

I hope I managed to communicate my concerns and questions. I'd be interested in other peoples' approach to this tricky problem. So far we thought we chose the best possible path but I am not so sure any more....

I look forward to a fruitful discussion and hope to read your comments and ideas soon.

Joachim

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/d7198a44-01fe-4317-acde-e68b01a6af65n%40googlegroups.com.

I always tried to keep transactions short. So my pattern would have been that if you have a list it's not in a transaction. When someone goes to edit an object, start a unit of work, refresh that object within the unit of work and open the dialog on it. By refreshing you should force it to re-read proxies as well, so it's a little bit slower to open but known to be consistent with the DB. And try to have the dialog take a single object, and to have changes that might happen within it not affect the rest of the list. Otherwise it gets a bit more complicated.

On Thu, Nov 5, 2020 at 2:32 AM jtuchel <[hidden email]> wrote:

As you may have read in my previous question about commits for Transactions with lots of objects loeded from the database, we are having issues with big numbers of objects in the caches.

This leads me to an old question I always had when it comes to effectively use objects across Transaction borders.

Example:

1. We load a list of objects into memory in one dialog. Here we commit some change to some object (like the time somebody took a look at this list).
2. The user double clicks one of these objects and opens a dialog to make some changes to this object and commits these changes and closes the dialog
3. The initialt list of objects is updated and the user can choose to work on the details of another object

I guess this is not a very weird and strange scenario, and yet it is complicated from the perspective of an ORM. How do we make sure none of the Transactions (UOW) gets out of sync with the database?

What is the best practice for such a scenario.

1. Do we use long UOWs by using #commitUnitOfWorkAndContinue?
   
2. Is it best to use commitUnitOfWork and make sure the UOW in the dialog re-reads its object in a new UOW in order to track the changes? Simply handing the object to the dialog after a commitUOW for the logging of last access would detach the object from the database (the new Transaction doesn't know this object because it is not in the new Transaction's undoMap)
3. Do people use #inTransactionDo: and take the full responsibility of all the bookkeeping?

We chose to use the ...AndContinue variants, because we thought this is the best compromise between speed and comfort. But it seems this approach has its limits when the number of objects gets large (see the other post).

We also thought that approach #2 might be dangerous, because something like

self session commitUnitOfWork.

self showDetailsDialogFor: self selectedObject.

has the danger of detaching the selected object or some objects it references from the database. It would then probably mean it is much better to just call other dialogs with the id of the "root object" to reload for editing, just to make sure the dialog reads the object and all its referenced objects in its own Transaction...

I hope I managed to communicate my concerns and questions. I'd be interested in other peoples' approach to this tricky problem. So far we thought we chose the best possible path but I am not so sure any more....

I look forward to a fruitful discussion and hope to read your comments and ideas soon.

Joachim

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/d7198a44-01fe-4317-acde-e68b01a6af65n%40googlegroups.com.

Hi Joachim,

At Washington Mutual, we had this problem with GemStone. We ended up building our own mementos for objects before doing the rollback. The VAST write barrier was involved, but it's been 18 years and I don't remember the details.

Store has this problem as well. It really doesn't need the undoMap because all of the modified Store objects are derived from image resident objects of a different class. Niall and I found this shortly before I left Cincom. I don't know whether he has done something to fix it or not.  What he did is likely to be Store specific though, and not part of Glorp (I think).

I would ask Alan and yourself if it might make sense to create ReadOnlyGlorpSession as a subclass of GlorpSession. ReadOnlyGlorpSession would retrieve objects from the database and instantiate them as immutable using the VA or VW immutability. It would not register the objects in the undoMap and it would not allow units of work or transactions. It would cache objects to avoid performance issues with recursive data structures and duplicate references. This assumes that there is an obvious line between objects being modified and objects never being modified. It could be problematic if you tried to read an object in using a read only session and then tried to modify it subsequently. I probably haven't sniffed out all of the implementation complications...

Tom

On 11/6/2020 1:44 AM, jtuchel wrote:

I am currently trying to do this in one of our dialogs as a prototype. But there are still a few things I don't really get my head around.

As a first attempt. I just started my details/edit dialog with

self dbSession rollbackUnitOfWork; beginUnitOfWork; refresh: myModelObjejct.

This does work. Every object I navigate to is re-read, so it seems like the proxifying of the tranisitveClosure etc. work as expected.

There is a problem, however: My dialog has an entry field (actualla a couple of them) which reads data from the database. Lots of objects are being read in order to make suggestions in this entry field. And there are certain business checks going on on this object and potentially lots of referenced objects. They all get read from the database. That is not a performance problem per se, but they all end up in the unitOfWork's cache and undoMap and whatnot. So the effect of this is close to zero, as long as these objects are read within the currentUnitOfWork.

The solution to this sounds easy: postpone the beginUnitOfWork until something is actually changed in the business object. But....

• this also means postponing the refresh of the business object until we actually change it
  
• this also means we need some kind of  transient memento for the business object and all objects that may be modified along the way while the user works on this dialog. We'd have to do something like #registerTransitiveClosure, just to find out which objects to copy as transient proxies of their persistent counterparts. Or we create such mementos by hand, which is likely to be error prone and may open flooding gates for errors that go undetected for a while (how could you possibly write tests this? "Is there any object we modified that is detached at the moment? I guess a WriteBarrier could come in handy at this problem...)

I'd be interested in opinions and ideas. How do you guys handle these issues?

I know this is not a new problem at all. This is a question every user of an ORM has to ask herself. We thought we found a good solution that carries us a long way in the use of ...AndContinue and just keeping everything, because our users typically have small numbers of objects. Until they hadn't.

Do you have pointers to literature on this complex of questions. I guess Object Transaction Framework is the keyword here? Has anybody built such a framework on top of Glorp? Is everybody implementing their own handmade ad-hoc version of this?

Joachim

alan.knight schrieb am Donnerstag, 5. November 2020 um 14:47:55 UTC+1:

I always tried to keep transactions short. So my pattern would have been that if you have a list it's not in a transaction. When someone goes to edit an object, start a unit of work, refresh that object within the unit of work and open the dialog on it. By refreshing you should force it to re-read proxies as well, so it's a little bit slower to open but known to be consistent with the DB. And try to have the dialog take a single object, and to have changes that might happen within it not affect the rest of the list. Otherwise it gets a bit more complicated.

On Thu, Nov 5, 2020 at 2:32 AM jtuchel <[hidden email]> wrote:

As you may have read in my previous question about commits for Transactions with lots of objects loeded from the database, we are having issues with big numbers of objects in the caches.

This leads me to an old question I always had when it comes to effectively use objects across Transaction borders.

Example:

1. We load a list of objects into memory in one dialog. Here we commit some change to some object (like the time somebody took a look at this list).
2. The user double clicks one of these objects and opens a dialog to make some changes to this object and commits these changes and closes the dialog
3. The initialt list of objects is updated and the user can choose to work on the details of another object

I guess this is not a very weird and strange scenario, and yet it is complicated from the perspective of an ORM. How do we make sure none of the Transactions (UOW) gets out of sync with the database?

What is the best practice for such a scenario.

1. Do we use long UOWs by using #commitUnitOfWorkAndContinue?
   
2. Is it best to use commitUnitOfWork and make sure the UOW in the dialog re-reads its object in a new UOW in order to track the changes? Simply handing the object to the dialog after a commitUOW for the logging of last access would detach the object from the database (the new Transaction doesn't know this object because it is not in the new Transaction's undoMap)
3. Do people use #inTransactionDo: and take the full responsibility of all the bookkeeping?

We chose to use the ...AndContinue variants, because we thought this is the best compromise between speed and comfort. But it seems this approach has its limits when the number of objects gets large (see the other post).

We also thought that approach #2 might be dangerous, because something like

self session commitUnitOfWork.

self showDetailsDialogFor: self selectedObject.

has the danger of detaching the selected object or some objects it references from the database. It would then probably mean it is much better to just call other dialogs with the id of the "root object" to reload for editing, just to make sure the dialog reads the object and all its referenced objects in its own Transaction...

I hope I managed to communicate my concerns and questions. I'd be interested in other peoples' approach to this tricky problem. So far we thought we chose the best possible path but I am not so sure any more....

I look forward to a fruitful discussion and hope to read your comments and ideas soon.

Joachim

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/d7198a44-01fe-4317-acde-e68b01a6af65n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/2016532e-2f74-4b46-ab06-a0bc34c7ebebn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/30116c56-3459-9612-2860-019493140421%40gmail.com.

Hi Joachim,

At Washington Mutual, we had this problem with GemStone. We ended up building our own mementos for objects before doing the rollback. The VAST write barrier was involved, but it's been 18 years and I don't remember the details.

Store has this problem as well. It really doesn't need the undoMap because all of the modified Store objects are derived from image resident objects of a different class. Niall and I found this shortly before I left Cincom. I don't know whether he has done something to fix it or not.  What he did is likely to be Store specific though, and not part of Glorp (I think).

I would ask Alan and yourself if it might make sense to create ReadOnlyGlorpSession as a subclass of GlorpSession. ReadOnlyGlorpSession would retrieve objects from the database and instantiate them as immutable using the VA or VW immutability. It would not register the objects in the undoMap and it would not allow units of work or transactions. It would cache objects to avoid performance issues with recursive data structures and duplicate references. This assumes that there is an obvious line between objects being modified and objects never being modified. It could be problematic if you tried to read an object in using a read only session and then tried to modify it subsequently. I probably haven't sniffed out all of the implementation complications...

Tom

On 11/6/2020 1:44 AM, jtuchel wrote:

I am currently trying to do this in one of our dialogs as a prototype. But there are still a few things I don't really get my head around.

As a first attempt. I just started my details/edit dialog with

self dbSession rollbackUnitOfWork; beginUnitOfWork; refresh: myModelObjejct.

This does work. Every object I navigate to is re-read, so it seems like the proxifying of the tranisitveClosure etc. work as expected.

There is a problem, however: My dialog has an entry field (actualla a couple of them) which reads data from the database. Lots of objects are being read in order to make suggestions in this entry field. And there are certain business checks going on on this object and potentially lots of referenced objects. They all get read from the database. That is not a performance problem per se, but they all end up in the unitOfWork's cache and undoMap and whatnot. So the effect of this is close to zero, as long as these objects are read within the currentUnitOfWork.

The solution to this sounds easy: postpone the beginUnitOfWork until something is actually changed in the business object. But....

• this also means postponing the refresh of the business object until we actually change it
  
• this also means we need some kind of  transient memento for the business object and all objects that may be modified along the way while the user works on this dialog. We'd have to do something like #registerTransitiveClosure, just to find out which objects to copy as transient proxies of their persistent counterparts. Or we create such mementos by hand, which is likely to be error prone and may open flooding gates for errors that go undetected for a while (how could you possibly write tests this? "Is there any object we modified that is detached at the moment? I guess a WriteBarrier could come in handy at this problem...)

I'd be interested in opinions and ideas. How do you guys handle these issues?

I know this is not a new problem at all. This is a question every user of an ORM has to ask herself. We thought we found a good solution that carries us a long way in the use of ...AndContinue and just keeping everything, because our users typically have small numbers of objects. Until they hadn't.

Do you have pointers to literature on this complex of questions. I guess Object Transaction Framework is the keyword here? Has anybody built such a framework on top of Glorp? Is everybody implementing their own handmade ad-hoc version of this?

Joachim

alan.knight schrieb am Donnerstag, 5. November 2020 um 14:47:55 UTC+1:

I always tried to keep transactions short. So my pattern would have been that if you have a list it's not in a transaction. When someone goes to edit an object, start a unit of work, refresh that object within the unit of work and open the dialog on it. By refreshing you should force it to re-read proxies as well, so it's a little bit slower to open but known to be consistent with the DB. And try to have the dialog take a single object, and to have changes that might happen within it not affect the rest of the list. Otherwise it gets a bit more complicated.

On Thu, Nov 5, 2020 at 2:32 AM jtuchel <[hidden email]> wrote:

As you may have read in my previous question about commits for Transactions with lots of objects loeded from the database, we are having issues with big numbers of objects in the caches.

This leads me to an old question I always had when it comes to effectively use objects across Transaction borders.

Example:

1. We load a list of objects into memory in one dialog. Here we commit some change to some object (like the time somebody took a look at this list).
2. The user double clicks one of these objects and opens a dialog to make some changes to this object and commits these changes and closes the dialog
3. The initialt list of objects is updated and the user can choose to work on the details of another object

I guess this is not a very weird and strange scenario, and yet it is complicated from the perspective of an ORM. How do we make sure none of the Transactions (UOW) gets out of sync with the database?

What is the best practice for such a scenario.

1. Do we use long UOWs by using #commitUnitOfWorkAndContinue?
   
2. Is it best to use commitUnitOfWork and make sure the UOW in the dialog re-reads its object in a new UOW in order to track the changes? Simply handing the object to the dialog after a commitUOW for the logging of last access would detach the object from the database (the new Transaction doesn't know this object because it is not in the new Transaction's undoMap)
3. Do people use #inTransactionDo: and take the full responsibility of all the bookkeeping?

We chose to use the ...AndContinue variants, because we thought this is the best compromise between speed and comfort. But it seems this approach has its limits when the number of objects gets large (see the other post).

We also thought that approach #2 might be dangerous, because something like

self session commitUnitOfWork.

self showDetailsDialogFor: self selectedObject.

has the danger of detaching the selected object or some objects it references from the database. It would then probably mean it is much better to just call other dialogs with the id of the "root object" to reload for editing, just to make sure the dialog reads the object and all its referenced objects in its own Transaction...

I hope I managed to communicate my concerns and questions. I'd be interested in other peoples' approach to this tricky problem. So far we thought we chose the best possible path but I am not so sure any more....

I look forward to a fruitful discussion and hope to read your comments and ideas soon.

Joachim

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/d7198a44-01fe-4317-acde-e68b01a6af65n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/2016532e-2f74-4b46-ab06-a0bc34c7ebebn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/30116c56-3459-9612-2860-019493140421%40gmail.com.

Hi Estaban,

I would think that you need to keep the rowmaps or alternatively, a cache of all read only primarykeys for each class of read only objects. Simple schemas might not require this, but as soon as you have many-to-many relationships, you need a way to avoid loading duplicate instances. Persons with multiple addresses and addresses with multiple persons are a good example. If you're only loading a few objects this way, duplication might not matter, but if you have lots, it could lead to more bloat than the rowmaps do.

Tom

On 11/8/2020 6:05 PM, Esteban Maringolo wrote:

Hi,

I would favor an immutability based read-only session, this would save both memory and time by not keeping the rowmaps around.

For an ORM we built in another company (and that is still in use) we used Dolphin's immutability exception handling, so after reading an object was marked as immutable, and when there was an attempt to mutate it we handled the signalled exception and used that to mark the object as dirty, etc. Of course no rowmap was available in memory, and neither in memory rollbacks, but it was seldom the case that we couldn't handle that situation by reloading the object from the DB and doing a one way become if extremely necessary.

Regards,

ps: I'm currently in the need for something like this because I have small web app that uses GLORP and Seaside, so each Seaside session [1] has its own GLORP session [2], and this in turn has its own DB connection [3] as well, and for the end user the data never changes (it's read only), only the backend UI/APIs modify the data, my only concern is that the read-only sessions show the most recent data committed to the DB from other sessions. As much I have 200 concurrent sessions at peak load, but saving me from keeping all that state that I'm not going to use would be useful.

[1] This isn't harmful, and it is lightweight.

[2] I might need to pool these sessions, but I don't want to get into situations where GLORP "phases" overlap each other

[3] Pharo currently doesn't have a Pooled Database Accessor, so I'm relying on pg_bounce on the server side, and so far works transparently.

Esteban A. Maringolo

On Fri, Nov 6, 2020 at 9:01 AM Tom Robinson <[hidden email]> wrote:

Hi Joachim,

At Washington Mutual, we had this problem with GemStone. We ended up building our own mementos for objects before doing the rollback. The VAST write barrier was involved, but it's been 18 years and I don't remember the details.

Store has this problem as well. It really doesn't need the undoMap because all of the modified Store objects are derived from image resident objects of a different class. Niall and I found this shortly before I left Cincom. I don't know whether he has done something to fix it or not.  What he did is likely to be Store specific though, and not part of Glorp (I think).

I would ask Alan and yourself if it might make sense to create ReadOnlyGlorpSession as a subclass of GlorpSession. ReadOnlyGlorpSession would retrieve objects from the database and instantiate them as immutable using the VA or VW immutability. It would not register the objects in the undoMap and it would not allow units of work or transactions. It would cache objects to avoid performance issues with recursive data structures and duplicate references. This assumes that there is an obvious line between objects being modified and objects never being modified. It could be problematic if you tried to read an object in using a read only session and then tried to modify it subsequently. I probably haven't sniffed out all of the implementation complications...

Tom

On 11/6/2020 1:44 AM, jtuchel wrote:

I am currently trying to do this in one of our dialogs as a prototype. But there are still a few things I don't really get my head around.

As a first attempt. I just started my details/edit dialog with

self dbSession rollbackUnitOfWork; beginUnitOfWork; refresh: myModelObjejct.

This does work. Every object I navigate to is re-read, so it seems like the proxifying of the tranisitveClosure etc. work as expected.

There is a problem, however: My dialog has an entry field (actualla a couple of them) which reads data from the database. Lots of objects are being read in order to make suggestions in this entry field. And there are certain business checks going on on this object and potentially lots of referenced objects. They all get read from the database. That is not a performance problem per se, but they all end up in the unitOfWork's cache and undoMap and whatnot. So the effect of this is close to zero, as long as these objects are read within the currentUnitOfWork.

The solution to this sounds easy: postpone the beginUnitOfWork until something is actually changed in the business object. But....

• this also means postponing the refresh of the business object until we actually change it
  
• this also means we need some kind of  transient memento for the business object and all objects that may be modified along the way while the user works on this dialog. We'd have to do something like #registerTransitiveClosure, just to find out which objects to copy as transient proxies of their persistent counterparts. Or we create such mementos by hand, which is likely to be error prone and may open flooding gates for errors that go undetected for a while (how could you possibly write tests this? "Is there any object we modified that is detached at the moment? I guess a WriteBarrier could come in handy at this problem...)

I'd be interested in opinions and ideas. How do you guys handle these issues?

I know this is not a new problem at all. This is a question every user of an ORM has to ask herself. We thought we found a good solution that carries us a long way in the use of ...AndContinue and just keeping everything, because our users typically have small numbers of objects. Until they hadn't.

Do you have pointers to literature on this complex of questions. I guess Object Transaction Framework is the keyword here? Has anybody built such a framework on top of Glorp? Is everybody implementing their own handmade ad-hoc version of this?

Joachim

alan.knight schrieb am Donnerstag, 5. November 2020 um 14:47:55 UTC+1:

I always tried to keep transactions short. So my pattern would have been that if you have a list it's not in a transaction. When someone goes to edit an object, start a unit of work, refresh that object within the unit of work and open the dialog on it. By refreshing you should force it to re-read proxies as well, so it's a little bit slower to open but known to be consistent with the DB. And try to have the dialog take a single object, and to have changes that might happen within it not affect the rest of the list. Otherwise it gets a bit more complicated.

On Thu, Nov 5, 2020 at 2:32 AM jtuchel <[hidden email]> wrote:

As you may have read in my previous question about commits for Transactions with lots of objects loeded from the database, we are having issues with big numbers of objects in the caches.

This leads me to an old question I always had when it comes to effectively use objects across Transaction borders.

Example:

1. We load a list of objects into memory in one dialog. Here we commit some change to some object (like the time somebody took a look at this list).
2. The user double clicks one of these objects and opens a dialog to make some changes to this object and commits these changes and closes the dialog
3. The initialt list of objects is updated and the user can choose to work on the details of another object

I guess this is not a very weird and strange scenario, and yet it is complicated from the perspective of an ORM. How do we make sure none of the Transactions (UOW) gets out of sync with the database?

What is the best practice for such a scenario.

1. Do we use long UOWs by using #commitUnitOfWorkAndContinue?
   
2. Is it best to use commitUnitOfWork and make sure the UOW in the dialog re-reads its object in a new UOW in order to track the changes? Simply handing the object to the dialog after a commitUOW for the logging of last access would detach the object from the database (the new Transaction doesn't know this object because it is not in the new Transaction's undoMap)
3. Do people use #inTransactionDo: and take the full responsibility of all the bookkeeping?

We chose to use the ...AndContinue variants, because we thought this is the best compromise between speed and comfort. But it seems this approach has its limits when the number of objects gets large (see the other post).

We also thought that approach #2 might be dangerous, because something like

self session commitUnitOfWork.

self showDetailsDialogFor: self selectedObject.

has the danger of detaching the selected object or some objects it references from the database. It would then probably mean it is much better to just call other dialogs with the id of the "root object" to reload for editing, just to make sure the dialog reads the object and all its referenced objects in its own Transaction...

I hope I managed to communicate my concerns and questions. I'd be interested in other peoples' approach to this tricky problem. So far we thought we chose the best possible path but I am not so sure any more....

I look forward to a fruitful discussion and hope to read your comments and ideas soon.

Joachim

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/d7198a44-01fe-4317-acde-e68b01a6af65n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/2016532e-2f74-4b46-ab06-a0bc34c7ebebn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/30116c56-3459-9612-2860-019493140421%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/CAJMgPC%2B-RdDqAnjX0HywKgPs6iJkzL7v%2Bc-cWS2J%2BfBHEZZ3jQ%40mail.gmail.com.

Hi Tom,

But aren't these lookups going through the cache first when read? [1]

Last time I checked the `ObjectBuilder>>buildObjectsForRow:` checked whether the instance was in the cache first via `#findInstanceForRow:proxyType:`.

In any case I'm not at the point where that memory use is the next factor to optimize, as I said, pooling Glorp sessions and having "fresh" instances all the time without having to "rebuild the world" on each query are my current next things to optimize. 

I have an explicit inReadOnlyUnitOfWorkDo: to avoid any accidental mutation to be stored at the end of the block, it simply does an #rollbackUnitOfWork instead of committing.

Thanks!

[1] I actually have a "modified" version of the Cache to properly handle hierarchical type resolvers (so I have a single cache for all the hierarchy).

Esteban A. Maringolo

On Mon, Nov 9, 2020 at 1:48 PM Tom Robinson <[hidden email]> wrote:

Hi Estaban,

I would think that you need to keep the rowmaps or alternatively, a cache of all read only primarykeys for each class of read only objects. Simple schemas might not require this, but as soon as you have many-to-many relationships, you need a way to avoid loading duplicate instances. Persons with multiple addresses and addresses with multiple persons are a good example. If you're only loading a few objects this way, duplication might not matter, but if you have lots, it could lead to more bloat than the rowmaps do.

Tom

On 11/8/2020 6:05 PM, Esteban Maringolo wrote:

Hi,

I would favor an immutability based read-only session, this would save both memory and time by not keeping the rowmaps around.

For an ORM we built in another company (and that is still in use) we used Dolphin's immutability exception handling, so after reading an object was marked as immutable, and when there was an attempt to mutate it we handled the signalled exception and used that to mark the object as dirty, etc. Of course no rowmap was available in memory, and neither in memory rollbacks, but it was seldom the case that we couldn't handle that situation by reloading the object from the DB and doing a one way become if extremely necessary.

Regards,

ps: I'm currently in the need for something like this because I have small web app that uses GLORP and Seaside, so each Seaside session [1] has its own GLORP session [2], and this in turn has its own DB connection [3] as well, and for the end user the data never changes (it's read only), only the backend UI/APIs modify the data, my only concern is that the read-only sessions show the most recent data committed to the DB from other sessions. As much I have 200 concurrent sessions at peak load, but saving me from keeping all that state that I'm not going to use would be useful.

[1] This isn't harmful, and it is lightweight.

[2] I might need to pool these sessions, but I don't want to get into situations where GLORP "phases" overlap each other

[3] Pharo currently doesn't have a Pooled Database Accessor, so I'm relying on pg_bounce on the server side, and so far works transparently.

Esteban A. Maringolo

On Fri, Nov 6, 2020 at 9:01 AM Tom Robinson <[hidden email]> wrote:

Hi Joachim,

At Washington Mutual, we had this problem with GemStone. We ended up building our own mementos for objects before doing the rollback. The VAST write barrier was involved, but it's been 18 years and I don't remember the details.

Store has this problem as well. It really doesn't need the undoMap because all of the modified Store objects are derived from image resident objects of a different class. Niall and I found this shortly before I left Cincom. I don't know whether he has done something to fix it or not.  What he did is likely to be Store specific though, and not part of Glorp (I think).

I would ask Alan and yourself if it might make sense to create ReadOnlyGlorpSession as a subclass of GlorpSession. ReadOnlyGlorpSession would retrieve objects from the database and instantiate them as immutable using the VA or VW immutability. It would not register the objects in the undoMap and it would not allow units of work or transactions. It would cache objects to avoid performance issues with recursive data structures and duplicate references. This assumes that there is an obvious line between objects being modified and objects never being modified. It could be problematic if you tried to read an object in using a read only session and then tried to modify it subsequently. I probably haven't sniffed out all of the implementation complications...

Tom

On 11/6/2020 1:44 AM, jtuchel wrote:

I am currently trying to do this in one of our dialogs as a prototype. But there are still a few things I don't really get my head around.

As a first attempt. I just started my details/edit dialog with

self dbSession rollbackUnitOfWork; beginUnitOfWork; refresh: myModelObjejct.

This does work. Every object I navigate to is re-read, so it seems like the proxifying of the tranisitveClosure etc. work as expected.

There is a problem, however: My dialog has an entry field (actualla a couple of them) which reads data from the database. Lots of objects are being read in order to make suggestions in this entry field. And there are certain business checks going on on this object and potentially lots of referenced objects. They all get read from the database. That is not a performance problem per se, but they all end up in the unitOfWork's cache and undoMap and whatnot. So the effect of this is close to zero, as long as these objects are read within the currentUnitOfWork.

The solution to this sounds easy: postpone the beginUnitOfWork until something is actually changed in the business object. But....

• this also means postponing the refresh of the business object until we actually change it
  
• this also means we need some kind of  transient memento for the business object and all objects that may be modified along the way while the user works on this dialog. We'd have to do something like #registerTransitiveClosure, just to find out which objects to copy as transient proxies of their persistent counterparts. Or we create such mementos by hand, which is likely to be error prone and may open flooding gates for errors that go undetected for a while (how could you possibly write tests this? "Is there any object we modified that is detached at the moment? I guess a WriteBarrier could come in handy at this problem...)

I'd be interested in opinions and ideas. How do you guys handle these issues?

I know this is not a new problem at all. This is a question every user of an ORM has to ask herself. We thought we found a good solution that carries us a long way in the use of ...AndContinue and just keeping everything, because our users typically have small numbers of objects. Until they hadn't.

Do you have pointers to literature on this complex of questions. I guess Object Transaction Framework is the keyword here? Has anybody built such a framework on top of Glorp? Is everybody implementing their own handmade ad-hoc version of this?

Joachim

alan.knight schrieb am Donnerstag, 5. November 2020 um 14:47:55 UTC+1:

I always tried to keep transactions short. So my pattern would have been that if you have a list it's not in a transaction. When someone goes to edit an object, start a unit of work, refresh that object within the unit of work and open the dialog on it. By refreshing you should force it to re-read proxies as well, so it's a little bit slower to open but known to be consistent with the DB. And try to have the dialog take a single object, and to have changes that might happen within it not affect the rest of the list. Otherwise it gets a bit more complicated.

On Thu, Nov 5, 2020 at 2:32 AM jtuchel <[hidden email]> wrote:

As you may have read in my previous question about commits for Transactions with lots of objects loeded from the database, we are having issues with big numbers of objects in the caches.

This leads me to an old question I always had when it comes to effectively use objects across Transaction borders.

Example:

1. We load a list of objects into memory in one dialog. Here we commit some change to some object (like the time somebody took a look at this list).
2. The user double clicks one of these objects and opens a dialog to make some changes to this object and commits these changes and closes the dialog
3. The initialt list of objects is updated and the user can choose to work on the details of another object

I guess this is not a very weird and strange scenario, and yet it is complicated from the perspective of an ORM. How do we make sure none of the Transactions (UOW) gets out of sync with the database?

What is the best practice for such a scenario.

1. Do we use long UOWs by using #commitUnitOfWorkAndContinue?
   
2. Is it best to use commitUnitOfWork and make sure the UOW in the dialog re-reads its object in a new UOW in order to track the changes? Simply handing the object to the dialog after a commitUOW for the logging of last access would detach the object from the database (the new Transaction doesn't know this object because it is not in the new Transaction's undoMap)
3. Do people use #inTransactionDo: and take the full responsibility of all the bookkeeping?

We chose to use the ...AndContinue variants, because we thought this is the best compromise between speed and comfort. But it seems this approach has its limits when the number of objects gets large (see the other post).

We also thought that approach #2 might be dangerous, because something like

self session commitUnitOfWork.

self showDetailsDialogFor: self selectedObject.

has the danger of detaching the selected object or some objects it references from the database. It would then probably mean it is much better to just call other dialogs with the id of the "root object" to reload for editing, just to make sure the dialog reads the object and all its referenced objects in its own Transaction...

I hope I managed to communicate my concerns and questions. I'd be interested in other peoples' approach to this tricky problem. So far we thought we chose the best possible path but I am not so sure any more....

I look forward to a fruitful discussion and hope to read your comments and ideas soon.

Joachim

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/d7198a44-01fe-4317-acde-e68b01a6af65n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/2016532e-2f74-4b46-ab06-a0bc34c7ebebn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/30116c56-3459-9612-2860-019493140421%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/CAJMgPC%2B-RdDqAnjX0HywKgPs6iJkzL7v%2Bc-cWS2J%2BfBHEZZ3jQ%40mail.gmail.com.

Hi Joachim,

At Washington Mutual, we had this problem with GemStone. We ended up building our own mementos for objects before doing the rollback. The VAST write barrier was involved, but it's been 18 years and I don't remember the details.

Store has this problem as well. It really doesn't need the undoMap because all of the modified Store objects are derived from image resident objects of a different class. Niall and I found this shortly before I left Cincom. I don't know whether he has done something to fix it or not.  What he did is likely to be Store specific though, and not part of Glorp (I think).

I would ask Alan and yourself if it might make sense to create ReadOnlyGlorpSession as a subclass of GlorpSession. ReadOnlyGlorpSession would retrieve objects from the database and instantiate them as immutable using the VA or VW immutability. It would not register the objects in the undoMap and it would not allow units of work or transactions. It would cache objects to avoid performance issues with recursive data structures and duplicate references. This assumes that there is an obvious line between objects being modified and objects never being modified. It could be problematic if you tried to read an object in using a read only session and then tried to modify it subsequently. I probably haven't sniffed out all of the implementation complications...

Tom

On 11/6/2020 1:44 AM, jtuchel wrote:

I am currently trying to do this in one of our dialogs as a prototype. But there are still a few things I don't really get my head around.

As a first attempt. I just started my details/edit dialog with

self dbSession rollbackUnitOfWork; beginUnitOfWork; refresh: myModelObjejct.

This does work. Every object I navigate to is re-read, so it seems like the proxifying of the tranisitveClosure etc. work as expected.

There is a problem, however: My dialog has an entry field (actualla a couple of them) which reads data from the database. Lots of objects are being read in order to make suggestions in this entry field. And there are certain business checks going on on this object and potentially lots of referenced objects. They all get read from the database. That is not a performance problem per se, but they all end up in the unitOfWork's cache and undoMap and whatnot. So the effect of this is close to zero, as long as these objects are read within the currentUnitOfWork.

The solution to this sounds easy: postpone the beginUnitOfWork until something is actually changed in the business object. But....

• this also means postponing the refresh of the business object until we actually change it
  
• this also means we need some kind of  transient memento for the business object and all objects that may be modified along the way while the user works on this dialog. We'd have to do something like #registerTransitiveClosure, just to find out which objects to copy as transient proxies of their persistent counterparts. Or we create such mementos by hand, which is likely to be error prone and may open flooding gates for errors that go undetected for a while (how could you possibly write tests this? "Is there any object we modified that is detached at the moment? I guess a WriteBarrier could come in handy at this problem...)

I'd be interested in opinions and ideas. How do you guys handle these issues?

I know this is not a new problem at all. This is a question every user of an ORM has to ask herself. We thought we found a good solution that carries us a long way in the use of ...AndContinue and just keeping everything, because our users typically have small numbers of objects. Until they hadn't.

Do you have pointers to literature on this complex of questions. I guess Object Transaction Framework is the keyword here? Has anybody built such a framework on top of Glorp? Is everybody implementing their own handmade ad-hoc version of this?

Joachim

alan.knight schrieb am Donnerstag, 5. November 2020 um 14:47:55 UTC+1:

I always tried to keep transactions short. So my pattern would have been that if you have a list it's not in a transaction. When someone goes to edit an object, start a unit of work, refresh that object within the unit of work and open the dialog on it. By refreshing you should force it to re-read proxies as well, so it's a little bit slower to open but known to be consistent with the DB. And try to have the dialog take a single object, and to have changes that might happen within it not affect the rest of the list. Otherwise it gets a bit more complicated.

On Thu, Nov 5, 2020 at 2:32 AM jtuchel <[hidden email]> wrote:

As you may have read in my previous question about commits for Transactions with lots of objects loeded from the database, we are having issues with big numbers of objects in the caches.

This leads me to an old question I always had when it comes to effectively use objects across Transaction borders.

Example:

1. We load a list of objects into memory in one dialog. Here we commit some change to some object (like the time somebody took a look at this list).
2. The user double clicks one of these objects and opens a dialog to make some changes to this object and commits these changes and closes the dialog
3. The initialt list of objects is updated and the user can choose to work on the details of another object

I guess this is not a very weird and strange scenario, and yet it is complicated from the perspective of an ORM. How do we make sure none of the Transactions (UOW) gets out of sync with the database?

What is the best practice for such a scenario.

1. Do we use long UOWs by using #commitUnitOfWorkAndContinue?
   
2. Is it best to use commitUnitOfWork and make sure the UOW in the dialog re-reads its object in a new UOW in order to track the changes? Simply handing the object to the dialog after a commitUOW for the logging of last access would detach the object from the database (the new Transaction doesn't know this object because it is not in the new Transaction's undoMap)
3. Do people use #inTransactionDo: and take the full responsibility of all the bookkeeping?

We chose to use the ...AndContinue variants, because we thought this is the best compromise between speed and comfort. But it seems this approach has its limits when the number of objects gets large (see the other post).

We also thought that approach #2 might be dangerous, because something like

self session commitUnitOfWork.

self showDetailsDialogFor: self selectedObject.

has the danger of detaching the selected object or some objects it references from the database. It would then probably mean it is much better to just call other dialogs with the id of the "root object" to reload for editing, just to make sure the dialog reads the object and all its referenced objects in its own Transaction...

I hope I managed to communicate my concerns and questions. I'd be interested in other peoples' approach to this tricky problem. So far we thought we chose the best possible path but I am not so sure any more....

I look forward to a fruitful discussion and hope to read your comments and ideas soon.

Joachim

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/d7198a44-01fe-4317-acde-e68b01a6af65n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "glorp-group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

To view this discussion on the web visit https://groups.google.com/d/msgid/glorp-group/2016532e-2f74-4b46-ab06-a0bc34c7ebebn%40googlegroups.com.