voyage mongo and transactionality

I have used OmniBase extensively long ago with Dolphin  Smalltalk (the Dolphin repository can be read & writen by Visual  Works too).

Even the Dolphin repository can be under Windows and Linux (using Samba). You can distribute instances ODBContainer through a LAN.

OmniBase use "multiversion concurrency control" to hold different version of the same object (in different transactions).

OmniBase is pretty cool but if need a lot of indexes (indexes on more than one inst var ) you have to solve the problem yourself using OmniBase b-trees.

regards

bruno

It may be irrelevant, but I have been playing recently with OmniBase, which is a fully object-oriented database system, now over 20 years old, but it still works very well for my uses. David Gorišek, the author, claims that it has ACID properties. From my reading, updates operate on a proxy object, which is not written to the database until an explicit commit is given. A second transaction accessing the same object will still see the original until the change is committed. What happens to a proxy which is never committed is not clear, but if Gorišek is right, the stored data can never be contaminated. I think a proxy in this sense is equivalent to a memento.

 

Thanks to Esteban Lorenzano, OmniBase is now available on Pharo. The code is ancient, there is no documentation and obviously no support, but it might be worth while for someone to try some software archaeology and put it to use. I have found it possible to create and maintain a small database of natural language information, and access is fairly quick and easy – and it’s all Smalltalk.

It claims to store all kinds of Smalltalk objects, except block closures, and skimming through the code it seems to incorporate a serializer similar to Fuel.

 

The only documentation I have found is a slideshow at https://www.slideshare.net/esug/omni-baseobjectdatabase. I have found out a few things about it, if anyone is interested.

 

Peter Kenny

 

 

From: Pharo-users [hidden email] On Behalf Of Norbert Hartl
Sent: 09 October 2019 18:08
To: Any question about pharo is welcome [hidden email]
Subject: Re: [Pharo-users] voyage mongo and transactionality

 

 

Am 09.10.2019 um 16:48 schrieb "[hidden email]" <[hidden email]>:

 

This is a tricky mine field. Sometimes you need a lot of business functionality in objects referenced in your objects that are currently in the editor. So I'm still to see a project in which the memento pattern really worked for more complex scenarios. How deep do you dive to have enough memento objects to provide the functionality needed. I guess you can do that with some sort of object-level transaction framework that automatically creates mementos of whatever object is being navigated to during some kind of processing-context. I guess slots could be of use here. But this is not trivial for general cases.

 

Yes it is tricky. You can have copies of business objects but you have always references to the business objects not pointing to the copy. 

And you need to know which objects should be tracked. In Gemstone IIRC it is easy as it is the time the object is copied from the stone to the gem it is registered in the current transaction. So you can check it and committing if it changed because you have to write it back. The important point here might be get noticed when a reference is acquired. In pharo it is not that easy but could be done if object would be reified and interceptable. 

In my experience, this problem area makes for the other 70% of the time spent on developing GUI or Web applications, besides the 60% for GUI design and implementation and 25% business logic...

70% + 60% + 25% + 30% = 185%

 

sounds indeed very realistic if it comes to project planning. 😛There is even a rule saying that for the first 90% of the project you need the first 90% of time and for the last 10% of the project you need the second 90% of time. 

I'd be interested to learn about patterns to handle such more complex things. We constantly travel back and forth between implementing stuff in the GUI handlers (copying values to the GUI classes that access themselves during GUI operations and push values to the business objects when the users clicks on OK), using mementos (which most of the times are nets of mementos that are created manually - "we know what we'll touch in this Editor") and operating on business objects directly and relying on the persistence mechanism (Glorp in our case) and its rollback behaviour. All three have lots of weaknesses and seem to have their place nevertheless.

So this is a very interesting discussion and I think this is an area that has not been solved yet.

I think it isn‘t solved and I find every piece of information about it very interesting.

 

Norbert

Joachim

 

 

 

 

 

 

Am 09.10.19 um 16:25 schrieb James Foster:

Thanks for the explanation. And, yes, this is an artifact of your design; if you put intermediate values into domain objects then they will remain in your domain objects to be seen later. >From what you’ve described, I don’t see how it would be any different in a non-image environment (Java, C#, etc.), unless you re-read the entire object graph from the database. As someone else mentioned, this would be a good place for the Memento Pattern.

 

James

On Oct 9, 2019, at 1:59 AM, Jonathan van Alteren <[hidden email]> wrote:

 

Hi James,

 

I see how my explanation might be unclear.

 

We have a main form for the agenda and a subform for an item, which is shown using Seaside call/answer. The save button of the subform is clicked, which adds the item to the underlying agenda model object, but the save button of the main form is not clicked by the user. The callback for the main save button sends the save message to the agenda object, causing the database to be updated.

 

So yes, the browser does submit the data on the subform, it's the main form component that doesn't receive the save button callback. I realize that this is in large part an issue with our design. However, the way object persistence seems to work in the image environment plays a large role. 

 

 

Kind regards,

Jonathan van Alteren

Founding Member | Object Guild
[hidden email]

On 8 Oct 2019, 15:41 +0200, James Foster <[hidden email]>, wrote:

On Oct 8, 2019, at 3:05 AM, Jonathan van Alteren <[hidden email]> wrote:

We've encountered an issue where a user makes changes to an agenda, but does not click the Save button. Instead, the user closes the browser or uses the navigation to go to a different part of the application. When navigating back to the original agenda, the changes made previously (e.g. items added) are still being displayed, even though they were never explicitly saved.

Here is what I don’t understand: how did the change get from the user’s client agent (browser) to the server? If you make a change to a field in a form and then close the browser, who sent the change to the server? If you show the save domain value in a different location, with a dynamically-generated id and name (so it isn’t cached in the browser), or written to the Pharo Transcript, does the value still change? That is, are you sure that the change is in the reflected in the Smalltalk image and not just somehow cached in the browser?

James

 

 

-- 

-----------------------------------------------------------------------

Objektfabrik Joachim Tuchel          [hidden email]

Fliederweg 1                         http://www.objektfabrik.de

D-71640 Ludwigsburg                  http://joachimtuchel.wordpress.com

Telefon: +49 7141 56 10 86 0         Fax: +49 7141 56 10 86 1

Hi Joachim,

Thank you for your feedback.

It feels good to know we're not alone in this :-)

Well, I guess there are far more people in this boat than actually know they are ;-)

Unfortunately, the things you describe are familiar to me. My business partner Dave West has a lot of experience with applying a behavioral, 'pure' object design approach. We're looking hard into simplifying these matters through the application of inversion of control and by making objects as autonomous as they can possible be.

sigh, This is really, really hard. You'd need a guard in the background that autonomously register all objects that get visited/accessed/modified during an operation and create a memento just before this actually happens. Ideally, this would only happen if the upcoming operation on that objects modifies its state once the operation happens. Or ot would have to clean up non-modified ones after the operation finished. I guess this second alternative is even feasible using slots and such, but there remain a few questions about synchronizing all this and the underlying database transactions...

At the same time, we haven't found the pot of gold at the end of the rainbow yet ;-)

You know, sometimes I fear it is better not to have anybody come back from that end of the rainbow. They might have found out there is no pot or anything similar to one ;-)

Nowadays, a fair amount of applications have a very direct way of handling user interactions and resulting state changes. For example, the settings of my Firefox browser doesn't have a save button anymore. Anything I change looks to be automatically persisted. Perhaps there is some value in that approach for 'enterprise' applications as well, although I think there will remain a lot of use cases where an explicit save will be needed for whatever reason.

Hmm. I am afraid there is a certain distance between changes that only affect a single setting value and more complex values that imply a lot of recomputation of values in a more complex model. Sometimes you also need to modify or at least check consequences of changes that don't fit onto a single screen, and there you need to find ways to allow a user to step back from what they just started to do...

Let's keep talking about this. I need to do some research on the memento pattern first :-) If you have any information sources you can point me to, I would greatly appreciate that.

I don't follow any interesting research on mementos. Too much work to do for that. The memento pattern itself is just "make a copy that the user can play with without any consequences to the original object". Works perfect for a single object or a well-defined object graph. But I am not aware of any working solutions that would solve our problems for more complex graphs, or even a general one... If you find something, let us know!

Joachim

Kind regards,

Jonathan van Alteren

Founding Member | Object Guild
[hidden email]

On 9 Oct 2019, 16:49 +0200, [hidden email] [hidden email], wrote:

This is a tricky mine field. Sometimes you need a lot of business functionality in objects referenced in your objects that are currently in the editor. So I'm still to see a project in which the memento pattern really worked for more complex scenarios. How deep do you dive to have enough memento objects to provide the functionality needed. I guess you can do that with some sort of object-level transaction framework that automatically creates mementos of whatever object is being navigated to during some kind of processing-context. I guess slots could be of use here. But this is not trivial for general cases.

In my experience, this problem area makes for the other 70% of the time spent on developing GUI or Web applications, besides the 60% for GUI design and implementation and 25% business logic...

I'd be interested to learn about patterns to handle such more complex things. We constantly travel back and forth between implementing stuff in the GUI handlers (copying values to the GUI classes that access themselves during GUI operations and push values to the business objects when the users clicks on OK), using mementos (which most of the times are nets of mementos that are created manually - "we know what we'll touch in this Editor") and operating on business objects directly and relying on the persistence mechanism (Glorp in our case) and its rollback behaviour. All three have lots of weaknesses and seem to have their place nevertheless.

So this is a very interesting discussion and I think this is an area that has not been solved yet.

Joachim

Am 09.10.19 um 16:25 schrieb James Foster:

Thanks for the explanation. And, yes, this is an artifact of your design; if you put intermediate values into domain objects then they will remain in your domain objects to be seen later. From what you’ve described, I don’t see how it would be any different in a non-image environment (Java, C#, etc.), unless you re-read the entire object graph from the database. As someone else mentioned, this would be a good place for the Memento Pattern.

James

On Oct 9, 2019, at 1:59 AM, Jonathan van Alteren <[hidden email]> wrote:

Hi James,

I see how my explanation might be unclear.

We have a main form for the agenda and a subform for an item, which is shown using Seaside call/answer. The save button of the subform is clicked, which adds the item to the underlying agenda model object, but the save button of the main form is not clicked by the user. The callback for the main save button sends the save message to the agenda object, causing the database to be updated.

So yes, the browser does submit the data on the subform, it's the main form component that doesn't receive the save button callback. I realize that this is in large part an issue with our design. However, the way object persistence seems to work in the image environment plays a large role. 

Kind regards,

Jonathan van Alteren

Founding Member | Object Guild
[hidden email]

On 8 Oct 2019, 15:41 +0200, James Foster <[hidden email]>, wrote:

On Oct 8, 2019, at 3:05 AM, Jonathan van Alteren <[hidden email]> wrote:

We've encountered an issue where a user makes changes to an agenda, but does not click the Save button. Instead, the user closes the browser or uses the navigation to go to a different part of the application. When navigating back to the original agenda, the changes made previously (e.g. items added) are still being displayed, even though they were never explicitly saved.

Here is what I don’t understand: how did the change get from the user’s client agent (browser) to the server? If you make a change to a field in a form and then close the browser, who sent the change to the server? If you show the save domain value in a different location, with a dynamically-generated id and name (so it isn’t cached in the browser), or written to the Pharo Transcript, does the value still change? That is, are you sure that the change is in the reflected in the Smalltalk image and not just somehow cached in the browser?

James

--  
-----------------------------------------------------------------------
Objektfabrik Joachim Tuchel          [hidden email]
Fliederweg 1                         http://www.objektfabrik.de
D-71640 Ludwigsburg                  http://joachimtuchel.wordpress.com
Telefon: +49 7141 56 10 86 0         Fax: +49 7141 56 10 86 1

-- 
-----------------------------------------------------------------------
Objektfabrik Joachim Tuchel          [hidden email]
Fliederweg 1                         http://www.objektfabrik.de
D-71640 Ludwigsburg                  http://joachimtuchel.wordpress.com
Telefon: +49 7141 56 10 86 0         Fax: +49 7141 56 10 86 1

Hi guys,

I came to this post today due to having what can be described as an
unexpected voyage behavior and was greatly impressed by all the detailed
descriptions and explanations. Thanks everybody for that!

Also thanks Peter for mentioning that port I did back in the day (~2009) for
using OmniBase [1] in Aiflowing [2] where it achieved sub-second page
renderings with a full ACID, multi-image scalable setup. Is not actively
maintained but I think is a piece of software that is worth not having it
lost. The concept alone, filesystem based B-Tree object persistance, is neat
and can be the foundation of a lot of stuff.

Now I wonder what Esteban's fork has that couldn't be a merge request, I
would gladly take a look if open.

I feel bad about you loosing time with testBTreeKeyLocking because you won't
find the issue inside smalltalk. As it depends on the OS' file system to
detect concurrent changes in an object, you need the kernel of that system
to be supporting mandatory lockings [3] which is a feature that linux and
macOS does not have turned on by default. As mentioned in the readme of the
project you need to activate it in order to OmniBase raise an exception
preventing you of a race condition writing an object (coming it from a
transaction from another image or from another thread in that very image)
that would create db corruption or inconsistency. Note that, with mandatory
locking active in your OS kernel, the transaction that comes later will
raise an exception hence that test will pass. By the way, Windows has
mandatory locking by default.

*Back to the issue about voyage* here, I see the whole issue happening
simply because Voyage is caching by default hence forcing all Voyage users
to deal with one of the nastiest problems in computing: cache invalidation.

Yesterday I faced this issue:
1. changed a mongo document from a mongo client and
2. querying it from Voyage revealed different state (the state before 1).

This makes no sense at all. It makes Voyage not to be useful for
interoperation with other applications and languages, it makes it impossible
to scale an application in more than one Pharo image (all the main well
proven cloud platforms requires stateless built artifacts to horizontally
scale apps), on top of the lack of horizontal scaling issue there is the
issue of high availability that now you can't do because you are forced to
have a single point of failure (one vm behind the load balancer instead of
N).

But all those issues go away if Voyage simply does what a persistent client
is normally expected to do: save and query documents in the mongo
repository. By trying to do too much, like deciding that caching all inside
the session open in a given image is a valid design, it actually becomes
less powerful and surprises users in the worst way. Might be valid for a
prototype but certainly not for high-availability and high-load demanding
applications in production.

Caching is a great thing but is an application design decision, not a
persistency client decision. That concern is not for the client but for the
engineers who use it to make an application that might have /some/ of its
persisted models cached but certainly not all, except on very limited
scenarios. If you think about it, this is how things become niche.

Now in an application where I'm using Voyage expecting that queries return
what is actually in mongo I have to architect things in a way that
workaround this issue, like purging that cache somehow with some kind of
policy (time or a given event).

I like Voyage and I'm grateful for all the effort Esteban and the
contributors are putting into it. I might add a merge request myself. Is
documented, works well, has really nice features and is intuitive. It's just
this /cache all by default/ part that I don't think belongs to it and if
removed it would be a more powerful mongo client.



[1] https://github.com/sebastianconcept/OmniBase
[2] https://pharo.org/success/AirFlowing
[3] In *nix systems it requires mandatory file locking, so be sure you use
mand on your fstab.
https://www.hackinglinuxexposed.com/articles/20030623.html



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html