Voyage: how to delete references to objects.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Voyage: how to delete references to objects.

Olivier Auverlot
Hi,

I'm using Voyage in a Pharo application.

I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.

How can I remove properly a book and the reference to the book ?

Best regards
Olivier ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

NorbertHartl

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert


Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

Olivier Auverlot
thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert



Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

NorbertHartl

Am 17.04.2014 um 15:05 schrieb olivier auverlot <[hidden email]>:

thanks Robert for the explications.

It’s Norbert btw. :)

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

You need to explicitly save it as we don’t have something like write barriers in the image. So

aComicCollection 
removeBook: aBook;
save

is needed.

Norbert

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert




Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

Olivier Auverlot
Norbert,

Sorry for my mistake ;-)



2014-04-17 15:18 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 15:05 schrieb olivier auverlot <[hidden email]>:

thanks Robert for the explications.

It’s Norbert btw. :)

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

You need to explicitly save it as we don’t have something like write barriers in the image. So

aComicCollection 
removeBook: aBook;
save

is needed.

Norbert

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert





Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

Sabine Manaa
In reply to this post by Olivier Auverlot
Hi Olivier,

possibly the option 
VOMongoContainer>>enableMissingContent;
is useful to avoid 

Here is an example:

Regards
Sabine




On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <[hidden email]> wrote:
thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:


Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert






If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html
To start a new topic under Pharo Smalltalk Users, email [hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

NorbertHartl
Sabine,

I read the case now the first time. Just one note. Having references in both directions is probably not the best idea if using a non-object store. For navigating the graph it might be good but for persisting it is not. The serialization is a lot more complex if you do and the gain is nothing. For most databases with proper indexes you can request both directions easily and fast. So having one reference pointing from A to B you get the opposite direction for free. 

Norbert

Am 17.04.2014 um 15:22 schrieb Sabine Knöfel <[hidden email]>:

Hi Olivier,

possibly the option 
VOMongoContainer>>enableMissingContent;
is useful to avoid 

Here is an example:

Regards
Sabine




On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:


Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert






If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html
To start a new topic under Pharo Smalltalk Users, email <a href="x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=1" target="_top" rel="nofollow" link="external">[hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML



View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

Sabine Manaa
Hi Norbert,

for me the gain within smalltalk is huge because I want to navigate from person to trip and from trip to person by the attributes without asking mongo for the correspondent object via indexes.

Don't you think that this is a question of design of the object model and should not be decided by database arguments? 

Regards
Sabine


On Thu, Apr 17, 2014 at 3:40 PM, NorbertHartl [via Smalltalk] <[hidden email]> wrote:
Sabine,

I read the case now the first time. Just one note. Having references in both directions is probably not the best idea if using a non-object store. For navigating the graph it might be good but for persisting it is not. The serialization is a lot more complex if you do and the gain is nothing. For most databases with proper indexes you can request both directions easily and fast. So having one reference pointing from A to B you get the opposite direction for free. 

Norbert

Am 17.04.2014 um 15:22 schrieb Sabine Knöfel <[hidden email]>:

Hi Olivier,

possibly the option 
VOMongoContainer>>enableMissingContent;
is useful to avoid 

Here is an example:

Regards
Sabine




On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:


Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert






If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html
To start a new topic under Pharo Smalltalk Users, email <a href="x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=1" target="_top" rel="nofollow" link="external">[hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML



View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.




If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755102.html
To start a new topic under Pharo Smalltalk Users, email [hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

NorbertHartl

Am 17.04.2014 um 15:52 schrieb Sabine Knöfel <[hidden email]>:

Hi Norbert,

for me the gain within smalltalk is huge because I want to navigate from person to trip and from trip to person by the attributes without asking mongo for the correspondent object via indexes.

Don't you think that this is a question of design of the object model and should not be decided by database arguments? 

I think there isn’t only one _or_ the other. Of course it is a lot more beautiful to have set up proper objects and collections and do everything in a smalltalkish way. That does only count as long as you stay on the island (smalltalk image). As soon as you start to play with the outer world you have to change your rules. If you agree it isn’t perfectly beautiful if you have a proper object model that freezes your image, right? :)
IMHO the beauty comes from a mixture of both that fit them together without restricting one or the other.

So I would try to avoid storing both directions. But which one to skip? If I assume that a person can have a lot of trips than I would skip the collection inside Person. The more trips a Person has the bigger the serialized document becomes. This makes it first slower to operate on the database and will fail if you reach a document size of 16MB because that is the maximum Mongo supports. Most of the time this will be a non-issue but for now I take it into account.
So the data model will have a class Trip with an instVar pointing to Person. And Person will have an instVar trips that is transient (see VOMongoTransientDescription). The implementation for Person then goes

Person>>#trips
^ trips ifNil: [ trips := self fetchTrips ]

Person>>#fetchTrips
^ Trips selectMany: [: each| … ]

Person>>#addTrip: aTrip
aTrip person: self.
aTrip save

So the Trip will have a reference to Person in the database but Person will have no reference to Trip. Person will retrieve its trips as soon as they are used in a really smalltalkish way :). Person acts as a smalltalk object and as a database object cache at the same time.
So this an implementation of Person that on the inside takes the database nature in regard but on the outside the interface is pure smalltalk.

Still not good?

Norbert


Regards
Sabine


On Thu, Apr 17, 2014 at 3:40 PM, NorbertHartl [via Smalltalk] <<a href="x-msg://58/user/SendEmail.jtp?type=node&amp;node=4755106&amp;i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
Sabine,

I read the case now the first time. Just one note. Having references in both directions is probably not the best idea if using a non-object store. For navigating the graph it might be good but for persisting it is not. The serialization is a lot more complex if you do and the gain is nothing. For most databases with proper indexes you can request both directions easily and fast. So having one reference pointing from A to B you get the opposite direction for free. 

Norbert

Am 17.04.2014 um 15:22 schrieb Sabine Knöfel <[hidden email]>:

Hi Olivier,

possibly the option 
VOMongoContainer>>enableMissingContent;
is useful to avoid 

Here is an example:

Regards
Sabine




On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <<a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;amp;node=4755097&amp;amp;i=0">x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:


Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert






If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html
To start a new topic under Pharo Smalltalk Users, email <a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;amp;node=4755097&amp;amp;i=1">x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=1" target="_top" rel="nofollow" link="external">[hidden email] 
To unsubscribe from Pharo Smalltalk Users, click here.
NAML



View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.




If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755102.html
To start a new topic under Pharo Smalltalk Users, email <a href="x-msg://58/user/SendEmail.jtp?type=node&amp;node=4755106&amp;i=1" target="_top" rel="nofollow" link="external">[hidden email] 
To unsubscribe from Pharo Smalltalk Users, click here.
NAML



View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

EstebanLM
In reply to this post by NorbertHartl

On 17 Apr 2014, at 15:18, Norbert Hartl <[hidden email]> wrote:


Am 17.04.2014 um 15:05 schrieb olivier auverlot <[hidden email]>:

thanks Robert for the explications.

It’s Norbert btw. :)

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

You need to explicitly save it as we don’t have something like write barriers in the image. So

aComicCollection 
removeBook: aBook;
save

is needed.

yes, and also (if you want to be clean):

aBook remove. 

(assuming aBook is also persistent). 

But as Sabina points: #enableMissingContent in CommicCollection will help you to simplify things, then you just need: "aBook remove", and next time you read aComicCollection it will “clean” the invalid reference. 
I call that “eventual integrity” :P 
but beware of it… is considered “power voyage programming” :)

Esteban


Norbert

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert





Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

EstebanLM
In reply to this post by NorbertHartl

On 17 Apr 2014, at 16:26, Norbert Hartl <[hidden email]> wrote:


Am 17.04.2014 um 15:52 schrieb Sabine Knöfel <[hidden email]>:

Hi Norbert,

for me the gain within smalltalk is huge because I want to navigate from person to trip and from trip to person by the attributes without asking mongo for the correspondent object via indexes.

Don't you think that this is a question of design of the object model and should not be decided by database arguments? 

I think there isn’t only one _or_ the other. Of course it is a lot more beautiful to have set up proper objects and collections and do everything in a smalltalkish way. That does only count as long as you stay on the island (smalltalk image). As soon as you start to play with the outer world you have to change your rules. If you agree it isn’t perfectly beautiful if you have a proper object model that freezes your image, right? :)
IMHO the beauty comes from a mixture of both that fit them together without restricting one or the other.

So I would try to avoid storing both directions. But which one to skip? If I assume that a person can have a lot of trips than I would skip the collection inside Person. The more trips a Person has the bigger the serialized document becomes. This makes it first slower to operate on the database and will fail if you reach a document size of 16MB because that is the maximum Mongo supports. Most of the time this will be a non-issue but for now I take it into account.
So the data model will have a class Trip with an instVar pointing to Person. And Person will have an instVar trips that is transient (see VOMongoTransientDescription). The implementation for Person then goes

Person>>#trips
^ trips ifNil: [ trips := self fetchTrips ]

Person>>#fetchTrips
^ Trips selectMany: [: each| … ]

Person>>#addTrip: aTrip
aTrip person: self.
aTrip save

So the Trip will have a reference to Person in the database but Person will have no reference to Trip. Person will retrieve its trips as soon as they are used in a really smalltalkish way :). Person acts as a smalltalk object and as a database object cache at the same time.
So this an implementation of Person that on the inside takes the database nature in regard but on the outside the interface is pure smalltalk.

Still not good?

yes, no good :)
I mean: I understand there are times when you need it, but I would prefer a framework to solve that for me (then I can think just in my objects). 
And if the framework does not do a good job, then is a bad framework (even if made by me :P) and it needs to be improved. 

having to think in persistence is a cancer that needs to be removed :P

Esteban


Norbert


Regards
Sabine


On Thu, Apr 17, 2014 at 3:40 PM, NorbertHartl [via Smalltalk] <<a href="x-msg://58/user/SendEmail.jtp?type=node&amp;node=4755106&amp;i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
Sabine,

I read the case now the first time. Just one note. Having references in both directions is probably not the best idea if using a non-object store. For navigating the graph it might be good but for persisting it is not. The serialization is a lot more complex if you do and the gain is nothing. For most databases with proper indexes you can request both directions easily and fast. So having one reference pointing from A to B you get the opposite direction for free. 

Norbert

Am 17.04.2014 um 15:22 schrieb Sabine Knöfel <[hidden email]>:

Hi Olivier,

possibly the option 
VOMongoContainer>>enableMissingContent;
is useful to avoid 

Here is an example:

Regards
Sabine




On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <<a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;amp;node=4755097&amp;amp;i=0">x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:


Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert






If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html
To start a new topic under Pharo Smalltalk Users, email <a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;amp;node=4755097&amp;amp;i=1">x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=1" target="_top" rel="nofollow" link="external">[hidden email] 
To unsubscribe from Pharo Smalltalk Users, click here.
NAML



View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.




If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755102.html
To start a new topic under Pharo Smalltalk Users, email <a href="x-msg://58/user/SendEmail.jtp?type=node&amp;node=4755106&amp;i=1" target="_top" rel="nofollow" link="external">[hidden email] 
To unsubscribe from Pharo Smalltalk Users, click here.
NAML



View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.


Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

NorbertHartl

Am 17.04.2014 um 18:05 schrieb Esteban Lorenzano <[hidden email]>:


On 17 Apr 2014, at 16:26, Norbert Hartl <[hidden email]> wrote:


Am 17.04.2014 um 15:52 schrieb Sabine Knöfel <[hidden email]>:

Hi Norbert,

for me the gain within smalltalk is huge because I want to navigate from person to trip and from trip to person by the attributes without asking mongo for the correspondent object via indexes.

Don't you think that this is a question of design of the object model and should not be decided by database arguments? 

I think there isn’t only one _or_ the other. Of course it is a lot more beautiful to have set up proper objects and collections and do everything in a smalltalkish way. That does only count as long as you stay on the island (smalltalk image). As soon as you start to play with the outer world you have to change your rules. If you agree it isn’t perfectly beautiful if you have a proper object model that freezes your image, right? :)
IMHO the beauty comes from a mixture of both that fit them together without restricting one or the other.

So I would try to avoid storing both directions. But which one to skip? If I assume that a person can have a lot of trips than I would skip the collection inside Person. The more trips a Person has the bigger the serialized document becomes. This makes it first slower to operate on the database and will fail if you reach a document size of 16MB because that is the maximum Mongo supports. Most of the time this will be a non-issue but for now I take it into account.
So the data model will have a class Trip with an instVar pointing to Person. And Person will have an instVar trips that is transient (see VOMongoTransientDescription). The implementation for Person then goes

Person>>#trips
^ trips ifNil: [ trips := self fetchTrips ]

Person>>#fetchTrips
^ Trips selectMany: [: each| … ]

Person>>#addTrip: aTrip
aTrip person: self.
aTrip save

So the Trip will have a reference to Person in the database but Person will have no reference to Trip. Person will retrieve its trips as soon as they are used in a really smalltalkish way :). Person acts as a smalltalk object and as a database object cache at the same time.
So this an implementation of Person that on the inside takes the database nature in regard but on the outside the interface is pure smalltalk.

Still not good?

yes, no good :)
I mean: I understand there are times when you need it, but I would prefer a framework to solve that for me (then I can think just in my objects). 
And if the framework does not do a good job, then is a bad framework (even if made by me :P) and it needs to be improved. 

Sure. I was talking about _real_ stuff you can do _now_. I like to think just in objects, too. But we have a separation in hard disc and memory so we need to take care about persistence. I would also prefer to think just in programming and not have to use a stupid device like a keyboard and a mouse just in order to type silly characters that when interpreted probably come close to what I meant. But for the time being I need to stick to it, I guess.

having to think in persistence is a cancer that needs to be removed :P

Yep, we just need a common memory space where we can waste Zetabytes of memory to do little programs. Ah, and did I mention it needs to be infinitely fast because I have a certain use case and I don’t like to optimize it because I think that is cancer, too? Very academic view, well done! :)

Norbert

Esteban


Norbert


Regards
Sabine


On Thu, Apr 17, 2014 at 3:40 PM, NorbertHartl [via Smalltalk] <<a href="x-msg://58/user/SendEmail.jtp?type=node&amp;node=4755106&amp;i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
Sabine,

I read the case now the first time. Just one note. Having references in both directions is probably not the best idea if using a non-object store. For navigating the graph it might be good but for persisting it is not. The serialization is a lot more complex if you do and the gain is nothing. For most databases with proper indexes you can request both directions easily and fast. So having one reference pointing from A to B you get the opposite direction for free. 

Norbert

Am 17.04.2014 um 15:22 schrieb Sabine Knöfel <[hidden email]>:

Hi Olivier,

possibly the option 
VOMongoContainer>>enableMissingContent;
is useful to avoid 

Here is an example:

Regards
Sabine




On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <<a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;amp;node=4755097&amp;amp;i=0">x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:


Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert






If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html
To start a new topic under Pharo Smalltalk Users, email <a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;amp;node=4755097&amp;amp;i=1">x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=1" target="_top" rel="nofollow" link="external">[hidden email] 
To unsubscribe from Pharo Smalltalk Users, click here.
NAML



View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.




If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755102.html
To start a new topic under Pharo Smalltalk Users, email <a href="x-msg://58/user/SendEmail.jtp?type=node&amp;node=4755106&amp;i=1" target="_top" rel="nofollow" link="external">[hidden email] 
To unsubscribe from Pharo Smalltalk Users, click here.
NAML



View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

pharo4Stef@free.fr
In reply to this post by EstebanLM
esteban

you should do a pass on the voyage documentation chapter and add this kind of infomration.
We can pair write if you want.

stef

On 17/4/14 17:56, Esteban Lorenzano wrote:

On 17 Apr 2014, at 15:18, Norbert Hartl <[hidden email]> wrote:


Am 17.04.2014 um 15:05 schrieb olivier auverlot <[hidden email]>:

thanks Robert for the explications.

It’s Norbert btw. :)

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

You need to explicitly save it as we don’t have something like write barriers in the image. So

aComicCollection 
removeBook: aBook;
save

is needed.

yes, and also (if you want to be clean):

aBook remove. 

(assuming aBook is also persistent). 

But as Sabina points: #enableMissingContent in CommicCollection will help you to simplify things, then you just need: "aBook remove", and next time you read aComicCollection it will “clean” the invalid reference. 
I call that “eventual integrity” :P 
but beware of it… is considered “power voyage programming” :)

Esteban


Norbert

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert






Reply | Threaded
Open this post in threaded view
|

Re: Voyage: how to delete references to objects.

Olivier Auverlot
Hi,

This morning, I arrived early at office because I was impatient to test the proposed solutions. Thank for your help, my problem is solved and I understand better the usage of Voyage and Mongo.

We learn something each day and it's cool !

Best regards
Olivier ;-) 


2014-04-17 21:56 GMT+02:00 [hidden email] <[hidden email]>:
esteban

you should do a pass on the voyage documentation chapter and add this kind of infomration.
We can pair write if you want.

stef


On 17/4/14 17:56, Esteban Lorenzano wrote:

On 17 Apr 2014, at 15:18, Norbert Hartl <[hidden email]> wrote:


Am 17.04.2014 um 15:05 schrieb olivier auverlot <[hidden email]>:

thanks Robert for the explications.

It’s Norbert btw. :)

I agree with you that's the best way to remove a book is to delete the reference in  ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

You need to explicitly save it as we don’t have something like write barriers in the image. So

aComicCollection 
removeBook: aBook;
save

is needed.

yes, and also (if you want to be clean):

aBook remove. 

(assuming aBook is also persistent). 

But as Sabina points: #enableMissingContent in CommicCollection will help you to simplify things, then you just need: "aBook remove", and next time you read aComicCollection it will “clean” the invalid reference. 
I call that “eventual integrity” :P 
but beware of it… is considered “power voyage programming” :)

Esteban


Norbert

Olivier ;-)


2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert