Smalltalk › Pharo › Pharo Smalltalk Users

Voyage: how to delete references to objects.

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

14 messages Options

Olivier Auverlot

Voyage: how to delete references to objects.

Hi,

I'm using Voyage in a Pharo application.

I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.

How can I remove properly a book and the reference to the book ?

Best regards
Olivier ;-)

NorbertHartl

Re: Voyage: how to delete references to objects.

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

Olivier Auverlot

Re: Voyage: how to delete references to objects.

thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>

From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

NorbertHartl

Re: Voyage: how to delete references to objects.

Am 17.04.2014 um 15:05 schrieb olivier auverlot <[hidden email]>:

thanks Robert for the explications.

It’s Norbert btw. :)

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

You need to explicitly save it as we don’t have something like write barriers in the image. So

aComicCollection

removeBook: aBook;

save

is needed.

Norbert

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>

From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

Olivier Auverlot

Re: Voyage: how to delete references to objects.

Norbert,

Sorry for my mistake ;-)

2014-04-17 15:18 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 15:05 schrieb olivier auverlot <[hidden email]>:

thanks Robert for the explications.

It’s Norbert btw. :)

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

You need to explicitly save it as we don’t have something like write barriers in the image. So

aComicCollection
removeBook: aBook;

save

is needed.

Norbert

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>

From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

Sabine Manaa

Re: Voyage: how to delete references to objects.

In reply to this post by Olivier Auverlot

Hi Olivier,

possibly the option

VOMongoContainer>>enableMissingContent;

is useful to avoid

http://forum.world.st/Voyage-image-freeze-VOMongoError-Lazy-reference-not-found-td4703739.html

Here is an example:

http://esug.org/data/ESUG2013/4-Thu/03-ESUG2013%20-%20VoyageTutorial.pdf

Regards

Sabine

On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <[hidden email]> wrote:

thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>

From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

If you reply to this email, your message will be added to the discussion below:

http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html

To start a new topic under Pharo Smalltalk Users, email [hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

NorbertHartl

Re: Voyage: how to delete references to objects.

Sabine,

I read the case now the first time. Just one note. Having references in both directions is probably not the best idea if using a non-object store. For navigating the graph it might be good but for persisting it is not. The serialization is a lot more complex if you do and the gain is nothing. For most databases with proper indexes you can request both directions easily and fast. So having one reference pointing from A to B you get the opposite direction for free.

Norbert

Am 17.04.2014 um 15:22 schrieb Sabine Knöfel <[hidden email]>:

Hi Olivier,

possibly the option
VOMongoContainer>>enableMissingContent;
is useful to avoid
http://forum.world.st/Voyage-image-freeze-VOMongoError-Lazy-reference-not-found-td4703739.html

Here is an example:
http://esug.org/data/ESUG2013/4-Thu/03-ESUG2013%20-%20VoyageTutorial.pdf

Regards
Sabine

On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <<a href="x-msg://56/user/SendEmail.jtp?type=node&node=4755097&i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:

thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>

From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

If you reply to this email, your message will be added to the discussion below:

http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html

To start a new topic under Pharo Smalltalk Users, email <a href="x-msg://56/user/SendEmail.jtp?type=node&node=4755097&i=1" target="_top" rel="nofollow" link="external">[hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

Sabine Manaa

Re: Voyage: how to delete references to objects.

Hi Norbert,

for me the gain within smalltalk is huge because I want to navigate from person to trip and from trip to person by the attributes without asking mongo for the correspondent object via indexes.

Don't you think that this is a question of design of the object model and should not be decided by database arguments?

Regards

Sabine

On Thu, Apr 17, 2014 at 3:40 PM, NorbertHartl [via Smalltalk] <[hidden email]> wrote:

Sabine,

I read the case now the first time. Just one note. Having references in both directions is probably not the best idea if using a non-object store. For navigating the graph it might be good but for persisting it is not. The serialization is a lot more complex if you do and the gain is nothing. For most databases with proper indexes you can request both directions easily and fast. So having one reference pointing from A to B you get the opposite direction for free.

Norbert

Am 17.04.2014 um 15:22 schrieb Sabine Knöfel <[hidden email]>:

Hi Olivier,

possibly the option
VOMongoContainer>>enableMissingContent;
is useful to avoid
http://forum.world.st/Voyage-image-freeze-VOMongoError-Lazy-reference-not-found-td4703739.html

Here is an example:
http://esug.org/data/ESUG2013/4-Thu/03-ESUG2013%20-%20VoyageTutorial.pdf

Regards
Sabine

On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <<a href="x-msg://56/user/SendEmail.jtp?type=node&node=4755097&i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:

thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>

From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

If you reply to this email, your message will be added to the discussion below:

http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html

To start a new topic under Pharo Smalltalk Users, email <a href="x-msg://56/user/SendEmail.jtp?type=node&node=4755097&i=1" target="_top" rel="nofollow" link="external">[hidden email]

To unsubscribe from Pharo Smalltalk Users, click here.
NAML

View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

If you reply to this email, your message will be added to the discussion below:

http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755102.html

To start a new topic under Pharo Smalltalk Users, email [hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

NorbertHartl

Re: Voyage: how to delete references to objects.

Am 17.04.2014 um 15:52 schrieb Sabine Knöfel <[hidden email]>:

Hi Norbert,

for me the gain within smalltalk is huge because I want to navigate from person to trip and from trip to person by the attributes without asking mongo for the correspondent object via indexes.

Don't you think that this is a question of design of the object model and should not be decided by database arguments?

I think there isn’t only one _or_ the other. Of course it is a lot more beautiful to have set up proper objects and collections and do everything in a smalltalkish way. That does only count as long as you stay on the island (smalltalk image). As soon as you start to play with the outer world you have to change your rules. If you agree it isn’t perfectly beautiful if you have a proper object model that freezes your image, right? :)

IMHO the beauty comes from a mixture of both that fit them together without restricting one or the other.

So I would try to avoid storing both directions. But which one to skip? If I assume that a person can have a lot of trips than I would skip the collection inside Person. The more trips a Person has the bigger the serialized document becomes. This makes it first slower to operate on the database and will fail if you reach a document size of 16MB because that is the maximum Mongo supports. Most of the time this will be a non-issue but for now I take it into account.

So the data model will have a class Trip with an instVar pointing to Person. And Person will have an instVar trips that is transient (see VOMongoTransientDescription). The implementation for Person then goes

Person>>#trips

^ trips ifNil: [ trips := self fetchTrips ]

Person>>#fetchTrips

^ Trips selectMany: [: each| … ]

Person>>#addTrip: aTrip

aTrip person: self.

aTrip save

So the Trip will have a reference to Person in the database but Person will have no reference to Trip. Person will retrieve its trips as soon as they are used in a really smalltalkish way :). Person acts as a smalltalk object and as a database object cache at the same time.

So this an implementation of Person that on the inside takes the database nature in regard but on the outside the interface is pure smalltalk.

Still not good?

Norbert

Regards
Sabine

On Thu, Apr 17, 2014 at 3:40 PM, NorbertHartl [via Smalltalk] <<a href="x-msg://58/user/SendEmail.jtp?type=node&node=4755106&i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
Sabine,

I read the case now the first time. Just one note. Having references in both directions is probably not the best idea if using a non-object store. For navigating the graph it might be good but for persisting it is not. The serialization is a lot more complex if you do and the gain is nothing. For most databases with proper indexes you can request both directions easily and fast. So having one reference pointing from A to B you get the opposite direction for free.

Norbert

Am 17.04.2014 um 15:22 schrieb Sabine Knöfel <[hidden email]>:

Hi Olivier,

possibly the option
VOMongoContainer>>enableMissingContent;
is useful to avoid
http://forum.world.st/Voyage-image-freeze-VOMongoError-Lazy-reference-not-found-td4703739.html

Here is an example:
http://esug.org/data/ESUG2013/4-Thu/03-ESUG2013%20-%20VoyageTutorial.pdf

Regards
Sabine

On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <<a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=0">x-msg://56/user/SendEmail.jtp?type=node&node=4755097&i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html
To start a new topic under Pharo Smalltalk Users, email <a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=1">x-msg://56/user/SendEmail.jtp?type=node&node=4755097&i=1" target="_top" rel="nofollow" link="external">[hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755102.html
To start a new topic under Pharo Smalltalk Users, email <a href="x-msg://58/user/SendEmail.jtp?type=node&node=4755106&i=1" target="_top" rel="nofollow" link="external">[hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

EstebanLM

Re: Voyage: how to delete references to objects.

In reply to this post by NorbertHartl

On 17 Apr 2014, at 15:18, Norbert Hartl <[hidden email]> wrote:

Am 17.04.2014 um 15:05 schrieb olivier auverlot <[hidden email]>:

thanks Robert for the explications.

It’s Norbert btw. :)

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

You need to explicitly save it as we don’t have something like write barriers in the image. So

aComicCollection
removeBook: aBook;
save

is needed.

yes, and also (if you want to be clean):

aBook remove.

(assuming aBook is also persistent).

But as Sabina points: #enableMissingContent in CommicCollection will help you to simplify things, then you just need: "aBook remove", and next time you read aComicCollection it will “clean” the invalid reference.

I call that “eventual integrity” :P

but beware of it… is considered “power voyage programming” :)

Esteban

Norbert

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>

From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

EstebanLM

Re: Voyage: how to delete references to objects.

In reply to this post by NorbertHartl

On 17 Apr 2014, at 16:26, Norbert Hartl <[hidden email]> wrote:

Am 17.04.2014 um 15:52 schrieb Sabine Knöfel <[hidden email]>:

Hi Norbert,

for me the gain within smalltalk is huge because I want to navigate from person to trip and from trip to person by the attributes without asking mongo for the correspondent object via indexes.

Don't you think that this is a question of design of the object model and should not be decided by database arguments?

I think there isn’t only one _or_ the other. Of course it is a lot more beautiful to have set up proper objects and collections and do everything in a smalltalkish way. That does only count as long as you stay on the island (smalltalk image). As soon as you start to play with the outer world you have to change your rules. If you agree it isn’t perfectly beautiful if you have a proper object model that freezes your image, right? :)
IMHO the beauty comes from a mixture of both that fit them together without restricting one or the other.

So I would try to avoid storing both directions. But which one to skip? If I assume that a person can have a lot of trips than I would skip the collection inside Person. The more trips a Person has the bigger the serialized document becomes. This makes it first slower to operate on the database and will fail if you reach a document size of 16MB because that is the maximum Mongo supports. Most of the time this will be a non-issue but for now I take it into account.
So the data model will have a class Trip with an instVar pointing to Person. And Person will have an instVar trips that is transient (see VOMongoTransientDescription). The implementation for Person then goes

Person>>#trips
^ trips ifNil: [ trips := self fetchTrips ]

Person>>#fetchTrips
^ Trips selectMany: [: each| … ]

Person>>#addTrip: aTrip
aTrip person: self.
aTrip save

So the Trip will have a reference to Person in the database but Person will have no reference to Trip. Person will retrieve its trips as soon as they are used in a really smalltalkish way :). Person acts as a smalltalk object and as a database object cache at the same time.
So this an implementation of Person that on the inside takes the database nature in regard but on the outside the interface is pure smalltalk.

Still not good?

yes, no good :)

I mean: I understand there are times when you need it, but I would prefer a framework to solve that for me (then I can think just in my objects).

And if the framework does not do a good job, then is a bad framework (even if made by me :P) and it needs to be improved.

having to think in persistence is a cancer that needs to be removed :P

Esteban

Norbert

Regards
Sabine

On Thu, Apr 17, 2014 at 3:40 PM, NorbertHartl [via Smalltalk] <<a href="x-msg://58/user/SendEmail.jtp?type=node&node=4755106&i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
Sabine,

I read the case now the first time. Just one note. Having references in both directions is probably not the best idea if using a non-object store. For navigating the graph it might be good but for persisting it is not. The serialization is a lot more complex if you do and the gain is nothing. For most databases with proper indexes you can request both directions easily and fast. So having one reference pointing from A to B you get the opposite direction for free.

Norbert

Am 17.04.2014 um 15:22 schrieb Sabine Knöfel <[hidden email]>:

Hi Olivier,

possibly the option
VOMongoContainer>>enableMissingContent;
is useful to avoid
http://forum.world.st/Voyage-image-freeze-VOMongoError-Lazy-reference-not-found-td4703739.html

Here is an example:
http://esug.org/data/ESUG2013/4-Thu/03-ESUG2013%20-%20VoyageTutorial.pdf

Regards
Sabine

On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <<a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=0">x-msg://56/user/SendEmail.jtp?type=node&node=4755097&i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html
To start a new topic under Pharo Smalltalk Users, email <a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=1">x-msg://56/user/SendEmail.jtp?type=node&node=4755097&i=1" target="_top" rel="nofollow" link="external">[hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755102.html
To start a new topic under Pharo Smalltalk Users, email <a href="x-msg://58/user/SendEmail.jtp?type=node&node=4755106&i=1" target="_top" rel="nofollow" link="external">[hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

NorbertHartl

Re: Voyage: how to delete references to objects.

Am 17.04.2014 um 18:05 schrieb Esteban Lorenzano <[hidden email]>:

On 17 Apr 2014, at 16:26, Norbert Hartl <[hidden email]> wrote:

Am 17.04.2014 um 15:52 schrieb Sabine Knöfel <[hidden email]>:

Hi Norbert,

for me the gain within smalltalk is huge because I want to navigate from person to trip and from trip to person by the attributes without asking mongo for the correspondent object via indexes.

Don't you think that this is a question of design of the object model and should not be decided by database arguments?

I think there isn’t only one _or_ the other. Of course it is a lot more beautiful to have set up proper objects and collections and do everything in a smalltalkish way. That does only count as long as you stay on the island (smalltalk image). As soon as you start to play with the outer world you have to change your rules. If you agree it isn’t perfectly beautiful if you have a proper object model that freezes your image, right? :)
IMHO the beauty comes from a mixture of both that fit them together without restricting one or the other.

So I would try to avoid storing both directions. But which one to skip? If I assume that a person can have a lot of trips than I would skip the collection inside Person. The more trips a Person has the bigger the serialized document becomes. This makes it first slower to operate on the database and will fail if you reach a document size of 16MB because that is the maximum Mongo supports. Most of the time this will be a non-issue but for now I take it into account.
So the data model will have a class Trip with an instVar pointing to Person. And Person will have an instVar trips that is transient (see VOMongoTransientDescription). The implementation for Person then goes

Person>>#trips
^ trips ifNil: [ trips := self fetchTrips ]

Person>>#fetchTrips
^ Trips selectMany: [: each| … ]

Person>>#addTrip: aTrip
aTrip person: self.
aTrip save

So the Trip will have a reference to Person in the database but Person will have no reference to Trip. Person will retrieve its trips as soon as they are used in a really smalltalkish way :). Person acts as a smalltalk object and as a database object cache at the same time.
So this an implementation of Person that on the inside takes the database nature in regard but on the outside the interface is pure smalltalk.

Still not good?

yes, no good :)
I mean: I understand there are times when you need it, but I would prefer a framework to solve that for me (then I can think just in my objects).
And if the framework does not do a good job, then is a bad framework (even if made by me :P) and it needs to be improved.

Sure. I was talking about _real_ stuff you can do _now_. I like to think just in objects, too. But we have a separation in hard disc and memory so we need to take care about persistence. I would also prefer to think just in programming and not have to use a stupid device like a keyboard and a mouse just in order to type silly characters that when interpreted probably come close to what I meant. But for the time being I need to stick to it, I guess.

having to think in persistence is a cancer that needs to be removed :P

Yep, we just need a common memory space where we can waste Zetabytes of memory to do little programs. Ah, and did I mention it needs to be infinitely fast because I have a certain use case and I don’t like to optimize it because I think that is cancer, too? Very academic view, well done! :)

Norbert

Esteban

Norbert

Regards
Sabine

On Thu, Apr 17, 2014 at 3:40 PM, NorbertHartl [via Smalltalk] <<a href="x-msg://58/user/SendEmail.jtp?type=node&node=4755106&i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
Sabine,

I read the case now the first time. Just one note. Having references in both directions is probably not the best idea if using a non-object store. For navigating the graph it might be good but for persisting it is not. The serialization is a lot more complex if you do and the gain is nothing. For most databases with proper indexes you can request both directions easily and fast. So having one reference pointing from A to B you get the opposite direction for free.

Norbert

Am 17.04.2014 um 15:22 schrieb Sabine Knöfel <[hidden email]>:

Hi Olivier,

possibly the option
VOMongoContainer>>enableMissingContent;
is useful to avoid
http://forum.world.st/Voyage-image-freeze-VOMongoError-Lazy-reference-not-found-td4703739.html

Here is an example:
http://esug.org/data/ESUG2013/4-Thu/03-ESUG2013%20-%20VoyageTutorial.pdf

Regards
Sabine

On Thu, Apr 17, 2014 at 3:07 PM, Olivier Auverlot [via Smalltalk] <<a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=0">x-msg://56/user/SendEmail.jtp?type=node&node=4755097&i=0" target="_top" rel="nofollow" link="external">[hidden email]> wrote:
thanks Robert for the explications.

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>
From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755092.html
To start a new topic under Pharo Smalltalk Users, email <a href="<a href="x-msg://56/user/SendEmail.jtp?type=node&amp;node=4755097&amp;i=1">x-msg://56/user/SendEmail.jtp?type=node&node=4755097&i=1" target="_top" rel="nofollow" link="external">[hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/Voyage-how-to-delete-references-to-objects-tp4755088p4755102.html
To start a new topic under Pharo Smalltalk Users, email <a href="x-msg://58/user/SendEmail.jtp?type=node&node=4755106&i=1" target="_top" rel="nofollow" link="external">[hidden email]
To unsubscribe from Pharo Smalltalk Users, click here.
NAML

View this message in context: Re: Voyage: how to delete references to objects.
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

pharo4Stef@free.fr

Re: Voyage: how to delete references to objects.

In reply to this post by EstebanLM

esteban

you should do a pass on the voyage documentation chapter and add this kind of infomration.
We can pair write if you want.

stef

On 17/4/14 17:56, Esteban Lorenzano wrote:

On 17 Apr 2014, at 15:18, Norbert Hartl <[hidden email]> wrote:

Am 17.04.2014 um 15:05 schrieb olivier auverlot <[hidden email]>:

thanks Robert for the explications.

It’s Norbert btw. :)

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

You need to explicitly save it as we don’t have something like write barriers in the image. So

aComicCollection

removeBook: aBook;

save

is needed.

yes, and also (if you want to be clean):

aBook remove.

(assuming aBook is also persistent).

But as Sabina points: #enableMissingContent in CommicCollection will help you to simplify things, then you just need: "aBook remove", and next time you read aComicCollection it will “clean” the invalid reference.

I call that “eventual integrity” :P

but beware of it… is considered “power voyage programming” :)

Esteban

Norbert

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>

From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert

Olivier Auverlot

Re: Voyage: how to delete references to objects.

Hi,

This morning, I arrived early at office because I was impatient to test the proposed solutions. Thank for your help, my problem is solved and I understand better the usage of Voyage and Mongo.

We learn something each day and it's cool !

Best regards

Olivier ;-)

2014-04-17 21:56 GMT+02:00 [hidden email] <[hidden email]>:

esteban

you should do a pass on the voyage documentation chapter and add this kind of infomration.
We can pair write if you want.

stef

On 17/4/14 17:56, Esteban Lorenzano wrote:

On 17 Apr 2014, at 15:18, Norbert Hartl <[hidden email]> wrote:

Am 17.04.2014 um 15:05 schrieb olivier auverlot <[hidden email]>:

thanks Robert for the explications.

It’s Norbert btw. :)

I agree with you that's the best way to remove a book is to delete the reference in ComicsCollection.

But how to do that ? Must I simply remove the reference in the ordered collection ? Voyage will syncronize automatically the data in memory with the content of the database ? Must I force the save of the data after removing the reference ?

You need to explicitly save it as we don’t have something like write barriers in the image. So

aComicCollection

removeBook: aBook;

save

is needed.

yes, and also (if you want to be clean):

aBook remove.

(assuming aBook is also persistent).

But as Sabina points: #enableMissingContent in CommicCollection will help you to simplify things, then you just need: "aBook remove", and next time you read aComicCollection it will “clean” the invalid reference.

I call that “eventual integrity” :P

but beware of it… is considered “power voyage programming” :)

Esteban

Norbert

Olivier ;-)

2014-04-17 14:10 GMT+02:00 Norbert Hartl <[hidden email]>:

Am 17.04.2014 um 13:53 schrieb olivier <[hidden email]>:

> Hi,
>
> I'm using Voyage in a Pharo application.
>
> I have two MongoDB collections which are ComicsCollection and ComicsBook. Each book is attached to a instance of ComicsCollection. The reference of each book is stored in an ordered collection (in the instance of ComicsCollection). The problem is that if I remove a book, the reference to the book is not deleted from ComicsCollection.
>
> How can I remove properly a book and the reference to the book ?
>

From the smalltalk image view the problem is usually exactly the opposite: You don’t delete objects but you remove references to them. As voyage maps objects it is a good idea to stay in the object realm. So you should rather remove the book from the collection. This way you won’t get errors just garbage. The „real“ problem that arises then is that the book would still be in the database. Just like it is in the image but there is no garbage collector for mongo. To decide that from the image is not possible. You load only a sub graph from the database into image memory. So you don’t know if there are other objects referencing the book.

If your records fit all in memory you could load all collections and books and build the difference between all referenced books in the collections and the total amount of books. The difference will be the set of objects not being referenced by a collection and those can be deleted.

If the records do not fit in memory an alternative strategy would be needed. I started to research if it is possible to build a map/reduce based garbage collector for common cases but got distracted. But I have the same problem so I will need to pick it up some time.

hope that helps,

Norbert