Voyage: GC collecting random objects?

Hi, 

Voyage should keep a weak cache... objects should be disposed as soon as they are not used anymore. 

Now... it is saying "Reference not found", or "Lazy reference not found"?

Esteban

On Feb 8, 2013, at 11:46 AM, Bernat Romagosa <[hidden email]> wrote:

Hi!

We've been trying to migrate the data of our Iliad webapp to Voyage, and so far everything was going great, until the garbage collector collected objects it shouldn't have... Here's what we did:

We used to have all objects stored in an Instances classVar belonging to their own classes, so:

1. I had our classes return true to #isVoyageRoot
   
2. ourClasses do: [ :c | (c classVarNamed: #Instances) do: [ :i | i save ]]
   
3. I checked MongoDB and all was looking great. From Pharo, we could get all our instances and explore all of their relationships by OurClass selectAll. Everything was working, so we thought we didn't need the image stored instances anymore, thus:
   
4. ourClasses do: [ :c | c classVarNamed: #Instances put: nil ]
   
5. Smalltalk garbageCollect
   
6. RMUser selectAll → (long delay) → BOOM → 'Reference  not found RMUser: OID(3725800612)'
   

So, should we actually keep references to our instances in the image? I doubt this is the case, cos that'd mean we still have the 2Gb storage limitation, and then having an external DB would not be helping much...

Thanks a lot in advance!

Bernat.

p.s. I've also tried saving copies of my objects, but the result's been the same, after garbage-collecting, I lose references.

--
Bernat Romagosa.

yes, this is what happens: 

You have a graph that can look like this: 

A->b->c->D

where A and D represent roots of voyage (#isVoyageRoot = true)

you removed D, but A is not updated (the logic for auto update references is too complex, and this is a known limitation of voyage). 

So, in a next cycle you try to bring A to the image, and A is trying to load eagerly an instance of D... which does not exists anymore, so you have the error. 

To prevent this kind of problems, I use something that I call "eventual integrity" :) you can configure your collection to not fail when you try to get it and it does not exists. For achieve that, you have to add a setting to the class you want to take as "lazy deletable" (is a very dangerous property to just allow it by defect).

You have to do something like this:

D class>>#mongoContainer

<mongoContainer>

^VOMongoContainer new

enableMissingContent;

yourself

note that who is enabling missing content is D, not A :)

hope it helps.

Cheers,

Esteban

ps: y si no se entiende nada, mandame un mail privado y lo explico en español :)

On Feb 8, 2013, at 12:12 PM, Bernat Romagosa <[hidden email]> wrote:

It says "Reference not found", and the method signaling the error is VOMongoMaterializer >> #missingContentFor:id:

I can try to set up a minimal test case and sent it to you, would that help?

Thanks!

2013/2/8 Esteban Lorenzano <[hidden email]>

Hi, 

Voyage should keep a weak cache... objects should be disposed as soon as they are not used anymore. 

Now... it is saying "Reference not found", or "Lazy reference not found"?

Esteban

On Feb 8, 2013, at 11:46 AM, Bernat Romagosa <[hidden email]> wrote:

Hi!

We've been trying to migrate the data of our Iliad webapp to Voyage, and so far everything was going great, until the garbage collector collected objects it shouldn't have... Here's what we did:

We used to have all objects stored in an Instances classVar belonging to their own classes, so:

1. I had our classes return true to #isVoyageRoot
   
2. ourClasses do: [ :c | (c classVarNamed: #Instances) do: [ :i | i save ]]
   
3. I checked MongoDB and all was looking great. From Pharo, we could get all our instances and explore all of their relationships by OurClass selectAll. Everything was working, so we thought we didn't need the image stored instances anymore, thus:
   
4. ourClasses do: [ :c | c classVarNamed: #Instances put: nil ]
   
5. Smalltalk garbageCollect
   
6. RMUser selectAll → (long delay) → BOOM → 'Reference  not found RMUser: OID(3725800612)'
   

So, should we actually keep references to our instances in the image? I doubt this is the case, cos that'd mean we still have the 2Gb storage limitation, and then having an external DB would not be helping much...

Thanks a lot in advance!

Bernat.

p.s. I've also tried saving copies of my objects, but the result's been the same, after garbage-collecting, I lose references.

--
Bernat Romagosa.

--
Bernat Romagosa.

Hmm I'm not sure I understand. Now I've tried not even having the objects stored in a classvar at all, but saving them from a materialized fuel file straight away:

(FLMaterializer materializeFromFileNamed: 'mongodb.obj') do: [ :each | each save ].

It's all ok and working, until I do Smalltalk garbageCollect and there goes the "Reference not found" error again, without me having removed any object.

Would making ALL of my classes respond true to #isVoyageRoot fix the issue?

Thanks again!

Bernat.

p.s. se entiende el idioma, no el contenido, pero eso es culpa mía! jeje

2013/2/8 Esteban Lorenzano <[hidden email]>

yes, this is what happens: 

You have a graph that can look like this: 

A->b->c->D

where A and D represent roots of voyage (#isVoyageRoot = true)

you removed D, but A is not updated (the logic for auto update references is too complex, and this is a known limitation of voyage). 

So, in a next cycle you try to bring A to the image, and A is trying to load eagerly an instance of D... which does not exists anymore, so you have the error. 

To prevent this kind of problems, I use something that I call "eventual integrity" :) you can configure your collection to not fail when you try to get it and it does not exists. For achieve that, you have to add a setting to the class you want to take as "lazy deletable" (is a very dangerous property to just allow it by defect).

You have to do something like this:

D class>>#mongoContainer

<mongoContainer>

^VOMongoContainer new

enableMissingContent;

yourself

note that who is enabling missing content is D, not A :)

hope it helps.

Cheers,

Esteban

ps: y si no se entiende nada, mandame un mail privado y lo explico en español :)

On Feb 8, 2013, at 12:12 PM, Bernat Romagosa <[hidden email]> wrote:

It says "Reference not found", and the method signaling the error is VOMongoMaterializer >> #missingContentFor:id:

I can try to set up a minimal test case and sent it to you, would that help?

Thanks!

2013/2/8 Esteban Lorenzano <[hidden email]>

Hi, 

Voyage should keep a weak cache... objects should be disposed as soon as they are not used anymore. 

Now... it is saying "Reference not found", or "Lazy reference not found"?

Esteban

On Feb 8, 2013, at 11:46 AM, Bernat Romagosa <[hidden email]> wrote:

Hi!

We've been trying to migrate the data of our Iliad webapp to Voyage, and so far everything was going great, until the garbage collector collected objects it shouldn't have... Here's what we did:

We used to have all objects stored in an Instances classVar belonging to their own classes, so:

1. I had our classes return true to #isVoyageRoot
   
2. ourClasses do: [ :c | (c classVarNamed: #Instances) do: [ :i | i save ]]
   
3. I checked MongoDB and all was looking great. From Pharo, we could get all our instances and explore all of their relationships by OurClass selectAll. Everything was working, so we thought we didn't need the image stored instances anymore, thus:
   
4. ourClasses do: [ :c | c classVarNamed: #Instances put: nil ]
   
5. Smalltalk garbageCollect
   
6. RMUser selectAll → (long delay) → BOOM → 'Reference  not found RMUser: OID(3725800612)'
   

So, should we actually keep references to our instances in the image? I doubt this is the case, cos that'd mean we still have the 2Gb storage limitation, and then having an external DB would not be helping much...

Thanks a lot in advance!

Bernat.

p.s. I've also tried saving copies of my objects, but the result's been the same, after garbage-collecting, I lose references.

--
Bernat Romagosa.

--
Bernat Romagosa.

--
Bernat Romagosa.