Smalltalk › Pharo › Pharo Smalltalk Users

why is adding instance variables so slow?

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

18 messages Options

Peter Uhnak

why is adding instance variables so slow?

Hi,

does anyone know why adding instance variables is so slow?

I did some quick benchmarks (see below), resulting in more than order of magnitude speed difference between having it in the class definition and adding it later.

In fact it is still much faster to delete the class and then recreate it with the instance variables in the d efinition. For four arguments it is till 8x faster to delete the class four times and recreate it then just add the variable. Unfortunately I cannot just trash the classes (they have methods and inheritance).

So the question is: why is it so slow? can I somehow improve the performance?

Thanks,
Peter

Benchmarks:

[
cls := Object subclass: #Some1.
cls removeFromSystem.
] bench. "'91 per second'"

[
cls := Object subclass: #Some2 instanceVariableNames: 'variable'.
cls removeFromSystem
] bench. "'90 per second'"

[
cls := Object subclass: #Some3.
cls addInstVarNamed: 'variable'.
cls removeFromSystem.
] bench. "'7 per second'"

[
cls := Object subclass: #Some4.
cls removeFromSystem.
cls := Object subclass: #Some4 instanceVariableNames: 'variable'.
cls removeFromSystem.
] bench. "'43 per second'"

[
cls := Object subclass: #Some3.
cls addInstVarNamed: 'variable1'.
cls addInstVarNamed: 'variable2'.
cls addInstVarNamed: 'variable3'.
cls addInstVarNamed: 'variable4'.
cls removeFromSystem.
] bench. "'2 per second'"

[
cls := Object subclass: #Some4.
cls removeFromSystem.
cls := Object subclass: #Some4 instanceVariableNames: 'variable1 variable2 variable3 variable4'.
cls removeFromSystem.
] bench. "'44 per second'"

[
cls := Object subclass: #Some5.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1'.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2'.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3'.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3 variable4'.
cls removeFromSystem.
] bench. "'17.269 per second'"

Denis Kudriashov

Re: why is adding instance variables so slow?

Hi.

I profiled a bit and found problem:

PharoClassInstaller>>migrateClasses: old to: new using: anInstanceModification

instanceModification := anInstanceModification.

old ifEmpty: [ ^ self ].

[

1 to: old size do: [ :index |

self updateClass: (old at: index) to: (new at: index)].

old elementsForwardIdentityTo: new.

" Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: "

" If we don't clean up this garbage, a second update would revive them with a wrong layout! "

" (newClass rather than oldClass, since they are now both newClass) "

Smalltalk garbageCollect.

] valueUnpreemptively

Commenting garbage collection here increases performance 10 times.

Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method.

Can we remove manual garbage collection here? Why it is needed?

2017-04-12 9:34 GMT+02:00 Peter Uhnak <[hidden email]>:

Hi,

does anyone know why adding instance variables is so slow?

I did some quick benchmarks (see below), resulting in more than order of magnitude speed difference between having it in the class definition and adding it later.

In fact it is still much faster to delete the class and then recreate it with the instance variables in the d efinition. For four arguments it is till 8x faster to delete the class four times and recreate it then just add the variable. Unfortunately I cannot just trash the classes (they have methods and inheritance).

So the question is: why is it so slow? can I somehow improve the performance?

Thanks,
Peter

Benchmarks:

[
cls := Object subclass: #Some1.
cls removeFromSystem.
] bench. "'91 per second'"

[
cls := Object subclass: #Some2 instanceVariableNames: 'variable'.
cls removeFromSystem
] bench. "'90 per second'"

[
cls := Object subclass: #Some3.
cls addInstVarNamed: 'variable'.
cls removeFromSystem.
] bench. "'7 per second'"

[
cls := Object subclass: #Some4.
cls removeFromSystem.
cls := Object subclass: #Some4 instanceVariableNames: 'variable'.
cls removeFromSystem.
] bench. "'43 per second'"

[
cls := Object subclass: #Some3.
cls addInstVarNamed: 'variable1'.
cls addInstVarNamed: 'variable2'.
cls addInstVarNamed: 'variable3'.
cls addInstVarNamed: 'variable4'.
cls removeFromSystem.
] bench. "'2 per second'"

[
cls := Object subclass: #Some4.
cls removeFromSystem.
cls := Object subclass: #Some4 instanceVariableNames: 'variable1 variable2 variable3 variable4'.
cls removeFromSystem.
] bench. "'44 per second'"

[
cls := Object subclass: #Some5.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1'.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2'.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3'.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3 variable4'.
cls removeFromSystem.
] bench. "'17.269 per second'"

Guillermo Polito

Re: why is adding instance variables so slow?

On Wed, Apr 12, 2017 at 10:51 AM, Denis Kudriashov <[hidden email]> wrote:

Hi.

I profiled a bit and found problem:

PharoClassInstaller>>migrateClasses: old to: new using: anInstanceModification
instanceModification := anInstanceModification.
old ifEmpty: [ ^ self ].
[
1 to: old size do: [ :index |
self updateClass: (old at: index) to: (new at: index)].
old elementsForwardIdentityTo: new.
" Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: "
" If we don't clean up this garbage, a second update would revive them with a wrong layout! "
" (newClass rather than oldClass, since they are now both newClass) "
Smalltalk garbageCollect.
] valueUnpreemptively

Commenting garbage collection here increases performance 10 times.
Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method.

Can we remove manual garbage collection here? Why it is needed?

Well, there is the comment that explains it and makes pretty good sense.

2017-04-12 9:34 GMT+02:00 Peter Uhnak <[hidden email]>:
Hi,

does anyone know why adding instance variables is so slow?

I did some quick benchmarks (see below), resulting in more than order of magnitude speed difference between having it in the class definition and adding it later.

In fact it is still much faster to delete the class and then recreate it with the instance variables in the d efinition. For four arguments it is till 8x faster to delete the class four times and recreate it then just add the variable. Unfortunately I cannot just trash the classes (they have methods and inheritance).

So the question is: why is it so slow? can I somehow improve the performance?

Thanks,
Peter

Benchmarks:

[
cls := Object subclass: #Some1.
cls removeFromSystem.
] bench. "'91 per second'"

[
cls := Object subclass: #Some2 instanceVariableNames: 'variable'.
cls removeFromSystem
] bench. "'90 per second'"

[
cls := Object subclass: #Some3.
cls addInstVarNamed: 'variable'.
cls removeFromSystem.
] bench. "'7 per second'"

[
cls := Object subclass: #Some4.
cls removeFromSystem.
cls := Object subclass: #Some4 instanceVariableNames: 'variable'.
cls removeFromSystem.
] bench. "'43 per second'"

[
cls := Object subclass: #Some3.
cls addInstVarNamed: 'variable1'.
cls addInstVarNamed: 'variable2'.
cls addInstVarNamed: 'variable3'.
cls addInstVarNamed: 'variable4'.
cls removeFromSystem.
] bench. "'2 per second'"

[
cls := Object subclass: #Some4.
cls removeFromSystem.
cls := Object subclass: #Some4 instanceVariableNames: 'variable1 variable2 variable3 variable4'.
cls removeFromSystem.
] bench. "'44 per second'"

[
cls := Object subclass: #Some5.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1'.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2'.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3'.
cls removeFromSystem.
cls := Object subclass: #Some5 instanceVariableNames: 'variable1 variable2 variable3 variable4'.
cls removeFromSystem.
] bench. "'17.269 per second'"

Denis Kudriashov

Re: why is adding instance variables so slow?

2017-04-12 10:55 GMT+02:00 Guillermo Polito <[hidden email]>:

PharoClassInstaller>>migrateClasses: old to: new using: anInstanceModification
instanceModification := anInstanceModification.
old ifEmpty: [ ^ self ].
[
1 to: old size do: [ :index |
self updateClass: (old at: index) to: (new at: index)].
old elementsForwardIdentityTo: new.
" Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: "
" If we don't clean up this garbage, a second update would revive them with a wrong layout! "
" (newClass rather than oldClass, since they are now both newClass) "
Smalltalk garbageCollect.
] valueUnpreemptively

Commenting garbage collection here increases performance 10 times.
Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method.

Can we remove manual garbage collection here? Why it is needed?

Well, there is the comment that explains it and makes pretty good sense.

But is does not explain why these bad zombies exist. We investigates possible reasons and could not reproduce them. We will try remove garbage collection here in Pharo 7

Guillermo Polito

Re: why is adding instance variables so slow?

On Wed, Apr 12, 2017 at 11:35 AM, Denis Kudriashov <[hidden email]> wrote:

2017-04-12 10:55 GMT+02:00 Guillermo Polito <[hidden email]>:
PharoClassInstaller>>migrateClasses: old to: new using: anInstanceModification
instanceModification := anInstanceModification.
old ifEmpty: [ ^ self ].
[
1 to: old size do: [ :index |
self updateClass: (old at: index) to: (new at: index)].
old elementsForwardIdentityTo: new.
" Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: "
" If we don't clean up this garbage, a second update would revive them with a wrong layout! "
" (newClass rather than oldClass, since they are now both newClass) "
Smalltalk garbageCollect.
] valueUnpreemptively

Commenting garbage collection here increases performance 10 times.
Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method.

Can we remove manual garbage collection here? Why it is needed?

Well, there is the comment that explains it and makes pretty good sense.

But is does not explain why these bad zombies exist. We investigates possible reasons and could not reproduce them. We will try remove garbage collection here in Pharo 7

No, this will break stuff! I'll try to explain what does it mean by zombie instances to make some sense:

- Imagine that you have class A + 10 instances of A.

- We add an instance variable to A.

- this means the class builder will generate class A' that is the new version of A.

- then, it migrates all instances of A to class A'.

This migration is not magic:

- 10 new instances of A' are created

- the state is migrated from the instances of A to A'

- each instance of A is becomed into its corresponding instance of A'

- finally we become class A into A'

This step will make that old instances of A now have:

- the old format

- but point to the new class A

If we do not garbage collect, this means that doing

A allInstances

will return not only the new 10 instances of A, but the old instances of A'.

And that will break LOOOTS of stuff.

Denis Kudriashov

Re: why is adding instance variables so slow?

2017-04-12 13:17 GMT+02:00 Guillermo Polito <[hidden email]>:

1) each instance of A is becomed into its corresponding instance of A'
2) finally we become class A into A'
This step will make that old instances of A now have:
- the old format
- but point to the new class A

step 1) ensures that there are no instances of class A anymore.

Check following script:

c1 := Class1 new.
c2 := Class2 new.
c1 becomeForward: c2.
Class1 allInstances "=> #()".

And full migration is executed in high priority uninterrupted process to ensure that between 1) and 2) nobody will instantiate Class1

tesonep@gmail.com

Re: why is adding instance variables so slow?

In reply to this post by Guillermo Polito

Hi,

in non spur, the only improve that I think it can be made safely is moving the garbage collect operation to the migration of instances side, only executing the garbage collect if there are instances.

In spur the garbage collect it is not needed anymore, because the way the become is working is not the same.

In the old implementation, the old instances where in the image until they are removed by the garbage collector. So you can access them with the allInstances. In the new implementation, the old instances are marked as a forwarded to the new instances.

So, there is no need for the garbage collect.

This should be tried but I think it can work.

Cheers,

Pablo

On Wed, Apr 12, 2017 at 1:17 PM, Guillermo Polito <[hidden email]> wrote:

On Wed, Apr 12, 2017 at 11:35 AM, Denis Kudriashov <[hidden email]> wrote:

2017-04-12 10:55 GMT+02:00 Guillermo Polito <[hidden email]>:
PharoClassInstaller>>migrateClasses: old to: new using: anInstanceModification
instanceModification := anInstanceModification.
old ifEmpty: [ ^ self ].
[
1 to: old size do: [ :index |
self updateClass: (old at: index) to: (new at: index)].
old elementsForwardIdentityTo: new.
" Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: "
" If we don't clean up this garbage, a second update would revive them with a wrong layout! "
" (newClass rather than oldClass, since they are now both newClass) "
Smalltalk garbageCollect.
] valueUnpreemptively

Commenting garbage collection here increases performance 10 times.
Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method.

Can we remove manual garbage collection here? Why it is needed?

Well, there is the comment that explains it and makes pretty good sense.

But is does not explain why these bad zombies exist. We investigates possible reasons and could not reproduce them. We will try remove garbage collection here in Pharo 7

No, this will break stuff! I'll try to explain what does it mean by zombie instances to make some sense:

- Imagine that you have class A + 10 instances of A.

- We add an instance variable to A.
- this means the class builder will generate class A' that is the new version of A.
- then, it migrates all instances of A to class A'.
This migration is not magic:
- 10 new instances of A' are created
- the state is migrated from the instances of A to A'
- each instance of A is becomed into its corresponding instance of A'
- finally we become class A into A'
This step will make that old instances of A now have:
- the old format
- but point to the new class A

If we do not garbage collect, this means that doing

A allInstances

will return not only the new 10 instances of A, but the old instances of A'.
And that will break LOOOTS of stuff.

Pablo Tesone.
[hidden email]

Stephane Ducasse-3

Re: why is adding instance variables so slow?

In reply to this post by Denis Kudriashov

But I do not get how doing that would handle the old instances?

Because you want to migrate the old instances.

Stef

On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <[hidden email]> wrote:

2017-04-12 13:17 GMT+02:00 Guillermo Polito <[hidden email]>:
1) each instance of A is becomed into its corresponding instance of A'
2) finally we become class A into A'
This step will make that old instances of A now have:
- the old format
- but point to the new class A

step 1) ensures that there are no instances of class A anymore.
Check following script:
c1 := Class1 new.
c2 := Class2 new.
c1 becomeForward: c2.
Class1 allInstances "=> #()".

And full migration is executed in high priority uninterrupted process to ensure that between 1) and 2) nobody will instantiate Class1

Denis Kudriashov

Re: why is adding instance variables so slow?

2017-04-14 9:19 GMT+02:00 Stephane Ducasse <[hidden email]>:

But I do not get how doing that would handle the old instances?
Because you want to migrate the old instances.

But my example shows that old instances not exist after becomeFormard operation. Or what you mean?

Denis Kudriashov

Re: why is adding instance variables so slow?

In reply to this post by tesonep@gmail.com

2017-04-12 13:29 GMT+02:00 [hidden email] <[hidden email]>:

Hi,
in non spur, the only improve that I think it can be made safely is moving the garbage collect operation to the migration of instances side, only executing the garbage collect if there are instances.

In spur the garbage collect it is not needed anymore, because the way the become is working is not the same.

I checked on prespur cogVM: my script produced same result as on spur.

Igor Stasenko

Re: why is adding instance variables so slow?

In reply to this post by Stephane Ducasse-3

On 14 April 2017 at 10:19, Stephane Ducasse <[hidden email]> wrote:

But I do not get how doing that would handle the old instances?
Because you want to migrate the old instances.

there are no such thing as 'bad zombies', if they are there, it means you either don't care about migrating data

or again, don't care about doing #becomeForward-ing them properly.

In any case i don't see how GC could help to fix these issues. You either have consistency or don't have it,

and GC cannot do anything magical to fix it.

Stef

On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <[hidden email]> wrote:

2017-04-12 13:17 GMT+02:00 Guillermo Polito <[hidden email]>:
1) each instance of A is becomed into its corresponding instance of A'
2) finally we become class A into A'
This step will make that old instances of A now have:
- the old format
- but point to the new class A

step 1) ensures that there are no instances of class A anymore.
Check following script:
c1 := Class1 new.
c2 := Class2 new.
c1 becomeForward: c2.
Class1 allInstances "=> #()".

And full migration is executed in high priority uninterrupted process to ensure that between 1) and 2) nobody will instantiate Class1

Best regards,
Igor Stasenko.

tesonep@gmail.com

Re: why is adding instance variables so slow?

In reply to this post by Denis Kudriashov

Hi Denis,

in the script you are not generating instances of Class1, the problem is with the instances.

Cheers,

Pablo

On Fri, Apr 14, 2017 at 10:04 AM, Denis Kudriashov <[hidden email]> wrote:

2017-04-12 13:29 GMT+02:00 [hidden email] <[hidden email]>:
Hi,
in non spur, the only improve that I think it can be made safely is moving the garbage collect operation to the migration of instances side, only executing the garbage collect if there are instances.

In spur the garbage collect it is not needed anymore, because the way the become is working is not the same.

I checked on prespur cogVM: my script produced same result as on spur.

Pablo Tesone.
[hidden email]

Igor Stasenko

Re: why is adding instance variables so slow?

In reply to this post by Guillermo Polito

On 12 April 2017 at 14:17, Guillermo Polito <[hidden email]> wrote:

On Wed, Apr 12, 2017 at 11:35 AM, Denis Kudriashov <[hidden email]> wrote:

2017-04-12 10:55 GMT+02:00 Guillermo Polito <[hidden email]>:
PharoClassInstaller>>migrateClasses: old to: new using: anInstanceModification
instanceModification := anInstanceModification.
old ifEmpty: [ ^ self ].
[
1 to: old size do: [ :index |
self updateClass: (old at: index) to: (new at: index)].
old elementsForwardIdentityTo: new.
" Garbage collect away the zombie instances left behind in garbage memory in #updateInstancesFrom: "
" If we don't clean up this garbage, a second update would revive them with a wrong layout! "
" (newClass rather than oldClass, since they are now both newClass) "
Smalltalk garbageCollect.
] valueUnpreemptively

Commenting garbage collection here increases performance 10 times.
Then commenting class update loop increases performance 3 times more. But this loop is required. It adopts all instances of old class to new one. And time here spent in #allInstances method.

Can we remove manual garbage collection here? Why it is needed?

Well, there is the comment that explains it and makes pretty good sense.

But is does not explain why these bad zombies exist. We investigates possible reasons and could not reproduce them. We will try remove garbage collection here in Pharo 7

No, this will break stuff! I'll try to explain what does it mean by zombie instances to make some sense:

- Imagine that you have class A + 10 instances of A.

- We add an instance variable to A.
- this means the class builder will generate class A' that is the new version of A.
- then, it migrates all instances of A to class A'.
This migration is not magic:
- 10 new instances of A' are created
- the state is migrated from the instances of A to A'
- each instance of A is becomed into its corresponding instance of A'
- finally we become class A into A'
This step will make that old instances of A now have:
- the old format
- but point to the new class A

If we do not garbage collect, this means that doing

A allInstances

will return not only the new 10 instances of A, but the old instances of A'.
And that will break LOOOTS of stuff.

if you run #allInstances and in between you will trigger adding instance var & GC etc etc..
you'll have everything broken.. because there are things didn't meant to work in certain scenarios.
IIRC allInstances is highly dependns on NOT having full GC while doing it, and that's why all loops that

doing it is highly conservative & cautious about creating new objects while iterating over heap.

That's the nature how #allInstance works, and you could have a tons of issues with it regardless , if you

do full GC manually, or it triggered by VM itself. So, this is nothing to do with migrating instances of class.

Best regards,
Igor Stasenko.

tesonep@gmail.com

Re: why is adding instance variables so slow?

In reply to this post by Igor Stasenko

Hi, I think the problem was not clearly explained. This is the scenario that is problematic.

This scenario does not happen in the new Spur implementation of forwarding, but I think is happening in the old one.

1. You have, let's say 3 instances of ClassA.

2. You add a new instance variable to ClassA. It produces

2a. A new ClassAv2 is created with the instances variables of ClassA and the newone

2b. 3 Instances of ClassAv2 are created

2c. The values of the instance variables of ClassA are copied to the ones in ClassAv2 (the ones missing are left in nil).

2d. The 3 instances of ClassA are becomed forward to the 3 instances of ClassAv2

2e. The ClassA is becomed forward ClassAv2

3. You add a new instance variable to ClassAv2. It produces

3a. A new ClassAv3 is created with the instances variables of ClassAv2 and the newone

3b. 3 Instances of ClassAv3 are created

3c. The values of the instance variables of ClassAv2 are copied to the ones in ClassAv3 (the ones missing are left in nil).

3d. The 3 instances of ClassAv2 are becomed forward to the 3 instances of ClassAv3

3e. The ClassAv2 is becomedFormeward ClassAv3

4. All the instances of ClassAV3 have the correct format and everything works.

What is the problem:

===============

- When you do the first add instance variable, the old instances (the one from ClassA) which are smaller (has 1 instance variable less)

have its class changed (after you perform a become of ClassA to ClassAv2). So if you try to use them everything will explode, because you will trying to access an instance variable that does not exists.

These instances are not referenced by anyone, however if you perform a ClassAv2>>allInstances you will find them. So if you modify the class adding two variables, one after another the second time

you will be accessing the invalid instances.

Considering the differences in the Become implementation

============================================

However, the main difference is the implementation of the become forward.

Let's start with the new implementation, as it has not problems.

When you do a become forward, from object a to b, the primitive replaces the object a with a forwarder to b.

When this forwarded is accessed the references to it are rewrited.

If the objects are the same size (not this scenario) the object b replaces object a. It does not produces an error because the old.

In the become forward the old instances are not keeped.

In the old implementation the whole image is scanned, changing the references to the old instances, replacing with references to the new instances.

The old instances are not removed, just kept there to let the GC do its work.

Again if the objects are the same size there is special behavior.

I hope know the problem is better explained

Cheers,

Pablo

On Fri, Apr 14, 2017 at 10:50 AM, Igor Stasenko <[hidden email]> wrote:

On 14 April 2017 at 10:19, Stephane Ducasse <[hidden email]> wrote:
But I do not get how doing that would handle the old instances?
Because you want to migrate the old instances.

+1
there are no such thing as 'bad zombies', if they are there, it means you either don't care about migrating data
or again, don't care about doing #becomeForward-ing them properly.
In any case i don't see how GC could help to fix these issues. You either have consistency or don't have it,
and GC cannot do anything magical to fix it.

Stef

On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <[hidden email]> wrote:

2017-04-12 13:17 GMT+02:00 Guillermo Polito <[hidden email]>:
1) each instance of A is becomed into its corresponding instance of A'
2) finally we become class A into A'
This step will make that old instances of A now have:
- the old format
- but point to the new class A

step 1) ensures that there are no instances of class A anymore.
Check following script:
c1 := Class1 new.
c2 := Class2 new.
c1 becomeForward: c2.
Class1 allInstances "=> #()".

And full migration is executed in high priority uninterrupted process to ensure that between 1) and 2) nobody will instantiate Class1

--
Best regards,
Igor Stasenko.

Pablo Tesone.
[hidden email]

Igor Stasenko

Re: why is adding instance variables so slow?

i don't see much difference comparing to old implementation of #becomeForward:

the only issue is that #allInstances could report same instance twice

first, it will find an old instance (that is forwards to new one) and so, add it to results array,

and then walking the heap further will find a new version of same instance.

That's why , it think #allInstances should use IdentitySet-behavior to mitigate such problem,
and always look if there's already same object captured to avoid reporting it twice.
Or else we can declare it as a feature and warn users of #allInstances that they could have duplications,

so in case if it important to them, they could always do 'allInstances asIdentitySet'

But again, this is soooo orthogonal to adding instance variables to class... maybe we should rename the topic instead

and speak about #allInstances behavior? :)

On 14 April 2017 at 12:09, [hidden email] <[hidden email]> wrote:

Hi, I think the problem was not clearly explained. This is the scenario that is problematic.
This scenario does not happen in the new Spur implementation of forwarding, but I think is happening in the old one.

1. You have, let's say 3 instances of ClassA.
2. You add a new instance variable to ClassA. It produces
2a. A new ClassAv2 is created with the instances variables of ClassA and the newone
2b. 3 Instances of ClassAv2 are created
2c. The values of the instance variables of ClassA are copied to the ones in ClassAv2 (the ones missing are left in nil).
2d. The 3 instances of ClassA are becomed forward to the 3 instances of ClassAv2
2e. The ClassA is becomed forward ClassAv2

3. You add a new instance variable to ClassAv2. It produces
3a. A new ClassAv3 is created with the instances variables of ClassAv2 and the newone
3b. 3 Instances of ClassAv3 are created
3c. The values of the instance variables of ClassAv2 are copied to the ones in ClassAv3 (the ones missing are left in nil).
3d. The 3 instances of ClassAv2 are becomed forward to the 3 instances of ClassAv3
3e. The ClassAv2 is becomedFormeward ClassAv3

4. All the instances of ClassAV3 have the correct format and everything works.

What is the problem:
===============

- When you do the first add instance variable, the old instances (the one from ClassA) which are smaller (has 1 instance variable less)
have its class changed (after you perform a become of ClassA to ClassAv2). So if you try to use them everything will explode, because you will trying to access an instance variable that does not exists.
These instances are not referenced by anyone, however if you perform a ClassAv2>>allInstances you will find them. So if you modify the class adding two variables, one after another the second time
you will be accessing the invalid instances.

Considering the differences in the Become implementation
============================================

However, the main difference is the implementation of the become forward.
Let's start with the new implementation, as it has not problems.

When you do a become forward, from object a to b, the primitive replaces the object a with a forwarder to b.
When this forwarded is accessed the references to it are rewrited.
If the objects are the same size (not this scenario) the object b replaces object a. It does not produces an error because the old.
In the become forward the old instances are not keeped.

In the old implementation the whole image is scanned, changing the references to the old instances, replacing with references to the new instances.
The old instances are not removed, just kept there to let the GC do its work.
Again if the objects are the same size there is special behavior.

I hope know the problem is better explained

Cheers,
Pablo

On Fri, Apr 14, 2017 at 10:50 AM, Igor Stasenko <[hidden email]> wrote:

On 14 April 2017 at 10:19, Stephane Ducasse <[hidden email]> wrote:
But I do not get how doing that would handle the old instances?
Because you want to migrate the old instances.

+1
there are no such thing as 'bad zombies', if they are there, it means you either don't care about migrating data
or again, don't care about doing #becomeForward-ing them properly.
In any case i don't see how GC could help to fix these issues. You either have consistency or don't have it,
and GC cannot do anything magical to fix it.

Stef

On Wed, Apr 12, 2017 at 1:26 PM, Denis Kudriashov <[hidden email]> wrote:

2017-04-12 13:17 GMT+02:00 Guillermo Polito <[hidden email]>:
1) each instance of A is becomed into its corresponding instance of A'
2) finally we become class A into A'
This step will make that old instances of A now have:
- the old format
- but point to the new class A

step 1) ensures that there are no instances of class A anymore.
Check following script:
c1 := Class1 new.
c2 := Class2 new.
c1 becomeForward: c2.
Class1 allInstances "=> #()".

And full migration is executed in high priority uninterrupted process to ensure that between 1) and 2) nobody will instantiate Class1

--
Best regards,
Igor Stasenko.

--
Pablo Tesone.
[hidden email]

Best regards,
Igor Stasenko.

Denis Kudriashov

Re: why is adding instance variables so slow?

In reply to this post by tesonep@gmail.com

2017-04-14 10:51 GMT+02:00 [hidden email] <[hidden email]>:

Hi Denis,
in the script you are not generating instances of Class1, the problem is with the instances.

But it generates:

c1 := Class1 new.

Or what kind of generation you mean?

Denis Kudriashov

Re: why is adding instance variables so slow?

In reply to this post by tesonep@gmail.com

2017-04-14 11:09 GMT+02:00 [hidden email] <[hidden email]>:

Hi, I think the problem was not clearly explained. This is the scenario that is problematic.
This scenario does not happen in the new Spur implementation of forwarding, but I think is happening in the old one.

1. You have, let's say 3 instances of ClassA.
2. You add a new instance variable to ClassA. It produces
2a. A new ClassAv2 is created with the instances variables of ClassA and the newone
2b. 3 Instances of ClassAv2 are created
2c. The values of the instance variables of ClassA are copied to the ones in ClassAv2 (the ones missing are left in nil).
2d. The 3 instances of ClassA are becomed forward to the 3 instances of ClassAv2
2e. The ClassA is becomed forward ClassAv2

I still not see why my example not reflects all these steps. I checked also scenario with class becoming:

c1 := Class1 new.
c2 := Class2 new.
c1 becomeForward: c2.
Class1 becomeForward: Class2.
Class2 allInstances "=>#(aClass2)"

It also works in Cog

Tudor Girba-2

Re: why is adding instance variables so slow?

Hi,

Please let’s move this discussion on pharo-dev. And keep going :)

Cheers,
Doru

> On Apr 14, 2017, at 12:09 PM, Denis Kudriashov <[hidden email]> wrote:
>
>
> 2017-04-14 11:09 GMT+02:00 [hidden email] <[hidden email]>:
> Hi, I think the problem was not clearly explained. This is the scenario that is problematic.
> This scenario does not happen in the new Spur implementation of forwarding, but I think is happening in the old one.
>
> 1. You have, let's say 3 instances of ClassA.
> 2. You add a new instance variable to ClassA. It produces
> 2a. A new ClassAv2 is created with the instances variables of ClassA and the newone
> 2b. 3 Instances of ClassAv2 are created
> 2c. The values of the instance variables of ClassA are copied to the ones in ClassAv2 (the ones missing are left in nil).
> 2d. The 3 instances of ClassA are becomed forward to the 3 instances of ClassAv2
> 2e. The ClassA is becomed forward ClassAv2
>
> I still not see why my example not reflects all these steps. I checked also scenario with class becoming:
>
> c1 := Class1 new.
> c2 := Class2 new.
> c1 becomeForward: c2.
> Class1 becomeForward: Class2.
> Class2 allInstances "=>#(aClass2)"
>
> It also works in Cog

--
www.tudorgirba.com
www.feenk.com

"What we can governs what we wish."