Smalltalk › Pharo › Pharo Smalltalk Developers

TraitedMetaclass / Behaviour bug?

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

9 messages Options

alistairgrant

TraitedMetaclass / Behaviour bug?

Hi Everyone,

TraitedMetaclass stores methods in two different instance variables:

- methodDict (inherited from Behavior)
- localMethods

And the same method can be different in each dictionary.

Different browsers show different versions, i.e.:

- The System Browser uses the method in methodDict.
- Calypso's "Senders of" browser uses localMethods.
- The Iceberg versions browser uses localMethods.

It appears that the method in methodDict is the one that is actually
used when a message is sent, but it would be nice to get confirmation.

To reproduce this in a clean Pharo 8 image:

- Open the System Browser on LGitBuf class>>signature_free: (the
methodDict version).
- Open the system browser on TLGitCalloutTrait>>call:options: and then
press "Senders". The resulting window will include LGitBuf
class>>signature_free:, but the localMethods version.

In this example, #signature_free: isn't defined by the trait, and
isn't overwritten, so I would expect that the method in both
dictionaries should be the same or it should only be in one
dictionary.

Can someone say what the intended behaviour is?

Thanks,
Alistair

alistairgrant

Re: TraitedMetaclass / Behaviour bug?

P.S. Two implications of this:

1. The code you're looking at may not be the code that is executed.
If in doubt, use the standard System Browser (not senders,
implementers, etc.).
2. Iceberg may not be loading extension methods that override an
existing method correctly, so the old version is left as the
executable code. There may be happening in other situations as well,
I haven't checked.

Thanks,
Alistair

On Wed, 18 Sep 2019 at 11:04, Alistair Grant <[hidden email]> wrote:

>
> Hi Everyone,
>
> TraitedMetaclass stores methods in two different instance variables:
>
> - methodDict (inherited from Behavior)
> - localMethods
>
> And the same method can be different in each dictionary.
>
> Different browsers show different versions, i.e.:
>
> - The System Browser uses the method in methodDict.
> - Calypso's "Senders of" browser uses localMethods.
> - The Iceberg versions browser uses localMethods.
>
> It appears that the method in methodDict is the one that is actually
> used when a message is sent, but it would be nice to get confirmation.
>
> To reproduce this in a clean Pharo 8 image:
>
> - Open the System Browser on LGitBuf class>>signature_free: (the
> methodDict version).
> - Open the system browser on TLGitCalloutTrait>>call:options: and then
> press "Senders". The resulting window will include LGitBuf
> class>>signature_free:, but the localMethods version.
>
> In this example, #signature_free: isn't defined by the trait, and
> isn't overwritten, so I would expect that the method in both
> dictionaries should be the same or it should only be in one
> dictionary.
>
> Can someone say what the intended behaviour is?
>
> Thanks,
> Alistair

Guillermo Polito

Re: TraitedMetaclass / Behaviour bug?

In reply to this post by alistairgrant

Hi Alistair,

we are checking this with Pablo here, some questions inline

> El 18 sept 2019, a las 11:04, Alistair Grant <[hidden email]> escribió:
>
> Hi Everyone,
>
> TraitedMetaclass stores methods in two different instance variables:
>
> - methodDict (inherited from Behavior)
> - localMethods
>
> And the same method can be different in each dictionary.

Yes, trais actually work by flattening the methods of traits into the user class.
- localMethods defines the methods actually defined in the class
- methodDict has the raw low-level flattened version the VM uses for the lookup

>
> Different browsers show different versions, i.e.:
>
> - The System Browser uses the method in methodDict.

Which is this System browser? Calypso system browser?

> - Calypso's "Senders of" browser uses localMethods.

and this is Calypso senders, right?

> - The Iceberg versions browser uses localMethods.
>
> It appears that the method in methodDict is the one that is actually
> used when a message is sent, but it would be nice to get confirmation.
>
> To reproduce this in a clean Pharo 8 image:
>
> - Open the System Browser on LGitBuf class>>signature_free: (the
> methodDict version).
> - Open the system browser on TLGitCalloutTrait>>call:options: and then
> press "Senders". The resulting window will include LGitBuf
> class>>signature_free:, but the localMethods version.
>
> In this example, #signature_free: isn't defined by the trait, and
> isn't overwritten, so I would expect that the method in both
> dictionaries should be the same or it should only be in one
> dictionary.
>
> Can someone say what the intended behaviour is?

We are tracking this, it seems you have found two bugs :D.

1st bug) Calypso and the senders/implementors should be in sync and show the same (always the local methods for traited classes)

Moreover, (I’m confirming this with Pablo on line right now) the local methods dictionary should be just a subset of the method dictionary.
In other words, the invariant is that methods in localMethods should be included in the methoddict, they should not differ as they are differing there.

2nd bug) UFFI? seems to be messing with this invariant :)

When a UFFI binding is executed for the very first time, the ffi call definition gets compiled and a new compiled method is generated performing that call.
When a session finishes (the image stops/restarts) ffi bindings are reset (because their compilation is platform dependent).

What we are seeing there is that somehow UFFI managed to reset a binding method and put a wrong one in the local method

I’ve seen this bug before but never got the insight of what was producing it (neither put too much time to dig on it either :P).
Another possibility is that the culprit are the LibGit bindings, which extend UFFI binding compilation.

>
> Thanks,
> Alistair
>

Guillermo Polito

Re: TraitedMetaclass / Behaviour bug?

In reply to this post by alistairgrant

About bug1) I’ve found that Calypso query scopes use sometimes #methods and sometimes #localMethods

ClyLocalClassScope >> methodsDo: aBlock

self classesDo: [ :eachClass |
self metaLevelsOf: eachClass do: [ :concreteMetaLevelClass |
concreteMetaLevelClass methods do: aBlock ] ]

ClyClassHierarchyScope >> methodsDo: aBlock

self classesDo: [ :eachClass |
self metaLevelsOf: eachClass do: [ :concreteMetaLevelClass |
concreteMetaLevelClass methods do: aBlock ] ]

ClyPackageScope >> methodsDo: aBlock
self classesDo: [:class |
class localMethods do: aBlock.
class classSide localMethods do: aBlock].

self packagesDo: [ :package |
package extensionMethods do: aBlock ]

ClySystemEnvironmentScope >> methodsDo: aBlock

self classesDo: [:class |
class instanceSide localMethods do: aBlock.
class classSide localMethods do: aBlock]

We should get an homogeneous version of this.

Pablo is checking the UFFI compilation now :)

> El 18 sept 2019, a las 14:20, Alistair Grant <[hidden email]> escribió:
>
> P.S. Two implications of this:
>
> 1. The code you're looking at may not be the code that is executed.
> If in doubt, use the standard System Browser (not senders,
> implementers, etc.).
> 2. Iceberg may not be loading extension methods that override an
> existing method correctly, so the old version is left as the
> executable code. There may be happening in other situations as well,
> I haven't checked.
>
> Thanks,
> Alistair
>
> On Wed, 18 Sep 2019 at 11:04, Alistair Grant <[hidden email]> wrote:
>>
>> Hi Everyone,
>>
>> TraitedMetaclass stores methods in two different instance variables:
>>
>> - methodDict (inherited from Behavior)
>> - localMethods
>>
>> And the same method can be different in each dictionary.
>>
>> Different browsers show different versions, i.e.:
>>
>> - The System Browser uses the method in methodDict.
>> - Calypso's "Senders of" browser uses localMethods.
>> - The Iceberg versions browser uses localMethods.
>>
>> It appears that the method in methodDict is the one that is actually
>> used when a message is sent, but it would be nice to get confirmation.
>>
>> To reproduce this in a clean Pharo 8 image:
>>
>> - Open the System Browser on LGitBuf class>>signature_free: (the
>> methodDict version).
>> - Open the system browser on TLGitCalloutTrait>>call:options: and then
>> press "Senders". The resulting window will include LGitBuf
>> class>>signature_free:, but the localMethods version.
>>
>> In this example, #signature_free: isn't defined by the trait, and
>> isn't overwritten, so I would expect that the method in both
>> dictionaries should be the same or it should only be in one
>> dictionary.
>>
>> Can someone say what the intended behaviour is?
>>
>> Thanks,
>> Alistair
>

alistairgrant

Re: TraitedMetaclass / Behaviour bug?

In reply to this post by Guillermo Polito

Hi Guille,

Replies inline...

On Wed, 18 Sep 2019 at 14:44, Guillermo Polito
<[hidden email]> wrote:

>
> Hi Alistair,
>
> we are checking this with Pablo here, some questions inline
>
> > El 18 sept 2019, a las 11:04, Alistair Grant <[hidden email]> escribió:
> >
> > Hi Everyone,
> >
> > TraitedMetaclass stores methods in two different instance variables:
> >
> > - methodDict (inherited from Behavior)
> > - localMethods
> >
> > And the same method can be different in each dictionary.
>
> Yes, trais actually work by flattening the methods of traits into the user class.
> - localMethods defines the methods actually defined in the class
> - methodDict has the raw low-level flattened version the VM uses for the lookup

So methodDict and localMethods should always contain the same instance
of CompiledMethod for a given selector, right?

>
> >
> > Different browsers show different versions, i.e.:
> >
> > - The System Browser uses the method in methodDict.
>
> Which is this System browser? Calypso system browser?

Yep, as in World Menu -> Tools -> System Browser

> > - Calypso's "Senders of" browser uses localMethods.
>
> and this is Calypso senders, right?

Yes, pressing the "Senders" button from the System Browser (Calypso).

> > - The Iceberg versions browser uses localMethods.
> >
> > It appears that the method in methodDict is the one that is actually
> > used when a message is sent, but it would be nice to get confirmation.
> >
> > To reproduce this in a clean Pharo 8 image:
> >
> > - Open the System Browser on LGitBuf class>>signature_free: (the
> > methodDict version).
> > - Open the system browser on TLGitCalloutTrait>>call:options: and then
> > press "Senders". The resulting window will include LGitBuf
> > class>>signature_free:, but the localMethods version.
> >
> > In this example, #signature_free: isn't defined by the trait, and
> > isn't overwritten, so I would expect that the method in both
> > dictionaries should be the same or it should only be in one
> > dictionary.
> >
> > Can someone say what the intended behaviour is?
>
> We are tracking this, it seems you have found two bugs :D.
>
> 1st bug) Calypso and the senders/implementors should be in sync and show the same (always the local methods for traited classes)

From what you've written above (and below), the two dictionaries
should always have the same instance of CompiledMethod so it shouldn't
matter where it comes from.

> Moreover, (I’m confirming this with Pablo on line right now) the local methods dictionary should be just a subset of the method dictionary.
> In other words, the invariant is that methods in localMethods should be included in the methoddict, they should not differ as they are differing there.
>
> 2nd bug) UFFI? seems to be messing with this invariant :)
>
> When a UFFI binding is executed for the very first time, the ffi call definition gets compiled and a new compiled method is generated performing that call.
> When a session finishes (the image stops/restarts) ffi bindings are reset (because their compilation is platform dependent).
>
> What we are seeing there is that somehow UFFI managed to reset a binding method and put a wrong one in the local method
>
> I’ve seen this bug before but never got the insight of what was producing it (neither put too much time to dig on it either :P).
> Another possibility is that the culprit are the LibGit bindings, which extend UFFI binding compilation.

I haven't looked at this, but what I saw was the localMethods was
correct and the methodDict was old:

Redefine a base system method (LGitCommit>>commit_author:) using an
extension method resulted in methodDict containing the original
definition and the localMethods containing the overridden method.
What I would expect is that the overridden method is what is used
everywhere.

I'm not sure if this is related to UFFI or not.

I'm on Discord if you'd like to chat real-time (back around 15:25).

Thanks!
Alistair

alistairgrant

Re: TraitedMetaclass / Behaviour bug?

In reply to this post by Guillermo Polito

Inline below...

Cheers,
Alistair
(on phone)

On Wed., 18 Sep. 2019, 15:05 Guillermo Polito, <[hidden email]> wrote:

About bug1) I’ve found that Calypso query scopes use sometimes #methods and sometimes #localMethods

ClyLocalClassScope >> methodsDo: aBlock

self classesDo: [ :eachClass |
self metaLevelsOf: eachClass do: [ :concreteMetaLevelClass |
concreteMetaLevelClass methods do: aBlock ] ]

ClyClassHierarchyScope >> methodsDo: aBlock

self classesDo: [ :eachClass |
self metaLevelsOf: eachClass do: [ :concreteMetaLevelClass |
concreteMetaLevelClass methods do: aBlock ] ]

ClyPackageScope >> methodsDo: aBlock
self classesDo: [:class |
class localMethods do: aBlock.
class classSide localMethods do: aBlock].

self packagesDo: [ :package |
package extensionMethods do: aBlock ]

ClySystemEnvironmentScope >> methodsDo: aBlock

self classesDo: [:class |
class instanceSide localMethods do: aBlock.
class classSide localMethods do: aBlock]

We should get an homogeneous version of this.

Pablo is checking the UFFI compilation now :)

Right. I wrote that it shouldn't matter whether methodDict or local methods is used, but I agree that there should be a single accessor that is used everywhere.

Thanks!

Alistair

> El 18 sept 2019, a las 14:20, Alistair Grant <[hidden email]> escribió:
>
> P.S. Two implications of this:
>
> 1. The code you're looking at may not be the code that is executed.
> If in doubt, use the standard System Browser (not senders,
> implementers, etc.).
> 2. Iceberg may not be loading extension methods that override an
> existing method correctly, so the old version is left as the
> executable code. There may be happening in other situations as well,
> I haven't checked.
>
> Thanks,
> Alistair
>
> On Wed, 18 Sep 2019 at 11:04, Alistair Grant <[hidden email]> wrote:
>>
>> Hi Everyone,
>>
>> TraitedMetaclass stores methods in two different instance variables:
>>
>> - methodDict (inherited from Behavior)
>> - localMethods
>>
>> And the same method can be different in each dictionary.
>>
>> Different browsers show different versions, i.e.:
>>
>> - The System Browser uses the method in methodDict.
>> - Calypso's "Senders of" browser uses localMethods.
>> - The Iceberg versions browser uses localMethods.
>>
>> It appears that the method in methodDict is the one that is actually
>> used when a message is sent, but it would be nice to get confirmation.
>>
>> To reproduce this in a clean Pharo 8 image:
>>
>> - Open the System Browser on LGitBuf class>>signature_free: (the
>> methodDict version).
>> - Open the system browser on TLGitCalloutTrait>>call:options: and then
>> press "Senders". The resulting window will include LGitBuf
>> class>>signature_free:, but the localMethods version.
>>
>> In this example, #signature_free: isn't defined by the trait, and
>> isn't overwritten, so I would expect that the method in both
>> dictionaries should be the same or it should only be in one
>> dictionary.
>>
>> Can someone say what the intended behaviour is?
>>
>> Thanks,
>> Alistair
>

Guillermo Polito

Re: TraitedMetaclass / Behaviour bug?

In reply to this post by alistairgrant

El 18 sept 2019, a las 15:05, Alistair Grant <[hidden email]> escribió:

TraitedMetaclass stores methods in two different instance variables:

- methodDict (inherited from Behavior)
- localMethods

And the same method can be different in each dictionary.

Yes, trais actually work by flattening the methods of traits into the user class.
- localMethods defines the methods actually defined in the class
- methodDict has the raw low-level flattened version the VM uses for the lookup

So methodDict and localMethods should always contain the same instance
of CompiledMethod for a given selector, right?

Yes it should.

We are tracking this, it seems you have found two bugs :D.

1st bug) Calypso and the senders/implementors should be in sync and show the same (always the local methods for traited classes)

From what you've written above (and below), the two dictionaries
should always have the same instance of CompiledMethod so it shouldn't
matter where it comes from.

Just for the record, yes and no :).

With the new trait implementation, we have actually a modular MOP.

All classes (whether traited or not) understand:

- #localMethods => returning the methods as they are implemented in the class

- #methods => returning the actual methods executed by the VM

For normal classes, #localMethods is defined as follows:

localMethods

^ self methods

And methods understand

- #origin => where that method is actually defined

For example, in a traited class a lookup should be done in the trait composition, otherwise is the class where the method is installed.

Tools should work against #localMethods and #origin, which hide how the actual implementation stores the things :).

We already had in the past a lot of tools that hardcoded/duplicated the internals of Traits for example.

The other advantage of this is that, imagine you want to extend Pharo with:

- multimethods => localMethods will have a single multi-method implementation, and the implementation side will contain maybe an expanded version with type checks or even one method per type combination

- optional parameters => a single method with many parameters should be expanded to many methods with less parameters + default values

The idea is that tools do not hardcode the implementation details of these extensions, so the public API is #localMethods and #origin :)

Of course, one could design a tool showing #methods, which will show the low-level view of classes.

Moreover, (I’m confirming this with Pablo on line right now) the local methods dictionary should be just a subset of the method dictionary.
In other words, the invariant is that methods in localMethods should be included in the methoddict, they should not differ as they are differing there.

2nd bug) UFFI? seems to be messing with this invariant :)

When a UFFI binding is executed for the very first time, the ffi call definition gets compiled and a new compiled method is generated performing that call.
When a session finishes (the image stops/restarts) ffi bindings are reset (because their compilation is platform dependent).

What we are seeing there is that somehow UFFI managed to reset a binding method and put a wrong one in the local method

I’ve seen this bug before but never got the insight of what was producing it (neither put too much time to dig on it either :P).
Another possibility is that the culprit are the LibGit bindings, which extend UFFI binding compilation.

I haven't looked at this, but what I saw was the localMethods was
correct and the methodDict was old:

Redefine a base system method (LGitCommit>>commit_author:) using an
extension method resulted in methodDict containing the original
definition and the localMethods containing the overridden method.
What I would expect is that the overridden method is what is used
everywhere.

I'm not sure if this is related to UFFI or not.

Pablo tracked this down to possibly

FFIMethodRegistry >> removeMethod: aMethod

aMethod methodClass methodDict

at: aMethod selector

put: (aMethod propertyValueAt: #ffiNonCompiledMethod)

alistairgrant

Re: TraitedMetaclass / Behaviour bug?

Hi Guille,

On Wed, 18 Sep 2019 at 15:22, Guillermo Polito
<[hidden email]> wrote:

>
>
>
> El 18 sept 2019, a las 15:05, Alistair Grant <[hidden email]> escribió:
>
>
> [snip/]
>
> From what you've written above (and below), the two dictionaries
> should always have the same instance of CompiledMethod so it shouldn't
> matter where it comes from.
>
>
> Just for the record, yes and no :).
>
> With the new trait implementation, we have actually a modular MOP.
> All classes (whether traited or not) understand:
> - #localMethods => returning the methods as they are implemented in the class
> - #methods => returning the actual methods executed by the VM
>
> For normal classes, #localMethods is defined as follows:
>
> localMethods
> ^ self methods
>
> And methods understand
> - #origin => where that method is actually defined
> For example, in a traited class a lookup should be done in the trait composition, otherwise is the class where the method is installed.
>
> Tools should work against #localMethods and #origin, which hide how the actual implementation stores the things :).
> We already had in the past a lot of tools that hardcoded/duplicated the internals of Traits for example.
>
> The other advantage of this is that, imagine you want to extend Pharo with:
> - multimethods => localMethods will have a single multi-method implementation, and the implementation side will contain maybe an expanded version with type checks or even one method per type combination
> - optional parameters => a single method with many parameters should be expanded to many methods with less parameters + default values
>
> The idea is that tools do not hardcode the implementation details of these extensions, so the public API is #localMethods and #origin :)
> Of course, one could design a tool showing #methods, which will show the low-level view of classes.

Thanks for the clarification, that makes sense and will help with
tracking this further.

> > I haven't looked at this, but what I saw was the localMethods was
> > correct and the methodDict was old:
> >
> > Redefine a base system method (LGitCommit>>commit_author:) using an
> > extension method resulted in methodDict containing the original
> > definition and the localMethods containing the overridden method.
> > What I would expect is that the overridden method is what is used
> > everywhere.
> >
> > I'm not sure if this is related to UFFI or not.
>
>
> Pablo tracked this down to possibly
>
> FFIMethodRegistry >> removeMethod: aMethod
>
> aMethod methodClass methodDict
> at: aMethod selector
> put: (aMethod propertyValueAt: #ffiNonCompiledMethod)

That does look bad :-)

Using the test class included below:

MethodDictionaryConsistencyTest new inconsistentMethodDefinitions

answers a collection of 240 methods where methodDict and localMethods
aren't the same instance. All of the methods class names begin with
'LGit' :-)

There may still be one more bug involved since the scenario I
mentioned above regarding Iceberg and extension methods doesn't seem
to fit the FFI description (it calls an ffi method, but isn't one
itself).

Thanks,
Alistair

MethodDictionaryConsistencyTest.st (1K) Download Attachment

Denis Kudriashov

Re: TraitedMetaclass / Behaviour bug?

In reply to this post by Guillermo Polito

Hi Guille.

ср, 18 сент. 2019 г. в 14:22, Guillermo Polito <[hidden email]>:

Tools should work against #localMethods and #origin, which hide how the actual implementation stores the things :).

I am a bit confused by this sentence. My original design of scopes was purely based on #localMethods and #origin to follow this idea.

We had long dispute with Esteban about my approach and I remember you and Pablo also supported Esteban opinion. At the end he changed the implementation to use #methods and #methodClass instead. But it was not completely finished and therefore there are still some bugs like this one.

So did you change the opinion or do I misunderstand your point?

We already had in the past a lot of tools that hardcoded/duplicated the internals of Traits for example.

The other advantage of this is that, imagine you want to extend Pharo with:
- multimethods => localMethods will have a single multi-method implementation, and the implementation side will contain maybe an expanded version with type checks or even one method per type combination
- optional parameters => a single method with many parameters should be expanded to many methods with less parameters + default values

The idea is that tools do not hardcode the implementation details of these extensions, so the public API is #localMethods and #origin :)
Of course, one could design a tool showing #methods, which will show the low-level view of classes.

Moreover, (I’m confirming this with Pablo on line right now) the local methods dictionary should be just a subset of the method dictionary.
In other words, the invariant is that methods in localMethods should be included in the methoddict, they should not differ as they are differing there.

2nd bug) UFFI? seems to be messing with this invariant :)

When a UFFI binding is executed for the very first time, the ffi call definition gets compiled and a new compiled method is generated performing that call.
When a session finishes (the image stops/restarts) ffi bindings are reset (because their compilation is platform dependent).

What we are seeing there is that somehow UFFI managed to reset a binding method and put a wrong one in the local method

I’ve seen this bug before but never got the insight of what was producing it (neither put too much time to dig on it either :P).
Another possibility is that the culprit are the LibGit bindings, which extend UFFI binding compilation.

I haven't looked at this, but what I saw was the localMethods was
correct and the methodDict was old:

Redefine a base system method (LGitCommit>>commit_author:) using an
extension method resulted in methodDict containing the original
definition and the localMethods containing the overridden method.
What I would expect is that the overridden method is what is used
everywhere.

I'm not sure if this is related to UFFI or not.

Pablo tracked this down to possibly

FFIMethodRegistry >> removeMethod: aMethod

aMethod methodClass methodDict
at: aMethod selector
put: (aMethod propertyValueAt: #ffiNonCompiledMethod)