From MethodProperties to Pragmas

Hi,

I changed the implementation of the class MethodProperty to match the
one of the pragma-framework found in VisualWorks. MethodProperties in
Squeak only provided a small subset of the functionality available in
VisualWorks and with a completely different interface. I am in favour
-- if they are available and make sense -- not to introduce yet
another concept, but stick to 'standards'.

What I did:

- MethodProperty was renamed to 'Pragma', it provides the same
protocol as the class in VisualWorks, however it is implemented
slightly different.

- The Squeak Parser was changed to deal with all kind of pragmas, such
as <foo>, <foo: nil>, <foo: -1 bar: 'zork'>, etc; MethodProperties
only supported the one argument form <foo: #bar>. Primitives are also
parsed as Pragmas, what has the nice side-effect that one can look for
all senders of #primitive: and #primitve:module:. Unfortunately the
FFI package introduces a very ugly hack to define its external
functions. External functions are parsed the same way as before;
however I feel the need to change that, so that the FFI package does
not need to patch the parser anymore.

- CompiledMethod efficiently stores the pragmas within the literal
frame as implemented by Marcus Denker. The new implementation takes
care not to waste memory: methods without pragmas have the
literal-slot set to nil, methods with pragmas store an array of Pragma
instances. There is no dictionary required.

- The parser of the submitted version does not implement the mechanism
of VisualWorks to force the use to declare the pragmas he is using
with <pragmas: #class> and <pragmas: #instance>. If requested I can
submit an enhanced version that provides this feature.

- The Pragma class implements a complete protocol to efficiently find
pragmas in classes and class-hierarchies.

- All the parts of the implementation are heavily testsed in the class
PragmaTest (25 run, 25 passes, 0 expected failures, 0 failures, 0
errors, 0 unexpected passes).

To load the code you have to file-in the attached change-set, it is a
hand-modified one to make sure everything is done in the right order.
At the end it will recompile the image and all PragmaTests should
pass. I tested it on the latest Squeak3.9a-6715.

Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

> To load the code you have to file-in the attached change-set, it is a
> hand-modified one to make sure everything is done in the right order.
> At the end it will recompile the image and all PragmaTests should
> pass. I tested it on the latest Squeak3.9a-6715.

I just tested it in Squeak3.9a-6716 and it works fine. All the pragma
tests pass and there are no other tests that break after filing-in the
change-set. I really hope that this change will be considered to be
included, I need it for several things were the old implementation was
not powerful enough.

Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

Hi Lukas,

I will have a look.

But one thing is *essential* for me: I need an interface for storing  
state
in compiledMethods, things that do not appear textually as pragmas but
that you would normally do with intance variables (if Squeak would
allow them...).

e.g. I want to be able to cache the name of a method (for a fast  
#who) and bytesurgeon
needs space to store a lot of information like original unmodified  
method,
number of temps, things like that. For all sorts of experiments this  
is essential.

I will check how to merge those two things. e.g. why not keep the  
idea of properties and
put the Pragmas into a property? Empty pragmas should take no space  
here, too. Of cource
non-empty ones would be a bit less efficient, as they would mean to  
have a PropertyDict and
the collection of Pragmas.

And then of course we should make CompiledMethods to be real objects  
at some point.
Tim's changeset now is over 8 years old, I think...

       Marcus

On 20.01.2006, at 12:38, Lukas Renggli wrote:

Hi Marcus,

> But one thing is *essential* for me: I need an interface for storing state
> in compiledMethods, things that do not appear textually as pragmas but
> that you would normally do with intance variables (if Squeak would
> allow them...).

ok I see your point. What we have both in mind is something very
different and it probably makes sense to keep both approaches
separate, the properties and the pragmas. Forget about the change-set
I posted, I will go through the whole thing again a post a new
solution, that is optimal for both of us:

- Every compiled method has got a dictionary that is lazily
initialized, as with your original approach.

- However those properties will be not related to the source-code
anymore, they can be solely used as a replacement for
instance-variables, e.g., they do not map to things like <key: value>
and they disappear if the method gets recompiled.

- Moreover the keys of properties do not show up as answer to
#hasLiteralThorough:, since this does not make sense in my opinion if
the user can't see them in the source.

- Pragmas will be implemented on top of this with the key #pragmas. I
have some optimizations in mind that avoids unnecessary allocations
and that will make searching and accessing pragmas almost as fast as
with the original approach. Pragmas will show up in the result of
#hasLiteralTrough: since they are visible in the source-code.

Does that make sense?

> I will check how to merge those two things. e.g. why not keep
> the idea of properties and put the Pragmas into a property?
> Empty pragmas should take no space here, too. Of cource
> non-empty ones would be a bit less efficient, as they would
> mean to have a PropertyDict and the collection of Pragmas.

I will publish a new change-set tomorrow afternoon. I am almost there.
Luckily we have both tests, so we can check if what we want still
works ;-)

Cheers,
Lukas

PS: Why not put the whole #who-array #( class selector ) into the last
literal? Unfortunately #hasLiteral: returns wrong results, e.g., for
class-references it returns all methods of the class itself. Since
#hasLiteral: is implemented as a primitive it cannot be fixed easily
not to iterate over the last two elements.

PPS: Another problem with that last literal is that the methods of a
class point to a wrong value if the class gets renamed. Is this
intentional? Why is #who and #selector not using the last literal? Why
is there no accessor #actualClass?

Hehe ... I am getting into CompiledMethods ;-)

--
Lukas Renggli
http://www.lukas-renggli.ch

On 20.01.2006, at 21:40, Lukas Renggli wrote: ok

> - Moreover the keys of properties do not show up as answer to
> #hasLiteralThorough:, since this does not make sense in my opinion if
> the user can't see them in the source.
>
ok

> - Pragmas will be implemented on top of this with the key #pragmas. I
> have some optimizations in mind that avoids unnecessary allocations
> and that will make searching and accessing pragmas almost as fast as
> with the original approach. Pragmas will show up in the result of
> #hasLiteralTrough: since they are visible in the source-code.

> Does that make sense?


Yes, that looks like exactly what's needed.
>
> I will publish a new change-set tomorrow afternoon. I am almost there.
> Luckily we have both tests, so we can check if what we want still
> works ;-)
>

Great!

>
> PS: Why not put the whole #who-array #( class selector ) into the last
> literal?

Methods with super sends in them need to store the class binding in the
last literal. That's where the vm then knows which method to call.
(send, with lookup starting in the super of the class encoded in the
last literal).
So the content of the last literal is fixed as soon as you have any
super send in the method.

> Unfortunately #hasLiteral: returns wrong results, e.g., for
> class-references it returns all methods of the class itself. Since
> #hasLiteral: is implemented as a primitive it cannot be fixed easily
> not to iterate over the last two elements.
>

Yes, that's why senders calls
Behavior>>whichSelectorsReferTo: literal special: specialFlag byte:  
specialByte
which filters out hits on the last literal.

> PPS: Another problem with that last literal is that the methods of a
> class point to a wrong value if the class gets renamed. Is this
> intentional?

Intentional not, but unavoidable. We would need to change the class
rename and the method install routines to set the class correctly.

And we need to take care to have Traits behave correctly...  
compiledMethods
in Traits should return the Trait. So we need to take care with Trait
methods that contain super-sends...

> Why is #who and #selector not using the last literal?

In my image, they do. I have not come around to move that over.
I'm now making another pass over the ClosurCompiler to move everyting
inside the image that we want there. Especially merge all overrides.

> Hehe ... I am getting into CompiledMethods ;-)
>

;-)

I have to admit I am quite clueless of why there was no iteration on
making all that a bit more general and late-bound in the early phase
of Squeak development... if CompiledMethods would be real objects,
everything would be so much cleaner.

     Marcus

Hi Marcus,

attached you find the files to make both of us happy, hopefully ;-)

Some changes I did to your property code:

- moved #properties and #properties: to the private method-category,
they should not be called directly to avoid unnecessary allocations.
- removed MethodProperties and replaced with an IdentityDictionary to
avoid another indirection.
- #propertyAt:, #propertyRemove: now throw an exception if the element
is not available to conform to Squeak standards.
- if the properties become empty because of #propertyRemove:, the
dictionary is removed and replaced with nil.
- added nice comments to all the new methods.

Not much changes were necessary for the Pragmas, they are only
slightly slower than before. Else my old comments (to the parser)
still apply.

I tested it in Squeak3.9-6717: load Pragma.cs first, then load the mcz
package to add and update the tests.

Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

Hi,

I did a 3.9a6719 with the changeset and the tests added.

On 21.01.2006, at 15:05, Lukas Renggli wrote:

Hello,

I was too fast yesterday... I am sure that this is what I want, but I  
think we
should do another interation and get feedback from the Tweak people.

So as the stuff was not yet announced, I think about going back to 6717
for another round of bug-fixes / integration of externally managed  
packages
and have a round of discussions with Andreas about if we can have  
something
that we all agree on.

I am sure that we want the semantics of what Lukas proposes, as the  
Tweak
annotations-as-properties are just useless for everything but Tweak.

Slow process, but better, I think.

        Marcus

On 21.01.2006, at 21:54, Marcus Denker wrote: