A reengineering the CompiledMethod trailers

Hello guys,

please take a look at changeset i just uploaded:

http://bugs.squeak.org/view.php?id=7428

There is an initial implementation of CompiledMethodTrailer with some
test coverage.

My motivation behind this is simple:

1. no more poking with raw bytes, like this:

holdsTempNames
        "Are tempNames stored in trailer bytes"

        | flagByte |
        flagByte := self last.
        (flagByte = 0 or: [flagByte = 251 "some source-less methods have flag
= 251, rest = 0"
                        and: [(1 to: 3) allSatisfy: [:i | (self at: self size - i) = 0]]])
                ifTrue: [^ false].  "No source pointer & no temp names"
        flagByte < 252 ifTrue: [^ true].  "temp names compressed"
        ^ false "Source pointer"

or this:

endPC
        "Answer the index of the last bytecode."
        | size flagByte |
        "Can't create a zero-sized CompiledMethod so no need to use last for
the errorEmptyCollection check.
         We can reuse size."
        size := self size.
        flagByte := self at: size.
        flagByte = 0 ifTrue:
                ["If last byte = 0, may be either 0, 0, 0, 0 or just 0"
                1 to: 4 do: [:i | (self at: size - i) = 0 ifFalse: [^size - i]]].
        flagByte < 252 ifTrue:
                ["Magic sources (temp names encoded in last few bytes)"
                ^flagByte <= 127
                        ifTrue: [size - flagByte - 1]
                        ifFalse: [size - (flagByte - 128 * 128) - (self at: size - 1) - 2]].
        "Normal 4-byte source pointer"
        ^size - 4

2. method's trailer can be used for storing a variety of stuff, not
just source pointer or temps names. And it will be easy to add new
kinds of trailers.
    A most useful, as to me, and which i've added initially is:
  - being able to embed the source code in trailer, so a compiled
method and its source lives together in image.
     Some of you have expressed this idea before, so here it is.
  - being able to retrieve the method's source using other way, than
through SourceFiles or embedded in trailer.
     I added two kinds of trailers for that:
     - get method's source by class+selector where it installed to.
     - get method's source by class+some string identifier

  - yours. Please tell me, what kind you want to have in addition.

3.  a source pointer could surpass the 32Mb limit. A CompiledMethod
already can encode a source pointer value of any size.
  Its now only a matter of fixing the source pointer encoding logic to
enable having .source and .changes files above 32Mb.

Please, review my code and send me your comments and wishes.
At next stage i will create a changeset which will put this stuff in
use, as well as cleanup lots of places in CompiledMethod and
compiler/decompiler.

--
Best regards,
Igor Stasenko AKA sig.

At 12:14 AM +0200 12/14/09, Igor Stasenko apparently wrote:
Not sure how this relates exactly but there has been quite a bit of work in the past re: Virtual Image 4.0 format.
See:
New Compiled Method Format
<http://wiki.squeak.org/squeak/750>

and:
VI4 project
<http://wiki.squeak.org/squeak/2119>

Here is an old related message from Tim Rowledge on Oct 20, 2005:
At 1:23 PM -0700 10/20/05, tim Rowledge apparently wrote:
Ken G. Brown

2009/12/14 karl ramberg <[hidden email]>: its already there, sorry didn't mentioned that explicitly.
It encodes the temp names string using either qCompress, or Zip
compression by checking which one is best.
Same for source code.

> just variable names so decompiled code without full source get readable
> karl
>
>
>
>



--
Best regards,
Igor Stasenko AKA sig.

2009/12/14 Ken G. Brown <[hidden email]>: Oh, the changes in that project is much wider and requiring changing
the image format and VM.
My changes is much less significant and don't require rebuilding a whole image.
It is gradual change which doesn't changes the way how VM deals with
compiled methods,
but rather will change the way how tools (compiler/decompiler etc)
handing the compiled methods metadata encoding in trailing bytes.
And it backwards compatible with 99% of existing compiled methods.
Unless you start using new trailer kinds, it will remain the same.




--
Best regards,
Igor Stasenko AKA sig.

Ken G. Brown wrote:
> At 12:14 AM +0200 12/14/09, Igor Stasenko apparently wrote:
> Not sure how this relates exactly but there has been quite a bit of work in the past re: Virtual Image 4.0 format.
> See:
> New Compiled Method Format
> <http://wiki.squeak.org/squeak/750>
>
> and:
> VI4 project
> <http://wiki.squeak.org/squeak/2119>

It doesn't relate. There may be some inspiration for how to deal with
source pointers but to be honest, I find Igors approach vastly more
useful since it doesn't immediately add a megabyte or more to the image
size (which would happen if you go to an explicit source pointer
representation) but rather makes the encoding explicitly accessible.

Go Igor!

Cheers,
   - Andreas

PS. The only person who I'd like to explicitly comment (even if only to
say "that's fine") is Eliot, since he might have some additional
thoughts about some of this stuff which relate to Cog.

On Sun, Dec 13, 2009 at 05:01:17PM -0800, Andreas Raab wrote:
+1

FYI, and just to make sure it does not get overlooked, there is also
this from Markus Denker:
 
  Mantis 4369: [Patch] Both source files now with 512MB capacity
  http://bugs.squeak.org/view.php?id=4369

Dave

2009/12/15 David T. Lewis <[hidden email]>:
it adds a sourcePointer ivar to MethodProperties.
But i wonder, why its still limits the max pointer value, since once
you got ivar for it, you can store virtually anything there..
My changes is less intrusive - it won't change the space required to
hold the sourcePointer, nor the place where it held (4 bytes in
trailer) unless you allow to have large .source/.changes files and
therefore may need more than 4 bytes to encode the sourcepointer for
some of the methods.

> Dave
>
>
>



--
Best regards,
Igor Stasenko AKA sig.

Eliot Miranda wrote:
Amen.

Cheers,
   - Andreas

2009/12/15 Eliot Miranda <[hidden email]>:
Oh yeah.. SourceFilesArray. A quite representative illustration of
lack of abstraction.

I would call it at SourceCodeProvider and give a protocol:
#sourceCodeAt: sourcePointer
and
#methodStampAt: sourcePointer
and no, no more RemoteString crap.

Also, i tried to offer something to those who want to store sources
differently than system's default.
This is the reason why i added encoding of an arbitrary string identifier.
So, the scheme could be like: First, request comes to a class, where
the method installed, then class may either
answer the source code, or pass request further.
At the end of the day, you could use UUIDs for each method, and store
source code in some database.
Then provide a layer which talks with database for retrieving source
code. Since UUIDs do not need to be changed (in contrast to
sourcePointer, which needs to be changed if you compress the .changes
or .sources files), you can assign it once and keep forever.

The class+selector case is added to support metaprogramming. When you
having an anonymous class (or something which behaves like class), it
is pointless to log the source code in .changes when compiling the
methods, but still you could be able to store it somewhere, so user,
while debugging will see original source, instead of decompiled one.

>
>
>



--
Best regards,
Igor Stasenko AKA sig.