Smalltalk › Pharo › Pharo Smalltalk Developers

[Pharo-users] RB AST serialization

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

9 messages Options

Daniel Galdames

[Pharo-users] RB AST serialization

Hi, I'm working on a serialization of a RB AST, is in a very very earlier state, it can encode simple ASTs into a string but today i changed the encode methods, so now you can't decode the generated string :-P .Some node from AST-Core can't be encoded yet (RBPragmaNode, RBArrayNode, RBLiteralArrayNode) because I'm not sure what to encode, and the nodes aren't documented, at least in Pharo 1.0.

Any comments and suggestions will be appreciated!! :-)

I uploaded the code in squeaksource:

http://www.squeaksource.com/AstSerializer.html

I'm a bit new to Smalltalk and Pharo, so feel free to comment...
Greetings,
  Daniel Galdames G.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Lukas Renggli

Re: [Pharo-users] RB AST serialization

Hi Daniel,

I haven't looked at your code yet, but wonder what your goal is?

How is your serialization different from

source := aProgramNode source

?

How is your deserialization different from

RBParser parseMethod: source

?

Cheers,
Lukas

2010/4/27 Daniel Galdames <[hidden email]>:

> Hi, I'm working on a serialization of a RB AST, is in a very very earlier
> state, it can encode simple ASTs into a string but today i changed the
> encode methods, so now you can't decode the generated string :-P .Some node
> from AST-Core can't be encoded yet (RBPragmaNode, RBArrayNode,
> RBLiteralArrayNode) because I'm not sure what to encode, and the nodes
> aren't documented, at least in Pharo 1.0.
> Any comments and suggestions will be appreciated!! :-)
> I uploaded the code in squeaksource:
>
> http://www.squeaksource.com/AstSerializer.html
>
> I'm a bit new to Smalltalk and Pharo, so feel free to comment...
>
> Greetings,
>
> Daniel Galdames G.
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

--
Lukas Renggli
www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Stéphane Ducasse

Re: [Pharo-users] RB AST serialization

In reply to this post by Daniel Galdames

Daniel what would be really cool is to see if we can compact/zip AST because
marcus is really interested into slimbinary libraries and having a really strong compaction
of AST would be an interesting basis for a lot of experience

On Apr 27, 2010, at 6:45 AM, Daniel Galdames wrote:

> Hi, I'm working on a serialization of a RB AST, is in a very very earlier state, it can encode simple ASTs into a string but today i changed the encode methods, so now you can't decode the generated string :-P .Some node from AST-Core can't be encoded yet (RBPragmaNode, RBArrayNode, RBLiteralArrayNode) because I'm not sure what to encode, and the nodes aren't documented, at least in Pharo 1.0.
> Any comments and suggestions will be appreciated!! :-)
> I uploaded the code in squeaksource:
> http://www.squeaksource.com/AstSerializer.html
>
> I'm a bit new to Smalltalk and Pharo, so feel free to comment...
> Greetings,
> Daniel Galdames G.
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Marcus Denker-4

Re: [Pharo-users] RB AST serialization

In reply to this post by Lukas Renggli

On Apr 27, 2010, at 7:48 AM, Lukas Renggli wrote:

> Hi Daniel,
>
> I haven't looked at your code yet, but wonder what your goal is?
>
> How is your serialization different from
>
> source := aProgramNode source
>
> ?
>
> How is your deserialization different from
>
> RBParser parseMethod: source
>
> ?

In Reflectivity, we have an AST for every method. In addition, the AST can be
annotated with non-source visible annotations.

So what I want to explore is to have a very compact representations of ASTs, but one
that does not lose information *and* that allows the AST to be annotated with non-textual
objects even when it's compressed.

Marcus

--
Marcus Denker -- http://www.marcusdenker.de
INRIA Lille -- Nord Europe. Team RMoD.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Lukas Renggli

Re: [Pharo-users] RB AST serialization

> So what I want to explore is to have a very compact representations of ASTs, but one
> that does not lose information *and* that allows the AST to be annotated with non-textual
> objects even when it's compressed.

Indeed, annotations (or properties) get lost when serializing an AST
as source code. However I doubt that a different encoding of the AST
as a string is any more efficient than just storing the (compressed)
original source code (see the tests of AstSerializer). Also having a
"custom pretty printer" and a "custom parser" makes it kind of
difficult to have code that cannot be represented with the standard
AST, as this is for example the case in Helvetia.

What about storing the annotations (or properties) separate the the
source string and map them back to the AST after parsing?

Lukas

>
> Marcus
>
> --
> Marcus Denker -- http://www.marcusdenker.de
> INRIA Lille -- Nord Europe. Team RMoD.
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

--
Lukas Renggli
www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Stéphane Ducasse

Re: [Pharo-users] RB AST serialization

what theo d hondt told us a while ago is that pascal or oberon
they did not store bytecode but compressed ast and it was shorter than bytecode

Stef

On Apr 27, 2010, at 1:44 PM, Lukas Renggli wrote:

>> So what I want to explore is to have a very compact representations of ASTs, but one
>> that does not lose information *and* that allows the AST to be annotated with non-textual
>> objects even when it's compressed.
>
> Indeed, annotations (or properties) get lost when serializing an AST
> as source code. However I doubt that a different encoding of the AST
> as a string is any more efficient than just storing the (compressed)
> original source code (see the tests of AstSerializer). Also having a
> "custom pretty printer" and a "custom parser" makes it kind of
> difficult to have code that cannot be represented with the standard
> AST, as this is for example the case in Helvetia.
>
> What about storing the annotations (or properties) separate the the
> source string and map them back to the AST after parsing?
>
> Lukas
>
>
>
>
>
>
>>
>> Marcus
>>
>> --
>> Marcus Denker -- http://www.marcusdenker.de
>> INRIA Lille -- Nord Europe. Team RMoD.
>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>
>
>
> --
> Lukas Renggli
> www.lukas-renggli.ch
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Lukas Renggli

Re: [Pharo-users] RB AST serialization

> what theo d hondt told us a while ago is that pascal or oberon
> they did not store bytecode but compressed ast and it was shorter than bytecode

Sounds strange (imagine if you have comments in your code). Are there
any references on this?

Lukas

--
Lukas Renggli
www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Stéphane Ducasse

Re: [Pharo-users] RB AST serialization

>
> Lukas
>
> --
> Lukas Renggli
> www.lukas-renggli.ch
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Marcus Denker-4

Re: [Pharo-users] RB AST serialization

In reply to this post by Lukas Renggli

On Apr 27, 2010, at 2:33 PM, Lukas Renggli wrote:

>> what theo d hondt told us a while ago is that pascal or oberon
>> they did not store bytecode but compressed ast and it was shorter than bytecode
>
> Sounds strange

Why?

> (imagine if you have comments in your code).

There are no comments in bytecode, either. The idea is *not* to replace source code.

> Are there
> any references on this?
>
There is a lot of work on this, e.g. slim binaries:

http://portal.acm.org/citation.cfm?id=265576

or lots of work on grammar-driven compression of code, and much more.
Not that it's exactly slim binaries that we want to implement here.

We want to learn about compact serialization of AST like structures. Just learn.
First serialization. Than compression.

And the nice thing is that I am payed for exploring strange things :-)

And I think we should pospone discussion after we have learned. It's so easy to kill all energy
on a mailinglist in a minute.

Marcus

--
Marcus Denker -- http://www.marcusdenker.de
INRIA Lille -- Nord Europe. Team RMoD.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project