Breaking another dependency to Compiler (related to Traits I think)

> Hi guys. Today, there is a part of the system coupled with Compiler. Issue is: http://code.google.com/p/pharo/issues/detail?id=6110
>
> Now, if you see the method:
>
> getSourceReplacingSelectorWith: newSelector
> | oldKeywords newKeywords args newSelectorWithArgs source oldSelector s |
> source := self sourceCode.
> oldSelector := self parserClass new parseSelector: source.
> oldSelector = newSelector ifTrue: [ ^ source ].
> oldKeywords := oldSelector keywords.
> newKeywords := (newSelector ifNil: [self defaultSelector]) keywords.
> [oldKeywords size = newKeywords size] assert.
> args := (self methodClass parserClass new
> parseArgsAndTemps: source string notifying: nil) copyFrom: 1 to: self numArgs.
> newSelectorWithArgs := String streamContents: [:stream |
> newKeywords withIndexDo: [:keyword :index |
> stream nextPutAll: keyword.
> stream space.
> args size >= index ifTrue: [
> stream nextPutAll: (args at: index); space]]].
> s := source string readStream.
> oldKeywords do: [ :each | s match: each ].
> args isEmpty ifFalse: [ s match: args last ].
> ^newSelectorWithArgs trimBoth, s upToEnd
>
> The 2 magic lines are:
>
> oldSelector := self parserClass new parseSelector: source.
> oldSelector = newSelector ifTrue: [ ^ source ].
>
> So, why we cannot just replace "self parserClass new parseSelector: source." with "self selector". Well, I think (talking with Guille) this ONLY because of Trait transformation/aliases that associates a selector with a method of a different selector.
>
> If this is the case, it means that 99% of the cases,"oldSelector = newSelector ifTrue: [ ^ source ].  "  is true. Because those aliases are only used in one trait in all the image ().
>
> So...if that is the case, can't we get the oldSelector from elsewhere  rather than needing to use the Compiler to get the selector from the source code ?
>

>
> On Aug 14, 2012, at 2:01 PM, Mariano Martinez Peck <[hidden email]> wrote:
>
>> Hi guys. Today, there is a part of the system coupled with Compiler. Issue is: http://code.google.com/p/pharo/issues/detail?id=6110
>>
>> Now, if you see the method:
>>
>> getSourceReplacingSelectorWith: newSelector
>>       | oldKeywords newKeywords args newSelectorWithArgs source oldSelector s |
>>       source := self sourceCode.
>>       oldSelector := self parserClass new parseSelector: source.
>>       oldSelector = newSelector ifTrue: [ ^ source ].
>>       oldKeywords := oldSelector keywords.
>>       newKeywords := (newSelector ifNil: [self defaultSelector]) keywords.
>>       [oldKeywords size = newKeywords size] assert.
>>       args := (self methodClass parserClass new
>>               parseArgsAndTemps: source string notifying: nil) copyFrom: 1 to: self numArgs.
>>       newSelectorWithArgs := String streamContents: [:stream |
>>               newKeywords withIndexDo: [:keyword :index |
>>                       stream nextPutAll: keyword.
>>                       stream space.
>>                       args size >= index ifTrue: [
>>                               stream nextPutAll: (args at: index); space]]].
>>       s := source string readStream.
>>       oldKeywords do: [ :each | s match: each ].
>>       args isEmpty ifFalse: [ s match: args last ].
>>       ^newSelectorWithArgs trimBoth, s upToEnd
>>
>> The 2 magic lines are:
>>
>> oldSelector := self parserClass new parseSelector: source.
>>       oldSelector = newSelector ifTrue: [ ^ source ].
>>
>> So, why we cannot just replace "self parserClass new parseSelector: source." with "self selector". Well, I think (talking with Guille) this ONLY because of Trait transformation/aliases that associates a selector with a method of a different selector.
>>
>> If this is the case, it means that 99% of the cases,"oldSelector = newSelector ifTrue: [ ^ source ].  "  is true. Because those aliases are only used in one trait in all the image ().
>>
>> So...if that is the case, can't we get the oldSelector from elsewhere  rather than needing to use the Compiler to get the selector from the source code ?
>>
> As we now encode the selector in the CompiledMethod itself, this can be relaced with
>
>         oldSelector := self selector.
>
>
> But what you have left is the arguments...
>
> args := (self methodClass parserClass new
>                 parseArgsAndTemps: source string notifying: nil) copyFrom: 1 to: self numArgs.
>
> The only way to get the the argument names (temps) of a method is to compile the source... no way around it.
>
> (I personally think this is another example why bytecode as the model for reflection is just wrong)
>
>
>         Marcus
>
>
> --
> Marcus Denker -- http://marcusdenker.de
>
>

>
> It would ease the queries + manipulation, in the tools a lot!
>
> CompiledMethod>>getSourceReplacingSelectorWith: aSelector
> ....
>
> would be instead:
>
> CompiledMethod>>withSelector: aSelector
>  ^ (self ast withSelector: aSelector) source.

>
>
> I've been using an AST, instead of the CompiledMethods directly for
> this reason, to deal with more expressive power.
>
> Fernando
>
> On Tue, Aug 14, 2012 at 2:56 PM, Marcus Denker <[hidden email]> wrote:
>>
>> On Aug 14, 2012, at 2:01 PM, Mariano Martinez Peck <[hidden email]> wrote:
>>
>>> Hi guys. Today, there is a part of the system coupled with Compiler. Issue is: http://code.google.com/p/pharo/issues/detail?id=6110
>>>
>>> Now, if you see the method:
>>>
>>> getSourceReplacingSelectorWith: newSelector
>>>       | oldKeywords newKeywords args newSelectorWithArgs source oldSelector s |
>>>       source := self sourceCode.
>>>       oldSelector := self parserClass new parseSelector: source.
>>>       oldSelector = newSelector ifTrue: [ ^ source ].
>>>       oldKeywords := oldSelector keywords.
>>>       newKeywords := (newSelector ifNil: [self defaultSelector]) keywords.
>>>       [oldKeywords size = newKeywords size] assert.
>>>       args := (self methodClass parserClass new
>>>               parseArgsAndTemps: source string notifying: nil) copyFrom: 1 to: self numArgs.
>>>       newSelectorWithArgs := String streamContents: [:stream |
>>>               newKeywords withIndexDo: [:keyword :index |
>>>                       stream nextPutAll: keyword.
>>>                       stream space.
>>>                       args size >= index ifTrue: [
>>>                               stream nextPutAll: (args at: index); space]]].
>>>       s := source string readStream.
>>>       oldKeywords do: [ :each | s match: each ].
>>>       args isEmpty ifFalse: [ s match: args last ].
>>>       ^newSelectorWithArgs trimBoth, s upToEnd
>>>
>>> The 2 magic lines are:
>>>
>>> oldSelector := self parserClass new parseSelector: source.
>>>       oldSelector = newSelector ifTrue: [ ^ source ].
>>>
>>> So, why we cannot just replace "self parserClass new parseSelector: source." with "self selector". Well, I think (talking with Guille) this ONLY because of Trait transformation/aliases that associates a selector with a method of a different selector.
>>>
>>> If this is the case, it means that 99% of the cases,"oldSelector = newSelector ifTrue: [ ^ source ].  "  is true. Because those aliases are only used in one trait in all the image ().
>>>
>>> So...if that is the case, can't we get the oldSelector from elsewhere  rather than needing to use the Compiler to get the selector from the source code ?
>>>
>> As we now encode the selector in the CompiledMethod itself, this can be relaced with
>>
>>         oldSelector := self selector.
>>
>>
>> But what you have left is the arguments...
>>
>> args := (self methodClass parserClass new
>>                 parseArgsAndTemps: source string notifying: nil) copyFrom: 1 to: self numArgs.
>>
>> The only way to get the the argument names (temps) of a method is to compile the source... no way around it.
>>
>> (I personally think this is another example why bytecode as the model for reflection is just wrong)
>>
>>
>>         Marcus
>>
>>
>> --
>> Marcus Denker -- http://marcusdenker.de
>>
>>
>
>

> On Wed, 15 Aug 2012, Fernando Olivero wrote:
>
>> Hi, great point Marcus.  I was wondering why cant a compiled method
>> know the "ast" that generated it?
>
> Would you store all ASTs in memory all the time or recreate it every time
> on demand?
>
>>
>> It would ease the queries + manipulation, in the tools a lot!
>>
>> CompiledMethod>>getSourceReplacingSelectorWith: aSelector
>> ....
>>
>> would be instead:
>>
>> CompiledMethod>>withSelector: aSelector
>>  ^ (self ast withSelector: aSelector) source.
>
> Is there an implementation of AST that can reproduce the exact same source
> code?
>
>
> Levente
>
>>
>>
>> I've been using an AST, instead of the CompiledMethods directly for
>> this reason, to deal with more expressive power.
>>
>> Fernando
>>
>> On Tue, Aug 14, 2012 at 2:56 PM, Marcus Denker <[hidden email]> wrote:
>>>
>>> On Aug 14, 2012, at 2:01 PM, Mariano Martinez Peck <[hidden email]> wrote:
>>>
>>>> Hi guys. Today, there is a part of the system coupled with Compiler. Issue is: http://code.google.com/p/pharo/issues/detail?id=6110
>>>>
>>>> Now, if you see the method:
>>>>
>>>> getSourceReplacingSelectorWith: newSelector
>>>>       | oldKeywords newKeywords args newSelectorWithArgs source oldSelector s |
>>>>       source := self sourceCode.
>>>>       oldSelector := self parserClass new parseSelector: source.
>>>>       oldSelector = newSelector ifTrue: [ ^ source ].
>>>>       oldKeywords := oldSelector keywords.
>>>>       newKeywords := (newSelector ifNil: [self defaultSelector]) keywords.
>>>>       [oldKeywords size = newKeywords size] assert.
>>>>       args := (self methodClass parserClass new
>>>>               parseArgsAndTemps: source string notifying: nil) copyFrom: 1 to: self numArgs.
>>>>       newSelectorWithArgs := String streamContents: [:stream |
>>>>               newKeywords withIndexDo: [:keyword :index |
>>>>                       stream nextPutAll: keyword.
>>>>                       stream space.
>>>>                       args size >= index ifTrue: [
>>>>                               stream nextPutAll: (args at: index); space]]].
>>>>       s := source string readStream.
>>>>       oldKeywords do: [ :each | s match: each ].
>>>>       args isEmpty ifFalse: [ s match: args last ].
>>>>       ^newSelectorWithArgs trimBoth, s upToEnd
>>>>
>>>> The 2 magic lines are:
>>>>
>>>> oldSelector := self parserClass new parseSelector: source.
>>>>       oldSelector = newSelector ifTrue: [ ^ source ].
>>>>
>>>> So, why we cannot just replace "self parserClass new parseSelector: source." with "self selector". Well, I think (talking with Guille) this ONLY because of Trait transformation/aliases that associates a selector with a method of a different selector.
>>>>
>>>> If this is the case, it means that 99% of the cases,"oldSelector = newSelector ifTrue: [ ^ source ].  "  is true. Because those aliases are only used in one trait in all the image ().
>>>>
>>>> So...if that is the case, can't we get the oldSelector from elsewhere  rather than needing to use the Compiler to get the selector from the source code ?
>>>>
>>> As we now encode the selector in the CompiledMethod itself, this can be relaced with
>>>
>>>         oldSelector := self selector.
>>>
>>>
>>> But what you have left is the arguments...
>>>
>>> args := (self methodClass parserClass new
>>>                 parseArgsAndTemps: source string notifying: nil) copyFrom: 1 to: self numArgs.
>>>
>>> The only way to get the the argument names (temps) of a method is to compile the source... no way around it.
>>>
>>> (I personally think this is another example why bytecode as the model for reflection is just wrong)
>>>
>>>
>>>         Marcus
>>>
>>>
>>> --
>>> Marcus Denker -- http://marcusdenker.de
>>>
>>>
>>
>>

> I've been using RB, which allows to recreate on demand the ast, but in
> my system i cached those values.
>
>
> ast := RBParser parseMethod: 'parsingAMethod
> | yeah |
> yeah := self doesAMethodNode remember        its            source.
> yeah
> ifTrue:[ ^ #hurray]'.
>
> ast source.
> ast formattedCode.
>
> ast := (Object>>#at:) parseTree.
> ast source.
> ast formattedCode
>
>
> Regarding the memory usage, in my system only the methods which have a
> "method shape" opened in the IDE, store their ast, thus i cannot
> provide numbers of the overhead. But why should one extra object more
> per method be a problem?
>

>
> On Aug 16, 2012, at 9:32 AM, Fernando Olivero <[hidden email]> wrote:
>
>> I've been using RB, which allows to recreate on demand the ast, but in
>> my system i cached those values.
>>
>>
>> ast := RBParser parseMethod: 'parsingAMethod
>>       | yeah |
>>       yeah := self doesAMethodNode remember        its            source.
>>       yeah
>>               ifTrue:[ ^ #hurray]'.
>>
>> ast source.
>> ast formattedCode.
>>
>> ast := (Object>>#at:) parseTree.
>> ast source.
>> ast formattedCode
>>
>>
>> Regarding the memory usage, in my system only the methods which have a
>> "method shape" opened in the IDE, store their ast, thus i cannot
>> provide numbers of the overhead. But why should one extra object more
>> per method be a problem?
>>
>
> It's not one object... it's a tree. I did once a system that stored the AST for each method
> (uncompressed) in addition to what is there now (CompiledMethod).
> The system was 120MB instead of 20MB. (based on Squeak 3.9)
>
> Which means, even then smaller than Eclipse ;-)
>
>         Marcus
>
>
> --
> Marcus Denker -- http://marcusdenker.de
>

>
> On Aug 16, 2012, at 4:05 PM, Fernando Olivero <[hidden email]> wrote:
>
>> NICE! I would say lets go for it, 120MB is nothing nowadays. The IDE
>> could be simplied and be more powerful.
>>
>
> it kind of changes quite drasitically what you can do concerning behavioral reflection...
>
> As for space, compression should solve that quite nicely.

>
>
> Fernando
>
> On Thu, Aug 16, 2012 at 9:35 AM, Marcus Denker <[hidden email]> wrote:
>>
>> On Aug 16, 2012, at 9:32 AM, Fernando Olivero <[hidden email]> wrote:
>>
>>> I've been using RB, which allows to recreate on demand the ast, but in
>>> my system i cached those values.
>>>
>>>
>>> ast := RBParser parseMethod: 'parsingAMethod
>>>       | yeah |
>>>       yeah := self doesAMethodNode remember        its            source.
>>>       yeah
>>>               ifTrue:[ ^ #hurray]'.
>>>
>>> ast source.
>>> ast formattedCode.
>>>
>>> ast := (Object>>#at:) parseTree.
>>> ast source.
>>> ast formattedCode
>>>
>>>
>>> Regarding the memory usage, in my system only the methods which have a
>>> "method shape" opened in the IDE, store their ast, thus i cannot
>>> provide numbers of the overhead. But why should one extra object more
>>> per method be a problem?
>>>
>>
>> It's not one object... it's a tree. I did once a system that stored the AST for each method
>> (uncompressed) in addition to what is there now (CompiledMethod).
>> The system was 120MB instead of 20MB. (based on Squeak 3.9)
>>
>> Which means, even then smaller than Eclipse ;-)
>>
>>         Marcus
>>
>>
>> --
>> Marcus Denker -- http://marcusdenker.de
>>
>
>

> On Thu, 16 Aug 2012, Fernando Olivero wrote:
>
>> NICE! I would say lets go for it, 120MB is nothing nowadays. The IDE
>> could be simplied and be more powerful.
>
> Let's be a bit more realistic. The current garbage collector is choking if
> you add 100 MB of small objects to your image. Caching the recently
> accessed ASTs sounds like a better idea.
>
>>
>> For example, in gaucho i'm using the ast for pretty printing of the
>> source, and for styling unsed variables differently, etc...
>
> You didn't answer yet if RB's AST can reproduce the exact same source
> string or not.
>
>
> Levente
>
>>
>>
>> Fernando
>>
>> On Thu, Aug 16, 2012 at 9:35 AM, Marcus Denker <[hidden email]> wrote:
>>>
>>> On Aug 16, 2012, at 9:32 AM, Fernando Olivero <[hidden email]> wrote:
>>>
>>>> I've been using RB, which allows to recreate on demand the ast, but in
>>>> my system i cached those values.
>>>>
>>>>
>>>> ast := RBParser parseMethod: 'parsingAMethod
>>>>       | yeah |
>>>>       yeah := self doesAMethodNode remember        its            source.
>>>>       yeah
>>>>               ifTrue:[ ^ #hurray]'.
>>>>
>>>> ast source.
>>>> ast formattedCode.
>>>>
>>>> ast := (Object>>#at:) parseTree.
>>>> ast source.
>>>> ast formattedCode
>>>>
>>>>
>>>> Regarding the memory usage, in my system only the methods which have a
>>>> "method shape" opened in the IDE, store their ast, thus i cannot
>>>> provide numbers of the overhead. But why should one extra object more
>>>> per method be a problem?
>>>>
>>>
>>> It's not one object... it's a tree. I did once a system that stored the AST for each method
>>> (uncompressed) in addition to what is there now (CompiledMethod).
>>> The system was 120MB instead of 20MB. (based on Squeak 3.9)
>>>
>>> Which means, even then smaller than Eclipse ;-)
>>>
>>>         Marcus
>>>
>>>
>>> --
>>> Marcus Denker -- http://marcusdenker.de
>>>
>>
>>

> I read a paper once (at the University of New Brunswick (Canada))
> library in which the grammar for a language was used to construct a
> data compressor for syntaxly correct programs.  The output was a
> compressed form of a parse/syntax tree with comments and white space
> included. Once decompressed, the tree could be used to reconstruct
> the source.  I don't know if a compressed syntax tree would be very useful for
> programming tasks but it could be used to compress Smalltalk's source
> file and changes file.  This assumes the time saved by filing in
> smaller chucks of programs segments would greater than the cost of
> decompression.

>
> On Aug 16, 2012, at 6:13 PM, Ralph Boland wrote:
>
>> I read a paper once (at the University of New Brunswick (Canada))
>> library in which the grammar for a language was used to construct a
>> data compressor for syntaxly correct programs.  The output was a
>> compressed form of a parse/syntax tree with comments and white space
>> included. Once decompressed, the tree could be used to reconstruct
>> the source.  I don't know if a compressed syntax tree would be very useful for
>> programming tasks but it could be used to compress Smalltalk's source
>> file and changes file.  This assumes the time saved by filing in
>> smaller chucks of programs segments would greater than the cost of
>> decompression.
>
> do you have the reference?
> One guy told me and in Oberon or something like that the ast was compressed long time ago.

>>> I read a paper once (at the University of New Brunswick (Canada))
>>> library in which the grammar for a language was used to construct a
>>> data compressor for syntaxly correct programs.  The output was a
>>> compressed form of a parse/syntax tree with comments and white space
>>> included. Once decompressed, the tree could be used to reconstruct
>>> the source.  I don't know if a compressed syntax tree would be very useful for
>>> programming tasks but it could be used to compress Smalltalk's source
>>> file and changes file.  This assumes the time saved by filing in
>>> smaller chucks of programs segments would greater than the cost of
>>> decompression.
>>
>> do you have the reference?
>> One guy told me and in Oberon or something like that the ast was compressed long time ago.
>
> Franz, M. & Kistler, T. (1997), 'Slim Binaries', Commun. ACM 40 (12) , 87--94 .
> http://dl.acm.org/citation.cfm?id=265576