invalid sources for unregistered CompiledMethod

> Most CompiledMethod are registered in a methodDictionary.
> But some are not.
>
> (CompiledMethod allInstances reject: [:e | e methodClass ~~ nil and:
> [e selector ~~ nil and: [e == (e methodClass methodDictionary at: e
> selector ifAbsent: [nil])]]]) size
>
> This is often because a BlockClosure has been registered somewhere in
> a registry (a ServiceRegistry for example).
> The objects pointers chain is something like:
> a SeviceRegistry -> a ServiceAction -> a BlockClosure -> a
> MethodContext -> a CompiledMethod...
>
> Another source of MethodContext instances is a concurrent Process
> running in the image.
>
> If ever recompiled, a new instance of the CompiledMethod is installed
> in the methodDictionary.
> But the old CompiledMethod is still there, being pointed to by above chains.
> The old CompiledMethod has a source code pointer pointing to
> sourceCode somewhere in the change Log, and so far it's fine.
>
> But what if you compress the change log ?
> The registered CompiledMethod all get a valid new pointer to the new
> source location.
> But the unregistered ones retain their old pointer now pointing
> randomly in the change file.
> This is bad.
>
> (CompiledMethod allInstances select: [:e | e selector =
> #id:text:button:description:action: and: [(e methodClass
> methodDictionary at: e selector ifAbsent: [nil]) ~~ e]]) anyOne
> getSource
>
> Now if you try to decompile the method, either there is an Error like
> here, or unfortunately the random sourceCode is valid a valid chunk of
> Smalltalk and you open the door to potentially strange things... It
> could be misleading if you try to understand what the hell this
> SeviceAction could be doing.
>
> If you try to decompile the BlockClosure, and getSource does not point
> to valid code, it doesn't work, you get nil... This is another bug,
> because #decompile should revert to homeMethod decompile
> decompileString...
>
> I propose that #condenseChanges and the like, anihilate source code
> pointers of unregistered CompiledMethod.
> This should solve the issue.
> What do you think ?
>

> Most CompiledMethod are registered in a methodDictionary.
> But some are not.
>
> (CompiledMethod allInstances reject: [:e | e methodClass ~~ nil and:
> [e selector ~~ nil and: [e == (e methodClass methodDictionary at: e
> selector ifAbsent: [nil])]]]) size
>
> This is often because a BlockClosure has been registered somewhere in
> a registry (a ServiceRegistry for example).
> The objects pointers chain is something like:
> a SeviceRegistry -> a ServiceAction -> a BlockClosure -> a
> MethodContext -> a CompiledMethod...
>
> Another source of MethodContext instances is a concurrent Process
> running in the image.
>
> If ever recompiled, a new instance of the CompiledMethod is installed
> in the methodDictionary.
> But the old CompiledMethod is still there, being pointed to by above chains.
> The old CompiledMethod has a source code pointer pointing to
> sourceCode somewhere in the change Log, and so far it's fine.
>
> But what if you compress the change log ?
> The registered CompiledMethod all get a valid new pointer to the new
> source location.
> But the unregistered ones retain their old pointer now pointing
> randomly in the change file.
> This is bad.
>
> (CompiledMethod allInstances select: [:e | e selector =
> #id:text:button:description:action: and: [(e methodClass
> methodDictionary at: e selector ifAbsent: [nil]) ~~ e]]) anyOne
> getSource
>
> Now if you try to decompile the method, either there is an Error like
> here, or unfortunately the random sourceCode is valid a valid chunk of
> Smalltalk and you open the door to potentially strange things... It
> could be misleading if you try to understand what the hell this
> SeviceAction could be doing.
>
> If you try to decompile the BlockClosure, and getSource does not point
> to valid code, it doesn't work, you get nil... This is another bug,
> because #decompile should revert to homeMethod decompile
> decompileString...
>
> I propose that #condenseChanges and the like, anihilate source code
> pointers of unregistered CompiledMethod.
> This should solve the issue.
> What do you think ?
>

> Ah, I forgot to say that if you recompile the methodClass, it won't
> trigger recompilation of those unregistered methods.
> Some actions like removing an inst var can then trigger very bad
> things when you try to execute those unregistered methods, like
> corrupting memory.
>
> A security in the class builder would be to:
> - mutate all unregistered CompiledMethod pointing to this methodClass
>  before the oldClass becomeForward: newClass
>
> Then, a security in the VM should prevent those CompiledMethod mutant
> to be executed.
> (or maybe the mutant could simply be modified to send a message like
> self mustBeRecompiled).
>
> What do you think of this one ?
>

>
> 2009/11/6 Nicolas Cellier <[hidden email]>:
>> Most CompiledMethod are registered in a methodDictionary.
>> But some are not.
>>
>> (CompiledMethod allInstances reject: [:e | e methodClass ~~ nil and:
>> [e selector ~~ nil and: [e == (e methodClass methodDictionary at: e
>> selector ifAbsent: [nil])]]]) size
>>
>> This is often because a BlockClosure has been registered somewhere in
>> a registry (a ServiceRegistry for example).
>> The objects pointers chain is something like:
>> a SeviceRegistry -> a ServiceAction -> a BlockClosure -> a
>> MethodContext -> a CompiledMethod...
>>
>> Another source of MethodContext instances is a concurrent Process
>> running in the image.
>>
>> If ever recompiled, a new instance of the CompiledMethod is installed
>> in the methodDictionary.
>> But the old CompiledMethod is still there, being pointed to by above chains.
>> The old CompiledMethod has a source code pointer pointing to
>> sourceCode somewhere in the change Log, and so far it's fine.
>>
>> But what if you compress the change log ?
>> The registered CompiledMethod all get a valid new pointer to the new
>> source location.
>> But the unregistered ones retain their old pointer now pointing
>> randomly in the change file.
>> This is bad.
>>
>> (CompiledMethod allInstances select: [:e | e selector =
>> #id:text:button:description:action: and: [(e methodClass
>> methodDictionary at: e selector ifAbsent: [nil]) ~~ e]]) anyOne
>> getSource
>>
>> Now if you try to decompile the method, either there is an Error like
>> here, or unfortunately the random sourceCode is valid a valid chunk of
>> Smalltalk and you open the door to potentially strange things... It
>> could be misleading if you try to understand what the hell this
>> SeviceAction could be doing.
>>
>> If you try to decompile the BlockClosure, and getSource does not point
>> to valid code, it doesn't work, you get nil... This is another bug,
>> because #decompile should revert to homeMethod decompile
>> decompileString...
>>
>> I propose that #condenseChanges and the like, anihilate source code
>> pointers of unregistered CompiledMethod.
>> This should solve the issue.
>> What do you think ?
>>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

> Nicolas Cellier wrote:
>>
>> But what if you compress the change log ?
>
> Perhaps it's time to finally get rid of source pointers and just keep
> the source code as objects in the image, referenced directly from the
> CompiledMethod.
>
> External sources were necessary 25 years ago when an image without
> sources consumed 3/4 of the total RAM of the machine, but it's not  
> like
> that anymore.
>

>
> On Nov 7, 2009, at 6:05 PM, Martin McClure wrote:
>
>> Nicolas Cellier wrote:
>>>
>>> But what if you compress the change log ?
>>
>> Perhaps it's time to finally get rid of source pointers and just keep
>> the source code as objects in the image, referenced directly from the
>> CompiledMethod.
>>
>> External sources were necessary 25 years ago when an image without
>> sources consumed 3/4 of the total RAM of the machine, but it's not
>> like
>> that anymore.
>>
> +100
>
> I did that once as an experiment: searching for strings in all the
> code gets *fast*.
>
>        Marcus
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

>>>
>>> Perhaps it's time to finally get rid of source pointers and just  
>>> keep
>>> the source code as objects in the image, referenced directly from  
>>> the
>>> CompiledMethod.
>>>
>>> External sources were necessary 25 years ago when an image without
>>> sources consumed 3/4 of the total RAM of the machine, but it's not
>>> like
>>> that anymore.
>>>
>> +100
>>
>> I did that once as an experiment: searching for strings in all the
>> code gets *fast*.
>>
> I'd say +16M
>
> This is what i got when condensing sources:
> 16249263 2009-10-19 20:25 PharoV10.sources

>
> On Nov 7, 2009, at 7:13 PM, Nicolas Cellier wrote:
>>>>
>>>> Perhaps it's time to finally get rid of source pointers and just
>>>> keep
>>>> the source code as objects in the image, referenced directly from
>>>> the
>>>> CompiledMethod.
>>>>
>>>> External sources were necessary 25 years ago when an image without
>>>> sources consumed 3/4 of the total RAM of the machine, but it's not
>>>> like
>>>> that anymore.
>>>>
>>> +100
>>>
>>> I did that once as an experiment: searching for strings in all the
>>> code gets *fast*.
>>>
>> I'd say +16M
>>
>> This is what i got when condensing sources:
>> 16249263 2009-10-19 20:25 PharoV10.sources
>
>
> yes, but nevertheless, this is how much percent of the Ram of your
> machine?
>
> Two notes:
>
> - We could compress the source-strings. That should lead to something
> like
> 5MB increase.
>
> - We should invest into a general mechanism of swapping in unused
> objects on demand.
> It's fun that there is a special case for the source, but all the
> other resources happily live
> in the image (bitmaps, sounds... in general objects not part of the
> working set, e.g. I guess 80% of
> all compiled methods...).
>
> Marcus
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

> Adrian Lienhard wrote:
>> Yes, in "normal" use of Pharo, holding the sources in memory would
>> only consume a fraction of the available RAM. Some applications,
>> however, have several hundred images running at the same time on a
>> server. In this case the picture looks very different again.
>
> It would be easy to strip all sources from production images, but the
> simple way of doing that would mean any debugging would be done on
> decompiled code.
>
>>
>> I think Marcus' idea of swapping in/out ressources is very
>> interesting. We could get rid of the ancient sources handling but
>> still have a low, or even lower, memory footprint.
>>
>
> This would be even better.
> ImageSegments are a start in this direction.

>
> I think Marcus' idea of swapping in/out ressources is very  
> interesting. We could get rid of the ancient sources handling but  
> still have a low, or even lower, memory footprint.
>
> Cheers,
> Adrian
>
> On Nov 7, 2009, at 19:21 , Marcus Denker wrote:
>
> >
> > On Nov 7, 2009, at 7:13 PM, Nicolas Cellier wrote:
> >>>>
> >>>> Perhaps it's time to finally get rid of source pointers and just
> >>>> keep
> >>>> the source code as objects in the image, referenced directly from
> >>>> the
> >>>> CompiledMethod.
> >>>>
> >>>> External sources were necessary 25 years ago when an image without
> >>>> sources consumed 3/4 of the total RAM of the machine, but it's not
> >>>> like
> >>>> that anymore.
> >>>>
> >>> +100
> >>>
> >>> I did that once as an experiment: searching for strings in all the
> >>> code gets *fast*.
> >>>
> >> I'd say +16M
> >>
> >> This is what i got when condensing sources:
> >> 16249263 2009-10-19 20:25 PharoV10.sources
> >
> >
> > yes, but nevertheless, this is how much percent of the Ram of your
> > machine?
> >
> > Two notes:
> >
> > - We could compress the source-strings. That should lead to something
> > like
> > 5MB increase.
> >
> > - We should invest into a general mechanism of swapping in unused
> > objects on demand.
> > It's fun that there is a special case for the source, but all the
> > other resources happily live
> > in the image (bitmaps, sounds... in general objects not part of the
> > working set, e.g. I guess 80% of
> > all compiled methods...).
> >
> > Marcus
> >
> > _______________________________________________
> > Pharo-project mailing list
> > [hidden email]
> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project