Capturing messages

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Capturing messages

Alejandro Pulver
Hello,

I would like to intercept all messages sent in the system, as part of a small project to investigate if the information can be used in useful ways (autocompletion, object compatibility detection, etc).

It will also involve filtering, aggregating and enabling/disabling it for space and time conservation,

I'm new to Pharo, but had some success by changing the main perform (perform:withArguments: IIRC) method in Object, and making others use it instead of the primitives.
Someone told me it doesn't capture most messages because of optimizations done by the system.

My other thought was to use the profiler's spyOn as an approximation.

What do you think?

Thanks in Advance,
Alejandro
Reply | Threaded
Open this post in threaded view
|

Re: Capturing messages

Paul DeBruicker
Maybe you should use MethodWrappers.


http://forum.world.st/MethodWrappers-td3829576.html



On 09/23/2013 09:56 AM, Alejandro Pulver wrote:

> Hello,
>
> I would like to intercept all messages sent in the system, as part of a
> small project to investigate if the information can be used in useful
> ways (autocompletion, object compatibility detection, etc).
>
> It will also involve filtering, aggregating and enabling/disabling it
> for space and time conservation,
>
> I'm new to Pharo, but had some success by changing the main perform
> (perform:withArguments: IIRC) method in Object, and making others use it
> instead of the primitives.
> Someone told me it doesn't capture most messages because of
> optimizations done by the system.
>
> My other thought was to use the profiler's spyOn as an approximation.
>
> What do you think?
>
> Thanks in Advance,
> Alejandro


Reply | Threaded
Open this post in threaded view
|

Re: Capturing messages

Alejandro Pulver
Thanks for the reply, but the links mentioned in the forum are down. Also, I would a solution that works in Pharo 2.

Wrapping every method in the system doesn't look like an efficient solution. I know about Bifrost but it's implemented in 1.0 and not likely to be ported to 3 soon (according to a discussion I found recently).

Do you know a reason why the profiler of changing perform won't work?



On Mon, Sep 23, 2013 at 3:11 PM, Paul DeBruicker <[hidden email]> wrote:
Maybe you should use MethodWrappers.


http://forum.world.st/MethodWrappers-td3829576.html



On 09/23/2013 09:56 AM, Alejandro Pulver wrote:
> Hello,
>
> I would like to intercept all messages sent in the system, as part of a
> small project to investigate if the information can be used in useful
> ways (autocompletion, object compatibility detection, etc).
>
> It will also involve filtering, aggregating and enabling/disabling it
> for space and time conservation,
>
> I'm new to Pharo, but had some success by changing the main perform
> (perform:withArguments: IIRC) method in Object, and making others use it
> instead of the primitives.
> Someone told me it doesn't capture most messages because of
> optimizations done by the system.
>
> My other thought was to use the profiler's spyOn as an approximation.
>
> What do you think?
>
> Thanks in Advance,
> Alejandro



Reply | Threaded
Open this post in threaded view
|

Re: Capturing messages

Marcus Denker-4

On Sep 24, 2013, at 9:09 AM, Alejandro Pulver <[hidden email]> wrote:

> Thanks for the reply, but the links mentioned in the forum are down. Also, I would a solution that works in Pharo 2.
>
> Wrapping every method in the system doesn't look like an efficient solution. I know about Bifrost but it's implemented in 1.0 and not likely to be ported to 3 soon (according to a discussion I found recently).
>
> Do you know a reason why the profiler of changing perform won't work?
>
perform: is not called when doing a message send. Reason: it would be too slow to explicitly call a method to do a message send. Everything is in the VM.

We are working (slowly) on a new version of Reflectivity (Partial Behavioral Reflection) that is the basis of bifrost.
But even then, if you replace every message send by multiple ones, the result will be a *very* slow system.

This is in the end the real problem of behavioral reflection: it is not only difficult to find a good model and implement it in a way that it works
for real even in strange cases (e.g. kernel classes that are used by the reflective layer itself).

The real challenge is to make it fast. And that is not easy: just replacing the message send byte code by o multiple reflective sends will
slow things down quite a lot, regardless how clever you are.

        Marcus

signature.asc (210 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Capturing messages

Alejandro Pulver
I see. But what about sampling? It shouldn't slow down the program nearly as much as these "instrumentation" like mechanisms.

And as I'm going to count them anyways, missing the most unused ones won't be such a problem.

That combined with complete capture for a small run (like a test or block) would be enough for our needs.


On Tue, Sep 24, 2013 at 4:19 AM, Marcus Denker <[hidden email]> wrote:

On Sep 24, 2013, at 9:09 AM, Alejandro Pulver <[hidden email]> wrote:

> Thanks for the reply, but the links mentioned in the forum are down. Also, I would a solution that works in Pharo 2.
>
> Wrapping every method in the system doesn't look like an efficient solution. I know about Bifrost but it's implemented in 1.0 and not likely to be ported to 3 soon (according to a discussion I found recently).
>
> Do you know a reason why the profiler of changing perform won't work?
>
perform: is not called when doing a message send. Reason: it would be too slow to explicitly call a method to do a message send. Everything is in the VM.

We are working (slowly) on a new version of Reflectivity (Partial Behavioral Reflection) that is the basis of bifrost.
But even then, if you replace every message send by multiple ones, the result will be a *very* slow system.

This is in the end the real problem of behavioral reflection: it is not only difficult to find a good model and implement it in a way that it works
for real even in strange cases (e.g. kernel classes that are used by the reflective layer itself).

The real challenge is to make it fast. And that is not easy: just replacing the message send byte code by o multiple reflective sends will
slow things down quite a lot, regardless how clever you are.

        Marcus

Reply | Threaded
Open this post in threaded view
|

Re: Capturing messages

Marcus Denker-4

On Sep 24, 2013, at 9:41 AM, Alejandro Pulver <[hidden email]> wrote:

> I see. But what about sampling? It shouldn't slow down the program nearly as much as these "instrumentation" like mechanisms.
>
> And as I'm going to count them anyways, missing the most unused ones won't be such a problem.
>
> That combined with complete capture for a small run (like a test or block) would be enough for our needs.
>
>

Yes, one could just use some random selection (I really like the idea of shotgun sequencing… http://en.wikipedia.org/wiki/Shotgun_sequencing)
The resulting information should be quite good.

But the first step is that we need a really good, very dynamic, very general and powerful basis… this is what we want to build
with Reflectivity2 (and Reflectivity3 will have to hook deeper into the JIT of the VM, but that is another story).

Related to step 3), another source for interesting data is the VM itself (e.g. caches).

But step by step…

The idea is to take the new possibilities we have in Pharo3 and make a well engineered realization of partial behavioral reflection,
with two extensions:
        -> meta links on instance  variables. Very cool possibilities now that instance variables are reified objects ("Slot")
        -> object specific meta links by integrating with a good proxy implementation

This will already bring amazing capabilities to the system (and subsume MethodWrappers, simple CLOS style MOPs, perfect for
breakpoints, watchpoints and debugging in general,  and it can be the basis for AOP like things).

        Marcus


signature.asc (210 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Capturing messages

Alejandro Pulver
Thanks for the idea, I'll try emulating shotgun sequencing by increasing the sampling rate for short periods of time.

But a better alternative would be to enable and disable the simulator (context's runSimulated) for exact sequences (even if shorter, to avoid slowdowns). Is it possible with the current implementation?

It's not that I don't appreciate the ideas for the new reflection framework. but I'm looking for something to use for this project. And we have only a few weeks. Actually, the most important part is not the system integration but analyzing the structure of those message sequences (like word analysis in natural language processing).


On Tue, Sep 24, 2013 at 4:59 AM, Marcus Denker <[hidden email]> wrote:

On Sep 24, 2013, at 9:41 AM, Alejandro Pulver <[hidden email]> wrote:

> I see. But what about sampling? It shouldn't slow down the program nearly as much as these "instrumentation" like mechanisms.
>
> And as I'm going to count them anyways, missing the most unused ones won't be such a problem.
>
> That combined with complete capture for a small run (like a test or block) would be enough for our needs.
>
>

Yes, one could just use some random selection (I really like the idea of shotgun sequencing… http://en.wikipedia.org/wiki/Shotgun_sequencing)
The resulting information should be quite good.

But the first step is that we need a really good, very dynamic, very general and powerful basis… this is what we want to build
with Reflectivity2 (and Reflectivity3 will have to hook deeper into the JIT of the VM, but that is another story).

Related to step 3), another source for interesting data is the VM itself (e.g. caches).

But step by step…

The idea is to take the new possibilities we have in Pharo3 and make a well engineered realization of partial behavioral reflection,
with two extensions:
        -> meta links on instance  variables. Very cool possibilities now that instance variables are reified objects ("Slot")
        -> object specific meta links by integrating with a good proxy implementation

This will already bring amazing capabilities to the system (and subsume MethodWrappers, simple CLOS style MOPs, perfect for
breakpoints, watchpoints and debugging in general,  and it can be the basis for AOP like things).

        Marcus


Reply | Threaded
Open this post in threaded view
|

Re: Capturing messages

Marcus Denker-4

On Sep 24, 2013, at 4:30 PM, Alejandro Pulver <[hidden email]> wrote:

> Thanks for the idea, I'll try emulating shotgun sequencing by increasing the sampling rate for short periods of time.
>
> But a better alternative would be to enable and disable the simulator (context's runSimulated) for exact sequences (even if shorter, to avoid slowdowns). Is it possible with the current implementation?
>
The problem is that runSimulated is *very* slow as it interpretes bytecode using an interpreter written in Smalltalk.

Maybe an idea would be to use some sampling that uses VM Support… I don't know much about it, though. See MessageTally and AndreasSystemProfiler.

> It's not that I don't appreciate the ideas for the new reflection framework. but I'm looking for something to use for this project. And we have only a few weeks. Actually, the most important part is not the system integration but analyzing the structure of those message sequences (like word analysis in natural language processing).
>
Yes, I understand. We are working hard to improve the system… but it's a lot of work.

        Marcus

signature.asc (210 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Capturing messages

Goubier Thierry
In reply to this post by Alejandro Pulver
There is also a code tracer working in 2.0 that I maintain and use from
time to time, called Jejak.

It would require a small extension and recompiling a lot of methods, but
it could be made to trace and record all method entries (taking care of
not tracing its own code :)).

It will slow down the VM by a significant amount, of course.

Thierry

Le 24/09/2013 16:30, Alejandro Pulver a écrit :

> Thanks for the idea, I'll try emulating shotgun sequencing by increasing
> the sampling rate for short periods of time.
>
> But a better alternative would be to enable and disable the simulator
> (context's runSimulated) for exact sequences (even if shorter, to avoid
> slowdowns). Is it possible with the current implementation?
>
> It's not that I don't appreciate the ideas for the new reflection
> framework. but I'm looking for something to use for this project. And we
> have only a few weeks. Actually, the most important part is not the
> system integration but analyzing the structure of those message
> sequences (like word analysis in natural language processing).
>
>
> On Tue, Sep 24, 2013 at 4:59 AM, Marcus Denker <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>     On Sep 24, 2013, at 9:41 AM, Alejandro Pulver <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>      > I see. But what about sampling? It shouldn't slow down the
>     program nearly as much as these "instrumentation" like mechanisms.
>      >
>      > And as I'm going to count them anyways, missing the most unused
>     ones won't be such a problem.
>      >
>      > That combined with complete capture for a small run (like a test
>     or block) would be enough for our needs.
>      >
>      >
>
>     Yes, one could just use some random selection (I really like the
>     idea of shotgun sequencing…
>     http://en.wikipedia.org/wiki/Shotgun_sequencing)
>     The resulting information should be quite good.
>
>     But the first step is that we need a really good, very dynamic, very
>     general and powerful basis… this is what we want to build
>     with Reflectivity2 (and Reflectivity3 will have to hook deeper into
>     the JIT of the VM, but that is another story).
>
>     Related to step 3), another source for interesting data is the VM
>     itself (e.g. caches).
>
>     But step by step…
>
>     The idea is to take the new possibilities we have in Pharo3 and make
>     a well engineered realization of partial behavioral reflection,
>     with two extensions:
>              -> meta links on instance  variables. Very cool
>     possibilities now that instance variables are reified objects ("Slot")
>              -> object specific meta links by integrating with a good
>     proxy implementation
>
>     This will already bring amazing capabilities to the system (and
>     subsume MethodWrappers, simple CLOS style MOPs, perfect for
>     breakpoints, watchpoints and debugging in general,  and it can be
>     the basis for AOP like things).
>
>              Marcus
>
>

--
Thierry Goubier
CEA list
Laboratoire des Fondations des Systèmes Temps Réel Embarqués
91191 Gif sur Yvette Cedex
France
Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95