Fwd: Re: [Pharo-dev] OSProcess in 3.0

 

How to include aioplugin in pharovm?

See messages below for the reason why.

Phil

---------- Message transféré ----------
De : "David T. Lewis" <[hidden email]>
Date : 16 oct. 2014 01:30
Objet : Re: [Pharo-dev] OSProcess in 3.0
À : "Pharo Development List" <[hidden email]>
Cc :

On Tue, Oct 14, 2014 at 09:35:44PM +0200, Thierry Goubier wrote:
>
>
> Le 14/10/2014 14:15, David T. Lewis a ?crit :
> >On Tue, Oct 14, 2014 at 07:19:10AM +0200, Thierry Goubier wrote:
> >>Hi Dave,
> >>
> >>took me a while to do some testing. It seems there is an issue, but I
> >>couldn't find out where. First, the data:
> >>
> >>Loading a package from a local gitfiletree repository (get from the
> >>command a zipped git archive containing the package) :
> >>
> >>4.5.11, 4.6.8: 2132 msec.
> >>4.5.12, 4.6.9: 50847 msec.
> >>
> >>I tried to rewrite the main command, remove the closePipes, etc... No
> >>change in the runtime, so your right.
> >>
> >>Profiling doesn't help much: execution time is mostly idle time (but
> >>with a significant increase on PipeableOSProcess commands: goes from
> >>invisible to 2700 ms. A user break in the middle shows it delay waiting
> >>on output of the external process).
> >>
> >>Only hint I could see is that it seems to be linked with the size of the
> >>output of the command.
> >
> >Thierry,
> >
> >Aha! I am quite sure that you have located the problem. There must be
> >something preventing the OS process from running to completion with
> >a larger amount of output data. It definitely sounds like a bug in my
> >latest update to Pipeable OSProcess.
>
> The strange thing is that it runs to completion (i.e. I got no errors;
> if any data was missing or corrupted on output, my load would fail); but
> a lot slower than it should be.
>
> >Thank you very much for finding this. I will get it fixed as quickly
> >as I can.
>
> Thanks. I'm unable to help you much, I'm afraid :(

Thierry,

I do not yet have a solution to this, but I have been able to confirm
that the performance difference that you see is primarily associated with
the AioPlugin not being present in your VM.

When I changed PipeableOSProcess to handle pipe handle closing properly,
I used a buffered output stream that is driven by aio events. When data
is available to be read from the external OS process, an aio notification
pulls the available data into the buffered stream, and when no more data
is available, the pipe (file) handles are closed.

If the plugin is not present in the VM, I provided a polling mechanism
to read data from the external OS process. Apparently this is very
inefficient, and I am sure it could be greatly improved. That is the
cause of the extremely slow performance that you are seeing.

So in summary: The enhancement to PipeableOSProcess relies on an aio
event driven reader, and if the AIO plugin is missing, the fallback code
relies on polling, and is horribly inefficient.

More to follow, and thank you again for identifying the problem.

If you can convice your VM vendor to include AioPlugin in your VM, that
would be helpful also ;-)

Dave

 

On 16 Oct 2014, at 21:01, Eliot Miranda <[hidden email]> wrote:

Hi Phil,

On Wed, Oct 15, 2014 at 10:11 PM, [hidden email] <[hidden email]> wrote:

 

How to include aioplugin in pharovm?

Ask Esteban and other members of the Pharo team.  The AioPlugin is already included in the Squeak Cog and Interpreter VMs.  Since the build process is different for Pharo (CMake and VMMakerConfigurqation (sp?) based) telling you how my builds work won't help.  But if you want you could try and use one of the latest VMs from my website (see the Downloads link) and you should find that'll work as a Pharo VM.

AioPlugin is since always in the mac version of the Pharo VM. 

I don’t know why is not in the others, but my best guess is because I was not maintaining them at the time and when I took over I didn’t revised the process :P

I will see what I can do :)

Esteban

HTH

Eliot

See messages below for the reason why.

Phil

---------- Message transféré ----------
De : "David T. Lewis" <[hidden email]>
Date : 16 oct. 2014 01:30
Objet : Re: [Pharo-dev] OSProcess in 3.0
À : "Pharo Development List" <[hidden email]>
Cc : 

On Tue, Oct 14, 2014 at 09:35:44PM +0200, Thierry Goubier wrote:

>
>
> Le 14/10/2014 14:15, David T. Lewis a ?crit :
> >On Tue, Oct 14, 2014 at 07:19:10AM +0200, Thierry Goubier wrote:
> >>Hi Dave,
> >>
> >>took me a while to do some testing. It seems there is an issue, but I
> >>couldn't find out where. First, the data:
> >>
> >>Loading a package from a local gitfiletree repository (get from the
> >>command a zipped git archive containing the package) :
> >>
> >>4.5.11, 4.6.8: 2132 msec.
> >>4.5.12, 4.6.9: 50847 msec.
> >>
> >>I tried to rewrite the main command, remove the closePipes, etc... No
> >>change in the runtime, so your right.
> >>
> >>Profiling doesn't help much: execution time is mostly idle time (but
> >>with a significant increase on PipeableOSProcess commands: goes from
> >>invisible to 2700 ms. A user break in the middle shows it delay waiting
> >>on output of the external process).
> >>
> >>Only hint I could see is that it seems to be linked with the size of the
> >>output of the command.
> >
> >Thierry,
> >
> >Aha! I am quite sure that you have located the problem. There must be
> >something preventing the OS process from running to completion with
> >a larger amount of output data. It definitely sounds like a bug in my
> >latest update to Pipeable OSProcess.
>
> The strange thing is that it runs to completion (i.e. I got no errors;
> if any data was missing or corrupted on output, my load would fail); but
> a lot slower than it should be.
>
> >Thank you very much for finding this. I will get it fixed as quickly
> >as I can.
>
> Thanks. I'm unable to help you much, I'm afraid :(

Thierry,

I do not yet have a solution to this, but I have been able to confirm
that the performance difference that you see is primarily associated with
the AioPlugin not being present in your VM.

When I changed PipeableOSProcess to handle pipe handle closing properly,
I used a buffered output stream that is driven by aio events. When data
is available to be read from the external OS process, an aio notification
pulls the available data into the buffered stream, and when no more data
is available, the pipe (file) handles are closed.

If the plugin is not present in the VM, I provided a polling mechanism
to read data from the external OS process. Apparently this is very
inefficient, and I am sure it could be greatly improved. That is the
cause of the extremely slow performance that you are seeing.

So in summary: The enhancement to PipeableOSProcess relies on an aio
event driven reader, and if the AIO plugin is missing, the fallback code
relies on polling, and is horribly inefficient.

More to follow, and thank you again for identifying the problem.

If you can convice your VM vendor to include AioPlugin in your VM, that
would be helpful also ;-)

Dave

-- 
best,

Eliot