Proper way to use many PipeableOSProcesses ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Proper way to use many PipeableOSProcesses ?

Damien Pollet
Hi,

I need to pass a bunch of strings through a unix command. I'm using
this repeatedly:

cat := (PipeableOSProcess command: '/bin/cat')
        nextPutAll: aString;
        close.
cat output "do something with that"

After roughly 120 iterations, I get an error "cannot create OS pipe"
from ExternalPipe>>makePipe, and all subsequent PipeableOSProcesses do
that until I relaunch the image. Did I miss something? should I
flush/wait/finalize something?

--
Damien Pollet
type less, do more [ | ] http://typo.cdlm.fasmz.org

Reply | Threaded
Open this post in threaded view
|

Re: Proper way to use many PipeableOSProcesses ?

David T. Lewis
Hi Damien,

The #close on PipeableOSProcess will only close the input pipe (the
OS pipe that connects Squeak to the stdin of the external process).
It leaves the output and error pipes open (I guess that #close was a
poor selector name, since it implies "closing" the whole thing). As
a result, you will eventually run out of file handles for the Squeak
VM process, and that is why you get the error.

You can just add a #closePipes when you have finished reading the
process output, and this should clean things up properly:

  outputs := OrderedCollection new.
  1000 timesRepeat:
  [cat := PipeableOSProcess command: '/bin/cat'.
  cat nextPutAll: 'fooBarBaz'; close.
  outputs add: cat output.
  cat closePipes].
  outputs inspect

If you are using an operating system with a /proc file system (ie Linux),
it is helpful to look at /proc/<squeakpid>/fd/* to see the open file
descriptors. If you are accumulating file descriptors in that directory,
then you have a leak such at this one.

Dave

On Sun, Dec 09, 2007 at 07:22:31PM +0100, Damien Pollet wrote:

> Hi,
>
> I need to pass a bunch of strings through a unix command. I'm using
> this repeatedly:
>
> cat := (PipeableOSProcess command: '/bin/cat')
> nextPutAll: aString;
> close.
> cat output "do something with that"
>
> After roughly 120 iterations, I get an error "cannot create OS pipe"
> from ExternalPipe>>makePipe, and all subsequent PipeableOSProcesses do
> that until I relaunch the image. Did I miss something? should I
> flush/wait/finalize something?
>

Reply | Threaded
Open this post in threaded view
|

Re: Proper way to use many PipeableOSProcesses ?

Damien Pollet
Indeed it worked like a breeze this way. I thought the output and
error pipes were closed automatically by the unix process when it
exits.

thanks!

On 10/12/2007, David T. Lewis <[hidden email]> wrote:

> Hi Damien,
>
> The #close on PipeableOSProcess will only close the input pipe (the
> OS pipe that connects Squeak to the stdin of the external process).
> It leaves the output and error pipes open (I guess that #close was a
> poor selector name, since it implies "closing" the whole thing). As
> a result, you will eventually run out of file handles for the Squeak
> VM process, and that is why you get the error.
>
> You can just add a #closePipes when you have finished reading the
> process output, and this should clean things up properly:
>
>   outputs := OrderedCollection new.
>   1000 timesRepeat:
>   [cat := PipeableOSProcess command: '/bin/cat'.
>   cat nextPutAll: 'fooBarBaz'; close.
>   outputs add: cat output.
>   cat closePipes].
>   outputs inspect
>
> If you are using an operating system with a /proc file system (ie Linux),
> it is helpful to look at /proc/<squeakpid>/fd/* to see the open file
> descriptors. If you are accumulating file descriptors in that directory,
> then you have a leak such at this one.
>
> Dave
>
> On Sun, Dec 09, 2007 at 07:22:31PM +0100, Damien Pollet wrote:
> > Hi,
> >
> > I need to pass a bunch of strings through a unix command. I'm using
> > this repeatedly:
> >
> > cat := (PipeableOSProcess command: '/bin/cat')
> >       nextPutAll: aString;
> >       close.
> > cat output "do something with that"
> >
> > After roughly 120 iterations, I get an error "cannot create OS pipe"
> > from ExternalPipe>>makePipe, and all subsequent PipeableOSProcesses do
> > that until I relaunch the image. Did I miss something? should I
> > flush/wait/finalize something?
> >
>
>


--
Damien Pollet
type less, do more [ | ] http://typo.cdlm.fasmz.org