OSProcess left-over pipe handles?

Hi -

Today I noticed that we're starting to accumulate lots of pipe handles
on our servers when running for a while. I believe these are the results
of OSProcess usage and I was wondering what the correct way to clean
these up would be. lsof lists all of them just like here:

squeak    16372 Teleplace  824r     FIFO                0,6            
2214233 pipe
squeak    16372 Teleplace  825r     FIFO                0,6            
2217936 pipe
squeak    16372 Teleplace  826r     FIFO                0,6            
2215398 pipe
squeak    16372 Teleplace  827r     FIFO                0,6            
2214561 pipe
squeak    16372 Teleplace  828r     FIFO                0,6            
2214590 pipe
squeak    16372 Teleplace  829r     FIFO                0,6            
2215321 pipe
squeak    16372 Teleplace  830r     FIFO                0,6            
2215385 pipe


so it's a bit difficult to find out what's going on. Any ideas for how
to track this down and a way to clean them up?

Cheers,
  - Andreas

Andreas Raab wrote:
> Hi -
>
> Today I noticed that we're starting to accumulate lots of pipe handles
> on our servers when running for a while. I believe these are the
> results of OSProcess usage and I was wondering what the correct way to
> clean these up would be. lsof lists all of them just like here:

without going to see anything in squeak (so, maybe totally wrong),
usually when you call pipe() in unix, you then fork() and pass one of
the two descriptors to the child process. The common practice is to
close() the file descriptors passed to the child process. This has to be
done in the parent process after forking. Otherwise the parent keeps
open the end of the pipe() corresponding to the child.

Of course, after finishing you also hahve to close() the other end of
the pipe() in the parent. The child process usually dies, hence closing
all file descriptors. But, if the child doesn't die, then you have to be
careful to also close the file descriptors belonging to the parent, and
the proper way to do it is to close them just after forking, in the
child process... something like:

        pipe(&parentFd, &childFd);
        fork()

          parent: close(childFd)            child: close(parentFd)

          ...                                             ...

          close(parentFd)                      close(childFd)


    gera

Gerardo Richarte wrote:
> without going to see anything in squeak (so, maybe totally wrong),
> usually when you call pipe() in unix, you then fork() and pass one of
> the two descriptors to the child process. The common practice is to
> close() the file descriptors passed to the child process. This has to be
> done in the parent process after forking. Otherwise the parent keeps
> open the end of the pipe() corresponding to the child.

Thanks. We're using PipeableOSProcess to be able to get to the output of
various commands.

> Of course, after finishing you also hahve to close() the other end of
> the pipe() in the parent. The child process usually dies, hence closing
> all file descriptors. But, if the child doesn't die, then you have to be
> careful to also close the file descriptors belonging to the parent, and
> the proper way to do it is to close them just after forking, in the
> child process... something like:

Hm ... not sure what OSProcess does in that regard. Anyone know?

Thanks for the info!

Cheers,
   - Andreas

On Fri, Dec 11, 2009 at 07:17:50PM -0800, Andreas Raab wrote:
> Gerardo Richarte wrote:
> >without going to see anything in squeak (so, maybe totally wrong),
> >usually when you call pipe() in unix, you then fork() and pass one of
> >the two descriptors to the child process. The common practice is to
> >close() the file descriptors passed to the child process. This has to be
> >done in the parent process after forking. Otherwise the parent keeps
> >open the end of the pipe() corresponding to the child.

Yes, that's right. OSProcess does exactly this, it closes the descriptors
that will be used by the child process and keeps open the descriptors
that will be used for writing to the stdin of the child process, and
reading from the stdout and stderr of the child process.

Hi Andreas,

If you are using PipeableOSProcess class>>command:, then the most
likely source of the problem is that you need to explicitly close the
pipeFromOutput after reading the output data. PipeableOSProcess closes
as many of the pipe handles as possible, but the handle for the
pipe output (your reader end for the stdout from the external process)
is left open to permit you to read the output data after the external
process has exited.

To confirm this, use and object explorer on the PipeableOSProcess and
drill down to the pipe endpoints and check to make sure they are all
#closed.

You might consider using ProxyPipeline class>>command: rather than
PipeableOSProcess class>>command. In that case you can use #upToEndOfFile
rather than #upToEnd to obtain the output. This will read all available
output, waiting as needed to allow the external process to exit. A side
effect of the end of file detection is that the pipe reader will be
closed, hence no left over file handles.

ProxyPipeline is more complex than PipeableOSProcess, but the automatic
cleanup may make it more reliable for you overall.

A couple other notes that might be relevant:

- Pipe handles are not cleaned up by finalization. The reason for
this is that I explicitly disabled the handle registration in
AttachableFileStream class>>register: in order to prevent performance
issues associated with the weak registry.  Basically I found that running
the CommandShell unit tests was enough to cause performance problems
for the entire image, so I turned off the registration for file streams
associated with OS pipes. If you are not using OSProcess too heavily,
you might consider turning the registration back on as an extra measure
of safety. But do keep a close eye on the size of the weak registry,
because PipeableOSProcess uses a lot of file streams to connect to
all those pipes.

- Aside from failing to explicitly close pipes, the other possible (but
very unlikely) source of handle leaks would be a failure in the child
process cleanup mechanism, which results in zombie child processes and
lots of open handles. Fortunately, as long as you do not use a Pharo
image you are not likely to encounter this problem ;-)

HTH,
Dave

Hi David

> If you are using PipeableOSProcess class>>command:, then the most
[..]
> You might consider using ProxyPipeline class>>command: rather than
> PipeableOSProcess class>>command. In that case you can use #upToEndOfFile
In which package are PipeableOSProcess and ProxyPipeline define? I can't
find them in my image with OSProcess-dtl.53 loaded.

Thanks,

Udo

On Sun, Dec 13, 2009 at 12:29:13AM +0100, Udo Schneider wrote:
> Hi David
>
> >If you are using PipeableOSProcess class>>command:, then the most
> [..]
> >You might consider using ProxyPipeline class>>command: rather than
> >PipeableOSProcess class>>command. In that case you can use #upToEndOfFile
> In which package are PipeableOSProcess and ProxyPipeline define? I can't
> find them in my image with OSProcess-dtl.53 loaded.

Hi Udo,

These are in CommandShell at http://squeaksource.com/CommandShell.
A description is at http://wiki.squeak.org/squeak/1914. Originally
this was part of the OSProcess package, but I split CommandShell
into a separate package when OSProcess got too large.

Dave

Hi David,

> These are in CommandShell at http://squeaksource.com/CommandShell.
> A description is at http://wiki.squeak.org/squeak/1914. Originally
> this was part of the OSProcess package, but I split CommandShell
> into a separate package when OSProcess got too large.
Great! Thanks for the info - I'm currently doing the whole pipes stuff
manually. Again a chance to through away some code I wrote.
The best code is the one you don't have to write/maintain :-)

CU,

Udo