Does PipeableOSProcess require some finalization/clean up?

> In a fully updated trunk 4.4 alpha image (#12041), load
> OSProcess-dtl.70 and CommandShell-dtl.61.  And with the latest Cog VM
> ('Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.157] Croquet
> Cog 4.0.2550') on MacOS 10.7.4.  If I evaluate:
>
> -------------------------------------
> v := [1 to: 256 do: [:i |
> (PipeableOSProcess command: 'echo hello') output.
> Transcript print: i; cr.
> (Delay forMilliseconds: 50) wait.]].
> v fork
> -------------------------------------
>
> and when i reaches around 242, I get an error that says "cannot create
> OS pipe" (you actually don't get the reporting to Transcript but...).
> Garbage collection does not clear the issue (as the comment in
> #register: of AttachableFileStream suggests).  With lsof, I can see
> that it keeps all pipes.
>
> Do I need to make some more calls to have them clean up?  (Or, is this
> another symptom of the file open/close issue with CogVM on Mac VM?)
>
> -- Yoshiki

>
> Hi Yoshiki,
>
> Yes, a PipeableOSProcess does require cleanup to close open pipe
> handles. I am away and cannot look at the code right now, but check
> #closePipes, #close, etc. This is not a CogVM or Mac VM issue.
>
> When running in a CommandShell, the cleanup happens automatically,
> but not when you are interacting directly with an individual
> PipeableOSProcess.
>
> The resources in question are the unix file handles associated with
> OS pipe endpoints. For example, the standard output from your
> external command is connected to your image via an OS pipe, and
> the reader end of that pipe is represented in your image as a
> file stream. Just as with a file, you need to keep that file stream
> open until you are done using it, and the underlying OS file
> handle does not go away until the file stream is closed.
>
> The reason that garbage collection does not free up the handles
> is that "OSProcess thisOSProcess allMyChildren" has a dictionary
> of all child processes that have been run, including the PipeableOSProcess
> that you ran. I think that if you empty that dictionary, the
> garbage collection will probably finalize the pipes for you.
> But the better thing to do is close the pipes explicitly.
>
> Dave
>
>
> On Wed, May 16, 2012 at 09:45:55AM -0700, Yoshiki Ohshima wrote:
> > In a fully updated trunk 4.4 alpha image (#12041), load
> > OSProcess-dtl.70 and CommandShell-dtl.61.  And with the latest Cog VM
> > ('Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.157] Croquet
> > Cog 4.0.2550') on MacOS 10.7.4.  If I evaluate:
> >
> > -------------------------------------
> > v := [1 to: 256 do: [:i |
> > (PipeableOSProcess command: 'echo hello') output.
> > Transcript print: i; cr.
> > (Delay forMilliseconds: 50) wait.]].
> > v fork
> > -------------------------------------
> >
> > and when i reaches around 242, I get an error that says "cannot create
> > OS pipe" (you actually don't get the reporting to Transcript but...).
> > Garbage collection does not clear the issue (as the comment in
> > #register: of AttachableFileStream suggests).  With lsof, I can see
> > that it keeps all pipes.
> >
> > Do I need to make some more calls to have them clean up?  (Or, is this
> > another symptom of the file open/close issue with CogVM on Mac VM?)
> >
> > -- Yoshiki
>

>   Thank you, Dave!
>
> I now understand that going through CommandShell was the sanctioned
> way of using PipeableOSProcess.  I'll just modify our code to close it
> explicitly.  (And it seems to work beter.)
>
> I tried clearing the allMyChildren directory but did not seem to free
> these fds.  I tried:
>
> PipeableOSProcess allSubInstancesDo: [:e | e closePipes]
>
> but this do it seems to kill the VM.

>
> On Wed, May 16, 2012 at 12:53:26PM -0700, Yoshiki Ohshima wrote:
> >   Thank you, Dave!
> >
> > I now understand that going through CommandShell was the sanctioned
> > way of using PipeableOSProcess.  I'll just modify our code to close it
> > explicitly.  (And it seems to work beter.)
> >
> > I tried clearing the allMyChildren directory but did not seem to free
> > these fds.  I tried:
> >
> > PipeableOSProcess allSubInstancesDo: [:e | e closePipes]
> >
> > but this do it seems to kill the VM.
>
> Eeek! This should not be happening. PipeableOSProcess>>closePipes uses
> StandardFileStream>>primCloseNoError: which does an fclose() to close
> the pipe descriptor. This should be failure-proof, but I might be missing
> something.
>
> Is this reproduceable? It does not seem to happen on my Linux PC, but
> maybe it is an intermittent failure of some kind.

> At Wed, 16 May 2012 19:47:15 -0400,
> David T. Lewis wrote:
> >
> > On Wed, May 16, 2012 at 12:53:26PM -0700, Yoshiki Ohshima wrote:
> > >   Thank you, Dave!
> > >
> > > I now understand that going through CommandShell was the sanctioned
> > > way of using PipeableOSProcess.  I'll just modify our code to close it
> > > explicitly.  (And it seems to work beter.)
> > >
> > > I tried clearing the allMyChildren directory but did not seem to free
> > > these fds.  I tried:
> > >
> > > PipeableOSProcess allSubInstancesDo: [:e | e closePipes]
> > >
> > > but this do it seems to kill the VM.
> >
> > Eeek! This should not be happening. PipeableOSProcess>>closePipes uses
> > StandardFileStream>>primCloseNoError: which does an fclose() to close
> > the pipe descriptor. This should be failure-proof, but I might be missing
> > something.
> >
> > Is this reproduceable? It does not seem to happen on my Linux PC, but
> > maybe it is an intermittent failure of some kind.
>
> This is reproduceable in my combination of things.  It does not have
> to be allSubInstancesDo:, but allInstancesDo: does it (of course).
>
> Oh, the trick seems that I shouldn't close the notifier.  Do the
> first part:
> -------------------------
> v := [1 to: 256 do: [:i |
> (PipeableOSProcess command: 'echo hello') output.
> (Delay forMilliseconds: 50) wait.]].
> v fork.
> -------------------------
>
> and while the notifier is shown on the screen, evaluate:
>
> -------------------------
> PipeableOSProcess allInstancesDo: [:e | e closePipes]
> -------------------------
>

>
> On Wed, May 16, 2012 at 07:38:36PM -0700, Yoshiki Ohshima wrote:
> > At Wed, 16 May 2012 19:47:15 -0400,
> > David T. Lewis wrote:
> > >
> > > On Wed, May 16, 2012 at 12:53:26PM -0700, Yoshiki Ohshima wrote:
> > > >   Thank you, Dave!
> > > >
> > > > I now understand that going through CommandShell was the sanctioned
> > > > way of using PipeableOSProcess.  I'll just modify our code to close it
> > > > explicitly.  (And it seems to work beter.)
> > > >
> > > > I tried clearing the allMyChildren directory but did not seem to free
> > > > these fds.  I tried:
> > > >
> > > > PipeableOSProcess allSubInstancesDo: [:e | e closePipes]
> > > >
> > > > but this do it seems to kill the VM.
> > >
> > > Eeek! This should not be happening. PipeableOSProcess>>closePipes uses
> > > StandardFileStream>>primCloseNoError: which does an fclose() to close
> > > the pipe descriptor. This should be failure-proof, but I might be missing
> > > something.
> > >
> > > Is this reproduceable? It does not seem to happen on my Linux PC, but
> > > maybe it is an intermittent failure of some kind.
> >
> > This is reproduceable in my combination of things.  It does not have
> > to be allSubInstancesDo:, but allInstancesDo: does it (of course).
> >
> > Oh, the trick seems that I shouldn't close the notifier.  Do the
> > first part:
> > -------------------------
> > v := [1 to: 256 do: [:i |
> > (PipeableOSProcess command: 'echo hello') output.
> > (Delay forMilliseconds: 50) wait.]].
> > v fork.
> > -------------------------
> >
> > and while the notifier is shown on the screen, evaluate:
> >
> > -------------------------
> > PipeableOSProcess allInstancesDo: [:e | e closePipes]
> > -------------------------
> >
>
> Thanks Yoshiki,
>
> I see it now. I cannot explain what is happening. The system is
> behaving as if the VM was blocked on a read, but I don't think
> that this is the actual problem. Interesting ...

> At Thu, 17 May 2012 10:16:53 -0400,
> David T. Lewis wrote:
> >
> > On Wed, May 16, 2012 at 07:38:36PM -0700, Yoshiki Ohshima wrote:
> > > At Wed, 16 May 2012 19:47:15 -0400,
> > > David T. Lewis wrote:
> > > >
> > > > On Wed, May 16, 2012 at 12:53:26PM -0700, Yoshiki Ohshima wrote:
> > > > >   Thank you, Dave!
> > > > >
> > > > > I now understand that going through CommandShell was the sanctioned
> > > > > way of using PipeableOSProcess.  I'll just modify our code to close it
> > > > > explicitly.  (And it seems to work beter.)
> > > > >
> > > > > I tried clearing the allMyChildren directory but did not seem to free
> > > > > these fds.  I tried:
> > > > >
> > > > > PipeableOSProcess allSubInstancesDo: [:e | e closePipes]
> > > > >
> > > > > but this do it seems to kill the VM.
> > > >
> > > > Eeek! This should not be happening. PipeableOSProcess>>closePipes uses
> > > > StandardFileStream>>primCloseNoError: which does an fclose() to close
> > > > the pipe descriptor. This should be failure-proof, but I might be missing
> > > > something.
> > > >
> > > > Is this reproduceable? It does not seem to happen on my Linux PC, but
> > > > maybe it is an intermittent failure of some kind.
> > >
> > > This is reproduceable in my combination of things.  It does not have
> > > to be allSubInstancesDo:, but allInstancesDo: does it (of course).
> > >
> > > Oh, the trick seems that I shouldn't close the notifier.  Do the
> > > first part:
> > > -------------------------
> > > v := [1 to: 256 do: [:i |
> > > (PipeableOSProcess command: 'echo hello') output.
> > > (Delay forMilliseconds: 50) wait.]].
> > > v fork.
> > > -------------------------
> > >
> > > and while the notifier is shown on the screen, evaluate:
> > >
> > > -------------------------
> > > PipeableOSProcess allInstancesDo: [:e | e closePipes]
> > > -------------------------
> > >
> >
> > Thanks Yoshiki,
> >
> > I see it now. I cannot explain what is happening. The system is
> > behaving as if the VM was blocked on a read, but I don't think
> > that this is the actual problem. Interesting ...
>
> Just another data point: the crash report shows that VM is blocked on
> a read on Mac, too.
>
> -- Yoshiki

>
> David T. Lewis wrote
> >
> > the VM is blocking on a read from an OS pipe...
> >
> > 0xff7d49a0 M ExternalPipe>upToEnd -1053201688: a(n) ExternalPipe
> > 0xff7d49c0 M [] in PipeableOSProcess>closePipes -1053202016: a(n)
> > PipeableOSProcess
> >
>
> #closePipes is calling #upToEnd. Could this be another case of failing to
> call #setNonBlockingOutput?