In a fully updated trunk 4.4 alpha image (#12041), load
OSProcess-dtl.70 and CommandShell-dtl.61. And with the latest Cog VM ('Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.157] Croquet Cog 4.0.2550') on MacOS 10.7.4. If I evaluate: ------------------------------------- v := [1 to: 256 do: [:i | (PipeableOSProcess command: 'echo hello') output. Transcript print: i; cr. (Delay forMilliseconds: 50) wait.]]. v fork ------------------------------------- and when i reaches around 242, I get an error that says "cannot create OS pipe" (you actually don't get the reporting to Transcript but...). Garbage collection does not clear the issue (as the comment in #register: of AttachableFileStream suggests). With lsof, I can see that it keeps all pipes. Do I need to make some more calls to have them clean up? (Or, is this another symptom of the file open/close issue with CogVM on Mac VM?) -- Yoshiki |
Hi Yoshiki,
Yes, a PipeableOSProcess does require cleanup to close open pipe handles. I am away and cannot look at the code right now, but check #closePipes, #close, etc. This is not a CogVM or Mac VM issue. When running in a CommandShell, the cleanup happens automatically, but not when you are interacting directly with an individual PipeableOSProcess. The resources in question are the unix file handles associated with OS pipe endpoints. For example, the standard output from your external command is connected to your image via an OS pipe, and the reader end of that pipe is represented in your image as a file stream. Just as with a file, you need to keep that file stream open until you are done using it, and the underlying OS file handle does not go away until the file stream is closed. The reason that garbage collection does not free up the handles is that "OSProcess thisOSProcess allMyChildren" has a dictionary of all child processes that have been run, including the PipeableOSProcess that you ran. I think that if you empty that dictionary, the garbage collection will probably finalize the pipes for you. But the better thing to do is close the pipes explicitly. Dave On Wed, May 16, 2012 at 09:45:55AM -0700, Yoshiki Ohshima wrote: > In a fully updated trunk 4.4 alpha image (#12041), load > OSProcess-dtl.70 and CommandShell-dtl.61. And with the latest Cog VM > ('Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.157] Croquet > Cog 4.0.2550') on MacOS 10.7.4. If I evaluate: > > ------------------------------------- > v := [1 to: 256 do: [:i | > (PipeableOSProcess command: 'echo hello') output. > Transcript print: i; cr. > (Delay forMilliseconds: 50) wait.]]. > v fork > ------------------------------------- > > and when i reaches around 242, I get an error that says "cannot create > OS pipe" (you actually don't get the reporting to Transcript but...). > Garbage collection does not clear the issue (as the comment in > #register: of AttachableFileStream suggests). With lsof, I can see > that it keeps all pipes. > > Do I need to make some more calls to have them clean up? (Or, is this > another symptom of the file open/close issue with CogVM on Mac VM?) > > -- Yoshiki |
Thank you, Dave!
I now understand that going through CommandShell was the sanctioned way of using PipeableOSProcess. I'll just modify our code to close it explicitly. (And it seems to work beter.) I tried clearing the allMyChildren directory but did not seem to free these fds. I tried: PipeableOSProcess allSubInstancesDo: [:e | e closePipes] but this do it seems to kill the VM. -- Yoshiki At Wed, 16 May 2012 15:33:39 -0400, David T. Lewis wrote: > > Hi Yoshiki, > > Yes, a PipeableOSProcess does require cleanup to close open pipe > handles. I am away and cannot look at the code right now, but check > #closePipes, #close, etc. This is not a CogVM or Mac VM issue. > > When running in a CommandShell, the cleanup happens automatically, > but not when you are interacting directly with an individual > PipeableOSProcess. > > The resources in question are the unix file handles associated with > OS pipe endpoints. For example, the standard output from your > external command is connected to your image via an OS pipe, and > the reader end of that pipe is represented in your image as a > file stream. Just as with a file, you need to keep that file stream > open until you are done using it, and the underlying OS file > handle does not go away until the file stream is closed. > > The reason that garbage collection does not free up the handles > is that "OSProcess thisOSProcess allMyChildren" has a dictionary > of all child processes that have been run, including the PipeableOSProcess > that you ran. I think that if you empty that dictionary, the > garbage collection will probably finalize the pipes for you. > But the better thing to do is close the pipes explicitly. > > Dave > > > On Wed, May 16, 2012 at 09:45:55AM -0700, Yoshiki Ohshima wrote: > > In a fully updated trunk 4.4 alpha image (#12041), load > > OSProcess-dtl.70 and CommandShell-dtl.61. And with the latest Cog VM > > ('Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.157] Croquet > > Cog 4.0.2550') on MacOS 10.7.4. If I evaluate: > > > > ------------------------------------- > > v := [1 to: 256 do: [:i | > > (PipeableOSProcess command: 'echo hello') output. > > Transcript print: i; cr. > > (Delay forMilliseconds: 50) wait.]]. > > v fork > > ------------------------------------- > > > > and when i reaches around 242, I get an error that says "cannot create > > OS pipe" (you actually don't get the reporting to Transcript but...). > > Garbage collection does not clear the issue (as the comment in > > #register: of AttachableFileStream suggests). With lsof, I can see > > that it keeps all pipes. > > > > Do I need to make some more calls to have them clean up? (Or, is this > > another symptom of the file open/close issue with CogVM on Mac VM?) > > > > -- Yoshiki > |
On Wed, May 16, 2012 at 12:53:26PM -0700, Yoshiki Ohshima wrote:
> Thank you, Dave! > > I now understand that going through CommandShell was the sanctioned > way of using PipeableOSProcess. I'll just modify our code to close it > explicitly. (And it seems to work beter.) > > I tried clearing the allMyChildren directory but did not seem to free > these fds. I tried: > > PipeableOSProcess allSubInstancesDo: [:e | e closePipes] > > but this do it seems to kill the VM. Eeek! This should not be happening. PipeableOSProcess>>closePipes uses StandardFileStream>>primCloseNoError: which does an fclose() to close the pipe descriptor. This should be failure-proof, but I might be missing something. Is this reproduceable? It does not seem to happen on my Linux PC, but maybe it is an intermittent failure of some kind. Thanks, Dave |
At Wed, 16 May 2012 19:47:15 -0400,
David T. Lewis wrote: > > On Wed, May 16, 2012 at 12:53:26PM -0700, Yoshiki Ohshima wrote: > > Thank you, Dave! > > > > I now understand that going through CommandShell was the sanctioned > > way of using PipeableOSProcess. I'll just modify our code to close it > > explicitly. (And it seems to work beter.) > > > > I tried clearing the allMyChildren directory but did not seem to free > > these fds. I tried: > > > > PipeableOSProcess allSubInstancesDo: [:e | e closePipes] > > > > but this do it seems to kill the VM. > > Eeek! This should not be happening. PipeableOSProcess>>closePipes uses > StandardFileStream>>primCloseNoError: which does an fclose() to close > the pipe descriptor. This should be failure-proof, but I might be missing > something. > > Is this reproduceable? It does not seem to happen on my Linux PC, but > maybe it is an intermittent failure of some kind. This is reproduceable in my combination of things. It does not have to be allSubInstancesDo:, but allInstancesDo: does it (of course). Oh, the trick seems that I shouldn't close the notifier. Do the first part: ------------------------- v := [1 to: 256 do: [:i | (PipeableOSProcess command: 'echo hello') output. (Delay forMilliseconds: 50) wait.]]. v fork. ------------------------- and while the notifier is shown on the screen, evaluate: ------------------------- PipeableOSProcess allInstancesDo: [:e | e closePipes] ------------------------- -- Yoshiki |
On Wed, May 16, 2012 at 07:38:36PM -0700, Yoshiki Ohshima wrote:
> At Wed, 16 May 2012 19:47:15 -0400, > David T. Lewis wrote: > > > > On Wed, May 16, 2012 at 12:53:26PM -0700, Yoshiki Ohshima wrote: > > > Thank you, Dave! > > > > > > I now understand that going through CommandShell was the sanctioned > > > way of using PipeableOSProcess. I'll just modify our code to close it > > > explicitly. (And it seems to work beter.) > > > > > > I tried clearing the allMyChildren directory but did not seem to free > > > these fds. I tried: > > > > > > PipeableOSProcess allSubInstancesDo: [:e | e closePipes] > > > > > > but this do it seems to kill the VM. > > > > Eeek! This should not be happening. PipeableOSProcess>>closePipes uses > > StandardFileStream>>primCloseNoError: which does an fclose() to close > > the pipe descriptor. This should be failure-proof, but I might be missing > > something. > > > > Is this reproduceable? It does not seem to happen on my Linux PC, but > > maybe it is an intermittent failure of some kind. > > This is reproduceable in my combination of things. It does not have > to be allSubInstancesDo:, but allInstancesDo: does it (of course). > > Oh, the trick seems that I shouldn't close the notifier. Do the > first part: > ------------------------- > v := [1 to: 256 do: [:i | > (PipeableOSProcess command: 'echo hello') output. > (Delay forMilliseconds: 50) wait.]]. > v fork. > ------------------------- > > and while the notifier is shown on the screen, evaluate: > > ------------------------- > PipeableOSProcess allInstancesDo: [:e | e closePipes] > ------------------------- > Thanks Yoshiki, I see it now. I cannot explain what is happening. The system is behaving as if the VM was blocked on a read, but I don't think that this is the actual problem. Interesting ... Dave |
At Thu, 17 May 2012 10:16:53 -0400,
David T. Lewis wrote: > > On Wed, May 16, 2012 at 07:38:36PM -0700, Yoshiki Ohshima wrote: > > At Wed, 16 May 2012 19:47:15 -0400, > > David T. Lewis wrote: > > > > > > On Wed, May 16, 2012 at 12:53:26PM -0700, Yoshiki Ohshima wrote: > > > > Thank you, Dave! > > > > > > > > I now understand that going through CommandShell was the sanctioned > > > > way of using PipeableOSProcess. I'll just modify our code to close it > > > > explicitly. (And it seems to work beter.) > > > > > > > > I tried clearing the allMyChildren directory but did not seem to free > > > > these fds. I tried: > > > > > > > > PipeableOSProcess allSubInstancesDo: [:e | e closePipes] > > > > > > > > but this do it seems to kill the VM. > > > > > > Eeek! This should not be happening. PipeableOSProcess>>closePipes uses > > > StandardFileStream>>primCloseNoError: which does an fclose() to close > > > the pipe descriptor. This should be failure-proof, but I might be missing > > > something. > > > > > > Is this reproduceable? It does not seem to happen on my Linux PC, but > > > maybe it is an intermittent failure of some kind. > > > > This is reproduceable in my combination of things. It does not have > > to be allSubInstancesDo:, but allInstancesDo: does it (of course). > > > > Oh, the trick seems that I shouldn't close the notifier. Do the > > first part: > > ------------------------- > > v := [1 to: 256 do: [:i | > > (PipeableOSProcess command: 'echo hello') output. > > (Delay forMilliseconds: 50) wait.]]. > > v fork. > > ------------------------- > > > > and while the notifier is shown on the screen, evaluate: > > > > ------------------------- > > PipeableOSProcess allInstancesDo: [:e | e closePipes] > > ------------------------- > > > > Thanks Yoshiki, > > I see it now. I cannot explain what is happening. The system is > behaving as if the VM was blocked on a read, but I don't think > that this is the actual problem. Interesting ... Just another data point: the crash report shows that VM is blocked on a read on Mac, too. -- Yoshiki |
On Thu, May 17, 2012 at 09:54:11AM -0700, Yoshiki Ohshima wrote:
> At Thu, 17 May 2012 10:16:53 -0400, > David T. Lewis wrote: > > > > On Wed, May 16, 2012 at 07:38:36PM -0700, Yoshiki Ohshima wrote: > > > At Wed, 16 May 2012 19:47:15 -0400, > > > David T. Lewis wrote: > > > > > > > > On Wed, May 16, 2012 at 12:53:26PM -0700, Yoshiki Ohshima wrote: > > > > > Thank you, Dave! > > > > > > > > > > I now understand that going through CommandShell was the sanctioned > > > > > way of using PipeableOSProcess. I'll just modify our code to close it > > > > > explicitly. (And it seems to work beter.) > > > > > > > > > > I tried clearing the allMyChildren directory but did not seem to free > > > > > these fds. I tried: > > > > > > > > > > PipeableOSProcess allSubInstancesDo: [:e | e closePipes] > > > > > > > > > > but this do it seems to kill the VM. > > > > > > > > Eeek! This should not be happening. PipeableOSProcess>>closePipes uses > > > > StandardFileStream>>primCloseNoError: which does an fclose() to close > > > > the pipe descriptor. This should be failure-proof, but I might be missing > > > > something. > > > > > > > > Is this reproduceable? It does not seem to happen on my Linux PC, but > > > > maybe it is an intermittent failure of some kind. > > > > > > This is reproduceable in my combination of things. It does not have > > > to be allSubInstancesDo:, but allInstancesDo: does it (of course). > > > > > > Oh, the trick seems that I shouldn't close the notifier. Do the > > > first part: > > > ------------------------- > > > v := [1 to: 256 do: [:i | > > > (PipeableOSProcess command: 'echo hello') output. > > > (Delay forMilliseconds: 50) wait.]]. > > > v fork. > > > ------------------------- > > > > > > and while the notifier is shown on the screen, evaluate: > > > > > > ------------------------- > > > PipeableOSProcess allInstancesDo: [:e | e closePipes] > > > ------------------------- > > > > > > > Thanks Yoshiki, > > > > I see it now. I cannot explain what is happening. The system is > > behaving as if the VM was blocked on a read, but I don't think > > that this is the actual problem. Interesting ... > > Just another data point: the crash report shows that VM is blocked on > a read on Mac, too. > > -- Yoshiki The Cog VM produces a very useful stack dump that confirms that the VM is blocking on a read from an OS pipe. I cannot explain exactly how we are reaching this state, but apparently closing all those pipe handles with "PipeableOSProcess allInstancesDo: [:e | e closePipes]" is not a good thing to do while the debugger window is open ;) Dave SIGUSR1 Thu May 17 21:51:17 2012 Squeak VM version: 4.0-2550 #12 Wed Apr 18 09:27:54 PDT 2012 gcc 4.1.2 Built from: CoInterpreter VMMaker.oscog-eem.157 uuid: 9a11acbe-4a62-43ce-81de-934f351598cc Apr 4 2012 With: StackToRegisterMappingCogit VMMaker.oscog-eem.157 uuid: 9a11acbe-4a62-43ce-81de-934f351598cc Apr 4 2012 Revision: VM: r2550 http://www.squeakvm.org/svn/squeak/branches/Cog Plugins: r2545 http://squeakvm.org/svn/squeak/trunk/platforms/Cross/plugins Build host: Linux mcqfes 2.6.18-128.el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686 i686 i386 GNU/Linux plugin path: /usr/local/bin/../lib/squeak/4.0-2550 [default: /usr/local/lib/squeak/4.0-2550/] C stack backtrace: /usr/local/bin/../lib/squeak/4.0-2550/squeak[0x805b821] /usr/local/bin/../lib/squeak/4.0-2550/squeak[0x805ba8d] [0xffffe410] All Smalltalk process stacks (active first): Process 0xc0c32e28 priority 40 0xff7d4914 M AttachableFileStream(StandardFileStream)>readInto:startingAt:count: -1053201620: a(n) AttachableFileStream 0xff7d493c M AttachableFileStream(StandardFileStream)>next:into:startingAt: -1053201620: a(n) AttachableFileStream 0xff7d4960 M AttachableFileStream(PositionableStream)>nextInto: -1053201620: a(n) AttachableFileStream 0xff7d4988 M AttachableFileStream>upToEnd -1053201620: a(n) AttachableFileStream 0xff7d49a0 M ExternalPipe>upToEnd -1053201688: a(n) ExternalPipe 0xff7d49c0 M [] in PipeableOSProcess>closePipes -1053202016: a(n) PipeableOSProcess |
Administrator
|
#closePipes is calling #upToEnd. Could this be another case of failing to call #setNonBlockingOutput?
Cheers,
Sean |
On Thu, May 17, 2012 at 07:29:29PM -0700, Sean P. DeNigris wrote:
> > David T. Lewis wrote > > > > the VM is blocking on a read from an OS pipe... > > > > 0xff7d49a0 M ExternalPipe>upToEnd -1053201688: a(n) ExternalPipe > > 0xff7d49c0 M [] in PipeableOSProcess>closePipes -1053202016: a(n) > > PipeableOSProcess > > > > #closePipes is calling #upToEnd. Could this be another case of failing to > call #setNonBlockingOutput? Good catch Sean, that's exactly what's happening. The #closePipes does a few things in addition to closing the pipes, because it is intended for use in a process pipeline. Specifically, it is used to clean up the pipes for the last process in a pipeline of processes, in which case the output of that last process will have been set to nonblocking. I'm not sure that this is a case of failing to call #setNonBlockingOutput (because in general the output of an external process *should* be in blocking mode). But calling #upToEnd on a blocking file stream will lock up the VM if no data is available, and that's exactly what is happening here. This should be preventable ... hmmm ... Dave |
Free forum by Nabble | Edit this page |