PipeableOSProcess incremental output

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

PipeableOSProcess incremental output

Sean P. DeNigris
Administrator
How do I get the output and error of a PipeableOSProcess command line-by-line as it comes, like CommandShell does?

I found #setNonBlockingOutput for the output, but "pipeFromError setNonBlocking" doesn't seem to do the same for the errors.

Thanks.
Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: PipeableOSProcess incremental output

Sean P. DeNigris
Administrator
I got a little further. #updateErrorPipelineStream helped to get the error messages as they came.

p := PipeableOSProcess command: '/usr/local/bin/HandBrakeCLI -e x264 -q 20 -m --start-at duration:1 --stop-at duration:5 -i /Volumes/DISK_01 -o "/path/to/DISK_01.mp4"'.
[p isComplete] whileFalse: [
        p updateErrorPipelineStream.
        Transcript show: p errorUpToEnd ].
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: PipeableOSProcess incremental output

Sean P. DeNigris
Administrator
Sean P. DeNigris wrote
p := PipeableOSProcess command: '/usr/local/bin/HandBrakeCLI -e x264 -q 20 -m --start-at duration:1 --stop-at duration:5 -i /Volumes/DISK_01 -o "/path/to/DISK_01.mp4"'.
Now I'm running into weirdness with the output:
* "p pipeFromOutput reader isEmpty not" always returns true.
* "p atEnd" -> In AttachableFileStream>>#primSetPosition:to:, I get a primitive failure

Are these bugs?

Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: PipeableOSProcess incremental output

David T. Lewis
In reply to this post by Sean P. DeNigris
On Thu, Oct 27, 2011 at 04:58:34PM -0700, Sean P. DeNigris wrote:
> How do I get the output and error of a PipeableOSProcess command line-by-line
> as it comes, like CommandShell does?
>

In PipeableOSProcess, the output stream is #pipeFromOutput and
the error stream is #errorPipelineStream. For example:

  proc := PipeableOSProcess command: 'ls -l * NOSUCHFILE'.
  out := proc pipeFromOutput.
  err := proc errorPipelineStream.

>From any of these streams you can read character by character
with #next, or you can read the entire contents of the stream
with #upToEnd or #upToEndOfFile. On a unix system, lines of
output in a pipe will usually be separated by line feed characters,
so you can read character by character looking for a Character lf.

But actually you do not even need to access those streams
directly, because PipeableOSProcess itself behaves like a stream.
You can read output directly from a PipeableOSProcess with
#next, #next:, #upToEnd, #upToEndOfFile, #errorUpToEnd, and
#errorUpToEndOfFile.

To illustrate the idea, one way to get the output of a command
line by line as follows:

  (PipeableOSProcess command: 'ls -l')
       upToEndOfFile findTokens: String lf

> I found #setNonBlockingOutput for the output, but "pipeFromError
> setNonBlocking" doesn't seem to do the same for the errors.
>

In PipeableOSProcess this is all handled automatically, so
you can just use the output and error streams as described
above.

Dave
 
_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: PipeableOSProcess incremental output

Sean P. DeNigris
Administrator
Thanks David. Is there a way to query whether output/errors are available? I saw some mention of an event driven mechanism, but wasn't clear how to use it.

Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: PipeableOSProcess incremental output

David T. Lewis
On Fri, Oct 28, 2011 at 09:28:25AM -0700, Sean P. DeNigris wrote:
> Thanks David. Is there a way to query whether output/errors are available? I
> saw some mention of an event driven mechanism, but wasn't clear how to use
> it.

The event-driven stuff is "under the covers". It basically watches
for available data, for example when data is available on the
stderr stream of the external process it is automatically pushed
into the errorPipelineStream.

The easiest way to see if data is available is to read from the
stream and see what you get. For example, if you do #upToEnd and
get an empty string, then there is no data available. Note that
#upToEnd gives you all of the currently available data (there
may be more on the way, for example if the external process is
something that runs continuously and puts out data once in a
while). If you are dealing with an external program that you
expect to run once to completion, then you can use #upToEndOfFile
to read everything until the process is complete.

The output and error streams are just streams of data containing
whatever happens to come out of the external process. If you are
interested in error conditions, you can check exit status of the
external process after it runs to completion. Note that the
external process may not exit until after you have read its
output data (it will block on writing to the pipe if it needs
to write a lot of data and you have not read any of it yet),
so in general you will want to read the output data, then check
exit status afterwards (and maybe read the errorOutputStream
also).

Dave

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: PipeableOSProcess incremental output

Sean P. DeNigris
Administrator
In reply to this post by Sean P. DeNigris
Sean P. DeNigris wrote
I got a little further. #updateErrorPipelineStream helped to get the error messages as they came.
Dave, based on your answers, this seems like it should not be necessary, yet without it #errorUpToEnd returned an empty string even though there was output available. Is there an internal flush that's not getting done?

Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: PipeableOSProcess incremental output

David T. Lewis
On Sat, Nov 26, 2011 at 07:40:42AM -0800, Sean P. DeNigris wrote:

>
> Sean P. DeNigris wrote
> >
> > I got a little further. #updateErrorPipelineStream helped to get the error
> > messages as they came.
> >
>
> Dave, based on your answers, this seems like it should not be necessary, yet
> without it #errorUpToEnd returned an empty string even though there was
> output available. Is there an internal flush that's not getting done?

PipeableOSProcess is designed to be used in a pipeline of connected
process proxies (or other types of PipeJunction). In that context,
The standard output of each PipeJunction is connected to the standard
input of the next, and the error output (if any) from each PipeJunction
needs to go somewhere. The errorPipelineStream is shared by multiple
process proxies in the pipeline, and provides a place to accumulate
the error output from any of the process proxies. The updates to this
accumulator stream are triggered by change notifications that happen
as data becomes available on the error output streams of each process
proxy.

HTH,
Dave

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: PipeableOSProcess incremental output

Sean P. DeNigris
Administrator
David T. Lewis wrote
PipeableOSProcess is designed...
I learned a lot from your answer (I always forget I can chain them together), but I'm not clear whether it answered my question :)

Let me rephrase... In my original example:
    PipeableOSProcess command: '/usr/local/bin/HandBrakeCLI -e x264 -q 20 -m --start-at duration:1 --stop-at duration:5 -i /Volumes/DISK_01 -o "/path/to/DISK_01.mp4"'

As a naive user, I would assume that "p errorUpToEnd" would show me the error output. I found that it did not unless I first sent "p updateErrorPipelineStream". From your answer, it sounds like the error stream should be automatically notified and updated when new data is available, but this doesn't seem to be happening.

Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: PipeableOSProcess incremental output

David T. Lewis
On Mon, Nov 28, 2011 at 06:03:18AM -0800, Sean P. DeNigris wrote:

>
> David T. Lewis wrote
> >
> > PipeableOSProcess is designed...
> >
>
> I learned a lot from your answer (I always forget I can chain them
> together), but I'm not clear whether it answered my question :)
>
> Let me rephrase... In my original example:
>     PipeableOSProcess command: '/usr/local/bin/HandBrakeCLI -e x264 -q 20 -m
> --start-at duration:1 --stop-at duration:5 -i /Volumes/DISK_01 -o
> "/path/to/DISK_01.mp4"'
>
> As a naive user, I would assume that "p errorUpToEnd" would show me the
> error output. I found that it did not unless I first sent "p
> updateErrorPipelineStream". From your answer, it sounds like the error
> stream should be automatically notified and updated when new data is
> available, but this doesn't seem to be happening.

Try using #errorUpToEndOfFile rather than #errorUpToEnd. This will
wait for all available data, as opposed to just the data that is
currently available.

The reason that you are getting an empty string is that you are
reading errorUpToEnd before that error output has been completely
updated. The data will not be available until the external process
has written it to its stderr, flushed the steam (usually by exiting),
and your Squeak VM has detected data available, your OSProcess
has read the data, and all of those buffered streams have been
updated. This all takes place asynchronously with semaphores being
signaled and background processes waking up to update the data.

You can also illustrate this simply by inserting a delay between
starting the command and reading its error output. If you wait
for a few hundred milliseconds, the data will magically appear
and the errorUpToEnd will now give you the results you expect.

HTH,
Dave

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners