PipeableOSProcess incremental output

How do I get the output and error of a PipeableOSProcess command line-by-line as it comes, like CommandShell does?

I found #setNonBlockingOutput for the output, but "pipeFromError setNonBlocking" doesn't seem to do the same for the errors.

Thanks.
Sean

I got a little further. #updateErrorPipelineStream helped to get the error messages as they came.

p := PipeableOSProcess command: '/usr/local/bin/HandBrakeCLI -e x264 -q 20 -m --start-at duration:1 --stop-at duration:5 -i /Volumes/DISK_01 -o "/path/to/DISK_01.mp4"'.
[p isComplete] whileFalse: [
        p updateErrorPipelineStream.
        Transcript show: p errorUpToEnd ].

Sean P. DeNigris wrote

p := PipeableOSProcess command: '/usr/local/bin/HandBrakeCLI -e x264 -q 20 -m --start-at duration:1 --stop-at duration:5 -i /Volumes/DISK_01 -o "/path/to/DISK_01.mp4"'.

Now I'm running into weirdness with the output:
* "p pipeFromOutput reader isEmpty not" always returns true.
* "p atEnd" -> In AttachableFileStream>>#primSetPosition:to:, I get a primitive failure

Are these bugs?

Sean

On Thu, Oct 27, 2011 at 04:58:34PM -0700, Sean P. DeNigris wrote:
> How do I get the output and error of a PipeableOSProcess command line-by-line
> as it comes, like CommandShell does?
>

In PipeableOSProcess, the output stream is #pipeFromOutput and
the error stream is #errorPipelineStream. For example:

  proc := PipeableOSProcess command: 'ls -l * NOSUCHFILE'.
  out := proc pipeFromOutput.
  err := proc errorPipelineStream.

>From any of these streams you can read character by character
with #next, or you can read the entire contents of the stream
with #upToEnd or #upToEndOfFile. On a unix system, lines of
output in a pipe will usually be separated by line feed characters,
so you can read character by character looking for a Character lf.

But actually you do not even need to access those streams
directly, because PipeableOSProcess itself behaves like a stream.
You can read output directly from a PipeableOSProcess with
#next, #next:, #upToEnd, #upToEndOfFile, #errorUpToEnd, and
#errorUpToEndOfFile.

To illustrate the idea, one way to get the output of a command
line by line as follows:

  (PipeableOSProcess command: 'ls -l')
       upToEndOfFile findTokens: String lf

> I found #setNonBlockingOutput for the output, but "pipeFromError
> setNonBlocking" doesn't seem to do the same for the errors.
>

In PipeableOSProcess this is all handled automatically, so
you can just use the output and error streams as described
above.

Dave
 
_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

Thanks David. Is there a way to query whether output/errors are available? I saw some mention of an event driven mechanism, but wasn't clear how to use it.

Sean

On Fri, Oct 28, 2011 at 09:28:25AM -0700, Sean P. DeNigris wrote:
> Thanks David. Is there a way to query whether output/errors are available? I
> saw some mention of an event driven mechanism, but wasn't clear how to use
> it.

The event-driven stuff is "under the covers". It basically watches
for available data, for example when data is available on the
stderr stream of the external process it is automatically pushed
into the errorPipelineStream.

The easiest way to see if data is available is to read from the
stream and see what you get. For example, if you do #upToEnd and
get an empty string, then there is no data available. Note that
#upToEnd gives you all of the currently available data (there
may be more on the way, for example if the external process is
something that runs continuously and puts out data once in a
while). If you are dealing with an external program that you
expect to run once to completion, then you can use #upToEndOfFile
to read everything until the process is complete.

The output and error streams are just streams of data containing
whatever happens to come out of the external process. If you are
interested in error conditions, you can check exit status of the
external process after it runs to completion. Note that the
external process may not exit until after you have read its
output data (it will block on writing to the pipe if it needs
to write a lot of data and you have not read any of it yet),
so in general you will want to read the output data, then check
exit status afterwards (and maybe read the errorOutputStream
also).

Dave

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

Sean P. DeNigris wrote

I got a little further. #updateErrorPipelineStream helped to get the error messages as they came.

Dave, based on your answers, this seems like it should not be necessary, yet without it #errorUpToEnd returned an empty string even though there was output available. Is there an internal flush that's not getting done?

Sean

On Sat, Nov 26, 2011 at 07:40:42AM -0800, Sean P. DeNigris wrote:
PipeableOSProcess is designed to be used in a pipeline of connected
process proxies (or other types of PipeJunction). In that context,
The standard output of each PipeJunction is connected to the standard
input of the next, and the error output (if any) from each PipeJunction
needs to go somewhere. The errorPipelineStream is shared by multiple
process proxies in the pipeline, and provides a place to accumulate
the error output from any of the process proxies. The updates to this
accumulator stream are triggered by change notifications that happen
as data becomes available on the error output streams of each process
proxy.

HTH,
Dave

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

David T. Lewis wrote

PipeableOSProcess is designed...

I learned a lot from your answer (I always forget I can chain them together), but I'm not clear whether it answered my question :)

Let me rephrase... In my original example:
    PipeableOSProcess command: '/usr/local/bin/HandBrakeCLI -e x264 -q 20 -m --start-at duration:1 --stop-at duration:5 -i /Volumes/DISK_01 -o "/path/to/DISK_01.mp4"'

As a naive user, I would assume that "p errorUpToEnd" would show me the error output. I found that it did not unless I first sent "p updateErrorPipelineStream". From your answer, it sounds like the error stream should be automatically notified and updated when new data is available, but this doesn't seem to be happening.

Sean

On Mon, Nov 28, 2011 at 06:03:18AM -0800, Sean P. DeNigris wrote:
Try using #errorUpToEndOfFile rather than #errorUpToEnd. This will
wait for all available data, as opposed to just the data that is
currently available.

The reason that you are getting an empty string is that you are
reading errorUpToEnd before that error output has been completely
updated. The data will not be available until the external process
has written it to its stderr, flushed the steam (usually by exiting),
and your Squeak VM has detected data available, your OSProcess
has read the data, and all of those buffered streams have been
updated. This all takes place asynchronously with semaphores being
signaled and background processes waking up to update the data.

You can also illustrate this simply by inserting a delay between
starting the command and reading its error output. If you wait
for a few hundred milliseconds, the data will magically appear
and the errorUpToEnd will now give you the results you expect.

HTH,
Dave

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners