ProxyPipeLine terminates external command prematurely

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

ProxyPipeLine terminates external command prematurely

JohnReed Maffeo
I have been trying to get Squeak to execute a complex command for me. I tried OSProcess and CommandShell , but OSProcess does not work and gives me no errors (that I can see), CommandShell does not work because it interprets the command line I give it as multiple commands because my command string contains #& which appears to be a special character in #findTokens:keep.
 
When I tried ProxyPipeLine it looks likes it starts to work because a zero length file (zlf) is written to the target directory, but that is all I get. If I run the same command line that I feed to ProxyPipeLine to an external Treminal session, the zero length file is written to the target directory and then the program runs for several seconds and the file is populated. This makes me think there is a timming issue somewhere.
 
My test process is:
1 In a Workspace, enter the command string and Inspect it.
2. In the Inspector, Explore self value.
3. The zlf gets written to the directory
4. The ProxyPipeline says "a PipeableOSProcess on an ExternalUnixOSProcess with pid 53180 on /opt/local/bin/rtmpdump (complete, normal termination with status 1)"
5. The zlf stays zero
6. Tried again from 1. with waitForAllToComplete, but 'pipeline did not complete evaluation'
 
This is the closest I have gotten to getting the results I am looking for. I think I need some way to tell the pipeline to stay alive while the pipe completes its business.
 
--- my invocation
ProxyPipeline fromString: '/opt/local/bin/rtmpdump -r "rtmp://bbcodspdns.fcod.llnwd.net:1935/a5999/e1?as=adobe-hmac-sha256&av=1&te=connect&mp=prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4&et=1410651181&fmta-token=bcc4153708137784417c3b5350dd4636d4abe4884d85f3d650ff08d14357bb65" -a "a5999/e1?as=adobe-hmac-sha256&av=1&te=connect&mp=prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4&et=1410651181&fmta-token=bcc4153708137784417c3b5350dd4636d4abe4884d85f3d650ff08d14357bb65" -y "mp4:prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4" -o /Volumes/MediaVault/BBC/79151303.flv ' shell: CommandShell new
-- end
This is a time sensitive command and this example is not repeatable after a few days.
 
Thanks in advance for any suggetions,
jrm
 

_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: ProxyPipeLine terminates external command prematurely

David T. Lewis
Hi JohnReed,

I am away and cannot be of much help right now, but I'm afraid that I may
have given you bad advice in my last response. CommandShell does all the
command parsing in Smalltalk, and it may not be smart enough to handle the
complex parameters in the example you gave here. So it may be necessary to
let the real Unix shell do the parsing, which is exactly what you were
originally were trying to do - sorry.

I'll try to give you a better answer in a day or so.

Note, I have a hard time reading your examples in my mail system, so if
you are able to post messages in plain text rather than html that would
help me to understand what you are trying to do.

Dave


> <html><head></head><body><div style="font-family: Verdana;font-size:
> 12.0px;"><div>
> <div style="font-family: Verdana; font-size: 12px; line-height: normal;">I
> have been trying to get Squeak to execute a complex command for me. I
> tried OSProcess and CommandShell , but OSProcess does not work and gives
> me no errors (that I can see), CommandShell does not work because it
> interprets the command line I give it as multiple commands because my
> command string contains #&amp; which appears to be a special character in
> #findTokens:keep.</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">&nbsp;</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">When I tried ProxyPipeLine it looks likes it starts to work
> because a zero length file (zlf) is written to the target directory, but
> that is all I get. If I run the same command line that I feed to
> ProxyPipeLine to an external Treminal session, the&nbsp;<span
> style="line-height: 19.2px;">zero length file is written to the target
> directory and then the program runs for several seconds and the file is
> populated. This makes me think there is a timming issue
> somewhere.</span></div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">&nbsp;</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">My test process is:</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height: normal;">1
> In a Workspace, enter the command string and Inspect it.</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">2. In the Inspector, Explore self value.</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">3. The zlf gets written to the directory</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">4. The ProxyPipeline says &quot;a PipeableOSProcess on an
> ExternalUnixOSProcess with pid 53180 on /opt/local/bin/rtmpdump (complete,
> normal termination with status 1)&quot;</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">5. The zlf stays zero</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">6. Tried again from 1. with&nbsp;waitForAllToComplete,
> but&nbsp;&#39;pipeline did not complete evaluation&#39;</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">&nbsp;</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">This is the closest I have gotten to getting the results I am
> looking for. I think I need some way to tell the pipeline to stay alive
> while the pipe completes its business.</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">&nbsp;</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">--- my invocation</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">ProxyPipeline fromString: &#39;/opt/local/bin/rtmpdump -r
> &quot;rtmp://bbcodspdns.fcod.llnwd.net:1935/a5999/e1?as=adobe-hmac-sha256&amp;av=1&amp;te=connect&amp;mp=prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4&amp;et=1410651181&amp;fmta-token=bcc4153708137784417c3b5350dd4636d4abe4884d85f3d650ff08d14357bb65&quot;
> -a
> &quot;a5999/e1?as=adobe-hmac-sha256&amp;av=1&amp;te=connect&amp;mp=prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4&amp;et=1410651181&amp;fmta-token=bcc4153708137784417c3b5350dd4636d4abe4884d85f3d650ff08d14357bb65&quot;
> -y
> &quot;mp4:prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4&quot;
> -o /Volumes/MediaVault/BBC/79151303.flv &#39; shell: CommandShell
> new</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">-- end</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">This is a time sensitive command and this example is not
> repeatable after a few days.</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">&nbsp;</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">Thanks in advance for any suggetions,</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">jrm</div>
>
> <div style="font-family: Verdana; font-size: 12px; line-height:
> normal;">&nbsp;</div>
> </div></div></body></html>
> _______________________________________________
> Beginners mailing list
> [hidden email]
> http://lists.squeakfoundation.org/mailman/listinfo/beginners
>


_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: ProxyPipeLine terminates external command prematurely

JohnReed Maffeo

> Sent: Monday, September 15, 2014 at 6:53 AM
> From: "David T. Lewis" <[hidden email]>
> To: "A friendly place to get answers to even the most basic questions about Squeak." <[hidden email]>
> Subject: Re: [Newbies] ProxyPipeLine terminates external command prematurely
>
> Hi JohnReed,
>
> I am away and cannot be of much help right now, but I'm afraid that I may
> have given you bad advice in my last response. CommandShell does all the
> command parsing in Smalltalk, and it may not be smart enough to handle the
> complex parameters in the example you gave here. So it may be necessary to
> let the real Unix shell do the parsing, which is exactly what you were
> originally were trying to do - sorry.
>
> I'll try to give you a better answer in a day or so.
>
> Note, I have a hard time reading your examples in my mail system, so if
> you are able to post messages in plain text rather than html that would
> help me to understand what you are trying to do.
>
> Dave

--snip html ugliness
Dave, thank you for replying. Sorry for the HTML junk. I thought it was turned off and it should be now. I think I am asking the framework to do more than it was designed to do. OSProcess and CommandShell seem to eat #&, and ProxyPipeLine seems to terminate prematurely. I have begun work on a package(is that the right word?) to create a shell command file in Squeak that I can save to the local file system and then execute that shell command file using OSProcess. My UNIX skills are a bit rusty, but have done some prototyping and think it will work. It will be fragile, and OS and application specific, but it will work for me.

If I had a clue of how to start, I would try to create a new method for OSProcess, #justDoThisAndNothingMore.
(all this work because my iPod Touch will not play aac files in flash wrappers :-(

-- original message
I have been trying to get Squeak to execute a complex command for me. I tried OSProcess and CommandShell , but OSProcess does not work and gives me no errors (that I can see), CommandShell does not work because it interprets the command line I give it as multiple commands because my command string contains #& which appears to be a special character in #findTokens:keep.
 
When I tried ProxyPipeLine it looks likes it starts to work because a zero length file (zlf) is written to the target directory, but that is all I get. If I run the same command line that I feed to ProxyPipeLine to an external Treminal session, the zero length file is written to the target directory and then the program runs for several seconds and the file is populated. This makes me think there is a timming issue somewhere.
 
My test process is:
1 In a Workspace, enter the command string and Inspect it.
2. In the Inspector, Explore self value.
3. The zlf gets written to the directory
4. The ProxyPipeline says "a PipeableOSProcess on an ExternalUnixOSProcess with pid 53180 on /opt/local/bin/rtmpdump (complete, normal termination with status 1)"
5. The zlf stays zero
6. Tried again from 1. with waitForAllToComplete, but 'pipeline did not complete evaluation'
 
This is the closest I have gotten to getting the results I am looking for. I think I need some way to tell the pipeline to stay alive while the pipe completes its business.
 
--- my invocation (this is an abbreviated version of the whole command that I am using for testing))
ProxyPipeline fromString: '/opt/local/bin/rtmpdump -r "rtmp://bbcodspdns.fcod.llnwd.net:1935/a5999/e1?as=adobe-hmac-sha256&av=1&te=connect&mp=prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4&et=1410651181&fmta-token=bcc4153708137784417c3b5350dd4636d4abe4884d85f3d650ff08d14357bb65" -a "a5999/e1?as=adobe-hmac-sha256&av=1&te=connect&mp=prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4&et=1410651181&fmta-token=bcc4153708137784417c3b5350dd4636d4abe4884d85f3d650ff08d14357bb65" -y "mp4:prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4" -o /Volumes/MediaVault/BBC/79151303.flv ' shell: CommandShell new
-- end
NOTE: This is a time sensitive command and this example is not repeatable after a few days.
 
Thanks in advance for any suggetions,
jrm
_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Reply | Threaded
Open this post in threaded view
|

Re: ProxyPipeLine terminates external command prematurely

David T. Lewis

On Mon, Sep 15, 2014 at 07:47:33PM +0200, JohnReed Maffeo wrote:

>
> > Sent: Monday, September 15, 2014 at 6:53 AM
> > From: "David T. Lewis" <[hidden email]>
> > To: "A friendly place to get answers to even the most basic questions about Squeak." <[hidden email]>
> > Subject: Re: [Newbies] ProxyPipeLine terminates external command prematurely
> >
> > Hi JohnReed,
> >
> > I am away and cannot be of much help right now, but I'm afraid that I may
> > have given you bad advice in my last response. CommandShell does all the
> > command parsing in Smalltalk, and it may not be smart enough to handle the
> > complex parameters in the example you gave here. So it may be necessary to
> > let the real Unix shell do the parsing, which is exactly what you were
> > originally were trying to do - sorry.
> >
> > I'll try to give you a better answer in a day or so.
> >
> > Note, I have a hard time reading your examples in my mail system, so if
> > you are able to post messages in plain text rather than html that would
> > help me to understand what you are trying to do.
> >
> > Dave
>
> --snip html ugliness
> Dave, thank you for replying. Sorry for the HTML junk. I thought it was turned off and it should be now. I think I am asking the framework to do more than it was designed to do. OSProcess and CommandShell seem to eat #&, and ProxyPipeLine seems to terminate prematurely. I have begun work on a package(is that the right word?) to create a shell command file in Squeak that I can save to the local file system and then execute that shell command file using OSProcess. My UNIX skills are a bit rusty, but have done some prototyping and think it will work. It will be fragile, and OS and application specific, but it will work for me.
>
> If I had a clue of how to start, I would try to create a new method for OSProcess, #justDoThisAndNothingMore.
> (all this work because my iPod Touch will not play aac files in flash wrappers :-(
Hi JohnReed,

I did some tinkering with the command line that you were trying to run,
and I think the issue amounts to finding a way to pass the command line
(complete with funny characters that would be confusing to Smalltalk) to
a real Unix shell, and having the Unix shell do the work. But figuring out
how a Unix shell parses as string before passing it along to the program
that you are trying to run can be a bit tricky.

I am attaching a workspace that illustrates an approach that I think may
work for your problem. I put it into a workspace to make sure that the
example does not get mangled in email, but I'll also put in in this mail
below so you can read it.

We are probably getting a bit out of newbie territory here, so don't be
shy about asking additional questions on the main squeak-dev list.

Let me know if this helps. I think I'd like to come up with a better
way to handle problems like this, and your example makes for a nice
use case.

Dave

--- example workspace below ---

"Work out how to pass a command line with strange characters to a Unix shell for execution"

"This is a copy of the Unix command line that you want to run"
aCommandToTest := '/opt/local/bin/rtmpdump -r "rtmp://bbcodspdns.fcod.llnwd.net:1935/a5999/e1?as=adobe-hmac-sha256&av=1&te=connect&mp=prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4&et=1410651181&fmta-token=bcc4153708137784417c3b5350dd4636d4abe4884d85f3d650ff08d14357bb65" -a "a5999/e1?as=adobe-hmac-sha256&av=1&te=connect&mp=prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4&et=1410651181&fmta-token=bcc4153708137784417c3b5350dd4636d4abe4884d85f3d650ff08d14357bb65" -y "mp4:prod_rb2_flv_aac_low_intl/iplayerstream/p025mlpk_b007gwqn_1410009892525.mp4" -o /Volumes/MediaVault/BBC/79151303.flv'.

"I do not have the rtmpdump program on my system, so I will just pass the command to /bin/echo to see how it gets parsed by the Unix shell."
unixCommandLine := '/bin/echo ', aCommandToTest.

"Make a pipeable OS process running a Unix shell program"
sh := PipeableOSProcess shell.

"A pipeable OS process is dangerous, because it was designed to be used in CommandShell,
and its output pipe has been left in the default non-blocking mode. This can lock up your VM
if you read from the blocking pipe with no data available. Set it to non-blocking mode to
be safe."
sh pipeFromOutput setNonBlocking.

"Write the command line to the Unix shell"
sh exec: unixCommandLine.

"Close the stdin to the shell, thus writing the command line and exiting the shell when done"
sh close.

"Read the output of the command. See also #upToEnd and #errorUpToEnd. Here
we use #upToEndOfFile because the output pipe is non-blocking, we have closed
the input to the shell, and we want to read all available output even though the shell
may still be working on it at the moment. You could also delay for 20 msec and then
read #upToEnd to get the same result."
result := sh upToEndOfFile.

"We are done using the shell and reading from its output pipe, so close any open pipe handles"
sh closePipes.

"The Unix shell will have stripped out the double-quote characters when it parsed the
command, and before passing it to the /bin//echo program for execution. The /bin/echo
command echos that string exactly, and the shell adds a Character lf terminator. So if
you remove the double-quotes from the original command line, and compare it to the
result string with trailing line feed removed, they are the same."

(aCommandToTest copyWithout: $") = result allButLast "==> true"




_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners

JohnReedMaffeoOSProcess.text.gz (1K) Download Attachment