[squeak-dev] reading and writing from STDIN and STDOUT

> I am competing in a friendly Code Smackdown at work.  Smalltalk vs.
> All on-comers (Haskell, Erlang, LISP, Java, C and Python).  The
> challenge is to write a client for a Chinese Poker game.
>
> Everything is going well except communicating with the Python-based
> server.  I am having trouble reading and writing from STDIN and
> STDOUT.  I've installed OSProcess which should give me access to these
> streams, but I don't see how to read one line at a time from STDIN and
> then get a handle on STDOUT to write a line.  Anyone have pointers or
> sample code?
>
> Also, the image needs to be launched by Python with code like...
>
>    subprocess.Popen(["/Application/Squeak/Contents/MacOS/Squeak VM
> Opt", "/Users/jrogers/Squeak/images/chinese_poker.image", "Squeak
> Client"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>
> This will obviously need to change for a server running on another OS.
>  The question I have here is how to start a process reading from STDIN
> as soon as the VM is fired up.  Should I just create an object that is
> waiting on STDIN, reading a line at a time, before saving the image?
> That way it would just keep trying to read when the image was fired up
> again.

> On Tue, May 12, 2009 at 04:59:27PM -0400, Jason Rogers wrote:
>> I am competing in a friendly Code Smackdown at work.  Smalltalk vs.
>> All on-comers (Haskell, Erlang, LISP, Java, C and Python).  The
>> challenge is to write a client for a Chinese Poker game.
>>
>> Everything is going well except communicating with the Python-based
>> server.  I am having trouble reading and writing from STDIN and
>> STDOUT.  I've installed OSProcess which should give me access to these
>> streams, but I don't see how to read one line at a time from STDIN and
>> then get a handle on STDOUT to write a line.  Anyone have pointers or
>> sample code?
>>
>> Also, the image needs to be launched by Python with code like...
>>
>>    subprocess.Popen(["/Application/Squeak/Contents/MacOS/Squeak VM
>> Opt", "/Users/jrogers/Squeak/images/chinese_poker.image", "Squeak
>> Client"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>
>> This will obviously need to change for a server running on another OS.
>>  The question I have here is how to start a process reading from STDIN
>> as soon as the VM is fired up.  Should I just create an object that is
>> waiting on STDIN, reading a line at a time, before saving the image?
>> That way it would just keep trying to read when the image was fired up
>> again.
>
> Hi Jason,
>
> I'm assuming that you are using OS X or other unix based OS (note
> that OSProcess does not do pipes correctly on Windows, so you won't
> be able to do that).
>
> Please install CommandShell in addition to OSProcess, as that will give
> you better ways to control external processes connected by pipes. I'm
> not clear from your question as to whether Python or Squeak is the
> "client" in this scenario, but it sounds like you will want to have
> Squeak making requests to a Python server, but you want Python to
> set up the pipes and invoke Squeak as a subprocess. I don't have
> an example of this scenario exactly, but the following may help.
>
> For an example of Squeak as the client, but where Squeak (not Python)
> does all the work of setting up the pipes connections to the server,
> see PipeableOSProcess class>>tkExample. This shows Squeak running
> a tk/tcl interpreter and putting up a trivial UI with buttons connected
> to Squeak.
>
> For an example of Squeak as the server, communicating through its
> STDIN and STDOUT streams (which might be pipes set up by Python in
> your scenario), see class ExternalCommandShell. After evaluating
> "ExternalCommandShell start" your Squeak image will read command
> lines from STDIN and respond on STDOUT. The ExternalCommandShell
> will evaluate input commands and Smalltalk expressions, returning
> its output to the STDOUT stream. This does not sound like what you
> want for your project, but it should give you an idea of how these
> things can be done.
>
> In general, the STDIN steam is "OSProcess thisOSProcess stdIn" and
> the STDOUT and STDERR streams are "OSProcess thisOSProcess stdOut"
> and "OSProcess thisOSProcess stdErr". Do not save these in variables,
> but evaluate them at runtime so your Squeak image can find the right
> input/output streams even after an image restart. That means you can
> start a Process in Squeak that reads from stdIn and writes to stdOut.
> Leave the process running when you save the image, and it will carry
> on where it left off whenever your Python program restarts the Squeak
> image.
>
> Note that the stdIn stream for Squeak is (and must be) non-blocking in
> order to prevent locking up the Squeak VM on a blocked read. That means
> that your process will need to read available data from stdIn, possibly
> looping until you get a complete string of whatever you are looking for.
> For example, if ten characters are currently available in your STDIN
> input stream, then "OSProcess thisOSProcess stdIn next: 10000" will
> answer a string of length ten.
>
> ExternalCommandShell does this all in an event-driven manner with
> no polling, so you can copy that if you want to get fancy. Otherwise
> a simple polling loop in your process will probably work well enough.
> Put a 100 ms delay in whatever loop is reading from stdIn an you will
> get tolerable performance.
>
> Dave
>
>
>

> On Tue, May 12, 2009 at 04:59:27PM -0400, Jason Rogers wrote:
>> I am competing in a friendly Code Smackdown at work.  Smalltalk vs.
>> All on-comers (Haskell, Erlang, LISP, Java, C and Python).  The
>> challenge is to write a client for a Chinese Poker game.
>>
>> Everything is going well except communicating with the Python-based
>> server.  I am having trouble reading and writing from STDIN and
>> STDOUT.  I've installed OSProcess which should give me access to these
>> streams, but I don't see how to read one line at a time from STDIN and
>> then get a handle on STDOUT to write a line.  Anyone have pointers or
>> sample code?
>>
>> Also, the image needs to be launched by Python with code like...
>>
>>    subprocess.Popen(["/Application/Squeak/Contents/MacOS/Squeak VM
>> Opt", "/Users/jrogers/Squeak/images/chinese_poker.image", "Squeak
>> Client"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>
>> This will obviously need to change for a server running on another OS.
>>  The question I have here is how to start a process reading from STDIN
>> as soon as the VM is fired up.  Should I just create an object that is
>> waiting on STDIN, reading a line at a time, before saving the image?
>> That way it would just keep trying to read when the image was fired up
>> again.
>
> Hi Jason,
>
> I'm assuming that you are using OS X or other unix based OS (note
> that OSProcess does not do pipes correctly on Windows, so you won't
> be able to do that).
>
> Please install CommandShell in addition to OSProcess, as that will give
> you better ways to control external processes connected by pipes. I'm
> not clear from your question as to whether Python or Squeak is the
> "client" in this scenario, but it sounds like you will want to have
> Squeak making requests to a Python server, but you want Python to
> set up the pipes and invoke Squeak as a subprocess. I don't have
> an example of this scenario exactly, but the following may help.
>
> For an example of Squeak as the client, but where Squeak (not Python)
> does all the work of setting up the pipes connections to the server,
> see PipeableOSProcess class>>tkExample. This shows Squeak running
> a tk/tcl interpreter and putting up a trivial UI with buttons connected
> to Squeak.
>
> For an example of Squeak as the server, communicating through its
> STDIN and STDOUT streams (which might be pipes set up by Python in
> your scenario), see class ExternalCommandShell. After evaluating
> "ExternalCommandShell start" your Squeak image will read command
> lines from STDIN and respond on STDOUT. The ExternalCommandShell
> will evaluate input commands and Smalltalk expressions, returning
> its output to the STDOUT stream. This does not sound like what you
> want for your project, but it should give you an idea of how these
> things can be done.
>
> In general, the STDIN steam is "OSProcess thisOSProcess stdIn" and
> the STDOUT and STDERR streams are "OSProcess thisOSProcess stdOut"
> and "OSProcess thisOSProcess stdErr". Do not save these in variables,
> but evaluate them at runtime so your Squeak image can find the right
> input/output streams even after an image restart. That means you can
> start a Process in Squeak that reads from stdIn and writes to stdOut.
> Leave the process running when you save the image, and it will carry
> on where it left off whenever your Python program restarts the Squeak
> image.
>
> Note that the stdIn stream for Squeak is (and must be) non-blocking in
> order to prevent locking up the Squeak VM on a blocked read. That means
> that your process will need to read available data from stdIn, possibly
> looping until you get a complete string of whatever you are looking for.
> For example, if ten characters are currently available in your STDIN
> input stream, then "OSProcess thisOSProcess stdIn next: 10000" will
> answer a string of length ten.
>
> ExternalCommandShell does this all in an event-driven manner with
> no polling, so you can copy that if you want to get fancy. Otherwise
> a simple polling loop in your process will probably work well enough.
> Put a 100 ms delay in whatever loop is reading from stdIn an you will
> get tolerable performance.
>
> Dave

> On Tue, May 12, 2009 at 8:59 PM, David T. Lewis <[hidden email]> wrote:
>> On Tue, May 12, 2009 at 04:59:27PM -0400, Jason Rogers wrote:
>>> I am competing in a friendly Code Smackdown at work.  Smalltalk vs.
>>> All on-comers (Haskell, Erlang, LISP, Java, C and Python).  The
>>> challenge is to write a client for a Chinese Poker game.
>>>
>>> Everything is going well except communicating with the Python-based
>>> server.  I am having trouble reading and writing from STDIN and
>>> STDOUT.  I've installed OSProcess which should give me access to these
>>> streams, but I don't see how to read one line at a time from STDIN and
>>> then get a handle on STDOUT to write a line.  Anyone have pointers or
>>> sample code?
>>>
>>> Also, the image needs to be launched by Python with code like...
>>>
>>>    subprocess.Popen(["/Application/Squeak/Contents/MacOS/Squeak VM
>>> Opt", "/Users/jrogers/Squeak/images/chinese_poker.image", "Squeak
>>> Client"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>>
>>> This will obviously need to change for a server running on another OS.
>>>  The question I have here is how to start a process reading from STDIN
>>> as soon as the VM is fired up.  Should I just create an object that is
>>> waiting on STDIN, reading a line at a time, before saving the image?
>>> That way it would just keep trying to read when the image was fired up
>>> again.
>>
>> Hi Jason,
>>
>> I'm assuming that you are using OS X or other unix based OS (note
>> that OSProcess does not do pipes correctly on Windows, so you won't
>> be able to do that).
>>
>> Please install CommandShell in addition to OSProcess, as that will give
>> you better ways to control external processes connected by pipes. I'm
>> not clear from your question as to whether Python or Squeak is the
>> "client" in this scenario, but it sounds like you will want to have
>> Squeak making requests to a Python server, but you want Python to
>> set up the pipes and invoke Squeak as a subprocess. I don't have
>> an example of this scenario exactly, but the following may help.
>>
>> For an example of Squeak as the client, but where Squeak (not Python)
>> does all the work of setting up the pipes connections to the server,
>> see PipeableOSProcess class>>tkExample. This shows Squeak running
>> a tk/tcl interpreter and putting up a trivial UI with buttons connected
>> to Squeak.
>>
>> For an example of Squeak as the server, communicating through its
>> STDIN and STDOUT streams (which might be pipes set up by Python in
>> your scenario), see class ExternalCommandShell. After evaluating
>> "ExternalCommandShell start" your Squeak image will read command
>> lines from STDIN and respond on STDOUT. The ExternalCommandShell
>> will evaluate input commands and Smalltalk expressions, returning
>> its output to the STDOUT stream. This does not sound like what you
>> want for your project, but it should give you an idea of how these
>> things can be done.
>>
>> In general, the STDIN steam is "OSProcess thisOSProcess stdIn" and
>> the STDOUT and STDERR streams are "OSProcess thisOSProcess stdOut"
>> and "OSProcess thisOSProcess stdErr". Do not save these in variables,
>> but evaluate them at runtime so your Squeak image can find the right
>> input/output streams even after an image restart. That means you can
>> start a Process in Squeak that reads from stdIn and writes to stdOut.
>> Leave the process running when you save the image, and it will carry
>> on where it left off whenever your Python program restarts the Squeak
>> image.
>>
>> Note that the stdIn stream for Squeak is (and must be) non-blocking in
>> order to prevent locking up the Squeak VM on a blocked read. That means
>> that your process will need to read available data from stdIn, possibly
>> looping until you get a complete string of whatever you are looking for.
>> For example, if ten characters are currently available in your STDIN
>> input stream, then "OSProcess thisOSProcess stdIn next: 10000" will
>> answer a string of length ten.
>>
>> ExternalCommandShell does this all in an event-driven manner with
>> no polling, so you can copy that if you want to get fancy. Otherwise
>> a simple polling loop in your process will probably work well enough.
>> Put a 100 ms delay in whatever loop is reading from stdIn an you will
>> get tolerable performance.
>>
>> Dave
>
> Thanks again Dave for the help.  My tack was to "copy"
> ExternalCommandShell's pattern.  StdIn is working fine, but when I
> write to StdOut nothing seems to actually "get out".  Any ideas?
>
> I'm running on Mac, I don't have the AioPlugin installed (I've
> commented out the inform: that tells about the plugin not being there
> and therefore polling will be used).  I'm not verbose enough at
> building plugins.
>
> Any ideas why?  I can post my Monticello file(s) somewhere if that would help.

> On Tue, May 12, 2009 at 8:59 PM, David T. Lewis <[hidden email]> wrote:
> >
> > ExternalCommandShell does this all in an event-driven manner with
> > no polling, so you can copy that if you want to get fancy. Otherwise
> > a simple polling loop in your process will probably work well enough.
> > Put a 100 ms delay in whatever loop is reading from stdIn an you will
> > get tolerable performance.
> >
> > Dave
>
> Thanks again Dave for the help.  My tack was to "copy"
> ExternalCommandShell's pattern.  StdIn is working fine, but when I
> write to StdOut nothing seems to actually "get out".  Any ideas?

> On Fri, May 15, 2009 at 01:31:57AM -0400, Jason Rogers wrote:
>> On Tue, May 12, 2009 at 8:59 PM, David T. Lewis <[hidden email]> wrote:
>> >
>> > ExternalCommandShell does this all in an event-driven manner with
>> > no polling, so you can copy that if you want to get fancy. Otherwise
>> > a simple polling loop in your process will probably work well enough.
>> > Put a 100 ms delay in whatever loop is reading from stdIn an you will
>> > get tolerable performance.
>> >
>> > Dave
>>
>> Thanks again Dave for the help.  My tack was to "copy"
>> ExternalCommandShell's pattern.  StdIn is working fine, but when I
>> write to StdOut nothing seems to actually "get out".  Any ideas?
>
> Try doing a #flush after you write to stdOut, like this:
>
>  OSProcess thisOSProcess stdOut
>     nextPutAll: 'hello world';
>     nextPut: Character lf;
>     flush.
>
> On unix (OS X), the I/O is done through the C stdio library. There is
> normally some buffering going on at that level, so if you flush the
> stream it should push the bytes out the tube.
>
>> I'm running on Mac, I don't have the AioPlugin installed (I've
>> commented out the inform: that tells about the plugin not being there
>> and therefore polling will be used).  I'm not verbose enough at
>> building plugins.
>
> Don't worry about it.  AioPlugin is used by CommandShell to do the
> event-driven I/O. If the plugin is not present, it gives you a warning
> and falls back on polling loops instead of using the "data ready"
> events from AioPlugin. For your application, polling loops should
> be good enough.
>
> Dave
>
>
>

>
>  I'm experiencing a similar problem: I have a method that calls an external function
>  via FFI. This function reports its progress via stdout. I'd like such output could be
>  redirected either to Transcript or to a squeak shell (worse). But that's not
>  happening. The routine prints on the shell where squeak was started. When squeak is
>  started in the form of something like: nohup squeak > /dev/null 2> /dev/null & the
>  routine redirects to the "device" (meaning /dev/tty...).
>  One consequence is that how there's no IO perceived by squeak, everything freezes
>  until routine is done (about 12 hours).
>  What would be the best way of redirecting stdio (file descriptor) to Transcript?

> Or direct stdout to a named pipe (unix mkfifo command) and read from
> the named pipe. If you use a named pipe, you will need to open it as
> a file stream in Squeak, then set the file stream to non-blocking mode
> before you use it, otherwise you will lock up your Squeak VM. Search
> for methods with 'setnonblocking' in the name. It's somewhere in
> OSProcess, but I don't have Squeak with me to check right now.
>
> Dave
>
>
>
>  

>
> David T. Lewis escreveu:
>
>> Do you mean that the external library is writing to stdout, and you
>> want to be able to see that output from within the Squeak image? In
>> that case you want Squeak to be reading its own standard output, and
>> redirecting it to the Transcript?
>>
>> (...)
>
>    Yes, I have something like (the most time consuming routine):
>        fInterface apiFannTrain: theFANN
>            onFile: trainingDataFileName
>            times: theMaxEpochs
>            reporting: numberOfEpochs
>            toPrecision: theDesiredError.
>    Where apiFannTrain is:
>    apiFannTrain: aFann onData: aTrainData times: aNumber1 reporting: aNumber2 toPrecision: aFloat
>        < cdecl: void 'fann_train_on_data' ( Fann* FannTrainData* ulong ulong float ) module: 'floatfann' >
>        ^ self externalCallFailed.
>    And that routine outputs progress to stdout. Problems are:
>     1. As I cannot redirect stdout from inside squeak, I'm not able to watch progress inside squeak (Transcript, morph... you name it)
>     2. As this kind of output doesn't create context change within squeak VM, apiFannTrain freezes VM until its completion (which may
>        take more than 12 hours). Geez, it's a long time with frozen squeak...
>
>    I just figured out that if it was possible to redirect stdout to something inside squeak it would allow me to fix the two problems.
>
> By the way, as the apiFannTrain is not preemptive from inside squeak,
> there's no use in "fork" it.