Launching other programs from within Smalltalk

I have a problem (well probably several, but this is the first I
found!) with my first attempt at a serious Smalltalk application. I
wonder if some kind person could give me some tips as to where to look
for information.

I am trying to use Smalltalk to provide a civilised front end to a
large and complex batch-oriented Fortran program (actually the US
Census Bureau's latest X-12 seasonal adjustment program).  The X-12
program is run from a command line which specifies where to find the
parameter file and where to write the output files, as well as various
switches determining the form of output. There may well be several
runs of the X-12 program in analysing one series, as different
parameter combinations are tried.

What I need to do, therefore, is to start the X-12 program running
with the appropriate command line from within my Smalltalk
application, detect when X-12 has finished running (typically under 10
seconds on my machine) and then read the various output files into my
application. My questions relate mainly to the first step.

I have found the method KernelLibrary>>system:, which seems to do what
I want, but is marked as deprecated. All it does is to invoke
KernelLibrary>>winExec:uCmdShow:, which is not deprecated. So my first
question is why the difference? Am I asking for any trouble by using
KernelLibrary>>system:, or should I use winExec:uCmdShow: in
preference? Or is there some other route altogether? I have also found
KernelLibrary>>createProcess:….., which might be another possibility,
but this has ten parameters and I am referred to the Win32 SDK help
for explanation of how to use them. Does this give any advantages, and
if so has anyone produced a wrapper which makes it easy to use with
default parameters?

My second requirement is to know when X-12 has finished. There is a
crude way to do this, because X-12 writes a log file at the end of the
run. A loop which tests say once a second for the existence of the log
file will do the trick, but something which triggers an event in my
Smalltalk application as soon as X-12 finishes would be neater.

Sorry for the lengthy explanation, and thanks in advance for any help
you can give.

Peter Kenny

Peter,

take a look at Bob Jarvis' ExternalProccess package (available here:
http://www.nls.net/mp/jarvis/Bob/DolphinGoodies.htm).

It should address the issues you are having.

CU,

Udo

"Peter Kenny" <[hidden email]> schrieb im Newsbeitrag
news:[hidden email]...

Peter Kenny wrote:

> What I need to do, therefore, is to start the X-12 program running
> with the appropriate command line from within my Smalltalk
> application, detect when X-12 has finished running (typically under 10
> seconds on my machine) and then read the various output files into my
> application. My questions relate mainly to the first step.

For some reason -- possibly reflecting a similar lack of focus in the Windows
APIs -- Dolphin has never had what I would call a well-rounded ability to start
sub-processes.

I suggest you take a look at CRTLibrary>>spawnForOutput: which is one example
of how to spawn a process and capture its output, if that's what you need to
do.  If you don't need to capture the output then
CRTLibrary>>_spawnvp:cmdname:argv: may do what you want.

Actually, I prefer to use a simple wrapper for the standard C library function
system().  I define it as follows:

===============
CRTLibrary>>system: aString
 "Spawn a new process to execute the given command line.

  int system(char *cmdline);

 Implementation Note: Overlapped so as to block only the calling process, as
the
 spawned external process may run for a lengthy time.
 "

 <overlap cdecl: sdword system lpstr>
#CUadded.
 ^self invalidCall
"
[CRTLibrary default system: 'command.com'] fork
"
===============

It is almost identical to #_spawnvp:cmdname:argv: except that it's marginally
easier to use, and is based on a standard C function, none of that Windows muck
;-)

In either case #_spawnvp:blah:blah: or #system: will block the calling Process,
but you can #fork off a new Process which will execute it, and then, say,
trigger a notification in your main application to say that it's finished.

Bob Jarvis has a more sophisticated package which appears (I haven't yet had
occasion to use it myself) to provide finer-grained control over spawned
Windows processes.  I had a little trouble re-discovering the link, but I think
that:
   <http://www.nls.net/mp/jarvis/Bob/DolphinGoodies.htm>
is correct.

As far as I know, nobody has ever described a way to avoid the window that pops
up when command line programs are executed from a "normal" Windows program.  I
suspect that it is possible, because I've seen other programs that seem to
manage the trick.  Does anyone know how that's done ?

    -- chris

Chris,

I'm using Bob's package extensivly to call remote proccess (sometimes up to
10 per second) and I didn't see
this popup effect.

But when I remember correctly I redirected standard output to a file ...
maybe this is the trick.

CU,

Udo


"Chris Uppal" <[hidden email]> schrieb im
Newsbeitrag news:[hidden email]...
> As far as I know, nobody has ever described a way to avoid the window that
pops
> up when command line programs are executed from a "normal" Windows
program.  I
> suspect that it is possible, because I've seen other programs that seem to
> manage the trick.  Does anyone know how that's done ?

Udo,

> But when I remember correctly I redirected standard output to a file ...
> maybe this is the trick.

Ditto on Bob's ExternalProcess package; I have used it for a while now
with good results.  You might have the answer to my one problem: I'm
having diffulty capturing output from compilers.  For example, it would
be nice to capture all of the output from InnoSetups's compiler, and (no
doubt over time) teach my code what constitutes a problem.  Likewise for
LaTeX.  Eventually, it might be nice to drive MinGW and/or MSVC to build
  projects (especially DLLs) as part of automated builds.  For now, I'd
settle for Inno's output :)

Do you have any examples of how you use ExternalProcess for such purposes?

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Bill Schwab wrote:
Some commandline tools need to be run in a command shell before you see
the output.
So if runCommand: 'tool.exe -flag' doesn't answer the stdout output you
might try runCommand: 'cmd.exe /c tool.exe -flag'
Another possibility is that the output is written to stderr instead of
stdout.

  - Pieter



runCommand: aString
        | stdoutName process tmpFile |
        stdoutName := File temporaryFilename.
        process := ExternalProcess new
                secondsToWait: 60;
                stdoutFilename: stdoutName;
                directory: self directory;
                commandLine: aString.

        status := ''.
        [process executeSync]
                on: ExternalProcessWaitFailure do: [ :e | ^status := 'Error while
waiting for external process. ']
                on: ExternalProcessWaitTimeout do: [ :e | ^status := 'Error, external
process execution time-out. '].
        process close.

        tmpFile := FileStream read: stdoutName.
        status := tmpFile contents.
        tmpFile close.
        File delete: stdoutName.

        ^status !

Udo,

> I'm using Bob's package extensivly to call remote proccess (sometimes up
> to 10 per second) and I didn't see
> this popup effect.

Just tried it and neither do I.  Thank you.  And thank you to Bob too.


> But when I remember correctly I redirected standard output to a file ...
> maybe this is the trick.

It seems that stdout/stderr are just discarded unless you tell the
ExternalProcess to trap them to a file.  Which, as it happens, is just the
behaviour I wanted :-)

    -- chris

Bill,

> I'm
> having diffulty capturing output from compilers.  For example, it would
> be nice to capture all of the output from InnoSetups's compiler

Does something like:

    (ExternalProcess new)
        commandLine: 'whatever';
        stdoutFilename: 'C:\Temp\whatever.txt';
        stderrFilename: 'C:\Temp\whatever.err';
        executeSync.

not work for you ?

For me that captures stuff written to stdout and stderr from a (VC-compiled) C
program.  Output sent explicitly to the console with cprintf() seems to be just
lost, but I doubt if most of the applications you mention would use that, being
good little UNIX programs at heart.

    -- chris

"Udo Schneider" <[hidden email]> wrote in message news:<[hidden email]>...
> Peter,
>
> take a look at Bob Jarvis' ExternalProccess package (available here:
> http://www.nls.net/mp/jarvis/Bob/DolphinGoodies.htm).
>
> It should address the issues you are having.
>

Many thanks to Udo (and also to Chris Uppal who gave the same link). I
have now downloaded Bob Jarvis's Goodies, and it looks as though the
ExternalProcess class will do just what I want. There are a few
formerly loose ends (in my mind anyway – they may always have been
obvious to others) which I think I can now tie up:

a. I was on the right track in looking for a wrapper for
KernelLibrary>>createProcess:…, because that is exactly what
ExternalProcess is.
b. I can solve the problem of knowing when the external process has
terminated, either by using ExternalProcess>>execSync or by starting
asynchronously and testing regularly to see if the process is alive.
In the former case, if I fork off the external process launch, I
presume it is only the forked process that is suspended until the
external program completes.
c. The ability to redirect the standard output to a file is very
useful, because the X-12 program often reports problems to the console
output. If as Udo says this also stops the system opening a command
line window, so much the better.

As a bonus, Bob's goodies contain two or three other items which could
be useful in my later work, e.g. the Linear Algebra package. So a very
successful first use of this group as a source of help; you may well
hear from me again!

Thanks to all.

Peter

Just one minor mystery about using ExternalProcess - I can't find any
way of making the 'directory' parameter have any effect. I assumed
that all addresses in the command line would be evaluated relative to
the stated directory, but it just doesn't happen. I have to put full
file paths for all the command line arguments.

This is no big deal for my program, and the command lines are
invisible to users so they will not be confused, but it's just a
puzzle. Does anyone have any ideas?

(I have found, by the way, that stdoutFilemame etc. are relative to
the home directory of the Smalltalk application that launches the
external process, which I suppose is logical, but seems to add further
confusion.)

Peter

Just one minor mystery about using ExternalProcess - I can't find any
way of making the 'directory' parameter have any effect. I assumed
that all addresses in the command line would be evaluated relative to
the stated directory, but it just doesn't happen. I have to put full
file paths for all the command line arguments.

This is no big deal for my program, and the command lines are
invisible to users so they will not be confused, but it's just a
puzzle. Does anyone have any ideas?

(I have found, by the way, that stdoutFilemame etc. are relative to
the home directory of the Smalltalk application that launches the
external process, which I suppose is logical, but seems to add further
confusion.)

Peter

Kenny,

the directory you are providing is used to set the current working directory
of the executable your executable.

You still have to provide an absolute path to your executble ..... but if
you are providing additional files to your
executable or if the executbale opens files on it's own you can use relative
paths. These relative paths will then be
resolved relative to the current working directory.

CU,

Udo

"Peter Kenny" <[hidden email]> schrieb im Newsbeitrag
news:[hidden email]...

"Udo Schneider" <[hidden email]> wrote in message
news:40d0b341$[hidden email]...
>
> the directory you are providing is used to set the current working
directory
> of the executable your executable.
>
> You still have to provide an absolute path to your executble ..... but if
> you are providing additional files to your
> executable or if the executbale opens files on it's own you can use
relative
> paths. These relative paths will then be
> resolved relative to the current working directory.

Thanks Udo - I had not tried enough combinations of cases. As you say, the
first item on the command line (i.e. the address of the executable) has to
be a full path, but the other arguments can be relative to the specified
directory. I just tried it, and it works.

Peter

PS Apologies to all for the duplicate posting of my last message - some
unexplained finger trouble.

Bill Schwab <[hidden email]> wrote in message news:<cank65$guc$[hidden email]>...
> You might have the answer to my one problem: I'm
> having diffulty capturing output from compilers.  For example, it would
> be nice to capture all of the output from InnoSetups's compiler, and (no
> doubt over time) teach my code what constitutes a problem.  Likewise for
> LaTeX.  Eventually, it might be nice to drive MinGW and/or MSVC to build
>   projects (especially DLLs) as part of automated builds.  For now, I'd
> settle for Inno's output :)
>
> Do you have any examples of how you use ExternalProcess for such purposes?

Thanks for the kind words, Bill.

As far as capturing stdout and stderr, you *should* just need to
provide the filenames you want and it should work fine.  I use this
regularly in my RcsSourceManager package.  Take a look at
RcsSourceManager>>basicExecuteCommand:inDirectory:waitForSeconds: for
an example.  (If you decide to load the RcsSourceManager package
please use an image you don't care about - unless you have RCS
installed on your machine somewhere I fear it might cause problems.
Better safe than sorry...).  Something like

        tmpStdinName := File temporaryFilename.
        tmpStdoutName := File temporaryFilename.
        tmpStderrName := File temporaryFilename.

        process := ExternalProcess new
                                commandLine: aStringCommand;
                                directory: aDirectoryString;
                                secondsToWait: anIntegerSeconds;
                                stdinFilename: tmpStdinName;
                                stdoutFilename: tmpStdoutName;
                                stderrFilename: tmpStderrName;
                                yourself.

        process executeSync

should work fine (in fact, the above is the exact code used in the
RcsSourceManager method I mentioned earlier).

Looking at the ExternalProcess code it appears that you should really
provide an absolute path with your stdin, stdout, and stderr filenames
- otherwise these files will be created or looked for relative to the
directory where your Dolphin EXE has its current working directory,
*not* relative to the 'directory' arg to ExternalProcess.  The
'directory' arg to ExternalProcess only controls the CWD of the
started process - it isn't used for anything else.  ExternalProcess
makes no assumptions about where your files are located, but that
means you have to specify everything with an absolute path.

For a test you might try running your InnoSetup compiler in a command
prompt window, using command-line redirection to send its output to a
file.  If that works then I'll hope it's just a matter of getting the
filenames set up properly.  The only other thing I can think of that
could get in the way would be if for some reason files that are
created under your account can't be inherited by spawned processes -
I'm pretty weak on Windows security stuff but I think there might be
some way to restrict this.  Just thinking aloud, sort of...

Anyways, let me know if you have further problems with ExternalProcess
and I'll be happy to help as best I can.

Bob

[hidden email] (Peter Kenny) wrote in message news:<[hidden email]>...
I agree this is perhaps a bit inconvenient but it's intentional.  The
'directory' ivar/accessors specify the intial working directory of the
started process - that's it.  It's not used anywhere else.  So for now
you should supply absolute paths with your stdin/out/err filenames.

It looks like some additional class and package comments are in order
here.  I'll get on this as soon as I can.

"Bob Jarvis" <[hidden email]> wrote in message
news:[hidden email]...
> [hidden email] (Peter Kenny) wrote in message
news:<[hidden email]>...
Bob

My question came from confusion in my mind, partly due to lack of experience
in this field. Thanks to Udo Schneider's explanation in this thread, I now
have it clear (I hope). But there are apparently others with similar
problems, if I understand correctly a query from Terry Arnold to this group
on June 16. The situation can be complicated with many directories and paths
involved, so I have worked out the following rather long-winded explanation
to try to cover all cases. If it is of any use to you for your modified
comments, please help yourself.

There may be four directories involved in spawning an External Process (any
two or more could be the same, but for generality let them be different):

a. The current working directory (CWD) of the Smalltalk application which
spawns the process.
b. The directory containing the executable which is to be spawned.
c. The CWD of the spawned External Process.
d. The directory in which the redirected standard outputof the External
Process is to be stored.

The first of these is given, of course. The others are specified, if
required, as follows:

1. The first item on the command line specifies b., either absolutely or
relative to a.
2. The 'directory' argument specifies c., either absolutely or relative to
a. If the 'directory' argument is not used, c. is the same as a.
3. The 'stdoutFilename' parameter specifies d., either absolutely or
relative to a. (and similarly for stdin and stderr).
4. Any files referenced by the External Process which are passed as command
line arguments are specified either absolutely or relative to c.

(In Terry Arnold's case, he was spawning by using CRTLibrary>>_spawnvp:...,
which does not have a 'directory' parameter, so he wanted to solve the
problem by making a. the same as the desired c.)

Many thanks for providing such an excellent Goodies package. My only problem
is that I have spent too much time browsing around to see what all the
things do!

Peter Kenny

"Peter Kenny" <[hidden email]> wrote in message news:<[hidden email]>...
> The situation can be complicated with many directories and paths
> involved, so I have worked out the following rather long-winded explanation
> to try to cover all cases. If it is of any use to you for your modified
> comments, please help yourself.

<excellent explanation snipped>

Thanks for the explanation of how the various directories interact
with one another.  I'll probably use this verbatim.

> Many thanks for providing such an excellent Goodies package. My only problem
> is that I have spent too much time browsing around to see what all the
> things do!

You're quite welcome.  I'm glad you and others are finding it useful.
If you haven't already discovered them you should take a look at Ian
Bartholomew's goodies, located at

        http://www.idb.me.uk/

His ChunkBrowser package can be a real life saver.

Bob

Bob,

> Thanks for the kind words, Bill.

Thanks for the great tool!


> As far as capturing stdout and stderr, you *should* just need to
> provide the filenames you want and it should work fine.  I use this
> regularly in my RcsSourceManager package.  Take a look at
> RcsSourceManager>>basicExecuteCommand:inDirectory:waitForSeconds: for
> an example.  (If you decide to load the RcsSourceManager package
> please use an image you don't care about - unless you have RCS
> installed on your machine somewhere I fear it might cause problems.
> Better safe than sorry...).  Something like

Thanks to you and others offering replies.  It's time for me to take
another look at this.  The caveats about the file names might be
relevant, as might questions about the command interpreter.  It is also
possible that I was imagining the whole thing<g>, but I recall being
very surprised at the output I was not seeing.



Below, I've included what I _think_ is something I assembled based on
your other exmaples.  I am uncertain, because it is packaged with
ExternalProcess, not separately as I'd expect if I had written it.  If
it is mine, feel free to ignore it, or to use it as-is or modified.
Perhaps a better approach would be accept a monadic valuable for each of
the output streams.  It would be more general, and the behavior below
could be written in terms of it.


> For a test you might try running your InnoSetup compiler in a command
> prompt window, using command-line redirection to send its output to a
> file.  If that works then I'll hope it's just a matter of getting the
> filenames set up properly.  

That's very possible, and your proposed test is a good one.  I assume I
did run InnoSetup that way, but maybe not.  Maybe there is a special
switch to get it to dump everything to the standard streams???


 > The only other thing I can think of that
> could get in the way would be if for some reason files that are
> created under your account can't be inherited by spawned processes -
> I'm pretty weak on Windows security stuff but I think there might be
> some way to restrict this.  Just thinking aloud, sort of...

That's a good question too.  Given the number of directories on my
machines that Windows thinks are "invalid" (seems to be precisely those
created by my code and/or unzip utilities), it's worth a look.

Have a good one,

Bill


----------------------------

!ExternalProcess class methodsFor!

executeCommand:aStringCommand inDirectory:aDirectoryString
waitForSeconds:anIntegerSeconds
        "Execute a command by launching a separate process."

                | tmpStdinName tmpStdoutName tmpStderrName tmpFile text process
startTemps endTemps diffTemps |

        startTemps := (File find: aDirectoryString, '\_*') collect: [
:aWin32FindData | aWin32FindData fileName ].

        text := ''.

        tmpStdinName := File temporaryFilename.
        tmpStdoutName := File temporaryFilename.
        tmpStderrName := File temporaryFilename.

        process := ExternalProcess new
                                commandLine: aStringCommand;
                                directory: aDirectoryString;
                                secondsToWait: anIntegerSeconds;
                                stdinFilename: tmpStdinName;
                                stdoutFilename: tmpStdoutName;
                                stderrFilename: tmpStderrName;
                                yourself.

        [ process executeSync ]
                on: ExternalProcessWaitFailure do: [ :e | text := 'Unable to wait for
the command to complete' ]
                on: ExternalProcessWaitTimeout do: [ :e | text := 'The command issued
did not terminate.  This may ',
                                                                        'be because of a problem with the read-only attributes ',
                                                                        'on the files involved.' ]
                on: Error do: [ :e | text := 'An ', e class name, ' error occurred,
error text=', e description ].


        "
        tmpFile := FileStream read:tmpStderrName.
        text := text, tmpFile contents.
        tmpFile close."
        ( Array with:tmpStdoutName with:tmpStderrName ) do:[ :fileName |
                tmpFile := FileStream read:tmpStderrName.
                text := text, tmpFile contents.
                tmpFile close.
        ].

        File delete: tmpStdinName.
        File delete: tmpStdoutName.
        File delete: tmpStderrName.

        endTemps := (File find: aDirectoryString, '\_*') collect: [
:aWin32FindData | aWin32FindData fileName ].
        diffTemps := endTemps difference: startTemps.
        diffTemps do: [ :aFilename | File delete: aFilename ].

        ^text
! !
!ExternalProcess class categoriesFor:
#executeCommand:inDirectory:waitForSeconds:!public! !



--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Hello all,

Bob emailed me with a one-line change that appears to be the answer to
my problem.  While slightly embarrassing (all the more so because I
remember looking for just such a mistake), it probably underscores the
value in a one-time solution to the details of managing the temp files.
   Again, feel free to use this as-is, or modified as you see fit.

Any comments on the stdout/stderr block idea?

Thanks, Bob!

Bill


--------------

!ExternalProcess class methodsFor!

executeCommand:aStringCommand inDirectory:aDirectoryString
waitForSeconds:anIntegerSeconds
        "Execute a command by launching a separate process."

                | tmpStdinName tmpStdoutName tmpStderrName tmpFile text process
startTemps endTemps diffTemps |

        startTemps := (File find: aDirectoryString, '\_*') collect: [
:aWin32FindData | aWin32FindData fileName ].

        text := ''.

        tmpStdinName := File temporaryFilename.
        tmpStdoutName := File temporaryFilename.
        tmpStderrName := File temporaryFilename.

        process := ExternalProcess new
                                commandLine: aStringCommand;
                                directory: aDirectoryString;
                                secondsToWait: anIntegerSeconds;
                                stdinFilename: tmpStdinName;
                                stdoutFilename: tmpStdoutName;
                                stderrFilename: tmpStderrName;
                                yourself.

        [ process executeSync ]
                on: ExternalProcessWaitFailure do: [ :e | text := 'Unable to wait for
the command to complete' ]
                on: ExternalProcessWaitTimeout do: [ :e | text := 'The command issued
did not terminate.  This may ',
                                                                        'be because of a problem with the read-only attributes ',
                                                                        'on the files involved.' ]
                on: Error do: [ :e | text := 'An ', e class name, ' error occurred,
error text=', e description ].


        "
        tmpFile := FileStream read:tmpStderrName.
        text := text, tmpFile contents.
        tmpFile close."
        ( Array with:tmpStdoutName with:tmpStderrName ) do:[ :fileName |
                "6-04 - thanks to Bob for spotting this
                tmpFile := FileStream read:tmpStderrName."
                tmpFile := FileStream read:fileName.
                text := text, tmpFile contents.
                tmpFile close.
        ].

        File delete: tmpStdinName.
        File delete: tmpStdoutName.
        File delete: tmpStderrName.

        endTemps := (File find: aDirectoryString, '\_*') collect: [
:aWin32FindData | aWin32FindData fileName ].
        diffTemps := endTemps difference: startTemps.
        diffTemps do: [ :aFilename | File delete: aFilename ].

        ^text
! !
!ExternalProcess class categoriesFor:
#executeCommand:inDirectory:waitForSeconds:!public! !




--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Bill Schwab <[hidden email]> wrote in message news:<cba7da$1768$[hidden email]>...
> Bob emailed me with a one-line change that appears to be the answer to
> my problem.

Hey, that's happened to all of us, and it'll happen again.  :-)

> Any comments on the stdout/stderr block idea?

I guess I'm either too tired or too dense, but I don't understand how
this would work.  Could you give a bit more explanation of what you're
thinking of?  Code fragments of how you'd expect to use it would be a
big help.  Thanks for helping to improve things.

Bob Jarvis