OSProcess - working on gnuplot

Dave,

I am slowly gathering ideas on how to write an interface to gnuplot.  It will probably be enough to use PipeableOSPRocess class>>command: to grab output from it, but it will be inefficient for producing large numbers of graphs.

Ideally, I would like to run the program and obtain output without creating so much as a named pipe; it appears to be possible to route output to standard out, so there is a potential for it to work.  For a single graph, think I see how to make that go.

For large numbers of graphs, one could allow gnuplot to write the files; that might even be preferred in many situations, but right now I appear to be designing that way because I don't know another way - I'm trying to leave Windows behind me :)

Is there a clean way to start gnuplot, leave it running, write to its stdin, read from its stdout almost as if it were a TCP server, and then close it when I'm ready to do so? I tried running CommandShell thinking that might let me demo it, but appears to be stuck back in MVC??

Bill



_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi,

You don't have to reinvent the wheel all the time:
http://www.squeaksource.com/GNUPlot.html

Cheers,
Levente

On Sun, 15 Nov 2009, Schwab,Wilhelm K wrote:

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

It makes explicit reference to win32, so I excluded it on that basis.  Are you saying it runs elsewhere?



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Levente Uzonyi
Sent: Sunday, November 15, 2009 3:26 PM
To: [hidden email]
Subject: Re: [Pharo-project] OSProcess - working on gnuplot

Hi,

You don't have to reinvent the wheel all the time:
http://www.squeaksource.com/GNUPlot.html

Cheers,
Levente

On Sun, 15 Nov 2009, Schwab,Wilhelm K wrote:

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

It runs on all platforms, including win32.

Levente

On Sun, 15 Nov 2009, Schwab,Wilhelm K wrote:

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Sun, Nov 15, 2009 at 03:04:49PM -0500, Schwab,Wilhelm K wrote:
Hi Bill,

You should probably first check out the suggestion from Levente:

> You don't have to reinvent the wheel all the time:
> http://www.squeaksource.com/GNUPlot.html

But your question is interesting to me, because the first free software
thing I ever wrote was gl_plot, which appeared in comp.sources.unix volume
18, and was used for generating graphics to support gnuplot on PC unix
(Microport System V/AT) back before PC unix had graphics support. It's
still out there in case you want to run gnuplot on your 286 unix pc:

  http://ftp.isc.org/pub/usenet/comp.sources.unix/volume18/gl_plot

Anyway, back to the new millenium. Yes you should definitely be able to
control a gnuplot process with a connected PipeableOSProcess. That's
what it is designed for.

> Is there a clean way to start gnuplot, leave it running, write to its
> stdin, read from its stdout almost as if it were a TCP server, and then
> close it when I'm ready to do so?

Sure, see PipeableOSProcess class>>tkExample for an example of how to do
this. Use /usr/bin/gnuplot instead of /usr/bin/wish of course. However
(and this is a big caveat at the moment), OSProcess and CommandShell
currently do not work properly on Pharo for reasons that I don't yet
understand. So definitely check out Levente's suggestion as a first step.

> I tried running CommandShell thinking
> that might let me demo it, but appears to be stuck back in MVC??

CommandShell has user interfaces for both Morphic and MVC. Pharo does
not have MVC, but that should not effect the Morphic side, other than
the annoyance of having to skip over the errors when you load it.

Dave


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Dave,

ProcessWrapper looks like a win32-only project to me, and Sig's gnuplot package appears to use OSProcess for unix.   Please let me know if I am missing something.

It is true that OSProcess' unit tests cause Pharo to experience an ugly death, but a lot seems to work.  I am having some success with code such as

 gp := PipeableOSProcess command:'gnuplot'.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)'.

but I cannot get any output without shutting down the process.  Are any of the problems you describe relevant to that, or am I asking too much, or perhaps just going about this incorrectly?

Bill



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of David T. Lewis
Sent: Sunday, November 15, 2009 6:57 PM
To: [hidden email]
Subject: Re: [Pharo-project] OSProcess - working on gnuplot

On Sun, Nov 15, 2009 at 03:04:49PM -0500, Schwab,Wilhelm K wrote:
Hi Bill,

You should probably first check out the suggestion from Levente:

> You don't have to reinvent the wheel all the time:
> http://www.squeaksource.com/GNUPlot.html

But your question is interesting to me, because the first free software thing I ever wrote was gl_plot, which appeared in comp.sources.unix volume 18, and was used for generating graphics to support gnuplot on PC unix (Microport System V/AT) back before PC unix had graphics support. It's still out there in case you want to run gnuplot on your 286 unix pc:

  http://ftp.isc.org/pub/usenet/comp.sources.unix/volume18/gl_plot

Anyway, back to the new millenium. Yes you should definitely be able to control a gnuplot process with a connected PipeableOSProcess. That's what it is designed for.

> Is there a clean way to start gnuplot, leave it running, write to its
> stdin, read from its stdout almost as if it were a TCP server, and
> then close it when I'm ready to do so?

Sure, see PipeableOSProcess class>>tkExample for an example of how to do this. Use /usr/bin/gnuplot instead of /usr/bin/wish of course. However (and this is a big caveat at the moment), OSProcess and CommandShell currently do not work properly on Pharo for reasons that I don't yet understand. So definitely check out Levente's suggestion as a first step.

> I tried running CommandShell thinking
> that might let me demo it, but appears to be stuck back in MVC??

CommandShell has user interfaces for both Morphic and MVC. Pharo does not have MVC, but that should not effect the Morphic side, other than the annoyance of having to skip over the errors when you load it.

Dave


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi!

On Mon, 16 Nov 2009, Schwab,Wilhelm K wrote:

> Dave,
>
> ProcessWrapper looks like a win32-only project to me, and Sig's gnuplot package appears to use OSProcess for unix.   Please let me know if I am missing something.
>

Though I'm not Dave, I can tell you that ProcessWrapper is for win32 only.
It exists because PipeableOSProcess doesn't work under win32.

> It is true that OSProcess' unit tests cause Pharo to experience an ugly death, but a lot seems to work.  I am having some success with code such as
>
> gp := PipeableOSProcess command:'gnuplot'.
> gp
> exec:'set output ""';
> exec:'set terminal png';
> exec:'plot sin(x)'.
>

You still don't have to reinvent the wheel. If you don't want to use
Igor's project - which you should IMO - you can still check out his code
and see how he uses PipeableOSProcess to do the thing you want to do.

Levente

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Mon, Nov 16, 2009 at 05:36:08PM -0500, Schwab,Wilhelm K wrote:
You may be hanging up trying to read a blocking stream from the gnuplot program.
Try using #setNonBlockingOutput to prevent this. The following three examples
all work for me (but I ran this on a Squeak image).

Your original example, but with #setNonBlockingOutput:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)'.
 gp close. "close stdin to gnuplot, like <ctl>D"
 gp upToEndOfFile inspect. "stdout, the graphics output"
 gp errorUpToEndOfFile inspect. "stderr output if any"

If you want to keep gnuplot running in background, you will have to read
data as it becomes available using #upToEnd rather than #upToEndOfFile:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)'.
 (Delay forMilliseconds: 1000) wait. "allow gnuplot to send data"
 gp upToEnd inspect. "all available stdout data"
 gp errorUpToEnd inspect.  "all available stderr data"
 gp close.

You can cause the background gnuplot interpreter to exit by sending it
the "exit" command, or by doing "gp close" as in the examples above:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)';
        exec: 'exit'. "exit gnuplot, similar effect to gp close"
 gp upToEndOfFile inspect.
 gp errorUpToEndOfFile inspect.

If the above examples do not work for you, it is very likely due to the
current inconsistencies between OSProcess and Pharo, for which I do not yet
have a solution (I would welcome any tips here, I tried again last night to
figure out what's wrong and gave up in frustration :-/ ).

HTH,
Dave


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Dave,

Thanks for the examples, and the reminder about Squeak/Pharo differences.  I'll see what I can figure out.  You might have already explained why one of my tests failed.   I do want the option to keep gnuplot running.  It might turn out that the cases in which that matters are precisely the ones in which gnuplot should be allowed to write the output files, but I'd rather not start out with that approach.

Re the problems in Pharo, one perhaps completely worthless point: I ran some of the unit tests with Pharo launched from a terminal, and got Ipv6 related messages.  It made no sense to me, but (I think) that code is whacked, and if there is any relationship, it *might* be the problem.  Usual caveats about this being a potential red herring apply.

Bill



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of David T. Lewis
Sent: Monday, November 16, 2009 7:40 PM
To: [hidden email]
Subject: Re: [Pharo-project] OSProcess - working on gnuplot

On Mon, Nov 16, 2009 at 05:36:08PM -0500, Schwab,Wilhelm K wrote:
You may be hanging up trying to read a blocking stream from the gnuplot program.
Try using #setNonBlockingOutput to prevent this. The following three examples all work for me (but I ran this on a Squeak image).

Your original example, but with #setNonBlockingOutput:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)'.
 gp close. "close stdin to gnuplot, like <ctl>D"
 gp upToEndOfFile inspect. "stdout, the graphics output"
 gp errorUpToEndOfFile inspect. "stderr output if any"

If you want to keep gnuplot running in background, you will have to read data as it becomes available using #upToEnd rather than #upToEndOfFile:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)'.
 (Delay forMilliseconds: 1000) wait. "allow gnuplot to send data"
 gp upToEnd inspect. "all available stdout data"
 gp errorUpToEnd inspect.  "all available stderr data"
 gp close.

You can cause the background gnuplot interpreter to exit by sending it the "exit" command, or by doing "gp close" as in the examples above:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)';
        exec: 'exit'. "exit gnuplot, similar effect to gp close"
 gp upToEndOfFile inspect.
 gp errorUpToEndOfFile inspect.

If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ).

HTH,
Dave


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Dave,

That works - thanks!  Knowing how much data to read will be a problem.  My assumption was that I would have an image reader pull from the stream, taking the hint when it finds a complete image.  Sadly, they do bad things to the stream, and I am not sure how to correct that.

Bill



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Schwab,Wilhelm K
Sent: Monday, November 16, 2009 7:59 PM
To: [hidden email]
Subject: Re: [Pharo-project] OSProcess - working on gnuplot

Dave,

Thanks for the examples, and the reminder about Squeak/Pharo differences.  I'll see what I can figure out.  You might have already explained why one of my tests failed.   I do want the option to keep gnuplot running.  It might turn out that the cases in which that matters are precisely the ones in which gnuplot should be allowed to write the output files, but I'd rather not start out with that approach.

Re the problems in Pharo, one perhaps completely worthless point: I ran some of the unit tests with Pharo launched from a terminal, and got Ipv6 related messages.  It made no sense to me, but (I think) that code is whacked, and if there is any relationship, it *might* be the problem.  Usual caveats about this being a potential red herring apply.

Bill



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of David T. Lewis
Sent: Monday, November 16, 2009 7:40 PM
To: [hidden email]
Subject: Re: [Pharo-project] OSProcess - working on gnuplot

On Mon, Nov 16, 2009 at 05:36:08PM -0500, Schwab,Wilhelm K wrote:
You may be hanging up trying to read a blocking stream from the gnuplot program.
Try using #setNonBlockingOutput to prevent this. The following three examples all work for me (but I ran this on a Squeak image).

Your original example, but with #setNonBlockingOutput:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)'.
 gp close. "close stdin to gnuplot, like <ctl>D"
 gp upToEndOfFile inspect. "stdout, the graphics output"
 gp errorUpToEndOfFile inspect. "stderr output if any"

If you want to keep gnuplot running in background, you will have to read data as it becomes available using #upToEnd rather than #upToEndOfFile:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)'.
 (Delay forMilliseconds: 1000) wait. "allow gnuplot to send data"
 gp upToEnd inspect. "all available stdout data"
 gp errorUpToEnd inspect.  "all available stderr data"
 gp close.

You can cause the background gnuplot interpreter to exit by sending it the "exit" command, or by doing "gp close" as in the examples above:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)';
        exec: 'exit'. "exit gnuplot, similar effect to gp close"
 gp upToEndOfFile inspect.
 gp errorUpToEndOfFile inspect.

If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ).

HTH,
Dave


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi Dave,

On Nov 17, 2009, at 01:40 , David T. Lewis wrote:

[...]

> If the above examples do not work for you, it is very likely due to  
> the
> current inconsistencies between OSProcess and Pharo, for which I do  
> not yet
> have a solution (I would welcome any tips here, I tried again last  
> night to
> figure out what's wrong and gave up in frustration :-/ ).


What is the simplest thing to do to reproduce the problem?

Do we know if it is on all platforms? Does it work in a Squeak Trunk  
image?

Maybe I or other people can help to get more insights about what goes  
wrong. OSProcess is certainly a very important package for Pharo, so  
we should try to get it working again.

Thanks,
Adrian
___________________
http://www.adrian-lienhard.ch/


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

2 Bill & others.

I started gnuplot to make an abstract layer for its functionality.
Its not really matter, how it invokes the OSProcess, so feel free to
replace/improve that part of code.

And of course, its not for a single OS. There are GPWrapper which
should deal with platform-specific crap,
while rest of API should simply reflect the gnuplot api in convenient way.

It was tested & run on windoze & unix(es).
Potentially, it could be split on 2 packages (feel free to do it):
  - basic classes
  - platform-specific / OS-specific stuff

Also, if i remember, the API is not fully covered, so feel free to add
new protocol(s).
Repository is writable, so do not hesitate to put your hands on it.


--
Best regards,
Igor Stasenko AKA sig.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Yes!
What are the problems?
I read the mail three times but I could not get it.

Stef

On Nov 17, 2009, at 9:30 AM, Adrian Lienhard wrote:


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Stef,

The only problems I have found are in the unit tests, which somewhere along the line cause a vm crash.  Otherwise, it seems to work.  Until it either proves unstable or otherwise lets me down, I'm trying to press forward with it, since I agree that it is REALLY important that this and other other FFI-like features work and work well.

In general, I find the Linux vm to be too willing to crash w/o leaving a trace, or at least a trace that I know how to find(??).

Bill



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Stéphane Ducasse
Sent: Tuesday, November 17, 2009 8:02 AM
To: [hidden email]
Subject: Re: [Pharo-project] OSProcess - working on gnuplot

Yes!
What are the problems?
I read the mail three times but I could not get it.

Stef

On Nov 17, 2009, at 9:30 AM, Adrian Lienhard wrote:


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Tue, Nov 17, 2009 at 09:30:16AM +0100, Adrian Lienhard wrote: Hi Adrian,

There are unit tests for OSProcess as well as for CommandShell, and these
should all be green. Yes, all tests pass on Squeak trunk (see attached).

I get inconsistent results on a Pharo image. I just tried again tonight
on pharo1.0-10495-rc1dev09.11.3, and things look better than the last
time I tried (which might be coincidental, I'm running various VMs and
maybe I got myself confused the other evening).

I'm not sure, but I think I am seeing two types of problem. The easy
one is that I'm getting test failures in AioEventHandlerTestCase that
are probably caused by changes in the socket code for Pharo compared to
Squeak. I have not found the cause yet, but I'm not worried about it
because it's probably just a change in the socket code for Pharo, and
I probably will just need to adjust the tests for this.

The harder problem is that I am getting intermittent failures, most
often showing up in CommandShellTestCase (which does a lot of pipes,
event handling, and OS process handling). I think that I am seeing
failures in reading from pipes, which in turn results in failure to
find end of file and clean up exiting processes. This causes the tests
to get "stuck", and if I look in a terminal window, I can see that many
of the processes spawned by the unit tests are in zombie mode, indicating
that the grimReaperProcess running in Pharo/Squeak is failing to
clean up processes that are ready to exit.

I think that when I ran this on earlier Pharo images I was getting
errors that had something to do with the fileID of a file stream
(sorry I did not save the specific errors), so I am wondering if
anything may have changed that would affect access to the fileID
handle for a file stream. I have also seen VM crashes when running
the tests, which suggests possibly passing the wrong kind of object
to a primitive that expects a fileID byte array (and of course it
also means that I need to do a better job of validating parameters
passed to primitives in OSPP, but that's a separate topic ;)

> Maybe I or other people can help to get more insights about what goes  
> wrong. OSProcess is certainly a very important package for Pharo, so  
> we should try to get it working again.

Thanks for offering to help. I suspect that this will turn out to be
one or two simple problems, and I just have not found them yet.

Dave


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Tue, Nov 17, 2009 at 03:20:41PM -0500, Schwab,Wilhelm K wrote:
> Stef,
>
> The only problems I have found are in the unit tests, which somewhere along the line cause a vm crash.  Otherwise, it seems to work.  Until it either proves unstable or otherwise lets me down, I'm trying to press forward with it, since I agree that it is REALLY important that this and other other FFI-like features work and work well.
>
> In general, I find the Linux vm to be too willing to crash w/o leaving a trace, or at least a trace that I know how to find(??).

Hi Bill,

I am not sure what was causing the crashes (I have seen this too, although
not on the latest Pharo that I tried tonight). But I am quite sure that
you should not blame the VM, and you should not blame Pharo either. In
such a case, the fault lies with the person who wrote the plugin that
crashed (that would be me). I expect that the unit tests ended up passing
the "wrong" object as a parameter to some primitive in OSProcessPlugin,
which in turn led to a crash. Most likely the symptoms (VM crash) will
go away whenever we figure out the issues with OSProcess on Pharo, but
only correct solution is to write the primitives with enough parameter
checking that a VM crash cannot happen.

If I see this problem again, I'll try running the VM under a gdb debugger
to find out which primitive is causing the crash. That's probably the
only way to find it, because the VM cannot prevent a sloppy plugin from
doing a segfault and exiting.

Dave


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Dave,

It is nice to see a programmer accept blame for something.  Too many hide behind the  euphemism du jour ("issue") as if they had nothing to do with it.

A long time ago, I read an article that objected to the term "bug," suggesting that it was too easy to pass of as something relatively harmless; the author favored the term "defect."  It is even more applicable now.  If you want to have fun, get across a table with a programmer who measures his/her worth by listing their MS certifications, and insist on using "defect" every time they pass something off as an "issue."  The looks of horror can be quite entertaining.

But I digress.  Ok, you blew it<g>.  Feel better?  You're man enough to take the heat (not that we're going to give you any).  However, the vm is partially at fault, because (AFICT), it fails to trap crashes.

IMHO, the worst thing a program can do is lock up.  The next worst thing it can do is disappear w/o leaving a reason for it.  So, I *do* blame the vm for not doing more to help us find this.

Bill



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of David T. Lewis
Sent: Tuesday, November 17, 2009 11:09 PM
To: [hidden email]
Subject: Re: [Pharo-project] OSProcess - working on gnuplot

On Tue, Nov 17, 2009 at 03:20:41PM -0500, Schwab,Wilhelm K wrote:
> Stef,
>
> The only problems I have found are in the unit tests, which somewhere along the line cause a vm crash.  Otherwise, it seems to work.  Until it either proves unstable or otherwise lets me down, I'm trying to press forward with it, since I agree that it is REALLY important that this and other other FFI-like features work and work well.
>
> In general, I find the Linux vm to be too willing to crash w/o leaving a trace, or at least a trace that I know how to find(??).

Hi Bill,

I am not sure what was causing the crashes (I have seen this too, although not on the latest Pharo that I tried tonight). But I am quite sure that you should not blame the VM, and you should not blame Pharo either. In such a case, the fault lies with the person who wrote the plugin that crashed (that would be me). I expect that the unit tests ended up passing the "wrong" object as a parameter to some primitive in OSProcessPlugin, which in turn led to a crash. Most likely the symptoms (VM crash) will go away whenever we figure out the issues with OSProcess on Pharo, but only correct solution is to write the primitives with enough parameter checking that a VM crash cannot happen.

If I see this problem again, I'll try running the VM under a gdb debugger to find out which primitive is causing the crash. That's probably the only way to find it, because the VM cannot prevent a sloppy plugin from doing a segfault and exiting.

Dave


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

 
The image readers heap a lot of abuse on the streams given to them (reset vs. restoring position, sending #binary, #close, etc.).  With PNG, I was able to create a modified reader that uses the IEND chunk to stop the read rather than the end of stream, which is hideous for some uses (such as mine).  My "fix" could be applied to the existing code.  Any interest?

My wrapper around the stdout pipe is still failing me, but I think I see why.  With a proper page-reading trick, I will hopefully be able to get it to work, which I define here as having the PNG read stop at the end of the data.  Not now though: it's getting too late for this nonsense :)

Bill



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Schwab,Wilhelm K
Sent: Monday, November 16, 2009 9:28 PM
To: [hidden email]
Subject: Re: [Pharo-project] OSProcess - working on gnuplot

Dave,

That works - thanks!  Knowing how much data to read will be a problem.  My assumption was that I would have an image reader pull from the stream, taking the hint when it finds a complete image.  Sadly, they do bad things to the stream, and I am not sure how to correct that.

Bill



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Schwab,Wilhelm K
Sent: Monday, November 16, 2009 7:59 PM
To: [hidden email]
Subject: Re: [Pharo-project] OSProcess - working on gnuplot

Dave,

Thanks for the examples, and the reminder about Squeak/Pharo differences.  I'll see what I can figure out.  You might have already explained why one of my tests failed.   I do want the option to keep gnuplot running.  It might turn out that the cases in which that matters are precisely the ones in which gnuplot should be allowed to write the output files, but I'd rather not start out with that approach.

Re the problems in Pharo, one perhaps completely worthless point: I ran some of the unit tests with Pharo launched from a terminal, and got Ipv6 related messages.  It made no sense to me, but (I think) that code is whacked, and if there is any relationship, it *might* be the problem.  Usual caveats about this being a potential red herring apply.

Bill



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of David T. Lewis
Sent: Monday, November 16, 2009 7:40 PM
To: [hidden email]
Subject: Re: [Pharo-project] OSProcess - working on gnuplot

On Mon, Nov 16, 2009 at 05:36:08PM -0500, Schwab,Wilhelm K wrote:
You may be hanging up trying to read a blocking stream from the gnuplot program.
Try using #setNonBlockingOutput to prevent this. The following three examples all work for me (but I ran this on a Squeak image).

Your original example, but with #setNonBlockingOutput:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)'.
 gp close. "close stdin to gnuplot, like <ctl>D"
 gp upToEndOfFile inspect. "stdout, the graphics output"
 gp errorUpToEndOfFile inspect. "stderr output if any"

If you want to keep gnuplot running in background, you will have to read data as it becomes available using #upToEnd rather than #upToEndOfFile:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)'.
 (Delay forMilliseconds: 1000) wait. "allow gnuplot to send data"
 gp upToEnd inspect. "all available stdout data"
 gp errorUpToEnd inspect.  "all available stderr data"
 gp close.

You can cause the background gnuplot interpreter to exit by sending it the "exit" command, or by doing "gp close" as in the examples above:

 gp := PipeableOSProcess command:'gnuplot'.
 gp setNonBlockingOutput.
 gp
        exec:'set output ""';
        exec:'set terminal png';
        exec:'plot sin(x)';
        exec: 'exit'. "exit gnuplot, similar effect to gp close"
 gp upToEndOfFile inspect.
 gp errorUpToEndOfFile inspect.

If the above examples do not work for you, it is very likely due to the current inconsistencies between OSProcess and Pharo, for which I do not yet have a solution (I would welcome any tips here, I tried again last night to figure out what's wrong and gave up in frustration :-/ ).

HTH,
Dave


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi Dave,

Just to make sure I'm looking at the right code: The latest version  
(for Pharo) are in the projects OSProcess and CommandShell on  
SqueakSource.com? MC complains about missing classes when loading; I  
just ignored them (probably they are not relevant as they seem related  
to the GUI).

Running all tests crashes the Mac VM.

But even the simple test of chdir fails:

(UnixProcessAccessorTestCase selector: #testChDir) debug

It does not seem to invoke any Pharo-specific code. So I wonder if I  
miss something trivial. UnixOSProcessPlugin.bundle exists in my VM's  
Contents/Resources/ directory.

Adrian

On Nov 18, 2009, at 04:55 , David T. Lewis wrote:


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Wed, Nov 18, 2009 at 09:51:24AM +0100, Adrian Lienhard wrote:
> Hi Dave,
>
> Just to make sure I'm looking at the right code: The latest version  
> (for Pharo) are in the projects OSProcess and CommandShell on  
> SqueakSource.com?

Yes, these are the correct versions to use.

> MC complains about missing classes when loading; I  
> just ignored them (probably they are not relevant as they seem related  
> to the GUI).

These are the MVC user interface classes for CommandShell, and it is
safe to ignore them. The tests do not required these classes.

>
> Running all tests crashes the Mac VM.

I have seen crashes on my Linux VM also (testing with an earlier Pharo
image), although it was not happening yesterday when I was testing.
Presumably there is inadequate parameter checking in one or more of
the OSPP primitives, and whatever is going wrong in the tests has
exposed this.

>
> But even the simple test of chdir fails:
>
> (UnixProcessAccessorTestCase selector: #testChDir) debug
>
> It does not seem to invoke any Pharo-specific code. So I wonder if I  
> miss something trivial. UnixOSProcessPlugin.bundle exists in my VM's  
> Contents/Resources/ directory.

I do not have a Mac to test with. I would think that the fact that you
have gotten the VM to crash is an indication that you have located the
plugin bundle though ;) Just to double check, try doing this in a
workspace:

  OSProcess accessor osppModuleVersionString ==> '4.3.3'

The #osppModuleVersionString method will make a call to the plugin
without doing much else, so that is a way to verify that you have
loaded the bundle. Also, if the version is not fairly close to the
latest (4.3.3) it might be an issue.

On OS X, I don't think that you have the AioPlugin or XDisplayControlPlugin
available, so some of the tests will fail as a result of this. I
don't have time right now, but I'll try removing these plugins from
my system and see what happens. But it should be a few simple test
failures, not the kind of problem you are seeing.

(As a side note, I use a Linux PC, and John and I originally got
OSProcess working on Mac entirely through email with a bit of
help from a couple other Mac users. Talk about a lengthy
edit-compile-test-debug development cycle! But it was a good
challenge and quite gratifying when we got it all working.).

Thanks,
Dave


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project