OSProcess - can it pipe 1 MB?

> Dave,
>
> Sure, but then the command shell must then be doing something that OSProcess should do: transfer "large" (1MB isn't much these days) blobs in pieces that are manageable.  The questions are where does it belong, and what if anything is necessary to ensure the pipe is ready for the next piece?
>
> One thought would be
>
> AttachableFileStream>>nextPutAll:blob
> | in |
> in := ReadStream on:blob.
> [ in atEnd ] whileFalse:[
>      self nextPutAll:( in next:32000).
> ].
>
> but it should run into the same problem after a few times through the loop.  It needs some "flow control" and could easily belong elsewhere in the hierarch??  The above assumes the (should be changed) silent truncation of #next:.
>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] On Behalf Of David T. Lewis [[hidden email]]
> Sent: Sunday, August 29, 2010 12:37 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] OSProcess - can it pipe 1 MB?
>
> On Sun, Aug 29, 2010 at 11:36:02AM -0400, Schwab,Wilhelm K wrote:
> > I am trying to pipe roughly 1 MB of data into a program on Linux; I know it can handle the load, because I have tested it with by cat-ing a script and the data.  I do not yet promise that my Pharo code is correct; if the program did not respond or returned junk, I would look to myself.  But I am getting errors suggesting that AttachableFileStream is failing on writing to the pipe, and it's very suspicious that it simply can't handle the size of the string.  Any ideas?
> >
> > Bill
>
> Try googling "maximum size of write to linux pipe". Unix pipes
> have a capacity limit, and the write will fail if you try to write
> too large a chunk all at once.
>
> Dave
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

> Dave,
>
> Sure, but then the command shell must then be doing something that OSProcess should do: transfer "large" (1MB isn't much these days) blobs in pieces that are manageable.  The questions are where does it belong, and what if anything is necessary to ensure the pipe is ready for the next piece?
>
> One thought would be
>
> AttachableFileStream>>nextPutAll:blob
> | in |
> in := ReadStream on:blob.
> [ in atEnd ] whileFalse:[
>      self nextPutAll:( in next:32000).
> ].
>
> but it should run into the same problem after a few times through the loop.  It needs some "flow control" and could easily belong elsewhere in the hierarch??  The above assumes the (should be changed) silent truncation of #next:.
>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] On Behalf Of David T. Lewis [[hidden email]]
> Sent: Sunday, August 29, 2010 12:37 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] OSProcess - can it pipe 1 MB?
>
> On Sun, Aug 29, 2010 at 11:36:02AM -0400, Schwab,Wilhelm K wrote:
> > I am trying to pipe roughly 1 MB of data into a program on Linux; I know it can handle the load, because I have tested it with by cat-ing a script and the data.  I do not yet promise that my Pharo code is correct; if the program did not respond or returned junk, I would look to myself.  But I am getting errors suggesting that AttachableFileStream is failing on writing to the pipe, and it's very suspicious that it simply can't handle the size of the string.  Any ideas?
> >
> > Bill
>
> Try googling "maximum size of write to linux pipe". Unix pipes
> have a capacity limit, and the write will fail if you try to write
> too large a chunk all at once.
>
> Dave
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

> Dave,
>
> Why does
>
> primWrite: id from: stringOrByteArray startingAt: startIndex count: count
> "Write count bytes onto this file from the given string or byte array starting at the given index. Answer the number of bytes written."
>
> raise an error?  Perhaps it or a related method should answer the number of bytes written.  There should be a compromise between locking up the VM and having no idea how quickly the other side can accept data.  Given a count of what was written (as suggested in the comment, and perhaps zero bytes if the other side is struggling), then one could design a reasonable loop with a delay to provide the needed flow control.  Otherwise, it's just guesswork.
>
> Bill
>
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] On Behalf Of David T. Lewis [[hidden email]]
> Sent: Sunday, August 29, 2010 3:05 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] OSProcess - can it pipe 1 MB?
>
> It does not belong in OSProcess unless it's something that the underlying
> operating system actually supports, which is not the case here. As far
> as I know, there is no Unix system call for determining how much data a
> pipe is ready to accept.
>
> You'll need to manage your flow control yourself. You may also need to
> call #setNonBlocking on the pipe writer, otherwise you can block on
> write and lock up the virtual machine(I think you know that already but
> just in case). If the process at the other end of your pipe is keeping
> up with the data, it won't be a problem, but if you try to write to a
> full pipe without pulling something out of the other end, you will lock
> up your VM completely unless the stream has been set to non-blocking
> mode.
>
> Dave
>
> On Sun, Aug 29, 2010 at 02:28:50PM -0400, Schwab,Wilhelm K wrote:
> > Dave,
> >
> > Sure, but then the command shell must then be doing something that OSProcess should do: transfer "large" (1MB isn't much these days) blobs in pieces that are manageable.  The questions are where does it belong, and what if anything is necessary to ensure the pipe is ready for the next piece?
> >
> > One thought would be
> >
> > AttachableFileStream>>nextPutAll:blob
> > | in |
> > in := ReadStream on:blob.
> > [ in atEnd ] whileFalse:[
> >      self nextPutAll:( in next:32000).
> > ].
> >
> > but it should run into the same problem after a few times through the loop.  It needs some "flow control" and could easily belong elsewhere in the hierarch??  The above assumes the (should be changed) silent truncation of #next:.
> >
> > Bill
> >
> >
> > ________________________________________
> > From: [hidden email] [[hidden email]] On Behalf Of David T. Lewis [[hidden email]]
> > Sent: Sunday, August 29, 2010 12:37 PM
> > To: [hidden email]
> > Subject: Re: [Pharo-project] OSProcess - can it pipe 1 MB?
> >
> > On Sun, Aug 29, 2010 at 11:36:02AM -0400, Schwab,Wilhelm K wrote:
> > > I am trying to pipe roughly 1 MB of data into a program on Linux; I know it can handle the load, because I have tested it with by cat-ing a script and the data.  I do not yet promise that my Pharo code is correct; if the program did not respond or returned junk, I would look to myself.  But I am getting errors suggesting that AttachableFileStream is failing on writing to the pipe, and it's very suspicious that it simply can't handle the size of the string.  Any ideas?
> > >
> > > Bill
> >
> > Try googling "maximum size of write to linux pipe". Unix pipes
> > have a capacity limit, and the write will fail if you try to write
> > too large a chunk all at once.
> >
> > Dave
> >
> >
> > _______________________________________________
> > Pharo-project mailing list
> > [hidden email]
> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
> >
> > _______________________________________________
> > Pharo-project mailing list
> > [hidden email]
> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project