Generators vs Pipes (again)

In response to my post about Pipes being an improvement over generators
Andreas quite rightly pointed out ways in which his idea of generators
is better than
my idea of pipes.  However, I still feel that pipes is very useful
generalization
of generators.

   1)  Andreas ideas of generators is really my idea of generators
(similar to iterators)
combined with the idea of coroutines.  Now I think generators without
the use of
coroutines is still very useful and I will assume  Andreas agrees.
Furthermore, I think coroutines are a great idea that every language
should have, including
Smalltalk.  In particular, as pointed out by Andreas, they are
especially useful in
combination with (my idea of) generators.  However, if Andreas is allowed to use
coroutines with generators then I should be able to use coroutines in
combination
with my pipes idea.
Revisiting my pipes idea three components but now with coroutines
available we have:
  a)  Generators (but now Andreas' idea of generators).
       Note that since my idea now includes Andreas'  it cannot be
argued that his idea
       is more powerful; one can only argue that all the extra
functionality I propose
       is not worth the additional complexity involved.
  b)  Absorbers.  Absorbers are the reverse of generators and they
similarly become
       more powerful if a coroutine capability is available.
  c)  Pipes.  Pipes provide the generator capability and the Absorber capability
       concurrently.  Pipes also become more powerful if a coroutine
capability is
       available but the coroutine capability must be more powerful
because the pipe
       may want to yield to the object feeding it data (when the pipe
is acting like an
       absorber) and may also want to yield to the object it feeds data to
       (when acting like a generator).

Clearly my argument in my original post that pipes are a very useful construct
is even stronger if coroutines are available.

 2)  In the generator software that Andreas is developing,
       I think that Andreas is combining the idea of generator and the
idea of coroutine
       into one package and that this is a mistake!
      The coroutine idea should be its own package
      (if not part of the language definition itself).
      This way coroutines may be used wherever they prove to be useful and not
      just in generators.  In particular, I hope that they are
powerful enough to be
      used in pipes both for input and output at the same time if necessary.
      For me the prototypical example of the use of coroutines is of
four players
      playing a game of cards.  As each player plays a card control is
transferred to the
      next player.  It should be natural to implement this card game
in Smalltalk using
      the coroutine package.  The pipe example though is more complex
and may need
      more thought.

 3) I don't know what a coroutine package should look like exactly. I
only know that
     with a coroutine package:
       a) It should be possible to build Andreas idea of generator, or
my idea of absorber.
       b)  It should be powerful enough to be used for Pipes
            (both input and output concurrently).  I don't believe
Andreas' generator
            package will provide this.
       c)  We should really think through what we want it to look like
before adding it.
            Perhaps ideally it should be supported by syntax and even
if we cannot now
            change the syntax to support coroutines we should
acknowledge the ideal.

  4) As pointed out in my original posting,  my idea of pipes is
really stolen from the
      (Bash in my case) Linux shell.  In Linux, pipes are used between
processes and
      thus, since Smalltalk supports multiple processes (threads),  a
full implementation of
      Andreas' generators or my pipes should include their use in
communicating between
      Smalltalk processes (threads).

   5) Apart from Andreas' very useful feedback no one provided any
feedback whatsoever.
       If my idea is a good one it would be nice to hear.
       More importantly if it stinks telling me and others why is very useful.
       Better to be sent back to the drawing board than to be left in the dark.
       Lastly, I doubt my ideas are the last word on pipes.  Ideas
from others on how to
       improve upon it would be the most useful comments of all.  Are
my ideas so out
       of sync with the Squeak community that they are completely ignored,
       save for Andreas?

My original posting was long so I have not repeated it here.  It
should be easy to find in
the archives and I can forward a copy to anyone who cannot find it.
Andreas'  (first) response to my posting follows:

> > I do not understand all the commotion over generators.

> Generators transform a push model (#do:) into a pull model (#next).
> Let's say you have two algorithms one that produces values via a push
> model and one that consumes values via a pull model:

> producer := SomeAlgorithm new.
> producer do:[:value| "produces the value from the algorithm"].

> consumer := AnotherAlgoritm new.
> consumer process: inputStream. "consumes the value from inputStream"

> Plugging these two together can be a royal pain. You basically have the
> following three options:

> 1) Preproduce all the values from the producer and create a stream that
> holds them:

> values := Array streamContents:[:s| producer do:[:each| s nextPut: each]].
> consumer process: values readStream.

> This can work very well if the number of values is limited and if you
> know you're going to consume all the produced values (doesn't work that
> well with infinite sequences of numbers).

> 2) Rewrite either one of the algorithms. Can work very well if it's your
> own code and reasonably simple, and can be extremely painful if it's
> complex.

> 3) Use a Generator. With a generator, the solution becomes simply:

> consumer process: (Generator on:[:yield| producer do: yield]).

> When you can neither use solution 1) or 2) generators simplify your life
> *greatly*. In fact I'd say they are indispensable if you're dealing
> with a complex existing environment.

> Cheers,
>  - Andreas





--
Had a wife and kids in Florida, Jack (Nicklaus)
Went out for a "ride" but I couldn't get back
As a ten timer being hunted down
OR
When the wife found out who I was knowing (biblically speaking)
I hit a hydrant and I just kept going (till I hit a tree).
...

On Sat, Feb 13, 2010 at 11:46:23PM -0700, Ralph Boland wrote:
>
>   4) As pointed out in my original posting,  my idea of pipes is really stolen from the
>       (Bash in my case) Linux shell.  In Linux, pipes are used between processes and
>       thus, since Smalltalk supports multiple processes (threads),  a full implementation of
>       Andreas' generators or my pipes should include their use in communicating between
>       Smalltalk processes (threads).
>

I don't have anything to say one way or the other about generators,
coroutines, etc. But I do agree with your point that the concept of
pipes is quite useful. As a minor correction, I have to point out
that the concept has nothing to do with Linux, it was one of the
fundamental ideas that Unix got right very early on in more or less
the same time frame that Smalltalk was being invented.

Your original posting put forward quite a few interesting ideas, but
taken down to the basics, it seems to me that if you just implemented
the read and write stream protocols (which IIUC is the fundamental idea)
on top of a SharedQueue, then you have a pipe.  You can imagine all
kinds of variations (e.g.  pipes between images, various performance
optimizations, and so forth), but read and write streams on a SharedQueue
should cover the basics with very little new invention required.

In the CommandShell package (which loosely emulates a unix shell with
a perly mashup of Smalltalky things) I used character-oriented OSPipe
and InternalPipe to implement unix style pipe behavior. I always
thought it would be a good idea to replace this with a generalized
pipe of objects (not just characters), and use SmartRefStream to
convert objects to external form when sending them through an OSPipe.

So FWIW I like your suggestion of adopting the concept of pipes in
Smalltalk.

Dave