Generator example: Nested log file compression

Folks -

Now that we have both a compression algorithm and generators, let's put
them together. Attached is a (slightly more compact) version of my
compression script for Eliot that can be used like this:

Array
        compress: 'AAAABCBCBCABCABCA' readStream
        to:[:count :pattern| Transcript show: count printString,'x', (String
withAll: pattern); space]
        lookahead: 20.

But more interestingly, we can now nest the algorithm using generators:

Installer installUrl:
'http://source.lukas-renggli.ch/continuations/Generator-ar.5.mcz'.

generator := Generator on:[:g|
        Array compress:'AABAAB' readStream
                        to:[:repeat :pattern| g yield: repeat printString, 'x', (String
withAll: pattern)]
                        lookahead: 20.
].

Array
        compress: generator
        to:[:repeat :pattern| Transcript cr; show: repeat printString, 'x ',
pattern, ' ']
        lookahead: 20.

The result of which is 2x #('2xA' '1xB'), i.e., AABAAB just as expected.

The advantage of using generators in this case is that I can whip up the
nested version without having to think of eventual consequences in terms
of space (yes, I could conceivably run two passes but would that work if
Eliot's log file is GBs in size?) or how to restructure the algorithm
(yes, you could rewrite the algorithm but why waste my valuable time?).

*That* is the value of generators to me. They solve a certain class of
problems straightforwardly and intuitively that one needs to work around
in awkward ways otherwise. It's simply a way of being more productive
when you have a problem that fits that pattern (which is not all that
uncommon).

Cheers,
   - Andreas

>>>>> "Andreas" == Andreas Raab <[hidden email]> writes:

Andreas> Now that we have both a compression algorithm and generators, let's put them
Andreas> together. Attached is a (slightly more compact) version of my compression
Andreas> script for Eliot that can be used like this:

Could I nudge someone to look at Xtreams instead?  I think you're reinventing
a lot of stuff that's already done, and from what I understand, Cincom is
putting Xtreams under the MIT license.

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[hidden email]> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Randal L. Schwartz wrote:
>>>>>> "Andreas" == Andreas Raab <[hidden email]> writes:
>
> Andreas> Now that we have both a compression algorithm and generators, let's put them
> Andreas> together. Attached is a (slightly more compact) version of my compression
> Andreas> script for Eliot that can be used like this:
>
> Could I nudge someone to look at Xtreams instead?  I think you're reinventing
> a lot of stuff that's already done, and from what I understand, Cincom is
> putting Xtreams under the MIT license.

Maybe I'm missing something but isn't Xtreams a stream *framework*?
Generator is a single class with a total of 13 methods. Not exactly what
I'd consider as reinventing a lot of stuff (in particular given that the
implementation is both fun, educational, and mind-twisting). As for
licensing: Wake me up when it actually happened and there is code to
download. You of all people should know how long license changes can
take :-)

Cheers,
   - andreas