( picoVerse-:( convert Generators from version 2.1 to version 6 ) )

I have a large library of Packages for Dolphin Smalltalk version 2.1
patch level 2.  The ones that are probably of the most interest to the
Smalltalk community are the ones about Generators.  I would like to
port the Packages to Dolphin Smalltalk 6 community version.  But X6
doesn't seem to run on Windows 98.  It said that it would bring up a
Dialog asking me where the unlocked image should go and when I clicked
the [OK] it just closed and dissapeared.  Not very Smalltalk like.  I
imagine I can get Dolphin Smalltalk 2.1 to work under Windows XP?  And
then I could move things over from V2.1 to V6 Class by Class and Method
by Method?  I should think that there should be a way to just suck
everything from a 2.1 Package over to version 6.   This is how I used
to do it:  I made up an Object called a Crystal which had a Dictionary
of < <instance variable name><instance variable value> > pairs.  I
would search out all the instances of a Class and turn them all into
Crystals with the Class name in the Crystals.  Each instance would
>>become: a Crystal of itself.  Then I could change the shape of the original Class by adding instance variables and recompile it with no instances of itself in the Image.  Then I would unCrystalize all the Crystals back into instances of the Class.  Now the new instance variables would be there as well as the old ones.  This was before there was refactoring.  And it worked.  It's too hard to port Packages from one version to another.  Packages should be designed so that new versions will suck up old Packages from old versions.  Once a Package is in the image there should be a [Convert To Version X Package] menu item in the Package Browser.  And then you just save the version 6 Package to disk.  So each Package system should be a super set of the previous one.  Also it would be nice if each version of Dolphin could run the images of previous versions.  Because if I install Version 6 on my Version 2.1 machine now all my 2.1 images don't work anymore because Version 6 has taken over.  I have to reinstall version 2.1 to get those images working again.  What a mess.  Not good planning.  My next idea is to install version 6 and then install version 2.1 and once version 6 is running its virgin image then I could try to port Class by Class and method by method over to version 6 into new Packages under XP.  Or maybe I should try to port to version 5( the free version from the magazine ) and then port from there to version 6 since version 5 works under Windows 98.  And maybe it is easier to get from 2.1 to 6 via 5.   Please help!!!



Where Collections are breadth first Generators are depth first.  This
can make many things easy that are hard or impossible or impossibly
baroque using Collections.  You can have infinite sized Generators.
You mostly have a Generator on a Collection like a Stream is on a
Collection.  But you can also have a Generator on another Generator so
you can build them up incrementally like a little machine using select:
reject: collect: etc. etc. etc.  You can create Generators like little
robots that can be used as the headwaters of other Generators.  I use
Generators a lot.  Recently I made a method which uses Generators to
fish through a human readable journal file and collect up data which
could then be graphed.  It took three days to make.  It's rock solid.
As soon as it works it works all the time.  No bugs.  The Generator
methods are extensively documented with tons of usage ideas.  The
writing style is a little screwy though.  It took years to make.  There
is an untested full design for generalizing Generators to sequences of
trees where changes in shape to previous trees also changes the shape
of following trees.  There is select: reject: collect: for trees.
There is a generalization of Tree recursion using Blocks the way they
are used with Collections.  There are Lisp like lists as Generators or
Trees.  There are tons of example methods that make use of Generators.
You can have Generators on Generators on FileStreams or on Strings or
on Streams or on Collections.  All sorts of things and almost limitless
combinations.  Frankly the Generator stuff makes the Collections and
Streams by themselves look hopeless.  An image without Generators is an
empty place.  How do you get anything done without the KEGGenerators?
Is the question.  Yes you can try to use Collections and Streams to do
the same thing but it's messy and you keep finding places where a
Generator would do it better.  There is a FiniteStateMachine based in
the KEGGenerator>>sortInto: method.  You can make simple
FiniteStateMachines the same way you use select: reject: collect: by
using the sortInto: method to make the FiniteStateMachine.  Which is a
Generator so it can be put into larger Generators.
You use the linear way of embedding:

          (
          (
          (
          ( aCollection asGenerator
          ) collect:[ :x | ........x......... ]
          ) select:[ :x | .....x............ ]
          ) reject:[ :x | ............x...... ]
          ) etc:[ :x | .........x............. ]

which allows you to embed deeper without making a mess.  Now you have
built a new Generator but you have not iterated it yet because
Generators are Lazy.  You have to send it >>asOrderedCollection or some
such thing to get it to iterate.

Kego,

Hang on. It states quite clearly in the system requirements on the
website that the Dolphin 6 development system doesn't run under Windows
98.

http://www.object-arts.com/content/navigation/products/systemRequirement
s.html

Why then are you trying to port your packages to run under an operating
system that you don't have? Doesn't make much sense to me.

> > > Because if I install
> > > Version 6 on my Version 2.1 machine now all my 2.1 images don't
> > > work anymore because Version 6 has taken over.  I have to
> > > reinstall version 2.1 to get those images working again.  What a
> > > mess.  Not good planning.  

Most of the more recent versions of Dolphin will co-exist with Dolphin
6. To be honest we haven't tried Dolphin 2.1 because, not only is it
eight (8) years old but also it is 4 (four) major releases behind.

Cripes, I tried to run some free Java game (on my son's PC you
understand) the other day and it didn't run because it required a Java
VM of 1.4.1 and the machine had 1.4.2 on it. Naturally, I cursed
"bloody Java" under my breath but I didn't write to complain to either
the game producer or Sun because both pieces of software were free and
I should hardly expect anything better.

> > > My next idea is to install version 6
> > > and then install version 2.1 and once version 6 is running its
> > > virgin image then I could try to port Class by Class and method
> > > by method over to version 6 into new Packages under XP.  Or maybe
> > > I should try to port to version 5( the free version from the
> > > magazine ) and then port from there to version 6 since version 5
> > > works under Windows 98.  And maybe it is easier to get from 2.1
> > > to 6 via 5.   Please help!!!

Again, why do you want to get to D6 when you don't seem to have the OS
to run it?

Frankly, it seems to me that you are only attempting to port your stuff
because there is, once again, a free version of Dolphin. My guess is
that, if we hadn't released a free DCE6, then you wouldn't be
complaining about lack of backwards compatibility because you wouldn't
be attempting the port.

 From my (admittedly biased) point of view this seems overly critical.
What level of support do you honestly expect from a free product? Maybe
I'm just old fashioned but this new era of Open Source software and
"free beer" seem to have distorted most peoples expectations.

Sigh!


Regards

--
Andy Bower
Dolphin Support
www.object-arts.com

Hello [hidden email],

> I have a large library of Packages for Dolphin Smalltalk version 2.1
> patch level 2.  The ones that are probably of the most interest to the
> Smalltalk community are the ones about Generators.  I would like to
> port the Packages to Dolphin Smalltalk 6 community version.  But X6
> doesn't seem to run on Windows 98.  

The Smalltalk Companion book comes with a limited edition of Dolphin 5.1,
so maybe you can use it as a stepping stone to D6.

Hello [hidden email],

> combinations.  Frankly the Generator stuff makes the Collections and
> Streams by themselves look hopeless.  An image without Generators is

I've been playing with the idea of adding generators "a la python" to Smalltalk,
but never had the time. Can you give more details on your implementation?

[hidden email] wrote:

> I have a large library of Packages for Dolphin Smalltalk version 2.1
> patch level 2.

Can't you just load those packages into D6 ?  That's supposed to work, though I
wouldn't be too surprised if so large a jump in versions exposed some lurking
bug.

If the jump is too big, then maybe you could load them into, say, D3 before
saving them[*], and then try again.  If necessary try D4 too, and so on.  If
you don't have or can't find the intermediate versions, then maybe someone
around here could help.

([*] Dolphin always saves in the "current" format, not the one that the package
file had when it was loaded.)


> Where Collections are breadth first Generators are depth first.

Sounds interesting and/or fun.

I believe that the original concept of Smalltalk ReadStreams was that they were
intended to be external iterators too, but in practise, too many assumptions
(e.g. I/O related) have been built into that part of the hierarchy to make it
easy to use them that way.

    -- chris

The hierarchy is basically
KEGGenerator
    KEGAbstractGenerator
        KEGStringGenerator

and then there are methods all over
the place like >>asGenerator.  So
you can wrap Collections and Streams
in Generators and then create more
Generators by wrapping those into
Generators.  You send the >>asGenerator
message to a Collection or a Stream
and then you get a Generator which can
be wrapped in Generators.  If you send
>>asGenerator to any old Object then
a singleton Generator is returned.
All the Collection iteration methods work
for Generators and mostly you can use
Collections and Generators interchangeably.
There are tons and tons of Generator iteration
methods that are not present for Collections
because it just wouldn't work.  Or they are
just not natural for Collections.

I am thinking about getting Windows XP.
I would like to port to X6 because it is the latest version and because
I hope that I could make the Packages available to both the free and
not free versions of X6.  I would like to make Generators available to
the Dolphin Smalltalk community.  I would like to become part of the
Dolphin Smalltalk community.

I guess in a post further down it says that I should port to like X5
and then port to X6 from there.  That seems to work.

I didn't mean to sound critical but just suggestive.  I guess most of
the things I was asking for are there in Dolphin.  I can install X2.1
Packages into X5 and then save them as X5 Packages.  From there I can
go to X6.

Yeah that is my idea too.

> ([*] Dolphin always saves in the "current" format, not the one that the package
> file had when it was loaded.)

Thanks for the info that Dolphin always saves Packages in the format of
the image not the original format.  That is really useful to me.

Hello Chris,

> Sounds interesting and/or fun.
>
> I believe that the original concept of Smalltalk ReadStreams was that
> they were intended to be external iterators too, but in practise, too
> many assumptions (e.g. I/O related) have been built into that part of
> the hierarchy to make it easy to use them that way.


Apparently,  Alan Knight has being playing with this in VW:
http://www.cincomsmalltalk.com/userblogs/knight/blogView?showComments=true&entry=3320514455

Andy

I think that KEGGenerators should be included in all
of the different Smalltalks.  Either as an addon package
or as an integrated part of the base image.  Right now
it has the GPL license.  But other licenses could work
if that would mean that Generators could be included
in the Smalltalk base image.   Why would the GPL
get in the way?  If someone changes one of the KEGGenerator
methods then they just have to make a Package with those
changes in it.  If they add methods they don't have to do
anything.  I think that you should be able to make a
Package A that knows about a Package B and can overwrite
methods and Classes in B when it gets filed in and finds
Package B in the image.

> Apparently, Alan Knight has being playing with this in VW:
> http://www.cincomsmalltalk.com/userblogs/knight/blogView?showComments...

I think that wrapping a Stream in a Generator is
actually much better than trying to turn a Stream
into a Generator by adding collecting: selecting: rejecting:
methods to Stream.  Streams just were not made to be
Generators.  It is true that a KEGGenerator can be ON
anything that responds to >>reset and >>next so Streams
can be considered to be Generators since they respond
to >>reset and >>next.  Anything that responds to >>reset
and >>next can be considered to be a Generator and
KEGGenerator can be on it.  Also KEGGenerator>>detect:
is either the empty sequence or a singleton sequence.

Try this with Collections:
    | three | ( three := false ).
    (
    (
    ( #( 1 2 3 4 5 ) asGenerator
    ) collect:[ :x | ( x = 3 )ifTrue:[ three := true ] ]
    ) select:[ :x | three ]
    ) asArray

which will return #( 3 4 5 ).

You see you can have local variables passing
info between the various Generators you are
building up. >>collect: returns a KEGAbstractGenerator A
and >>select: returns a KEGAbstractGenerator B
and >>asArray is where B is sent >>reset followed by
>>next over and over until nil is returned and then the
finished Array is returned.

But actually if you wanted to really do this you
would do:
    (
    ( #( 1 2 3 4 5 ) asGenerator
    ) after:[ :x | x = 3 ]
    ) asArray

Well no, this would return #( 4 5 ) so maybe the
above example is a way of implementing
KEGGenerator>>afterIncluding: .

KEGGenerator>>afterIncluding: okBlock
    | first |
   ^(
    (
    ( self abstractGenerator
    ) reset:[ ( self reset ). first := false ]
      next:[ self next ]
    ) collect:[ :x | ( okBlock value:( x ) )ifTrue:[ first := true ] ]
    ) select:[ :x | first ]

So

    | three | ( three := false ).
    (
    (
    ( #( 1 2 3 4 5 ) asGenerator
    ) collect:[ :x | ( x = 3 )ifTrue:[ three := true ] ]
    ) select:[ :x | three ]
    ) asArray

becomes

    ( #( 1 2 3 4 5 ) asGenerator afterIncluding:[ :x | x = 3 ] )

you could inline the >>collect: and >>select:
within the >>afterIncluding in a single >>reset:next:
for added speed but it would be messier.

The main Generator Class structure is:

KEGGenerator
    KEGAbstractGenerator
        KEGStringGenerator

KEGAbstractGenerator has
    resetBlock               and
    nextBlock                instance variables.

And the
KEGGenerator>>collect: method looks like
    | next |
    ^( self abstractGenerator )
         reset:[ self reset ]
         next:[ ( next := ( self next ) )==nil ifFalse:[ next ] ]

KEGAbstractGenerator>>reset: reset next: next          looks like
    ( resetBlock := reset ).
    ( nextBlock := next ).

If an AbstractGenerator>>next message returns nil then then
it will cause all the Generators it feeds into to return nil also
from then on.  Essentially all Generators are infinite in length
but they consist of a non nil prefix sequence followed by an infinite
number
of nils .  KEGGenerator>>asArray makes an OrderedCollection
and then converts that into an Array.  As soon as >>asArray
hits a nil then it stops adding to the OrderedCollection.

In this sort of way Generators of different sizes can be
combined and you don't need to worry all the time that
they be of equal sizes.  As soon as one of them runs
out of elements the right thing will be done like if you
did

gen1 := #( 1 2 3 )asGenerator .
gen2 := #( 1 2 3 4 5 ) asGenerator.
gen3 := gen1 pairUp:[ :x :y | x + y ]with: gen2

then as soon as either gen1 or gen2 runs out of elements
then the KEGAbstractGenerator returned by >>pairUp:with:
will start returning nil to all subsequent >>next messages
until it receives a >>reset message.  So

( gen3 asArray ) will return #( 2 4 6 )

KEGGenerator>>asArray looks like
    ^( self asOrderedCollection ) asArray

KEGGenerator>>asOrderedCollection looks like
    | answer |
    ( answer := OrderedCollection new ).
    ( self do:[ :x | answer add:( x ) ] ).
    ^( answer )

KEGGenerator>>do: aBlock is the main iterator and looks exactly like
    | next |
     next := self first .
     [ next notNil ]
          whileTrue:[
               aBlock value: next .
               next := self next
          ] .

So the Generator we created above look like

( headwaters )->( Gen1 )->( Gen2 )

where nextBlocks contain the pointers.

In the above collect: select: example the
headWaters is returned by #( 1 2 3 4 5 ) asGenerator.
and Gen1 is returned by the >>collect: message
and Gen2 is returned by the >>select: message.

Suppose you put Gen2 into a local variable gen2 instead
of sending Gen2 the asArray message.

    | three gen2 | ( three := false ).
    ( gen2 :=
    (
    ( #( 1 2 3 4 5 ) asGenerator
    ) collect:[ :x | ( x = 3 )ifTrue:[ three := true ] ]
    ) select:[ :x | three ]
    ).

You can make a Gen3 Generator ( in local variable gen3 )
and evaluate:

    ( gen2 on: gen3 )

and Gen3 will become the new headWaters
instead of ( #( 1 2 3 4 5 ) asGenerator ).

Therefore you can make a generalized gen3
and reuse it on various gen2s.

So if you are in a situation where you need
to make a lot of new gen2's one for each
turn around an iteration loop but each one
of those has the same initial gen3 headwaters.
Then you can make gen3 once and then reuse
it over and over once for each new gen2.

So then

( headWaters )->( Gen1 )->( Gen2 )

becomes

( Gen3 )->( Gen1 )->( Gen2 )

because the >>on: method inserts
Gen3 in place of the headWaters which
is on a Stream.  This situation comes up
often enough that a special method makes
an empty Generator on a Stream
KEGGenerator class>>streamGenerator
which returns an empty KEGGenerator on
an empty Stream on an empty OrderedCollection
which serves as an initial marker headWaters Generator.

Here is something you can do with Generators
that is inconvenient with Collections.

    | ignore |
    ignore := SomeClass new ."<---( needs to be unique )"
   ^(
    (
    ( someGenerator
    ) collect:[ :x |
        ( x isOK )ifTrue:[ x someProcessing ]ifFalse:[
        ( x isOK1 )ifTrue:[ x someProcessing1 ]
        ifFalse:[ ignore ] ] ]
    ) reject:[ :x | x == ignore ]
    )

which is a combination >>select: with an embedded
>>collect: which most of the time will probably return
a sequence that is smaller in size than the original
someGenerator sequence.  This happens often enough
that you would actually use the
KEGGenerator>>collect:ignoring: method instead of the
>>collect: >>reject: above.

Collections and Generators and Streams etc all
respond to the >>isSequence message by returning
true.  But String>>isSequence returns false.  This
is so that Generators will not break Strings up into
sequences of Characters.  So that you can have
sequences of Strings which is what you want
instances of the KEGStringGenerator Class to be.

If you want a String to become a sequence of Characters
you would do ( aString asCharacter ) which would return
a KEGGenerator on the Characters in aString.

Since Generators are depth first you can have
a Generator on the lines of a File by doing:

linesGen := ( FileStream read:'somePath' ) asGenerator linesStrGen .

( I think )

which returns a KEGStringGenerator of all the lines in the File F
designated by 'somePath' . But none of those lines are in RAM
yet.  And now you can do

sections := linesGen sectionsEndingIn:[ :line | 'someText'
containedIn:( line ) ].

which actually returns a KEGAbstractGenerator which is
actually a sequence of OrderedCollections O1 O2 O3 ...
where each of the O1 and O2 and O3 etc are == identical.
So the sequence actually looks like < O1 O1 O1 O1 ... >.

But each of the O1's has different contents.  This is because
>>sectionsEndingIn: returns a KEGAbstractGenerator that is
doing a depth first thing when it iterates via the >>do: message.

So each section of File F is a sequence of text line Strings
ending with a line that contains 'someText' somewhere in it.
Each section looks like:

line1
line2
line3
...
linewith'someText'somewhereInIt

Each section is loaded into the OrderedCollection O1

OrderedCollection(
line1
line2
line3
...
linewith'someText'somewhereInIt )

( sections do:[ :sectionOC |
    someOtherFile nextPutAll:( sectionOC asString ) ] )

will cause the >>do: message to be sent to the
sections KEGAbstractGenerator which will repeatedly
send the >>next message to the sections Generator.
Each of those >>next message will return O1 but
each time with different contents.  Different text lines
from the different sections in File F.

O1 is loaded into the sectionOC local variable over
and over again but each time with different contents.

So each section in File F is only in RAM until the
do: Block exits and then its lines get garbage collected.

But the O1 OrderedCollection gets reused over and over.

O1 does not get garbage collected until the do: loop
itself exits.

So you can write code that looks like Collection code
except that you don't have to keep all those O1 O2 O3...
OrderedCollections in RAM all at the same time.
One single O1 OrderedCollection is reused over and
over again with different String contents each time.

These kinds of RAM savings are always popping up
in every KEGGenerator situation it seems.

RAM is cheap but Generators cut down on the need
to use up lots of RAM for intermediate Collections that
are only needed to make other Collections until finally
the result Collection is found.

If BlockClosures can be called over and over again
in a speedy manner that would make Generators
fast also.

I think that KEGGenerators should be included in all
of the different Smalltalks.  Either as an addon package
or as an integrated part of the base image.

Try this with Collections:
    | three | ( three := false ).
    (
    (
    ( #( 1 2 3 4 5 ) asGenerator
    ) collect:[ :x | ( x = 3 )ifTrue:[ three := true ] ]
    ) select:[ :x | three ]
    ) asArray

actually has a bug in it it should be

Try this with Collections:
    | three | ( three := false ).
    (
    (
    ( #( 1 2 3 4 5 ) asGenerator
    ) collect:[ :x | ( x = 3 )ifTrue:[ three := true ] . x ]
    ) select:[ :x | three ]
    ) asArray

KEGGenerator>>afterIncluding: okBlock
    | first |
   ^(
    (
    ( self abstractGenerator
    ) reset:[ ( self reset ). first := false ]
      next:[ self next ]
    ) collect:[ :x | ( okBlock value:( x ) )ifTrue:[ first := true ] .
x ]
    ) select:[ :x | first ]