[squeak-dev] Squeak 2010-ish

Squeakers, Squeakians, Squeakites, and Squeakods,

Allow me to mark the end of 2008 by sharing with you a little story and a wish list for 2010.  [and before anyone asks, yes I have free time and will help]  Also allow me to admit that I am a bad Squeak user, in that I abuse the hell out of the system.  I come from a school of programming that believes only in the reality of the machine itself, and often feel that computer scientists should never be allowed near a computer, let alone allowed to program one. :)

That said, last month I was playing around with prototyping an application in Squeak, in a fumbly experimental way, poking around in dark corners most people never bother with.  And it struck me that most of the problems that I was having, that Craig Latta tries to address in Spoon, and that causes forks like Pharo, all stem from a fundamental misconception Squeakers have about how their environment should work.  And much of the issue stems from my favorite whipping boy, the Smalltalk system dictionary object.  

Oh how I love to abuse that object!  I just love "polluting" the global namespace of my images with all sorts of often used objects and doodads.  Take your favorite object and just jam it in there, and poof!  You can start inserting it all over the place, and like magick you stop having to pass along useless arguments.  If you play this game long enough, you quickly get to a place were few if any selectors actually take any arguments!

And classes, now there's where you can have real fun.  I like writting methods that programatically generate classes, and then mutate them based upon real world results.  Why use a variable that stays largely static for long periods of time, when you can just recompile the method on the off chance it changes?  Need a one of a kind one off object?  Just abuse the entire notion of class, and make a classy-object and pop it in Smalltalk for good measure.

Let's look at the style of some the code that I've been writing:

        myMethod := 'someObject someMethod' compile.
        myClass addMethod: myMethod named: #goDoSomething.
        myObject := myClass clone.
        myObject addMethod: 'someOtherPrototype doSomethingElse' compile name: #goDoSomething.

And as you can see, I'm doing some pretty nasty stuff.  I've extended ByteString and ByteArray to invoke the compiler on their contents and produce CompiledMethods, and then wrapped a bit of Behavior's stuff and stuck it in Object because I feel the distinction between classes and objects are counter productive.

So why in the world would someone start smashing apart the whole Smalltalk OO world by abusing globals, dumping classes, and start writing self-modifying code in Smalltalk?  Well partially because we should be able to, but more importantly because Smalltalk code should be able to.  If you're writing Smalltalk that writes Smalltalk, development processes that serve a human well, do not necessarily serve a program well.  Especially, when to that software bits are bits, and has no real understanding of a distinction between source, code, and data.

The experiences and experiments that I've been conducting over the past year, all seem to point to some fundamental weaknesses in the overall system architecture:

1.) Primatives can not be added to the VM on the fly, by just assembling some code and running it.

2.) The global namespace (aka Smalltalk) is static, but not hyper-static (where in all top level name changes only affect code going forward)

3.) Class and Trait names are global variables. (which is doubly a problem when you consider #2)

4.) Objects and processes all live within one space, but are often unreasonably hard to reference (anonymous)

5.) The operation of one conceptually distinct group of objects (say an application), can easily interfere with the operating of a second conceptually distinct group of object (say the system).

6.) It is difficult / impossible to share a single object, between two distinct implementations of the system, without resorting to simulating the system within itself (which is itself a simulation of the system within itself).


Now I admit, these weaknesses may seem a bit odd or even objectionable, and many system level squeakers are going to claim that if you look at the system from a different perspective they may not even exist.  But from my POV, what this boils down to is:

1.) I can't test new hardware in a running squeak image
2.) I can't load and unload applications, without affecting the underlying system
3.) I can't load and unload multiple versions of a "module" without substantial hackery
4.) I can't run 2 of the same process at once, and be sure that they don't affect eachother's internal state
5.) I can't extend the interface procedurally, without risking breaking my debugger!
6.) I can't test multiple versions of my code base against the same set of live objects!


The project that I'm working on does involved embedded systems, robotics, AI, and evolutionary programming.  Most of the road blocks I've run into in Squeak, are a product of what is clearly early design trade offs made by people who never imagined one would want to do half the things I'd like to do with their toy.  These problems, however, also effect every other developer out there who is trying to write more conventional applications.  

What I really want is a way to treat ImageSegments as their own physically distinct machine, with their own attached Process list and Smalltalk.  The top level Smalltalk System Dictionary, would not be a global dictionary, but a hyper-static dictionary, like the Forth dictionary or the ML top level namespace, where you can multiply redefine globals in effect producing multiple contexts within that global space (hence order of loading mattering).  And then, I'd like it if these ImageSegments using either their out pointers mechanism or something else could reference eachother's objects, and send each other messages, but with some ability to easily name those out pointers in the local context.  Finally, I'd really love it if the VM had a simple primative that simply let you invoked to content of any ByteArray as native code.  (This would also have the interesting side effect that you could eventually rewrite the entire low level VM entirely in live objects inside Squeak using Squeak, dumping Slang).

Obviously, I'm not saying Squeak sucks, or that the problems require rewriting Squeak in its entirety.  I think the problem is one of familiarity.  We as programmers are far too familiar with the warts in our system, and stop looking at them for what they are.  Right now, Squeak is a lot like DOS.  You can do one thing at a time, and every project requires a whole new application.  What we need is a Squeak that is a lot more like a Mach microkernel, capable of keeping lots of balls in the air at once, and easily extensible in "user space".

Anywhoo, that's my two last cents for 2008.

Best wishes for 2009,

Dave
       

Hi Dave,

it is now one-two more hours here, until Happy New Year, but this year we
are doing the child care and so I have some time for addressing your
concerns: the kids went to bed already ;)

On Wed, 31 Dec 2008 18:41:15 +0100, you wrote:

> Squeakers, Squeakians, Squeakites, and Squeakods,
>
> Allow me to mark the end of 2008 by sharing with you a little story and  
> a wish list for 2010.  [and before anyone asks, yes I have free time and  
> will help]  Also allow me to admit that I am a bad Squeak user, in that  
> I abuse the hell out of the system.  I come from a school of programming  
> that believes only in the reality of the machine itself, and often feel  
> that computer scientists should never be allowed near a computer, let  
> alone allowed to program one. :)

The (your) "typical" computer scientist is often not a good programmer and
the converse also holds often; what's wrong with people having between min
and max of talent in their fingers?

> That said, last month I was playing around with prototyping an  
> application in Squeak, in a fumbly experimental way, poking around in  
> dark corners most people never bother with.  And it struck me that most  
> of the problems that I was having, that Craig Latta tries to address in  
> Spoon, and that causes forks like Pharo, all stem from a fundamental  
> misconception Squeakers have about how their environment should work.

I agree to some degree; too many "mistakes" made it into the "monolithic"
.image and people who could clean up, the mess they produced themselves
for the long run, are often no longer around.

Sort of, life happens.

> And much of the issue stems from my favorite whipping boy, the Smalltalk  
> system dictionary object.
>
> Oh how I love to abuse that object!  I just love "polluting" the global  
> namespace of my images with all sorts of often used objects and  
> doodads.  Take your favorite object and just jam it in there, and poof!  
> You can start inserting it all over the place, and like magick you stop  
> having to pass along useless arguments.

The protocol for adding new things at the global level is very small and
centered precisely: one can easily put an interceptor at work and (at
least) record (FBiSmalltalk at: Time now put: evidence) or so.

No need to go soo deep as if it where a C system. In project NewCompiler
it has a BytecodeGenerator "I generate bytecodes in response to
'instructions' messages being sent to me" which also helps to avoid many
mistakes.

> So why in the world would someone start smashing apart the whole  
> Smalltalk OO world by abusing globals, dumping classes, and start  
> writing self-modifying code in Smalltalk?  Well partially because we  
> should be able to, but more importantly because Smalltalk code should be  
> able to.  If you're writing Smalltalk that writes Smalltalk, development  
> processes that serve a human well, do not necessarily serve a program  
> well.  Especially, when to that software bits are bits, and has no real  
> understanding of a distinction between source, code, and data.

There is a whole project dedicated to "self-modifying code in Smalltalk",
over at

- http://code.google.com/p/moebius-st/

I'm sure the Moebius team and you would benefit from each other, towards
self-modifying code in Smalltalk.

> The experiences and experiments that I've been conducting over the past  
> year, all seem to point to some fundamental weaknesses in the overall  
> system architecture:
>
> 1.) Primatives can not be added to the VM on the fly, by just assembling  
> some code and running it.

... that's part of the Moebius project.

> 2.) The global namespace (aka Smalltalk) is static, but not hyper-static  
> (where in all top level name changes only affect code going forward)

... this seems to be addressed elegantly with NewSpeak (for which we still
have to wait until it gets released) but, Igor+my have also worked out
concepts which address "local" "globality" at the runtime and no longer at
the compile-time level (could work in classic Squeak as well -- a small
compiler change and an even smaller bug fix in the stock VM would be all
that is needed).

But what's missing is that you talk about the hands which then have to
refactor all the until-now existing source code? in order to benefit from
such an isolation level?

> 3.) Class and Trait names are global variables. (which is doubly a  
> problem when you consider #2)

Right. Yes. Damn'it yes (!)

> 4.) Objects and processes all live within one space, but are often  
> unreasonably hard to reference (anonymous)

This is perhaps a little bit OT but *any* solution to 4) seems to require
distributed GC/alloc to work acceptable on *one* CPU, which we haven't seen
yet. This is perhaps analogous to the "I" in AI ;)

> 5.) The operation of one conceptually distinct group of objects (say an  
> application), can easily interfere with the operating of a second  
> conceptually distinct group of object (say the system).

I'm sure you had something in mind, other than this: the app level
irrefutably calls the system level, or: the client level  irrefutably
calls the service level. but what?

> 6.) It is difficult / impossible to share a single object, between two  
> distinct implementations of the system, without resorting to simulating  
> the system within itself (which is itself a simulation of the system  
> within itself).

Really? and I always thought that, for example a model can be seen+shared
by completely different viewers. What would be the problem that you are
concerned with here?

> Now I admit, these weaknesses may seem a bit odd or even objectionable,  
> and many system level squeakers are going to claim that if you look at  
> the system from a different perspective they may not even exist.  But  
> from my POV, what this boils down to is:
>
> 1.) I can't test new hardware in a running squeak image

... this must be a mistake, since hardware doesn't run in software (other
that in a simulator). if I where a piece of hardware I would object to
running in a Smalltalk system ;)

> 2.) I can't load and unload applications, without affecting the  
> underlying system

Right. In Squeak the developers often mixed up service level and client
level, and so produced .images from which the app can no longer be
unloaded, unfortunately. I once hoped that the Pharo folks would address
this but they do: they throw things out manually (no offense!).

But isn't this a typical "for me it works" scenario? ;)

(OT*0.5) can't we always go back to earlier releases of Squeak in which
the not unlodable did not exist?

> 3.) I can't load and unload multiple versions of a "module" without  
> substantial hackery

Right, and good luck anyways.

> 4.) I can't run 2 of the same process at once, and be sure that they  
> don't affect eachother's internal state

Why not? what would be wrong with Semaphore and friends?

> 5.) I can't extend the interface procedurally, without risking breaking  
> my debugger!

... the "interface procedurally" doesn't parse here. sorry.

> 6.) I can't test multiple versions of my code base against the same set  
> of live objects!

Right. I therefore proposed to have multiple method dictionaries per class
in development .images and for production to consolidate the multiple
md's again (for example in your case one md per version). sent several
emails to several people but they where not interested (perhaps they want
"their" "own" solution to that; really, I don't know).

> The project that I'm working on does involved embedded systems,  
> robotics, AI, and evolutionary programming.  Most of the road blocks  
> I've run into in Squeak, are a product of what is clearly early design  
> trade offs made by people who never imagined one would want to do half  
> the things I'd like to do with their toy.

And if predicting the future where so easy as you seemingly describe it
here, why would they care about your needs? did you send them your
proposals and requirements?

> These problems, however, also effect every other developer out there who  
> is trying to write more conventional applications.
>
> What I really want is a way to treat ImageSegments as their own  
> physically distinct machine, with their own attached Process list and  
> Smalltalk.

Aha. now *you* are proposing a "one thing does it all" solution. doesn't
that make you automagically belong to "people who never imagined one would
want to do half the things that other people would like to do with that
toy" ? and if not, please tell us why not.

I'm suprised you don't use one of those which available for Squeak. did
you check Exupery which precisely does that? or perhaps Huemul? both are
based on and do "it" for Squeak.

> (This would also have the interesting side effect that you could  
> eventually rewrite the entire low level VM entirely in live objects  
> inside Squeak using Squeak, dumping Slang).

Yes. see the Moebius project which I mentioned above.

> Obviously, I'm not saying Squeak sucks, or that the problems require  
> rewriting Squeak in its entirety.

No, right. but incrementally ;)

> I think the problem is one of familiarity.  We as programmers are far  
> too familiar with the warts in our system, and stop looking at them for  
> what they are.  Right now, Squeak is a lot like DOS.

No. it simply isn't. it's perhaps the way and portions of Squeak that you
are using that makes you believe the DOS religion. I for example don't use
comparable-to-DOS things when I use Squeak as developer and/or user.

> You can do one thing at a time, and every project requires a whole new  
> application.

I didn't see anything else than this association during the last 30 years
or so. yes, an application is a project.

> What we need is a Squeak that is a lot more like a Mach microkernel,  
> capable of keeping lots of balls in the air at once, and easily  
> extensible in "user space".

I'm neutral to that microkernel "one thing does it all" solution. and I
know that Huemul did it and has it.

But I believe that Squeak needs even more succesfull applications,
"besides" Etoys, Croquet, Plop, CMSbox, Sophie, Seaside+friends, Scratch,
and so.

> Anywhoo, that's my two last cents for 2008.
>
> Best wishes for 2009,

Thanks, and same to you :-)

/Klaus

> Dave
>


--
"If at first, the idea is not absurd, then there is no hope for it".
Albert Einstein

Hi David,

David J. Goehrig wrote:
> Squeakers, Squeakians, Squeakites, and Squeakods,
> Allow me to mark the end of 2008 by sharing with you a little story
> and a wish list for 2010.

How do you feel about your message now, one year later? I am interested
in pretty much all of the things you discuss in the message: in
particular, in gaining a clearer understanding of the interaction
between a hosting and the hosted system: the VM and image, if you like,
or recursively, processes in the image and the InterpreterSimulator'd
subimage.

> 1.) Primatives can not be added to the VM on the fly, by just
assembling some code and running it.

This is the only one of your list that's low-hanging fruit, really, if
you don't mind poking at the VM sources a bit that is. A primitive for
getting the system to set %eip (%rip) to the start of a given ByteArray
would do the trick ;-)

Perhaps Alien will help here?

> 1.) I can't test new hardware in a running squeak image
> 2.) I can't load and unload applications, without affecting the underlying system
> 3.) I can't load and unload multiple versions of a "module" without substantial hackery
> 4.) I can't run 2 of the same process at once, and be sure that they don't affect eachother's internal state
> 5.) I can't extend the interface procedurally, without risking breaking my debugger!
> 6.) I can't test multiple versions of my code base against the same set of live objects!

These, to me, are (still) *excellent* requirements for a Smalltalk of
the Future!

Regards,
  Tony