If python goes Etoys, let's go scripting then - with the best IDE for creating and a clean license for running(!) scripts

Dear squeak masters and friends,

Professor Nierstrasz (liking the IDE of Squeak and Smalltalk a lot,  
but starting to build Ruby scripts for convenience (#!) reasons)  
asked Marcus and me about the possibility of having Squeak as a good  
script writing engine on the one hand and a script executor on the  
other hand.

The underlying idea is simple: use a full fledged Squeak as your IDE  
but then file out your smalltalk script you created (in a more human  
readable version, thus without those "!!" and also edible in Emacs  
and friends, maybe looking like the scripts Andreas and Bert proposed  
on Squeak-dev recently) and run it on top of some minimal working  
image in a #!/usr/local/bin/squeak way. Ruby on rails will be on the  
next version of OS-X, but what does us stop to have things like  
Seaside there in the follow up?

I think currently it is the license. Especially for the stuff needed  
to run the "scripts".
Whereas I would not care for the open source kindness of the Squeak  
license, many people do...

As far as I understand the situation, a minimal working image could  
be soon made OSI-compliant using this apple open source license  
APSL2, no?
Isn't it mainly morphs on the image side? These would not be  
necessary in the minimal script, but maybe I am too "blue eyed" here  
again.

But for the VM side: Is it too naive from me to think that it is only a:

  "YES, I HEREBY DECLARE ALL MY CHANGES TO THE APSL2 LICENSED VM FROM  
SQUEAK 1.1 ALSO TO BE APSL2 LICENSED"
(please sign here ...)

from the few VM gurus (I hopefully caught here as addressees) which  
prevents us from doing so?

So please, dear VM gurus, tell me, this would be possible to  
relicense your great work of the last years to something OSI  
compliant- and then do it, - or that this has already been done, or  
that it is unnecessary for some good reason. I think changing the  
_current_ VM license to APSL2 could accelerate the rest of the  
process a lot.

Then - having this "scripting language plot" established, we might be  
able to tear some more people to the "Etoys Omniuser plot" which in  
my eyes stays the master plan of course...

And: I may be ignorant! Is there any discussion about this issue I am  
missing, or points that I see in a wrong perspective?

Cheers,

Markus

p.s. Please forgive receiving this as a doublet, but it was silly of  
me to send this only to a few people yesterday - learn sth. new every  
day... ;-)
The alternative would have been to send it to squeak-board, but now I  
opted to have the discussion (hopefully there is more action) public  
and not closed - it is about the openness of squeak in the end...

About script, is not what is about GNU Smalltalk, plus it have a clean
licence, a JIT compiler but yes a poor IDE environment

Hilaire

Le mercredi 23 août 2006 à 08:27 +0200, Markus Gaelli a écrit :

On 8/23/06, Hilaire Fernandes <[hidden email]> wrote:
> About script, is not what is about GNU Smalltalk, plus it have a clean
> licence, a JIT compiler but yes a poor IDE environment

and it uses the changeset syntax

--
 Damien Pollet
 type less, do more

Well, I got a lot of flack for my "it's not worth it" response to  
Markus' original post. Very well, I'll make another attempt to  
explain why I think this idea is a lot harder than it sounds. First,  
let me define some terms:

IDE - This is a program that allows one to view and manipulate  
another program in terms of it's semantic elements, such as classes  
and  methods, rather than in terms of the sequence of characters that  
will be fed to a parser. IDEs might happen to display text, but they  
also provide tools like class browsers, refactoring and other  
transformations, auto-completion of identifiers etc, things that  
require a higher level model of the program than text. Examples  
include various Smalltalk implementations, Eclipse, Visual Studio, IDEA.

Scripting language - a programming language and execution model where  
the program is stored as text until it is executed. Immediately prior  
to execution, the runtime environment is created, the program's  
source code is parsed and executed, and then the runtime environment  
is destroyed. This is an important point - the state of the runtime  
environment is not preserved when execution terminates, and one  
invocation of a program cannot influence future invocations.

Now, one might quibble over my definition of "scripting language."  
Fine, I agree that it's not a good general definition of everyday use  
of the term. But it's an important feature of languages like Ruby,  
Python, Perl, Javascript, and PHP and one that makes IDEs for those  
languages particularly hard to write.

Damien Pollet brought up the key issue in designing a Smalltalk-bases  
scripting language - should the syntax be declarative or imperative?  
(Yeah, that again.)

Imperative syntax gives us a lot of flexibility and power in the  
language. A lot of the current fascination with Ruby stems from Java  
programmers discovering what can be done with imperative class  
definitions. The Ruby pickaxe book explains this well:

        In languages such as C++ and Java, class definitions are processed  
at compile time:
        the compiler loads up symbol tables, works out how much storage to  
allocate, constructs
        dispatch tables, and does all those other obscure things we'd rather  
not think too hard
        about. Ruby is different. In Ruby, class and module definitions are  
executable code.
       
Executable definitions is how metaprogramming is done in scripting  
languages. Ruby on Rails gets a lot of milage out of this,  
essentially by adding class-side methods that can be called from  
within these executable class definitions to generate a lot of boring  
support code. In Java, we can't modify class definitions at runtime,  
and that's why Java folks use so much XML configuration.

Python does this too - http://docs.python.org/ref/class.html. Perl5  
is pretty weird, but Perl6 is slated to handle class definition this  
way as well. Javascript doesn't have class definitions, but we can  
build up pseudoclasses by creating objects and assigning functions to  
their properties.

When writing an executable class definition, we have the full power  
of the language available. You can create methods inside of  
conditionals to tailor the class to it's environment. You can use eval
() to create methods by manipulating strings. You can send messages  
to other parts of the system. You can do anything.

I'm making a big deal out of this, because I think it's a really,  
really important feature of modern scripting languages.

Declarative syntax, on the other hand, gives us a lot of flexibility  
and power in the tools. Java, C++ and C# have declarative class  
definitions. This means that IDEs can read in the source code, create  
a semantic model of it, manipulate that model in response to user  
commands, and write it back out as source code. The source code has a  
cannonical represenation as text, so the code that's produced is  
similar to the code that was read in, with the textual changes  
proportional to the semantic changes that were made in between.

This is really hard to do with scripting languages, because we can't  
create the semantic units of the program just by parsing the source  
code. You actually have to execute it to fully create the program's  
structure. This is problematic to an IDE for many reasons: the  
program might take a long time to run, it might have undesirable side  
effects (like deleting files), and in the end, there's no way to tell  
whether the program structure we end up with is dependent on the  
input to the program.

Even if we did have a way to glean the program structure from a  
script, there would be no way to write it back out again as source  
code. All of the metaprogramming in the script would be undone,  
partially evaluated, as it were, and we'd be stuck with whatever  
structures were created on that particular invocation of the script.

So, it would appear that we can have either a powerful language, or  
powerful tools, but not both at the same time. And looking around,  
it's notable that there are no good IDEs for scripting languages, but  
none of the languages that have good IDEs lend themselve to  
metaprogramming.

There is, of course, one exception. Smalltalk.

With Smalltalk, we have the best of both worlds. A highly dynamic  
language where metaprogramming is incredibily easy, and at the same  
time, a very powerful IDE. We can do this because we sidestep the  
whole issue of declarative vs. imperative syntax by not having any  
syntax at all.

In Smalltalk, classes and methods are created by executing Smalltalk  
code, just like in scripting languages. That code creates objects  
which reflect the semantic elements of the program, just like in the  
IDEs for compiled languages. One might say that programs in compiled  
languages are primarily state, while programs in scripting languages  
are primarily behavior. Smalltalk programs are object-oriented; they  
have both state and behavior. The secret ingredient that makes this  
work is the image - Smalltalk programs don't have to be represented  
as text.

And that's why a Smalltalk-like scripting language wouldn't be  
worthwhile. It leaves out the very thing that makes Smalltalk work so  
well - the image. It would have to have syntax for creating classes -  
either imperatively or declaratively. We'd end up limiting either the  
language or the tools, or if we tried hard enough, both. There are  
certainly no shortage of languages that have tried to be "Smalltalk,  
but with source code in files."

I'd much rather see a Smalltalk that let me create small, headless  
images, tens or hundreds of kilobytes in size, with just the little  
bits of functionality I need for a particular task. If they had good  
libraries for file I/O, processing text on stdin/stdout and executing  
other commandline programs, they'd fill the "scripting language"  
niche very well. If they could be created and edited by a larger IDE  
image, they'd have the Smalltalk tools advantage as well.

I have high hopes for Spoon in this regard. Between shrinking, remote  
messaging and Flow, it's already got most of the ingredients. It just  
needs to be packaged with a stripped down VM, and integrated into the  
host operating system.

Colin

Thank you.

On 8/24/06, Colin Putney <[hidden email]> wrote:
> I'm making a big deal out of this, because I think it's a really,
> really important feature of modern scripting languages.

Yes and that's why I'm not so fond of the changeset/pepsi syntax
especially because smalltalk tends to have more small classes and
methods that other languages.

> There is, of course, one exception. Smalltalk.
>
> With Smalltalk, we have the best of both worlds. A highly dynamic
> language where metaprogramming is incredibily easy, and at the same
> time, a very powerful IDE. We can do this because we sidestep the
> whole issue of declarative vs. imperative syntax by not having any
> syntax at all.

Well, ignoring some aethetic details, I consider that a changeset is
readable syntax.

Rather what happens is that the IDE forgets all the metaprogramming
actions that, in a scripting language, would structure the code. At
best you get an history with the changes file (which would map to a
trace of the metaprogramming part in a script).

And anyway, the kind of scripts that I'd like to write in smalltalk
rarely (ab)use metaprograming like bigger programs inevitably do.
Example: watch *.tex and call make when one changes. It's more than 15
lines in ruby because I tried to be clever, added a few options and
threw in some ANSI color bling-bling :) I don't think I even declare a
class, it's just a loop.

> I'd much rather see a Smalltalk that let me create small, headless
> images, tens or hundreds of kilobytes in size, with just the little
> bits of functionality I need for a particular task. If they had good
> libraries for file I/O, processing text on stdin/stdout and executing
> other commandline programs, they'd fill the "scripting language"
> niche very well. If they could be created and edited by a larger IDE
> image, they'd have the Smalltalk tools advantage as well.

Granted.

> I have high hopes for Spoon in this regard.

Me too. Pressure, pressure :)

--
 Damien Pollet
 type less, do more

On 24 août 06, at 09:09, Colin Putney wrote:

> I'd much rather see a Smalltalk that let me create small, headless  
> images, tens or hundreds of kilobytes in size, with just the little  
> bits of functionality I need for a particular task. If they had  
> good libraries for file I/O, processing text on stdin/stdout and  
> executing other commandline programs, they'd fill the "scripting  
> language" niche very well. If they could be created and edited by a  
> larger IDE image, they'd have the Smalltalk tools advantage as well.

sure this is the point!
But having a simple syntax for declaring methods like pepsi (instead  
of using !!) can make it.

by the way I'm trying to understand why we cannot have a declarative  
syntax for Smalltalk that still can be interpreted dynamically or  
not. May be I'm not clear enough about what is the declarative  
syntax: for me it means that I can read it and can analyze it and get  
a model of the program (this does not exclude executing it on the fly  
when I load the program).

Stef

> I have high hopes for Spoon in this regard. Between shrinking,  
> remote messaging and Flow, it's already got most of the  
> ingredients. It just needs to be packaged with a stripped down VM,  
> and integrated into the host operating system.

Yes me too. Even if sometimes I would like to be able to touch the  
spoon "community"

Stef

In a script we would expect everything to be executable (rather than  
simply declared), but what is a nice way to define new methods?  A  
declarative syntax would just state that methods are defined, rather  
than sending a message to a class asking it to add or redefine a method.

MyClass compile: 'mymethod ^ self'

just does not cut it.

But the declarative syntax:

MyClass mymethod [ ^ self ]

is not right either, since this can be interpreted as sending  
mymethod to MyClass and then sending a block to the result (which  
makes no sense).

Something like this would work, but is ugly to read:

MyClass defineMethod: #mymethod withBody: [ ^ self ]

MyClass defineMethod: #+ withBody: [ :other | ^ self add: other ]

MyClass defineMethod: #a:b: withBody: [ :anA : aB | ^ ... ]

It seems like some new syntax would be needed if you want readable  
and editable scripts.

Maybe some kind of bang-notation to separate method declarations from  
executable code is not so bad!

!MyClass>>mymethod
        ^ self !

Oscar

On Aug 24, 2006, at 13:19, stéphane ducasse wrote:

> by the way I'm trying to understand why we cannot have a  
> declarative syntax for Smalltalk that still can be interpreted  
> dynamically or not. May be I'm not clear enough about what is the  
> declarative syntax: for me it means that I can read it and can  
> analyze it and get a model of the program (this does not exclude  
> executing it on the fly when I load the program).

I think that a script should not define new methods or classes.  A
script should just execute them.

I think it would be fine to say that a script always executes in an
image.  If you want to define new classes then you define a new image.
 Or, you could make the first line of the script file in  some
packages.

The big value of a scripting system is to let you write short programs
with little fuss.  If your script gets large then you should rewrite
it.

It is amusing that everybody is interested in defining the language,
but nobody seems to want to define the proper class library for a
scripting system.  The important part is string matching and I/O, not
the format for the language.  Make a script be the same as a
workspace, except that there is an automatic "do it" on the entire
script.

-Ralph

On 24 août 06, at 15:02, Ralph Johnson wrote:

yes read my wishes list I sent to hernan:
        - define a nice package for accessing the file system
                (I want to be able to write '/' asDir filesAndFolderRecursivelyDo:  
[:each | Transcript show: each name , ' ', each size]

On Aug 24, 2006, at 15:02, Ralph Johnson wrote:

> I think that a script should not define new methods or classes.  A
> script should just execute them.

Hm.  But my Ruby scripts consist of class definitions and methods,  
and then a call to single method.

If I cannot define new methods or class then what is the point of an  
OO scripting language?

Oscar

On 8/24/06, Oscar Nierstrasz <[hidden email]> wrote:

> Hm.  But my Ruby scripts consist of class definitions and methods,
> and then a call to single method.

Ruby has to do it this way because it doesn't have an image.

To me, scripting isn't very OO.  However, a good scripting system
needs proper domain abstractions.  OO is good for defining the domain
abstractions.  If you just want to bash some files, you don't need to
define new classes.  But if you want to think of the files as "Paper
Reviews" or "Job Applications" then you will want to make some
classes.

People who write scripts will learn the Smalltalk expression syntax
and collection protocol.  Then they will want to write classes, so
they will learn to use the image.  They will already know a lot of
classes, so it will not be as big a shock as it is now.

-Ralph

Ralph Johnson wrote:

> On 8/24/06, Oscar Nierstrasz <[hidden email]> wrote:
>
>> Hm.  But my Ruby scripts consist of class definitions and methods,
>> and then a call to single method.
>
>
> Ruby has to do it this way because it doesn't have an image.

>
> To me, scripting isn't very OO.  However, a good scripting system
> needs proper domain abstractions.  OO is good for defining the domain
> abstractions.  If you just want to bash some files, you don't need to
> define new classes.  But if you want to think of the files as "Paper
> Reviews" or "Job Applications" then you will want to make some
> classes.

I think that a scripting system should allow the user to decompose the
problem into smaller ones, because if not, the script could be ending in
a 100 lines of plain code... and at the same time I think that classes
and objects are too much for the type of problems that one specs to
solve when scripting...
Therefore (and don't kill me for this idea), I think the script language
should support the definition of "procedures or functions" and no more
than that...
Doing so you can decompose a cohesive problem into smaller ones easily,
no "bigger" abstractions are needed, and if the problem calls for bigger
abstractions then scripting is not the solution any more...


--
______________________________
Lic. Hernán A. Wilkinson
Gerente de Desarrollo y Tecnología
Mercap S.R.L.
Tacuari 202 - 7mo Piso - Tel: 54-11-4878-1118
Buenos Aires - Argentina
http://www.mercapsoftware.com
---------------------------------------------------------------------
Este mensaje es confidencial. Puede contener informacion amparada
por el secreto profesional. Si usted ha recibido este e-mail por error,
por favor comuniquenoslo inmediatamente via e-mail y tenga la
amabilidad de eliminarlo de su sistema; no debera copiar el mensaje
ni divulgar su contenido a ninguna persona. Muchas gracias.
 
This message is confidential. It may also contain information that is
privileged or otherwise legally exempt from disclosure. If you have
received it by mistake please let us know by e-mail immediately and
delete it from your system; you should also not copy the message nor
disclose its contents to anyone. Thanks.
 ---------------------------------------------------------------------

Reading this I'm not sure everybody is aware of how the chunk format
works, because it actually is not declarative.

A chunk is everything from the current stream position to the next bang
(!). Each chunk is simply executed as a "do it". The result is ignored.

However, empty chunks (whitespace only) are a special marker.

If there is an empty chunk, then the result of evaluating the following
chunk is treated as special reader object. It is sent the #scanFrom:
message with the actual input stream as argument. For example,
'methodsFor:' answers a ClassCategoryReader which parses each chunk as
method until it encounters an empty chunk.

Perhaps it's useful to rearrange the bangs a bit for illustration:

        Object subclass: #Foo
                instanceVariableNames: 'foo'
                classVariableNames: ''
                poolDictionaries: ''
                category: 'Foo'

        ! ! Foo methodsFor: 'accessing'

        ! foo
                ^foo

        ! foo: aFoo
                foo := aFoo

        ! ! "end Foo methodsFor: 'accessing'"

        (Foo new foo: 'bar') inspect

The most annoying thing about that format is that you have to double
bangs in your code. Also, since one and the same symbol (!) is used to
mark start and end and separator, its hardly readable. But otherwise
it's pretty much genius, allowing even to embed binary data in your
script (pretty much like a shell's "<<id" here-document construct).

So I imagine a similar construct could be used for a human-friendly
scripting syntax. I'd prefer to only add minimal syntax and let the rest
be handled by regular methods.

- Bert -

Oscar Nierstrasz schrieb:

On 8/24/06, Ralph Johnson <[hidden email]> wrote:
> I think that a script should not define new methods or classes.  A
> script should just execute them.

Hmm. But often a script does need at least some procedural code (i.e.
methods private to it) or to make ad-hoc class extensions or to define
ad-hoc classes too (unless you want to use dictionaries as basic data
structures...)

--
 Damien Pollet
 type less, do more

On 24 août 06, at 17:42, Bert Freudenberg wrote:

> Reading this I'm not sure everybody is aware of how the chunk  
> format works, because it actually is not declarative.

exactly this is why this can be a pain to perform some analysis on cs.

> The most annoying thing about that format is that you have to  
> double bangs in your code. Also, since one and the same symbol (!)  
> is used to mark start and end and separator, its hardly readable.

Exact

> But otherwise it's pretty much genius, allowing even to embed  
> binary data in your script (pretty much like a shell's "<<id" here-
> document construct).
>
> So I imagine a similar construct could be used for a human-friendly  
> scripting syntax. I'd prefer to only add minimal syntax and let the  
> rest be handled by regular methods.

Yes but the syntax of pespi is cool too. :)

Even if I understand now what colin wanted to say about scripting or  
the fact that in ruby your code represent actions that will be  
performed when loaded, while a fileout (modulo class and doti) or a  
java program is more the result that is saved.

Stef

Interesting discussion, as I have been wrestling with this very issue  
in my Objective-Smalltalk / stsh work for some time now.

On Aug 24, 2006, at 14:13 , Oscar Nierstrasz wrote:


> MyClass defineMethod: #mymethod withBody: [ ^ self ]

This is close to what I currently have, but it's not really very pretty.

> It seems like some new syntax would be needed if you want readable  
> and editable scripts.

While I agree, I do find that in practical use of stsh, my scripts  
tend to generally not need new method definitions very much.  
Scripting does seem to be somewhat different from (other) OO  
programming, though I do also think that the transition should be a  
smooth one, so it should be easy and elegant to define methods and  
classes.  At least that's one of my motivations for having something  
like (Objective-)Smalltalk as a scripting language:  there should be  
no breaks as a script evolves into something larger.

And maybe if it were easier I would do more of it in my scripts!


>
> Maybe some kind of bang-notation to separate method declarations  
> from executable code is not so bad!
>
> !MyClass>>mymethod
> ^ self !

That is machine readable, not human read-/writable, IMHO.

One idea that's been on my mind for a bit is that maybe for an OO  
scripting system, classes should be defined in separate script files?  
So the script file itself can define a sort of implicit class context,  
but this can vary.  Hmm..

Marcel

On 8/24/06, Marcel Weiher <[hidden email]> wrote:
> Interesting discussion, as I have been wrestling with this very issue
> in my Objective-Smalltalk / stsh work for some time now.

can we play with stsh ? besides systrace shell google only found a
mail from you :)

> > MyClass defineMethod: #mymethod withBody: [ ^ self ]
>
> This is close to what I currently have, but it's not really very pretty.

My idea was to find a nice hybrid syntax of method patterns and blocks
as a notation for CompiledMethods...

#MyClass << Object.   "MyClass should become a valid identifier at this point"
MyClass instVars: #(x y);  "why is it a string in current smalltalks ???"
       
        >> [myMethod: foo with: bar|  "but parsing this sucks"
                |a b c|
                foo: qux
        ]; "semicolumn strikes back :D but here it's a regular cascade"
       
        >> anotherMethodHere

Here are other ideas to give actual selectors to blocks:

##sel1: arg1 sel2: arg2 [ |locals| body ] "all this denotes a CompiledMethod"

#sel1:arg1#sel2:arg2  "enhanced symbol, where each pair selectorPart +
argument name are separated, but no space and the method declaration
needs to be a kwd message to give the body as a block

--
 Damien Pollet
 type less, do more

> Interesting discussion, as I have been wrestling with this very  
> issue in my Objective-Smalltalk / stsh work for some time now.

Is stsh available ? If yes, where ?
According to Google STSH means sensitive thyroid stimulating hormone

cheers,
Alexandre
--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.cs.tcd.ie/Alexandre.Bergel
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

> MyClass instVars: #(x y);  "why is it a string in current smalltalks ???"

The symbols #x and #y will not be instantiated
 until you use it as a method selector.

best,
Ale.


----- Original Message -----
From: "Damien Pollet" <[hidden email]>
To: "The general-purpose Squeak developers list"
<[hidden email]>
Sent: Thursday, August 24, 2006 4:39 PM
Subject: Re: Re: Scripting languages and IDEs (was: If python goes EToys...)

point" CompiledMethod"
>
> #sel1:arg1#sel2:arg2  "enhanced symbol, where each pair selectorPart +
> argument name are separated, but no space and the method declaration
> needs to be a kwd message to give the body as a block
>
> --
>  Damien Pollet
>  type less, do more
>