Info on Smalltalk DSLs or Metaprogramming...

Hi,

I've seen several online articles about writing Domain Specific  
Languages in Ruby. All the articles mention that the idea of DSLs  
comes largely from Lisp and Smalltalk. I've found descriptions on  
writing DSLs in Lisp--but I can't seem to find any good references  
for Smalltalk.

Does anyone have any helpful pointers? If I'm going to follow the  
Ruby model, then I need to be able to execute arbitrary strings as  
code (which Compiler evaluate: and related functions seems to  
handle). I also need to be able to add new methods to classes/
instances at runtime--which I haven't found.

Any help would be greatly appreciated.

-Rich-

Is there anyone who has the power to rename packages on squeaksource?

many thanks in advance

Keith

               
___________________________________________________________
All New Yahoo! Mail – Tired of Vi@gr@! come-ons? Let our SpamGuard protect you. http://uk.docs.yahoo.com/nowyoucan.html

2006/8/14, Keith Hodges <[hidden email]>:
> Is there anyone who has the power to rename packages on squeaksource?

Packages or projects?

Philippe

Philippe Marschall wrote:
> 2006/8/14, Keith Hodges <[hidden email]>:
>> Is there anyone who has the power to rename packages on squeaksource?
>
> Packages or projects?
>
> Philippe
>
>
Sorry I meant to say projects!


Keith

               
___________________________________________________________
Yahoo! Messenger - with free PC-PC calling and photo sharing. http://uk.messenger.yahoo.com

Here is a blog entry of mine about using the Refactoring Browsers code
to generate classes for me:

http://wilkes.blogspot.com/2005/12/simple-code-generation-in-smalltalk.html

The Behavior>>compile: method will add a new method onto a class.

For instance methods:
MyClass compile: 'someAttribute
    ^ someAttribute'

For class methods:
MyClass class compile: 'aClassMethod
   "Do some stuff"'


On 8/14/06, Rich Warren <[hidden email]> wrote:

Great! Thanks. That should let me bootstrap the things I need.

Does anyone know of any general references to writing interpreters/
DSLs/Metaprogramming in SmallTalk?

-Rich-

On Aug 14, 2006, at 6:55 AM, Wilkes Joiner wrote:

On 8/14/06, Rich Warren <[hidden email]> wrote:
> Great! Thanks. That should let me bootstrap the things I need.
>
> Does anyone know of any general references to writing interpreters/
> DSLs/Metaprogramming in SmallTalk?

Writing interpreters is so easy in Smalltalk that people don't even
talk about how to do it.  See the "Interpreter" pattern in "Design
Patterns".  This was my attempt to explain it, though we did it in a
language independent way.  Basically, you just build a class
hierarchy.  In fact, I bet that most of the time Smalltalkers use the
Interpreter pattern, they don't think of what they write as an
interpreter.  This means that they don't think of the class hierarchy
as a grammar or the tree of objects as an abstract syntax tree.  They
don't think of the instance creation methods as their parser, even if
they work hard to make the syntax natural.

Smalltalkers usually don't use the phrase "DSL", but instead talk
about frameworks for a particular domain.  Also, look at
http://adaptiveobjectmodel.com for some papers on DSL that are either
in Smalltalk or are similar to ones that are.  Again, the papers
usually don't mention the phrase "DSL".

-Ralph Johnson

In my tests, I've noticed that Behavior>>compile: actually changes  
the image. Is there any way to make less-permanent changes (e.g. just  
changing a single instance)?

-Rich-


On Aug 14, 2006, at 6:55 AM, Wilkes Joiner wrote:

There is another approach available in Smalltalk that is not available
in many other languages.  The syntax of Smalltalk as is allows for a
pretty good DSL to be built out of simple classes and blocks with good
selector naming.  You really do not need to dynamically add code to the
system to get a DSL when the syntax for Smalltalk is so flexible and has
good readability.  You can certainly do "modeling" and generate the code
from some other representation.  But, then you lose all the value of the
Smalltalk tools.  If the user wants to model orders just create an Order
class and let them go to town with Orderish methods and blocks.  Then
they can use the best debugger on the planet (that actually works when
you want to change the source, etc).  You can even persist in-flight
execution and many other things that other environments do not support,
letting that DSL support more than just in-memory execution.
Integrating database accesses and distributed communications is not a
barrier.

Michael


-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Rich
Warren
Sent: Wednesday, August 16, 2006 6:56 PM
To: The general-purpose Squeak developers list
Subject: Re: Info on Smalltalk DSLs or Metaprogramming...

In my tests, I've noticed that Behavior>>compile: actually changes  
the image. Is there any way to make less-permanent changes (e.g. just  
changing a single instance)?

-Rich-


On Aug 14, 2006, at 6:55 AM, Wilkes Joiner wrote:

There is another approach available in Smalltalk that is not available in
many other languages.  The syntax of Smalltalk as is allows for a pretty
good DSL to be built out of simple classes and blocks with good selector
naming.  You really do not need to dynamically add code to the system to get
a DSL when the syntax for Smalltalk is so flexible and has good readability.
You can certainly do "modeling" and generate the code from some other
representation.  But, then you lose all the value of the Smalltalk tools.
If the user wants to model orders just create an Order class and let them go
to town with Orderish methods and blocks.  Then they can use the best
debugger on the planet (that actually works when you want to change the
source, etc).  You can even persist in-flight execution and many other
things that other environments do not support, letting that DSL support more
than just in-memory execution.  Integrating database accesses and
distributed communications is not a barrier.

Michael

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Rich
Warren
Sent: Wednesday, August 16, 2006 6:56 PM
To: The general-purpose Squeak developers list
Subject: Re: Info on Smalltalk DSLs or Metaprogramming...

In my tests, I've noticed that Behavior>>compile: actually changes  
the image. Is there any way to make less-permanent changes (e.g. just  
changing a single instance)?

-Rich-


On Aug 14, 2006, at 6:55 AM, Wilkes Joiner wrote:

It may not be necessary (probably isn't). It may not even be wise.  
But I'm still curious--how far can I stretch the boundaries?

  The specific project I'm working on is an AI research project as  
part of my Ph.D. Ideally I would like to have a configuration text  
file that (with an easily-human-readable syntax) configures the  
system for each trial--one line per trial.

Why not just code it all in smalltalk? I want non-programmers to be  
able to look at the configuration file, and look at the data and  
easily understand  the initial conditions for a specific run and how  
it correlates with the resulting data. Having the configuration file  
as a separate artifact will greatly help--but only if I can make it  
easy to read (basically code that doesn't look like code).

I'd like to create a mini-language for the config file whose syntax  
basically consists of objects, selectors and operators. I'm still  
feeling out the infrastructure's requirements--and I'm trying to  
decide whether to implement everything in Smalltalk or Ruby. I have  
good examples for implementing a DSL in Ruby. I think I can port most  
of the ideas easily enough to Smalltalk, but some of the more  
complicated examples do make changes to classes at runtime (I'm not  
sure I'll need that--but I'm also not sure I won't).

So far,they're still running fairly neck-and-neck. In most ways,  
they're both excellent approaches. Here's my thoughts:

Ruby:

I know Ruby better.

All my DSL examples are in Ruby.

For DSL's, Ruby's syntax seems slightly more flexible than Smalltalk  
(or maybe I just know how to abuse it more).

 From my research so far, Ruby as better support for distributed  
objects (especially Rinda). This might be important if we need to  
move the code over to the cluster.

Ruby can more-easily create actual child processes, which might let  
me plow through the data faster on a dual-core machine (though, I  
believe this might lock it into only running on Unix boxes).

I have better Ruby reference books.

I have good instructions for creating charts/graphs directly in Ruby.

Smalltalk:

It would be a good excuse to get more proficient in Smalltalk.

I prefer working in the Smalltalk environments.

Creating a cross-platform GUI will be easier in Squeak.

While getting multiple processes is a pain (as far as I know, I'd  
need to run multiple instances of squeak), I can save and quit squeak  
while the application is running. When I launch squeak, it will start  
up again exactly where it left off. This may actually be the killer  
feature. I can run the experiments on my laptop overnight. Even if  
they don't finish by morning, I can shut things down and take the  
laptop with me to class--without losing any data.

I'm sure there are a lot of other cool reasons to pick Smalltalk that  
I just don't know about, because I'm still kind of low on the  
learning curve.

-Rich-

On Aug 16, 2006, at 10:26 PM, Michael Latta wrote:

On Wed, Aug 16, 2006 at 11:36:33PM -1000, Rich Warren wrote:
>
> Ruby can more-easily create actual child processes, which might let  
> me plow through the data faster on a dual-core machine (though, I  
> believe this might lock it into only running on Unix boxes).

Given that you are expecting to run on a unix box anyway, you can
do this quite easily in Squeak, see OSProcess and CommandShell on
SqueakMap. If you want to run multiple Squeak images on multiple
CPUs, look at #forkSqueak.

But I would be very surprised if you need this in practice. You can
manipulate really large amounts of data directly in Squeak, and
you'll be amazed how fast it can be.  Once you load your data into
Squeak, everything is in memory, and Squeak itself is quite fast.

Not so obvious is that when you load your data into well designed
objects, you actually can use a lot less memory than you might
have expected (no need for multiple copies of the same pieces of
data as often is the case in practice with file-based or database
oriented applications). The end result is that you often end up
realizing that you don't need a database after all, once it's in
a Squeak image that you can save at will, that's all you need.
You can crunch a lot of data all in one image with no need to
fuss with child processes, and no need to worry about exchanging
data between the processes or saving results to files.

When brings us back to your DLS question. Michael Latta's advice
is very good: Try implementing your language as classes and
methods that read well and make sense to people. you can read the
source code for this in from a configuration file if you want.
If that turns out not to be sufficient for your needs, you can
implement any language syntax you like using SmaCC (parser
generator like lex/yacc for Smalltalk). SmaCC is on SqueakMap
also.

Dave

On Aug 17, 2006, at 12:52 AM, David T. Lewis wrote:

Aha. Again, it's the low on the learning curve curse. I guess I  
shouldn't be surprised. Both use the same basic threading model. I  
should have suspected a similar OS process forking abilities.

>
> But I would be very surprised if you need this in practice. You can
> manipulate really large amounts of data directly in Squeak, and
> you'll be amazed how fast it can be.  Once you load your data into
> Squeak, everything is in memory, and Squeak itself is quite fast.

You're probably right, given that using both cores will give me a 2x  
boost, max (and that's being overly optimistic). So the speed boost  
you mention might largely compensate. Struggling to get it running on  
both cores feels like trying to eek out more speed by implementing it  
in highly profiled and optimized (and therefor non-portable) c code  
and assembly. I don't need to go down that road.

I do have a few machines on my home network whose idle cycles I could  
borrow. That's where something like Rinda becomes more attractive.  
And, if I work out the bugs in my home network, then I could run it  
on the school's 32-processor cluster. Doing that should give me an  
order of magnitude boost in speed--which could make a big difference,  
depending on how many simulations we want to run (I'm guessing at  
least in the mid- to high- 100s), and how long each simulation will  
take.

Regarding the DSL itself, I am currently planning on trying to  
implement it using just classes and methods. Using the config file  
approach, I'll read in a line at a time, and evaluate each line.  But  
I'll check out SmaCC, just in case I need the extra power.

Thanks,

-Rich-

Hi rich


I would suggest to consider doing that in Smalltalk using the  
smalltalk syntax as a try and report the problems
you get because the syntax of Smalltalk was made for kids and it is  
uniform and simple.

Can you give an example of what you want to express?


Trial name: 'FirstTrial' resultIn:  'Not so good'


Stef

+ 1

this is why we often do not launch a metacompiler such as Smacc for  
simple languages.
We often use class and selector the parsing, debugging, storage is  
free. Even the refactoring.
Especially during a PhD I would work at that level and in 10 years  
when your language will
be the mainstream language for trial you create a real compiler. This  
is at least the advice
I gave to some friends of mine when they started to open Yacc/lex  
even before understanding
deeply what they wanted to express.

Stef

On 17 août 06, at 12:52, David T. Lewis wrote:

> When brings us back to your DLS question. Michael Latta's advice
> is very good: Try implementing your language as classes and
> methods that read well and make sense to people. you can read the
> source code for this in from a configuration file if you want.
> If that turns out not to be sufficient for your needs, you can
> implement any language syntax you like using SmaCC (parser
> generator like lex/yacc for Smalltalk). SmaCC is on SqueakMap
> also.

> All my DSL examples are in Ruby.
>
> For DSL's, Ruby's syntax seems slightly more flexible than
> Smalltalk (or maybe I just know how to abuse it more).
>

Might I suggest that Smalltalk itself, isn't so much a language, as it is a
notation for creating DSL's.  Smalltalk for example, models predicate logic,
with simple objects and blocks, no special keywords such as
if/unless/do/while are necessary.  Ruby can't do this, and requires special
syntax in the language, due to its inability to pass multiple blocks and its
lack of Smalltalk style keyword selectors.  I'm curious where you see Ruby's
syntax being more flexible?

Here is some recent experience in Java that has relevance to the multi-core
discussion:

1) My system has 8 logical processors (4 cores with HyperThreading).
2) My first attempt a making a set of tasks concurrent got about a -10%
improvement.  With good profilers and several days of work I got that down
to a 2X speedup (using 8 cores).  Individual parts of the process are
getting 100% scalability, and others are still single threaded (which is
very common).

While it is tempting to think that multiple cores means one application will
run faster, in practice it is much harder to make happen, and FAR FAR harder
to debug due to the unpredictability of it.

You mention that you will be running multiple simulations.  Running each
simulation in a separate image is far more likely to produce good
scalability.  You can easily use various methods to move control objects
from one image to another to coordinate the simulations and migrate data
from one image to another at the beginning of a simulation.  I would pursue
that approach much more than trying to get 2 cores to run faster.  2 cores
is useful for multi-processing, but almost useless for multi-tasking as you
generally want a thread to supervise the other threads, which drops you back
down to 1 task oriented core.

Michael

On Aug 17, 2006, at 6:44 AM, Ramon Leon wrote:

Before I get too far into this, let me again point out that my sense  
that Ruby handles DSLs (as I am currently thinking about them)  
possibly better than Smalltalk may simply be because I am more  
familiar with Ruby syntax than Smalltalk. And because I have several  
good examples that give clear examples of different methods for  
building Ruby classes/methods/etc. to support DSLs.

So, please, please, please. If anyone knows of any good examples that  
show how to do these things in Smalltalk, please let me know. That  
might give me a better handle on how to approach this in Smalltalk,  
and might just win me over.

First off, I'm talking exclusively about using internal DSLs here.  
That is, I will be using the natural features of a general purpose  
language. I'm not talking about building a full-fledged parser/
interpreter. However, I do want to be able to read in and interpret  
arbitrary text files at runtime.

I think the approach should be similar in both the Ruby and Smalltalk  
case. Take a line of text. Evaluate it as code in a context so that  
the individual words in the line get interpreted as objects/messages/
methods/arguments/whatever.

If this is the case, then I can see a few places where Ruby would be  
simpler. For example, if you're executing text as code within an  
object, you don't need 'self' when referencing methods. As a result,  
most Ruby DSLs execute inside objects--I suspect Smalltalk DSLs don't  
do this.

To me, this seems to imply that in Smalltalk, the first word on each  
line will typically be a class, followed immediately by either a  
class method or a constructor. Ruby DSLs can do this as well, but  
they can also start immediately with a method call on the containing  
class. That, potentially, gives Ruby a considerable bit of extra  
flexibility.

Second, in Ruby methods that take arguments do not need to end in a  
colon. This can help keep the DSL syntax clean, particularly for  
single-argument methods. Ruby methods can also have default values  
set for the parameters (which I don't believe is available in  
Smalltalk), and if the final parameter is a hash, you can use the  
simplified hash syntax.

On the smalltalk side, I suspect cascading and the separate selector  
for each argument could produce highly readable DSL syntax--but I'd  
really like to see a few demo examples to get an idea of how it all  
fits together.

There may also be ways of doing things in Smalltalk syntax that could  
make DSLs easier--things that I don't know, or that I'm just not  
thinking about. For example, I don't see how avoiding if/unless/do/
while really gains anything (especially since 'if' is arguably more  
human readable than 'ifTrue:'). I also don't see a big gain in  
passing single blocks vs. multiple blocks. I doubt I'd use blocks in  
the DSL directly anyway (or 'if' for that matter)--since the block  
syntax is a bit ugly in both languages--and I can't imagine it being  
necessary.

Really, I'll probably be implementing something very simple--so it's  
probably a wash.

It looks like I'll need to set the following information:

Run name, ID, number of agents, target function, max error,  
visualization on/off, verbose on/off

In Ruby this could be done as follows (calls the new_run method and  
passes in a hash):

new_run name => 'my name', id => 0, agent_count =>  200,  target =>  
30sphere ,error => 1e-10, visualize => true

Since we are passing a hash into the new_run method, these arguments  
can be in any order. They can also be omitted (the target object  
presumably sets a default for anything not present).

The same thing in smalltalk would look like this (note, they can be  
in almost any order--'name:' is a constructor and must come first--
the others can be in any order and can be dropped):

TrialRun name: 'my name'; id: 0; agentCount: 200; target: 30sphere;  
error: 1e-10; visualize: true

or, for a prettier option (but one that sacrifices flexibility)

TrialRun name: 'my name' id: 0 agentCount: 200 target: 30sphere  
error: 1e-10 visualize: true verbose: false

None of them are really as readable as I would like, but any would  
do. The last one is the most readable, but if the number of optional  
flags started to grow--it would quickly become a real pain.

-Rich-

On Aug 17, 2006, at 7:37 AM, Michael Latta wrote:
This is really the approach I was looking at. I'd have a master  
controller that maintained a pool of available simulations. Slave  
processes would request a simulation, process it, then asynchronously  
return the results. The master would be idle most of the time, so the  
slaves should be able to grab most of the available CPU power  
(especially if I can place them on separate machines or across a  
cluster).

I'm not sure how to do distributed processing in squeak though. I'd  
rather not deal with it at the socket level (though that would be  
possible). rST looks promising, but I'm a little concerned about its  
stability (the wiki page says "Testers are needed!" which makes me  
pause).

The difficulty in finding a good way to do distributed processing  
surprises me. I thought distributed processing was supposed to be one  
of Smalltalk's fortes. Which probably means I'm just missing  
something obvious again.

-Rich-

Of course, the best ideas always come after I hit the send button.

It dawned on me that I could always preprocess the incoming string on  
either platform. A relatively trivial amount of code means I could  
write the (now case insensitive) DSL as:

name: my name, id: 0, number of agents: 200, target function:  
30sphere, maximum error: 1e-10, visualize

And have it converted into a simple format for either platform:

new_run :name => 'my name', :id => 0, :number_of_agents =>  
200,  :target_function => :30sphere , :error => 1e-10,  :visualize =>  
true

or

TrialRun new;  name: 'my name'; id: 0; numberOfAgents: 200;  
targetFunction: #30sphere; error: 1e-10; visualize: true

Then evaluate the string.

Of course, this is moving a big step towards the full parser/
interpreter approach. Well, maybe not that big a step, given the  
limited scope of the DSL, but a step.

-Rich-

On Aug 18, 2006, at 12:35 AM, Rich Warren wrote: