RE: Re: Java's modules rock? (was Re: election details *PLEASE READ*)

Hi, Andreas,

just one answer at your first question:
VW allow parcel to modify a method already on the image. Interesting behavior is the fact that, if and when you remove the parcel (thing you can do on a running application) the modification goes away, and the old behavior get restored.
This seems to me more on the Smalltalk stile of thinking: I can do what I want, obviously also shut at my foot.
But this is just my way of thinking.

ciao

Giorgio Ferraris

o base classes? Say, one that says

[hidden email] wrote:
> VW allow parcel to modify a method already on the image. Interesting behavior is the fact that, if and when you remove the parcel (thing you can do on a running application) the modification goes away, and the old behavior get restored.

That's what I expected.

> This seems to me more on the Smalltalk stile of thinking: I can do what I want, obviously also shut at my foot.

Well, that's great as long as it is *your* foot. Unfortunately, when
modifying a base class that other modules use you are not shooting your
own foot - you are shooting the *other* guys foot (since your module
will work happily with those modifications but the other modules won't)
and that's not okay for a module system.

Cheers,
   - Andreas

May be you should read the PhD of alex since it was about scoping  
class extension.

After thinking of lot another solution can be as in objective-C (I  
may be wrong) to support class extension
but not method redefinition.

Another approach could be selector namespace, since your changes are  
only local in your context, and you can still
have all the power of class extension.

If you read the OOPSLA paper on classbox you can see why subclassing  
does not work and why Swing has serious problems.

Stef

On 1 mars 07, at 11:41, Andreas Raab wrote:

On Mar 1, 2007, at 2:41 AM, Andreas Raab wrote:
> Well, that's great as long as it is *your* foot. Unfortunately,  
> when modifying a base class that other modules use you are not  
> shooting your own foot - you are shooting the *other* guys foot  
> (since your module will work happily with those modifications but  
> the other modules won't) and that's not okay for a module system.

It seems there have been some attempts at this in Squeak - islands?  
Squeak-E?  I haven't followed them.

I've done more than my share of freaky classloader programming.  The  
best way to think of them is as a hierarchical namespace.  A new  
class loader is a new namespace.  It can reference (but not extend -  
apart from inherit from) things in the parent classloader.

But this is easy given that Java freezes everything at compile time.  
No extensions to anything allowed post-compile (ok - post load).  
Unfortunately, the bar is higher in Smalltalk as we want to allow  
extension of anything.

I don't think it would be conceptually hard to do something similar  
in Smalltalk if you were to have class lookup work similarly to  
method lookup but along namespace hierarchies.  The other thing that  
would make extension safe is a copy on write semantic with class  
wherein extensions to classes that are not local to your namespace  
result in a new class with the same name in your namespace being  
created that derives from the class in the parent namespace.  Thus,  
your mods are kept local to your context.  It does make class binding  
harder as class names have to be resolved dynamically in the methods  
that reference them.

Tricky work I think, but doable if one had the will and the <evil>one  
million dollars</evil> to pay someone to do it. :-)

-Todd Blanchard

Andreas Raab <[hidden email]> writes:
> Well, that's great as long as it is *your* foot. Unfortunately, when
> modifying a base class that other modules use you are not shooting
> your own foot - you are shooting the *other* guys foot (since your
> module will work happily with those modifications but the other
> modules won't) and that's not okay for a module system.


This is one real issue, but other issues point in the opposite
direction.  It's not a clearcut question.


Note that the purist stance for modules does not work.  That way lies
the mythical components utopia [1], where arbitrary combinations of
modules Just Work.  In practice you need some way of dealing with
inter-module problems.  Whatever that mechanism is -- and for many
communities it can be as simple as maintaining a list of all the
modules in the "world" -- that mechanism will take pressure off of the
module system's built-in checks.

Also, notice that Smalltalkers just love adding methods to base
classes.  Have you ever tried writing Smalltalk code without the
abilitity to add anything to Object?  Java programmers want it, too!
I keep reading articles by guys trying to open up Java classes for
modification by outside modules.  I keep wondering whether they'd be
better off just doing the simple Smalltalk thing.

For Smalltalk, my inclination is to at least allow *additions* to
classes in other modules.  It's very useful, and it does not appear to
be a big enough source of bugs that it should be outlawed.
Practically every Smalltalk code-loding system allows this....


Lex


[1] http://www.lexspoon.org/blog/components-utopia.html

Lex Spoon wrote:
It is a very clearcut question as far as I am concerned. What I am
interested in, however, is not a components utopia (btw, your write-up
was the most painful thing I've ever read from you because it is so full
of baseless presumptions) but rather exploring the possibilities of
loading non-interacting modules side-by-side and from there enable
interactions between them. The way you are phrasing the problem you are
basically giving up without even trying.

For example, a simple, yet perfectly valid answer to the problem is to
load different versions of modules in different images, or, even more
extreme, on different computers. Mind you, these modules can still be
enabled to interact by using the network to communicate, however, for
the purpose of our discussion, they *are* perfectly isolated against
each other. And now tell me why this is "utopian" or why we shouldn't
enable something like that from within Squeak to minimize communication
overhead and space requirements where possible.

And once we start looking at it from that angle we see that -starting
from perfect isolation- there are scopes at which we expect modules to
be isolated (across machines) and scopes at which we expect them to
interact (for example, inside applications). With that basic model, we
can explore which interactions we can enable while still adhering to
whatever strictness of a isolation we'd like to implement. None of which
is utopian, or a halting problem or whatever; just deliberately chosen
tradeoffs. (to come back to the original thread, Java took one stand on
these issues and judging from the results they seem to work)

> Also, notice that Smalltalkers just love adding methods to base
> classes.  Have you ever tried writing Smalltalk code without the
> abilitity to add anything to Object?

Yes, I have. And if not for the need of having a very fast type test
(Object>>isFoo) I have rarely a need for them (and try to avoid them as
much as possible). Yes, there are situations in which modifications to
class Object can be useful, however, most of the time there is no need
for them and in the long term you are typically better off without them
than with them (partially because you write your code more defensively).

Besides, there are a few changes to the language that might dramatically
help with solving these issues - I have recently started to think about
traits in that way, e.g., to be able to "import" an API that immediately
delegates to some other implementor (this particular thought was
triggered by the realization of traits being the equivalent of an
#include<> statement and one of useful the things that does is giving
you access to a name space - in which case you can say "self do: arg"
and the trait might map that automatically into "FooImpl for: rcvr: do:
arg" or so - I'm not sure where this would lead to but the current uses
of traits really resonate with me as solving the wrong problem at the
wrong time in the wrong context ... and yet, I do think that there may
be a pony somewhere) Unfortunately, you'd still need that very fast type
test and possibly also the need to query for a default implementation.

In any case, I think that the "love to adding methods" is pretty much a
red herring in this discussion. It is one of the possible tradeoffs you
can make but it is certainly not the only one. And I'm sure we can find
solutions for it if we put our mind to the problem (as a matter of fact,
Dan made at least one concrete proposal towards that exact problem which
wasn't exactly utopian either).

Cheers,
   - Andreas

Now you are talking!  I think this is the way to start thinking about
all such problems and needs....

Andreas Raab wrote
At 10:07 PM 3/5/2007, Andreas Raab wrote:

You mention several fascinating ideas.  Let me just defend what I mean
on this one issue, and toss out a suggestion of my own.

Andreas Raab <[hidden email]> writes:
> What I am
> interested in, however, is not a components utopia (btw, your write-up
> was the most painful thing I've ever read from you because it is so
> full of baseless presumptions)

I agree it is painful in the abstract, because the concept is
inherently illogical!  Let us look at a concrete example:


"Well, that's great as long as it is *your* foot. Unfortunately, when
modifying a base class that other modules use you are not shooting
your own foot - you are shooting the *other* guys foot (since your
module will work happily with those modifications but the other
modules won't) and that's not okay for a module system."


Your argument here is that feature X allows modules to mess each other
up. The perfect module system must absolutely forbid modules to mess
each other up.  Therefore, X is disallowed.

The components utopia is the second to last sentence: The perfect
module system must absolutely forbid modules to mess each other up.
That's obviously too extreme, right?  Utopian, some might say?

Believe me, I do not mean to beat up on your simplification here.
It's just a simplification.  Perhaps, though, if you try to be more
precise here, then there is a deeper understanding to be had.  What,
precisely, does it mean for modules to allow *limited* interaction, as
opposed to the extremes of non-interaction or blatantly stepping all
over each other?

When considering it, keep in mind that even a network connection--a
terrific metaphor, by the way--is plenty for one module to mess up
another one.  Further, once you can mess up one module, you can also
mess up modules that that module talks to.  A partial-interference
property is hard to define!

Curiously, component systems can succeed even when they allow lots of
interference between packages.  I'm thinking of Debian packages.
Debian packages are your nightmare, Andreas: practically everything is
in a global scope, just begging for inter-package conflicts!  Yet,
Debian manages to produce 10k of these packages which all get along
together.  How can this be?


I think their success is due to how they set up their package sharing.
Debian has a great answer to the following question: what do you do
when, as is inevitable, two packages conflict?


Lex

Hi Lex -

Lex Spoon wrote:
> Your argument here is that feature X allows modules to mess each other
> up. The perfect module system must absolutely forbid modules to mess
> each other up.  Therefore, X is disallowed.

No, this is not my argument. My argument is that I don't get any say in
whether these modules should affect each other in this way or not. There
are perfectly reasonable situations (loading patches, trying to build an
application) where you *want* these modules to affect each other.

> The components utopia is the second to last sentence: The perfect
> module system must absolutely forbid modules to mess each other up.
> That's obviously too extreme, right?  Utopian, some might say?

It depends on what you mean by "absolutely forbid to mess each other
up". What I would expect from the "perfect" module system is to allow me
the formulate the equivalent of both, saying "I want these two modules
on different computers", in other words "perfectly isolated" or
alternatively to specify "I want these modules to affect each other", in
other words "patch module X with module Y". Both are perfectly
reasonable, and I simply don't get why you completely disregard the
former in favor of the latter. There is *nothing* utopian about it - I
run different versions of modules on different computers all the time
and I patch modules with others all the time. And they communicate with
each other all the time.

> Believe me, I do not mean to beat up on your simplification here.
> It's just a simplification.  Perhaps, though, if you try to be more
> precise here, then there is a deeper understanding to be had.  What,
> precisely, does it mean for modules to allow *limited* interaction, as
> opposed to the extremes of non-interaction or blatantly stepping all
> over each other?

I would define "limited interaction" as: Limited to *explicit*
interaction, e.g., sending messages requesting services contrasted with
*implicit* interaction, caused merely by loading a code base that has an
(unintended) side effect on another module. In other words, if module A
uses a public API to tell module B it wants "Foo" to mean "Bar" that is
much more acceptable than the effect that merely loading A has the
effect of "Foo" meaning "Bar" (the simple reason being that module B can
deny the request if it is explicit; and that other modules are aware
that Foo may mean Bar and can guard against that possible problem).

> Curiously, component systems can succeed even when they allow lots of
> interference between packages.  I'm thinking of Debian packages.
> Debian packages are your nightmare, Andreas: practically everything is
> in a global scope, just begging for inter-package conflicts!  Yet,
> Debian manages to produce 10k of these packages which all get along
> together.  How can this be?

Do they? Really? We are currently living through the equivalent of
DLL-hell on Linux, and let me tell you, it is *much* worse than the
situation currently on Windows and MacOS. MUCH worse. For example, on
Windows we ship certain DLLs with our commercial app (trivial; just drop
them into the binary directory) and that works with every binary on
Windows regardless of whether other versions of the same library are
installed on the system or not. On MacOS it's harder, but still doable;
you need to patch the binary a little to use "your versions" of the
library but again, it's doable. On Linux? Well, we just gave up on one
particular product we were planning to use simply because it couldn't
even agree on the C runtime libraries to use! Sorry Lex, but I think
Linux is not exactly the benchmark in this area.

> I think their success is due to how they set up their package sharing.
> Debian has a great answer to the following question: what do you do
> when, as is inevitable, two packages conflict?

You run them side-by-side. That strategy was ultimately the way out of
DLL-hell on windows (compared with previous versions, this problem has
ceased to exist since XP). Windows, starting with XP (and even more
extreme on Vista) has explicitly adapted that strategy to the point
where even patching system DLLs means your app gets their "private
versions" of those DLLs instead of silently side-effecting others. The
same is true on MacOS (with a bit more explicit programmer effort since
historically MacOS didn't have the problems). However, in both cases the
result is the same: Applications with different requirements in terms of
the libraries required can run side-by-side and *still* communicate with
each other using a variety of APIs. They are not *required* to affect
each other, they can if the integrator (usually the app) decides to, but
if the integrator doesn't, they can also live perfectly happy side-by-side.

Cheers,
   - Andreas

I want to try and extrapolate from the discussion and make another  
suggestion ;-)

I have the impression that the thread seems to revolve partly about  
two hidden and different assumptions (bear with me for the moment):
- Andreas seems to be defending an 'anticipated reuse' model, where a  
module is a well-encapsulated black-box entity. Indeed, running in a  
different image ( 'classloader' :-) ) quite strongly enforces this  
encapsulation.
. Lex seems to be more in favour of an 'unanticipated reuse' model  
where anybody can add elements.

Both models are at the extreme ends of the spectrum (more robust vs.  
more brittle, more closed vs. more extensible).

In the same vein however that object-oriented programming opened up  
black-box modules known from procedural languages (by differentiating  
two clients: the inheritor that has white-box access and the client,  
that can only call methods), I would propose to open-up the classical  
fully encapsulated solution you get with a closed module system by  
thinking about what inheritance between modules would look like.

That is really good.  So development is the same: build isolated modules.
But deployment is different either load the module as an isolated package or
as a subclass of another package and you get either patching a module or
running it separately.  The best part of your solution is that if the
namespace is visible then you can easily see the overrides created by a
subclassed package.  We could even allow tweaks to packages to remove
overrides to certain package methods, this could be stored as part of some
deployment descriptor.  This would allow you to publish different
integrations of a package for different purposes.  It would seem that the
visibility is key.  Then to Andreas' point: how do you enable communication
when necessary?  You use the namespace as part of the method call when
necessary.

It's something to think about.

Ron Teitelbaum

On 6 mars 07, at 07:07, Andreas Raab wrote:

I would like to understand what you would like to have here but I  
could not get it.
Can you explain it a bit?


Can you tell us more?

On 6 mars 07, at 07:07, Andreas Raab wrote:

> Yes, I have. And if not for the need of having a very fast type  
> test (Object>>isFoo) I have rarely a need for them (and try to  
> avoid them as much as possible). Yes, there are situations in which  
> modifications to class Object can be useful, however, most of the  
> time there is no need for them and in the long term you are  
> typically better off without them than with them (partially because  
> you write your code more defensively).

Hi andreas

I agree that extending Object is not always the right solution.  
Especially with isSomething...
But they are cases where been able to add method is important.  
Especially since subclassing does not work
since the clients of the class you would like to extend will not see  
the added methods.

I have the impression that either only been able to add (and not  
overrides) I think that override are really evil.
could be a solution or something in the vein of selectorNamespace  
since you control the scope of your changes and you
do not impact existing code.

Now I would really like to see a concrete problem you have because I  
still would like to work on a good module
system. I'm not satisfied with classboxes even if we learned a lot  
designing it.

Stef

On 9 mars 07, at 09:16, Andreas Raab wrote:

At berne guys have been working on changeboxes: having multiple  
versions of a module running in the same image, and I'm trying to  
digest what I would gain to be able to do that. Because I could build  
applications as on my mac that uses different version of components  
without
having to have them all at the same time in the same applications.

Did you encounter cases where you would have like to have different  
versions of the same component/modules running in the same image.

Indeed.

Stef

Hi Stef,

stephane ducasse wrote:

> At berne guys have been working on changeboxes: having multiple versions
> of a module running in the same image, and I'm trying to digest what I
> would gain to be able to do that. Because I could build applications as
> on my mac that uses different version of components without
> having to have them all at the same time in the same applications.
>
> Did you encounter cases where you would have like to have different
> versions of the same component/modules running in the same image.

I very real case: a web hosting of many applications for many customers,
for each customer a bit different one. This can be solved by other means
too, like namespaces in VW, but noone is quite satisfactory.

Best regards
Janko

--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si

stephane ducasse wrote:
> Did you encounter cases where you would have like to have different
> versions of the same component/modules running in the same image.

Sure. Consider Croquet: It relies on replicated computation which means
that the replicated islands must have identical code based to have the
same behavior. But what happens when you link to a space which uses a
different (older or newer) version of a module that is already loaded in
the space you are in? As far I can see there are only three options:]

1. Don't ever allow it. That's our current solution, i.e. all spaces
linked together must use the same code base. For the obvious reasons
this is neither particularly scalable nor desirable.

2. Force migration of "older" spaces. This may be an acceptable solution
if the number of spaces linked together is relatively small. However it
has the disadvantage that you create "waves" of updates since
participants are linked together in unforeseen ways (compare this to a
WWW where each web page can only be linked to another webpage of the
same version and where upgrading one would implicitly require upgrade
all the ones it links to - I think you can see the basic problem with
the approach).

3. Have these modules sit side-by-side. In reality, this is the only
scalable, secure and desirable solution.

Cheers,
   - Andreas

Roel Wuyts <[hidden email]> writes:
> I have the impression that the thread seems to revolve partly about
> two hidden and different assumptions (bear with me for the moment):
> - Andreas seems to be defending an 'anticipated reuse' model, where a
> module is a well-encapsulated black-box entity. Indeed, running in a
> different image ( 'classloader' :-) ) quite strongly enforces this
> encapsulation.
> . Lex seems to be more in favour of an 'unanticipated reuse' model
> where anybody can add elements.

Yes, that is fair generalization.  I would like the default module
format to allow unanticipated reuse.  Just look at the packages
Squeakers post for each other, and you will see all kinds of things
that modify the system.  How could you write something like Shout if a
module cannot change the system text editor?

Dealing with inter-module problems, as Andreas describes, is very
important!  However, to address those problems well, we need to treat
the suite of available modules as itself a thing we are developing.
Then we can file bugs against the suite itself and develop it just
like any other software artifact.

DLL hell is basically absent in Debian.  It is truly sweet to install
packages via Debian, because all the library dependencies are sorted
out already for you.

For anyone who is involved in a project facing DLL hell, I suggest
making an APT repository and a bug tracker and giving it a try.  This
setup eliminates the routine forms of DLL hell, and it turns the hard
cases into normal old bug reports.


To return to the topic at hand, it would be great to see better module
systems.  Classboxes are real cool, as are the ideas Andreas has
floated in this thread.  However, please leave in an override
mechanism.  Make the walls advisory, not mandatory, just like in
Smalltalk of today.

  -Lex

Lex Spoon wrote:
> Yes, that is fair generalization.  I would like the default module
> format to allow unanticipated reuse.  Just look at the packages
> Squeakers post for each other, and you will see all kinds of things
> that modify the system.  How could you write something like Shout if a
> module cannot change the system text editor?

Err, by refactoring it in a way so that you can? We have already various
abstractions (AppRegistry) and there is nothing wrong to have one that
allows for syntax highlighting - with a common protocol that allows
other clients to hook into it, instead of hacking the system text editor
directly. Coincidentally, this is *precisely* what I referred to earlier
wrt. explicit vs. implicit interactions. As it stands, loading an
alternative syntax highlighter will (almost necessarily) clash with
shout; with that interface both modules could use the same ground rules,
could be loaded side by side and could be activated depending on the
context you need them in.

Cheers,
   - Andreas

Hi all,
Since Shout has been mentioned, I'll jump in here.

>----- Original Message -----
>From: "Andreas Raab" <[hidden email]>


> Lex Spoon wrote:
> > Yes, that is fair generalization.  I would like the default module
> > format to allow unanticipated reuse.  Just look at the packages
> > Squeakers post for each other, and you will see all kinds of things
> > that modify the system.  How could you write something like Shout if a
> > module cannot change the system text editor?

This is slightly ambiguous, do you (Lex) mean...
Switch the system text editor to an alternative system text editor?
Or modify the code of the existing system text editor?

>
> Err, by refactoring it in a way so that you can? We have already various
> abstractions (AppRegistry) and there is nothing wrong to have one that
> allows for syntax highlighting - with a common protocol that allows
> other clients to hook into it, instead of hacking the system text editor
> directly.

When installled, Shout registers PluggableShoutMorph as the default
MorphicTextEditor under AppRegistry.
PluggableShoutMorph has a 'styler' instance variable which holds an
SHTextStylerST80. When the text changes, the styler is instructed to parse, and
style, the Text.

Now, if the styler instance variable were in PluggableTextMorph (or higher), any
alternative syntax highlighter could be plugged in. It would just need to
implement the public interface of SHTextStylerST80...
    #style: <Text>
    #styleInBackgroundProcess:<Text>
    #format: <Text>
    #unstyledTextFrom: <Text>
.. and PluggableTextMorph would need to include the hacks (!?) from
PluggableShoutMorph to update its contents from the re-styled Text.

> Coincidentally, this is *precisely* what I referred to earlier
> wrt. explicit vs. implicit interactions. As it stands, loading an
> alternative syntax highlighter will (almost necessarily) clash with
> shout;

Not if it registers itself as the default (or an alternative) MorphicTextEditor.
Or, uses Shout, but implements an alternative to SHTextStylerST80.

(Incidently, I think it would be good to have a MorphicCodeEditor in addition to
MorphicTextEditor).

> with that interface both modules could use the same ground rules,
> could be loaded side by side and could be activated depending on the
> context you need them in.

Yes, that would be good.

Cheers,
Andy

Andrew Tween wrote:
> When installled, Shout registers PluggableShoutMorph as the default
> MorphicTextEditor under AppRegistry.

Sorry, I wasn't aware of that. In this case, this whole discussion is
fairly irrelevant (and I get Lex's point even less) since you aren't
even modifying the basic text editor but rather plugging into the
appropriate interface for it. No overrides needed, just a plain old API
being used. Nevermind.

Cheers,
   - Andreas