1 day left for Squeak elections!

On Feb 25, 2006, at 4:54 AM, Andreas Raab wrote:

> I think a discussion about how Monticello fits a working style that  
> can be used to maintain a live system is overdue by now. Having  
> been there, having seen the immense pain Monticello inflicts on  
> both sides of the maintenance chain (not only is it a pain for the  
> person doing the maintenance, it is also a pain for the person on  
> the receiving end of the maintenance) I think we can say with some  
> certainty that Monticello fails in this regard and that another  
> approach is needed.

Good idea.

First, let me mention that Monticello was designed with maintaining  
live system in mind. I think where it falls down is with scale. If  
you're working on a system with lots of packages, packages containing  
lots of code, packages with lots of ancestry, or with changes cutting  
across lots of packages, it does get slow. Also, we try to deal with  
some aspects of this slowness with caching, which can take up a lot  
of memory. So, yes speed is a problem for anything but small projects.

You also mentioned that "it doesn't work the right way for  
incremental migrations." The strategy Monticello uses it to compare  
the version being loaded with what is already in the image and make  
only the necessary changes. Are you finding that this is the wrong  
strategy for your needs? Or perhaps that Monticello isn't executing  
that strategy well?

I also agree that a different approach is required. I don't think the  
way ancestry information is modelled in Monticello will allow for  
much more optimization, so we'll have to change something  
fundamental. I have some ideas about the direction to take, but I'd  
be interested in hearing what you think would be useful.

Colin

As far as I can tell, there are several fundamental things that  
Squeak (and Smalltalk) have not yet gotten right, but could. Among  
these are namespaces and version dependencies. (One example of the  
latter is that resource A uses B and C, but B uses version 1 of D,  
and C uses version 2 of D.)

Until these are confronted, I feel that Monticello (and change sets)  
will always be "half a loaf:"  they do what they do - often well -  
but don't address the fundamental issues brought to the fore by  
evolution of large, complex, independently developed yet interrelated  
systems.

-H

On Feb 25, 2006, at 7:05 PM, Colin Putney wrote:

Aside from general tool irritations (basically morphic problems along  
with simple UI design failings) the only big annoyance I have with MC  
is the way that it violates a very longstanding idiom - so  
longstanding that I think you would find it encoded in my mRNA - of  
always running the class initialise method after loading changes to a  
class. I can't begin to think of the number of times I've been  
screwed by that.

Mostly I just find MC works. I'm sure it could be better but really,  
it basically just works.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Who is General Failure and why is he reading my disk?

Colin Putney wrote:
> First, let me mention that Monticello was designed with maintaining live
> system in mind. I think where it falls down is with scale.

Actually, I think there are some very fundamental issues when using
Monticello in a live system, e.g., a system which has live instances of
the classes being changed. The most important ones I can think about
(which have nothing to do with scale) are renaming classes and series of
class migrations.

For example, when you rename class Foo into Bar Monticello is utterly
unable to recognize this and will happily convert all of your existing
instances into ObsoleteFoo's. That's deadly in a live system. Similarly,
if you change the shape of a class multiple times and require
intermediate steps to update existing instances, Monticello will often
(by combining all the changes into a single one) wreck utter havoc upon
the system. Or take "reclassifications" (e.g., moving methods between
packages) - unless you take great care, methods may simply be removed.
Or doIts - many a time we need to do one-time fix-ups and those may only
be appropriate at a single point in the evolution of the live system
(e.g., cannot be handled in an initialize method). Monticello has no
good way of dealing with that. And I won't even go into dependencies
(e.g., you need change A before change B before change C) since this is
only partly a Monticello issue (but MC does make it hard to serialize
changes properly).

All of these have caused me lots of trouble and none of them has much to
do with scale. I think that the key issue is that because of the
snapshot approach that Monticello uses, it simply lacks information.
Information like whether class Foo has been renamed to Bar (contrary to
adding Bar and removing Foo), information like whether a method has been
moved out of a package vs. it being removed from the system. And since
it looses the intent of the change it needs to guess later on and it
just guesses wrong as often as it guesses right.

Interestingly, this information *is* available in change sets which is
really at the root of my claim that change sets are better suited to
maintain live systems. Speed, scale, cross-cutting changes are (albeit
important) secondary to these more fundamental issues.

> You also mentioned that "it doesn't work the right way for incremental
> migrations." The strategy Monticello uses it to compare the version
> being loaded with what is already in the image and make only the
> necessary changes. Are you finding that this is the wrong strategy for
> your needs? Or perhaps that Monticello isn't executing that strategy well?

It's really both. If I rename class Morph to FooMorph I cannot afford
Monticello to nuke all Morphs. If I move Object>>size between packages I
cannot afford Monticello to nuke that method just because I load an
updated version of the package that contained #size before (even if I
load the other package later chances are that my system has long died
since then).

> I also agree that a different approach is required. I don't think the
> way ancestry information is modelled in Monticello will allow for much
> more optimization, so we'll have to change something fundamental. I have
> some ideas about the direction to take, but I'd be interested in hearing
> what you think would be useful.

Well, it's a hard problem. One idea might be to actually record the
changes and use that in cases where it is relevant (e.g., remove vs.
classification etc). I'm certain there are issues with that approach but
it might be worthwhile to think about.

Cheers,
   - Andreas

They seem to be good at different things.  Monticello is all about  
maintaining packages and in my view is a much *SAFER* mechanism for  
loading in new functionality (packaged code) than changesets.  For  
one thing, it does a good job of preflighting the changes and if its  
going to have issues it will tell you and you can bail on it or it  
will help you get to the desired final state.  So as a librarian - MC  
is DA BOMB.

Changesets seem better for system level stuff that involves lots of  
refactoring.  As an example, I've been trying to consolidate all of  
the code that knows something about URL's and Http into a coherent  
package.   Its scattered all over the image in half a dozen packages  
so this change will involve many package name changes, class name  
changes, moving methods, etc.  Monticello seems to act on packages,  
so to capture this work and get it applied, I think a changeset is a  
better tool.

However, when it comes to maintaining modules and packages (chunks of  
capability that are a "THING"), I feel that changesets are AWFUL,  
crap out more often than not because my image isn't in the starting  
state the changeset was based on and changeset loading doesn't  
preflight well.  If it fails I have an image in an indeterminate  
state and have to quit and relaunch to get back the last saved (and  
sane) state.  I can't begin to count the number of times I've set out  
to load a .sar file and had it crap out because something in my image  
wasn't in the state that file expected.

As an exercise, try grabbing a 3.6 basic image and see if, using  
old .sar's on SqueakMap, you can get an image configured with Kom,  
Seaside, Postgres, and GLORP and let me know how many tries it takes  
you.

This is because changesets don't capture the expected initial and  
final state - only the transitions - and that is seldom enough when  
you're trying to work in a team and build a single "thing".

We really need a tool that knows both.

-Todd Blanchard

On Feb 25, 2006, at 11:08 PM, Andreas Raab wrote:

On Feb 25, 2006, at 11:08 PM, Andreas Raab wrote:
>
>
> Well, it's a hard problem. One idea might be to actually record the  
> changes and use that in cases where it is relevant (e.g., remove  
> vs. classification etc). I'm certain there are issues with that  
> approach but it might be worthwhile to think about.

It's a hard problem in general, but dealing properly with class  
renames isn't all that hard - we just need to track classes by some  
kind of unique ID (which we assign the first time we see them and  
maintain from then on) rather than by name.  This would be a very  
shallow change to MC, and even making it backwards compatible  
shouldn't be *that* hard.

Avi

On Feb 25, 2006, at 10:04 PM, tim Rowledge wrote:

> Aside from general tool irritations (basically morphic problems  
> along with simple UI design failings) the only big annoyance I have  
> with MC is the way that it violates a very longstanding idiom - so  
> longstanding that I think you would find it encoded in my mRNA - of  
> always running the class initialise method after loading changes to  
> a class. I can't begin to think of the number of times I've been  
> screwed by that.

MC runs the initialize method only when the initialize method itself  
changes.  This is trying to mimic the semantics of changesets: if you  
were distributing a changeset that included some methods on a class  
but not the #initialize method, it wouldn't be run then either.  But  
you're saying you want it to be run every time any method in the  
class changes?   Or maybe that all initialize methods in the package  
should be run any time the package changes?

Avi

Todd Blanchard wrote:
> They seem to be good at different things.  Monticello is all about
> maintaining packages and in my view is a much *SAFER* mechanism for
> loading in new functionality (packaged code) than changesets.  For one
> thing, it does a good job of preflighting the changes and if its going
> to have issues it will tell you and you can bail on it or it will help
> you get to the desired final state.  So as a librarian - MC is DA BOMB.

Agreed. As a programmer who is trying to build a system from first
principles MC is vastly superior to change sets.

> However, when it comes to maintaining modules and packages (chunks of
> capability that are a "THING"), I feel that changesets are AWFUL, crap
> out more often than not because my image isn't in the starting state the
> changeset was based on and changeset loading doesn't preflight well.  If
> it fails I have an image in an indeterminate state and have to quit and
> relaunch to get back the last saved (and sane) state.  I can't begin to
> count the number of times I've set out to load a .sar file and had it
> crap out because something in my image wasn't in the state that file
> expected.

It's a funny set of tradeoffs since I could counter this with pointing
out that it is now pretty much impossible for anybody to keep up-to-date
with the latest alphas. All of your changes will be wiped out by loading
that latest one-line fix which means you have to rebuild your image from
first principles whenever anything changes (e.g., all the freaking time
;-) I wonder how many people these days actively track 3.9alpha (if at
all) and how many of those do *any* of their own work in these images.
Personally, I stopped doing it after a few more important changes got
wiped clean, where (in the past) I've typically used the latest-greatest
to make sure  things get tested properly.

> As an exercise, try grabbing a 3.6 basic image and see if, using old
> .sar's on SqueakMap, you can get an image configured with Kom, Seaside,
> Postgres, and GLORP and let me know how many tries it takes you.

Yup. Agreed. And countered by: Try grabbing a 3.8 image and get it to
3.9 using Monticello alone ;-) And then use the update stream and see it
fail, too, because there are apparently some dependencies which need to
be carefully observed and have been condensed by Monticello in the wrong
form. [Disclaimer: Haven't tested this very recently; I tried it a
couple of weeks ago and got totally hosed].

> This is because changesets don't capture the expected initial and final
> state - only the transitions - and that is seldom enough when you're
> trying to work in a team and build a single "thing".
>
> We really need a tool that knows both.

Agreed as well.

Cheers,
   - Andreas

Avi Bryant wrote:
> It's a hard problem in general, but dealing properly with class renames
> isn't all that hard - we just need to track classes by some kind of
> unique ID (which we assign the first time we see them and maintain from
> then on) rather than by name.  This would be a very shallow change to
> MC, and even making it backwards compatible shouldn't be *that* hard.

Please do. This is one of the primary show-stoppers.

Cheers,
   - Andreas

Avi said:
"It's a hard problem in general, but dealing properly with class renames
isn't all that hard - we just need to track classes by some kind of unique
ID (which we assign the first time we see them and maintain from then on)
rather than by name.  This would be a very shallow change to MC, and even
making it backwards compatible shouldn't be *that* hard."

Harmony, the module/versioning tool I developed about ten years ago, did
exactly that: it tracked classes, Modules, Subsystems and Systems by GUID (a
System is roughly analogous to an Envy config map--it specified the complete
set of classes and methods that would be resident in the image.)

Harmony could also export either "absolute" or "relative" versions of
modules (or subsystems, or systems.)  The "relative" exports were somewhat
like changesets, except that they were generated by version difference
computations.

Harmony permitted the same methods and class definitions to reside in any
number of modules, the same modules to reside in any number of Subsystems,
and the same Subsystesm to reside in any number of Systems. An extension
method was simply any method in a module in which the method's class
definition did not reside.  Version conflicts were won by whichever module
got loaded last--but the losing modules would be marked as changed.
Conflicts were only possible when loading modules a la carte, instead of as
part of a containing Subsystem or System.

Harmony, unlike Envy, neither required nor used a central server. It was
designed to allow team members to work "offline," and enable them to merge
their work with that of others whenever convenient.

What Harmony didn't do was provide any namespace support. I now realize that
that was a big mistake (another big mistake was not providing very much
documentation.) In fact, were I to attempt such a project again, there's a
variety of things I would do differently, although there would proably be
some resemblance to the feature set I describe above.

Craig Latta's Naiad system is also based on GUIDs.  Craig and I had many
discussions about the architecture and design of Harmony a decade ago.

When you combine Namespaces with GUIDs, you get a synergistic effect that is
better and more powerful than the sum of its parts.  The effect is to give
classes and modules a kind of "object identity" AT DESIGN/CODE TIME that is
persistable and transportable across address spaces. This is a key,
fundamental benefit of the Naiad/Spoon architecture.  Go Craig!

--Alan

On Sun, 26 Feb 2006 01:02:05 -0800, Alan Lovejoy  
<[hidden email]> wrote:

> persistable and transportable across address spaces. This is a key,
> fundamental benefit of the Naiad/Spoon architecture.  Go Craig!

My sentiments exactly.

On Sun, 26 Feb 2006 01:02:05 -0800, "Alan Lovejoy"
<[hidden email]> wrote:

> Harmony, the module/versioning tool I developed about ten years ago

Now that brings back memories :-)

Alan, I remember working with you on a system that used Harmony, but I'll
be damned if I can remember where (or when) it was...

Later,
Jon
--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Seeker (Micro Autonomous Underwater Vehicle)
           http://www.huv.com

On 26-Feb-06, at 5:41 AM, Jon Hylands wrote:
Almost certainly Interval


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Fractured Idiom:- L'ETAT, C'EST MOO - I'm boss around here

On Sun, 26 Feb 2006 07:49:17 -0800, tim Rowledge <[hidden email]> wrote:

> Almost certainly Interval

Of course :-)

Silly me...

Later,
Jon
--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Seeker (Micro Autonomous Underwater Vehicle)
           http://www.huv.com

Hi andreas

This is clear that MC could be better to work on bootstrapping and  
changing a living system. This is not what we would dream about, and  
we would prefer something that has the property of MC + support  
change on live objects. This is why for 3.9 we have been forced to  
use MC as slow changesets. Since we cannot take a mini-image and load  
the latest versions (but we should load all the intermediate state).  
Ideally we would like to have a nano kernel and a load script but  
this is the future.

Now I think that MC is a step forward and that it should improve. I  
can also tell you that
postscript in cs are also a problem to analyze code and build tools.  
This is the question between a declarative code
representation and not. We got already these discussions in the past.

Stef