Proposal for the coming versions

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Proposal for the coming versions

Cees De Groot
Hi,

Needless to say, we've been talking a lot of Squeak at the Smalltalk
conference. Literally from early morning to early morning, on Saturday
:-).

When discussing the immediate future, I promised to write up a small
proposal for the coming versions. Here it is - shoot at it:

Version 3.10 - Implement the Process
===========================
After 3.9, I think it would be a good idea to work on a version that
has as its main goal to implement the community process we have in
place (user -> mantis -> package team -> inbox -> release team,
roughly). The idea is to enhance SqueakMap with Team cards, to link
into Mantis from the image, etcetera, so that at least User and
Package Team can work with "single button presses". Some typical use
cases:

User: "Submit bug report". SqueakMap is used to find out what team
maintains the code that the user has patched. The team card in SM can
have stuff like the name of the Mantis category matching the Team's
bugs, the format the team prefers (changeset, .mcz or .mcd file, ...),
the place to send it to (a mail address, an MC repository, ...),
etcetera. The user gets a simple form containing most of the stuff
Mantis also asks for, and then the package submits the bug to Mantis.
As simple as the old method of mailing a bug or patch to the list, but
asking for more details in order to raise the quality of bug reports
and patches.

Team member: "Work as member of...". The SM card is fetched, and from
here maybe a little control panel is created that can fetch the next
bug report from Mantis ("Next" being defined as some heuristic over
age and severity, for example), automagically file in any attached
patch, and when done does a sort of transaction over mantis and the
team repository, closing the bug and committing to the team repository
with links in both Mantis and MC to each other.

Also, 3.10 could be the first version that is developed update stream-less.

Version 4.0: Brave New World
======================
After 3.10, and probably starting right away given the length of time
discussions are likely to take place ;-), version 4.0 will especially
integrate changes that will effect image format. Whatever they may be
- there's a long list of wishes, and the team lead should first and
foremost come up with a community process to turn the list of wishes
into a todo list.

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Edgar J. De Cleene
Cees De Groot puso en su mail :

> Also, 3.10 could be the first version that is developed update stream-less.
Cees, all seems clear except this point, could extend your concept ?



       
       
               
___________________________________________________________
1GB gratis, Antivirus y Antispam
Correo Yahoo!, el mejor correo web del mundo
http://correo.yahoo.com.ar 


Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Cees de Groot-2
On Mon, 13 Mar 2006 09:01:04 -0300, "Lic. Edgar J. De Cleene"
<[hidden email]> wrote:

>Cees De Groot puso en su mail :
>
>> Also, 3.10 could be the first version that is developed update stream-less.
>Cees, all seems clear except this point, could extend your concept ?
>
The conclusion from 3.9 development so far seems to be that combining
MC's model with the model of the update stream is painful. Because the
"upstream" code is managed and therefore normally delivered to the
release team by means of MC, removing MC from the picture is
difficult. Which means that the update stream must be removed.

My proposal is to not work with an update stream during "fast
development" - that is certainly up to and including the alpha stage,
but probably also up to and including the beta stage. After that, the
version is frozen (more so than until now - 3.8 saw entirely new
features rather than just bugfixes during its lifetime), and the
patches that need to be applied can be fed through the update stream
mechanism again.



Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

stéphane ducasse-2
In reply to this post by Cees De Groot
> Hi,
>
> Needless to say, we've been talking a lot of Squeak at the Smalltalk
> conference. Literally from early morning to early morning, on Saturday
> :-).

:)

Indeed I would like to see a much better process for 3.10.

Stef

> When discussing the immediate future, I promised to write up a small
> proposal for the coming versions. Here it is - shoot at it:
>
> Version 3.10 - Implement the Process
> ===========================
> After 3.9, I think it would be a good idea to work on a version that
> has as its main goal to implement the community process we have in
> place (user -> mantis -> package team -> inbox -> release team,
> roughly). The idea is to enhance SqueakMap with Team cards, to link
> into Mantis from the image, etcetera, so that at least User and
> Package Team can work with "single button presses". Some typical use
> cases:
>
> User: "Submit bug report". SqueakMap is used to find out what team
> maintains the code that the user has patched. The team card in SM can
> have stuff like the name of the Mantis category matching the Team's
> bugs, the format the team prefers (changeset, .mcz or .mcd file, ...),
> the place to send it to (a mail address, an MC repository, ...),
> etcetera. The user gets a simple form containing most of the stuff
> Mantis also asks for, and then the package submits the bug to Mantis.
> As simple as the old method of mailing a bug or patch to the list, but
> asking for more details in order to raise the quality of bug reports
> and patches.
>
> Team member: "Work as member of...". The SM card is fetched, and from
> here maybe a little control panel is created that can fetch the next
> bug report from Mantis ("Next" being defined as some heuristic over
> age and severity, for example), automagically file in any attached
> patch, and when done does a sort of transaction over mantis and the
> team repository, closing the bug and committing to the team repository
> with links in both Mantis and MC to each other.
>
> Also, 3.10 could be the first version that is developed update  
> stream-less.
>
> Version 4.0: Brave New World
> ======================
> After 3.10, and probably starting right away given the length of time
> discussions are likely to take place ;-), version 4.0 will especially
> integrate changes that will effect image format. Whatever they may be
> - there's a long list of wishes, and the team lead should first and
> foremost come up with a community process to turn the list of wishes
> into a todo list.
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Andreas.Raab
In reply to this post by Cees de Groot-2
Cees de Groot wrote:
> The conclusion from 3.9 development so far seems to be that combining
> MC's model with the model of the update stream is painful. Because the
> "upstream" code is managed and therefore normally delivered to the
> release team by means of MC, removing MC from the picture is
> difficult. Which means that the update stream must be removed.

Interesting. I'm not sure I buy that conclusion. Because, while I agree
that using MC with the update stream model is painful, it seems that the
cause of that pain is purely and unequivocally Monticello itself. The
update stream is already used as a last-resort kind of thing but
unfortunately (as discussed in previous messages) Monticello is unable
to deal with a multitude of issues that arise in system evolution. So
I'm really curious as to how you think removal of the update stream
would reduce the pain all by itself.

> My proposal is to not work with an update stream during "fast
> development" - that is certainly up to and including the alpha stage,
> but probably also up to and including the beta stage. After that, the
> version is frozen (more so than until now - 3.8 saw entirely new
> features rather than just bugfixes during its lifetime), and the
> patches that need to be applied can be fed through the update stream
> mechanism again.

If we apply this logic to 3.9 (just to understand the model you're
proposing) would this mean we'd still be in a situation where a user
who'd want to have a look at the latest alpha would effectively go
through a process by which she downloads a 3.8 image, figures out in
which order to load the latest (or new/removed) packages and does that
all by herself? Have you tried that lately? ;-)

[BTW, don't get me wrong here, I'd really *love* to see a model that
says "get 3.8 load some packages, have 3.9" work; this isn't a critique
of the idea just a little reality check since I've tried to work in such
a way for quite some time now and it's painfully obvious how much MC
lacks in this regard].

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Cees de Groot-2
On Mon, 13 Mar 2006 10:13:38 -0800, Andreas Raab <[hidden email]>
wrote:
>Because, while I agree
>that using MC with the update stream model is painful, it seems that the
>cause of that pain is purely and unequivocally Monticello itself.

Personally, I would point my finger at the use of MC instead of MC
itself (MC seems to be not fit for a particular purpose its creators
probably never reckoned with). Anyway, I wasn't assigning "blame", I
was just remarking that there was a conflict here.

>If we apply this logic to 3.9 (just to understand the model you're
>proposing) would this mean we'd still be in a situation where a user
>who'd want to have a look at the latest alpha would effectively go
>through a process by which she downloads a 3.8 image, figures out in
>which order to load the latest (or new/removed) packages and does that
>all by herself? Have you tried that lately? ;-)

Nope. My idea is that during "fast development", the team leader just
publishes images. And probably an MCC map or something similar showing
how the image is put together. So the user would effectively have to
download the image to hop across major upgrades. Minor upgrades would
likely be doable by just udating using the MCC map. For some value of
"major" and "minor", of course :)

Yes, that's bad (for people with low bandwidth, for starters).
However, it buys time to fix the issues with Monticello, so that
hopefully later on a "upgrade v.X to v.Y" button can be reinstated.


Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Andreas.Raab
Cees de Groot wrote:
> Personally, I would point my finger at the use of MC instead of MC
> itself (MC seems to be not fit for a particular purpose its creators
> probably never reckoned with).

According to Colin "Monticello was designed with maintaining live system
in mind" (see
http://lists.squeakfoundation.org/pipermail/squeak-dev/2006-February/101067.html)

> Anyway, I wasn't assigning "blame", I
> was just remarking that there was a conflict here.

Me neither. As I've said in the past, MC is good for what MC is good for
(namely managing source code in a "build" environment) but it strikes me
that you're proposing to apply it in just the way where it doesn't work
very well. But of course, that's just my experience. YMMV.

> Nope. My idea is that during "fast development", the team leader just
> publishes images. And probably an MCC map or something similar showing
> how the image is put together. So the user would effectively have to
> download the image to hop across major upgrades. Minor upgrades would
> likely be doable by just udating using the MCC map. For some value of
> "major" and "minor", of course :)

Um ... I might be slow or something but can you explain to me again how
this makes things simpler/easier/whatever? Where is (in terms of pain
associated with it) the difference between having that MCC map and
posting it to an update stream? (unless you don't do it at all ;-)

And using images this way ... uh, oh. I have a *really* bad feeling
about this one. You're right, it's often easier to do things in your
image and distribute that image. It's also a sure way of making it
impossible for anyone to track ongoing development, the release
progress, and the history of how things evolved in the image. All of
which are (in my understanding) very desirable properties. I fear that
this will lead to a process where you hand off "the image vX.Y" to a
small group of people (possibly one) and get back "the image vX.Y+1" and
hardly anyone will have looked at anything at all, noticed the oddities,
which clean ups are required along the way etc. Not saying that many
people today track the changes very closely but I even that seems
preferrable to a process where you don't get to see anything and have no
documentation of the changes the "official" image went through.

> Yes, that's bad (for people with low bandwidth, for starters).
> However, it buys time to fix the issues with Monticello, so that
> hopefully later on a "upgrade v.X to v.Y" button can be reinstated.

Nice try. You know just as well as I do that this ain't gonna happen ;-)
If only because when you give people the ability to modify an image "at
home" they'll use it for *precisely* those changes that are hard to deal
with in Monticello. Otherwise, why not use Monticello to begin with?

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Colin Putney

On Mar 13, 2006, at 2:59 PM, Andreas Raab wrote:

> Cees de Groot wrote:
>> Personally, I would point my finger at the use of MC instead of MC
>> itself (MC seems to be not fit for a particular purpose its creators
>> probably never reckoned with).
>
> According to Colin "Monticello was designed with maintaining live  
> system in mind" (see http://lists.squeakfoundation.org/pipermail/ 
> squeak-dev/2006-February/101067.html)

Yeah. I probably shouldn't have stated it that strongly. Our "live  
systems" at the time were web apps with data stored in an external  
database. So if by "maintaining live systems" you mean updating the  
code without out taking down the app, yeah. If you also mean "and  
migrating in-image objects transparently," well no, not quite.

On the other hand, truly "live systems" are a big part of what this  
community does, and I personally would like to see Monticello do a  
better job of serving that part of the community. As Avi mentioned,  
Monticello2 has been moving very, very slowly forward for the last  
year (2 years is a bit of an exaggeration). I honestly think, though,  
that it's the best vehicle for addressing these issues. I'm going to  
keep pushing it forward, hopefully making faster progress than I have  
been.

Colin

Reply | Threaded
Open this post in threaded view
|

RE: Proposal for the coming versions

Peter Crowther-2
In reply to this post by Cees De Groot
> From: Cees de Groot
> So the user would effectively have to
> download the image to hop across major upgrades.

So how do I carry my customised image content over between versions?  Am
I supposed to export it and re-import it?  Is everyone?

                - Peter

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Diego Gomez Deck
In reply to this post by Colin Putney
Hi folks,

What about preambles and postscripts in versions of Monticello-packages?

Let's state an example: We have a package named Foo, with the versions
Foo1, Foo2, Foo3, Foo4 and Foo5.

Foo3 and Foo5 versions have preamble and/or postscript.

If you have loaded Foo1 and you want to update to Foo5, version Foo3 has
to be loaded before loading Foo5.

This solution will force to load the intermediate versions where a
preamble/postscript exists, but it's not a big problem.

We need to load the intermediate versions to be sure the
preamble/postscript execute in a predictable way.

BTW This proposal don't include a fix for the class-renaming problem.

Cheers,

-- Diego


> On Mar 13, 2006, at 2:59 PM, Andreas Raab wrote:
>
> > Cees de Groot wrote:
> >> Personally, I would point my finger at the use of MC instead of MC
> >> itself (MC seems to be not fit for a particular purpose its creators
> >> probably never reckoned with).
> >
> > According to Colin "Monticello was designed with maintaining live  
> > system in mind" (see http://lists.squeakfoundation.org/pipermail/ 
> > squeak-dev/2006-February/101067.html)
>
> Yeah. I probably shouldn't have stated it that strongly. Our "live  
> systems" at the time were web apps with data stored in an external  
> database. So if by "maintaining live systems" you mean updating the  
> code without out taking down the app, yeah. If you also mean "and  
> migrating in-image objects transparently," well no, not quite.
>
> On the other hand, truly "live systems" are a big part of what this  
> community does, and I personally would like to see Monticello do a  
> better job of serving that part of the community. As Avi mentioned,  
> Monticello2 has been moving very, very slowly forward for the last  
> year (2 years is a bit of an exaggeration). I honestly think, though,  
> that it's the best vehicle for addressing these issues. I'm going to  
> keep pushing it forward, hopefully making faster progress than I have  
> been.
>
> Colin



Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Hans-Martin Mosner
In reply to this post by Peter Crowther-2
Peter Crowther wrote:

>>From: Cees de Groot
>>So the user would effectively have to
>>download the image to hop across major upgrades.
>>    
>>
>
>So how do I carry my customised image content over between versions?  Am
>I supposed to export it and re-import it?  Is everyone?
>  
>
For major upgrades, I'd suggest that this is the way to go.
When significant things change (such as redefinitions of major class
hierarchies, or modularization of the image, etc), providing an
automatic system which re-enacts this change in arbitrary images is
almost impossible.
Sometimes it works (that's where the upgrade stream gets used),
sometimes it won't.
Where it does not work, exporting code into fileouts or repositories
normally works pretty well, and saving content as project files is not
too bad either.

Cheers,
Hans-Martin

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Daniel Vainsencher-2
In reply to this post by Andreas.Raab
Hi everyone.

I think improving the process for those doing the development is
important, but agree with Andreas that the current proposal sounds like
we're giving up on users ability to update their images. If that's what
we want, we should just say so.

If we're going to go for a better system that has the benefits of MC and
supports image updates, we need to actually catalog, discuss, and solve
those nitty gritty problems that bug both sides of the development
process at the moment. And this will be a lot of work, and we can't
expect the MC guys to just solve it all for us, even if MC2 solves the
code issues, including performance and class-renames.

Before continuing with the bad news, just wanted to remind ourselves
that systematic treatment of live system updates is possible at least in
the sense that GemStone does it.

Here is a partial list of problems I met while working on the updates
for Traits:
- Having made some changes, I should know whether these are loadable
into a published image. This currently requires manually posting,
opening new image, loading. Some test server seems useful in this context.
- Doing live updates correctly requires capturing a sequence of steps
which know enough each about the previous state to keep things running
correctly. MC by itself does not model this at all. An external tool
(ScriptLoader, IIRC, in the current process) can capture this. But in
addition to capturing update sequences, we need to be able to merge
them. For example, suppose we and Squeakland create two unrelated update
sequences for overlapping sets of packages, starting from common state
A, going to B and C respectively, through B1, B2... and C1, C2... The
"did-both-updates" state D needs to be created (probably via MC merge),
but what are the stage D1,D2...? It might well be that the merge problem
needs to be solved manually, rather than automatically, but we
definitely need a better way to detect it, to capture the sequences, and
to change from one sequence to the other when we find problems
(currently doable only by updating the updates-list file, AFAIK).
- Recompiling the whole image is very slow. Therefore changing the shape
of many classes (in that example, all metaclasses) makes updates take a
very long time.
- It is hard to make some system level changes because none of the
existing tools in the image document all the VM assumptions. Therefore,
that bug from changing the shape of Behaviors that prevents some
primitives from running.

Daniel Vainsencher

Andreas Raab wrote:

> Cees de Groot wrote:
>
>> Personally, I would point my finger at the use of MC instead of MC
>> itself (MC seems to be not fit for a particular purpose its creators
>> probably never reckoned with).
>
>
> According to Colin "Monticello was designed with maintaining live
> system in mind" (see
> http://lists.squeakfoundation.org/pipermail/squeak-dev/2006-February/101067.html)
>
>
>> Anyway, I wasn't assigning "blame", I
>> was just remarking that there was a conflict here.
>
>
> Me neither. As I've said in the past, MC is good for what MC is good
> for (namely managing source code in a "build" environment) but it
> strikes me that you're proposing to apply it in just the way where it
> doesn't work very well. But of course, that's just my experience. YMMV.
>
>> Nope. My idea is that during "fast development", the team leader just
>> publishes images. And probably an MCC map or something similar showing
>> how the image is put together. So the user would effectively have to
>> download the image to hop across major upgrades. Minor upgrades would
>> likely be doable by just udating using the MCC map. For some value of
>> "major" and "minor", of course :)
>
>
> Um ... I might be slow or something but can you explain to me again
> how this makes things simpler/easier/whatever? Where is (in terms of
> pain associated with it) the difference between having that MCC map
> and posting it to an update stream? (unless you don't do it at all ;-)
>
> And using images this way ... uh, oh. I have a *really* bad feeling
> about this one. You're right, it's often easier to do things in your
> image and distribute that image. It's also a sure way of making it
> impossible for anyone to track ongoing development, the release
> progress, and the history of how things evolved in the image. All of
> which are (in my understanding) very desirable properties. I fear that
> this will lead to a process where you hand off "the image vX.Y" to a
> small group of people (possibly one) and get back "the image vX.Y+1"
> and hardly anyone will have looked at anything at all, noticed the
> oddities, which clean ups are required along the way etc. Not saying
> that many people today track the changes very closely but I even that
> seems preferrable to a process where you don't get to see anything and
> have no documentation of the changes the "official" image went through.
>
>> Yes, that's bad (for people with low bandwidth, for starters).
>> However, it buys time to fix the issues with Monticello, so that
>> hopefully later on a "upgrade v.X to v.Y" button can be reinstated.
>
>
> Nice try. You know just as well as I do that this ain't gonna happen
> ;-) If only because when you give people the ability to modify an
> image "at home" they'll use it for *precisely* those changes that are
> hard to deal with in Monticello. Otherwise, why not use Monticello to
> begin with?
>
> Cheers,
>   - Andreas
>


Reply | Threaded
Open this post in threaded view
|

RE: Proposal for the coming versions

Alan L. Lovejoy
In reply to this post by Colin Putney


Colin said:

"Yeah. I probably shouldn't have stated it that strongly. Our "live systems"
at the time were web apps with data stored in an external database. So if by
"maintaining live systems" you mean updating the code without out taking
down the app, yeah. If you also mean "and migrating in-image objects
transparently," well no, not quite.

On the other hand, truly "live systems" are a big part of what this
community does, and I personally would like to see Monticello do a better
job of serving that part of the community. "

A module/version manager such as Monticello can't by itself deal with every
issue that might arise in "hot loading" a new version of a module (or module
graph) on top of an already-resident version.  Either the classes in the
modules have to be designed to handle that, or else some sort of contraint
system must be used to ensure that the contraints declaratively specified
for each version of a class or module are satisfied (such as certain
instance variables not being nil.) Actually, the same is true of using
change sets: the change set has to include whatever code (and "do its") are
necessary to fix up whatever issues may be created by the change to the live
system.

A lot (but certainly not all) of the pain people experience when using
module/version management systems is ultimately due to the fact that the
classes they are changing weren't designed to properly interact with the way
module/version managers do business.  And the programmers may not be used to
the module/version manager paradigm. The "change model" is fundamentally
different.

In the change set paradigm, classes are sent the message #initialize when
first loaded.  Change sets may need to send the #initialize message to one
or more already-resident classes, and may need to send other messages to
either classes or other non-class instances.  It is the responsibility of
the change set author to ensure that the necessary "do its" are included,
and that they get executed at the appropriate moment.  Of course, there is
an O(n^2) issue here, in that the "repair logic" may need to be different
for every combination of source and targert version.

In the module/version manager paradigm, one must not only send #initialize
(or whatever the canonical message name may be) to classes when first
loaded, one must also be able to canonically inform each class, each module
and each meta-module (if such exist) that it has been (and is about to be)
changed to a new version, so that whatever "fix up" procedure may be
necessary can be executed. The "version change repair" logic may need to
differ for each version of each class, each module and each meta-module.

So the "change model" when using a module/version manager is fundamentally
different than the one associated with change sets, and requires
architectural and design differences in the code being "managed."  It puts
the responsibility for dealing with "live updates" squarely on the shoulders
of the authors/maintainers of the code, and does not permit the issue to be
just an afterthought that is only dealt with when constructing a change set.

In my experience, one can handle all the same sorts of live updates using
the module/version manager paradigm that one can handle when using change
sets--but only in conjuction with the appropriate architecture and design of
the application/library/system code under management.

--Alan






Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Cees de Groot-2
In reply to this post by Andreas.Raab
On Mon, 13 Mar 2006 11:59:16 -0800, Andreas Raab <[hidden email]>
wrote:

>Um ... I might be slow or something but can you explain to me again how
>this makes things simpler/easier/whatever? Where is (in terms of pain
>associated with it) the difference between having that MCC map and
>posting it to an update stream? (unless you don't do it at all ;-)
>
Well... in the update stream, you need to take care about load order,
about conversion scripts, stuff like that. Just a map won't be enough
to upgrade X to Y. For example, loading your UI work (ToolBuilder
etcetera) was relatively easy, not in the least because it was
well-prepared and well-documented. Loading it in a way that was
perfectly reproducable in an automatic way (ie "scriptable") took
quite a bit of time, it had to be split up over multiple update stream
versions, so it was a general pain to take that extra step.

>Nice try. You know just as well as I do that this ain't gonna happen ;-)

It depends. Personally, I don't think it is fair to burden the release
team with lots of work that maybe only others care about. That's why
I'm trying to find a process (and - remember the original mail -
implement that in 3.10) that minimizes the burden on the package and
especially release teams. If people complain that this won't satisfy
their crave for software archeology, let them step up and do something
about it (yup, exaggerating there - the points you point out are valid
but are they important enough?).

Anyway, I'm not married to the specifics of my proposal - I feel the
idea to implement the process is important, but how the technical
details turn out bothers me less, as long as the decision framework is
to err on the side of making life of the team members easy.

As far quality - there are many ways to ensure quality, an easy
overview of changes to the image is just one of them. It is another
thing we chatted on in Brussels (I don't mention with whom - it might
give the wrong impression that they're backing any of this ;-)). We
need to be more quality-conscious (for starters, we MUST - yes, that's
the RFC "MUST" - have all-green unit tests in every released image).
However, if the process+tools eats up all the time just to get stuff
done, there's nothing left to do about quality. In that case, it's
probably better to drop some functionality and buy time in exchange.


Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Cees de Groot-2
In reply to this post by Daniel Vainsencher-2
On Mon, 13 Mar 2006 23:27:04 +0200, Daniel Vainsencher
<[hidden email]> wrote:
>[...] the current proposal sounds like
>we're giving up on users ability to update their images.

In case I wasn't clear enough - yes, we are. At least for the
development cycle, as far as I'm concerned not for the maintenance
cycle.


>
>If we're going to go for a better system that has the benefits of MC and
>supports image updates, we need to actually catalog, discuss, and solve
>those nitty gritty problems that bug both sides of the development
>process at the moment.

Yes, we do. But that shouldn't halt the ongoing development of Squeak.
That's why I propose some emergency measures to keep it moving, while
at the same time we see how to make MC better suited to the job (or
write a new tool specifically for this sort of work, whatever).

Could you explain a bit about the Gemstone model you are referring to?
I only have marginal GS exposure, and I'm sure that I'm not the only
one.


Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Adrian Lienhard
In reply to this post by Daniel Vainsencher-2
Hi

It would be valuable to precisely identify the problems of MC because  
in my experience it works very well for 80% of the cases (which we  
can't say for change sets -- at least what I use MC for). So, the  
goal for 3.10 (or for a dedicated "task force") would be to improve  
those shortcomings.

I see three main areas of problems:

a) performance (although not necessarily a show stopper the slow  
speed makes working with MC for non-trivial projects quite annoying)

b) load order problems (I guess, an atomic load would solve all those  
problems):
(i) i.e., moving a class from package A to B, or circular  
dependencies between packages
(ii) modifying code that is used while loading itself

c) "live updates": upgrading state to the new version of the system,  
a task that change sets and the update stream was really good at but  
that MC does not support explicitly. Here we could argue that this is  
not MC's business, in which case we should find another solution that  
is less ad-hoc than the ScriptLoader as used in 3.9a now.

- other stuff: method overrides don't work correctly and extending  
class definition would be nice too (e.g., in package A add an  
instance variable to a class that is defined in package B).

Although MC2 is supposed to solve many of those problems, I don't  
think it is a good strategy to wait and leave those issues unfixed.  
Since MC is in the critical path of the development of Squeak, I  
think we should rather start today to fix what stands in our way.

Cheers,
Adrian

PS: for b) there seems to be a solution already but I wonder why this  
has never found any big attention, nor has been incorporated into MC  
in 3.9? Citing Colin: "SystemEditor is a small package that provides  
way to execute a series of change to the system atomically. It's  
going to be a critical part of Monticello2, and in the meantime, I  
think it would be valuable to update Monticello1 to use it as well."


On Mar 13, 2006, at 22:27 , Daniel Vainsencher wrote:

> Hi everyone.
>
> I think improving the process for those doing the development is  
> important, but agree with Andreas that the current proposal sounds  
> like we're giving up on users ability to update their images. If  
> that's what we want, we should just say so.
>
> If we're going to go for a better system that has the benefits of  
> MC and supports image updates, we need to actually catalog,  
> discuss, and solve those nitty gritty problems that bug both sides  
> of the development process at the moment. And this will be a lot of  
> work, and we can't expect the MC guys to just solve it all for us,  
> even if MC2 solves the code issues, including performance and class-
> renames.
>
> Before continuing with the bad news, just wanted to remind  
> ourselves that systematic treatment of live system updates is  
> possible at least in the sense that GemStone does it.
>
> Here is a partial list of problems I met while working on the  
> updates for Traits:
> - Having made some changes, I should know whether these are  
> loadable into a published image. This currently requires manually  
> posting, opening new image, loading. Some test server seems useful  
> in this context.
> - Doing live updates correctly requires capturing a sequence of  
> steps which know enough each about the previous state to keep  
> things running correctly. MC by itself does not model this at all.  
> An external tool (ScriptLoader, IIRC, in the current process) can  
> capture this. But in addition to capturing update sequences, we  
> need to be able to merge them. For example, suppose we and  
> Squeakland create two unrelated update sequences for overlapping  
> sets of packages, starting from common state A, going to B and C  
> respectively, through B1, B2... and C1, C2... The "did-both-
> updates" state D needs to be created (probably via MC merge), but  
> what are the stage D1,D2...? It might well be that the merge  
> problem needs to be solved manually, rather than automatically, but  
> we definitely need a better way to detect it, to capture the  
> sequences, and to change from one sequence to the other when we  
> find problems (currently doable only by updating the updates-list  
> file, AFAIK).
> - Recompiling the whole image is very slow. Therefore changing the  
> shape of many classes (in that example, all metaclasses) makes  
> updates take a very long time.
> - It is hard to make some system level changes because none of the  
> existing tools in the image document all the VM assumptions.  
> Therefore, that bug from changing the shape of Behaviors that  
> prevents some primitives from running.
>
> Daniel Vainsencher
>
> Andreas Raab wrote:
>
>> Cees de Groot wrote:
>>
>>> Personally, I would point my finger at the use of MC instead of MC
>>> itself (MC seems to be not fit for a particular purpose its creators
>>> probably never reckoned with).
>>
>>
>> According to Colin "Monticello was designed with maintaining live  
>> system in mind" (see http://lists.squeakfoundation.org/pipermail/ 
>> squeak-dev/2006-February/101067.html)
>>
>>> Anyway, I wasn't assigning "blame", I
>>> was just remarking that there was a conflict here.
>>
>>
>> Me neither. As I've said in the past, MC is good for what MC is  
>> good for (namely managing source code in a "build" environment)  
>> but it strikes me that you're proposing to apply it in just the  
>> way where it doesn't work very well. But of course, that's just my  
>> experience. YMMV.
>>
>>> Nope. My idea is that during "fast development", the team leader  
>>> just
>>> publishes images. And probably an MCC map or something similar  
>>> showing
>>> how the image is put together. So the user would effectively have to
>>> download the image to hop across major upgrades. Minor upgrades  
>>> would
>>> likely be doable by just udating using the MCC map. For some  
>>> value of
>>> "major" and "minor", of course :)
>>
>>
>> Um ... I might be slow or something but can you explain to me  
>> again how this makes things simpler/easier/whatever? Where is (in  
>> terms of pain associated with it) the difference between having  
>> that MCC map and posting it to an update stream? (unless you don't  
>> do it at all ;-)
>>
>> And using images this way ... uh, oh. I have a *really* bad  
>> feeling about this one. You're right, it's often easier to do  
>> things in your image and distribute that image. It's also a sure  
>> way of making it impossible for anyone to track ongoing  
>> development, the release progress, and the history of how things  
>> evolved in the image. All of which are (in my understanding) very  
>> desirable properties. I fear that this will lead to a process  
>> where you hand off "the image vX.Y" to a small group of people  
>> (possibly one) and get back "the image vX.Y+1" and hardly anyone  
>> will have looked at anything at all, noticed the oddities, which  
>> clean ups are required along the way etc. Not saying that many  
>> people today track the changes very closely but I even that seems  
>> preferrable to a process where you don't get to see anything and  
>> have no documentation of the changes the "official" image went  
>> through.
>>
>>> Yes, that's bad (for people with low bandwidth, for starters).
>>> However, it buys time to fix the issues with Monticello, so that
>>> hopefully later on a "upgrade v.X to v.Y" button can be reinstated.
>>
>>
>> Nice try. You know just as well as I do that this ain't gonna  
>> happen ;-) If only because when you give people the ability to  
>> modify an image "at home" they'll use it for *precisely* those  
>> changes that are hard to deal with in Monticello. Otherwise, why  
>> not use Monticello to begin with?
>>
>> Cheers,
>>   - Andreas
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Andreas.Raab
Adrian Lienhard wrote:
> It would be valuable to precisely identify the problems of MC because in
> my experience it works very well for 80% of the cases (which we can't
> say for change sets -- at least what I use MC for). So, the goal for
> 3.10 (or for a dedicated "task force") would be to improve those
> shortcomings.

Start here:
http://lists.squeakfoundation.org/pipermail/squeak-dev/2006-February/101071.html

If you can fix the issues listed there (class renames, incremental shape
modifications, migrations across these different versions, moving
classes/methods between packages, integration of system-wide doIts, load
order dependencies and all possible combinations of these) then I'd call
MC functionally complete to deal with full system maintenance. It'd
still be slow but at least a complete system maintenance tool. I'll stop
short of the full list, if you can show how one solution subsumes
another where possible.


Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Andreas.Raab
In reply to this post by Cees de Groot-2
Cees de Groot wrote:

>> Um ... I might be slow or something but can you explain to me again how
>> this makes things simpler/easier/whatever? Where is (in terms of pain
>> associated with it) the difference between having that MCC map and
>> posting it to an update stream? (unless you don't do it at all ;-)
>>
> Well... in the update stream, you need to take care about load order,
> about conversion scripts, stuff like that. Just a map won't be enough
> to upgrade X to Y. For example, loading your UI work (ToolBuilder
> etcetera) was relatively easy, not in the least because it was
> well-prepared and well-documented. Loading it in a way that was
> perfectly reproducable in an automatic way (ie "scriptable") took
> quite a bit of time, it had to be split up over multiple update stream
> versions, so it was a general pain to take that extra step.

I still don't quite get what you're saying. Was there any "extra effort"
associated with "making it perfectly reproducable in an automatic way"
(I'm quoting it because I don't understand what was involved in this
process) other than simply posting a "load this package" command? In
other words, what I don't understand (or don't know about) is what extra
effort was needed to get the packages in a form suitable for the update
stream?

>> Nice try. You know just as well as I do that this ain't gonna happen ;-)
>
> It depends. Personally, I don't think it is fair to burden the release
> team with lots of work that maybe only others care about. That's why

"only others care about" - "others" like in "customers"? I'd be careful
not to underestimate the interest of the casual Squeaker to just check
out what's going on in the latest versions. Not necessarily always
riding on the latest alpha but being able to "check in and see" where
things are.

> I'm trying to find a process (and - remember the original mail -
> implement that in 3.10) that minimizes the burden on the package and
> especially release teams. If people complain that this won't satisfy
> their crave for software archeology, let them step up and do something
> about it (yup, exaggerating there - the points you point out are valid
> but are they important enough?).

Good question. It is important to put the options squarely on the table
though, and in this case (and both Daniel and you in a response to
Daniel's post agree on that) it really means that this proposal is
essentially the end of the ability for anyone to update their images
automatically (which is really behind my comment, e.g., how likely is it
that somebody will retrace the steps after the fact to make it loadable).

Whether that is "important enough" or not is a good question. One thing
I believe is "important enough" however is the documentation aspect of
being able to update on your own. The main value here is that anyone can
review what changes have been done in any version and can follow along
and see where things stand, what has been changed and how. That's not to
say that the update stream is the only way to deal with that but given
the diverse set of people dealing with different releases I think it's
critical to have the process documented in some form. If the way in
which image vX.Y+1 was derived from vX.Y is unclear I think we're in for
trouble.

Cheers,
   - Andreas


Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Daniel Vainsencher-2
When we ask how important the update stream is, I think we need to
separate two questions:
-How important is it to be able to update every image forever across
versions?
-How important is it to be able to update most of the time, and only get
a new image once or twice per release when things break?

The latter is enough for the "keeping up with current events". The
former is what we need for "I don't care if I can export/import my
objects, I'll just upgrade the image" or for creating "history images"
which have full versions and so forth.

This is a scenario I worry about, and maybe I shouldn't: how should we
handle things like the Traits transition? if all we do is put out a new
image, then for (for example) Croquet to get Traits, for example, they
need to either do the same process themselves (but we didn't require
anyone to document the process, just put out a result) or take the new
squeak-dev image and turn it into Croquet.

Adrian and I worked pretty hard to make the Traits transition
reproducable, so it can be redone in other projects. So this is
possible, but it is sometimes a lot of work.

Is this issue important enough to do this work?

Daniel

Andreas Raab wrote:

> Cees de Groot wrote:
>
>>> Um ... I might be slow or something but can you explain to me again
>>> how this makes things simpler/easier/whatever? Where is (in terms of
>>> pain associated with it) the difference between having that MCC map
>>> and posting it to an update stream? (unless you don't do it at all ;-)
>>>
>> Well... in the update stream, you need to take care about load order,
>> about conversion scripts, stuff like that. Just a map won't be enough
>> to upgrade X to Y. For example, loading your UI work (ToolBuilder
>> etcetera) was relatively easy, not in the least because it was
>> well-prepared and well-documented. Loading it in a way that was
>> perfectly reproducable in an automatic way (ie "scriptable") took
>> quite a bit of time, it had to be split up over multiple update stream
>> versions, so it was a general pain to take that extra step.
>
>
> I still don't quite get what you're saying. Was there any "extra
> effort" associated with "making it perfectly reproducable in an
> automatic way" (I'm quoting it because I don't understand what was
> involved in this process) other than simply posting a "load this
> package" command? In other words, what I don't understand (or don't
> know about) is what extra effort was needed to get the packages in a
> form suitable for the update stream?
>
>>> Nice try. You know just as well as I do that this ain't gonna happen
>>> ;-)
>>
>>
>> It depends. Personally, I don't think it is fair to burden the release
>> team with lots of work that maybe only others care about. That's why
>
>
> "only others care about" - "others" like in "customers"? I'd be
> careful not to underestimate the interest of the casual Squeaker to
> just check out what's going on in the latest versions. Not necessarily
> always riding on the latest alpha but being able to "check in and see"
> where things are.
>
>> I'm trying to find a process (and - remember the original mail -
>> implement that in 3.10) that minimizes the burden on the package and
>> especially release teams. If people complain that this won't satisfy
>> their crave for software archeology, let them step up and do something
>> about it (yup, exaggerating there - the points you point out are valid
>> but are they important enough?).
>
>
> Good question. It is important to put the options squarely on the
> table though, and in this case (and both Daniel and you in a response
> to Daniel's post agree on that) it really means that this proposal is
> essentially the end of the ability for anyone to update their images
> automatically (which is really behind my comment, e.g., how likely is
> it that somebody will retrace the steps after the fact to make it
> loadable).
>
> Whether that is "important enough" or not is a good question. One
> thing I believe is "important enough" however is the documentation
> aspect of being able to update on your own. The main value here is
> that anyone can review what changes have been done in any version and
> can follow along and see where things stand, what has been changed and
> how. That's not to say that the update stream is the only way to deal
> with that but given the diverse set of people dealing with different
> releases I think it's critical to have the process documented in some
> form. If the way in which image vX.Y+1 was derived from vX.Y is
> unclear I think we're in for trouble.
>
> Cheers,
>   - Andreas
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Proposal for the coming versions

Adrian Lienhard
In reply to this post by Andreas.Raab

On Mar 14, 2006, at 10:56 , Andreas Raab wrote:

> Adrian Lienhard wrote:
>> It would be valuable to precisely identify the problems of MC  
>> because in my experience it works very well for 80% of the cases  
>> (which we can't say for change sets -- at least what I use MC  
>> for). So, the goal for 3.10 (or for a dedicated "task force")  
>> would be to improve those shortcomings.
>
> Start here:
> http://lists.squeakfoundation.org/pipermail/squeak-dev/2006- 
> February/101071.html
>
> If you can fix the issues listed there

wait, I've never talked about how to fix those problems. What I said  
was "I see three main areas of problems" (critical issues) and a bag  
of "other stuff" (still important but less than the first three groups).
What I wanted to say is: lets identify what is most critical and  
encourage people to work on this now, rather than talking about what  
would be the perfect solution that solves all problem (although  
that's nice too).

Of course, then, everybody has different needs (for example, it seems  
you care less about speed, whereas I would rather like to have more  
speed than having correctly working overrides because I can work  
around the latter but not the former) but I have the impression that  
with some concrete steps we could already go a step forward. Let's  
take the atomic load. It seems it would solve a couple of problems  
(e.g., load order dependencies, moving methods/classes between  
packages), no?

Adrian

> (class renames, incremental shape modifications, migrations across  
> these different versions, moving classes/methods between packages,  
> integration of system-wide doIts, load order dependencies and all  
> possible combinations of these) then I'd call MC functionally  
> complete to deal with full system maintenance. It'd still be slow  
> but at least a complete system maintenance tool. I'll stop short of  
> the full list, if you can show how one solution subsumes another  
> where possible.
>
>
> Cheers,
>   - Andreas
>


12