Proposal for the coming versions

> When discussing the immediate future, I promised to write up a small
> proposal for the coming versions. Here it is - shoot at it:
>
> Version 3.10 - Implement the Process
> ===========================
> After 3.9, I think it would be a good idea to work on a version that
> has as its main goal to implement the community process we have in
> place (user -> mantis -> package team -> inbox -> release team,
> roughly). The idea is to enhance SqueakMap with Team cards, to link
> into Mantis from the image, etcetera, so that at least User and
> Package Team can work with "single button presses". Some typical use
> cases:
>
> User: "Submit bug report". SqueakMap is used to find out what team
> maintains the code that the user has patched. The team card in SM can
> have stuff like the name of the Mantis category matching the Team's
> bugs, the format the team prefers (changeset, .mcz or .mcd file, ...),
> the place to send it to (a mail address, an MC repository, ...),
> etcetera. The user gets a simple form containing most of the stuff
> Mantis also asks for, and then the package submits the bug to Mantis.
> As simple as the old method of mailing a bug or patch to the list, but
> asking for more details in order to raise the quality of bug reports
> and patches.
>
> Team member: "Work as member of...". The SM card is fetched, and from
> here maybe a little control panel is created that can fetch the next
> bug report from Mantis ("Next" being defined as some heuristic over
> age and severity, for example), automagically file in any attached
> patch, and when done does a sort of transaction over mantis and the
> team repository, closing the bug and committing to the team repository
> with links in both Mantis and MC to each other.
>
> Also, 3.10 could be the first version that is developed update  
> stream-less.
>
> Version 4.0: Brave New World
> ======================
> After 3.10, and probably starting right away given the length of time
> discussions are likely to take place ;-), version 4.0 will especially
> integrate changes that will effect image format. Whatever they may be
> - there's a long list of wishes, and the team lead should first and
> foremost come up with a community process to turn the list of wishes
> into a todo list.
>
>

> On Mar 13, 2006, at 2:59 PM, Andreas Raab wrote:
>
> > Cees de Groot wrote:
> >> Personally, I would point my finger at the use of MC instead of MC
> >> itself (MC seems to be not fit for a particular purpose its creators
> >> probably never reckoned with).
> >
> > According to Colin "Monticello was designed with maintaining live  
> > system in mind" (see http://lists.squeakfoundation.org/pipermail/ 
> > squeak-dev/2006-February/101067.html)
>
> Yeah. I probably shouldn't have stated it that strongly. Our "live  
> systems" at the time were web apps with data stored in an external  
> database. So if by "maintaining live systems" you mean updating the  
> code without out taking down the app, yeah. If you also mean "and  
> migrating in-image objects transparently," well no, not quite.
>
> On the other hand, truly "live systems" are a big part of what this  
> community does, and I personally would like to see Monticello do a  
> better job of serving that part of the community. As Avi mentioned,  
> Monticello2 has been moving very, very slowly forward for the last  
> year (2 years is a bit of an exaggeration). I honestly think, though,  
> that it's the best vehicle for addressing these issues. I'm going to  
> keep pushing it forward, hopefully making faster progress than I have  
> been.
>
> Colin

>>From: Cees de Groot
>>So the user would effectively have to
>>download the image to hop across major upgrades.
>>    
>>
>
>So how do I carry my customised image content over between versions?  Am
>I supposed to export it and re-import it?  Is everyone?
>  
>

> Cees de Groot wrote:
>
>> Personally, I would point my finger at the use of MC instead of MC
>> itself (MC seems to be not fit for a particular purpose its creators
>> probably never reckoned with).
>
>
> According to Colin "Monticello was designed with maintaining live
> system in mind" (see
> http://lists.squeakfoundation.org/pipermail/squeak-dev/2006-February/101067.html)
>
>
>> Anyway, I wasn't assigning "blame", I
>> was just remarking that there was a conflict here.
>
>
> Me neither. As I've said in the past, MC is good for what MC is good
> for (namely managing source code in a "build" environment) but it
> strikes me that you're proposing to apply it in just the way where it
> doesn't work very well. But of course, that's just my experience. YMMV.
>
>> Nope. My idea is that during "fast development", the team leader just
>> publishes images. And probably an MCC map or something similar showing
>> how the image is put together. So the user would effectively have to
>> download the image to hop across major upgrades. Minor upgrades would
>> likely be doable by just udating using the MCC map. For some value of
>> "major" and "minor", of course :)
>
>
> Um ... I might be slow or something but can you explain to me again
> how this makes things simpler/easier/whatever? Where is (in terms of
> pain associated with it) the difference between having that MCC map
> and posting it to an update stream? (unless you don't do it at all ;-)
>
> And using images this way ... uh, oh. I have a *really* bad feeling
> about this one. You're right, it's often easier to do things in your
> image and distribute that image. It's also a sure way of making it
> impossible for anyone to track ongoing development, the release
> progress, and the history of how things evolved in the image. All of
> which are (in my understanding) very desirable properties. I fear that
> this will lead to a process where you hand off "the image vX.Y" to a
> small group of people (possibly one) and get back "the image vX.Y+1"
> and hardly anyone will have looked at anything at all, noticed the
> oddities, which clean ups are required along the way etc. Not saying
> that many people today track the changes very closely but I even that
> seems preferrable to a process where you don't get to see anything and
> have no documentation of the changes the "official" image went through.
>
>> Yes, that's bad (for people with low bandwidth, for starters).
>> However, it buys time to fix the issues with Monticello, so that
>> hopefully later on a "upgrade v.X to v.Y" button can be reinstated.
>
>
> Nice try. You know just as well as I do that this ain't gonna happen
> ;-) If only because when you give people the ability to modify an
> image "at home" they'll use it for *precisely* those changes that are
> hard to deal with in Monticello. Otherwise, why not use Monticello to
> begin with?
>
> Cheers,
>   - Andreas
>

> Hi everyone.
>
> I think improving the process for those doing the development is  
> important, but agree with Andreas that the current proposal sounds  
> like we're giving up on users ability to update their images. If  
> that's what we want, we should just say so.
>
> If we're going to go for a better system that has the benefits of  
> MC and supports image updates, we need to actually catalog,  
> discuss, and solve those nitty gritty problems that bug both sides  
> of the development process at the moment. And this will be a lot of  
> work, and we can't expect the MC guys to just solve it all for us,  
> even if MC2 solves the code issues, including performance and class-
> renames.
>
> Before continuing with the bad news, just wanted to remind  
> ourselves that systematic treatment of live system updates is  
> possible at least in the sense that GemStone does it.
>
> Here is a partial list of problems I met while working on the  
> updates for Traits:
> - Having made some changes, I should know whether these are  
> loadable into a published image. This currently requires manually  
> posting, opening new image, loading. Some test server seems useful  
> in this context.
> - Doing live updates correctly requires capturing a sequence of  
> steps which know enough each about the previous state to keep  
> things running correctly. MC by itself does not model this at all.  
> An external tool (ScriptLoader, IIRC, in the current process) can  
> capture this. But in addition to capturing update sequences, we  
> need to be able to merge them. For example, suppose we and  
> Squeakland create two unrelated update sequences for overlapping  
> sets of packages, starting from common state A, going to B and C  
> respectively, through B1, B2... and C1, C2... The "did-both-
> updates" state D needs to be created (probably via MC merge), but  
> what are the stage D1,D2...? It might well be that the merge  
> problem needs to be solved manually, rather than automatically, but  
> we definitely need a better way to detect it, to capture the  
> sequences, and to change from one sequence to the other when we  
> find problems (currently doable only by updating the updates-list  
> file, AFAIK).
> - Recompiling the whole image is very slow. Therefore changing the  
> shape of many classes (in that example, all metaclasses) makes  
> updates take a very long time.
> - It is hard to make some system level changes because none of the  
> existing tools in the image document all the VM assumptions.  
> Therefore, that bug from changing the shape of Behaviors that  
> prevents some primitives from running.
>
> Daniel Vainsencher
>
> Andreas Raab wrote:
>
>> Cees de Groot wrote:
>>
>>> Personally, I would point my finger at the use of MC instead of MC
>>> itself (MC seems to be not fit for a particular purpose its creators
>>> probably never reckoned with).
>>
>>
>> According to Colin "Monticello was designed with maintaining live  
>> system in mind" (see http://lists.squeakfoundation.org/pipermail/ 
>> squeak-dev/2006-February/101067.html)
>>
>>> Anyway, I wasn't assigning "blame", I
>>> was just remarking that there was a conflict here.
>>
>>
>> Me neither. As I've said in the past, MC is good for what MC is  
>> good for (namely managing source code in a "build" environment)  
>> but it strikes me that you're proposing to apply it in just the  
>> way where it doesn't work very well. But of course, that's just my  
>> experience. YMMV.
>>
>>> Nope. My idea is that during "fast development", the team leader  
>>> just
>>> publishes images. And probably an MCC map or something similar  
>>> showing
>>> how the image is put together. So the user would effectively have to
>>> download the image to hop across major upgrades. Minor upgrades  
>>> would
>>> likely be doable by just udating using the MCC map. For some  
>>> value of
>>> "major" and "minor", of course :)
>>
>>
>> Um ... I might be slow or something but can you explain to me  
>> again how this makes things simpler/easier/whatever? Where is (in  
>> terms of pain associated with it) the difference between having  
>> that MCC map and posting it to an update stream? (unless you don't  
>> do it at all ;-)
>>
>> And using images this way ... uh, oh. I have a *really* bad  
>> feeling about this one. You're right, it's often easier to do  
>> things in your image and distribute that image. It's also a sure  
>> way of making it impossible for anyone to track ongoing  
>> development, the release progress, and the history of how things  
>> evolved in the image. All of which are (in my understanding) very  
>> desirable properties. I fear that this will lead to a process  
>> where you hand off "the image vX.Y" to a small group of people  
>> (possibly one) and get back "the image vX.Y+1" and hardly anyone  
>> will have looked at anything at all, noticed the oddities, which  
>> clean ups are required along the way etc. Not saying that many  
>> people today track the changes very closely but I even that seems  
>> preferrable to a process where you don't get to see anything and  
>> have no documentation of the changes the "official" image went  
>> through.
>>
>>> Yes, that's bad (for people with low bandwidth, for starters).
>>> However, it buys time to fix the issues with Monticello, so that
>>> hopefully later on a "upgrade v.X to v.Y" button can be reinstated.
>>
>>
>> Nice try. You know just as well as I do that this ain't gonna  
>> happen ;-) If only because when you give people the ability to  
>> modify an image "at home" they'll use it for *precisely* those  
>> changes that are hard to deal with in Monticello. Otherwise, why  
>> not use Monticello to begin with?
>>
>> Cheers,
>>   - Andreas
>>
>
>

>> Um ... I might be slow or something but can you explain to me again how
>> this makes things simpler/easier/whatever? Where is (in terms of pain
>> associated with it) the difference between having that MCC map and
>> posting it to an update stream? (unless you don't do it at all ;-)
>>
> Well... in the update stream, you need to take care about load order,
> about conversion scripts, stuff like that. Just a map won't be enough
> to upgrade X to Y. For example, loading your UI work (ToolBuilder
> etcetera) was relatively easy, not in the least because it was
> well-prepared and well-documented. Loading it in a way that was
> perfectly reproducable in an automatic way (ie "scriptable") took
> quite a bit of time, it had to be split up over multiple update stream
> versions, so it was a general pain to take that extra step.

> Cees de Groot wrote:
>
>>> Um ... I might be slow or something but can you explain to me again
>>> how this makes things simpler/easier/whatever? Where is (in terms of
>>> pain associated with it) the difference between having that MCC map
>>> and posting it to an update stream? (unless you don't do it at all ;-)
>>>
>> Well... in the update stream, you need to take care about load order,
>> about conversion scripts, stuff like that. Just a map won't be enough
>> to upgrade X to Y. For example, loading your UI work (ToolBuilder
>> etcetera) was relatively easy, not in the least because it was
>> well-prepared and well-documented. Loading it in a way that was
>> perfectly reproducable in an automatic way (ie "scriptable") took
>> quite a bit of time, it had to be split up over multiple update stream
>> versions, so it was a general pain to take that extra step.
>
>
> I still don't quite get what you're saying. Was there any "extra
> effort" associated with "making it perfectly reproducable in an
> automatic way" (I'm quoting it because I don't understand what was
> involved in this process) other than simply posting a "load this
> package" command? In other words, what I don't understand (or don't
> know about) is what extra effort was needed to get the packages in a
> form suitable for the update stream?
>
>>> Nice try. You know just as well as I do that this ain't gonna happen
>>> ;-)
>>
>>
>> It depends. Personally, I don't think it is fair to burden the release
>> team with lots of work that maybe only others care about. That's why
>
>
> "only others care about" - "others" like in "customers"? I'd be
> careful not to underestimate the interest of the casual Squeaker to
> just check out what's going on in the latest versions. Not necessarily
> always riding on the latest alpha but being able to "check in and see"
> where things are.
>
>> I'm trying to find a process (and - remember the original mail -
>> implement that in 3.10) that minimizes the burden on the package and
>> especially release teams. If people complain that this won't satisfy
>> their crave for software archeology, let them step up and do something
>> about it (yup, exaggerating there - the points you point out are valid
>> but are they important enough?).
>
>
> Good question. It is important to put the options squarely on the
> table though, and in this case (and both Daniel and you in a response
> to Daniel's post agree on that) it really means that this proposal is
> essentially the end of the ability for anyone to update their images
> automatically (which is really behind my comment, e.g., how likely is
> it that somebody will retrace the steps after the fact to make it
> loadable).
>
> Whether that is "important enough" or not is a good question. One
> thing I believe is "important enough" however is the documentation
> aspect of being able to update on your own. The main value here is
> that anyone can review what changes have been done in any version and
> can follow along and see where things stand, what has been changed and
> how. That's not to say that the update stream is the only way to deal
> with that but given the diverse set of people dealing with different
> releases I think it's critical to have the process documented in some
> form. If the way in which image vX.Y+1 was derived from vX.Y is
> unclear I think we're in for trouble.
>
> Cheers,
>   - Andreas
>
>

> Adrian Lienhard wrote:
>> It would be valuable to precisely identify the problems of MC  
>> because in my experience it works very well for 80% of the cases  
>> (which we can't say for change sets -- at least what I use MC  
>> for). So, the goal for 3.10 (or for a dedicated "task force")  
>> would be to improve those shortcomings.
>
> Start here:
> http://lists.squeakfoundation.org/pipermail/squeak-dev/2006- 
> February/101071.html
>
> If you can fix the issues listed there

> (class renames, incremental shape modifications, migrations across  
> these different versions, moving classes/methods between packages,  
> integration of system-wide doIts, load order dependencies and all  
> possible combinations of these) then I'd call MC functionally  
> complete to deal with full system maintenance. It'd still be slow  
> but at least a complete system maintenance tool. I'll stop short of  
> the full list, if you can show how one solution subsumes another  
> where possible.
>
>
> Cheers,
>   - Andreas
>