RV: An idea, crazy or not? (Re: Very bad about Squeak in blogosfere)

> De: <[hidden email]>
> Responder a: The general-purpose Squeak developers list
> <[hidden email]>
> Fecha: Mon, 13 Aug 2007 10:47:04 +0200
> Para: The general-purpose Squeak developers list
> <[hidden email]>
> Asunto: An idea, crazy or not? (Re: Very bad about Squeak in blogosfere)
>
> Hi all!
>
> (long post, sorry!)
>
> Andreas Raab <[hidden email]> wrote:
>> Colin Putney wrote:
>>> It's stated a bit harshly, but yeah, that sounds basically accurate. The
>>> amazing thing is that, in spite of all that, Squeak is still such a
>>> wonderful platform to work with. I do use Squeak in production, and
>>> there are very few things I would trade it for.
>>
>> Well, yes, but you can't deny that the guy's got a point. The
>> frustration he's expressing is something that everyone has felt over the
>> years. And while there are various plain invalid points in his post
>> (like the fact that Squeak has bugs - I'm *shocked* to hear that of
>> course and would have never started three products if I'd known that ;-)
>> the main emerging point is valid: The lack of quality and maintenance.
>> The problems he cites are all known, some of them even have fixes but
>> there isn't enough traction in the community to make this all come
> ...
>> together. And of course the forks don't exactly help because we still
>> haven't figured out how to share code across the forks and consequently
>
> And Andreas continues to describe the problem that I have been thinking
> a bit about during my vacation.
>
> Last night I wrote it up and intended to bounce it around "privately"
> first on selected people, but reading this thread it is so inviting to
> share the idea so what the hell :) Forgive me if this post is long - but
> I want to paint my picture as best I can.
>
> IMO these are our major problems today (we have lots of minor ones too
> and the problems intermix):
>
> 1. More or less anarchy when it comes to leadership of Squeak-dev. The
> board is not taking charge, for a whole range of possible reasons. One
> obvious reason is that the guys elected are really good developers who
> tend to be very busy. Another reason may be that we have different
> expectations on what they should/could be doing. IMPORTANT: I am *not*
> placing any blame though - we are all in this together.
>
> 2. We are now "de facto" living in a highly forked world (Croquet,
> Sophie, Squeak-dev, Spoon, Squeakland, OLPC, yaddayadda...) but we don't
> have tools that support such a world!
>
> 3. The core Squeak is not evolving properly mainly due to the fact that
> most of the core codebase is "abandoned" - or in other words, noone
> feels responsible and/or authoritive to bring it forward.
>
> In short we have "leadership paralysis", a "forked world" and a largely
> "abandoned core". Other problems I missed? Of course! ...but lets ignore
> those for this post.
>
>
> Considering a few measures then:
>
> - Make a new fork with a strong leadership? Hmmm, could help with #1 and
> #3 above but does nothing to improve #2 and would probably be tons of
> work to succeed, and I personally don't have the required time to spend.
> Plus I am a hard core "squeak-dev" member - I don't *want* to fork. :)
>
> - Reshape the organisation of Squeak-dev? Nah, I am sick and tired of
> pulling such work, done just too much of that stuff already over the
> years, and it would possibly help with #1 and #3 (strong leadership) but
> again would not help at all with #2.
>
> So ok, forking is out and reshaping the organisation from the inside is
> out, even though I really would want much more initiative from the board
> in the future - no doubt about that.
>
>
> Hmmm. But what could be done then? Well, after seeing over and over
> again how really good TOOLS can change the way we interact in our
> community (Monticello, SqueakMap, SqueakSource, Universes etc) I tried
> to imagine a new infrastructure that would help with the three problems
> above, in a *natural* way.
>
> Let's look at #1 - the anarchy problem. If we don't try to solve it but
> instead try to *live with it* - what could that mean? The piece that
> suffers mostly from the anarchy is of course the base image. All our
> external packages prosper just fine anyway (kinda). So could we put some
> kind of "supertool" in place to maintain the base image that could be
> made to work *without* strong leadership? I think we can.
>
> What about #2 - the forking problem? Well, imagine this "supertool"
> being written in such a way that it can be used in all major forks - if
> its good and easy to adapt/install it would most probably also *be*
> used. Monticello has already showed this can be done.
>
> So instead of trying *not* to fork, we try to make it very easy to fork
> - or rather *easy to live* in a forked world of Squeak
> images/communities.
>
> But #3 - the rotting core problem? Again, if it was very easy to fix
> stuff in the core, publish these changes without having to ask anyone
> for permission and very easy to cherry-pick such fixes from other people
> - then hopefully the core would again start moving forward. As Andreas
> notes - fixes are sitting inside Croquet and my bet is that they are
> mainly sitting there because it is a bit "too much work" to push them
> out. I am aware of fixes sitting in Gjallar (at least a few) just
> because I didn't feel I had time to fix/prepare them sufficiently etc.
>
>
> So my "daft" idea is:
>
> Let's pull together a good team and make a Supertool that is written to
> be used in all forks that... well, what will it actually *do*? :)
>
>
> 1. Introduce a new kind of source unit - let's call it a Delta. The guys
> working with MC2/Monticello may have lots of noteworthy stuff to share
> about how this kind of thing could look, but I simply want a "changeset"
> for the 21st century. A patch. But a smarter one! The philosophy here
> being that changesets are nice given their simplicity and malleability -
> but they are not so nice in many other respects. But the concept of
> communicating with other developers using the natural "work unit" - a
> "commit" - is quite nice. In MC at least I tend to make too large
> snapshots because let's face it - MC is too slow for small commits -
> given how it works. We just don't suffer because it is so darn slick at
> merging :)
>
> 2. Create the notion of "delta streams". Yes, we had the update stream
> earlier - a sequential flow of changesets. It was kinda nifty, but
> imagine having tons of these streams originating both from larger
> cooperative projects (Croquet, Sophie, Seaside, Gjallar etc) and from
> individual developers ("Andreas Raab's kernel fixes", "Juan's
> refactorings of Morphic" etc). A Delta could appear in several streams
> so Croquet could have a single stream for each release AND multiple
> streams for different kinds of Deltas, like "base image fixes for
> Hedgehog" etc).
>
> 3. Make an efficient but simple storage model for deltas and delta
> streams. Here I am thinking KISS. If a delta could be represented as a
> single gzipped file (enabling dumb servers just like MC does) and a
> stream can be represented by file naming conventions (by a number
> suffix) - then a stream is just a directory or a zip of a bunch of delta
> files. Yes, very similar to gzipped changesets and the update stream. :)
> This enables us to use all "vanilla" tools available for dealing with
> files (http servers, ftp, rsync, email etc).
>
> 4. Make a simple but efficient transport. Let's say we set up a public
> server that syncs such directories of delta files (streams) from tons of
> places. And then offers it to all of us as a simple rsync. Each
> developer (or team) could then use rsync to mirror that mega tree of
> delta files onto our laptops/servers and the tool inside Squeak would
> just need to bother with reading the local filesystem - which makes it
> much more portable across Squeak dialects (and possibly even Smalltalk
> dialects - but let's not go there just yet). This also means we don't
> suffer from servers being down.
>
> 4. Make a simple model of Delta and DeltaStream in Squeak and let it
> mirror the filesystem. Add a simple API to the model á la Keith's
> Installer. Or hey, just enable Installer work with this model.
>
> 5. Make a great Morphic tool to work with them. Perhaps we could even
> rework the changesorter family of tools thus superceding changesets?
>
>
> Ok, imagine this supertool in your image. Imagine that all public
> streams are listed directly in the tool and new ones just "pop up" as
> they are added. Imagine that you can add your own private streams just
> like you can add repositories to sources.list in apt-get (you Debian
> people know what I mean). You can have purely private local streams on
> your harddrive or streams on a shared fileserver in your company/dev
> group, etc.
>
> Now you have this huge flood of Deltas in hundreds of streams at your
> fingertips. How would you use it?
>
> 1. Subscribe to some major streams and configure it to automatically
> load deltas whenever they appear when you press "update".
>
> 2. Set some rules that govern if the tool should just try to load - or
> ask for confirmation based on characteristics of the delta.
>
> 3. Autosubscribe to categories of streams. For example, if a new bug-fix
> stream appears for a package you use a lot - you might want to get it
> autosusbcribed - or again, have the tool ask you.
>
> 4. Of course, easily publish your own streams. We are talking single
> button, no getting permissions etc.
>
> 5. Push fixes to other people's packages as deltas onto a personal
> public fixes-stream. This means it does not matter what package you are
> bug fixing - you can *always* push the fix to your personal public
> fixes-stream and don't need to bother with getting permissions on
> SqueakSource or wherever it came from. This is a very IMPORTANT feature
> and should hopefully put an end to "lost fixes" or "fixes sitting inside
> images".
>
> We have too many fixes out there that just never get published because
> of the "hassle", and sure, someone says "it is very easy to upload to
> Mantis!" - but fixes on Mantis sets expectations on quality,
> documentation, follow up etc. A personal fixes stream like above sets no
> such expectations - it is there for the taking - but that is all.
>
> 6. Pull deltas. Selective cherry picking etc.
>
>
> Important features of Deltas:
>
> - Atomic load of a Delta. Either it loads cleanly or it does not load at
> all, and it ensures this by checking FIRST that everything is in place
> for it to be able to apply cleanly. This should prevent "failed loads"
> and broken images. The actual low level atomicity is another story but
> SystemEditor from MC2 perhaps? I dunno.
>
> - Revertable, if *possible*. A delta can be analyzed to see if it is
> revertable. If it includes doits or class initializations it is in
> theory not revertable, it may be in practice though! It can also be
> marked as DefinitelyNonRevertable.
>
> - A Delta is declarative. Another class should be responsible for
> actually applying them.
>
> - When a Delta is applied to an image we generate a reverse Delta which
> (when applied) will reverse the effects. Since the image may be in
> different states this reverse Delta needs to be constructed when the
> Delta is applied!
>
>
> A Delta contains:
>
> - A Preamble similar to ChangeSets with info fields
> - Developer (name, id, signature etc whatever)
> - Original stream (URL)
> - UUID
> - DefinitelyNotRevertable flag
> - Test doit (a non destructive, non interactive doit that should throw
> an exception in order to prevent loading!)
> - An ordered sequence of Actions, see below.
>
> An example list of different types of Actions are:
>
> - Change method (class name, old method src, new method src)
> - Add method (class name, method src, category name)
> - Remove method (class name, method src, category name)
> - Categorize method (class name, method name, old category name, new
> category name)
>
> - Create class (class name, super class name, definition, category)
> - Delete class (class name)
> - Change superclass of class (class name, old super class name, new
> super class name)
> - Rename class (old class name, new class name)
> - Categorize class (class name, old category name, new category name)
> - Change definition (class name, new definition, old definition)
> - Class comment change (new comment, old comment)
>
> - Class initialization
> - Doit (marked as revertable or not by author!)
>
> Note how these Actions contain "more" info than a ChangeSet - it
> contains information about what it was "before" in the image it
> originates from! The idea is to make Deltas "rich" with info so that we
> can apply them with more smarts.
>
> When applied to an image the Delta is copied to a directory with the
> same name as the image followed by "-applied-deltas". It is also logged
> in changesfile. If the Delta can not be cleanly reverted (based solely
> on its own contents) we generate reverse Deltas and store them in the
> same directory prefixed with "reverse-".
>
> Applying a Delta:
>
> 1. Verify that it can be applied (or signal CanNotApplyException). Do
> this by analyzing Actions and running any Test and see if it throws an
> Exception.
> - It can not be applied *cleanly* if something is "different" from
> expected.
> - It can be applied *dirty* if all changes can be applied in "some
> fashion". For example, a delete operation and the thing to delete is
> already deleted. The reverse Delta will then show this so that a revert
> will not put the thing back in.
> - It can be applied *partially* if just a subset of Actions are applied
> (by choice or since some just can't be applied). The reverse Delta will
> then show what was applied or not.
>
> 2. Verify that it can be reverted cleanly (or signal
> CanNotRevertCleanlyException). Do this by analyzing Actions.
>
> 3. Apply and generate the reverse Delta if needed.
>
> 4. Copy delta and any reverse delta to applied dir.
>
> 5. Log to changes file.
>
> 6. Report package touched (for running unit tests afterward!)
>
>
> What about merging then? Well, the concept here is to be much more
> "loose" when it comes to merging compared to say MC. It was inspired by
> something I read about git (Linus' SCM tool) - it doesn't try to be
> smart - it just tries to do the right thing when it is obvious. In other
> situations it just asks the developer. I think I like this approach. The
> Delta model should be easy enough for all developers to understand. It
> is "just" a changeset/patch after all.
>
> So if I am trying to apply a Delta (or several) and the "before state"
> is not as the Delta expects - we can either cop out, or let it be
> "smart", or just ask the user to decide what to do. BUT... I am not an
> SCM implementor - the MC/MC2 guys can clearly explain to me why/how this
> will not work or amend the idea so that it can work. I gladly admit my
> ignorance. ;)
>
>
> ...ok, enough blabbering. :) Does this sound plausible, useful or just
> plain dumb? Is MC2 already this and much more?
>
> ciao, Göran
>