Squeak, source control, subversion, versioning, monticello, all that good stuff.

Hi Squeakers. Bit of an internal monologue here about something that's
been concerning me about Squeak development for a while now: Source
control. Please excuse the verbosity.

Specifically, over time at my last two companies I've worked with others
on developing a development process using CVS (later Subversion) that
allowed development to happen on a fair-sized codebase, across a team of
up to about 30 or 40 developers. Nothing particularly special, it uses
the idea of "branching a lot" as opposed to "branching a little", but it
allows the tight integration of a bug tracking tool with ones source
control, and potentially (although the reality is often less idealistic,
obviously) keeps fixes for bugs nicely segregated from the rest of the
code until they're merged. Note that this has always been on file-based
languages (Java in this case). But with a major smalltalk-related
caveat, which I'll get on to shortly.

It's how I'd like to be able to work now with Squeak, but obviously
things are different in image-based-development-land.

Just to elaborate a little on the process: There's a "main branch" of
development, commonly referred to (in my experience: Note that my
terminology may differ a lot to what people are used to, so please bear
with me :) as the HEAD with cvs/svn. The state of this branch is
arbitrary, really - although generally for me it was always the latest
main development/unstable/latest&greatest version of the product the
source-controlled codebase represented.

We then also used the branch-per-product method: In an ideal world a new
version of the product would be released to the world against a known
cvs/svn tag. Further maintenence against that version would then be
performed in a branch made from that tag, and would henceforth live
seperately from the HEAD. This process could continue as futher
maintenence releases were made from *that* branch, etc etc. One could
end up with version numbers like, 1.1, 1.2, 1.2.1, 1.2.2, 1.2.2.1, say.
You get the picture I'm sure (I hope!).

We went a bit further (people may disagree here), and pushed branching
to the extreme: We branched for each and every defect change, naming
that branch against the bug number from our bug control software. Yes,
it took a lot of training to get the developers used to and happy with
this process, but once we did it became much, much easier to track
changes related to certain bugs, merge those changes into other branches
(ie back/forward porting bugs fixes between versions), review the
changes for a bug atomically, etc. We were pretty happy with it.

Of course one might point out that this model pretty much negated one of
the nicest parts of cvs/svn: that it allows developers to concurrently
work on the same files at once, and on face value you'd be right.
However, often we were working in a "large bug branch" that was really a
large branch related to a large new development, with at least two
developers (both assigned to the same bug) working on the same branch at
once, so here the concurrent modification came into play.

Now for the caveat: The model was based upon the original development
environment that my previous company were using before we switched to
Eclipse, VAJ (Visual Age forJava), in other words loosely based upon
development using ENVY - therefore loosely based on a reaonable
Smalltalk development practice.

Now I find myself at the point of trying to justify Squeak for
development here at my current job. I've got a lot of backing from
"upstairs", but I want to know that if we use Squeak we can have the
same level of granular control over source modification and control that
I've had with Java in the past. To my current understanding Monticello
just doesn't cut it - in fact (harsh as it may sound), the merging/diff
tools in Squeak generally seem inferior to svn or even VAJ/ENVY. I seem
to recall a post to this list that I read once from Avi saying that
Monticello was more of a version delivery system than a source version
control system - in other words that it was more useful as a way to
deliver the upgrade of a version of a complete package at once, rather
than people "checking code in an out" all the time as is more the case
with cvs/svn. Is this right?

Similarly, there was a system that Avi wrote a while back (I think this
was pre-MC) called DVS. However the archive on squeak dev isn't clear on
exactly what it was. From what I read it seemed much more a "snapshot
all of the image and shove it straight into whatever scm we're pointing
at" system, rather than dealing with individual files. I may however be
doing it an enormous injustice here.

Another thing that was recently mentioned was some work around MC2, but
I don't know if that will support the multi-developer process I
described above either, because of the general problem I've had so far:
Try as I might I cannot find enough information on how to manage a
smalltalk development within a multi-developer environment. Sure, I can
see how MC works on a high level. But for sharing code it seems like it
just won't work in the way I envisage it.

Therefore I find myself thinking that either I don't understand
MC/MC2/Envy and even Squeak development properly yet (highly likely as
I've only been using Squeak since last July, and even then not
exclusively by any means), or that the current systems out there don't
support what I want at all, or that the development process I've
described above is flawed.

So having gone through all that preamble (and that hopefully whatever
poor souls have made it this far now have a handle on the way I used to
work) I can ask the simpler question: Using Squeak how would one go
about developing a system in a safe, recordable and easy manner in a
multi-developer environment, while maintaining linkage with a
bug-tracking system such as bugzilla, in a way that is non-obtrusive* to
the developer?

Cheers for reading, I hope it makes sense.
Simon

* Here I'm talking about the auditing of changes against bug fixes: For
instance with subversion I have a number of tools available: Fisheye
from cenqua allows me to track who's checking what in where via RSS, and
scmbug glues svn and bugzilla together such that comments of the form
"bug 45: Some changes for this bug" will automatically put a comment
into bugzilla against that bug: Getting developers to responsibily and
happily use bug tracking software is quite a challenge!


This message has been scanned for viruses by BlackSpider MailControl - www.blackspider.com

On 1/27/06, Simon Kirk <[hidden email]> wrote:
> Hi Squeakers. Bit of an internal monologue here about something that's
> been concerning me about Squeak development for a while now: Source
> control. Please excuse the verbosity.
>
And please excuse me for snipping it all in my response :-).

The basic question is: can you do optimistic version control with MC
with a larger group. Modulo some performance problems (which, it turns
out, are related to source file writing and should be fairly simple to
remove), I think with MC(+MCC) you have everything you need in the
tools. You cannot (should not?) name versions, but I've used "special
comments" in CVS with great success (I first used CVS when it was a
bunch of shell scripts around RCS, so I'm reasonably experienced with
the system), and with MC that shouldn't be any different. The merge
tool is simple but sufficient (and easy enough to extend - try to
extend CVS while you're working with it ;-)), and branching support is
very good. Every MC version carries around its whole version history,
so developers can commit versions to private repositories and then a
final version to the shared repository; MC will see the gaps and just
skip over them.

It might not do 100% of what you want right away, but I am 100% sure
that no version control in the world will do it <snipped rant about
how bad CVS sucks>. So it will need work to make it good for your
organization, but it is good enough to start with and decently enough
coded to make it adaptable to your needs.

Note, by the way, that I would object against any single system under
development of a single 30-40 person team :-).

I know ENVY only from VA Smalltalk, and I can tell you confidently
that compared to MC it sucks.

On 27-Jan-06, at 9:22 AM, Cees De Groot wrote:
>
> I know ENVY only from VA Smalltalk, and I can tell you confidently
> that compared to MC it sucks.
Envy is most definitely a deadly sin.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
If the code and the comments disagree, then both are probably wrong

2006/1/27, tim Rowledge <[hidden email]>:
> Envy is most definitely a deadly sin.

sure? why?

Vaidotas Didžbalis

Am 27.01.2006 um 21:54 schrieb Vaidotas Didžbalis:

> 2006/1/27, tim Rowledge <[hidden email]>:
>> Envy is most definitely a deadly sin.
>
> sure? why?

Literally.

- Bert -

Vaidotas Didžbalis wrote:
By definition of the Catholic church, there are 7 deadly sins. Envy is
one of them.

(however, Envy the program is ok ;-)

David

On 1/28/06, David P Harris <[hidden email]> wrote:
> (however, Envy the program is ok ;-)
>
Personally, I don't envy people who envy Envy, but there's no arguing
about taste.

(actually there is, but that'll have to wait for my book review of The
Nature of Order and is waaaay off-topic)

On 27-Jan-06, at 4:52 PM, David P Harris wrote:
>>
> By definition of the Catholic church, there are 7 deadly sins. Envy  
> is one of them.
That, obviously, is a matter of faith and whether you choose to  
believe any of them.
>
> (however, Envy the program is ok ;-)
That, however, is a matter of observable fact and I can only disagree  
completely with your statement. FWIW Dave Thomas agreed with me on it.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful random insult:- A room temperature IQ.

On 1/28/06, tim Rowledge <[hidden email]> wrote:
> > (however, Envy the program is ok ;-)
> That, however, is a matter of observable fact and I can only disagree
> completely with your statement. FWIW Dave Thomas agreed with me on it.
>
That's an argument by authority.

> Useful random insult:- A room temperature IQ.
>
My room is around 293K at the moment. Yeah, that's about in the ballpark.

Cees can you explain me why it sucks. I'm new to MC.

I use ENVY to work every day. We use ENVY in a very particular way (we do collective code ownership and continuous integration, so we don't use all those "owner/manager" things, we don't use subapps or configuration expressions... and we use some in-house tools to handle dependencies and merging, if you want I can tell you more details -- I don't want to write a big boring mail--).

In my little experience with MC, I miss some things of ENVY (again maybe because I'm new to MC, and I use squeak in my spare time), those things are (in order of importance):

1. The lack of a reified concept of "application". I know that in Squeak are "categories", but they are only strings. Maybe an example can explain this point:
In our system we use GemStone. We try to make that our app portable between VAST and GS (this makes our lives more easy because debugging and developing directly in GS is a pain in the ass). So we had extended the applications to respond to the message #gemStoneSynchronizer. So all applications are polymorphic to this message. Each time that we want to upload the code to GS, an object simply iterates over all applications in dependency order (using a topological sort) get its "synchronizer" and executes it. The synchronization depends on the synchronizer implementation returned by the application (some applications are not uploaded to GS, some uploads a GS specific implementation, by default all applications returns a "null" synchronizer that does nothing).
This would be more difficult in Squeak , because you have to do an object that maps the category String with the specific synchronizer (for example I dint found a way to use a particular instance to MCPackage that depends on the application... oh well category).

2. Is there a way to do something similar to "configuration maps" in MC? (or Bundles is VW)

3. In ENVY each time that I save a method a version of it is created on the repository. In MC I can revert the changes on a method but I didn't found a similar functionality.

4. The "Make image consistent" is really useful when you have an image crash (I know in the end you have to check the doit.log like in Squeak to ensure that everything is back again, but I found that this in ENVY is much more easy: simply "Make image consistent" load or discard different methods, and then check that there is no Undefined classes in the image --this test is easy to do because the application object responds to #undefined with the classes that are missing)

5. I really miss the three way diff :) (we have done an special version that shows with colors those methods that can be merged without conflicts: the ancestor and the current version are the same)

On Fri, Jan 27, 2006 at 06:22:27PM +0100, Cees De Groot wrote:
> On 1/27/06, Simon Kirk <[hidden email]> wrote:
> > Hi Squeakers. Bit of an internal monologue here about something that's
> > been concerning me about Squeak development for a while now: Source
> > control.
>
> Note, by the way, that I would object against any single system under
> development of a single 30-40 person team :-).

Without a development team of this size, it is not possible to
assemble a large enough population of defects to objectively
measure the value the aforementioned control processes. In the
interest of the continued advancement of computer science, pray
allow the experiment to proceed.

Dave

p.s. Apologies to Simon whose questions are entirely valid ;)

On 27-Jan-06, at 5:08 PM, Cees De Groot wrote:

> On 1/28/06, tim Rowledge <[hidden email]> wrote:
>>> (however, Envy the program is ok ;-)
>> That, however, is a matter of observable fact and I can only disagree
>> completely with your statement. FWIW Dave Thomas agreed with me on  
>> it.
>>
> That's an argument by authority.
But consider the authorities...

>
>> Useful random insult:- A room temperature IQ.
>>
> My room is around 293K at the moment. Yeah, that's about in the  
> ballpark.
I would anticipate no less.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: SDJ: Send all Data to Japan

On Jan 27, 2006, at 11:23 AM, Simon Kirk wrote:

Hi Simon,

Perhaps a bit of history will give you the information you need to  
make a sensible choice. A few years ago I was in a similar situation  
to where you are now - moving from a file-based language managed by  
CVS, to Squeak managed by... what? Avi was consulting for some  
Seaside projects we were doing, and whipped up DVS as a way to let us  
continue using CVS with Squeak. DVS evolved to the point where we  
realized that CVS was holding us back rather than helping. So we  
added native versioning capabilities and renamed it Monticello.

It's interesting that you find the diff/merge capabilities of  
Monticello to be lacking, because really that's the one thing  
Monticello does really well. MC takes the same approach as CVS in  
that it allows concurrent development, with the ability merge  
parallel lines of development after they have diverged. In fact,  
Monticello does a better job of it than CVS, because it handles  
repeated merges without bogus conflicts. Also, it doesn't require you  
to update before committing - if someone else has committed since  
your checkout, you just end up creating a branched version that you  
can merge back into the trunk later.

If you want to have a strict branching discipline with Monticello, it  
wouldn't be hard. You could establish a convention for naming  
versions to indicate the branch it's on. You'd set that name when you  
save the first version on the new branch and Monticello would suggest  
version names that preserve that convention. On the other hand, we  
find it's pretty common to have more organic branching patterns in  
Monticello, simply because merging is so easy. With CVS you need a  
more structural discipline because it doesn't handle merges with  
complicated histories correctly. Have a look at this entry from Avi's  
blog for an example of what I mean:

http://www.cincomsmalltalk.com/userblogs/avi/blogView?
showComments=true&entry=3291004466

All in all, I think Monticello would be a good fit to your needs. It  
was written with exactly this sort of use in mind.

> * Here I'm talking about the auditing of changes against bug fixes:  
> For instance with subversion I have a number of tools available:  
> Fisheye from cenqua allows me to track who's checking what in where  
> via RSS, and scmbug glues svn and bugzilla together such that  
> comments of the form "bug 45: Some changes for this bug" will  
> automatically put a comment into bugzilla against that bug: Getting  
> developers to responsibily and happily use bug tracking software is  
> quite a challenge!

Yeah, this kind of thing is important, but not covered my Monticello  
per-se. Have a look at SqueakSource. It provides a web front end to a  
Monticello repository, including RSS feeds with commit messages. I  
don't know of anyone that has done bugzilla integration, but I  
imagine it could be done.

Good luck with this, I hope you're able to use Squeak for your project.

Colin

On 1/28/06, Diego Fernandez <[hidden email]> wrote:
>  I use ENVY to work every day.

Me too :-).

>  1. The lack of a reified concept of "application".

Yup, you're right there. I think we should help stamp out and
eradicate all and every non-reified thingy in Squeak.

But note that Monticello can handle arbitrary bits of code in a
package - the PackageInfo thing is just a convention. And there's no
UI for arbitrary stuff in a MC package but that's probably because
we're all so la^H^Hbusy.

Mentioning applications - ever tried to refactor your code? Like
moving classes between applications? I'm a bit of an ENVY/VAST n00b,
but I haven't found out yet how to accomplish that magical feat. The
menu entry is there, but it always pops up an error to the extent "bad
bad developer. You can NOT do that!".

Oh, and you mentioned "home brewn tool for merging". ENVY/Manager
merging support. 'nuff said about the topic.

>  2. Is there a way to do something similar to "configuration maps" in MC?
> (or Bundles is VW)
>
Yes - rudimentary form but surprisingly usable, MonticelloConfigurations.

>  3. In ENVY each time that I save a method a version of it is created on the
> repository. In MC I can revert the changes on a method but I didn't found a
> similar functionality.
>
In Squeak (and VW) these changes are kept in the .changes file.
However, if you hunt for 'Anvil', I once did a proof-of-concept that
logged every change in a Magma database.

Oh, and I don't like this feature too much. For starters, I had to
clone the whole library to my laptop (quarter of a gig) in order to be
able to work off-site. Of course, Envy says "I know what is best for
you, so I will only clone released editions, you dumb dumb developer".
Which is a sort of a bummer if the team's idea of using Envy is only
to release apps when an app release is imminent.

>  4. The "Make image consistent" is really useful when you have an image
> crash

Yes. Ok. That's good about VAST/Envy - that stuff and atomic loads.

But all in all, Envy has for me the wrong feature set. It doesn't do
the very basics of versioning (merging support is all but absent, you
can't comment editions, stuff like that) and tries very hard to impose
some sort of obsolete development model on you, with individual code
ownership and favoring locking over merging (it seems. I'm not sure,
but it smells that way). And refactoring your code, especially on the
class level (moving classes between apps, renaming classes) seems to
involve dark magic in VAST/Envy, and a tool that sits between me and
well-factored code sucks.

Whereas MC does do the very basics of versioning, and let people
decide all by themselves how they want to go about the process on top
of it. Which, I think, is the right way to attack this sort of stuff.

On Jan 27, 2006, at 6:59 PM, Diego Fernandez wrote:

<snip lots of valid criticism>

> 5. I really miss the three way diff :) (we have done an special  
> version that shows with colors those methods that can be merged  
> without conflicts: the ancestor and the current version are the same)

This confuses me, because Monticello is almost entirely built around  
being able to do a three way diff - I can't imagine a process where  
multiple people were using Monticello and *not* constantly doing  
three way diffs, including good conflict detection (more accurate  
than is possible with ENVY, which doesn't model branches very well).  
So - do you use the Merge button?  What do you miss from it?

Avi

On Jan 27, 2006, at 8:23 AM, Simon Kirk wrote:

No, I don't think I would have said that - I certainly don't agree  
with it :)  What do you find weak about the merge and diff tools in  
Monticello?  I suspect it may be a matter of being poorly documented  
(which they are) rather than technically inferior (I would, in fact,  
claim the opposite).  The one thing that's awkward to do in  
Monticello is to cherry pick a small number of changes from one  
branch and apply them to a different branch, rather than merging  
entire branches - in fact, Monticello goes smoothest if you have the  
(sometimes unrealistic) discipline and foresight to implement any  
change as an independent branch from the earliest point in the  
ancestry at which it could have been made, to maximize the potential  
places where you could merge it in.  MC2 will do this better, but  
even in MC1, the worst case is that you cherry pick "manually"  
without the ancestry getting correctly recorded and risk some  
spurious conflicts later on in face of a repeated merge.
>
> Similarly, there was a system that Avi wrote a while back (I think  
> this was pre-MC) called DVS. However the archive on squeak dev  
> isn't clear on exactly what it was. From what I read it seemed much  
> more a "snapshot all of the image and shove it straight into  
> whatever scm we're pointing at" system, rather than dealing with  
> individual files. I may however be doing it an enormous injustice  
> here.

Replace "all of the image" with "all of a package" and you've about  
got it.  I really wouldn't recommend going back to DVS.

> So having gone through all that preamble (and that hopefully  
> whatever poor souls have made it this far now have a handle on the  
> way I used to work) I can ask the simpler question: Using Squeak  
> how would one go about developing a system in a safe, recordable  
> and easy manner in a multi-developer environment, while maintaining  
> linkage with a bug-tracking system such as bugzilla, in a way that  
> is non-obtrusive* to the developer?

Here's a possible set of practices:
- Start out by having a mainline/HEAD branch in Monticello, named  
after the product.  If your package is "Foo", version names might  
look like Foo.2.0-avi.1, Foo.2.0-cwp.2, etc.
- When addressing an issue recorded in bugzilla, start by loading the  
earliest mainline release in which the bug is evident or in which the  
feature could be implemented.  Make the fixes/changes, and then  
commit and name the version after the issue, eg, Foo.bug45-avi.1.  
This may be a single version, or it may turn into a branch (Foo.bug45-
avi.2, Foo.bug45-avi.3, etc).  Automatically add the note to bugzilla  
when you see the version name of that form.
- Periodically integrate these bug branches into the appropriate  
product branches (load the latest version from the product branch,  
select the latest version on the bug branch, press Merge, deal with  
any conflicts, and then commit back the merged version as the next  
version on the product branch with an appropriate note).

Does that make sense?

Avi

Avi Bryant wrote:
> No, I don't think I would have said that - I certainly don't agree  with
> it :)  What do you find weak about the merge and diff tools in  
> Monticello?  I suspect it may be a matter of being poorly documented  
> (which they are) rather than technically inferior (I would, in fact,  
> claim the opposite).  

The worst thing right now for me is the "Keep" vs. "Reject" buttons in
the merge tool. I have to think *every single time* I use it.
Personally, I think that something like "keep local version" vs. "keep
repository version" (this could be abbreviated) would be much clearer.

Cheers,
   - Andreas

Hi Collin!

> On the other hand, we find it's pretty common to have more organic  
> branching patterns in Monticello,

What is an organic branching ?

Alexandre
--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.iam.unibe.ch/~bergel
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

Am 29.01.2006 um 08:41 schrieb Avi Bryant:

> What do you find weak about the merge and diff tools in Monticello?

The automatic merging works very nicely for us. I even find myself  
formatting methods so they produce a nice diff ;-) Like, we have a  
method that lists the versions of packages used to build an image.  
It's laid-out as one line per package, so in the diff each changed  
package stands out visually :)

However, *if* there are conflicts, tool support is lacking. Apart  
from the wording in the merge-browser's interface, which I got used  
to, I'd like to properly resolve conflicts right there while merging.  
Currently it's either-or: you have to keep your method or replace it  
with the incoming method.

As a work-around, I usually accept all the incoming methods, but open  
a method browser on each of them. After the merge I go to each one to  
properly adjust the code manually, using the changes-based diff to  
see what happened.

It'd be very nice to be able to edit the new method right there in  
the merge browser. Visually, a three-pane layout showing the old, the  
new, and the merged hand-edited version would be very nice. But only  
being able to edit the code in a single pane would be an improvement,  
too.

Fortunately, it happens not very often we actually have conflicts,  
and even then, the majority can be resolved by using either the old  
or the new version. But I guess the remaining cases are what people  
mean when they ask for a better 3-way merge.

- Bert -

300% true. Same with me. I try and fail most of the time and restart  
remembering where I pressed.

On 29 janv. 06, at 08:49, Andreas Raab wrote: