History and thoughts about "how" we work on the image together

Hi all!

(warning - long post - but possible containing some good info and
thoughts)

Hmmmmmm. Ok, triggered by Cees' post on his blog and other discsussions
on IRC I started wondering and questioning our established views on
"how" we should maintain and further develop the "official" Squeak
image. Hehe, we have *never* done that before, have we? ;)

Let us take a step back and look at what we have tried in the past:

1. In the beginning we had SqC, a tight team with full time dedicated
people. A few of the SqCers (Ted and Dan IIRC) worked as update stream
maintainers and did most "harvesting". It worked probably because of the
limited scale and the dedicated time. And because SqC held the rudder,
there was not much discussion going on. There were negative sides of
course - but if we focus on the actual work made on the image, this
model probably worked quite ok given the circumstances at the time.
After all, it is a classic model - a few people in control and most
others sending contributions that may or may not be accepted.

2. Then we moved to a model with a few people outside of SqC maintaining
the update stream and harvesting changes from the list (as before sent
as changesets to squeak-dev). This also worked for a while, but the
harvesters did not work full time nor did they get paid - so they got
really tired. :) And the rudder was not as firmly held, with both
positive and negative effects. One of the positive effects was that the
community probably had a more direct "saying" on what went in - it was a
first step in a gradual transfer of responsibility to the community at
large.

3. As a response to this (the seemingly unsustainable harvesting model)
an new model formed that is based on dividing the image into different
parts (using PackageInfo that came about when Monticello was built) and
then trying to get people to take personal responsibility of those parts
("Stewards") and form smaller development teams around them. Most of us
saw this as the only way forward - and we also recognized that this was
the way that all the "rest" of the Squeak artifacts are maintained (see
666 packages on SM) - so why shouldn't it work also for the parts in the
image?

With 3.9 this partitioning is now official since the 3.9 release team
took the partitioning and ran with it. A good move IMHO, but... (there
is always one, isn't there?)

...developers haven't really *flocked* to the different parts and the
small teams we all wanted to see haven't flourished. The 3.9 team ended
up acting more like harvesters than integrators. So the theory sounds
good - but the practice isn't following. Why is that so?

Perhaps it is simply because we haven't "pushed" this model enough in
the community. Are people aware that there now are different packages
waiting for Stewards to take them on? Are people aware of what that
means? Do we even have this model defined and described a little bit?
No.

So that is surely one of the reasons. But is it really the only one - a
lack of "marketing"?

I think there is at least one more reason. I think that the simple fact
is that a lot of the work on the base packages are done as "side
effects". Things you stumble on while you are writing the next Seaside
or whatever. Things that have bugged you for a long time and suddenly
you just fire up a browser and try to fix it. Try to scratch that damn
itch!

At least bug fixes and small enhancements come about this way I am
pretty sure. So the same people that typically do these small fixes,
possibly all over the place, can't really sign up for each and every PI
of the image. And most of these people are also pretty busy so just
signing up for one of these PIs is a "step" that is easiest not taken.

Just looking at myself - yes, I want to help with IO, Collections, a few
tools and probably a bit here and there related to packages I try to
maintain. And yes, I did join the IO team when this was fresh - but
alas, even *I* have dozed off.

So if the vision of "permanent" teams around a Steward for specific
parts of the image fails because of the fact that people aren't that
focused or have that amount of time or interest (or whatever the reason
is) - are there other ways to organize ourselves?

I don't have a clear "package" to offer. :) But I do think we can fix
some problems at least:

1. The Easy Contribution.

Now, is it *easy* for someone to "share" a fix or enhancement? Is it a
single button to push? Is it obvious how to do it? Can a newbie do it?
Is the mental barrier low enough or is sharing a fix connected with a
bunch of prerequisites like code quality, unit tests etc that makes it
seem "hard work" to do it? Have you ever skipped sending in a fix
because it just felt like a bit too much work at the moment you made the
fix? Truthfully? :)

Earlier it was kinda easy. You picked the changesorter, sliced out your
fix, wrote a preamble and fired it off with a "mail to list". You knew
that perhaps it would not be picked up, perhaps not - but at least this
first step was easy. And yes, we drowned in such contributions and
suffered from lack of harvesting - but that is still the second step -
the first step seemed to work pretty ok.

Today it is *almost* as easy. We still use changesets for fixes etc, but
we upload to a Mantis installation instead. It gives better tracking and
so on, but the steps to create an issue etc is more complex. You need to
get an account and figure out what to write in various fields and so on.
But sure, it is still fairly easy. I bet we could make it even easier
though.

2. The Bleeding Edge.

Today we really don't have a bleeding edge. This is an idea that has
been around in various shapes and forms, a separate alpha update stream
or whatever. This is also what most other "more conventional" OSS
projects have - the CVS/SVN or whatever repository. The classic model is
to have trusted developers that can just "commit" straight onto a
bleeding edge - and then it gets gradually frozen, tested, distilled,
reviewed or whatever when the times come for a release.

Our corresponding bleeding edge today would be a range of MC repos which
you would need to track and manually pull snapshots from - for each
package in the image. There are many. And the snapshots typically
interdepend on each other so you just can't pull the latest from them
all.

I bet that if we had the official image available in a similarly "easy"
way as a single CVS/SVN repo - it would be much easier for developers to
engage. Doesn't this idea go against the partitioning idea? Yeah,
probably. But perhaps we can make them live side by side!

Consider if we just set such a thing up and let developers "go wild".
The maintainers of each separate piece could then track this thing and
"shout" if they saw something go astray. Or simply roll back or "fix"
changes that missed the mark. And they could then take "sanctioned"
snapshots of their each package and put that in a repo.



Ok, this email is FAAAR too long already. I did have a number 3 too -
but it is still a bit vague so I will stop here.

regards, Göran

[hidden email] wrote:
> (warning - long post - but possible containing some good info and
> thoughts)

Interesting post. Couple of comments:

> With 3.9 this partitioning is now official since the 3.9 release team
> took the partitioning and ran with it. A good move IMHO, but... (there
> is always one, isn't there?)
>
> ...developers haven't really *flocked* to the different parts and the
> small teams we all wanted to see haven't flourished. The 3.9 team ended
> up acting more like harvesters than integrators. So the theory sounds
> good - but the practice isn't following. Why is that so?

I think it's simply because the 3.9 team never understood itself as an
integrator of externally maintained packages. Instead of delegating
fixes to the appropriate maintainers and waiting for those fixes to be
turned around they added them directly to the image. I can understand
why they did it (it's a lot faster to push this stuff directly) but when
it comes to enabling others this is a plain slap in the face.

Personally, I think that the only way to make progress in this matter,
to really enable and to push the maintenance issue, is to make the
community feel that actions have consequences. As long as the release
team will do the work for everyone they'll only burn out with little
positive effect overall. But what if there wouldn't be a single change
in Morphic in 3.9 because there isn't a team which has delivered a
Morphic package for the release? What if the fixes to Files and Sockets
are missing because there isn't an I/O team delivering a package for the
release?

I think you'd *very quickly* find a few people who find that they have
at least enough time to fix and integrate a bug here and there. And that
is a starting point, people grouping around that package to fix a bug
here or there.

> 1. The Easy Contribution.
[... snip ...]
> Today it is *almost* as easy. We still use changesets for fixes etc, but
> we upload to a Mantis installation instead. It gives better tracking and
> so on, but the steps to create an issue etc is more complex. You need to
> get an account and figure out what to write in various fields and so on.
> But sure, it is still fairly easy. I bet we could make it even easier
> though.

I know that people bitch and moan about Mantis but all I can say is that
compared with the alternatives, Mantis is hugely advantageous. Even the
threshold to post a bug is good in my understanding. It makes sure that
if you go to the length of reporting a bug (and it's really not *that*
much effort) you try to provide good and accurate information. 9 out of
10 bugs that I come across are clear, obvious, and for those that come
with a fix the fix is usually just fine the way it is. Meaning I can
avoid sifting through tons of unrelated emails (the BFAV problem) and
focus on solving or integrating the fixes. In other words, lowering the
bar is not necessarily advantageous.

> 2. The Bleeding Edge.
[... snip ...]
> Consider if we just set such a thing up and let developers "go wild".
> The maintainers of each separate piece could then track this thing and
> "shout" if they saw something go astray. Or simply roll back or "fix"
> changes that missed the mark. And they could then take "sanctioned"
> snapshots of their each package and put that in a repo.

I don't think so.

Cheers,
   - Andreas

--- Andreas Raab <[hidden email]> wrote:

Personally, I think using a bug tracker is a "MUST"
when you're dealing with so much
code/packages/environments/projects...  It gives you
history and traceability among other things (plus easy
reporting, tracking, etc.)

Regarding release 3.9 (I came back to Squeak after a
long pause), what bugs me the most is the fact that a
lot of packages don't load properly (or don't load at
all!) in 3.9.  I've always thought not loading your
code into a clean image prior to releasing is a
capital sin.

Yes, some would say it's easy to whine and complain
but I think we can definitely improve the process and
the user experience with a little more effort here...
like trying to load your packages into a clean
image...

Btw,  since I'm mostly interested in database stuff
(99% RDBMS), I'd volunteer to help/test/document the
ODBC/MySQL/WhateverRelationalDatabase area...

-----------------
Benoit St-Jean
Yahoo! Messenger: bstjean
A standpoint is an intellectual horizon of radius zero.
(Albert Einstein)

Hi Andreas and all!

Andreas Raab <[hidden email]> wrote:
I would guess that the 3.9 team disagrees :) - but it doesn't matter -
what matters is how we go forward from here.

Do we believe in the model? Will the next release team (3.10 or 4.0 or
both in parallell :)) work differently? Do we need to create some team
(or appoint a role in the current board at least) to get these teams
going - like explaining it more, trying to help people form teams etc?
Should we focus on central infrastructure for these teams or let them
"spread" any way they like? And so on.

Yeah, it might work. We would of course get a "breather" period while
teams get formed, but for the next release that might not be a bad idea.
Robustness, bug fixing, cleaning might be what we should focus on
anyway.

I don't :). I was just comparing the "Easyness" of making the
contribution. As we are totally in agreement with, we need an issue
tracker of course.

> compared with the alternatives, Mantis is hugely advantageous. Even the
> threshold to post a bug is good in my understanding. It makes sure that
> if you go to the length of reporting a bug (and it's really not *that*
> much effort) you try to provide good and accurate information. 9 out of
> 10 bugs that I come across are clear, obvious, and for those that come
> with a fix the fix is usually just fine the way it is. Meaning I can
> avoid sifting through tons of unrelated emails (the BFAV problem) and
> focus on solving or integrating the fixes. In other words, lowering the
> bar is not necessarily advantageous.

I agree with your points too. I am just afraid that we are missing lots
of "small fixes" because people don't take the time. I might be wrong.

> > 2. The Bleeding Edge.
> [... snip ...]
> > Consider if we just set such a thing up and let developers "go wild".
> > The maintainers of each separate piece could then track this thing and
> > "shout" if they saw something go astray. Or simply roll back or "fix"
> > changes that missed the mark. And they could then take "sanctioned"
> > snapshots of their each package and put that in a repo.
>
> I don't think so.

To clarify - such a "hot bed" could work like a "Zoo" or a place where
all these "trivial fixes" could "just be made" and then easily harvested
simply by using MC selective merging or something. Just a thought.

> Cheers,
>    - Andreas

regards, Göran

Hi!

Benoit St-Jean <[hidden email]> wrote:
> --- Andreas Raab <[hidden email]> wrote:
> > I know that people bitch and moan about Mantis but
> > all I can say is that
[SNIP]
> Personally, I think using a bug tracker is a "MUST"
> when you're dealing with so much
> code/packages/environments/projects...  It gives you
> history and traceability among other things (plus easy
> reporting, tracking, etc.)

I never said otherwise. I was talking about the easiness of the
contribution. I am even writing an issue tracker for gods sake. ;)

> Regarding release 3.9 (I came back to Squeak after a
> long pause), what bugs me the most is the fact that a
> lot of packages don't load properly (or don't load at
> all!) in 3.9.  I've always thought not loading your
> code into a clean image prior to releasing is a
> capital sin.

Eh... if you are talking about the 666 packages on SM - the release team
has nothing to do with the external packages - that is up to the
maintainers to move forward.

> Yes, some would say it's easy to whine and complain
> but I think we can definitely improve the process and
> the user experience with a little more effort here...
> like trying to load your packages into a clean
> image...

Again, this has not been the task for the release team. We could of
course change that - but I don't think that is reasonable.

regards, Göran

I agree.
We did what we did because we tried to still make the system move  
forward in presence of ghost team.

I read the blog of cees and it reminds me the post of someone  
thinking aloud but not
really actually doing something to improve the situation. Blogging is  
so easy.


Sure I think with 5 people payed full time we could do a lot.


I do not understand your sentence above so I will not reply.

> ...developers haven't really *flocked* to the different parts and the
> small teams we all wanted to see haven't flourished. The 3.9 team  
> ended
> up acting more like harvesters than integrators. So the theory sounds
> good - but the practice isn't following. Why is that so?

Indeed!

> Perhaps it is simply because we haven't "pushed" this model enough in
> the community. Are people aware that there now are different packages
> waiting for Stewards to take them on? Are people aware of what that
> means? Do we even have this model defined and described a little bit?
> No.

Exact.

Indeed. Doing something regularly a little at the time but steadily  
is really difficult.

Send a fix to who?


At least this is rationalized (I do not find my babies inside Mantis  
but I like the infrasttructure
it represents).

Have a serious look in the inbox of the squeakfoundation source to  
see that
it cannot work. What can work is that you have a guy responsible of  
package that sorts the
stuff he wants to get in. The main problem is cross cutting changes.  
because you need several
packages to be in sync.

Stef

--- [hidden email] wrote:
> I never said otherwise. I was talking about the
> easiness of the
> contribution. I am even writing an issue tracker for
> gods sake. ;)

I know Goran, I didn't mean the contrary!  Personnaly,
I think Mantis does a good job and is pretty easy to
use.  And I too am writing a bug tracker in
Smalltalk...  :)

> Eh... if you are talking about the 666 packages on
> SM - the release team
> has nothing to do with the external packages - that
> is up to the
> maintainers to move forward.

I know.  The 3.9 release had LOTS of stuff in it and
the release team did a great job but I was pretty
disappointed when most of the packages I tried to
install failed one after another...  As I said in an
earlier post, yep 3.9 is great but it would be MUCH
cooler if we could install all those cool packages and
not try to figure out why we get a debugger, errors,
etc. most of the time.
 
> Again, this has not been the task for the release
> team. We could of
> course change that - but I don't think that is
> reasonable.

No, that wouldn't make any sense like you say.  It's
everyone's responsibility to not only develop but
ideally also maintain their own stuff so their
packages are in sync with the newest release as
quickly as possible.


-----------------
Benoit St-Jean
Yahoo! Messenger: bstjean
A standpoint is an intellectual horizon of radius zero.
(Albert Einstein)

I missed this email originally it looks like.  But what if every MC
repository had two branches.  One for "bleeding edge" type things where
anyone from the comunity can write their changes, and a "production line"
repository where the package maintainer could publish what he liked from the
submissions to make the stable branch.  Then the squeak map concept could be
changed to just look in these places for the packages (or at least it could
be an option).

I don't know what package maintainers are/were in the current/previous
system, but in most OSS projects of any size they wind up being people who
mostly merge changes of others (and run the test suite against it, etc.,
etc.) AFAIK.

And taking a concept like this even further, change systems like DVS usually
have a way to maintain multiple branches at the same time (i.e. gcc 3.*
still maintained while 4.* is being worked on and even released).  Perhaps
if the back end repository where some kind of seaside application, as
opposed to a file system, then we could automatically have such branching.  
Someone checks in a new version for 4.* and the system just creates a new
section for it.  And the maintainer has some kind of switch to say in what
branch to look for the official release, so that squeakmap and friends know
where to look.

Just a thought.  Don't know if it's possible, reasonable or desirable. :)

I would just like to say that, after having participated in the previous
Stewarding effort to a considerable extent, I'm extremely interested in
experimenting with a more anarchic/distributed process.

In particular what I would like is to have a live system of 'updates'
where anyone with sufficiently high SqP ranking can submit an update
directly into the 'pool' and it is immediately available to everyone
subscribed to the 'pool'.  Then either through automatic or manual means
each 'update' receives votes (the number of votes each person can give
or take away may be based on their SqP ranking).  The idea is then that
someone who wants to live sort of on the edge, but not too close, can
set things up so that they only get automatically get updates that have
a vote of at least X and if they have an update and it's vote drops
below Y (X>Y) then it is automatically removed.

My idea then is that a release team would at some specified time build
an image from this pool of updates based partially on the votes and
partially on manual checking/verification.

Let me stress that this is something I would like to experiment with,
not immediately use for an official release.  I think this system has
possibilities but only something like real-world testing can really
address how well it is likely to work in practice.

Ken

So try :)

Stef

On 9 oct. 06, at 22:49, Ken Causey wrote: