[squeak-dev] A New Community Development Model

The objective of the existing 3.11 effort was to produce "a new
community development model" based around the idea of planning new
releases ahead as a coded task. This task does things like so...

0. Start with fixed starting point.
1. load latest Installer/Monticello tools
2. load the latest versions of important packages that are managed
externally e.g. SUnit / SqueakMap
3. (optionally) Major architectural integrations (i.e. closures) pre-fixes
4. load image fixes that have been approved for 3.11 in mantis
5. (optionally) load image fixes that are marked as testing for 3.11 in
mantis
6. reorganise the image into tidier categories and better packages as
supported by new features in PackageInfo
    e.g. Monticello1.5 is now split into Monticello.impl and
Monticello.test
7. (optionally) Other major architectural integrations (i.e. closures)
post-fixes
8. export the result as MC packages
9. (optionally) load a bunch of packages to make a full/dev image
9a release 3.11-dev 3.11-fun 3.11-web 3.11-magmaserver
10. (optionally) Run all tests
11. (optionally) remove some packages
12. (optionally) remove externally managed packages
10. (optionally) remove all tests
12a release 3.11-minimum
13. (optionally) remove all tools - mc/installer etc.
13a. release 3.11-kernel
14 repeat back to 0 until satisfied.

This coded task, allows someone to be working on SUnit for example in
parallel with someone working on Mantis Fixes. It is not sequential it
is an iterative build process, which results in a sequential progression
in the MC repositories (step 6)

An outline plan for 3.11 the image [step 5 above] has been present in
code form for over a year, we were simply continuing to make the tools
to make the rest of the process happen i.e. (steps 4,5,10,13 etc) on a
frequent (i.e daily basis). Once in place this will free us up to make
releases on a far more continuous basis, say every 2/3 months.

Since Andreas has summarily announced a "new new community development
model", as far as I can see, and I am willing to be corrected, this has
rendered the "old new community development model" outlined above
irrelevant.

I apologise for listening to the board when Randal said relax on 3.11
because we want to put relicencing on the critical path. Andreas took
the inaction that followed as a queue to interfere, while I was simply
trying to pay the bills for a month or so, and we are experiencing a
welcome heat wave (whoever said global warming was a bad thing!)

At present I cant see what Andreas' model is good for, ok so you
contribute for a few MC packages in a kind of undocumented free for
all.  I don't see how Andreas' new model presents any new philosophy or
way forward for the different forks of squeak, so apparently we are
being forced back to the old way of doing things. Again as far as I can
see Andreas' new development model has no place in it for planning a
re-organisation of the image improving the package structure, moving
packages out to be loadable, supporting derived smaller and larger images.

To demonstrate my point, who is doing the work in Andreas' new model.
Yes, that's right it's Andreas, as far as I can see we are back to there
being one bottleneck in the contribution process.

The old new model (above), defines an integration process that a number
of contributors may contribute to at different stages. If for example
you choose to adopt the project of improving SUnit, then you know that
your work will be included in the release because your latest code is
loaded on each build iteration, and fully included in the integration
testing that everyone sees, and can give you feedback from.

The old old (i.e. 3.10) way, relies upon the release team deciding
whether or not to use your work when they have time to get around to
checking it (which is invariably never) This resulted in the 3.10 team
refusing for no good reason to include the latest version of Installer,
thus wasting the effort that was put into making the latest installer,
leaving 3.10 users with an inferior installer that couldn't even upgrade
itself. The use of a relentlessly moving forward update model resulted
in 3.10 having a broken DateAndTime now which had to be fixed in 3.10.2

The old new 3.11 way (above) automatically includes your work, if it is
included in the coded build task, and gives everyone the tools to
integrate, and test the fully integrated result, and all derived images.
The bulk of the effort that has gone into making 3.11 better than 3.10
is in packages that are loaded externally, aiming to make radical
improvements to the code loading tools, the handling of packages, and
the running and categorisation of tests (I had the misfortune to try and
install something using the python package manager, believe me we are
not the worst in this regard). If you want to help, then please
volunteer on [hidden email] there are lots of small
projects that could be helpful.

When you submit a fix to mantis you are a contributor automatically to
the "apply fixes" steps 4,5 in the build process. The Andreas model is
returning to relentless moving forward in the control of one person
deciding what is in and what is out without the ability to craft the
build process.  We threw out the idea of a continuous updates mechanism
because this only tends to facilitate incremental additions, it is a
nightmare to carefully surgically reorganise or remove a chunk of code
if you have to keep the image working all the time, and have to maintain
the same package structure.

I do not have the energy or the will to compete, either you the public
want a new way of doing things or you don't. So unless Andreas
voluntarily seeks to understand and co-operate with the present but now
apparently obsoleted vision, I have no alternative but to wish you all
luck in the good ship Andreas, the new release team doing it the same
old fashioned way.

Lets take the suggestion for submitting tests as indicated in a previous
email. Has Andreas even looked at the already coded task step for
organising tests as loadable sub-packages according to a much tidier and
more logical design? He certainly hasn't commented on it to me. Has
Andreas even loaded the improved SUnit and looked at the new ways for
tagging and categorising tests?

If Andreas wants to actually contribute to the existing vision, then he
needs to propose an additional step in the coded build task, and show us
how his repositories and update process fits with the rest of the
process. For example if he was to say ... I have a vision... that of
making "Network" a subsystem that I will go off and make really good
streamlined version of that package. Great... you can go off and work on
Network to your hearts content, but it might not fit into the plan for
3.11 it could be scheduled into the code integration task for 3.12 when
we are ready and able to load the new revolutionary streamlined chunk
with atomic loading in one go. (atomic loading is on the roadmap for 3.11)

So like I said if you want to volunteer to help, I will see you on
[hidden email]  We have a vision, and we have lots
to do... but as I said, I cant yet see what relevance Andreas's new new
model is to it. He has proposed a technical idea or two but that in
itself does not a vision make.

The current 3.11 vision includes

- Bob building and integration server
- Crafting images through pre-coded build integration and testing tasks
according to a planned design (not just the whims of the person
integrating the next update)
- Code loading and management tools for all squeak images promoting
cross fertilisation (i.e. Level Playing Field)
- Package management including dependencies for all packages (again for
all squeak forks and kernel images) (Sake/Packages)
- SUnit/SSPec testing with categorisation of tests for different images
- Atomic (un)loading
- More granular package structure
- Split Tests out into paired MC packages that can be loaded/unloaded
- Automated collection of fixes tagged as ready on mantis.

So as in time honoured tradition, the board has demonstrated that it is
irrelevant to squeakers, since it doesn't seek to discuss its decisions
with squeakers before taking catastrophic actions. Heck it didn't even
think of mentioning this to the release team before it was presented to
you all. This approach to dealing with people is the antithesis of what
the "old new community development model" was intended to be all about,
that of inclusion, and valuing people's contributions.

This is the second time that the board has inadvertently cancelled the
3.11 vision as described above, and it probably wont be the last. So I
throw this one back to the board to decide what it really does want. In
the meantime I shall work on my tan.

regards

Keith

2009/7/3 Keith Hodges <[hidden email]>: (the mail is a bit long, so i will comment as i read).
So far, we established a quite a bit new repositories at source.squeak.org
/trunk
/test
/inbox

lets say, all core packages, which is going to include into core image
will be located in trunk repo.
So far , do you see any contradiction with your model?
I may be wrong, but i don't see anything hazardous to your vision. Its
a simply a common place where people putting their code.

Later, Andreas shown how to work with it using Monticello configurations. Okay.
But you can show us the same, but done by using Sake (by writing a
config there) and Installer.
And i inviting you doing so.
That's great!

> Since Andreas has summarily announced a "new new community development
> model", as far as I can see, and I am willing to be corrected, this has
> rendered the "old new community development model" outlined above
> irrelevant.
>
Instead, i think it just make things a little bit easier:
a group of contributors could use a single place to put core packages there.
They don't have to remember/know each package/changeset or source in
other forms location(s).
Okay, so lets focus on making your tools more visible to community.
Not only to release list.
Well, if we want some kind of 'official' image, we can build it using
your tools.
And obviously, there should be someone who oversees this process and
choosing what is going into image and what's not.
Again: i don't see how this contradicts with 'old new' model.

Keith.
I think your work is relevant, and could bring us to a new level.
But i can't force people to go read about it, or start using it and
none of board members can.
I can add much more words, but the above summarises the most of it.

> regards
>
> Keith
>
>



--
Best regards,
Igor Stasenko AKA sig.

A couple of typos
> is an iterative build process, which results in a sequential progression
> in the MC repositories (step 6)
>  
should read Step 8

> An outline plan for 3.11 the image [step 5 above] has been present in
>  
should read step 6

Keith

You are missing a big point here... Take a good look at Step 0, see what
it doesn't specify... it doesnt actually specify the starting image. You
see the same iterative build process could be applied to a 3.9 image or
one of my existing production images. Ok it might not work as well as
the image for which it has been designed but it would be feasible. While
you can load a small change set into a number of images, I doubt you
could load a full MC package.

Where in the coded task steps does it say collect a kernel package (a
moving target) from a trunk repo? It doesn't, in our model the MC repo
packages are the output not the input. - see step 8. So the two
approaches are opposite in methodology.

If I write a mantis fix, I need for the image to be in a vaguely known
state at step 4. If prior to step 4 I load any packages from the trunk
repo then the image will not be in a fixed state for the application of
a fix from mantis. If I apply the trunk repo changes after the fixes,
then the repo package will stamp over the fixes.

We already have a repository for 311, it is called squeaksource.com/311
if you have the good fortune to reorganise a bit of kernel code into a
standalone module, that can be treated as virtually external to the
image, then fine. This can be loaded as a completed "external package"
at step 2 and it will need no further fixes. For example I think
"Object-Tracer" has been published as an "external" package.


> I may be wrong, but i don't see anything hazardous to your vision. Its
> a simply a common place where people putting their code.
>  
Sure if it was just a place to put "your" code, then it would be a
directory with a changeset, or a DS in it. But it isnt, it is a
repository with a whole package in it. Each trunk package represents an
incrementally developed moving target for anyone who wants to contribute.

With Andreas' model when I write my code I am not coding to a fixed
starting point, I am adding my code incrementally to whatever the last
person did. So we have everyone working to a moving target, and everyone
potentially stepping on each others toes.

With my model you pick a jurastiction and you work on coding your
project, you don't have to share your repo with others working on other
tasks.
> Later, Andreas shown how to work with it using Monticello configurations. Okay.
>  
Again, I dont see where this step is in the build plan. In the planned
approach you develop your code in the context of a fully populated
image, with all the tests present, and then when the whole thing works
you export the finished MC packages. Andreas' model is the opposite.

If you want to take a discrete chunk of the core and manage it
externally as a set of MC packages then be my guest. But then propose it
as a project and add your load task to the build plan. It becomes an
externally managed project that needs an integration task. But don't
just sumarily take the whole core and start maintaining it and managing
it in a separate place.

I am not saying that the updates idea is not useful, indeed this was
once the proposed plan. i.e. once you have designed and produced the
final image, we look to make different ways of distibuting the image,
and atomically loaded MC packages is one option.
>
> Instead, i think it just make things a little bit easier:
> a group of contributors could use a single place to put core packages there.
> They don't have to remember/know each package/changeset or source in
> other forms location(s).
>  
MC doesnt manage changesets, it doesnt manage the pieces of the work in
progress it only manages the final results.
>
> Well, if we want some kind of 'official' image, we can build it using
> your tools.
> And obviously, there should be someone who oversees this process and
> choosing what is going into image and what's not.
> Again: i don't see how this contradicts with 'old new' model.
>
>  
Thats not what Andreas is saying... he is saying "right we have this new
repo, and over the weekend I will integrate the 3.11 fixes into it."

The old new model gives a person a jurastiction, lets say they are
working on a particular fix. They continue to work on that fix,  and the
build system integrates it. There is no step in which it passes over to
the release team leader who choses to update the repo with it. That
would be taking the fix out  of the hands of the person responsible for
it, and bringing back the old bottleneck model.
> I think your work is relevant, and could bring us to a new level.
> But i can't force people to go read about it, or start using it and
> none of board members can.
>  
But you can do the simple stuff, like talking, like including relevant
people in discussions, like not pissing people off. In the immortal
words of Jeremy Clarkson "How hard can it be?"

Basically ask yourself who would want to work in these inter-personal
conditions?

Keith

On Wed, 2009-07-01 at 20:03 -0700, Andreas Raab wrote: I've had a question about this.  If a developer writes a test, finds a
problem, and develops a fix is it the intention that she put the fix in
inbox but the test in tests?  Doesn't that just complaint things for
everyone in terms of linking the two?  What is the value of two separate
inboxes?

Ken

Ken Causey wrote:
To me, it's just a pre-filtering that says that anything in tests will
require less work (if any) for review during integration. Effectively I
would expect that if I merge tests this will work with no further
effort, whereas with inbox I would assume that I at least need to look
at what it does. If we find that this isn't a useful distinction we
should get rid of it but I think it would be useful to explicitly
encourage submitting tests.

Cheers,
   - Andreas

Andreas Raab wrote: Andreas Raab wrote: Tests are useful for all users of squeak whatever fork they are using.

A new test should be submitted to Mantis and tested in as many images as
possible, either manually or automatically. The test can be marked using
the test tagging feature of SUNit-improved indvalidate if there are any
images that it is known not to work in for good reason.

Maintainers of other forks can be informed to let them know of the new
test, and they can install it using Installer mantis. Better still get
bob to build an image of their fork with your tests in it and show them
that it is good.

Keith

On Sun, 2009-07-05 at 11:48 -0700, Andreas Raab wrote: OK, I can see your point about pre-filtering.  But then I have to ask
what or who is the consumer of the filtered results?

If human, I have to think it's just as easy to filter based on the
package name which clearly states 'Tests' or should and using just one
inbox simplifies thing by requiring only one place to look.  In much
simplifies things for submitters who don't have to put things in two
places and decide which is right when, relevant for newbies at least.

Ken

Andreas,

I'd like to suggest to add a note in the New Community Development
Model manifesto in regard of the good practices with commit messages
(in rules of engagement). This would ensure some form of uniformity.

Regards,
Ian

--
http://mecenia.blogspot.com/

First let me say that I like the proposal because I feel that  
contributing is much easier that way for me. I think the minimum amout  
of work to make a contribution seems smaller than with the old process.

I have a question though. If I press the Update button do I get all  
the newest versions of the packages automatically? Or only those for  
which an mcm-file was issued.

Thanks for your attention.
Bernhard

Am 02.07.2009 um 05:03 schrieb Andreas Raab:

Bernhard Pieber wrote:
> I have a question though. If I press the Update button do I get all the
> newest versions of the packages automatically? Or only those for which
> an mcm-file was issued.

You get all the packages automatically. The MCMs only describe
intermediate stages where inter-package dependencies require a specific
load order.

Cheers,
   - Andreas

Am 09.07.2009 um 19:49 schrieb Andreas Raab:
> Bernhard Pieber wrote:
>> I have a question though. If I press the Update button do I get all  
>> the newest versions of the packages automatically? Or only those  
>> for which an mcm-file was issued.
> You get all the packages automatically. The MCMs only describe  
> intermediate stages where inter-package dependencies require a  
> specific load order.
Hmm, then I don't understand why I the MCFileRepositoryInspecter shows  
packages in bold on the trunk repository after I have pressed the  
"load code updates" button in the Squeak flap. I thought those were  
packages which are not loaded in the image yet? The following package  
versions are in the trunk repository but not in my image:
Morphic-CandidatesForGo-auto.1.mcz
Installer-auto.1.mcz
SystemChangeNotification-auto.1.mcz
BabySRE-auto.1.mcz

The three update.mcm files are bold as well.

Thanks for your support.
Bernhard

Bernhard Pieber wrote:
> Hmm, then I don't understand why I the MCFileRepositoryInspecter shows
> packages in bold on the trunk repository after I have pressed the "load
> code updates" button in the Squeak flap. I thought those were packages
> which are not loaded in the image yet? The following package versions
> are in the trunk repository but not in my image:
> Morphic-CandidatesForGo-auto.1.mcz
> Installer-auto.1.mcz
> SystemChangeNotification-auto.1.mcz
> BabySRE-auto.1.mcz

Ah, I see. Simple answer: The updater will *only* consider packages that
are listed in the update.mcm file. The packages you see above are indeed
not loaded (they were created as a side-effect of a bulk-commit from
3.10.2 and are effectively orphaned at the moment). Since they are not
listed in the update.mcm the updater will not consider these for loading.

So indeed, my explanation was incomplete. It should have said: The
updater will update the packages listed in the last update.mcm to their
latest versions, not necessarily all packages in the repository.
Packages that are not present in the update.mcm will not be considered
for automatic updating.

> The three update.mcm files are bold as well.

Same issue, except that I think MCMs always show as bold since MC
doesn't have a way of determining whether a particular configuration has
been loaded.

Cheers,
   - Andreas