[squeak-dev] Responses to IRC Questions arising

I am writing this note to answer some questions/issues that were
expressed in todays irc logs about issues pertaining to installer and
LPF etc.

=========
Item 1
=========
kencausey: yeah, that's one solution to an explosion of custom images,
custom scripts instead
[20:35] matthewf: is lots of custom images bad?
[20:36] matthewf: I think not
[20:36] kencausey: It's bad for me trying to help people who come here
asking questions.
...
kencausey: matthewf: yes of course, but it's a pain in the *** to have
to try to replicate that to answer a single question.
[20:45]

My response:

I have started a "bzr" repository with 160Gb of space that can/will be
used to manage images what have been built with these custom scripts.
The automatic building tool "Bob" has yet to be written, but will be
based upon Sake.

So when a user is reading a particular "Squeak By Example" edition or
(insert favourite tutorial here) this may be associated with a specific
"check out" from the repository. Those attempting to answer beginners
questions in any script-built-image will be able to download the same
image that the beginner is using via simple bzr checkout.
=========
Item 2
=========

[10:59] edgardec: So, release team now was myself, jerome, ken and matthew

Do I get credit for causing all of this problem in the first place?

=========
Item 3
=========

snake_mountain: I am uisng Pidgin for IRC.  How's about IRCe?  Will not
install.
snake_mountain: Ah, oh.  "Error:  Unable to find function address>".
snake_mountain: ExternalWebBrowserUnizMozilla(Object)>>error.
....
edgardec: I bet you use Universes for having IRC installed ?
[13:30] snake_mountain: It was recommended last night right here on irc
edgardec: Maybe who build the package have some more in mind, but the
fact is you DON'T need any external
[13:32]

My response:

So here we have a package in universes which needs fixing. I think the
3.10 universe has been frozen, perhaps the maintainer nominated in the
universes browser could take a look at it. However this is where
Sake/Packages comes into its own. Sake/Packages has the same universes
package definitions, but encoded as methods on a Class. Because it is a
class you can override these methods to your hearts content in the
subclass,  in order to provide something with does work for you. If it
works you can then post the mcz back onto squeaksource for others to
take advantage of the new improved definition.

Anyone can edit these packages on squeaksource so the whole enterprise
is to be organic and socially policed, I have proposed
packages@[hidden email] as a forum for this, however the
current owner is not responding to emails (Marcus D?). To whom it may
concern, please could I volunteer to adopt that list?

So... I have now done this, fixing the IRCe package definition in
Packages-Squeak310

Installer sake addPackage: 'IRCe'; install.  "should now work"
Installer universe addPackage: 'IRCe'; install.  "remains broken
awaiting the maintainers attention"

=========
Item 4
=========
wBryce: What's the list of fixes planned for 3.10.1?
[14:54]

The current list according to the LPF collection is here

http://installer.pbwiki.com/MinorFixes-Squeak3:10

and here

http://installer.pbwiki.com/MinorFixesUnstable-Squeak3:10

The later being the current candidate list includes many very minor
items that could be safely included. (the images I personally use
include all of these)

==========
Item 5
==========
wBryce: edgardec, It may be worthwhile doing a quick 3.10.1 followed by
a 3.10.2 with a longer UAT/gamma period.
[15:03] edgardec: 3.10.2 ?
....
[15:03] edgardec: I wish do 3.11 instead

My response:

At present I propose that 3.11 be assembled as

LPF + Latest (MinorFixes, MajorFixes, PackageUpgrades) + Clean (remove
more packages)

i.e. generated by something like

squeak squeak3.10-7159gamma http:://installer.pbwiki.com/f/LPF.st
Installer do="Latest;Clean" SmalltalkImage save=squeak3.11a.image +quit

Edgar, you have considerable experience of removing packages form the
image, and it would be nice if you might help work on the "Clean" script
so that it may be applied all versions of squeak 3.7 through to 3.10
etc, in order to help everyone to tighten the belts of their production
images. How about it?

========
Item 6
========
wBryce: edgardec, I'd suggest figuring out a more automated process of
applying changes for 3.11. Even if it means you've got to be more fussy
about how you accept changes.
[15:10] wBryce: Also, what's the scope of the 3.11 release? Any big
changes, or another maintenence release?

My response: The brainstorm page that matthew and I are using for
throwing ideas around is at:  http://installer.pbwiki.com/311 you are
welcome to comment and contribute there if you wish.

[15:10] edgardec: wBryce: I agree, but how we automate mor ?
 
My response: Since I have been working for more than 18 months on
"automating this more", and we currently have a completely transparent
automated process with Installer + LPF + installer wiki + mantis etc,
what more automation could you possibly be asking for? I dont think it
is possible for squeak to fix itself, there is nothing more that can
possibly be automated so I am amazed by the question. Either that or I
am completely missing the point somewhere?

wBryce: Keith's got code to load Mantis issues. I wonder if that could
be used to automatically update an image.
[15:33]

My response: indeed.

wBryce: You'd want to make sure that the image building code was easy to
unload before a final release.

My Response: This is why Sake/Packages and Sake/Tasks are implemented as
simple methods on a class.
All you have to do to remove image building code is to remove the class.

Part of the Sake/Packages design is to support the idea that the
Sake/Packages package definitions be loaded temporarily for the purpose
of building an image and disposed of prior to deployment. It can be
loaded again on demand if needed.

=======
Item 7
=======

kencausey: One thing I don't care for about Keith's solution is that it
is easily hackable, so doing anything automatic with it, for general
release, is not a good idea.
....
kencausey: Well, it reads from web pages that are world writable, as
simple as that

My Response:

A) installer.pbwiki.com is currently password protected, and can have
any level of security you desire to make it non "hackable"
B) Use another wiki with more watertight security, Installer is not
hardwired to installer.pbwiki.com
Bb) Use local files if you want to, we can mange these in any SCM, I
dont think this would have much advantage over the wiki.
C) Installer mantis scripts can protect themselves from people hacking
fixes with new code by specifying a date. If the date doesn't match a
warning is given.
D) In theory you can download a zip archive of the whole site and use it
locally
E) If instead of Installer install: 'MinorFixes', you run Installer
view: 'MinorFixes' you get a worksace with the entire script generated
from the wiki. This can be executed as a doit or saved for use as a FileIn.
F) installer.pbwiki.com is NOT intended to be the final release
generating mechanism. It is a knowledge collection tool for collecting
knowledge and fixes from the community and experimenting. There are two
plans for generating the final release.
Plan 1: Using DS, DS will record the progress of all the scripts, the DS
can then be used to generate cs for update streams.
Plan 2: Sake/MCTasks run after the scripts saving the whole image as mcz
packages. The final golden master release image is generated from a
fresh image by loading the mcz's (with atomic loading of course).

=======
Item 8
=======
kencausey: Well, number one, Mantis issues have all sort of stuff
attached to them
[15:38] kencausey: Sometimes examples of what not to do
[15:38] kencausey: So some human is going to have to designate what to test
 
My Response: This is what Installer mantis does... it picks out the
script placed on the bug report page, the script specifies the actual
fix, and tests for that fix. Mantis users are able to write and adjust
the contents of the script as much as they wish.

kencausey: I could maybe even add something to Mantis to make it easier

My Response: Its fine, its not perfect but it has been working most
effectively for over a year. (If it aint broke...)
The current system whereby user-notes are used for the Installer mantis
script is a little tricky since one person cant edit a script posted by
someone else. Currently this is handled by appending a new script. Would
it be desirable/possible to have a mutually editable note for this purpose?

=======
Item 9
=======

wBryce: I was just idlely speculating if there was any way to automate
as much of the drudge work out of the release team's job.
[15:42] kencausey: Personally I think that the drudge work is useful as
it requires human interaction and opportunities for thought, examination
of the issue, the code, etc.
[15:42] kencausey: As it is bad code repeatedly slips through.
[15:42] kencausey: How is that going to improve with further automation
without more eyeballs?

My Response: I was once a passenger in a car whose driver would insist
on driving at 90 mph everywhere in the black mountains of Wales. His
excuse being that the faster he went in the wrong direction the sooner
he would realise that he was lost.

Removing the drugery is the whole point of using LPF and Installer image
building tools. With installer.wiki we have a framework which is A)
Public B) Provides space for Documentation of design intent and design
decisions C) Open to all interested parties D) Anyone can build the
latest image and test it locally for their platform E) Anyone can
dry-run their own variants and experiments as they wish.

Using code snippets that are publically visible on
http://installer.pbwiki.com lots of people can contribute to the release
in a small way or a large way as appropriate for them. i.e. Lots more
eyeballs. Take Nicolas Cellier, and look at all the tiny little
contributions he has made to FixesUnstable. Its about harnessing the
creativity of the community by creating opportunities to be inclusive,
rather than defaulting to an exclusive team.

The update stream is exclusive, being in the grip of a privileged few,
so thats a non starter IMHO.
Previous release teams though desiring to be inclusive have not
succeeded as they might have wished.

Here we are planning for inclusiveness, and we are including users of
past images as well as present ones. If someone wants to take 3.11 work
and migrate it to 3.7 the mechanism is there, now that we have a "Level
Playing Field" to start with.

======
Item 10
======
kencausey: The problem with relying on tests as it requires reasonable
test coverage
[15:43] kencausey: Which we are not even close to
[15:43] kencausey: And I'm afraid the closer we get the more annoying
testing is going to become

My response: 14 months ago I wrote code for automated testing, this
applied the "pareto principle" whereby 80% of the result is achievable
with 20% of the effort. I did this by recording the time taken by each
test case, and running the fastest tests first. This way 90%-95% of the
tests run in less than 5mins. I cant remember the exact figures, but it
was a significant win. This also highlights the slowest tests for
improvement, or less frequent running.

======
Item 11
======
wBryce: In my case, it would be worthwhile to have a build server that
built a Exupery dev image then ran all tests. So I found out when
something broke at the time, rather than when I next release.

My Response: you can do this now*

Write an Installer build script to build your image  exupery.st

#> squeak non-LPF-start.image http://installer.pbwiki.com/f/LPF.st 
Installer installUrl: 'file:/exupery.st' SmalltalkImage
save=exupery.image TestReporter runAllSuites  SmalltalkImage +quit

*some of the Launcher code for running TestReporter from the command
line has been lost and needs to be retrieved from earlier versions of
Launcher


=============
sorry its been a long one


cheers

Keith

El 5/10/08 11:23 PM, "Keith Hodges" <[hidden email]> escribió:

> Do I get credit for causing all of this problem in the first place?

I wish do Ferrari team.
It's Board who should give us rules as FIA have for all teams.
And others teams exist, like McLaren-Mercedes, Renault, Toyota, etc.
So you should start yours and work with people trust you and be comfortable
with you.

And I don't answer any flames war.

Edgar

2008/5/11 Edgar J. De Cleene <[hidden email]>: To Keith, Matthew, Edgar & all interested parties:

First, of course, it is up to release team to decide, how they see the
work should be organized.

But, IMO it would be better to consolidate resources and work in team,
instead of diminishing resources.

I think you better to discuss the points where and how you will cooperate.
So, lets think about how to organize the work and divide
responsibilities between parties/team members.

For instance, if next release can be based on fully automated
LPF/installer scripts i expect to see a detailed description with
step-by-step instructions how developers can step-in, add change and
step out, without chances that they will break/intrude on field which
is under responsibility of others.

So, i suggest you to work out a detailed workflow , write it on the
wall and strictly follow it during work on release.

--
Best regards,
Igor Stasenko AKA sig.

On Sun, May 11, 2008 at 02:01:57PM +0300, Igor Stasenko wrote:
Yeah. That's what we discussed in the last release team meeting,
which was like a month ago, and I still haven't posted the
summary. My excuse is that it was the last month of the school
semester (ya know, when all the projects are due). I'm working
on it; I'd like to make a nice set of documents like I did for
the relicense effort (which is currently on hold).

> For instance, if next release can be based on fully automated
> LPF/installer scripts i expect to see a detailed description with
> step-by-step instructions how developers can step-in, add change and
> step out, without chances that they will break/intrude on field which
> is under responsibility of others.
>
> So, i suggest you to work out a detailed workflow , write it on the
> wall and strictly follow it during work on release.

Definitely. I'm on it. Thanks for the reminder.

--
Matthew Fulmer -- http://mtfulmer.wordpress.com/

El 5/12/08 1:23 AM, "Matthew Fulmer" <[hidden email]> escribió:

The current release model is Ralph model.
No scripts, no external mumbo jumbo.
As example I resurrect ReleaseBuilder , used in 3.8 and older releases.
Bert at the time we start advice about how do this.
And Ralph says no Doits, no weird things.

So if we wish unload Traits, we must have a #unloadTraits as we have #
unloadMorphicClasses and #unloadSome in ReleaseBuilderFor3dot10.

In ReleaseBuilderFor3dot11 we have #unloadSomeMore2 ,#unloadSomeMore3 and #
prepareforUnloadBookMorphandFriends #prepareforUnloadEtoys #
prepareforUnloadNebraska

Keep in mind in the future we should converge to Pavel works
(MinimalMorphic) to Craig Spoon or to Juan Vuletich Morphic 3.0, the three
forks I have as example of how I wish future Squeak should be.

And remember I was in four teams before , the last two was both
MorphicTeams.

So maybe I know some...
And work for have it.

Now people say FunSqueak should be next full release, very happy with that.

In Smalltalks 2007 I show all the FunSqueak, six months ago...

Soon , if Lord wish, I become 59.

So to old to waste my time.

Or Board say I could do 3.11 and people liking to work on this with me start
to Smalltalk...

Or we could continue JustTalking.

Cheers.

Edgar

Dear Edgar,
> The current release model is Ralph model.
> No scripts, no external mumbo jumbo.
>  
The external "mumbo jumbo" as you describe it is only part of the story.

The "external mumbo jumbo" bit of the process is external for those
users who are starting from different starting points. A KernelImage or
a 3.7 image does not have a ReleaseBuilderFor3dot11 class in it, so
running ReleaseBuilderFor3dot11 unloadTraits will not work for those
users, particularly since it has not be designed to work for those
users. To write a ReleaseBuilderFor3dot11 unloadTraits method adopts
what I think is a short sighted approach to the future of squeak,
Matthew and I are trying to promote something more dynamic and interesting.

The "external mumbo jumbo" provides a collecting point for ideas, and
the ability to pick and mix components in order to build the image you
want in order to test ideas.

Case Study:

When I wrote Rio initially it relied upon numerous kernel-level changes.
This was a problem because most of my potential users would load it and
find that they didn't have the required dependencies. So I have a choice
to make, either,

1) write new code for the future that is only usable by users with all
of my latest bug fixes in the very latest image, (i.e. my use only) or
2) throttle back on innovations in order to provide a package which uses
a lowest common denominator, for everyone to be able to use it

Another example, Monticello, either

1) move MC forward only for users of the latest greatest squeak image, or
2) Make MC use the lowest common denominator code.

This is difficult to actually achieve since I am using the latest image
to work in. So I have to keep testing MC in 3.7 for backwards
compatibility, one might as well continue to use 3.7, but then what
happens if something in a later image becomes deprecated (e.g.
Collection-upTo:) in this case you have to continually test for forwards
compatibility.

We are aiming to provide an alternative to the two options above. To
move to a model where a package such as MC can be loaded and tested in
ALL images automatically. At the same time enabling the differences
between starting point images to be factored out of the main MC
package.  This loading and testing tool is the tool that I have
christened "Bob".

So... The "external mumbo jumbo" as you call it is external for a
reason, there is no need for all of it to remain external. At some point
it is able to hand over processing to internal code.

Sake/Packages does this. The "external mumbo jumbo" loads the
appropriate "internal mumbo jumbo" for the users image, be it, etoys2.3
from OLPC, or Croquet, 3.7 or 3.10. The internal mumbo jumbo defines all
of the packages available, how to load them, and unload them. Loading
and unloading can capture any bug fixes needed for any individual
package. All dependencies are also captured, using a code based model,
built upon the Sake scaffolding.

And then there is Sake/Tasks (work in progress) , again the "external
mumbo jumbo" loads the appropriate "internal mumbo jumbo" for the image
the user is using. The Sake/Tasks equivalent of your ReleaseBuilder
unloadTraits, would be.

TasksSqueak311 unloadTraits run "or perhaps"
TasksCommon unloadTraits run.

Being a task, unlike "ReleaseBuilder unloadTraits", it is not designed  
to be run exclusively in the context of a single carefully constructed
realease building script, it is designed to be more intelligent than
that, to be a bit more self aware, so that any user could decide at any
point no matter what image they have... I dont want traits any more,
lets unload them now.

As a task it doesnt run if it has been run already, it doesnt run if it
is not needed, and it can define pre-requisite checks,  and may have
dependencies upon other tasks. As a task it also comes with Tracing and
Logging tools for debugging what is happening when and where. All fine and good, but basic non-task methods lack any formalization for
pre-flight checks, dependencies or logging and tracing. They are one hit
wonders. You could argue, who cares, you only want to run them once. But
in a bigger more diverse release team trying out different ideas in
different images, in different scenarios, packaging this knowledge as a
Task makes it more useful overall.
> Keep in mind in the future we should converge to Pavel works
> (MinimalMorphic) to Craig Spoon or to Juan Vuletich Morphic 3.0, the three
> forks I have as example of how I wish future Squeak should be.
>
>  
As you state here, there are several options as to the way squeak could
or should go. We as a community are not yet used to working with this
level of ambiguity.

You want to push on ahead with 3.11, while the rest of us see that as
dependent upon more supporting tools such as I have described above. If
you want to rush on ahead of us then fine go ahead, but you actually end
up making our job harder, since it is more difficult to track 4 fixed
targets (3.7, 3.8, 3.9, 3.10) and 5 moving targets (OLPC, Croquet,
Sophie, 3.11, Pharo), than 4 fixed targets and only 4 moving targets.

Furthermore this will result in you taking on ownership of the 3.11
project and leaving us the rest of us behind with no image of our own in
which to showcase these ideas, you will force us to fork 3.10, and for
goodness sake we dont need another fork just yet.
6 months down the line you will be complaining about how you have had to
do all of the work yourself and that no one supported your effort
sufficiently to your liking.
> Now people say FunSqueak should be next full release, very happy with that.
>
> In Smalltalks 2007 I show all the FunSqueak, six months ago...
>
> Soon , if Lord wish, I become 59.
>  
I am sure the Lord is fine with that idea.
> So to old to waste my time.
>  
Then dont waste your time, get with the programme!
> Or Board say I could do 3.11 and people liking to work on this with me start
> to Smalltalk...
>
> Or we could continue JustTalking.
>  
As I have said before, we have not been JustTalking, Matthew for one has
been working hard, I have been dealing with personal issues for a bit,
and trying to pay the bills. There has been significant progress made if
you just open your eyes to see it.
> Cheers.
>
> Edgar
>
>  
Best regards

Keith
>
>