[squeak-dev] [Ann] Bob has built!

Dear All,

The long awaited Bob, the image builder has been building away! When Bob
is building himself, he will be released as a one click image.

This email is intended to give you an idea of how he works, in order
that we can begin to develop useful scripts for building one-click
images, testing etc.

Bob Components
=============
squeaksource/Sake Sake-Scheduler = Package providing the ability to run
actions/tasks periodically
squeaksource/Sake Sake-Bob = Main image building tasks, based upon sake.
squeaksource/Bob-Releases = Package where we/you add your build tasks,
for distributed bob build/test servers to pick up.

How Bob works
============
There is a Bob build/test server running at http://bob.warwick.st you
can see what he has built and what he is doing by looking around that
site. He is also available via vnc display 1. User/password for ftp/vnc
is as for the swiki.  If you are keen you can download the currently
running bob from http://bob.warwick.st/bob/home

Bob picks up his assignments from the Bob-Releases package, the
following is an example of a build instruction from that package.

Example Bob Build Tasks
===================

BobBuildDev>>#build0000

    self name: 'Squeak3.10.2-dev'.
    self dependsOn: { BobBuildBuildBase taskBuild }.
    self when: true.

    info by: 'Damien Cassou'.

    self image:
'ftp://bob.squeak.org/release/Squeak3.10.2-build/release/*.zip'.
   
    self linkRelease: false.
             
    "    info upload:
'ssh://[hidden email]/var/www/files/3.11/' , self name."
        info upload: 'link:///bob/release/' , info name.

        "http not yet supported"
        info download: 'ftp://[hidden email]/bob/release/' , self
name.

    self comment: 'Learning to breathe'.
""""""
    Installer sake install: 'Squeak-dev image'.  

========
In the above meta-data those lines sent to #self are meta-data items
that the framework knows about, those sent to #info are arbitrary
meta-data that Bob ignores but are included in *.info files that Bob
outputs.

#name - defines where the wip/output/release directories will be, and
what the final image and zip files will be named.

output will be written to:
/bob/output/Dev/build0000/<timestamp>/<timestamp>_Squeak3.10.2-dev.zip
/bob/output/Dev/build0000/<timestamp>/<timestamp>_Squeak3.10.2-dev.info
/bob/output/DevBuild/latest (symbolic link)
/bob/output/DevBuild/release (symbolic link)

The *.info file is the metadata filed out into a man & machine readable
format. Bob uses this to compare what he has already built, with what he
is being asked to build.

dependsOn: - uses the standard Sake method for defining dependencies,
i.e. references the actual task that is used for building the dependent.

#when:  - true/false/#dont/aDateAndTime/aTimeStamp/aDuration/aFile -
specify a time to build, a period, or a file whose creation time is
monitored to see if it changes.

#image: - a match that will find one zip file in which an image is
available.  The file may be a) local b) remote via ftp, or c) local in
the output tree having been generated from other tasks. b) remote files
are cached locally.

#imageCreationTime: aTime - if this field is specified then bob does NOT
poll the image file to see if it has changed. This is set so as not to
keep nagging ftp.squeak.org needlessly on images such as Squeak3.7-basic.

#linkRelease: true/false - indicates whether or not this is to be
designated a release via a symbolic link. Notice how the #image field is
picking up the latest "release" of 3.10.2-build via the "release"
symbolic link, rather than the "latest" symbolic link.

The result of the build is packaged into the
ftp://bob.warwick.st/bob/output directory tree, also visible via
http://bob.warwick.st/bob/output

#upload: aDirectory - a destination directory to send package in the
output directory to - options are:

1) ssh:// perform the upload via ssh (but not as part of the build, this
will be done by a separate uploading thread)
2) link:// perform the upload by linking the /bob/output/Dev/yymmdd-hhmm
to the bob/release/Dev/yymmdd-hhmm directory which may then be rsync'ed
to a mirror ftp site. This is the option that bob.warwick.st uses to
upload files to ftp.squeak.org, because the user that bob is running
under does not have access to ftp.squeak.org, but the user which
performs the rsync does. This allows bob.warwick.st to be made public
via vnc, without compromising the security of ftp.squeak.org
3) list:// perform the upload by adding the file to a list, which can
then be used as an input to rsync in an external process.
4) file:// perform the upload by copying the files to a directory locally.

#download: Bob will look at the download location to determine whether
or not an upload is required.

Multi-Bob-Collaboration
==================
For each instance of Bob the choice of targets to make is specified by
creating the output directory e.g. /bob/output/Dev. If the output
directory does not exist locally bob will not build that target.

Multiple distributed Bob's collaborate by using #image: that are
available via ftp, as generated by another bob. Thus your bob can wait
for my bob to release a new Dev image by pointing to
ftp://bob.warwick.st/bob/release/Squeak3.10.2-dev/release/*.zip  Only
one bob should be designated as an uploader to the final ftp site (see
BobConfig and discussion about Configurations later on)

Multiple Bob threads on a single server collaborate by using the "locks"
described below. Within Sake a task can raise a SakeBlock exception,
Sake then trys to find another task that it can do in the meantime,
returning to the blocked task later on. If a Sake task discovers that
all of its dependent tasks are Blocked it too raises a SakeBlock
exception. If a bob thread finds his tasks blocked (i.e. in progress) he
will try again later.

Blocking - Multi-bob-collaboration
=========================
When a build is in progress this is indicated by a BUILD-IN-PROGRESS
link appearing in the output directory. When an upload is in progress
this is also visible in the output directory like so:
"090308-1316.uploading".

BobBuildImage - Starting Image
=======================
Minimum Requirements for the starting image:

Bob expects to find the image to build on, in a zip file, either locally
or on an ftp site somewhere.
The image simply needs to support the execution of a script (full path)
from the command line. There are no other minimum requirements.

Configuration of Bob
===============
The top of the class heirarchy is like so.

BobBuild -> BobConfig -> BobBuildImage

BobConfig is purely used for configuration options. You will need to
configure the parameters, such as :

#configSqueakVM - path to the vm
#configBaseDir - the base directory in which to work, default is set to
'/bob'.
#configPackageCacheDir - a directory that all built images may use for
their package-cache.

This allows you to define a local configuration and to save it simply by
filing out/in the class BobConfig.

Build Scripts
=========
There are two parts to the build script:

1) #theScript, and 2) metadata on the instance side as shown below.

A number of useful script methods have been defined for you. Each is
generated into a chunk of the script file that is handed to the image on
start up. This is an example #theScript.

BobBuildImage>>theScript

    script reset.
   
     self
        scriptTranscriptLogToFileStart;
        scriptAddLPF;
        scriptAddInfoScript;
        scriptTranscriptLogToFileStop;
        scriptSaveImageAndQuit: true.  
   
The line #scriptAddInfoScript appends any script chunks you defined as
an appendix to your metadata method: e.g.

""""""
"self halt."
    Installer sake install: 'Squeak-dev image'.  

Debugging
========
On the class side of each target, the methods #configStepNeeded
#configStepAction, and #configStepScript are available, if you set any
of these to true, it will halt at the appropriate place for you to step
through the process.

For debugging build scripts, the script is gnerated with a halt at the
beginning of each chunk.

Comments and Description
====================
The comment can contain arbitrary text.
The info metadata can contain any additional fields that you care to use.

However #comment and #description meta data fields are concatenated with
the comment and description of any dependent images.

enjoy

Keith