[VW7.4] Working with multiple Store repositories

This is possibly a stupid question, but I'm currently thinking about
using three Store repositories at the same time, i.e.

    Repository 1: Bleeding Egde Development
    Repository 2: Internal Releases
    Repository 3: Published Products & Maintenance

All of them carry the same projects (with different version numbers, of
course). AFAIK it should be no problem to handle them as follows:

Daily work is published to (1) until something stable is achieved which
is worth internal use and testing. Then I connect and publish to (2),
which effectively gives all packages and bundles "older" version numbers
compared to (1). Once in a while, when a new product release is made
publicly available, I connect and publish to (3), which again leads to
different verions in that database.

When after several weeks or so a bug is reported by a customer, I
connect to (3) and fix it, publish it to (3) and deliver the fix. There
are now two possible cases regarding how to merge the fix into (2) and (1):

    a) The fix was already done during daily development => fine
    b) The bug happended to be new => publish the fix from (3) => (2)
and (1)

It seems to me this should be working fine. Probably two repositories
are enough. Did I miss something? Is this method really good?

Any suggestions are appreciated.

Andre

It sounds like a bit of a nightmare to me! :-) Trying to track the same version with different version numbers in three different repositories would probably be rather tricky and unreliable in practice.

Why not have one repository and three images? Store allows multiple version branches and merging, so you can easily merge fixes from the bleeding edge stuff back into the published product image or vice versa. We've been working this way for several years, and as far as I can tell it offers what you are seeking.

Where Store lets us down is in the user interface, which has naïve assumptions like "a more recently published version is good to load". That's great if you only ever work in one linear set of versions, but as soon as you are also adding fixes to branches of older versions you end up in a mess. We added several new commands to the Store menu, e.g. only showing new versions if they are children of the currently loaded version.

Steve

Steven Kelly wrote:
> Why not have one repository and three images? Store allows multiple version branches and merging, so you can easily merge fixes from the bleeding edge stuff back into the published product image or vice versa.

My automated build procedure is based on loading Store sources and
building parcels etc. from that.  The problem is that Store doesn't
support versioned prerequisites and always loads the most recent
prerequisite pundle (or asks for the version, which is impossible to
keep track of manually). Hence newer prerequisite pundles will load into
an old, incompatible release.

(Sorry, by no means will I enter dozens or even hundreds version numbers
in prerequisite properties by hand ;-)

Another issue is that "product" images lack the tools needed to maintain
bug fixing etc. Ok, that's easy to change. I could maintain a separate
"maintenance" development image for each released product. But I will
still need to keep track of the pundle versions that made up the "old
product" somehow. I could create some "ProductRelease" class, or better:
"MetaBundle",  which maintains the particular versions of all code
components of a product release. However, that's too much effort for me
at the moment.

It seems the only simple way to *exactly* restore an old product image
from Store sources (automatically, incl. prerequisites) is to have a
separate repository for it.

Andre

Andre Schnoor wrote:
> My automated build procedure is based on loading Store sources and
> building parcels etc. from that.  The problem is that Store doesn't
> support versioned prerequisites and always loads the most recent
> prerequisite pundle (or asks for the version, which is impossible to
> keep track of manually). Hence newer prerequisite pundles will load
> into an old, incompatible release.
[...]
> It seems the only simple way to *exactly* restore an old product
> image from Store sources (automatically, incl. prerequisites) is to
> have a separate repository for it.


At Soops we found that we do not need to separate development and
deployment repositories, we use configs/lineups[1,2] as top-level load
to build those different image configurations.

However we find it near-impossible to reliably rebuild old product
versions using a single repository, so we have opted to duplicate the
repository whenever we start to develop a new product version.

We synchronize VW upgrades to our repository duplication times.


[1]http://www.cincomsmalltalk.com/userblogs/travis/blogView?showComments=true&entry=3265388740
[2]http://swiki.cdegroot.com/lineups


HTH,

Reinout
-------

Reinout wrote:
> However we find it near-impossible to reliably rebuild old product
> versions using a single repository,

We simply make each product image keep track of the versions that it
contains. We can then load exactly those versions.

Actually what we do is archive a "preStrip" image for each product
version. This contains a fairly minimal development environment, from
which we strip using RTP. Like Andre says:

> Another issue is that "product" images lack the tools needed to
maintain
> bug fixing etc. Ok, that's easy to change. I could maintain a separate

> "maintenance" development image for each released product. But I will
> still need to keep track of the pundle versions that made up the "old
> product" somehow. I could create some "ProductRelease" class, or
better:
> "MetaBundle",  which maintains the particular versions of all code
> components of a product release. However, that's too much effort for
me
> at the moment.

We use an OrderedCollection. Not a particularly large amount of effort.
Here's the code to build what we use (I cut a bit of MetaCase-specific
stuff, hope it still works!). The format is something we have used for
over ten years, for historical reasons - I'm sure you can come up with
your own.

"SET VERSIONS DICT"
vColl := #{Store.Registry} value allPundlesSorted.
vColl := vColl select: [:pm |
        (#{Store.Registry} value enclosingComponentsFor: pm) isEmpty].
vColl := vColl select: [:pm | pundles contains: [:s | (s class == Array
ifTrue: [s first] ifFalse: [s]) asString = pm name]].
vColl := vColl collect: [:pundleModel | | stream name ver |
        stream := pundleModel databaseInfomation versionString
readStream.
        stream next.
        ver := stream upTo: $,.
        name := pundleModel name.
        name, '::',  ver].

> It seems the only simple way to *exactly* restore an old product image

> from Store sources (automatically, incl. prerequisites) is to have a
> separate repository for it.

Nope. There's NO WAY to *exactly* restore an old product image from
Store - or any other - sources. The .im file WILL be different. That's
why you have to save it. If you want exactly the same sources in your
.im, that's easy - just load the same packages and bundles.

Steve

Andre Schnoor wrote:
>The problem is that Store doesn't
>support versioned prerequisites and always loads the most recent
>prerequisite pundle (or asks for the version, which is impossible to
>keep track of manually). Hence newer prerequisite pundles will load into
>an old, incompatible release.

IMHO this is not true. Store does support versioned prerequisites, and when prerequisite versions are used, Store will not load newer versions. The Store tools could support this better, but this is another story.

>(Sorry, by no means will I enter dozens or even hundreds version numbers
>in prerequisite properties by hand ;-)

If you really have that many prerequisites, then it might be cheaper to build your own tool enhancements than either typing in all the version names or spending a lot of time trying to keep track of which changes you made in which repository (I agree with Steven Kelly that this sounds like a nightmare). At least that's what was done in the project I'm currently involved in. We have graphical views of pundle dependency trees (based on Hotdraw), menu entries like "set prerequisite versions from image" and other nice little tools. Things like these should not be too difficult to reproduce (unfortunately I'm not in a position to post these add-ons to the public repository, and I'm a bit too busy to clone them in my spare time).

>Another issue is that "product" images lack the tools needed to maintain
>bug fixing etc. Ok, that's easy to change. I could maintain a separate
>"maintenance" development image for each released product. But I will
>still need to keep track of the pundle versions that made up the "old
>product" somehow. I could create some "ProductRelease" class, or better:
>"MetaBundle",  which maintains the particular versions of all code
>components of a product release. However, that's too much effort for me
>at the moment.

Setting up configuration classes or configuration files and "top level bundles" is probably less effort than maintaining and synchronizing different repositories. We use one-click loadable top-level bundles with versioned prerequisites plus configuration files for automatic builds (using the subsystem initialization mechanism for starting a build process and batch files for passing parameters into the build). Building an image of a particular version from scratch (including runtime images and exe's for Windows) is just a doubleclick on a batch file for us.

>It seems the only simple way to *exactly* restore an old product image
>from Store sources (automatically, incl. prerequisites) is to have a
>separate repository for it.

In our environment, it definitely *is* possible to automatically reproduce images which contain exactly the same code and are initialized the same way. We use only one repository for at least a dozen different products sharing several dozen packages and bundles, with development of the next release and maintenance branches developed in parallel by ca. 40 developers. Of course, we have to keep the corresponding base images (virgin VisualWorks images with some preloaded tools) and batch / configuration files under version control, but they change only once per major release (new VisualWorks version, changes in tools, different runtime image name). As Steven Kelly has remarked, the resulting image files will always be slightly different based on random effects in the image (Timestamps etc.), but if your application does not rely on this kind of information (and it shouldn't), you can safely ignore this.

Best regards,
Joachim Geidel

Thank you Steven, Reinout and Joachim. It seems I have to review my
current source code management policy.

[hidden email] wrote:
> [...]
> If you really have that many prerequisites, then it might be cheaper to build your own tool enhancements than either typing in all the version names or spending a lot of time trying to keep track of which changes you made in which repository (I agree with Steven Kelly that this sounds like a nightmare). At least that's what was done in the project I'm currently involved in. We have graphical views of pundle dependency trees (based on Hotdraw), menu entries like "set prerequisite versions from image" and other nice little tools. Things like these should not be too difficult to reproduce (unfortunately I'm not in a position to post these add-ons to the public repository, and I'm a bit too busy to clone them in my spare time).
>  

Sounds good. I utilize a class "DeploymentConfiguration" which handles
"VersionSnapshots" taken from the current image. Probably I should
invest some more time to make these snapshots re-loadable and I'm done.

> [...] We use one-click loadable top-level bundles with versioned prerequisites plus configuration files for automatic builds (using the subsystem initialization mechanism for starting a build process and batch files for passing parameters into the build). Building an image of a particular version from scratch (including runtime images and exe's for Windows) is just a doubleclick on a batch file for us.
>  

I've done it in a very similar way. It already works perfectly for me,
except the loading of older releases.

I assume that "set prerequisites from image" you mentioned is simply
setting the prerequisite properties of the packages/bundles and marking
them dirty, right? That's fairly easy.

Andre

BTW: How big (old) can a single repository grow before it gets
fragmented or corrupted too much to be practically useful anymore? I
tend to publish even minor changes twice or three times daily and am
wondering if this will really last for several years ....

Andre

 >> But I will still need to keep track of the pundle versions that made
 >> up the "old product" somehow. I could create some "ProductRelease"
class,
>> or better: "MetaBundle",  which maintains the particular versions of all
 >> code components of a product release. However, that's too much effort
 >> for me at the moment.
>
> We use an OrderedCollection. Not a particularly large amount of effort.
> Here's the code to build what we use (I cut a bit of MetaCase-specific
> stuff, hope it still works!). The format is something we have used for
> over ten years, for historical reasons - I'm sure you can come up with
> your own.

This is something that we use, too. We finally found that the order is
not necessary to keep, we just save a 'map' of what pundles and versions
are loaded. With this map we step into the loading process and each
prerequisite is preferably loaded in the version given.

To build an image a file-in script is used which loads the map and the
top-level bundle or package of the application, which recursively
depends on everything needed.

With this system we are able to build an image in any point in time if
the appropriate map exists. We haven't used it much yet, it's currently
in testing stage.

Martin

From: [hidden email]
> I remember having problems with overrides included in two packages
> and I couldn't remove them. The only solution was to create a new
> repository and publish all code from a  master image from scratch.

You can remove these kinds of problems in the end, if you're willing to
work a bit. Here's Alan's advice:

----
The duplicate methods issue indicates an error in the database (I'm not
sure how it arises). It's not terribly harmful, but can be eliminated by
bringing up a versions browser, select the version in question, bring up
the list menu, then press control and hold it down while selecting
browse. This is an easy way to get an inspector on the Store object in
question. Then send it "self searchAndDestroyDuplicateMethods".
----

Of course we shouldn't have to work so hard: Store should provide a
proper UI for this (or just fix these override bugs - some presumably
have been fixed, since I see this problems less often these days, but
some definitely remain).

> Sounds great. Is there a regular method for purging old
packages/bundles
> from Store?

VW launcher | Store | Administration | Garbage Collection
...not that I've ever used it!

Steve