Caching mczs

I'm trying to figure out how to achieve the following:

Using metacello I want that all downloaded mczs are stored on disk and I want them to be used in subsequent invocations of the metacello configuration. Storing on disk I can achive using cacheRepository: on the version. But it does not seem to check the existence of an mcz in the cacheRepository. At least in my case the mczs are downloaded again if I call the metacello config again. Setting cacheRepository and repositoryOverrides produces weird errors. So I want to ask here first.

Norbert

On 02/07/2011 04:43 AM, Norbert Hartl wrote:
Norbert,

The problems are caused because you are trying to use the package-cache
as a primary repository. You should get satisfactory results if you use
a separate directory-based repository. There are a couple of reasons...

Firstly, there are times when Metacello disables the package-cache in
Gofer, so attempting to use the package-cache can cause problems for
Metacello.

Secondly, I don't know if you tried yet or not, but it isn't possible to
actually save an mcz file into the package-cache. Pharo, copies the
mcz file into the package-cache before running the storeVersion: code
and the storeVersion: code doesn't like it when a copy of the file
already exists in the target repository, so it's not real easy to use
the package-cache as a real repository....

Dale

Dale,
On 07.02.2011, at 19:17, Dale Henrichs wrote:
I'm not sure if I understand you. I'm doing this.

version := ((Smalltalk at: #ConfigurationOfMyProject) project latestVersion).
cacheRepository := MCDirectoryRepository new
        directory: (FileDirectory on: '/opt/gsbuilder/cache/myproject').
version cacheRepository: cacheRepository."
MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
   version load ]

This downloads all mczs into the cacheRepository. If I invoke it a second time it does it again. Trying to add

version repositoryOverrides: (Array with: cacheRepository).

didn't help but threw an error.

Later I discovered that using the cacheRepository at all prevents any code to be loaded into an image. It seems the cacheRepository is checked and if the package is there it is neither downloaded nor loaded into the image.

So, is there a way?

Norbert

On 02/07/2011 10:37 AM, Norbert Hartl wrote:

Norbert,

Okay, now I see what you are trying to do, sorry about that:)

The use case that I have tested is the following two step process:

   1. download mcz files to cache repository, using cacheRepository:
   2. load configuration from cache repository, using
      repositoryOverrides:

Using your example the download step looks like this:

   version := (Smalltalk at: #ConfigurationOfMyProject) project
                latestVersion.
   cacheRepository := MCDirectoryRepository new
         directory: (FileDirectory on: '/opt/gsbuilder/cache/myproject').
   version cacheRepository: cacheRepository.
   version fetch.

and the load step looks like this:

   version := (Smalltalk at: #ConfigurationOfMyProject) project
                latestVersion.
   overrideRepository := FileDirectory
                on: '/opt/gsbuilder/cache/myproject'.
   version overrideRepository: { overrideRepository }.
   version load.

I suppose that I _could_ make your version work (combined download and
load), I just hadn't considered it...I suppose at a minimum I should
throw an error if you attempt a load when a cacheRepository: is
specified ... or make it work.

Dale

Dale,

On 07.02.2011, at 21:01, Dale Henrichs wrote:
Yes, I took this from the documentation.
Ok, I need to think about how to make this. I see you have two use cases

- always online. Loading resources and checking everything in image.
- always offline. If I need to have a sealed version I do a "snapshot" of the version and then I'm be able to replay at any time

but I think a use case

- cache mczs locally and use these instead on further attempts to download. This speeds up local image generation quite a bit and it tries to be nice to squeaksource ;) But the moste important thing to me is that it can act as a fallback. If once again squeaksource or lukas' server is offline some could work on the last known version.

Do you think it is feasible to do? If I integrate my software with a baseline of another software I can live with the last latest version that should be consistent in the cache.

Norbert
 

On 02/07/2011 12:15 PM, Norbert Hartl wrote:
Norbert,

You are correct about this being a third use case .... the package-cache
in some sense is supposed to function in the capacity you suggest ... so
in theory you shouldn't have to hit squeaksource to download an mcz file
... Metacello actually does use mcz file versions from the package-cache
when they are available, but there are certain circumstances where the
target repository is going to be hit anyway:

   in cases where you are using a #development version of a
   configuration, Metacello always tries to download the latest
   configuration mcz file.

   in cases where you are loading a #baseline, the latest version of the
   mcz files is required and we have to hit the repository to determine
   what the latest version is and if that version is already in the
   package-cache we use it

The trick is that when the primary repository is inaccessible this all
breaks down ... I've been reluctant to allow metacello to automagically
ignore the fact that a primary repository repository is inaccessible,
but given that squeaksource is very fragile, perhaps I should rethink
what could be done in this case ...

The trick would be to allow metacello to be run in a mode where the
package-cache is used to satisfy all of the mcz file needs (if possible)
... I suppose I could signal a "is it okay to use package-cache now" (or
a version setting for "emergency repository") after attempting a couple
of retries and just fallback to the package-cache...basically an
automatic switch over to the package-cache to satisfy requests for files
a particular repository when the repository wasn't accessible.

This wouldn't help you if you were downloading with a cold package-cache
or the version of mcz file you are trying to load hadn't been previously
downloaded, but in the use case you are talking about, it would work ...

As far as being nicer to squeak source during normal operation, that
would take some more work to figure out the best ways to cache
information ... and when to flush it ...

Dale