fetch for Metacello...

I've just finishing the initial implementation of using fetch with Metacello.

There are a couple of places where fetch is involved:

  - during a "normal" load, all of the mcz files are pre-fetched from the
    repositories. By default, during the fetch phase of loading, all of the
    mcz files are copied into an MCDictionaryRepository (in memory). Then during
    load phase the mcz files are loaded from the MCDictionary repository. The
    dictionary repository is dropped on the floor after the load and the
    repositoryGroup for each working copy is updated to point to the original
    repository

  - instead of using #load and #load: with a version, you may use #fetch and #fetch:
    to download mcz files to a local directory repository from which you can do loads.
    Right now, if you load from a local directory repository, the repositoryGroup
    for the mcz file points to the local repository (not the original http
    repository).

To do the implementation, I basically beefed up the repositoryOverrides...I've created a MetacelloLoaderPolicy class that manages the repositoryOverrides as well as which repository is used as the cache repository. To track record the original repository from which the mcz was downloaded, the loaderPolicy has a repositoryMap (mapping mcz file name to mcz repository.

For the "normal" load, nothing has changed, except that internally all of the mcz files are fetched before going into the load phase.

I would like a little bit of feedback (and early testers) for using fetch explicitly. At this point the idea is very simple:

  - use fetch to download all of the mcz files from a Configuration (including
    dependencies) to a local directory repository
  - load from the directory repository

Here is the FETCH script:

  | localRepository project version policy |
  localRepository :=
      MCDirectoryRepository new
          directory: (FileDirectory on: '/home/dhenrich/monticello').
  Gofer new
      squeaksource: 'MetacelloRepository';
      package: 'ConfigurationOfMagritte';
      load.
  project := (Smalltalk at: #ConfigurationOfMagritte) project.
  version := project version: '1.2.1.3'.

  policy := version loaderPolicy.
  policy cacheRepository: localRepository.  "set cache repository"
  policy useImage: false.       "fetch mcz file even if image is up-to-date"

  "Do the fetch"
  project fetchProject: policy. "make sure ConfigurationOfMagritte is fetched"
  version fetch: 'ALL'.

Here is the LOAD script:

  | localRepository project version policy |
  localRepository :=
      MCDirectoryRepository new
          directory: (FileDirectory on: '/home/dhenrich/monticello').
  Gofer new
      repository: localRepository; "load config from local repository"
      package: 'ConfigurationOfMagritte';
      load.
  project := (Smalltalk at: #ConfigurationOfMagritte) project.
  version := project version: '1.2.1.3'.

  "use repositoryOverrides: to use local repository"
  version repositoryOverrides: (Array with: localRepository).
  "Do the load"
  version load: 'ALL'.

Moving forward, the loaderPolicy will be involved in retries for failure during downloads (haven't gone there yet) as well as handling the "secondary Pharo repository".

Probably the biggest issue that I have with the above is that when loading from the local repository repositoryGroups for the mcz files don't reflect the "original repository" for the mcz file ... Metacello knows how to save an mcz file to the "original repository". In earlier implementations of Metacello, there was an OB command that would add the "original repository" to the repository group for an mcz file, so perhaps that capability could be resurrected.

The fetch pass produces a repositoryMap with this information, but there is no clean way to pass the repository map to the load script at this point in time ...

The above scripting API is at a raw stage, so I'd like feedback on different approaches, names of selectors, etc.

This stuff all works, so if you want to take it for a spin just load Metacello 1.0-beta.25. The version is under development, so if you hit snags let me know and I'll fix them ... I expect to change the names of some of the methods, so don't write any scripts in stone:)

Dale

What Fetch is for?
Is it the tool Esteban is supposed to work on?

Cheers,
Alexandre


On 7 Mar 2010, at 16:05, Dale Henrichs wrote:

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

No, it isn't replacing Loader .. if anything it will give Loader more to work with.

Fetch is really aimed at dealing with the instability of the HTTP repositories.

The first order feature is that the load is split into a fetch phase and a load phase. Doing the fetch first, means that you can get all of your downloads done first (and if any downloads fail ... deal with those issues). Typically the downloads are much faster relative to the compilation, so if you are launching a build, once the fetch phase is finished you can go off and have a beer or a nap, while you wait for the compilation to complete:)

Once I'd implemented the two pass load, it was a simple step to break out the fetch phase so that one could do a fetch independent of a load....Doing a separate fetch means that you can create your own local repository of all of the mcz files that you use and then arrange to build images and the like from the local repository without having to hit the web for mcz files.

#repositoryOverrides: has been a feature for a while, but there was no offical way to create the "cache repository". with fetch you can do that.

I would expect that Loader could be used to manage these local cache repositories "behind the scenes" even keeping track of the repositoryMaps itself.

Finally, Fetch provides the infrastructure for access to the "Pharo secondary repository" that was talked about on the Pharo list.

Dale

----- "Alexandre Bergel" <[hidden email]> wrote

| What Fetch is for?
| Is it the tool Esteban is supposed to work on?
|
| Cheers,
| Alexandre
|
|
| On 7 Mar 2010, at 16:05, Dale Henrichs wrote:
|
| > I've just finishing the initial implementation of using fetch with
|
| > Metacello.
| >
| > There are a couple of places where fetch is involved:
| >
| >  - during a "normal" load, all of the mcz files are pre-fetched from
|  
| > the
| >    repositories. By default, during the fetch phase of loading, all
|
| > of the
| >    mcz files are copied into an MCDictionaryRepository (in memory).
|
| > Then during
| >    load phase the mcz files are loaded from the MCDictionary  
| > repository. The
| >    dictionary repository is dropped on the floor after the load and
|
| > the
| >    repositoryGroup for each working copy is updated to point to the
|
| > original
| >    repository
| >
| >  - instead of using #load and #load: with a version, you may use  
| > #fetch and #fetch:
| >    to download mcz files to a local directory repository from which
|
| > you can do loads.
| >    Right now, if you load from a local directory repository, the  
| > repositoryGroup
| >    for the mcz file points to the local repository (not the original
|  
| > http
| >    repository).
| >
| > To do the implementation, I basically beefed up the  
| > repositoryOverrides...I've created a MetacelloLoaderPolicy class  
| > that manages the repositoryOverrides as well as which repository is
|
| > used as the cache repository. To track record the original  
| > repository from which the mcz was downloaded, the loaderPolicy has a
|  
| > repositoryMap (mapping mcz file name to mcz repository.
| >
| > For the "normal" load, nothing has changed, except that internally
|
| > all of the mcz files are fetched before going into the load phase.
| >
| > I would like a little bit of feedback (and early testers) for using
|
| > fetch explicitly. At this point the idea is very simple:
| >
| >  - use fetch to download all of the mcz files from a Configuration
|
| > (including
| >    dependencies) to a local directory repository
| >  - load from the directory repository
| >
| > Here is the FETCH script:
| >
| >  | localRepository project version policy |
| >  localRepository :=
| >      MCDirectoryRepository new
| >          directory: (FileDirectory on:
| '/home/dhenrich/monticello').
| >  Gofer new
| >      squeaksource: 'MetacelloRepository';
| >      package: 'ConfigurationOfMagritte';
| >      load.
| >  project := (Smalltalk at: #ConfigurationOfMagritte) project.
| >  version := project version: '1.2.1.3'.
| >
| >  policy := version loaderPolicy.
| >  policy cacheRepository: localRepository.  "set cache repository"
| >  policy useImage: false.       "fetch mcz file even if image is up-
|
| > to-date"
| >
| >  "Do the fetch"
| >  project fetchProject: policy. "make sure ConfigurationOfMagritte is
|  
| > fetched"
| >  version fetch: 'ALL'.
| >
| > Here is the LOAD script:
| >
| >  | localRepository project version policy |
| >  localRepository :=
| >      MCDirectoryRepository new
| >          directory: (FileDirectory on:
| '/home/dhenrich/monticello').
| >  Gofer new
| >      repository: localRepository; "load config from local
| repository"
| >      package: 'ConfigurationOfMagritte';
| >      load.
| >  project := (Smalltalk at: #ConfigurationOfMagritte) project.
| >  version := project version: '1.2.1.3'.
| >
| >  "use repositoryOverrides: to use local repository"
| >  version repositoryOverrides: (Array with: localRepository).
| >  "Do the load"
| >  version load: 'ALL'.
| >
| > Moving forward, the loaderPolicy will be involved in retries for  
| > failure during downloads (haven't gone there yet) as well as  
| > handling the "secondary Pharo repository".
| >
| > Probably the biggest issue that I have with the above is that when
|
| > loading from the local repository repositoryGroups for the mcz files
|  
| > don't reflect the "original repository" for the mcz file ...  
| > Metacello knows how to save an mcz file to the "original  
| > repository". In earlier implementations of Metacello, there was an
|
| > OB command that would add the "original repository" to the  
| > repository group for an mcz file, so perhaps that capability could
|
| > be resurrected.
| >
| > The fetch pass produces a repositoryMap with this information, but
|
| > there is no clean way to pass the repository map to the load script
|
| > at this point in time ...
| >
| > The above scripting API is at a raw stage, so I'd like feedback on
|
| > different approaches, names of selectors, etc.
| >
| > This stuff all works, so if you want to take it for a spin just load
|  
| > Metacello 1.0-beta.25. The version is under development, so if you
|
| > hit snags let me know and I'll fix them ... I expect to change the
|
| > names of some of the methods, so don't write any scripts in stone:)
| >
| > Dale
|
| --
| _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
| Alexandre Bergel  http://www.bergel.eu
| ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.