Metacello as a package management system for Pharo

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Metacello as a package management system for Pharo

Mariano Martinez Peck
Hi folks. First of all, sorry for the long email (I guess it will be long). After thinking and discussing hundred of times in the mailing lists, last week we all meet and talk about Metacello and Pharo. Dale, Stef, Marcus, Esteban, all were present. I want to write down all the ideas and solution in order to: not to forget them; and to get feedback and opinions from you.

Problems we want to face:

1) Right now a user CANNOT know which version of a project is the stable for a specific Pharo version. For example, suppose ConfigurationOfXXX. The user cannot know that XXX  3.45 is the stable for 1.0  and that XXX 5.7.3 is the one for 1.1....etc...so users end up confused not knowing which version to download. In addition, #latestVersion is not enough, because that will answer you the latest version, but that may not be the correct one for older Pharo images. Suppose in the previous case, latestVersion may answer 5.7.3 which may not work in Pharo 1.0..... We want the user to be able to autaticallly load the stable version for each pharo version without needing to know that.

2) Have a repository for each Pharo version so that someone can easily browse the available configurations for that Pharo version, and load them. Not only one as it is now woth MetacelloRepository.

3) Be self contained. Sometimes developers removed packages of versions from their repositories. In such case, Metacello cannot do anything and the load may not work anymore. We want to be able to reproduce the load and be able to load the same 10 years after.

Solution proposed:

The proposed solution may not even involve Metacello. You may have heard about Metacello Project Loader developer by Esteban Lorenzano. See http://www.smallworks.com.ar/en/community/GoferProjectLoader
The idea is rename that project to that it is a general name like GoferProjectManager or something like that, because now it will do load but also write :)
I will call it GoferProjectManager in this email.

The idea is that every developer of a configuration define the stable version for each Dialect version. There are two possible alternatives:

a) That each Conf class implements #stableVersion: spec, which could be something like:

ConfigurationOfXXX >> stableVersion: spec
      spec for: #Pharo1.0 do: [ ^ self version: '3.45' ].
      spec for: #Pharo1.1 do: [ ^ self version: '5.7.3' ].
      spec for: #Pharo1.2 do: [  self error: 'There is no stable version for Pharo 1.2' ].
      spec for: #Gemstone do: [ ^ self version: '4.2' ].
      .....

or something like that...the idea is that you define stable versions for each platform.

The idea is that now the user could do something like this:     ConfigurationOfXXX project stableVersion load
but he won't do that ;)

b) another approach is to specify something in each version, for example:

ConfigurationOfXXX >> version345: spec
    <version: '3.45' imports: #('1.2-baseline')>
   
    spec for: #common do: [
        spec blessing: #release.
        spec description: 'Blah balh...'.
        spec platformVersion: 'Pharo 1.0'


and

ConfigurationOfXXX >> version573: spec
    <version: '5.7.3' imports: #('1.2-baseline')>
   
    spec for: #common do: [
        spec blessing: #release.
        spec description: 'Blah balh...'.
        spec platformVersion: 'Pharo 1.1'

Now...any of those alternatives would solve problem 1)

Now to solve 3) the idea is that GoferProjectManager could provide something like this:

(Gofer project 'XXX') publishVersion: '3.45' for: 'Pharo 1.0'
or then
(Gofer project 'XXX') publishVersion: '5.7.3' for: 'Pharo 1.1'
etc...

Then this would do different things:

a) Automatically take the conf class (it is important the name conventions!!)
b) Commit it into the correct Pharo repository version. For example for 3.4.5 it will commit it in PharoMetacelloRepository10 and 5.7.3 to PharoMetacelloRepository11
c) Traverse all the dependencies of all needed packages and commit all of them to PharoPackagesContainer10  or PharoPackagesContainer11....etc
Dale sent few weeks ago a script that does that: takes a conf, a version and a repo and commits all the files to that repo.
  
With all this we solve 2). The idea is that for EACH pharo version we will have 2 repositories, one for the confs and another one for the packages (as a backup).

The last point is that when loading we should load everything from the PharoPackagesContainer10 or PharoPackagesContainer11...but we don't want to change all our confs (for example, they are pointing to they own repos or MetacelloRepository)....so the solution is that we use GoferProjectManager to load. Example:

Gofer project: 'XXX' loadStable

and loadStable would be something like:

self stableVersion repositoryOverrides: (self containerRepo); load

The idea is that it can use #repositoriesOverride to override with the correct repository...

So.....that's all. What do you think?  does this make sense?

Cheers

Mariano
Reply | Threaded
Open this post in threaded view
|

Re: Metacello as a package management system for Pharo

Miguel Cobá
El lun, 20-09-2010 a las 22:31 +0200, Mariano Martinez Peck escribió:

> Hi folks. First of all, sorry for the long email (I guess it will be
> long). After thinking and discussing hundred of times in the mailing
> lists, last week we all meet and talk about Metacello and Pharo. Dale,
> Stef, Marcus, Esteban, all were present. I want to write down all the
> ideas and solution in order to: not to forget them; and to get
> feedback and opinions from you.
>
> Problems we want to face:
>
> 1) Right now a user CANNOT know which version of a project is the
> stable for a specific Pharo version. For example, suppose
> ConfigurationOfXXX. The user cannot know that XXX  3.45 is the stable
> for 1.0  and that XXX 5.7.3 is the one for 1.1....etc...so users end
> up confused not knowing which version to download. In addition,
> #latestVersion is not enough, because that will answer you the latest
> version, but that may not be the correct one for older Pharo images.
> Suppose in the previous case, latestVersion may answer 5.7.3 which may
> not work in Pharo 1.0..... We want the user to be able to autaticallly
> load the stable version for each pharo version without needing to know
> that.
>
> 2) Have a repository for each Pharo version so that someone can easily
> browse the available configurations for that Pharo version, and load
> them. Not only one as it is now woth MetacelloRepository.
>
> 3) Be self contained. Sometimes developers removed packages of
> versions from their repositories. In such case, Metacello cannot do
> anything and the load may not work anymore. We want to be able to
> reproduce the load and be able to load the same 10 years after.

ok

>
> Solution proposed:
>
> The proposed solution may not even involve Metacello. You may have
> heard about Metacello Project Loader developer by Esteban Lorenzano.
> See http://www.smallworks.com.ar/en/community/GoferProjectLoader
> The idea is rename that project to that it is a general name like
> GoferProjectManager or something like that, because now it will do
> load but also write :)
> I will call it GoferProjectManager in this email.
>
> The idea is that every developer of a configuration define the stable
> version for each Dialect version. There are two possible alternatives:
>
> a) That each Conf class implements #stableVersion: spec, which could
> be something like:
>
> ConfigurationOfXXX >> stableVersion: spec
>       spec for: #Pharo1.0 do: [ ^ self version: '3.45' ].
>       spec for: #Pharo1.1 do: [ ^ self version: '5.7.3' ].
>       spec for: #Pharo1.2 do: [  self error: 'There is no stable
> version for Pharo 1.2' ].
>       spec for: #Gemstone do: [ ^ self version: '4.2' ].
>       .....

This implies that the stableVersion method will be heavily modified as
the software is ported to new platforms and new releases. Not a big
deal. Sometimes will imply that a merge will be needed when 2 developers
want  to maintain the configuration. Also, I feel that disconnects the
version method from the version string declaration/publication. again,
not a big issue either.

>
> or something like that...the idea is that you define stable versions
> for each platform.
>
> The idea is that now the user could do something like this:
> ConfigurationOfXXX project stableVersion load
> but he won't do that ;)
>
> b) another approach is to specify something in each version, for
> example:
>
> ConfigurationOfXXX >> version345: spec
>     <version: '3.45' imports: #('1.2-baseline')>
>    
>     spec for: #common do: [
>         spec blessing: #release.
>         spec description: 'Blah balh...'.
>         spec platformVersion: 'Pharo 1.0'
>
> and
>
> ConfigurationOfXXX >> version573: spec
>     <version: '5.7.3' imports: #('1.2-baseline')>
>    
>     spec for: #common do: [
>         spec blessing: #release.
>         spec description: 'Blah balh...'.
>         spec platformVersion: 'Pharo 1.1'
>
> Now...any of those alternatives would solve problem 1)


As you say they are equivalent ways to state the same information. I
like this because it is only a place to modify the new version number.
I personally like this, but also I can imagine that this could lead to
big methods with conditionals for functionality that applies to more
than one release.

Maybe the first option requires less work for Dale to support this
functionality.

>
> Now to solve 3) the idea is that GoferProjectManager could provide
> something like this:
>
> (Gofer project 'XXX') publishVersion: '3.45' for: 'Pharo 1.0'
> or then
> (Gofer project 'XXX') publishVersion: '5.7.3' for: 'Pharo 1.1'
> etc...
>
> Then this would do different things:
>
> a) Automatically take the conf class (it is important the name
> conventions!!)
> b) Commit it into the correct Pharo repository version. For example
> for 3.4.5 it will commit it in PharoMetacelloRepository10 and 5.7.3 to
> PharoMetacelloRepository11
> c) Traverse all the dependencies of all needed packages and commit all
> of them to PharoPackagesContainer10  or
> PharoPackagesContainer11....etc
> Dale sent few weeks ago a script that does that: takes a conf, a
> version and a repo and commits all the files to that repo.
>    
> With all this we solve 2). The idea is that for EACH pharo version we
> will have 2 repositories, one for the confs and another one for the
> packages (as a backup).

I like it. Now, when traversing dependencies, wouldn't be good to also
rewrite the repositories they point to so that they point to self
(repository). This way the problem 3 is solved too, because the
configuration already points to self and not the original repo?

This maybe could be easier if Metacello could have a way to indirect
repository names from the configuration so that the metacello version
projects only referred a keyworkd and not the real package and then on
commiting the package to the repo, it would be rewritten or overriden
with the container repository?

For example, instead of:

spec for: #squeakCommon do: [
  spec
    project: 'OSProcess' with: [
      spec
        className: 'ConfigurationOfOSProcess';
        loads: #('default');
        file: 'ConfigurationOfOSProcess';
        repository: 'http://www.squeaksource.com/MetacelloRepository' ].
  spec
    repository: 'http://www.squeaksource.com/Magma';
    package: 'WriteBarrier' with: [
      spec repository: 'http://www.squeaksource.com/WriteBarrier' ];

you'll write a baseline like:

spec for: #squeakCommon do: [
  spec
    project: 'OSProcess' with: [
      spec
        className: 'ConfigurationOfOSProcess';
        loads: #('default');
        file: 'ConfigurationOfOSProcess';
        repository: #osProcessRepository ].
  spec
    repository: 'http://www.squeaksource.com/Magma';
    package: 'WriteBarrier' with: [
      spec repository: #writeBarrierRepository ];

and will define the repositories somewhere else:

ConfigurationOfXXX>>repositoryFor: aSymbol
  ^ repoMap at: #aSymbol       "This will return the real repo"


And when commiting, maybe the GoferProjectManager will create a subclass
with a method overriding the repository map to

ConfigurationOfXXX>>repositoryFor: aSymbol
 ^ 'http://www.squeaksource.com/PharoPackagesContainerXX'


I haven't used repositoryOverrides so maybe this is nonsense and not
necessary and can be handled cleanly and better by Metacello itself. :)


>
> The last point is that when loading we should load everything from the
> PharoPackagesContainer10 or PharoPackagesContainer11...but we don't
> want to change all our confs (for example, they are pointing to they
> own repos or MetacelloRepository)....so the solution is that we use
> GoferProjectManager to load. Example:
>
> Gofer project: 'XXX' loadStable
>
> and loadStable would be something like:
>
> self stableVersion repositoryOverrides: (self containerRepo); load
>
> The idea is that it can use #repositoriesOverride to override with the
> correct repository...
>
> So.....that's all. What do you think?  does this make sense?
>
> Cheers
>
> Mariano


Overall I agree. The less the user has to type and know from where the
packages come the better for we all.

--
Miguel Cobá
http://miguel.leugim.com.mx

Reply | Threaded
Open this post in threaded view
|

Re: Metacello as a package management system for Pharo

Mariano Martinez Peck


On Mon, Sep 20, 2010 at 11:25 PM, Miguel Cobá <[hidden email]> wrote:
El lun, 20-09-2010 a las 22:31 +0200, Mariano Martinez Peck escribió:
> Hi folks. First of all, sorry for the long email (I guess it will be
> long). After thinking and discussing hundred of times in the mailing
> lists, last week we all meet and talk about Metacello and Pharo. Dale,
> Stef, Marcus, Esteban, all were present. I want to write down all the
> ideas and solution in order to: not to forget them; and to get
> feedback and opinions from you.
>
> Problems we want to face:
>
> 1) Right now a user CANNOT know which version of a project is the
> stable for a specific Pharo version. For example, suppose
> ConfigurationOfXXX. The user cannot know that XXX  3.45 is the stable
> for 1.0  and that XXX 5.7.3 is the one for 1.1....etc...so users end
> up confused not knowing which version to download. In addition,
> #latestVersion is not enough, because that will answer you the latest
> version, but that may not be the correct one for older Pharo images.
> Suppose in the previous case, latestVersion may answer 5.7.3 which may
> not work in Pharo 1.0..... We want the user to be able to autaticallly
> load the stable version for each pharo version without needing to know
> that.
>
> 2) Have a repository for each Pharo version so that someone can easily
> browse the available configurations for that Pharo version, and load
> them. Not only one as it is now woth MetacelloRepository.
>
> 3) Be self contained. Sometimes developers removed packages of
> versions from their repositories. In such case, Metacello cannot do
> anything and the load may not work anymore. We want to be able to
> reproduce the load and be able to load the same 10 years after.

ok

>
> Solution proposed:
>
> The proposed solution may not even involve Metacello. You may have
> heard about Metacello Project Loader developer by Esteban Lorenzano.
> See http://www.smallworks.com.ar/en/community/GoferProjectLoader
> The idea is rename that project to that it is a general name like
> GoferProjectManager or something like that, because now it will do
> load but also write :)
> I will call it GoferProjectManager in this email.
>
> The idea is that every developer of a configuration define the stable
> version for each Dialect version. There are two possible alternatives:
>
> a) That each Conf class implements #stableVersion: spec, which could
> be something like:
>
> ConfigurationOfXXX >> stableVersion: spec
>       spec for: #Pharo1.0 do: [ ^ self version: '3.45' ].
>       spec for: #Pharo1.1 do: [ ^ self version: '5.7.3' ].
>       spec for: #Pharo1.2 do: [  self error: 'There is no stable
> version for Pharo 1.2' ].
>       spec for: #Gemstone do: [ ^ self version: '4.2' ].
>       .....

This implies that the stableVersion method will be heavily modified as
the software is ported to new platforms and new releases. Not a big
deal. Sometimes will imply that a merge will be needed when 2 developers
want  to maintain the configuration.

Yes...what how many times would you have a conflic?  Anyway, as you said, a simple merge solves the problem.
 
Also, I feel that disconnects the
version method from the version string declaration/publication. again,
not a big issue either.

True. That's why we have also option b)
 

>
> or something like that...the idea is that you define stable versions
> for each platform.
>
> The idea is that now the user could do something like this:
> ConfigurationOfXXX project stableVersion load
> but he won't do that ;)
>
> b) another approach is to specify something in each version, for
> example:
>
> ConfigurationOfXXX >> version345: spec
>     <version: '3.45' imports: #('1.2-baseline')>
>
>     spec for: #common do: [
>         spec blessing: #release.
>         spec description: 'Blah balh...'.
>         spec platformVersion: 'Pharo 1.0'
>
> and
>
> ConfigurationOfXXX >> version573: spec
>     <version: '5.7.3' imports: #('1.2-baseline')>
>
>     spec for: #common do: [
>         spec blessing: #release.
>         spec description: 'Blah balh...'.
>         spec platformVersion: 'Pharo 1.1'
>
> Now...any of those alternatives would solve problem 1)


As you say they are equivalent ways to state the same information. I
like this because it is only a place to modify the new version number.
I personally like this, but also I can imagine that this could lead to
big methods with conditionals for functionality that applies to more
than one release.


Exactly. And this way it is a bit more complex to calculate. Suppose the user does ConfigurationOfXXX project stableVersion.
There you have to check in which Pharo you are, then, check ALL selectors that are version methods, and get the selector that as in the code the pharo version you are in. But how to do that?  maybe you have "Pharo 1.0" as a comment and maybe it is found? another is to run the execution over each method and then ask to the spec :)
Fnally, you will have several versions for the same Pharo version. Suppose first I set 3.45 as stable for Pharo 1.0 but then I do it for 3.46 so....we have to take the last one.

 
Maybe the first option requires less work for Dale to support this
functionality.


Yes, I think it is easier altought you may want to do the second one.
 
>
> Now to solve 3) the idea is that GoferProjectManager could provide
> something like this:
>
> (Gofer project 'XXX') publishVersion: '3.45' for: 'Pharo 1.0'
> or then
> (Gofer project 'XXX') publishVersion: '5.7.3' for: 'Pharo 1.1'
> etc...
>
> Then this would do different things:
>
> a) Automatically take the conf class (it is important the name
> conventions!!)
> b) Commit it into the correct Pharo repository version. For example
> for 3.4.5 it will commit it in PharoMetacelloRepository10 and 5.7.3 to
> PharoMetacelloRepository11
> c) Traverse all the dependencies of all needed packages and commit all
> of them to PharoPackagesContainer10  or
> PharoPackagesContainer11....etc
> Dale sent few weeks ago a script that does that: takes a conf, a
> version and a repo and commits all the files to that repo.
>
> With all this we solve 2). The idea is that for EACH pharo version we
> will have 2 repositories, one for the confs and another one for the
> packages (as a backup).

I like it. Now, when traversing dependencies, wouldn't be good to also
rewrite the repositories they point to so that they point to self
(repository). This way the problem 3 is solved too, because the
configuration already points to self and not the original repo?


That's a good idea :)
You mean...in order not to traverse it twice?  with this we could save the call to #repositoriesOverrides.
I guess it would be possible.

 
This maybe could be easier if Metacello could have a way to indirect
repository names from the configuration so that the metacello version
projects only referred a keyworkd and not the real package and then on
commiting the package to the repo, it would be rewritten or overriden
with the container repository?

For example, instead of:

spec for: #squeakCommon do: [
 spec
   project: 'OSProcess' with: [
     spec
       className: 'ConfigurationOfOSProcess';
       loads: #('default');
       file: 'ConfigurationOfOSProcess';
       repository: 'http://www.squeaksource.com/MetacelloRepository' ].
 spec
   repository: 'http://www.squeaksource.com/Magma';
   package: 'WriteBarrier' with: [
     spec repository: 'http://www.squeaksource.com/WriteBarrier' ];

you'll write a baseline like:

spec for: #squeakCommon do: [
 spec
   project: 'OSProcess' with: [
     spec
       className: 'ConfigurationOfOSProcess';
       loads: #('default');
       file: 'ConfigurationOfOSProcess';
       repository: #osProcessRepository ].
 spec
   repository: 'http://www.squeaksource.com/Magma';
   package: 'WriteBarrier' with: [
     spec repository: #writeBarrierRepository ];

and will define the repositories somewhere else:

ConfigurationOfXXX>>repositoryFor: aSymbol
 ^ repoMap at: #aSymbol       "This will return the real repo"


And when commiting, maybe the GoferProjectManager will create a subclass
with a method overriding the repository map to

ConfigurationOfXXX>>repositoryFor: aSymbol
 ^ 'http://www.squeaksource.com/PharoPackagesContainerXX'


Ok....I got the idea.
 


I haven't used repositoryOverrides so maybe this is nonsense and not
necessary and can be handled cleanly and better by Metacello itself. :)



And yes, I think itis handle by Metacello. But, as far as I understand, you can send the "#repositoryOverrides: newRepository" to a version. That will traverse all dependencies and then for each of them it traverse it does a "#repository: newRepository"  . Once this is done, you can said "load" to the version.

It seems they are 2 different approaches.

 
>
> The last point is that when loading we should load everything from the
> PharoPackagesContainer10 or PharoPackagesContainer11...but we don't
> want to change all our confs (for example, they are pointing to they
> own repos or MetacelloRepository)....so the solution is that we use
> GoferProjectManager to load. Example:
>
> Gofer project: 'XXX' loadStable
>
> and loadStable would be something like:
>
> self stableVersion repositoryOverrides: (self containerRepo); load
>
> The idea is that it can use #repositoriesOverride to override with the
> correct repository...
>
> So.....that's all. What do you think?  does this make sense?
>
> Cheers
>
> Mariano


Overall I agree. The less the user has to type and know from where the
packages come the better for we all.


:)
 
--
Miguel Cobá
http://miguel.leugim.com.mx