Metacello and FileTree/Cypress

I know that we are all familiar with the ConfigurationOf package that we use for managing project versions in Metacello.

If you've read through the Metacello Scripting API Glossary[1], then you've encountered the VersionOf package[2].

The basic idea is that the ConfigurationOf:

  Defines a set of versions for a project. There are baseline versions that define
  the structure of the project at various points in time and the versions reference
  a baseline version.

The VersionOf:

  Defines a single baseline version of a project. When using git, or another
  directory-based SCM, it is no longer necessary for Metacello to manage multiple
  versions of the project. In fact, it is only necessary for Metacello to specify
  the structure of the project.

If you look at a standard baseline definition:

  spec
      for: #'common'
      do: [
          spec blessing: #baseline.
          spec
              project: 'External'
              with: [
                  spec
                      version: '1.0-baseline';
                      repository: 'http://example.com/external' ].
          spec
              package: 'Sample-Core' with: [ spec requires: 'External'];
              package: 'Sample-Tests' with: [ spec requires: 'Sample-Core' ].
          spec
              group: 'default' with: #('Sample-Core') ].

I think that you can imagine that for a single platform one could reduce the baseline specification down to a bare minimum:

  spec
    project: 'External'
      with: [
        spec
          version: '1.0-baseline';
          repository: 'http://example.com/external' ].
  spec
    package: 'Sample-Core' with: [ spec requires: 'External'];
    package: 'Sample-Tests' with: [ spec requires: 'Sample-Core' ].
  spec
    group: 'default' with: #('Sample-Core')

however for many of the current mcz-file projects we end up with slightly different structures because of platform differences and the baseline spec looks more like the following:

        spec for: #'common' do: [
                spec
                        package: 'Grease-Core';
                        package: 'Grease-Tests-Core' with: [
                                spec requires: #('Grease-Core' ). ]. ].

        spec for: #'squeakCommon' do: [
                spec
                        package: 'Grease-Pharo-Core' with: [
                                spec requires: #('Grease-Core' ). ];
                        package: 'Grease-Tests-Pharo-Core' with: [
                                spec requires: #('Grease-Tests-Core' ). ].].

        spec for: #'pharo' do: [
                spec project: 'Refactoring-Core' with: [
                                spec
                                        className: 'ConfigurationOfRefactoringBrowser';
                                        versionString: #'stable';
                                        loads: #('Core' );
                                        repository: 'http://www.squeaksource.com/MetacelloRepository' ].
                spec
                        package: 'Grease-Slime' with: [
                                spec requires: #('Refactoring-Core' 'Grease-Core' ). ];
                        package: 'Grease-Tests-Slime' with: [
                                spec requires: #('Grease-Slime' ). ]. ].



with platform-specific structural differences.

These structural differences are necessary because the current Metacello model wants to use the same baseline version for each platform.

In working with the FileTree project over the last week, I created a branch per platform:

  pharo1.3
  squeak4.3
  gemstone2.4

With this approach we can return to needing a very simple structural specification.

For pharo1.3 the above would collapse to:

  spec
    package: 'Grease-Core';
    package: 'Grease-Tests-Core' with: [
      spec requires: #('Grease-Core' ). ].
  spec
    package: 'Grease-Pharo-Core' with: [
      spec requires: #('Grease-Core' ). ];
    package: 'Grease-Tests-Pharo-Core' with: [
      spec requires: #('Grease-Tests-Core' ). ].

  spec project: 'Refactoring-Core' with: [
    spec
      className: 'ConfigurationOfRefactoringBrowser';
      versionString: #'stable';
      loads: #('Core' );
      repository: 'http://www.squeaksource.com/MetacelloRepository' ].
  spec
    package: 'Grease-Slime' with: [
      spec requires: #('Refactoring-Core' 'Grease-Core' ). ];
    package: 'Grease-Tests-Slime' with: [
      spec requires: #('Grease-Slime' ). ].

and for squeak4.3 it would collapse to:

  spec
    package: 'Grease-Core';
    package: 'Grease-Tests-Core' with: [
      spec requires: #('Grease-Core' ). ].
  spec
    package: 'Grease-Pharo-Core' with: [
      spec requires: #('Grease-Core' ). ];
    package: 'Grease-Tests-Pharo-Core' with: [
      spec requires: #('Grease-Tests-Core' ). ].

Furthermore, when you are working with a single platform, the platform-specific packages (like Grease-Pharo-Core and Grease-Tests-Pharo-Core) don't actually need to be in separate packages, so the specifications become even simpler:

For pharo1.3 the above would collapse to:

  spec
    package: 'Grease-Core';
    package: 'Grease-Tests-Core' with: [
      spec requires: #('Grease-Core' ). ].
  spec project: 'Refactoring-Core' with: [
    spec
      className: 'ConfigurationOfRefactoringBrowser';
      versionString: #'stable';
      loads: #('Core' );
      repository: 'http://www.squeaksource.com/MetacelloRepository' ].
  spec
    package: 'Grease-Slime' with: [
      spec requires: #('Refactoring-Core' 'Grease-Core' ). ];
    package: 'Grease-Tests-Slime' with: [
      spec requires: #('Grease-Slime' ). ].

and for squeak4.3 it would collapse to:

  spec
    package: 'Grease-Core';
    package: 'Grease-Tests-Core' with: [
      spec requires: #('Grease-Core' ). ].

If we looked close enough at the Grease-Slime package, we might find that we could include the Grease-Slime package in the Grease-Core package further simplifying the structure to a simple common structural spec shared by all platforms:

  spec
    package: 'Grease-Core';
    package: 'Grease-Tests-Core' with: [
      spec requires: #('Grease-Core' ). ].

Of course if we wanted to make the loading of Grease-Slime optional (as we do with tests), then we would keep the Grease-Slime packages.

My point is that we can SIGNIFICANTLY reduce the complexity of the structural (baseline) specifications to the point where it no longer takes a degree in Metacello to specify the structure. The GST folks use package.xml[3] to specify the structural dependencies for their projects and we could do something very similar with a simple JSON file that is dead simple to modify and needs no complex Metacello tools to manage ...

Of course the cost of this simplicity is to adopt the branch per platform model and to "require" that projects undergo a "branch refactoring".

I don't like the idea of requiring a "branch refactoring", but then again I don't relish the idea of inventing a VersionOf structure that should never be really used in the first place ...

In reality, I think that the branch per platform model will improve the quality of projects and make it possible for folks to judge the status of the different platforms by looking at the network diagram[4] for the project.

I've attached a screen shot of the FileTree branch network and you can clearly see that the pharo1.3 and master branches are "ahead of" the gemstone2.4 and sqeuak4.3 branches, so as a user I would know that some merging was required for those two projects before I got the latest code ... hovering over the little dots in the diagram gives you commit comment and clicking on the dot takes you to the commit, so you can easily tell if you are in fact missing something important ...

Anyway, I'm interested in your thoughts and opinions about this particular issue ... I'll invent the VersionOf implementation if absolutely necessary or some other approach doesn't come up during the discussions..

Dale

[1] https://github.com/dalehenrich/MetacelloScriptingApiSpec/wiki/Glossaryofterms
[2] https://github.com/dalehenrich/MetacelloScriptingApiSpec/wiki/Glossaryofterms#wiki-versionof
[3] https://github.com/gnu-smalltalk/smalltalk/blob/master/packages/gtk/package.xml
[4] https://github.com/dalehenrich/filetree/network

Hi dale


> My point is that we can SIGNIFICANTLY reduce the complexity of the structural (baseline) specifications to the point where it no longer takes a degree in Metacello to specify the structure. The GST folks use package.xml[3] to specify the structural dependencies for their projects and we could do something very similar with a simple JSON file that is dead simple to modify and needs no complex Metacello tools to manage ...
>
> Of course the cost of this simplicity is to adopt the branch per platform model and to "require" that projects undergo a "branch refactoring".

I have difficulties to assess the impact of declaring dependencies at the package level. What happen if a local information gets wrong?
I do not know what branch refactoring means.

> I don't like the idea of requiring a "branch refactoring", but then again I don't relish the idea of inventing a VersionOf structure that should never be really used in the first place …

I do not know since I do not really understand what you mean but if it is having one specification for each platform I think that this is good.
>
> In reality, I think that the branch per platform model will improve the quality of projects and make it possible for folks to judge the status of the different platforms by looking at the network diagram[4] for the project.


> I've attached a screen shot of the FileTree branch network and you can clearly see that the pharo1.3 and master branches are "ahead of" the gemstone2.4 and sqeuak4.3 branches, so as a user I would know that some merging was required for those two projects before I got the latest code ... hovering over the little dots in the diagram gives you commit comment and clicking on the dot takes you to the commit, so you can easily tell if you are in fact missing something important …

I do not see how this is related to platform branch (even if I like the idea to separate them - this will help to reduce the complexity of certain versions), I imagine that
we could extract the same information without platform branch. Just more painful so to me this is the not a selling point.


I try to understand what you are saying and I see two point
        - dependency declared at the package level?
        - separation of platforms
       
Stef

----- Original Message -----
| From: "stephane ducasse" <[hidden email]>
| To: [hidden email]
| Sent: Tuesday, April 10, 2012 12:17:44 PM
| Subject: Re: [Metacello] Metacello and FileTree/Cypress
|
| Hi dale
|
|
| > My point is that we can SIGNIFICANTLY reduce the complexity of the
| > structural (baseline) specifications to the point where it no
| > longer takes a degree in Metacello to specify the structure. The
| > GST folks use package.xml[3] to specify the structural
| > dependencies for their projects and we could do something very
| > similar with a simple JSON file that is dead simple to modify and
| > needs no complex Metacello tools to manage ...
| >
| > Of course the cost of this simplicity is to adopt the branch per
| > platform model and to "require" that projects undergo a "branch
| > refactoring".
|
| I have difficulties to assess the impact of declaring dependencies at
| the package level.


Here's a better example (hopefully:), given this baseline spec from a ConfigurationOfGrease, where each platform has a 'Grease-XXX-Core' package:

  spec for: #common do: [
    spec
      package: 'Grease-Core';
      package: 'Grease-Tests-Core' with: [
        spec requires: #('Grease-Core' ). ]. ].

  spec for: #pharo do: [
    spec
      package: 'Grease-Pharo-Core' with: [
        spec requires: #('Grease-Core' ). ]. ].

  spec for: #squeak do: [
    spec
      package: 'Grease-Squeak-Core' with: [
        spec requires: #('Grease-Core' ). ]. ].

  spec for: #gemstone do: [
    spec
      package: 'Grease-GemStone-Core' with: [
        spec requires: #('Grease-Core' ). ]. ].

We have each of the plaform-specific packages fenced off in it's own #for:do: block because we are sharing the specification across all platforms.

If we were to write a specification that was for pharo only (i.e., the version of the specification is on the #pharo branch of the project) then the overall specification would be greatly simplified:

    spec
      package: 'Grease-Core';
      package: 'Grease-Tests-Core' with: [
        spec requires: #('Grease-Core' ). ];
      package: 'Grease-Pharo-Core' with: [
        spec requires: #('Grease-Core' ). ].

on the squeak branch the spec would be:

    spec
      package: 'Grease-Core';
      package: 'Grease-Tests-Core' with: [
        spec requires: #('Grease-Core' ). ];
      package: 'Grease-Squeak-Core' with: [
        spec requires: #('Grease-Core' ). ].

and so on.

With per platform branching, it is not necessary to include information about any other platform in the specification on the platform-specific branch.

| What happen if a local information gets wrong?

Not quite sure what you are getting at here. Currently we have this type of problem, only right now it is possible to make an editting mistake that impacts other platforms ... by isolating the changes on platform branches, the only platform that is affected is the one that is being worked on ..

Furthermore, using gitHub, developers will be able to insulate themselves from mistakes made in the public repository by forking the projects ... when you fork a project you control the rate at which you see updates and can apply your own testing and validation process before ACCEPTING any change into your fork of the project...
 
| I do not know what branch refactoring means.

What I mean by "branch refactoring" is that given the following specification on the pharo branch:

    spec
      package: 'Grease-Core';
      package: 'Grease-Tests-Core' with: [
        spec requires: #('Grease-Core' ). ];
      package: 'Grease-Pharo-Core' with: [
        spec requires: #('Grease-Core' ). ].

There is no practical reason to keep the Grease-Pharo-Core separate from the Grease-Core package. Today we do so because we need the Grease-Core package to be shared amongst the various platforms.

With git/github, one can manage the common changes without having to put them in a separate package. See the 'Guildelines for Contributing'[1] for the FileTree project to get an explanation of the process I am referring to.

Doing a "branch refactoring" involves merging the Grease-Pharo-Core package into the Grease-Core package on the pharo branch. If one does so, then the specification becomes:

   spec
      package: 'Grease-Core';
      package: 'Grease-Tests-Core' with: [
        spec requires: #('Grease-Core' ). ].

This form of the specification is nowhere near as complex as the original specification and there is the added benefit, that the specification itself can be managed as part of the common code base ... which means it becomes easy for other branches to track common structural changes as well as common code changes.
 

[1] https://github.com/dalehenrich/filetree/blob/master/doc/Contribute.md

|
| > I don't like the idea of requiring a "branch refactoring", but then
| > again I don't relish the idea of inventing a VersionOf structure
| > that should never be really used in the first place …
|
| I do not know since I do not really understand what you mean but if
| it is having one specification for each platform I think that this
| is good.

Yes, I do mean one specification for each platform and a greatly simplified specification to boot.
| >
| > In reality, I think that the branch per platform model will improve
| > the quality of projects and make it possible for folks to judge
| > the status of the different platforms by looking at the network
| > diagram[4] for the project.
|
|
| > I've attached a screen shot of the FileTree branch network and you
| > can clearly see that the pharo1.3 and master branches are "ahead
| > of" the gemstone2.4 and sqeuak4.3 branches, so as a user I would
| > know that some merging was required for those two projects before
| > I got the latest code ... hovering over the little dots in the
| > diagram gives you commit comment and clicking on the dot takes you
| > to the commit, so you can easily tell if you are in fact missing
| > something important …
|
| I do not see how this is related to platform branch (even if I like
| the idea to separate them - this will help to reduce the complexity
| of certain versions), I imagine that
| we could extract the same information without platform branch. Just
| more painful so to me this is the not a selling point.

The "lack of pain: part is a selling point to me:)

|
|
| I try to understand what you are saying and I see two point
| - dependency declared at the package level?
| - separation of platforms

I am tempted to restate that to:

  - dependency declared on per-platform basis
  - separate branches per platform

Dale

Dale we have 4 papers on the grill for saturday plus a lot more shit (like admin days for me).
I flagged these mails and I will read them carefully as soon as I can.


Stef

On Apr 10, 2012, at 11:36 PM, Dale Henrichs wrote:

Stef,

I appreciate the effort and the feedback ... I realize that folks are busy so it's important to me to be able to expect feedback at some point in the future and I will move forward carefully until I hear back from you ...

Dale

----- Original Message -----
| From: "stephane ducasse" <[hidden email]>
| To: [hidden email]
| Sent: Wednesday, April 11, 2012 10:38:54 AM
| Subject: Re: [Metacello] Metacello and FileTree/Cypress
|
| Dale we have 4 papers on the grill for saturday plus a lot more shit
| (like admin days for me).
| I flagged these mails and I will read them carefully as soon as I
| can.
|
|
| Stef
|
| On Apr 10, 2012, at 11:36 PM, Dale Henrichs wrote:
|
| >
| >
| > ----- Original Message -----
| > | From: "stephane ducasse" <[hidden email]>
| > | To: [hidden email]
| > | Sent: Tuesday, April 10, 2012 12:17:44 PM
| > | Subject: Re: [Metacello] Metacello and FileTree/Cypress
| > |
| > | Hi dale
| > |
| > |
| > | > My point is that we can SIGNIFICANTLY reduce the complexity of
| > | > the
| > | > structural (baseline) specifications to the point where it no
| > | > longer takes a degree in Metacello to specify the structure.
| > | > The
| > | > GST folks use package.xml[3] to specify the structural
| > | > dependencies for their projects and we could do something very
| > | > similar with a simple JSON file that is dead simple to modify
| > | > and
| > | > needs no complex Metacello tools to manage ...
| > | >
| > | > Of course the cost of this simplicity is to adopt the branch
| > | > per
| > | > platform model and to "require" that projects undergo a "branch
| > | > refactoring".
| > |
| > | I have difficulties to assess the impact of declaring
| > | dependencies at
| > | the package level.
| >
| >
| > Here's a better example (hopefully:), given this baseline spec from
| > a ConfigurationOfGrease, where each platform has a
| > 'Grease-XXX-Core' package:
| >
| >  spec for: #common do: [
| >    spec
| >      package: 'Grease-Core';
| >      package: 'Grease-Tests-Core' with: [
| >        spec requires: #('Grease-Core' ). ]. ].
| >
| >  spec for: #pharo do: [
| >    spec
| >      package: 'Grease-Pharo-Core' with: [
| >        spec requires: #('Grease-Core' ). ]. ].
| >
| >  spec for: #squeak do: [
| >    spec
| >      package: 'Grease-Squeak-Core' with: [
| >        spec requires: #('Grease-Core' ). ]. ].
| >
| >  spec for: #gemstone do: [
| >    spec
| >      package: 'Grease-GemStone-Core' with: [
| >        spec requires: #('Grease-Core' ). ]. ].
| >
| > We have each of the plaform-specific packages fenced off in it's
| > own #for:do: block because we are sharing the specification across
| > all platforms.
| >
| > If we were to write a specification that was for pharo only (i.e.,
| > the version of the specification is on the #pharo branch of the
| > project) then the overall specification would be greatly
| > simplified:
| >
| >    spec
| >      package: 'Grease-Core';
| >      package: 'Grease-Tests-Core' with: [
| >        spec requires: #('Grease-Core' ). ];
| >      package: 'Grease-Pharo-Core' with: [
| >        spec requires: #('Grease-Core' ). ].
| >
| > on the squeak branch the spec would be:
| >
| >    spec
| >      package: 'Grease-Core';
| >      package: 'Grease-Tests-Core' with: [
| >        spec requires: #('Grease-Core' ). ];
| >      package: 'Grease-Squeak-Core' with: [
| >        spec requires: #('Grease-Core' ). ].
| >
| > and so on.
| >
| > With per platform branching, it is not necessary to include
| > information about any other platform in the specification on the
| > platform-specific branch.
| >
| > | What happen if a local information gets wrong?
| >
| > Not quite sure what you are getting at here. Currently we have this
| > type of problem, only right now it is possible to make an editting
| > mistake that impacts other platforms ... by isolating the changes
| > on platform branches, the only platform that is affected is the
| > one that is being worked on ..
| >
| > Furthermore, using gitHub, developers will be able to insulate
| > themselves from mistakes made in the public repository by forking
| > the projects ... when you fork a project you control the rate at
| > which you see updates and can apply your own testing and
| > validation process before ACCEPTING any change into your fork of
| > the project...
| >
| > | I do not know what branch refactoring means.
| >
| > What I mean by "branch refactoring" is that given the following
| > specification on the pharo branch:
| >
| >    spec
| >      package: 'Grease-Core';
| >      package: 'Grease-Tests-Core' with: [
| >        spec requires: #('Grease-Core' ). ];
| >      package: 'Grease-Pharo-Core' with: [
| >        spec requires: #('Grease-Core' ). ].
| >
| > There is no practical reason to keep the Grease-Pharo-Core separate
| > from the Grease-Core package. Today we do so because we need the
| > Grease-Core package to be shared amongst the various platforms.
| >
| > With git/github, one can manage the common changes without having
| > to put them in a separate package. See the 'Guildelines for
| > Contributing'[1] for the FileTree project to get an explanation of
| > the process I am referring to.
| >
| > Doing a "branch refactoring" involves merging the Grease-Pharo-Core
| > package into the Grease-Core package on the pharo branch. If one
| > does so, then the specification becomes:
| >
| >   spec
| >      package: 'Grease-Core';
| >      package: 'Grease-Tests-Core' with: [
| >        spec requires: #('Grease-Core' ). ].
| >
| > This form of the specification is nowhere near as complex as the
| > original specification and there is the added benefit, that the
| > specification itself can be managed as part of the common code
| > base ... which means it becomes easy for other branches to track
| > common structural changes as well as common code changes.
| >
| >
| > [1]
| > https://github.com/dalehenrich/filetree/blob/master/doc/Contribute.md
| >
| > |
| > | > I don't like the idea of requiring a "branch refactoring", but
| > | > then
| > | > again I don't relish the idea of inventing a VersionOf
| > | > structure
| > | > that should never be really used in the first place …
| > |
| > | I do not know since I do not really understand what you mean but
| > | if
| > | it is having one specification for each platform I think that
| > | this
| > | is good.
| >
| > Yes, I do mean one specification for each platform and a greatly
| > simplified specification to boot.
| > | >
| > | > In reality, I think that the branch per platform model will
| > | > improve
| > | > the quality of projects and make it possible for folks to judge
| > | > the status of the different platforms by looking at the network
| > | > diagram[4] for the project.
| > |
| > |
| > | > I've attached a screen shot of the FileTree branch network and
| > | > you
| > | > can clearly see that the pharo1.3 and master branches are
| > | > "ahead
| > | > of" the gemstone2.4 and sqeuak4.3 branches, so as a user I
| > | > would
| > | > know that some merging was required for those two projects
| > | > before
| > | > I got the latest code ... hovering over the little dots in the
| > | > diagram gives you commit comment and clicking on the dot takes
| > | > you
| > | > to the commit, so you can easily tell if you are in fact
| > | > missing
| > | > something important …
| > |
| > | I do not see how this is related to platform branch (even if I
| > | like
| > | the idea to separate them - this will help to reduce the
| > | complexity
| > | of certain versions), I imagine that
| > | we could extract the same information without platform branch.
| > | Just
| > | more painful so to me this is the not a selling point.
| >
| > The "lack of pain: part is a selling point to me:)
| >
| > |
| > |
| > | I try to understand what you are saying and I see two point
| > | - dependency declared at the package level?
| > | - separation of platforms
| >
| > I am tempted to restate that to:
| >
| >  - dependency declared on per-platform basis
| >  - separate branches per platform
| >
| > Dale
|
|