feedback on using JSON to specify baselines for git repository

In mapping Metacello to a git-based repository system (versioned directory structure as opposed to versions of files in a directory) the first thing that happens is we no longer need to manage versions in the Metacello configuration ... when you checkout a directory structure from git the files that are present are the only version you need to worry about so it's only necessary to specify dependencies (project and package). When you add or remove packages, you will create a new version of the baseline spec in the directory and commit the whole enchilada ...

If you thought is was overkill to use a class to define all of the versions for a project, then burning a class per version is much worse ... I've toyed with some ideas of using doits instead of classes, but there is a very real issue that Pharo has a limit of 255 literals in a method and the Seaside30 configuration will easily exceed that limit if I attempted to put the spec in a single method/doit ...

I have been threatening to use a structured file format for the spec for awhile now and this will probably be a good time to make the change. Note that besides the method literal limit, there are problems with using live Smalltalk code to define version specs:

  - as it stands now, one cannot afford to reason about a configuration in a site
    like SqueakSource, because you have to load a package and execute arbitrary
    Smalltalk code ... a major security violation ...

Today I began playing with the idea of using JSON for the specification and you can see a JSON example of the spec file[1] for the FileTree project on github (yes I'm using git/github right now). For comparison you can look at the baseline method for the ConfigurationOfFileTree[2], which has exactly the same information.

I think JSON is just as readable as the configuration method version. If you want, you can look at the JSON variant for Metacello and compare it to a baseline method[4].

Of course, this is all work in progress, but I'd like to have early feedback if anyone has strong opnions or worries ...

The two metacello.json files that you are looking at were generated from the relevant configuration, so I think that it should be relatively easy to convert existing configurations (and vice versa) ...

Dale

[1] https://github.com/dalehenrich/filetree/blob/master/metacello.json
[2] https://github.com/dalehenrich/filetree/blob/master/ConfigurationOfFileTree.tree/ConfigurationOfFileTree/baseline100:.st
[3] https://github.com/dalehenrich/metacello/blob/MetacelloGitProject/metacello.json
[4] https://github.com/dalehenrich/metacello/blob/46e6eece660405ee4a527285497b2fe4bfe2e23a/ConfigurationOfGitMetacello.tree/ConfigurationOfGitMetacello/baseline:

Dale

I'm lost.
Now my point of view is that I would love to have a 1.0 for metacello based on MC so that we
can update the doc and freeze. In addition, I would like to avoid to depend on git because tomorrow
people may want something else….

Stef


On Feb 4, 2012, at 2:09 AM, Dale Henrichs wrote:

Stef,

Metacello will continue to work just fine for the standard MC. The traditional Monticello model will be fully supported along with a file-based model that can be used with svn, git, or any other file-based SCM ... The git model is based on Monticello so things will different, but not that different.

The new JSON-based file format that I am proposing will only be used for the file-based SCMs and will be completely compatible with the traditional model. If you recall, I have always talked about a structured file format for Metacello so supporting a JSON mapping is a natural extension of my plans.

I am making the detour into git mainly because I think that git offers some real critical features that will be more work to provide with traditional Monticello. I have spent the last year using Metacello/Monticello with an eye towards the requirements for better support of group development ...

As it stands right now, it is not practical for a group of developers to use Metacello/Monticello. I do see a path towards making it practical, but the bulk of work that is required is duplication of features that alread exist in git ...

Until last week I had always considered that the move to a file-based SCM for Smalltalk wasn't practical, but once I took a look at the work that Otto Behrens had done with FileTree (which stores Monticello packages as source in a directory structure while preserving the important semantics of isolating the key package components), it became obvious that a move to a file-based SCM _is_ finally practical. Add to that the work that Max Leske and Camillo Bruni have already done on providing a Smalltalk-based API for git, and you've got like 85% work done. All that really remains is a Metacello mapping and a scripting API and ....

I had already started the scripting API, so I think that with the potential benefits of git/github within reach, it makes sense to spend some time to develop the git connection a bit more and see if there are any major issues ...

Having spent the last 3 days on looking at the issue, it is pretty obvious that things are very close ... I plan to invest at least another week on the filetree-git side of things with the expectation that at the end of that time, I will start doing full time development for Metacello (and my other projects) using git/github, I will continue to deploy Metacello with the traditional mcz files and the GemSource repository.

I wouldn't be expecting individual developers to use git in the early stages as one will need to be familiar with git to make effective use of git, so all of the Metacello documentation will continue to be valid. I do think that for the projects where there are a number of developers doing concurrent development, the git model will be an major improvement over anything that can be done today.

If I hit a blockade along the way, I'll back off and finish up the work planned for the scripting API and 1.0-beta.32 bugs.

Personally I am excited at the prospect of being able to use git/github to manage my projects

Dale
 
----- Original Message -----
| From: "stephane ducasse" <[hidden email]>
| To: [hidden email]
| Sent: Saturday, February 4, 2012 5:47:29 AM
| Subject: Re: [Metacello] feedback on using JSON to specify baselines for git repository
|
| Dale
|
| I'm lost.
| Now my point of view is that I would love to have a 1.0 for metacello
| based on MC so that we
| can update the doc and freeze. In addition, I would like to avoid to
| depend on git because tomorrow
| people may want something else….
|
| Stef
|
|
| On Feb 4, 2012, at 2:09 AM, Dale Henrichs wrote:
|
| > In mapping Metacello to a git-based repository system (versioned
| > directory structure as opposed to versions of files in a
| > directory) the first thing that happens is we no longer need to
| > manage versions in the Metacello configuration ... when you
| > checkout a directory structure from git the files that are present
| > are the only version you need to worry about so it's only
| > necessary to specify dependencies (project and package). When you
| > add or remove packages, you will create a new version of the
| > baseline spec in the directory and commit the whole enchilada ...
| >
| > If you thought is was overkill to use a class to define all of the
| > versions for a project, then burning a class per version is much
| > worse ... I've toyed with some ideas of using doits instead of
| > classes, but there is a very real issue that Pharo has a limit of
| > 255 literals in a method and the Seaside30 configuration will
| > easily exceed that limit if I attempted to put the spec in a
| > single method/doit ...
| >
| > I have been threatening to use a structured file format for the
| > spec for awhile now and this will probably be a good time to make
| > the change. Note that besides the method literal limit, there are
| > problems with using live Smalltalk code to define version specs:
| >
| >  - as it stands now, one cannot afford to reason about a
| >  configuration in a site
| >    like SqueakSource, because you have to load a package and
| >    execute arbitrary
| >    Smalltalk code ... a major security violation ...
| >
| > Today I began playing with the idea of using JSON for the
| > specification and you can see a JSON example of the spec file[1]
| > for the FileTree project on github (yes I'm using git/github right
| > now). For comparison you can look at the baseline method for the
| > ConfigurationOfFileTree[2], which has exactly the same
| > information.
| >
| > I think JSON is just as readable as the configuration method
| > version. If you want, you can look at the JSON variant for
| > Metacello and compare it to a baseline method[4].
| >
| > Of course, this is all work in progress, but I'd like to have early
| > feedback if anyone has strong opnions or worries ...
| >
| > The two metacello.json files that you are looking at were generated
| > from the relevant configuration, so I think that it should be
| > relatively easy to convert existing configurations (and vice
| > versa) ...
| >
| > Dale
| >
| > [1]
| > https://github.com/dalehenrich/filetree/blob/master/metacello.json
| > [2]
| > https://github.com/dalehenrich/filetree/blob/master/ConfigurationOfFileTree.tree/ConfigurationOfFileTree/baseline100:.st
| > [3]
| > https://github.com/dalehenrich/metacello/blob/MetacelloGitProject/metacello.json
| > [4]
| > https://github.com/dalehenrich/metacello/blob/46e6eece660405ee4a527285497b2fe4bfe2e23a/ConfigurationOfGitMetacello.tree/ConfigurationOfGitMetacello/baseline:
|
|

On Feb 4, 2012, at 8:25 PM, Dale Henrichs wrote:

> Stef,
>
> Metacello will continue to work just fine for the standard MC. The traditional Monticello model will be fully supported along with a file-based model that can be used with svn, git, or any other file-based SCM ... The git model is based on Monticello so things will different, but not that different.

Ok :)

> The new JSON-based file format that I am proposing will only be used for the file-based SCMs and will be completely compatible with the traditional model. If you recall, I have always talked about a structured file format for Metacello so supporting a JSON mapping is a natural extension of my plans.

JSON is for saving the meta model? method? Class definitions ?
>
> I am making the detour into git mainly because I think that git offers some real critical features that will be more work to provide with traditional Monticello. I have spent the last year using Metacello/Monticello with an eye towards the requirements for better support of group development ...

Sure monticello does not support resources.

> As it stands right now, it is not practical for a group of developers to use Metacello/Monticello. I do see a path towards making it practical, but the bulk of work that is required is duplication of features that alread exist in git ...
>
> Until last week I had always considered that the move to a file-based SCM for Smalltalk wasn't practical, but once I took a look at the work that Otto Behrens had done with FileTree (which stores Monticello packages as source in a directory structure while preserving the important semantics of isolating the key package components), it became obvious that a move to a file-based SCM _is_ finally practical. Add to that the work that Max Leske and Camillo Bruni have already done on providing a Smalltalk-based API for git, and you've got like 85% work done. All that really remains is a Metacello mapping and a scripting API and ….

OK :)
I'm not bound to MC. Just something that works and has tool support.

Stef,

The JSON is used to define a single baseline version for the packages in a disk-based repository:

[
        {"baseline" : [
                {"common" :
                        [
                        {"package" : {
                                "name" : "MonticelloFileTree-Core"}},
                        {"group" : {
                                "name" : "default",
                                "includes" : ["MonticelloFileTree-Core"]}}]},
                {"gemstone" :
                        [
                        {"package" : {
                                "name" : "MonticelloFileTree-Core",
                                "includes" : ["MonticelloFileTree-GemStone-Extensions"]}},
                        {"package" : {
                                "name" : "MonticelloFileTree-GemStone-Extensions",
                                "requires" : ["MonticelloFileTree-Core"]}}]}
        ]}
]

as compared to:

!ConfigurationOfFileTree methodsFor: 'baselines' stamp: 'dkh 1/31/2012 15:19'!
baseline100: spec
    <version: '1.0-baseline'>
    spec
        for: #'common'
        do: [
            spec blessing: #'baseline'.
            spec package: 'MonticelloFileTree-Core'.
            spec group: 'default' with: #('MonticelloFileTree-Core') ].
    spec
        for: #'gemstone'
        do: [
            spec package: 'MonticelloFileTree-Core' with: [ spec includes: 'MonticelloFileTree-GemStone-Extensions' ].
            spec package: 'MonticelloFileTree-GemStone-Extensions' with: [ spec requires: 'MonticelloFileTree-Core' ] ]! !

There is straightforward mapping between the two.

It doesn't make sense to create a class per version (the case with directory-based repositories), so instead of continuing with Smalltalk code based specs (literal limitation looming), I think that going for a structured file representation of the single version makes sense and JSON seems to be a pretty good alterative ...

Dale
----- Original Message -----
| From: "stephane ducasse" <[hidden email]>
| To: [hidden email]
| Sent: Saturday, February 4, 2012 1:02:47 PM
| Subject: Re: [Metacello] feedback on using JSON to specify baselines for git repository
|
|
| On Feb 4, 2012, at 8:25 PM, Dale Henrichs wrote:
|
| > Stef,
| >
| > Metacello will continue to work just fine for the standard MC. The
| > traditional Monticello model will be fully supported along with a
| > file-based model that can be used with svn, git, or any other
| > file-based SCM ... The git model is based on Monticello so things
| > will different, but not that different.
|
| Ok :)
|
| > The new JSON-based file format that I am proposing will only be
| > used for the file-based SCMs and will be completely compatible
| > with the traditional model. If you recall, I have always talked
| > about a structured file format for Metacello so supporting a JSON
| > mapping is a natural extension of my plans.
|
| JSON is for saving the meta model? method? Class definitions ?
| >
| > I am making the detour into git mainly because I think that git
| > offers some real critical features that will be more work to
| > provide with traditional Monticello. I have spent the last year
| > using Metacello/Monticello with an eye towards the requirements
| > for better support of group development ...
|
| Sure monticello does not support resources.
|
| > As it stands right now, it is not practical for a group of
| > developers to use Metacello/Monticello. I do see a path towards
| > making it practical, but the bulk of work that is required is
| > duplication of features that alread exist in git ...
| >
| > Until last week I had always considered that the move to a
| > file-based SCM for Smalltalk wasn't practical, but once I took a
| > look at the work that Otto Behrens had done with FileTree (which
| > stores Monticello packages as source in a directory structure
| > while preserving the important semantics of isolating the key
| > package components), it became obvious that a move to a file-based
| > SCM _is_ finally practical. Add to that the work that Max Leske
| > and Camillo Bruni have already done on providing a Smalltalk-based
| > API for git, and you've got like 85% work done. All that really
| > remains is a Metacello mapping and a scripting API and ….
|
| OK :)
| I'm not bound to MC. Just something that works and has tool support.
|
| > I had already started the scripting API, so I think that with the
| > potential benefits of git/github within reach, it makes sense to
| > spend some time to develop the git connection a bit more and see
| > if there are any major issues ...
| >
| > Having spent the last 3 days on looking at the issue, it is pretty
| > obvious that things are very close ... I plan to invest at least
| > another week on the filetree-git side of things with the
| > expectation that at the end of that time, I will start doing full
| > time development for Metacello (and my other projects) using
| > git/github, I will continue to deploy Metacello with the
| > traditional mcz files and the GemSource repository.
| >
| > I wouldn't be expecting individual developers to use git in the
| > early stages as one will need to be familiar with git to make
| > effective use of git, so all of the Metacello documentation will
| > continue to be valid. I do think that for the projects where there
| > are a number of developers doing concurrent development, the git
| > model will be an major improvement over anything that can be done
| > today.
| >
| > If I hit a blockade along the way, I'll back off and finish up the
| > work planned for the scripting API and 1.0-beta.32 bugs.
| >
| > Personally I am excited at the prospect of being able to use
| > git/github to manage my projects
| >
| > Dale
| >
| > ----- Original Message -----
| > | From: "stephane ducasse" <[hidden email]>
| > | To: [hidden email]
| > | Sent: Saturday, February 4, 2012 5:47:29 AM
| > | Subject: Re: [Metacello] feedback on using JSON to specify
| > | baselines for git repository
| > |
| > | Dale
| > |
| > | I'm lost.
| > | Now my point of view is that I would love to have a 1.0 for
| > | metacello
| > | based on MC so that we
| > | can update the doc and freeze. In addition, I would like to avoid
| > | to
| > | depend on git because tomorrow
| > | people may want something else….
| > |
| > | Stef
| > |
| > |
| > | On Feb 4, 2012, at 2:09 AM, Dale Henrichs wrote:
| > |
| > | > In mapping Metacello to a git-based repository system
| > | > (versioned
| > | > directory structure as opposed to versions of files in a
| > | > directory) the first thing that happens is we no longer need to
| > | > manage versions in the Metacello configuration ... when you
| > | > checkout a directory structure from git the files that are
| > | > present
| > | > are the only version you need to worry about so it's only
| > | > necessary to specify dependencies (project and package). When
| > | > you
| > | > add or remove packages, you will create a new version of the
| > | > baseline spec in the directory and commit the whole enchilada
| > | > ...
| > | >
| > | > If you thought is was overkill to use a class to define all of
| > | > the
| > | > versions for a project, then burning a class per version is
| > | > much
| > | > worse ... I've toyed with some ideas of using doits instead of
| > | > classes, but there is a very real issue that Pharo has a limit
| > | > of
| > | > 255 literals in a method and the Seaside30 configuration will
| > | > easily exceed that limit if I attempted to put the spec in a
| > | > single method/doit ...
| > | >
| > | > I have been threatening to use a structured file format for the
| > | > spec for awhile now and this will probably be a good time to
| > | > make
| > | > the change. Note that besides the method literal limit, there
| > | > are
| > | > problems with using live Smalltalk code to define version
| > | > specs:
| > | >
| > | >  - as it stands now, one cannot afford to reason about a
| > | >  configuration in a site
| > | >    like SqueakSource, because you have to load a package and
| > | >    execute arbitrary
| > | >    Smalltalk code ... a major security violation ...
| > | >
| > | > Today I began playing with the idea of using JSON for the
| > | > specification and you can see a JSON example of the spec
| > | > file[1]
| > | > for the FileTree project on github (yes I'm using git/github
| > | > right
| > | > now). For comparison you can look at the baseline method for
| > | > the
| > | > ConfigurationOfFileTree[2], which has exactly the same
| > | > information.
| > | >
| > | > I think JSON is just as readable as the configuration method
| > | > version. If you want, you can look at the JSON variant for
| > | > Metacello and compare it to a baseline method[4].
| > | >
| > | > Of course, this is all work in progress, but I'd like to have
| > | > early
| > | > feedback if anyone has strong opnions or worries ...
| > | >
| > | > The two metacello.json files that you are looking at were
| > | > generated
| > | > from the relevant configuration, so I think that it should be
| > | > relatively easy to convert existing configurations (and vice
| > | > versa) ...
| > | >
| > | > Dale
| > | >
| > | > [1]
| > | > https://github.com/dalehenrich/filetree/blob/master/metacello.json
| > | > [2]
| > | > https://github.com/dalehenrich/filetree/blob/master/ConfigurationOfFileTree.tree/ConfigurationOfFileTree/baseline100:.st
| > | > [3]
| > | > https://github.com/dalehenrich/metacello/blob/MetacelloGitProject/metacello.json
| > | > [4]
| > | > https://github.com/dalehenrich/metacello/blob/46e6eece660405ee4a527285497b2fe4bfe2e23a/ConfigurationOfGitMetacello.tree/ConfigurationOfGitMetacello/baseline:
| > |
| > |
|
|

Some other questions:

        - with Git would be the merge done by git?
       

> The new JSON-based file format that I am proposing will only be used for the file-based SCMs and will be completely compatible with the traditional model. If you recall, I have always talked about a structured file format for Metacello so supporting a JSON mapping is a natural extension of my plans.

can you give one example?
Because we want to have a real code representation exchange format for logging changes:
        having method, and class is not enough.
        Now we have ring so we could expect that a new logger would log better the actions we are doing

> I am making the detour into git mainly because I think that git offers some real critical features that will be more work to provide with traditional Monticello. I have spent the last year using Metacello/Monticello with an eye towards the requirements for better support of group development ...
>
> As it stands right now, it is not practical for a group of developers to use Metacello/Monticello. I do see a path towards making it practical, but the bulk of work that is required is duplication of features that alread exist in git ...

That I do not know. For me I do not see problems beside resources management. I do not see the difference
for the rest because I can merge and the merge works quite well.
What I know is that we technical people are excited by technology. People are telling me that git is wonderful
and that I should stop using svn. Well so far Git just alienates me and I need a bad UI to manage simple pull requests.


> Until last week I had always considered that the move to a file-based SCM for Smalltalk wasn't practical, but once I took a look at the work that Otto Behrens had done with FileTree (which stores Monticello packages as source in a directory structure while preserving the important semantics of isolating the key package components), it became obvious that a move to a file-based SCM _is_ finally practical. Add to that the work that Max Leske and Camillo Bruni have already done on providing a Smalltalk-based API for git, and you've got like 85% work done. All that really remains is a Metacello mapping and a scripting API and ....
>
> I had already started the scripting API, so I think that with the potential benefits of git/github within reach, it makes sense to spend some time to develop the git connection a bit more and see if there are any major issues ...
>
> Having spent the last 3 days on looking at the issue, it is pretty obvious that things are very close ... I plan to invest at least another week on the filetree-git side of things with the expectation that at the end of that time, I will start doing full time development for Metacello (and my other projects) using git/github, I will continue to deploy Metacello with the traditional mcz files and the GemSource repository.
>
> I wouldn't be expecting individual developers to use git in the early stages as one will need to be familiar with git to make effective use of git, so all of the Metacello documentation will continue to be valid. I do think that for the projects where there are a number of developers doing concurrent development, the git model will be an major improvement over anything that can be done today.

Can you explain why because I do not see this major improvement?
We can merge, commit locally, sync with MC.
Even if I have no problem getting rid of a buggy squeak source (even 3.0) where I spend too much time
moving file around and MCfiles based management.

The real problem of MC is that we manage the metadata and ancestry in the image and it grows endlessly.

Dale,

Am 2012-02-04 um 23:31 schrieb Dale Henrichs:

I like the idea, but I think, YAML would be even “nicer”:

---
configuration: FileTree
stable: '1.0'
versions:
    - &1-0-baseline
        name: '1.0-baseline'
        timestamp: 2012-01-31T15:19Z
        author: dkh
        common:
            blessing: baseline
            packages:
                - name: MonticelloFileTree-Core
        gemstone:
            packages:
                - name: MonticelloFileTree-Core
                  includes:
                        - MonticelloFileTree-GemStone-Extensions
                - name: MonticelloFileTree-GemStone-Extensions
                  requires:
                        - MonticelloFileTree-Core
    - &1-0
        name: '1.0'
        imports: *1-0-baseline
        timestamp: 2012-02-05T17:40Z
        author: topa
        description: |
            This is just for demonstration.
            Newlines preserved here.
        common:
            blessing: beta
            packages:
                - name: MonticelloFileTree-Core
                  file: MonticelloFileTree-Core-topa.42
        gemstone:
            packages:
                - name: MonticelloFileTree-GemStone-Extensions
                  file: MonticelloFileTree-GemStone-Extensions-topa.23


YAML (http://yaml.org/) is a superset of json (see http://yaml.org/spec/1.2/spec.html#id2759572, http://stackoverflow.com/questions/1726802)

What do y'all think? I think its a little bit
better readable than pure json ;)

Best
        -Tobias

Tobias,

YAML looks a little more compact than JSON and is definitely more readable ... Is there a Smalltalk-base parser for YAML floating around?

Dale

----- Original Message -----
| From: "Tobias Pape" <[hidden email]>
| To: [hidden email]
| Sent: Sunday, February 5, 2012 8:57:26 AM
| Subject: Re: [Metacello] feedback on using JSON to specify baselines for git repository
|
| Dale,
|
| Am 2012-02-04 um 23:31 schrieb Dale Henrichs:
|
| > Stef,
| >
| > The JSON is used to define a single baseline version for the
| > packages in a disk-based repository:
| >
| > [
| > {"baseline" : [
| > {"common" :
| > [
| > {"package" : {
| > "name" : "MonticelloFileTree-Core"}},
| > {"group" : {
| > "name" : "default",
| > "includes" : ["MonticelloFileTree-Core"]}}]},
| > {"gemstone" :
| > [
| > {"package" : {
| > "name" : "MonticelloFileTree-Core",
| > "includes" : ["MonticelloFileTree-GemStone-Extensions"]}},
| > {"package" : {
| > "name" : "MonticelloFileTree-GemStone-Extensions",
| > "requires" : ["MonticelloFileTree-Core"]}}]}
| > ]}
| > ]
|
| I like the idea, but I think, YAML would be even “nicer”:
|
| ---
| configuration: FileTree
| stable: '1.0'
| versions:
|     - &1-0-baseline
|         name: '1.0-baseline'
|         timestamp: 2012-01-31T15:19Z
|         author: dkh
|         common:
|             blessing: baseline
|             packages:
|                 - name: MonticelloFileTree-Core
|         gemstone:
|             packages:
|                 - name: MonticelloFileTree-Core
|                   includes:
|                         - MonticelloFileTree-GemStone-Extensions
|                 - name: MonticelloFileTree-GemStone-Extensions
|                   requires:
|                         - MonticelloFileTree-Core
|     - &1-0
|         name: '1.0'
|         imports: *1-0-baseline
|         timestamp: 2012-02-05T17:40Z
|         author: topa
|         description: |
|             This is just for demonstration.
|             Newlines preserved here.
|         common:
|             blessing: beta
|             packages:
|                 - name: MonticelloFileTree-Core
|                   file: MonticelloFileTree-Core-topa.42
|         gemstone:
|             packages:
|                 - name: MonticelloFileTree-GemStone-Extensions
|                   file:
|                   MonticelloFileTree-GemStone-Extensions-topa.23
|
|
| YAML (http://yaml.org/) is a superset of json (see
| http://yaml.org/spec/1.2/spec.html#id2759572,
| http://stackoverflow.com/questions/1726802)
|
| What do y'all think? I think its a little bit
| better readable than pure json ;)
|
| Best
| -Tobias
|
|

Am 2012-02-05 um 19:06 schrieb Dale Henrichs:

> Tobias,
>
> YAML looks a little more compact than JSON and is definitely more readable ... Is there a Smalltalk-base parser for YAML floating around?

A quick search on the Squeaksources says no,
A googling the same.

However, a yaml-parser is overdue for smalltalk ;)
And since (yaml includes: json), probably the existing
json things can be used/extended.

What do  y'all think?

Best
        -Tobias

----- Original Message -----
| From: "stephane ducasse" <[hidden email]>
| To: [hidden email]
| Sent: Saturday, February 4, 2012 11:54:18 PM
| Subject: Re: [Metacello] feedback on using JSON to specify baselines for git repository
|
| Some other questions:
|
| - with Git would be the merge done by git?

Yes. In the beginning, the git (or SVN) tools that support the merge would be used to do the source level merges. Over time, if it is successful I imagine that image-level for doing the merge would be developed...

|
|
| > The new JSON-based file format that I am proposing will only be
| > used for the file-based SCMs and will be completely compatible
| > with the traditional model. If you recall, I have always talked
| > about a structured file format for Metacello so supporting a JSON
| > mapping is a natural extension of my plans.
|
| can you give one example?
| Because we want to have a real code representation exchange format
| for logging changes:
| having method, and class is not enough.
| Now we have ring so we could expect that a new logger would log
| better the actions we are doing

The JSON-based format is for Metacello specifications only. Using a class for ConfigurationOfXXX has been a convenient and expedient mechanism for defining configurations in Metacello, but using executable code to define a specification is not ideal:

  - one cannot extract specification information without executing
    code is bad from a security point of view
  - tools have to create classes and generate methods in order to
    write a specification, again not an ideal situation.

With the file based SCM systems (SVN and GIT) the Metacello configuration does not have to become the data base for all versions of the project. It is only necessary for Metacello to specify required projects and package dependencies, basically all we need is a baseline specification.

For a single baseline specification, one would need only one method in the current class-based system so including a class in the mix is basically overkill. I considered using a doit to define the specification, but thought that now would be a good time to go for the structured file to specify the baseline.

|
| > I am making the detour into git mainly because I think that git
| > offers some real critical features that will be more work to
| > provide with traditional Monticello. I have spent the last year
| > using Metacello/Monticello with an eye towards the requirements
| > for better support of group development ...
| >
| > As it stands right now, it is not practical for a group of
| > developers to use Metacello/Monticello. I do see a path towards
| > making it practical, but the bulk of work that is required is
| > duplication of features that alread exist in git ...
|
| That I do not know. For me I do not see problems beside resources
| management. I do not see the difference
| for the rest because I can merge and the merge works quite well.
| What I know is that we technical people are excited by technology.
| People are telling me that git is wonderful
| and that I should stop using svn. Well so far Git just alienates me
| and I need a bad UI to manage simple pull requests.

You are talking about merging individual mcz files working well and I agree. Monticello is a very nice system for dealing with versioned files.

As it stands today, there is no project merge capability. There is really no way for developers to create a branch at the project level, work for hours, days or weeks independently and then come back and merge their configuration changes and package changes. If you figured out which sets of packages changed, you would be able to merge the smalltalk code, but if you had made changes to the baselines and versions there is no merge capability...

When I started out with Metacello, I imagined that merging would be straightforward source code comparison. Today I don't think so. If you recall last spring Alexandre's style of using Metacello was to create a new project version for each commit. The basic problem wasn't that it was "wrong" to do so, it just made doing any sort of source-based merge impossible.

In order to merge two branches of a Metacello project that involve changes to more than just the same version, one must create two or three version specifications in memory at the same time. For an mcz file, you can keep multiple definitions because the mcz file is a serialized version of the meta data. For metacello, we have to load a class and execute a method for each of the three versions and if the version of the class in the initial image is dirty? What do we do?

Now technically there is a solution to this merge problem, but it involves at a minimum creating a completely different file format for configurations, not to mention a whole set of new tools ....

As for SVN and GIT, I think that the jury is still out on whether SVN is better or worse than GIT ... there isn't really a clear winner there.

The bulk of the work that I have and will be doing will be equally applicable to SVN and GIT ... we use SVN within GemStone and I intend to use this new work for managing the GemStone GLASS source in SVN directly rather than versioning a directory of mcz files ....

I believe that SVN and GIT can be used interchangeably especially with the initial versions of my work, since I will be relying on the developers using the SVN/GIT tools for managing their repositories.

On the other hand GitHub has a lot of features that we don't have in SqueakSource that would be real useful to have ... so because of GitHub, I am interested in making a push directed at git.

|
|
| > Until last week I had always considered that the move to a
| > file-based SCM for Smalltalk wasn't practical, but once I took a
| > look at the work that Otto Behrens had done with FileTree (which
| > stores Monticello packages as source in a directory structure
| > while preserving the important semantics of isolating the key
| > package components), it became obvious that a move to a file-based
| > SCM _is_ finally practical. Add to that the work that Max Leske
| > and Camillo Bruni have already done on providing a Smalltalk-based
| > API for git, and you've got like 85% work done. All that really
| > remains is a Metacello mapping and a scripting API and ....
| >
| > I had already started the scripting API, so I think that with the
| > potential benefits of git/github within reach, it makes sense to
| > spend some time to develop the git connection a bit more and see
| > if there are any major issues ...
| >
| > Having spent the last 3 days on looking at the issue, it is pretty
| > obvious that things are very close ... I plan to invest at least
| > another week on the filetree-git side of things with the
| > expectation that at the end of that time, I will start doing full
| > time development for Metacello (and my other projects) using
| > git/github, I will continue to deploy Metacello with the
| > traditional mcz files and the GemSource repository.
| >
| > I wouldn't be expecting individual developers to use git in the
| > early stages as one will need to be familiar with git to make
| > effective use of git, so all of the Metacello documentation will
| > continue to be valid. I do think that for the projects where there
| > are a number of developers doing concurrent development, the git
| > model will be an major improvement over anything that can be done
| > today.
|
| Can you explain why because I do not see this major improvement?
| We can merge, commit locally, sync with MC.
| Even if I have no problem getting rid of a buggy squeak source (even
| 3.0) where I spend too much time
| moving file around and MCfiles based management.
|
| The real problem of MC is that we manage the metadata and ancestry in
| the image and it grows endlessly.

I should amend my statement to include git and svn ... being able to manage projects at the source level where one can create project level branches and have a chance for merging the changes between two arbitrary branches will be a major step forward.

With mcz files the best one can do is manage a version of the Monticello directory, but all of the source is completely opaque.

In the file-base Monticello the meta data is stored in plain files and directory structure and the ancestry is taken care of the the underlying SCM (git or svn). For the FileTree package structure, the ancestry will be kept around only to make it possible to move the package into and out of a file-based repository without losing the ancestry (because it is required for Monticello-level merging).

| >
| > If I hit a blockade along the way, I'll back off and finish up the
| > work planned for the scripting API and 1.0-beta.32 bugs.
| >
| > Personally I am excited at the prospect of being able to use
| > git/github to manage my projects
| >
| > Dale
| >
| > ----- Original Message -----
| > | From: "stephane ducasse" <[hidden email]>
| > | To: [hidden email]
| > | Sent: Saturday, February 4, 2012 5:47:29 AM
| > | Subject: Re: [Metacello] feedback on using JSON to specify
| > | baselines for git repository
| > |
| > | Dale
| > |
| > | I'm lost.
| > | Now my point of view is that I would love to have a 1.0 for
| > | metacello
| > | based on MC so that we
| > | can update the doc and freeze. In addition, I would like to avoid
| > | to
| > | depend on git because tomorrow
| > | people may want something else….
| > |
| > | Stef
| > |
| > |
| > | On Feb 4, 2012, at 2:09 AM, Dale Henrichs wrote:
| > |
| > | > In mapping Metacello to a git-based repository system
| > | > (versioned
| > | > directory structure as opposed to versions of files in a
| > | > directory) the first thing that happens is we no longer need to
| > | > manage versions in the Metacello configuration ... when you
| > | > checkout a directory structure from git the files that are
| > | > present
| > | > are the only version you need to worry about so it's only
| > | > necessary to specify dependencies (project and package). When
| > | > you
| > | > add or remove packages, you will create a new version of the
| > | > baseline spec in the directory and commit the whole enchilada
| > | > ...
| > | >
| > | > If you thought is was overkill to use a class to define all of
| > | > the
| > | > versions for a project, then burning a class per version is
| > | > much
| > | > worse ... I've toyed with some ideas of using doits instead of
| > | > classes, but there is a very real issue that Pharo has a limit
| > | > of
| > | > 255 literals in a method and the Seaside30 configuration will
| > | > easily exceed that limit if I attempted to put the spec in a
| > | > single method/doit ...
| > | >
| > | > I have been threatening to use a structured file format for the
| > | > spec for awhile now and this will probably be a good time to
| > | > make
| > | > the change. Note that besides the method literal limit, there
| > | > are
| > | > problems with using live Smalltalk code to define version
| > | > specs:
| > | >
| > | >  - as it stands now, one cannot afford to reason about a
| > | >  configuration in a site
| > | >    like SqueakSource, because you have to load a package and
| > | >    execute arbitrary
| > | >    Smalltalk code ... a major security violation ...
| > | >
| > | > Today I began playing with the idea of using JSON for the
| > | > specification and you can see a JSON example of the spec
| > | > file[1]
| > | > for the FileTree project on github (yes I'm using git/github
| > | > right
| > | > now). For comparison you can look at the baseline method for
| > | > the
| > | > ConfigurationOfFileTree[2], which has exactly the same
| > | > information.
| > | >
| > | > I think JSON is just as readable as the configuration method
| > | > version. If you want, you can look at the JSON variant for
| > | > Metacello and compare it to a baseline method[4].
| > | >
| > | > Of course, this is all work in progress, but I'd like to have
| > | > early
| > | > feedback if anyone has strong opnions or worries ...
| > | >
| > | > The two metacello.json files that you are looking at were
| > | > generated
| > | > from the relevant configuration, so I think that it should be
| > | > relatively easy to convert existing configurations (and vice
| > | > versa) ...
| > | >
| > | > Dale
| > | >
| > | > [1]
| > | > https://github.com/dalehenrich/filetree/blob/master/metacello.json
| > | > [2]
| > | > https://github.com/dalehenrich/filetree/blob/master/ConfigurationOfFileTree.tree/ConfigurationOfFileTree/baseline100:.st
| > | > [3]
| > | > https://github.com/dalehenrich/metacello/blob/MetacelloGitProject/metacello.json
| > | > [4]
| > | > https://github.com/dalehenrich/metacello/blob/46e6eece660405ee4a527285497b2fe4bfe2e23a/ConfigurationOfGitMetacello.tree/ConfigurationOfGitMetacello/baseline:
| > |
| > |
|
|

Considering that YAML is more readable, I will look into this more tomorrow ...

Today is super bowl sunday and I've got some chicken wings to BBQ:)

Dale

----- Original Message -----
| From: "Tobias Pape" <[hidden email]>
| To: [hidden email]
| Sent: Sunday, February 5, 2012 10:18:39 AM
| Subject: Re: [Metacello] feedback on using JSON to specify baselines for git repository
|
|
| Am 2012-02-05 um 19:06 schrieb Dale Henrichs:
|
| > Tobias,
| >
| > YAML looks a little more compact than JSON and is definitely more
| > readable ... Is there a Smalltalk-base parser for YAML floating
| > around?
|
| A quick search on the Squeaksources says no,
| A googling the same.
|
| However, a yaml-parser is overdue for smalltalk ;)
| And since (yaml includes: json), probably the existing
| json things can be used/extended.
|
| What do  y'all think?
|
| Best
| -Tobias
|
|

I do not get why you want to rely on yet another syntax/parser.
Json looks much more robust than Yaml (imagine that your file is cut for example.
The tree structure of JSON looks to be much better.

Stef

On Feb 5, 2012, at 7:06 PM, Dale Henrichs wrote:

Am 2012-02-05 um 20:57 schrieb stephane ducasse:

> I do not get why you want to rely on yet another syntax/parser.
> Json looks much more robust than Yaml (imagine that your file is cut for example.
> The tree structure of JSON looks to be much better.
>
> Stef

json is a subset of yaml.

the point of configurations IMHO is
that they are
1. human readable
and
2. machine readable,
3. machine generatable.
(ordering intentional)

As far as I can see, the yaml version
has more potential to be human readable.

Best
        -Tobias

> Yes. In the beginning, the git (or SVN) tools that support the merge would be used to do the source level merges. Over time, if it is successful I imagine that image-level for doing the merge would be developed…

does it mean dropping all the merge tools from the image? and the three way merge?

> | can you give one example?
> | Because we want to have a real code representation exchange format
> | for logging changes:
> | having method, and class is not enough.
> | Now we have ring so we could expect that a new logger would log
> | better the actions we are doing
>
> The JSON-based format is for Metacello specifications only.

ah for Metacello so I'm even more confused. So not spec package: requires: []… anymore.
So we will have write metacello spec in JSON?

So in addition the spec in metacello we will have to load a JSON or YAML parser?

> Using a class for ConfigurationOfXXX has been a convenient and expedient mechanism for defining configurations in Metacello, but using executable code to define a specification is not ideal:
>
>  - one cannot extract specification information without executing
>    code is bad from a security point of view
>  - tools have to create classes and generate methods in order to
>    write a specification, again not an ideal situation.

Sure but you do not have to execute code to se the smalltalk syntax as a declarative one.
You do not need a JSON syntax to get a tree structure.

The problem of metacello is not the execution but the fact that it is not used declaratively.
Because
if you have a tree in JSON

[
  {"baseline" : [
                {"common" :
                        [
  {"package" : {
  "name" : "MonticelloFileTree-Core"}},

then you can have
        Baseline named: 'Common'
                addElement:
                        Package named: 'name'

You can use the smalltalk syntax as a declarative one. This is totally orthogonal.
So I really do not understand why JSON is better except that you will need yet another parser.


>
> With the file based SCM systems (SVN and GIT) the Metacello configuration does not have to become the data base for all versions of the project.
DNU

> It is only necessary for Metacello to specify required projects and package dependencies, basically all we need is a baseline specification.

I do not see why.
Because if I want to rely on specific versions or latest dev….


> For a single baseline specification, one would need only one method in the current class-based system so including a class in the mix is basically overkill. I considered using a doit to define the specification, but thought that now would be a good time to go for the structured file to specify the baseline.

you are losing me and I feel immensely sad because I have the impression that this is the revolution in metacello
and that we will have to reinvent the wheel and without a map packages are nothing …
So you killed my energy.

ah that.
But I do not see how Git can provide magic there.
To me git deal with texts or if it does not then it means that the logic can be apply to Metacello specs as well.


> When I started out with Metacello, I imagined that merging would be straightforward source code comparison. Today I don't think so. If you recall last spring Alexandre's style of using Metacello was to create a new project version for each commit. The basic problem wasn't that it was "wrong" to do so, it just made doing any sort of source-based merge impossible.

I do not get it.
You have two spec trees and you merge them. So I do not see why a git textual merge would succeeds and if it succeeds
why a merge based on objects = Metacello spec would not work.



> In order to merge two branches of a Metacello project that involve changes to more than just the same version, one must create two or three version specifications in memory at the same time. For an mcz file, you can keep multiple definitions because the mcz file is a serialized version of the meta data. For metacello, we have to load a class and execute a method for each of the three versions and if the version of the class in the initial image is dirty? What do we do?

I think that git is not the solution to the problem. The solution to the problem is to rethink what is a metacello spec.
Personally I see a metacello spec as a tree of declarations. Now if the three is generated then indeed this is a problem
because you have to execute the program but I do not see why the solution that would solved the problem by using git would not solve the problem without git. I think that the solution looks orthogonal to the software artifact used but may be I'm wrong.

> Now technically there is a solution to this merge problem, but it involves at a minimum creating a completely different file format for configurations, not to mention a whole set of new tools ….

Why not.
Because we have a declarative Smalltalk syntax.
Personnally I hate that I have to define method on a configuration like loadLatest load….
a ConfigurationOf is about data not execution.

> As for SVN and GIT, I think that the jury is still out on whether SVN is better or worse than GIT ... there isn't really a clear winner there.
>
> The bulk of the work that I have and will be doing will be equally applicable to SVN and GIT ... we use SVN within GemStone and I intend to use this new work for managing the GemStone GLASS source in SVN directly rather than versioning a directory of mcz files ....
>
> I believe that SVN and GIT can be used interchangeably especially with the initial versions of my work, since I will be relying on the developers using the SVN/GIT tools for managing their repositories.
>
> On the other hand GitHub has a lot of features that we don't have in SqueakSource that would be real useful to have ... so because of GitHub, I am interested in making a push directed at git.

Ok but I think that relying on an external logic to merge metacello configuration looks like dropping the ball on the floor
while now we could revisit and really simplify metacello and freeze in declaration the configuration.
May be I'm wrong but this is what you want to do with your git approach.

So may be I got it totally wrong but to me. The problem is in the executability of the DSL you built and the fact that you
need to interpret the data to get it. while with a structural spec and a fixed semantics I think that merge should be possible.

> I should amend my statement to include git and svn ... being able to manage projects at the source level where one can create project level branches and have a chance for merging the changes between two arbitrary branches will be a major step forward.
>
> With mcz files the best one can do is manage a version of the Monticello directory, but all of the source is completely opaque.
>
> In the file-base Monticello the meta data is stored in plain files and directory structure and the ancestry is taken care of the the underlying SCM (git or svn). For the FileTree package structure, the ancestry will be kept around only to make it possible to move the package into and out of a file-based repository without losing the ancestry (because it is required for Monticello-level merging).

You lost me again.
I thought you wanted to put only metacello in git.

Do you mean that you would version method as git objects and use the git ancestry management to merge?

Well this also means that we will need yet another parser to load anything in our system while we could use the Smalltalk syntax to represent a tree structure because this is what you want to express.
So I have more and more doubts about Metacello as the tool we need to manage configuration in the future.
I'm getting really sad that I pushed it so much. May be we should take the idea and rebuild it but this was not on my agenda.

Stef

On Feb 5, 2012, at 9:19 PM, Tobias Pape wrote:

Dale Henrichs wrote:
Dale, I understand your excitement, but it seems to me that git should
really be part of Metacello 2.0.  That is, it seems potentially "API"
changing to move away from the executable Smalltalk based Configuration
code.  I urge you to strongly consider the installed usage base and not
alienate users who have gone to the trouble of learning it in that form
- even if it is not ideal.

From my personal perspective as a Metacello user, rather than wait to
hit a blockade in new development I would advise focus on the smaller
(less exciting) task of bringing to completion the current scope of
Metacello 1.0 for inclusion in Pharo 1.4 with git/Metacello 2.0 left for
a later Pharo release.  At a minimum this will help the documentation to
keep up and contribute to an overall sense of stability for the Pharo
environment.

Stef,

1. The JSON parser used by Seaside is one class with 28 methods. I intend to copy
   the class and use it for Metacello, with possible mods to support YAML. For
   the very short term, I will do my "one week experiment" using JSON.

2. I am looking at JSON as a means to get away from the fact that Metacallo does
   not have a non-executable specification format and that is a GIANT SECURITY HOLE.

   I would love for a site like Bibliocello or SqueakSource to reason about Metacello
   versions, but with only an executable format to work with, a server site CANNOT
   create Metacello objects because of the security risks to do so.

   If we have a machine parsable and human readable format, the part of the problem is
   solved.

3. If I want to do true merges of configurations, I must be able to arrange to have 2-3
   versions of the ConfigurationOfXXX class installed in the image at one point in time.
   If I want to preserve any changes that have been made to the original configuration, I
   am in trouble.

   With a non-executable specification for a configuration, I can PARSE 3 different files
   and easily create the 3 instances of MetacelloProject that I need to do the merge.

   For mcz files, I can arrange to embed the JSON representation of the spec in a separate
   zip file directory...

4. In a file-based repository where the entire directory structure is versioned instead of
   the individual files, the class-based configuration does not make a lot of sense. In fact
   I only need a single baseline specification. So I have the choice of continuing with
   the executable specification or creating a parsable representation of the Metacello
   specifications

5. Since JSON has an existing (compact) parser for Smalltalk and JSON is a pretty readable
   format, I am proposing that JSON be used for the parsable Metacello specifications.

6. The parsable format (JSON) and executable format (ConfigurationOfXXX) can be used
   interchangeably. For the forseeable future the vast majority of developers will continue
   to use the executable format to create their specifications. If they happen to be using
   a file-based repository (git or svn), then they will have the option of creating/editting
   the parsable format if they prefer.

Dale


----- Original Message -----
| From: "stephane ducasse" <[hidden email]>
| To: [hidden email]
| Sent: Sunday, February 5, 2012 11:57:37 AM
| Subject: Re: [Metacello] feedback on using JSON to specify baselines for git repository
|
| I do not get why you want to rely on yet another syntax/parser.
| Json looks much more robust than Yaml (imagine that your file is cut
| for example.
| The tree structure of JSON looks to be much better.
|
| Stef
|
| On Feb 5, 2012, at 7:06 PM, Dale Henrichs wrote:
|
| > Tobias,
| >
| > YAML looks a little more compact than JSON and is definitely more
| > readable ... Is there a Smalltalk-base parser for YAML floating
| > around?
| >
| > Dale
| >
| > ----- Original Message -----
| > | From: "Tobias Pape" <[hidden email]>
| > | To: [hidden email]
| > | Sent: Sunday, February 5, 2012 8:57:26 AM
| > | Subject: Re: [Metacello] feedback on using JSON to specify
| > | baselines for git repository
| > |
| > | Dale,
| > |
| > | Am 2012-02-04 um 23:31 schrieb Dale Henrichs:
| > |
| > | > Stef,
| > | >
| > | > The JSON is used to define a single baseline version for the
| > | > packages in a disk-based repository:
| > | >
| > | > [
| > | > {"baseline" : [
| > | > {"common" :
| > | > [
| > | > {"package" : {
| > | > "name" : "MonticelloFileTree-Core"}},
| > | > {"group" : {
| > | > "name" : "default",
| > | > "includes" : ["MonticelloFileTree-Core"]}}]},
| > | > {"gemstone" :
| > | > [
| > | > {"package" : {
| > | > "name" : "MonticelloFileTree-Core",
| > | > "includes" : ["MonticelloFileTree-GemStone-Extensions"]}},
| > | > {"package" : {
| > | > "name" : "MonticelloFileTree-GemStone-Extensions",
| > | > "requires" : ["MonticelloFileTree-Core"]}}]}
| > | > ]}
| > | > ]
| > |
| > | I like the idea, but I think, YAML would be even “nicer”:
| > |
| > | ---
| > | configuration: FileTree
| > | stable: '1.0'
| > | versions:
| > |     - &1-0-baseline
| > |         name: '1.0-baseline'
| > |         timestamp: 2012-01-31T15:19Z
| > |         author: dkh
| > |         common:
| > |             blessing: baseline
| > |             packages:
| > |                 - name: MonticelloFileTree-Core
| > |         gemstone:
| > |             packages:
| > |                 - name: MonticelloFileTree-Core
| > |                   includes:
| > |                         - MonticelloFileTree-GemStone-Extensions
| > |                 - name: MonticelloFileTree-GemStone-Extensions
| > |                   requires:
| > |                         - MonticelloFileTree-Core
| > |     - &1-0
| > |         name: '1.0'
| > |         imports: *1-0-baseline
| > |         timestamp: 2012-02-05T17:40Z
| > |         author: topa
| > |         description: |
| > |             This is just for demonstration.
| > |             Newlines preserved here.
| > |         common:
| > |             blessing: beta
| > |             packages:
| > |                 - name: MonticelloFileTree-Core
| > |                   file: MonticelloFileTree-Core-topa.42
| > |         gemstone:
| > |             packages:
| > |                 - name: MonticelloFileTree-GemStone-Extensions
| > |                   file:
| > |                   MonticelloFileTree-GemStone-Extensions-topa.23
| > |
| > |
| > | YAML (http://yaml.org/) is a superset of json (see
| > | http://yaml.org/spec/1.2/spec.html#id2759572,
| > | http://stackoverflow.com/questions/1726802)
| > |
| > | What do y'all think? I think its a little bit
| > | better readable than pure json ;)
| > |
| > | Best
| > | -Tobias
| > |
| > |
|
|

Stef,

As far as I am concerned the beauty of this whole project is that the majority of the functionality of Monticello will be preserved. Monticello packages will still be defined and used, the Monticello Browser will still be used. etc.

The work that Camillo, Max, and Otto have done is to create an alternate mechanism for storing Monticello packages in a repository.

The mcz format is based on creating a zip repository and serializing the Monticello meta model into that format. Each mcz file contains a different version of the package.

The FileTree (Otto) and FSGit(Camillo and Max) projects simply map the package structure onto a directory structure and use embedded .st files for storing the source code. A single .st file contains the class defition, there are separate .st files for each of the methods, etc. When a package is loaded form a FileTree of FSGit repository, you have an honest to goodness Monticello meta model (snapshot) that can be used like any other snapshot.

The big difference is the ancestry information needs to be handled differently. For git and svn the ancestry information is not used for doing merges since svn and git have their own representation of the history of the repository. So when working strictly with git or svn, the merge tools for those repositories can and should be used, especially when it comes to the Metacello meta model.

When using git and svn one will still be able to use the Monticello 3 way merge code for merging packages ... everything works the same at the level of source code ...

The only remaining work to make git/svn truly usable from Smalltalk is to adapt Metacello so that it can be used to load the projects that are stored in git/svn ...

And that is the work that I am doing right now .... taking a serious look at what changes need to be made to Metacello to make this integration as seamless as possible ...


To move to git/svn does not require a revolution, it can be done in an evolutionary manner ... I intended Metacello to be flexible from the very beginning, since MC2 was on the horizon, so it isn't as big a stretch to incorporate git/svn as I had been thinking only a week ago.

I don't think you should be sad, Stef. I really think that a number of the things that you want for Pharo can be done much more effectively using git/svn than trying to build them from scratch in Smalltalk ...

Dale
----- Original Message -----
| From: "stephane ducasse" <[hidden email]>
| To: [hidden email]
| Sent: Sunday, February 5, 2012 12:19:28 PM
| Subject: Re: [Metacello] feedback on using JSON to specify baselines for git repository
|
| > Yes. In the beginning, the git (or SVN) tools that support the
| > merge would be used to do the source level merges. Over time, if
| > it is successful I imagine that image-level for doing the merge
| > would be developed…
|
| does it mean dropping all the merge tools from the image? and the
| three way merge?
|
| > | can you give one example?
| > | Because we want to have a real code representation exchange
| > | format
| > | for logging changes:
| > | having method, and class is not enough.
| > | Now we have ring so we could expect that a new logger would log
| > | better the actions we are doing
| >
| > The JSON-based format is for Metacello specifications only.
|
| ah for Metacello so I'm even more confused. So not spec package:
| requires: []… anymore.
| So we will have write metacello spec in JSON?
|
| So in addition the spec in metacello we will have to load a JSON or
| YAML parser?
|
| > Using a class for ConfigurationOfXXX has been a convenient and
| > expedient mechanism for defining configurations in Metacello, but
| > using executable code to define a specification is not ideal:
| >
| >  - one cannot extract specification information without executing
| >    code is bad from a security point of view
| >  - tools have to create classes and generate methods in order to
| >    write a specification, again not an ideal situation.
|
| Sure but you do not have to execute code to se the smalltalk syntax
| as a declarative one.
| You do not need a JSON syntax to get a tree structure.
|
| The problem of metacello is not the execution but the fact that it is
| not used declaratively.
| Because
| if you have a tree in JSON
|
| [
|   {"baseline" : [
| {"common" :
| [
|   {"package" : {
|   "name" : "MonticelloFileTree-Core"}},
|
| then you can have
| Baseline named: 'Common'
| addElement:
| Package named: 'name'
|
| You can use the smalltalk syntax as a declarative one. This is
| totally orthogonal.
| So I really do not understand why JSON is better except that you will
| need yet another parser.
|
|
| >
| > With the file based SCM systems (SVN and GIT) the Metacello
| > configuration does not have to become the data base for all
| > versions of the project.
| DNU
|
| > It is only necessary for Metacello to specify required projects and
| > package dependencies, basically all we need is a baseline
| > specification.
|
| I do not see why.
| Because if I want to rely on specific versions or latest dev….
|
|
| > For a single baseline specification, one would need only one method
| > in the current class-based system so including a class in the mix
| > is basically overkill. I considered using a doit to define the
| > specification, but thought that now would be a good time to go for
| > the structured file to specify the baseline.
|
| you are losing me and I feel immensely sad because I have the
| impression that this is the revolution in metacello
| and that we will have to reinvent the wheel and without a map
| packages are nothing …
| So you killed my energy.
|
| >
| > |
| > | That I do not know. For me I do not see problems beside resources
| > | management. I do not see the difference
| > | for the rest because I can merge and the merge works quite well.
| > | What I know is that we technical people are excited by
| > | technology.
| > | People are telling me that git is wonderful
| > | and that I should stop using svn. Well so far Git just alienates
| > | me
| > | and I need a bad UI to manage simple pull requests.
| >
| > You are talking about merging individual mcz files working well and
| > I agree. Monticello is a very nice system for dealing with
| > versioned files.
| >
| > As it stands today, there is no project merge capability. There is
| > really no way for developers to create a branch at the project
| > level, work for hours, days or weeks independently and then come
| > back and merge their configuration changes and package changes. If
| > you figured out which sets of packages changed, you would be able
| > to merge the smalltalk code, but if you had made changes to the
| > baselines and versions there is no merge capability…
|
| ah that.
| But I do not see how Git can provide magic there.
| To me git deal with texts or if it does not then it means that the
| logic can be apply to Metacello specs as well.
|
|
| > When I started out with Metacello, I imagined that merging would be
| > straightforward source code comparison. Today I don't think so. If
| > you recall last spring Alexandre's style of using Metacello was to
| > create a new project version for each commit. The basic problem
| > wasn't that it was "wrong" to do so, it just made doing any sort
| > of source-based merge impossible.
|
| I do not get it.
| You have two spec trees and you merge them. So I do not see why a git
| textual merge would succeeds and if it succeeds
| why a merge based on objects = Metacello spec would not work.
|
|
|
| > In order to merge two branches of a Metacello project that involve
| > changes to more than just the same version, one must create two or
| > three version specifications in memory at the same time. For an
| > mcz file, you can keep multiple definitions because the mcz file
| > is a serialized version of the meta data. For metacello, we have
| > to load a class and execute a method for each of the three
| > versions and if the version of the class in the initial image is
| > dirty? What do we do?
|
| I think that git is not the solution to the problem. The solution to
| the problem is to rethink what is a metacello spec.
| Personally I see a metacello spec as a tree of declarations. Now if
| the three is generated then indeed this is a problem
| because you have to execute the program but I do not see why the
| solution that would solved the problem by using git would not solve
| the problem without git. I think that the solution looks orthogonal
| to the software artifact used but may be I'm wrong.
|
| > Now technically there is a solution to this merge problem, but it
| > involves at a minimum creating a completely different file format
| > for configurations, not to mention a whole set of new tools ….
|
| Why not.
| Because we have a declarative Smalltalk syntax.
| Personnally I hate that I have to define method on a configuration
| like loadLatest load….
| a ConfigurationOf is about data not execution.
|
| > As for SVN and GIT, I think that the jury is still out on whether
| > SVN is better or worse than GIT ... there isn't really a clear
| > winner there.
| >
| > The bulk of the work that I have and will be doing will be equally
| > applicable to SVN and GIT ... we use SVN within GemStone and I
| > intend to use this new work for managing the GemStone GLASS source
| > in SVN directly rather than versioning a directory of mcz files
| > ....
| >
| > I believe that SVN and GIT can be used interchangeably especially
| > with the initial versions of my work, since I will be relying on
| > the developers using the SVN/GIT tools for managing their
| > repositories.
| >
| > On the other hand GitHub has a lot of features that we don't have
| > in SqueakSource that would be real useful to have ... so because
| > of GitHub, I am interested in making a push directed at git.
|
| Ok but I think that relying on an external logic to merge metacello
| configuration looks like dropping the ball on the floor
| while now we could revisit and really simplify metacello and freeze
| in declaration the configuration.
| May be I'm wrong but this is what you want to do with your git
| approach.
|
| So may be I got it totally wrong but to me. The problem is in the
| executability of the DSL you built and the fact that you
| need to interpret the data to get it. while with a structural spec
| and a fixed semantics I think that merge should be possible.
|
| > I should amend my statement to include git and svn ... being able
| > to manage projects at the source level where one can create
| > project level branches and have a chance for merging the changes
| > between two arbitrary branches will be a major step forward.
| >
| > With mcz files the best one can do is manage a version of the
| > Monticello directory, but all of the source is completely opaque.
| >
| > In the file-base Monticello the meta data is stored in plain files
| > and directory structure and the ancestry is taken care of the the
| > underlying SCM (git or svn). For the FileTree package structure,
| > the ancestry will be kept around only to make it possible to move
| > the package into and out of a file-based repository without losing
| > the ancestry (because it is required for Monticello-level
| > merging).
|
| You lost me again.
| I thought you wanted to put only metacello in git.
|
| Do you mean that you would version method as git objects and use the
| git ancestry management to merge?
|
|
|
|

On Feb 6, 2012, at 6:47 PM, Dale Henrichs wrote:

I still do not understand what you mean.

#( #baseline
        #( projectVersion '3.6' )
        #( url 'http://www.'))

Why this is not enough?
I do not get it.

Then the smalltalk message syntax is excellent to represent declarative structure

(Node
        name: #baseline
        son: (Node
                        name: 'ProjectVersion')

I still do not get why you need a different syntax

To me this is totally orthogonal to git usage or not. It depends on the specification of your inputs.
And this is why I said that the problem is the fact that you need to execute method to get objects
but not the technology used.

> 4. In a file-based repository where the entire directory structure is versioned instead of
>   the individual files, the class-based configuration does not make a lot of sense. In fact
>   I only need a single baseline specification. So I have the choice of continuing with
>   the executable specification or creating a parsable representation of the Metacello
>   specifications

As I said experience with a really declarative specification but in plain smalltalk syntax
Look at VisualiWorks UI specs this is exactly the same problem (see at the end of this mail)

Look at MSE format for Moose this is a declarative spec for model of source code.
Ask camillo because he is expert in Git and SCM and we discuss and he agreed with me. :)

> 5. Since JSON has an existing (compact) parser for Smalltalk and JSON is a pretty readable
>   format, I am proposing that JSON be used for the parsable Metacello specifications.
>
> 6. The parsable format (JSON) and executable format (ConfigurationOfXXX) can be used
>   interchangeably. For the forseeable future the vast majority of developers will continue
>   to use the executable format to create their specifications.

I hope not. It would be good to have one format for all.

> If they happen to be using
>   a file-based repository (git or svn), then they will have the option of creating/editting
>   the parsable format if they prefer.
>
> Dale
>
>
>
windowSpec
        "UIPainter new openOnClass: self andSelector: #windowSpec"

        <resource: #canvas>
        ^#(#{UI.FullSpec}
                #window:
                #(#{UI.WindowSpec}
                        #label: #(#{Kernel.UserMessage} #key: #UnlabeledCanvas #defaultString: 'Unlabeled Canvas' #catalogID: #labels)
                        #bounds: #(#{Graphics.Rectangle} 512 384 858 635 ) )
                #component:
                #(#{UI.SpecCollection}
                        #collection: #(
                                #(#{UI.TextEditorSpec}
                                        #layout: #(#{Graphics.LayoutFrame} 10 0 10 0 -10 1 -10 1 )
                                        #name: #textEditor
                                        #model: #textHolder
                                        #isReadOnly: true
                                        #tabRequiresControl: true ) ) ) )

On Feb 6, 2012, at 7:12 PM, Dale Henrichs wrote:

> Stef,
>
> As far as I am concerned the beauty of this whole project is that the majority of the functionality of Monticello will be preserved. Monticello packages will still be defined and used, the Monticello Browser will still be used. etc.
>
> The work that Camillo, Max, and Otto have done is to create an alternate mechanism for storing Monticello packages in a repository.

Yes I discussed with camillo: mapping on method per files and protocol for folder.

> The mcz format is based on creating a zip repository and serializing the Monticello meta model into that format. Each mcz file contains a different version of the package.

Yes
>
> The FileTree (Otto) and FSGit(Camillo and Max) projects simply map the package structure onto a directory structure and use embedded .st files for storing the source code. A single .st file contains the class defition, there are separate .st files for each of the methods, etc. When a package is loaded form a FileTree of FSGit repository, you have an honest to goodness Monticello meta model (snapshot) that can be used like any other snapshot.

Yes I asked camillo during lunch about it.

> The big difference is the ancestry information needs to be handled differently. For git and svn the ancestry information is not used for doing merges since svn and git have their own representation of the history of the repository. So when working strictly with git or svn, the merge tools for those repositories can and should be used, especially when it comes to the Metacello meta model.

Yes so this means no merging tool in Smalltalk anymore. Just plain unix one.

> When using git and svn one will still be able to use the Monticello 3 way merge code for merging packages ... everything works the same at the level of source code ...

Yes but I'm not sure because people will use Unix/github tools and may be this is better but it means that we
cannot change the behavior if we do not like it.

> The only remaining work to make git/svn truly usable from Smalltalk is to adapt Metacello so that it can be used to load the projects that are stored in git/svn ...
>
> And that is the work that I am doing right now .... taking a serious look at what changes need to be made to Metacello to make this integration as seamless as possible ...
>
> To move to git/svn does not require a revolution, it can be done in an evolutionary manner ... I intended Metacello to be flexible from the very beginning, since MC2 was on the horizon, so it isn't as big a stretch to incorporate git/svn as I had been thinking only a week ago.
>
> I don't think you should be sad, Stef. I really think that a number of the things that you want for Pharo can be done much more effectively using git/svn than trying to build them from scratch in Smalltalk …

I know. Now I would like to get some stable ground. Because we will put in production (we got an engineer and a postdoc working) on jenkins rules checking and distribution construction and for that we need a robust metacello. This is why I'm thinking that having a declarative syntax and a fixed semantics is probably a good move to have.
I guess that now we got nearly all the case and I do not understand why Metacello configuration could not handle as pure data on class side or in the manifest of a specific package (This is what we are doing for rule checking positive): a package will have a manifest class with only meta data.

Now people piss on monticello regularly but when monticello arrived it was one the first scm doing three way merge.
And instead of pissing on it, people should have improve it. It is boring to hear MC is bad after a while.

A real problem with MC is
        - the ancestor history is stored in the image 4mb now
        - does not handle well non code elements.

Now even if I prefer the git model over SVN, Git is too arcane for me. It adds a good level of complexity and I'm not blind.
I'm ok to learn it too. But having tool in your language is also quite cool.

Stef