Contributing to Pharo

We have the catalog. So how different it is.

And yes we would to change packages in nautilus.

Haha, I certainly understand the lack of time/manpower ... Just in case it wasn't clear, when I said "project browser" I wasn't talking about replacing the code browser (Nautilus), but replacing/augmenting the Monticello Browser - which should turn out to be a much simpler undertaking ---  "Project Browser" that manages the projects within the image - load, save, changes, etc.:

Dale

On 1/26/16 11:29 PM, Christophe Demarey wrote:

Hi Dale,

Le 27 janv. 2016 à 04:11, Dale Henrichs a écrit :

I've been wondering no-one in the Pharo community has started building a "Project Browser" to replace the "Package Browser" .... I've had a "project browser" in tODE for several years now and you work with a manageble list of Metacello projects like the following:

Of course, we have that in mind. We would love to have it but lack of time / manpower ...
We are not happy with the current browser (Nautilus). We could have something really more powerful. It will come but we have other projects to achieve before ;)
If someone comes with a better browser, we would love to set it as default.

Cheers,
Christophe

We have the catalog. So how different it is.

AFAIK the catalog only manages loads of projects ... to make it a  "Project Browser" you would add menu items for committing/saving projects (i.e., save all dirty packages and for git repos, do a git commit after updating packages); viewing the diffs for all packages in a project; and of course the load functionality ....

As I said somewhere else, I don't think that it is a major job to implement a "Project Browser", but it is something that really needs to be done for supporting git-based projects. There is a natural 1-to-1 mapping between a Metacello project and it's git repository and as you can see in the Git menu below, there are a number of additional git-specific functions that need to be supported ...

Dale

And yes we would to change packages in nautilus.

Le 27/1/16 15:13, Dale Henrichs a écrit :

Haha, I certainly understand the lack of time/manpower ... Just in case it wasn't clear, when I said "project browser" I wasn't talking about replacing the code browser (Nautilus), but replacing/augmenting the Monticello Browser - which should turn out to be a much simpler undertaking ---  "Project Browser" that manages the projects within the image - load, save, changes, etc.:

Dale

On 1/26/16 11:29 PM, Christophe Demarey wrote:

Hi Dale,

Le 27 janv. 2016 à 04:11, Dale Henrichs a écrit :

I've been wondering no-one in the Pharo community has started building a "Project Browser" to replace the "Package Browser" .... I've had a "project browser" in tODE for several years now and you work with a manageble list of Metacello projects like the following:

Of course, we have that in mind. We would love to have it but lack of time / manpower ...
We are not happy with the current browser (Nautilus). We could have something really more powerful. It will come but we have other projects to achieve before ;)
If someone comes with a better browser, we would love to set it as default.

Cheers,
Christophe

I think that this is important that we do not think only in terms of git.
People will jump on the next cool and better because nicer versioning system.
I think about fossil for example. I'm waiting for that. Because each time I pair program with smart git users
I see how this is "simple". So I would really be against having git menu in Pharo.
We should have an abstraction over the generic commands that a decentralized versioning system should
offer.

Stef

On 1/27/16 8:30 AM, stepharo wrote:

We have the catalog. So how different it is.

AFAIK the catalog only manages loads of projects ... to make it a  "Project Browser" you would add menu items for committing/saving projects (i.e., save all dirty packages and for git repos, do a git commit after updating packages); viewing the diffs for all packages in a project; and of course the load functionality ....

As I said somewhere else, I don't think that it is a major job to implement a "Project Browser", but it is something that really needs to be done for supporting git-based projects. There is a natural 1-to-1 mapping between a Metacello project and it's git repository and as you can see in the Git menu below, there are a number of additional git-specific functions that need to be supported ...

Dale

And yes we would to change packages in nautilus.

Le 27/1/16 15:13, Dale Henrichs a écrit :

Haha, I certainly understand the lack of time/manpower ... Just in case it wasn't clear, when I said "project browser" I wasn't talking about replacing the code browser (Nautilus), but replacing/augmenting the Monticello Browser - which should turn out to be a much simpler undertaking ---  "Project Browser" that manages the projects within the image - load, save, changes, etc.:

Dale

On 1/26/16 11:29 PM, Christophe Demarey wrote:

Hi Dale,

Le 27 janv. 2016 à 04:11, Dale Henrichs a écrit :

I've been wondering no-one in the Pharo community has started building a "Project Browser" to replace the "Package Browser" .... I've had a "project browser" in tODE for several years now and you work with a manageble list of Metacello projects like the following:

Of course, we have that in mind. We would love to have it but lack of time / manpower ...
We are not happy with the current browser (Nautilus). We could have something really more powerful. It will come but we have other projects to achieve before ;)
If someone comes with a better browser, we would love to set it as default.

Cheers,
Christophe

Totally agree with having an abstraction layer or common set of functions ...

As I've said before, I talk about git mainly because of github and because there needs to be a "first" and I chose git ...

Other than details, I think that most if not all of the information I plan to share is "disk-based SCM neutral" and applies equally to each SCM ....

Dale

I think that this is important that we do not think only in terms of git.
People will jump on the next cool and better because nicer versioning system.
I think about fossil for example. I'm waiting for that. Because each time I pair program with smart git users
I see how this is "simple". So I would really be against having git menu in Pharo.
We should have an abstraction over the generic commands that a decentralized versioning system should
offer.

Stef

Le 27/1/16 18:23, Dale Henrichs a écrit :

On 1/27/16 8:30 AM, stepharo wrote:

We have the catalog. So how different it is.

AFAIK the catalog only manages loads of projects ... to make it a  "Project Browser" you would add menu items for committing/saving projects (i.e., save all dirty packages and for git repos, do a git commit after updating packages); viewing the diffs for all packages in a project; and of course the load functionality ....

As I said somewhere else, I don't think that it is a major job to implement a "Project Browser", but it is something that really needs to be done for supporting git-based projects. There is a natural 1-to-1 mapping between a Metacello project and it's git repository and as you can see in the Git menu below, there are a number of additional git-specific functions that need to be supported ...

Dale

And yes we would to change packages in nautilus.

Le 27/1/16 15:13, Dale Henrichs a écrit :

Haha, I certainly understand the lack of time/manpower ... Just in case it wasn't clear, when I said "project browser" I wasn't talking about replacing the code browser (Nautilus), but replacing/augmenting the Monticello Browser - which should turn out to be a much simpler undertaking ---  "Project Browser" that manages the projects within the image - load, save, changes, etc.:

Dale

On 1/26/16 11:29 PM, Christophe Demarey wrote:

Hi Dale,

Le 27 janv. 2016 à 04:11, Dale Henrichs a écrit :

I've been wondering no-one in the Pharo community has started building a "Project Browser" to replace the "Package Browser" .... I've had a "project browser" in tODE for several years now and you work with a manageble list of Metacello projects like the following:

Of course, we have that in mind. We would love to have it but lack of time / manpower ...
We are not happy with the current browser (Nautilus). We could have something really more powerful. It will come but we have other projects to achieve before ;)
If someone comes with a better browser, we would love to set it as default.

Cheers,
Christophe

2016-01-27 17:30 GMT-03:00 Dale Henrichs <[hidden email]>:
>
> Totally agree with having an abstraction layer or common set of functions ...
>
> As I've said before, I talk about git mainly because of github and because there needs to be a "first" and I chose git ...
>
> Other than details, I think that most if not all of the information I plan to share is "disk-based SCM neutral" and applies equally to each SCM ....

AFAIU FileTree has that purpose, although I don't know if it is used
with something other than Git as a SCM.


Esteban A. Maringolo

So we are in sync then!

Totally agree with having an abstraction layer or common set of functions ...

As I've said before, I talk about git mainly because of github and because there needs to be a "first" and I chose git ...

Other than details, I think that most if not all of the information I plan to share is "disk-based SCM neutral" and applies equally to each SCM ....

Dale

On 01/27/2016 11:47 AM, stepharo wrote:

I think that this is important that we do not think only in terms of git.
People will jump on the next cool and better because nicer versioning system.
I think about fossil for example. I'm waiting for that. Because each time I pair program with smart git users
I see how this is "simple". So I would really be against having git menu in Pharo.
We should have an abstraction over the generic commands that a decentralized versioning system should
offer.

Stef

Le 27/1/16 18:23, Dale Henrichs a écrit :

On 1/27/16 8:30 AM, stepharo wrote:

We have the catalog. So how different it is.

AFAIK the catalog only manages loads of projects ... to make it a  "Project Browser" you would add menu items for committing/saving projects (i.e., save all dirty packages and for git repos, do a git commit after updating packages); viewing the diffs for all packages in a project; and of course the load functionality ....

As I said somewhere else, I don't think that it is a major job to implement a "Project Browser", but it is something that really needs to be done for supporting git-based projects. There is a natural 1-to-1 mapping between a Metacello project and it's git repository and as you can see in the Git menu below, there are a number of additional git-specific functions that need to be supported ...

Dale

And yes we would to change packages in nautilus.

Le 27/1/16 15:13, Dale Henrichs a écrit :

Haha, I certainly understand the lack of time/manpower ... Just in case it wasn't clear, when I said "project browser" I wasn't talking about replacing the code browser (Nautilus), but replacing/augmenting the Monticello Browser - which should turn out to be a much simpler undertaking ---  "Project Browser" that manages the projects within the image - load, save, changes, etc.:

Dale

On 1/26/16 11:29 PM, Christophe Demarey wrote:

Hi Dale,

Le 27 janv. 2016 à 04:11, Dale Henrichs a écrit :

I've been wondering no-one in the Pharo community has started building a "Project Browser" to replace the "Package Browser" .... I've had a "project browser" in tODE for several years now and you work with a manageble list of Metacello projects like the following:

Of course, we have that in mind. We would love to have it but lack of time / manpower ...
We are not happy with the current browser (Nautilus). We could have something really more powerful. It will come but we have other projects to achieve before ;)
If someone comes with a better browser, we would love to set it as default.

Cheers,
Christophe

Esteban,

FileTree is indeed neutral with regards to SCM, but Stef and I were
referring to a smalltalk api for managing the git (or other repos) in
terms of doing commits, checkouts and the like ... FileTree itself isn't
involved ...

Dale

On 01/27/2016 12:47 PM, Esteban A. Maringolo wrote:

I read too fast.
But my comment is still true.
We also want a new version browser. Monticello browser is too tied to ... Monticello. We would like a better approach for both git and monticello.
Esteban did a prototype for himself but he has no time to work on it, sadly

Le 27 janv. 2016 à 15:13, Dale Henrichs a écrit :

> On 28 Jan 2016, at 09:23, Christophe Demarey <[hidden email]> wrote:
>
> I read too fast.
> But my comment is still true.
> We also want a new version browser. Monticello browser is too tied to ... Monticello. We would like a better approach for both git and monticello.

It is totally wrong to put Git and Monticello at the same level, they are related but different things.

Monticello models our code and your changes to it. It also has a way of putting a package into a persistent representation that can be copied around. An MC server is nothing more than storage.

Filetree is another way of doing that, using a different representation, but it still needs MC for the high level stuff. Git is versioned storage. And there is a conflict there, yes.

But throwing away the MC code and changes model would leave you with file outs. You do not want to go there, really. Never, ever.

Le 28 janv. 2016 à 09:34, Sven Van Caekenberghe a écrit :
I fully agree.
The problem is that Monticello does too much things: code representation, versioning, loading of the code and call of #initialize methods, etc.
We want to keep a code representation but the question that comes: is it the better code representation? We also have ring, ficus (hierarchical model on top of ring). For code representation, we would like to represent code (from packages to method source code) for multiple versions, in a hierarchical way (to be able to query easily), potentially coming from a remote environment.
We also need to split the loading part (loading from the code representation) and the serialization / deserialization part.

At the end, we want an uniform way to represent code (not SCM related), a common API for versioning and different backends (git, Monticello). By the way, we can add the storage format: there are already different versions of the filetree format (with metadata, without, one file per method, one file per class). At some point, we need to handle this point of variability.

Excellent summary. Lots of work ;-)

On 01/28/2016 12:34 AM, Sven Van Caekenberghe wrote:
Sven,

If you look at my presentation on "Practical Git for Smalltalk"[1] from
2012, I take pains to differentiate between the Monticello package model
(definitions and snapshots) and the Monticello repository model (mcz files).

Noone is proposing to throw away the Monticello package model (to my
knowledge).

So in these discussions about supporting git we are talking about a
model where Filetree is still being used for Monticello packages on top
of a git storage strata ...

I have brought up the point about a "Metacello Project Browser" because
both git-based and mcz-based projects can benefit from a higher level of
management:

   - commit all dirty packages, then update and commit the
ConfigurationOf for mcz-based projects, OR
   - save all dirty packages and perform a git commit for git-based projects

   - perform a diff on all dirty packages in a project ...

   - etc.

Dale


[1]
http://www.stic.st/wp-content/conferences/2012/Wednesday/1415-Practical_Git_for_Smalltalk-Henrichs.pdf

Hi Dale,

2016-01-27 15:41 GMT+01:00 Dale Henrichs <[hidden email]>:

On 1/27/16 12:03 AM, Thierry Goubier wrote:

Hi Dale,

then a question: how does one query a baselineOf or a configOf to retrieve the current project state (i.e. packages) loaded in the image?

Thierry

Thierry,

Good question:)

To start with in tODE I have a class TDMetacelloProjectRegistrationDefinition which is part of the following hierarchy:

Object
 TDDefinition
  TDPackageBasedDefinition
   TDAbstractProjectRegistrationDefinition
    TDGitProjectRegistrationDefinition
    TDMetacelloProjectRegistrationDefinition
     TDMetacelloRegistrationDefinition
      TDMetacelloBaselineRegistrationDefinition
      TDMetacelloConfigurationRegistrationDefinition
      TDMetacelloHybridRegistrationDefinition
     TDMetacelloUnloadedProjectRegistrationDefinition

The method TDMetacelloProjectRegistrationDefinition>>workingCopies[1] calculates the list of MCWorkingCopy instances for the loaded packages associated with a project (including the configurationof or baselineof... or both).

The key Metacello methods called include:

  MetacelloProjectRegistration>>projectVersion[2]
  MetacelloMCVersion>>packages[3]
  MetacelloPackageSpec>>workingCopy[4]
  MetacelloMCProjectSpec>>projectPackage[5]

This is really the key information for doing all of the other operations for a "Project Browser" like "save all dirty packages and commit" and "give me diffs for all dirty packages in project"...

I was just asking that because I'd like to see that information merged as much as possible; separating concepts just strikes me as unneeded complexity, at least for baselines.

Not sure what you mean by "separating concepts and unneeded complexity". I provided a picture of some of the abstractions that I am using in tODE ... these abstractions are used in a number of commands, but I also tried to provide you with the basic implementation used in tODE in case you were interested in the fact that tODE includes the BaselineOf package in the list of packages for a project...

The simple answer is MetacelloMCVersion>>packages[3]

Configurations are a bit different, since a configuration can snapshot multiple versions. But I'd really like to make it simple and obvious how to setup a project.

I'm not sure what you are talking about here.

In Metacello a BaselineOf has a single version and the MetacelloMCVersion class manages the version ... Yes, a Configuration has multiple versions and each of those versions is an instance of MetacelloMCVersion....

Perhaps when you have time, you can provide a bit more information about what you are trying to do ...

I apologize if I provided too much information but when working with folks in different time zones, I am in the habit of packing as much information in a message as possible, because it's often a 24 hour turnaround for responses... (as it is in this case:)

Dale

Le 28/01/2016 20:55, Dale Henrichs a écrit :
This is me trying to express something and being unable to make it clear
:) I'll have something at one point later. Or maybe not. Doesn't really
matter; I'll wait until it becomes clearer to me.

> The simple answer is MetacelloMCVersion>>packages[3]

Thanks. I'll try something.

I was just looking at it from the angle of having to explain it to one
of my students (non-smalltalkers)... The simplest path to creating a
project.

I guess one must have a tool like Versionner.

> I apologize if I provided too much information but when working with
> folks in different time zones, I am in the habit of packing as much
> information in a message as possible, because it's often a 24 hour
> turnaround for responses... (as it is in this case:)

Don't worry about that, you already provided a lot, certainly enough for
what I was looking for: a way to organize groups of packages in the
image out of the Metacello baselines and configurations present, instead
of hacking stuff based on catalog keywords and package name prefix.

Now, it's time to implement... If I find that time :(

Thierry

On 01/28/2016 12:55 AM, Christophe Demarey wrote: Agreed ....
> By the way, we can add the storage format: there are already different versions of the filetree format (with metadata, without, one file per method, one file per class). At some point, we need to handle this point of variability.
On this point I think that we will always want to support multiple disk
formats .... in a perfect world, there is no reason not to have one file
per method and no Monticello meta data ... but perfection does not exist.

Monticello meta data was included in FileTree because it was needed over
the last 4 years so that developers could dip there toes into using git
without completely abandoning Monticello .... Now that we are gaining a
critical mass of users and the tool support is starting to show up,
dropping the meta data is a practical thing to do ... but there is code
committed on github that hasn't been updated in years that is still
being used and Monticello meta data is still present, so it is necessary
to "support the older formats".

The one file per method was chosen for two reasons.

The first reason has to do with the fact that disk based SCMs are
designed manage files as atomic units and in Smalltalk the method is the
atomic unit for source code ... it is very convenient to be able to
leverage native SCN files for a) versioning files and b) maintaining
version history for a method as it moves from class to class ...

The second reason has to do with the fact that there is no cross
platform fileout format for Smalltalk. Chunk format comes is "common"
but chunk format also relies on executing Smalltalk code to define
classes and there hasn't been a standard class creation method since the
second Smalltalk implementation showed up on the planet ... Technically,
class and method definitions can be derived from a file per class
approach, but without a standardized file format you end up with each
dialect isolating itself from the others and this is not desirable nor
necessary ... It just takes more work to come up with a file format that
can accommodate the different needs of the different dialects .... the
one method per file and a generic properties file for class
definitions[1] and the provision for allowing additional platform
specific property files and directories (like monticello meta data)
simplified the whole process ...

So I would like changing the disk format for sharing source code to be a
multi-platform effort - which means that format that is chosen is
flexible enough to allow dialects to include the meta data that they
need ... It is important to note that there has never been a guarantees
that all platforms would PRESERVE and WRITE foreign meta data ... the
only provision that exists is that ALL platforms can READ the format
while ignoring anything it doesn't recognize ....

Dale

[1]
https://raw.githubusercontent.com/CampSmalltalk/Cypress/master/img/CypressStructure-STIC2012.png

On 01/28/2016 12:16 PM, Thierry Goubier wrote:  From this perspective you should look at the MetacelloToolBox and
specifically MetacelloToolBox
class>>createBaselineOf:requiredProjects:packages:repositories:dependencies:groups:.


Versionner uses the MetacelloToolBox api and about 3 months ago I added
the basic support for creating a BaselineOf as I added a `project new`
command to tODE ... I would be glad to expand the MetacelloToolBox api
for BaselineOf if folks ask for specific functionality ...

The MetacelloToolBox is meant to be the common point for
programmatically creating and updating ConfigurationOf and BaselineOf ...

Dale

Hi Dale,

I am trying to understand this a little better. If a package containing metadata would be changed using a dialect which cannot interpret the metadata, wouldn’t or at least couldn’t it be broken or lost afterwards for a dialect which tries to interpret the metadata? At least if my assumption is correct that the metadata is related to the code in some way?

Bernhard

On 01/29/2016 09:02 AM, Bernhard Pieber wrote:
> Hi Dale,
>
> I am trying to understand this a little better. If a package containing metadata would be changed using a dialect which cannot interpret the metadata, wouldn’t or at least couldn’t it be broken or lost afterwards for a dialect which tries to interpret the metadata? At least if my assumption is correct that the metadata is related to the code in some way?
>
Bernhard,

Very good question.

You are absolutely correct, the act of writing a new version of a
package using a different dialect would indeed destroy the
platform-specific meta data.

However, it is NOT expected that two different dialects (with different
disk formats)  share the same commit.

If dialect A has committed a package, then dialect B is expected to be
able to read the package written by dialect A. Being able to read the
package means that all of the information that is common between the two
dialects will be preserved ... in FileTree, this means that:

   - all of the class and instance method source is readable and shared
   - the method category is readable and shared
   - the name, superclass, instance variables, class instance variables,
     pools, class category, and class vars for classes is readable and
shared

This is an awful lot common data .... things like traits or namespaces
are dialect specific and are ignored by dialects that don't have them
when reading the package ...

When dialect B has read, loaded/compiled and tested the code in the
package, the developer has a couple of choices to be made when writing
out a new package with her changes:

   - use the current branch
   - use a new branch

I think that using a new branch is the cleanest option. When I port a
project like Zinc[1] to GemStone[2], the master branch is preserved for
the Pharo-specific code. I create a gs_master branch where the GemStone
code goes. When Sven commits new code in his repo, I merge the changes
from the his new master branch into my gs_master and resolve conflicts
if any, run tests and I'm done.

The vast mjority of the code base is shared between GemStone and Pharo
and only the places where I made changes in porting Zinc to GemStone
have the potential for conflicts. Running the tests should highlight
any  impacts not covered by direct conflicts and if there were some
Pharo-specific meta data that I deleted on my branch that causes a test
to fail then that is up to me to research and fix.

When I have changes to feed back to Sven, I make those changes on a
separate "topic branch"[3] that is merged into gs_master for my use. I
then cherry-pick[4] the topic branch commit into a separate topic branch
off of the master branch (this picks up only the changes I made on the
topic branch) and open a pull request[5] against Svens repository ... if
Sven were using travis-ci, tests would be run automatically against a
merge of my changes into his master branch ... if the tests are green
and the code passes Sven's scrutiny in a code review he merges my
proposed changes into his master branch...

GemStone and Pharo share a common disk format (not 100% common), but
since git only merges the deltas for a commit, it is relatively easy to
keep the meta data differences isolated while still sharing the vast
bulk of the code and classes ....

Dale

[1] https://github.com/svenvc/zinc
[2] https://github.com/GsDevKit/zinc
[3] https://git-scm.com/book/en/v2/Git-Branching-Branching-Workflows
[4]
http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html
[5] https://help.github.com/articles/using-pull-requests/

Thanks Dale for all the explanations.

How Monticello and version control relate in the big picture is starting to make sense for me.

Now, I better understand why filetree ended up uses a file-per-method format, even though that is relatively hostile to git user interfaces optimised for other languages. There is really a need for a file-per-class exchange format, because that would works a lot better with the existing VCS ecosystem.

I think more package-based user interfaces would indeed be a very good idea, for browsing and for source code management.

Stef, I have the impression you think that git is popular because it is a new shiny toy. I disagree with this idea. Git is a typical worse-is-better tool. It's good enough for most people, but it still has many shortcomings. It is popular in spite of its shortcomings. It became popular as destination for projects shifting from CVS and Subversion. So it is unlikely to be displaced by a newer, incrementally shinier tools. Anything that will displace it will have to provide an improvement of a similar magnitude as the jump between centralised and distributed version control.

Still, I think it's a good idea not to restrict high level models to what git provides if that's a less than ideal fit to the image model.

I have a lot of ideas to improve browsing and source code management in Pharo. I can make no promises, but I would like to produce something there.