Released versions in Pharo Bootstrap

Guillermo Polito <[hidden email]>
wrote:
>...
>
> So yes, it may block upgrades, but until we have tools that allow us to
> cope with the complexity, I prefer to have reproducible versions where I
> can reproduce bugs in a reproducible way than an unpredictable version
> where I cannot grasp the cause of a problem :/.

I strongly advice against using Baselines for that. They are supposed to be
edited from hand, and using them for reproducible versions will break other
projects, as we've seen earlier with Configurations. AFAIK Metacello can
register which versions were loaded in which order, so that information
should be recorded for reproducibility.

Stephan

Cyril,

For development I tend to use Metacello locking instead of hard-wired dependencies in the BaselineOf and that works very well --- it completely avoids the need to edit a baseline for development purposes and this approach works really well for me ... perhaps we can discuss this in more detail in a separate thread or even private email? This all seems to be more complicated for you guys than has been my experience and of course you guys _appear_ to be completely ignoring the Metacello locking feature so I'm curious why ...

Dale

On Mon, Mar 5, 2018 at 4:51 PM, Guillermo Polito <[hidden email]> wrote:

I still do not understand... Also I do not understand your usage of the term "the merge". Can somebody give a **concrete** scenario? 

I'll try

For example the project MaterialDesignLite have two branches that are always here:

- master 

- development

Each commit on master is a stable release and end up with a tag as "v1.2.2". Since it's stable, BaselineOfMaterialDesignLite should depend only on fixed version of the dependencies. For example it should depend on MaterialDesignColor "v1.0.0".

On the branch dev, we want to get the patches and possibly the minor versions of the dependencies automatically. In the baseline we then want to depend on MaterialDesignColor "v1.x.x". Thus, we follow the changes of MaterialDesignColor while it's not a major release.

Because of this situation, BaselineOfMaterialDesignLite is different on the two branches. Later, if I want to merge development into master in order to release a new version, master will get the BaselineOfMaterialDesignLite with semantic versionning dependencies instead of the fixed dependencies. Before the release I'll need to change the Baseline to get fix dependencies once again. 

I hope I was clearer. :)

 

When you release a version, please do not move that version. You should then create new versions, with new numbers and new code. But never touch old versions with old numbers and old code. Like that I can download the same old code using the same old number to get the old version whenever I want :).

You can try to do it with branches, tags, different repositories, or even with zipfiles in mails. I don't care. Just don't modify releases and I'm happy with it.

 

--

Cyril Ferlicot
https://ferlicot.fr

Denis Kudriashov <[hidden email]>
wrote:
>
> It is clear that we need both baselines for that case.
> But because we will move Calypso to Pharo repo I think it is easy to manage
> strong versions directly in Pharo loading script for now (in #loadCalypso
> method).

Easy for now, yes. And also a bad example for others to follow. And
something that doesn't scale. Where is the increased modularity going to
come from if we still manage everything as a monorepo?

Stephan

On 03/05/2018 08:48 AM, Stephan Eggermont wrote: Stephan,

For this approach to work it is important that people not make a
practice of committing api-breaking changes and by definition each new
release of pharo breaks the fundamental api ...

I've been able to survive in this universe with Metacello over the
years, because I make a point of ensuring that the latest version of
Metacello works on all supported platform versions and my observation
has been that most Pharo projects ignore older versions of pharo for as
long as they can get away with it and this becomes the source of so many
issues I've seen ...

With github and travis-ci it is practical to support projects on
multiple platform versions ... if you want to ... this approach means
that when you move to a new platform version you have to use packaging
to isolate your version-specific changes and you probably have to
refactor your implementation to be able to manage these version-specific
changes ... only when you make api-breaking changes is it necessary to
start new branches or use a series of tags ...

Dale

On 03/05/2018 09:07 AM, Guillermo Polito wrote:
> On the other side, there is the fact that Metacello baselines are so
> far nice to describe release project dependencies, but they are not so
> nice to describe subprojects/development dependencies that may get
> edited along with the parent project. Kind of what we used to do with
> #bleedingEdge. I feel this is a complex problem, that not even SBT or
> maven that are there since years are capable of solving nicely... Tode
> and Iceberg metacello integration try to solve this problem by
> "ignoring the dependency and using the loaded repository" but this may
> not be successful either...

I have not been following this thread in detail, but this comment leads
me to believe that the issue that you guys are running into has to do
with trying to ongoing distributed development across multiple projects
and multiple platform versions ...

If so then I think that the moral equivalent of #bleedingEdge should be
a branch structure along the lines of:

   master branch --- the project release that is known to work on all
supported platform versions.
   dev_pharo6      --- the current #bleedingEdge for Pharo6.0
   dev_pharo7      --- the current #bleedingEdge for Pharo7.0

In an image where you want to use the dev_pharo7 for a project, you do a
Metacello lock on github://xxxx/xxx:dev_pharo7/src BEFORE loading the
project ... if there are multiple projects that need to all coordinate
development then you follow the same convention and use a Metacello lock
for each of the projects ... Executing expressions to lock projects is
tedious to use and manage.

In tODE I started using project entries[1] that are downloaded to disk
and shared by multiple images that allow a developer to specify what
branch/tag/commit they are interested in using ... each image that is
started up in a "cluster" uses the project entry to do a Metacello lock
on the project "automagically"... if there is a series of projects where
the developer needs to use dev_pharo6 they can arrange to edit the
project entry to reflect the proper branches for the set of projects
that need special _development_ treatment ... these .ston files can be
shared amongst groups of developers ... of course once the local clones
of these projects are on the local disk, then it is up to the developer
(or a set of scripts) to periodically update to the latest development
version of each of the projects ...

To share a specific configuration of development git repos, you need to
know that SHA anyway, so the image can produce a collection of project
entries with the SHA of the commit and those project entries can be
included in a bug report ...

For Rowan[2], I am using a "load specification"[3], which is a second
generation project entry .... the Rowan work is still evolving ...

Anyway, I think that putting the control of what is being used for
development completely in the developers hands and avoiding the need to
edit baselines is a good thing ... and I think that the idea of having
specifications that are shared by a cluster of images is a good way to 
go ...

Dale

[1] https://github.com/GsDevKit/GsDevKit_home/blob/gh-pages/Seaside32.ston
[2] https://github.com/dalehenrich/Rowan
[3] https://github.com/dalehenrich/Rowan/blob/master/specs/Rowan.ston

Dale

no private emails please.

stef

On Mon, Mar 5, 2018 at 6:57 PM, Dale Henrichs
<[hidden email]> wrote:

Denis,

See the "Lock Command Reference"[1] .. As I mention in a subsequent message I think that the best approach is to have meta specification for the configuration of a development environment that includes a spec for what should be loaded and this spec needs to be something that individual developers can customize for there local dev environment ... then the Metacello BaselineOf is used for production loading purposes and the load specification is used for customizing a particular development environment which may include a mix of additional projects that all _under development_ together ...

Dale

[1] https://github.com/Metacello/metacello/blob/master/docs/LockCommandReference.md#lock-command-reference

Hi Dale.

Where to read about locking? 

2018-03-05 18:57 GMT+01:00 Dale Henrichs <[hidden email]>:

Cyril,

For development I tend to use Metacello locking instead of hard-wired dependencies in the BaselineOf and that works very well --- it completely avoids the need to edit a baseline for development purposes and this approach works really well for me ... perhaps we can discuss this in more detail in a separate thread or even private email? This all seems to be more complicated for you guys than has been my experience and of course you guys _appear_ to be completely ignoring the Metacello locking feature so I'm curious why ...

Dale

On 03/05/2018 08:02 AM, Cyril Ferlicot wrote:

On Mon, Mar 5, 2018 at 4:51 PM, Guillermo Polito <[hidden email]> wrote:

I still do not understand... Also I do not understand your usage of the term "the merge". Can somebody give a **concrete** scenario? 

I'll try

For example the project MaterialDesignLite have two branches that are always here:

- master 

- development

Each commit on master is a stable release and end up with a tag as "v1.2.2". Since it's stable, BaselineOfMaterialDesignLite should depend only on fixed version of the dependencies. For example it should depend on MaterialDesignColor "v1.0.0".

On the branch dev, we want to get the patches and possibly the minor versions of the dependencies automatically. In the baseline we then want to depend on MaterialDesignColor "v1.x.x". Thus, we follow the changes of MaterialDesignColor while it's not a major release.

Because of this situation, BaselineOfMaterialDesignLite is different on the two branches. Later, if I want to merge development into master in order to release a new version, master will get the BaselineOfMaterialDesignLite with semantic versionning dependencies instead of the fixed dependencies. Before the release I'll need to change the Baseline to get fix dependencies once again. 

I hope I was clearer. :)

 

When you release a version, please do not move that version. You should then create new versions, with new numbers and new code. But never touch old versions with old numbers and old code. Like that I can download the same old code using the same old number to get the old version whenever I want :).

You can try to do it with branches, tags, different repositories, or even with zipfiles in mails. I don't care. Just don't modify releases and I'm happy with it.

 

--

Cyril Ferlicot
https://ferlicot.fr

That's fine ... but then it should be a separate thread ...

Dale


On 03/05/2018 11:16 AM, Stephane Ducasse wrote:

On Mon, Mar 5, 2018 at 8:07 PM, Dale Henrichs
<[hidden email]> wrote:
for Pharo master is the current released version and dev the bleeding edge

>
> In an image where you want to use the dev_pharo7 for a project, you do a
> Metacello lock on github://xxxx/xxx:dev_pharo7/src BEFORE loading the
> project ... if there are multiple projects that need to all coordinate
> development then you follow the same convention and use a Metacello lock for
> each of the projects ... Executing expressions to lock projects is tedious
> to use and manage.

Do you have a text explaining what is a lock?
I do not understand the problems locks solve.

In an image where you want to use the dev_pharo7 for a project, you do a
Metacello lock on github://xxxx/xxx:dev_pharo7/src BEFORE loading the
project ... if there are multiple projects that need to all coordinate
development then you follow the same convention and use a Metacello lock for
each of the projects ... Executing expressions to lock projects is tedious
to use and manage.

Do you have a text explaining what is a lock?
I do not understand the problems locks solve.

The "Lock Command Reference" hopefully does the job[1].

Dale

[1] https://github.com/Metacello/metacello/blob/master/docs/LockCommandReference.md#lock-command-reference

Hi Dale!

On Mon, Mar 5, 2018 at 8:07 PM, Dale Henrichs <[hidden email]> wrote:

On 03/05/2018 09:07 AM, Guillermo Polito wrote:

On the other side, there is the fact that Metacello baselines are so far nice to describe release project dependencies, but they are not so nice to describe subprojects/development dependencies that may get edited along with the parent project. Kind of what we used to do with #bleedingEdge. I feel this is a complex problem, that not even SBT or maven that are there since years are capable of solving nicely... Tode and Iceberg metacello integration try to solve this problem by "ignoring the dependency and using the loaded repository" but this may not be successful either...

I have not been following this thread in detail, but this comment leads me to believe that the issue that you guys are running into has to do with trying to ongoing distributed development across multiple projects and multiple platform versions ...

I don't know exactly the case of Calypso, but I don't think it is being maintained for other than latest Pharo 7.

The Calypso case as I understand it is that it has a couple of dependencies

 Calypso

   | - commander

   | - class annotations

The three projects are hosted in different repositories, but Denis wants to work on "bleeding edge" of the three of them at the same time.

Then when he loads calypso in a new image, if he just uses a baseline he will load specific versions of the dependencies and not the latest ones...

I think this is the particular use case for locking, isn't it?

Yes this is exactly the use case for locking.

 

If so then I think that the moral equivalent of #bleedingEdge should be a branch structure along the lines of:

  master branch --- the project release that is known to work on all supported platform versions.
  dev_pharo6      --- the current #bleedingEdge for Pharo6.0
  dev_pharo7      --- the current #bleedingEdge for Pharo7.0

In an image where you want to use the dev_pharo7 for a project, you do a Metacello lock on github://xxxx/xxx:dev_pharo7/src BEFORE loading the project ... if there are multiple projects that need to all coordinate development then you follow the same convention and use a Metacello lock for each of the projects ... Executing expressions to lock projects is tedious to use and manage.

It is not that complicated I think ^^.

I have found that "load specifications" are much easier to work with than "load expressions", similar to the way class definitions are superior to smalltalk code for creating classes, one can reason about a "load specification" for example it is possible to merge two load specifications to produce a single load specification whereas this kind of thing is not possible if you are given two arbitrary Smalltalk expressions to merge ...

 

In tODE I started using project entries[1] that are downloaded to disk and shared by multiple images that allow a developer to specify what branch/tag/commit they are interested in using ... each image that is started up in a "cluster" uses the project entry to do a Metacello lock on the project "automagically"... if there is a series of projects where the developer needs to use dev_pharo6 they can arrange to edit the project entry to reflect the proper branches for the set of projects that need special _development_ treatment ... these .ston files can be shared amongst groups of developers ... of course once the local clones of these projects are on the local disk, then it is up to the developer (or a set of scripts) to periodically update to the latest development version of each of the projects ...

YES. This gets closer to what I want. What I don't like from locking is that I have to:

  - know in advance all the projects (+ dependencies) I want to develop (specially if I'm not the main project developer)

  - know where they are stored in the disk

  - do an explicit locking on each of them

Haha, and that is why I created the project entries ("load specifications") in the first place:)

There is a missing abstraction that probably you get in Rowan or tODE with that project entry specification?

Yes, tODE project entries are an implementation that I have been using for several years now and over time I have found some limitations in the tODE implementation that I am attempting to correct in the Rowan implementation ... The Rowan spec I originally referenced[1] is actually "load expression" that is the equivalent of a Gofer load expression ... I have classes for a Metacello-based "load expression" but I haven't gotten to the point where I have integrated Metacello into Rowan.

[1] https://github.com/dalehenrich/Rowan/blob/master/specs/Rowan.ston

Because the reality is that personally I'm starting a new image every a couple of days, and having a specification for it would be much easier than locking here and there...

I want just to specify "My project X at development time requires Y, Z, H and Z and H are subprojects".

Then, he should realize he should lock Z and H for me.

Yes this is exactly the problem I was dealing with in the early implementation of tODE and with project entries.

You might want to look again at my FAST talk on "Dangerous Liaisons: Smalltalk, Files, And Git"[2] as I talk about "image clusters" that share a collection of project entries and the fact that each image needs to be able to have a private set of "project entries" ... now that you are experiencing the pain my talk might make more sense:)

... also I am thinking in terms of an "image specification" that is made up of a collection of Rowan "load specifications" ... Rowan has the notion of a Project Set and it is possible to load a set of projects. So it is not that big of a stretch to imagine a .ston image specification (collection load specifications) can be passed around and then customized if needed by individual developers and then project set can be used to create the image or images ....


[2] https://www.youtube.com/watch?v=Ejmqs0xLvSk

Now, If subprojects Z and H are in the same repository as X, that should not be complicated :).

In Denis' case it could be more complicated...

Right but once Denis gets his dependent projects cloned to local disk he can update each of the projects on his schedule by pulling later versions of the projects as he goes ... I lean to disk-based clones that can be shared by multiple images (an "image cluster")  ... tODE does not have the notion of image clusters, but I do have sets of images that are sharing a common set of projects where a number of those projects are on development-specific branches so Iknow that this concept makes sense ...

To share a specific configuration of development git repos, you need to know that SHA anyway, so the image can produce a collection of project entries with the SHA of the commit and those project entries can be included in a bug report ...

For Rowan[2], I am using a "load specification"[3], which is a second generation project entry .... the Rowan work is still evolving ...

Anyway, I think that putting the control of what is being used for development completely in the developers hands and avoiding the need to edit baselines is a good thing ... and I think that the idea of having specifications that are shared by a cluster of images is a good way to  go ...

Dale

[1] https://github.com/GsDevKit/GsDevKit_home/blob/gh-pages/Seaside32.ston
[2] https://github.com/dalehenrich/Rowan
[3] https://github.com/dalehenrich/Rowan/blob/master/specs/Rowan.ston

--

Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr Web: http://guillep.github.io Phone: +33 06 52 70 66 13

Le 16/03/2018 à 18:37, Denis Kudriashov a écrit :
That sounds like a nice scheme.

Thierry