OSSubprocess use

Hi,

I'm looking at OSSubprocess use as a requirement for GitFileTree and I
wonder if it is wise to list the dependency on OSSubprocess #stable or
to set to a specific version (v0.2)?

Thanks,

Thierry

Le 23/01/2016 15:44, Mariano Martinez Peck a écrit :
Yes. What is interesting is I can just target the baseline of
OSSubprocess in that case, using that url

github://marianopeck/OSSubprocess:v0.2.0/repository

which is convenient.

Thierry

Hi Both, Hi Dale,

Maybe soon enough we can find a way of integrating CI server test results for specific packages with Metacello and talk not about #stable, but #greenest.  Green means tests have passed, but i think the connotation of green=young is common too, no?  (viño verde).  This doesn't protect against API changes, when the tests get changed to match the new API, but that implies versioning the API separately from the package so one can ask for #greenest #api=1.23.  

Le 23/01/2016 17:17, Mariano Martinez Peck a écrit :
Yes, I know, I implemented part of that support into gitfiletree urls
(commit ids and tags don't work with git clone, of course).

> Yes, and not only tags, what everything (commits, branches, etc). See
> this post:
> http://blog.yuriy.tymch.uk/2015/07/pharo-and-github-versioning-revision-2.html
>
> This is the supported pattern:
>
> github://<username>/<projectname>[:<branch>|<tag>|<SHA>][/<path>]

I was thinking of the wildcard support Dale implemented ;)

Thierry


> :)
>
> --
> Mariano
> http://marianopeck.wordpress.com

Hi Eliot,

Le 23/01/2016 17:04, Eliot Miranda a écrit :
Well, if you take a Travis / Github integration and you fetch from
branch 1.23, then you should only merge into that branch with a pull
request if the tests are green. Moreover, with Metacello, baselines and
branches you allways target the newest version; as long as your CI
ensure that tests are green before merging into public/stable branches,
then that already works.

The branch methodology would look like that, then:

- one branch for each stable / long lived version with garanteed API
compatibility
   - v1.1
- one tag for each released version, when merged in their right overall
branch
   - v1.1.0, v1.1.1, v1.1.2, v1.1.3 (inside branch v1.1)
- development branches for the main work.
- and an integrated CI like travis so that no merge can be done in a
release branch without having everything green.

Like that, if you target v1.1, you'll get the latest green version on
API 1.1 :)

Thierry

We want to use Metacello to manage smalltalk dependencies in a way that
results in working software using a minimum amount of manual work. At
the moment there are several ways in which Metacello is used, some of
which result in more work and instability than others.

An inherent conflict in managing these dependencies is the level of
repeatability that is wanted and achievable, vs. the amount of manual
work needed. On the one hand we would like perfect repeatability. That
needs an exact description of the order in which specific versions of
packages were loaded in a specific image. In Metacello, we can achieve
this by making sure all package versions are fixed and the dependencies
are fixed to numeric versions.

The problem with this is that that not necessarily results in working
software. There are several reasons for this:
- Metacello only manages the smalltalk level dependencies. It does not
take into account changes in the vm necessitating image-level changes or
operating system and library changes.
- The image itself is not fully managed with Metacello (yet). Bug fixes
made after the creation of the Metacello version might be needed for the
software to work.
- Bugs might have been found and fixed in dependencies.
In the situation of a top-level configuration, i.e. one that no other
configuration depends on, this might be acceptable. Metacello
dependencies are defined one-directional. A configuration knows on which
configuration it depends, not which configurations depend on it.
Configurations that are published should therefore always be regarded as
possibly being used by others, and not only top-level.

If we consider the situation where others (B) are depending on
configurations (A) that are written like this, then the situation gets
worse:
- B might need an update because it is affected by an issue that does
not affect A.
- B might need a different load ordering, e.g. need to only partially
load dependencies of A, then something itself and then the other
dependencies of A.
Nevertheless, sometimes we need near-perfect repeatability. For that, a
snapshot of all loaded packages provides a reasonable alternative to a
Metacello configuration version.

On the other hand the least amount of manual updates results from always
depending on #development or #stable versions of dependencies. This
results in the following problems:
- If the API of a dependency changes, the configuration might break
silently.
- Projects might be slow to promote versions from #bleedingEdge to
#development to #stable, resulting in a slow uptake of improvements.
For configurations of simple packages that have an API that is very
stable or only expands, depending on #stable can be appropriate.

 From a modularity point of view, a configuration should be specific on
the packages in its own span of control, and as generous as possible in
what it accepts from its dependencies. In that way a high cohesion and
low coupling is achieved. We can achieve that by defining symbolic
versions that reflect observable change in behaviour. If we make changes
that should not affect users of the configuration, we create a new
numeric version and update the symbolic version in place to use this new
version. By default we only depend on these symbolic versions.

Stephan

Le 23/01/2016 18:42, Gabriel Cotelli a écrit :
> Well.. If you follow the semantic versioning guidelines
> <http://semver.org/> the dependency can be done in metacello with the
> new scripting API like this:
>
> github://marianopeck/OSSubprocess:v1.?/repository

Yes, this is what I was looking for.

But, thinking of it, if I do a v1 branch where I merge all the v1.x
versions as I release them, then

github://marianopeck/OSSubprocess:v1/repository

is enough to retrieve the latest subversion.

I'm just looking to see what is the simplest / less error prone setup;
to avoid the errors made updating a configuration by hand. If you
release say v1.5 and have to update a configuration in the meta repo by
hand with that number, then part of your release testing suppose you
also run that configuration to check you haven't introduced errors.

Thierry

Hi Thierry,

Listening to Stephan I think the key point is "with the minimum amount of manual work".  Of course I can use the process above.  But if what the tests were to run were specified in code or as package metadata, and the ci to run the tests was "available as a service" without my having to configure, then I wouldn't need to manually manage the process of only committing to the public version when the tests were green; the system would do that for me.

The only thing I'd have to do as a designer is name or version the API if it needed to evolve.

>
> Thierry
>

Le 23/01/2016 19:23, Eliot Miranda a écrit :
I certainly agree with that "minimum amount of manual work", and this is
why I'm looking for the less friction setup: one that doesn't require me
to update a few configurations in addition to merging and checking my
integration tests.

However, I would prefer to have, when releasing, a decision where I
decide to push something, when all tests are green, and not
automatically. As, for example, Dale has setup the FileTree repository,
then I can commit and push on separate branches, let CI check those, and
merge when I'm sure to a stable branch (when everything is green).

Now, having nightly builds automatically the latest green one... I guess
that can be done.

> The only thing I'd have to do as a designer is name or version the
> API if it needed to evolve.

But the setup you describe is a bleeding edge type of things, on which
some commits or pushes just fail the CI tests. So I'm not sure I'd like
to be a user of that.

Thierry

We want to use Metacello to manage smalltalk dependencies in a way that results in working software using a minimum amount of manual work. At the moment there are several ways in which Metacello is used, some of which result in more work and instability than others.

An inherent conflict in managing these dependencies is the level of repeatability that is wanted and achievable, vs. the amount of manual work needed. On the one hand we would like perfect repeatability. That needs an exact description of the order in which specific versions of packages were loaded in a specific image. In Metacello, we can achieve this by making sure all package versions are fixed and the dependencies are fixed to numeric versions.

This is one of the reasons that I have pushed for the use of git ... with git ALL of the packages are committed together so a single commit SHA is sufficient to ensure "that all package versions are fixed" without regard to numeric versions - the packages that work together are committed together.

The Metacello registry records the SHA of the commit that was loaded into an image so there is absolutely no ambiguity.

Dependencies are recorded in a BaselineOf that is also committed with the packages, so the dependencies are "guaranteed" to be in sync with the package versions ... a single SHA is all that is needed to exactly duplicate the formula used to load a project into an image ...

The problem with this is that that not necessarily results in working software. There are several reasons for this:
- Metacello only manages the smalltalk level dependencies. It does not take into account changes in the vm necessitating image-level changes or operating system and library changes.

There is no reason that project attirbutes should be restricted to only platform and platform versions .... the attribute list is generated by the running image, so a list of attributes that included additional information like vm type and version could be included and then it would only be necessary to use these new attributes in the baseline ...

- The image itself is not fully managed with Metacello (yet). Bug fixes made after the creation of the Metacello version might be needed for the software to work.

Yes, this is a critical point ...

- Bugs might have been found and fixed in dependencies.
In the situation of a top-level configuration, i.e. one that no other configuration depends on, this might be acceptable. Metacello dependencies are defined one-directional. A configuration knows on which configuration it depends, not which configurations depend on it. Configurations that are published should therefore always be regarded as possibly being used by others, and not only top-level.

If we consider the situation where others (B) are depending on configurations (A) that are written like this, then the situation gets worse:
- B might need an update because it is affected by an issue that does not affect A.
- B might need a different load ordering, e.g. need to only partially load dependencies of A, then something itself and then the other dependencies of A.
Nevertheless, sometimes we need near-perfect repeatability. For that, a snapshot of all loaded packages provides a reasonable alternative to a Metacello configuration version.

... and a list of project/sha/branch like the following:



provides near-perfect repeatability on a project by project basis --- there are ways using existing Metacello features to ensure that the exact SHA associated with each of the projects will be used to the exclusion of all others ...

On the other hand the least amount of manual updates results from always depending on #development or #stable versions of dependencies. This results in the following problems:
- If the API of a dependency changes, the configuration might break silently.
- Projects might be slow to promote versions from #bleedingEdge to #development to #stable, resulting in a slow uptake of improvements.
For configurations of simple packages that have an API that is very stable or only expands, depending on #stable can be appropriate.

The historical problem with generic symbolic versions like #development and #stable has  been that the semantics of the #stable version changes over time in the beginning it meant the version that was "stable on Pharo2.0", but when the same symbolic version to imply "stable on Pharo3.0", all was lost:)...

From a modularity point of view, a configuration should be specific on the packages in its own span of control, and as generous as possible in what it accepts from its dependencies. In that way a high cohesion and low coupling is achieved. We can achieve that by defining symbolic versions that reflect observable change in behaviour. If we make changes that should not affect users of the configuration, we create a new numeric version and update the symbolic version in place to use this new version. By default we only depend on these symbolic versions.

You are basically describing the principles behind semantic versioning[1] and  I've always been a fan of the symbolic version scheme that the Seaside project has been using (#release3, #release3.1, etc.), because it makes it possible to express a project dependency at the appropriate semantic version level...

For git repositories, one can create a tag for a particular SHA ("specific on the packages in its own span of control") and then use "tag pattern matching"[2], to provide a reasonable approximation of symbolic versions with a "git" repo.

-----

4 years ago, I was very aware of the fragility of the  ConfigurationOf ecosystem and at the time, was very glad to see that responsibility for the implementation for versions, branches and tags could be off-loaded onto a disk-based SCM - leaving Metacello to continue to specify package load order and external project dependencies...

In the subsequent years, I have been adding features to Metacello that I found were needed to support a "git-based" workflow (primarily the lock command[3])....

I have also been working on building a Smalltalk development environment that embraces "git" from the ground up and I consider that I've learned quite a few important lessons in this area ... I am not going to say that tODE[4] should be ported to Pharo, but I will say that tODE _is_ a proof of concept that a git-based development environment for Smalltalk can be built that preserves image-based Smalltalk development while leveraging "git" as the SCM... I should note that tODE bridges the ConfigurationOf and BaselineOf worlds with support for both ....

I have been "patiently waiting" for folks to come to accept/tolerate "git" and it seems that that time is getting close enough to begin trying to share the lessons that I've learned  --- but I can't "push on a rope" ---

Tool support is critical at this juncture and I am willing to share my experience (and code if desired) with the folks who will be building these next generation tools ---

As I mentioned in another post, the focus for the tools should shift from being package-centric to being project-centric ... a "git" repository contains all of the artifacts associated with a project, and in the "tODE model" there is a one to one mapping between a project and a git repository.

Of course there is more, but I will save that for sharing with folks who are curious and interested in learning from my mistakes and my successes:)

Dale


[1] http://semver.org
[2] https://github.com/dalehenrich/metacello-work/issues/277#issuecomment-58970696
[3] https://github.com/dalehenrich/metacello-work/blob/master/docs/LockCommandReference.md#lock-command-reference
[4] https://github.com/dalehenrich/tode#tode-the-object-centric-development-environment-

On Wed, Jan 27, 2016 at 12:59 PM, Dale Henrichs <[hidden email]> wrote:

I have also been working on building a Smalltalk development environment that embraces "git" from the ground up and I consider that I've learned quite a few important lessons in this area ... I am not going to say that tODE[4] should be ported to Pharo, but I will say that tODE _is_ a proof of concept that a git-based development environment for Smalltalk can be built that preserves image-based Smalltalk development while leveraging "git" as the SCM... I should note that tODE bridges the ConfigurationOf and BaselineOf worlds with support for both ....

I've been aware of tODE for a while, and though you probably mentioned it before, I didn't realise it worked with git.  I can't look at it right now, but I'll put it on the list. It definitely interests me.

 

Documentation is still sparse for tODE, but all of the command line commands have man pages:)

I plan to write some posts on this list about the workflows, structure and functionality currently used in tODE and GsDevKit_home that will cover the larger points.

In addition to the git support from within the "Project Browser":



I have a smalltalk API for calling the git command line and I use that for implementing all of the git functionality (so there's no Windows mapping - but then the GemStone server doesn't run on Windows).

In the code browser, you can get git history on a project, package, class and method level.

I've also built a `git mergetool` to a git-level merge from within the Smalltalk dev environment.

When you get the time let me know and I can give you more information ...

Dale