About balkanisation

On 11/7/16 7:36 AM, Esteban Lorenzano wrote: Well, I don't think that Iceberg is doing package loads and I don't
think that Iceberg will provide support for mcz repositories so I still
think there is a need for this Metacello Project Browser ... the
Metacello Project Browser would delegate to Iceberg for all of the
services that it does provide.

The Metacello Project Browser would not be a purely git/github tool ...
Metacello Projects are repository format neutral so svn, hg, etc. would
be equally supported from the Metacello Project Browser ... the standard
set of operations: save/load/diff/log are needed for all different
repository types ...

I don't think it's a lot of work to implement the Metacello Project
Browser tool and it is important that there be one place to look for and
manage the projects loaded in image .... this is the "unifying" bit ...
>
>> The unification comes because you can use one metaphor (project) to work with both Mcz and Filetree ... the underlying implementation will ultimately...
>>
>> The next layer of unified tools is when you look at version history, for a method, you need to provide the ability to view the git commit history for the method (if stored in git) or the image history ... git commit hisotry can be provided at the project/package/class/method ... whereever possible the equivalent mcz/git service should be provided so that the two systems are on an even par …
> also, this is supposed to be provided by iceberg.
>
>> The Monticello Browser and Iceberg GUI's don't go away, because it is often necessary to do work at the package level, but I think that putting focus on the project is the key ingredient to success for integrating git …
> Iceberg is not package-oriented. It just show repositories/packages/classes/methods… this is a good way to do it and I do not thing anything is lost like this.
> You need to take a second look at Iceberg :)
Well I am specifically referring to this issue[1] and this comment by
Guille[2] and Nico[3]. If Iceberg isn't going to load packages, then you
need a tool that will load a package ...

and as I say above this tool must work with ConfigurationOf and
BaselineOf for svn, hg, etc. repositories ...

[1] https://github.com/npasserini/iceberg/issues/184
[2] https://github.com/npasserini/iceberg/issues/184#issuecomment-254139549
[3] https://github.com/npasserini/iceberg/issues/184#issuecomment-257865003
>
>> Since git repos are persistent on disk and will be shared ... it is important that there be a way for developers to easily access the git repos for projects that have been cloned locally but not yet loaded in the image itself ... I am really struggling with getting how important this point as this is the also a point that ties a Catalog Browser and Project Browser together
> this is something to think… I do not get what catalog browser  can do here (but yes, a way to browse local repositories needs to be provided).
again in tODE, the Metacello Project Browser (aka project list) allows
one to view projects that are cloned to local disk but not loaded in the
current image ... note that I an using the term "project" not git
repository ... the catalog browser defines "load specs" for projects ...
but the developer is not able to control how the project gets loaded ...
a better more flexible catalog browser would actually download spec
objects into the Metacello Project Browser and allow developers to
adjust the "load spec" for their environment, share this "load spec"
with the other images in the "image cluster" and use the "load spec" to
clone github projects to local disk and then load from the Metacello
Project Browser ...


>
>> I've been using this approach for several years now and once you have the tools and can see at a glance what's "going on in your image" it is fun to work in a mixed environment ... all of this frustration that is bubbling on the list is largely due to not having these missing tools and underlying support --- I think …
> I do not think we are so far from your vision. I think you did not get it Iceberg right… please, take a second look :)
I would agree that not far is fair and I don't think that there is a ton
of work to be done, but it is work that needs to be done ... so going
the last little bit is necessary ...

Nico is open to discussion, but I am not sure that all of the things
that need to go into the Metacello Project Browser belong in Iceberg ....
> now… is true that now everything you say is already done… but this is general orientation :)
>
> Esteban
>
> ps: all of this is *clearly* not pharo-users but pharo-dev discussion, let’s move there.
>
>
oops now you tell me:)

Dale

And this one I don't understand. A smooth, git / iceberg oriented transition over Monticello/Metacello is perfectly doable... As Dale explained. A nice Iceberg gui reworking / making git usable is perfect.

But why make the transition so hard? You get Stef angry on a Sunday morning because he can't find things anymore... even if he is a strong proponent of the strategy he complains about ;)

No my point was not that.
My point is that it is important to pay attention and not to add more noise than necessary. Let us take the time and move alltogether.

Stef

Thierry

Hi,


On 07/11/16 02:50, Esteban Lorenzano wrote:
[...]
> What I mean is that we can help any transition, but at the end there is no way of having a “nice, coexisting” ecosystem: we will have one OR the other, or something that does not works at all, but we will not have seamlessly one AND the other (which does not means people using monticello will be forced to use git tools or vice-versa, just that you will need to chose one… right now many (many for real) of our problems come from the attempt of keeping our git support behaving as regular monticello… and that way of doing has a limit. A limit I think we already passed.
[...]

That's clarifying. Thanks a lot Esteban. I will try to keep myself on
the Monticello boat a little longer, until the filetree and Iceberg
stuff become clearer to me and I can see how to use them in a way that
doesn't interrupt the workflow I have now (as happens with Monticello
and Fossil).

> Anyway, if you can list what you think we will need for the transition, I will be very glad to see what we can do :)

Yes, Dale, your experience would be very appreciated.

Cheers,

Offray

Hi Dale,

2016-11-07 12:03 GMT+01:00 Dale Henrichs <[hidden email]>:

On 11/7/16 7:15 AM, Thierry Goubier wrote:

2016-11-07 11:05 GMT+01:00 Esteban Lorenzano <[hidden email]>:

On 7 Nov 2016, at 10:03, Thierry Goubier <[hidden email]> wrote:

Hi Esteban,

I cut out the rest, because I agree with all your points, except for...

2016-11-07 9:55 GMT+01:00 Esteban Lorenzano <[hidden email]>:

[ ... ]

Replacing Monticello with git goes in this direction:

[ ... ]

And this one I don't understand. A smooth, git / iceberg oriented transition over Monticello/Metacello is perfectly doable... As Dale explained. A nice Iceberg gui reworking / making git usable is perfect.

Well… I disagree with this. 

All my experience says the opposite: this is a convenience usage that in the long way does not match (the thing that we simulate mcz packages do not work… and makes things a lot harder to maintain later). 

Nico has worked a lot on this, maybe he has something to say.

I'd like to. Simulating mcz? That I don't get it.

Thierry, If I'm not mistaken, Esteban is referring to the fact that in FileTree we are still using Monticello to do the load of the packages and even when we are running metadataless, we end creating fake meta data to simulate an mcz ... you and I have had conversations about ways to eliminate this "requirement" because it is meaningless in a git context ...

Yes, this I understood. I do believe that what I suggested at one point (have the ability to compare versions with an 'isAncestorOf') would be very nice for that transition (work in mcz as well as on git with/without metadata).

II think that the work that you did with gitfiletree was very influential in making git a realistic option for the Pharo community and I think that the choices you made were excellent, but I think that for a transition it is important that git be presented as itself more fully and expose some of the features that set git apart (project/package/class/method diffs; multi-package commits; etc.) ... but I also feel that Monticello does not need to be diluted --- the existing Monticello tools should remain in place for the forseeable future ...

With the work that Richard Sargent did on the CypressReferenceImplementation, I would prefer to say that Cypress can provide an Alternative to Monticello rather than replace it ... the CypressReferenceImplementation includes a package loader so it isn't necessary to convert Filetree format on disk to Mcz format in the image --- without all of the ancestry, "latest version stuff" and package-cache the loading process becomes much, much simpler...

I think that both Monticello and Cypress can live together in the image ...

For me, MC is also the code / diff model behind: will Cypress rewrites all that too?

Well Cypress is not that much of a rewrite ... Cypress definitions are closely based upon the Monticello definition model... snapshots and loading based on comparing the snapshot of the repository against the snapshot of the image are also closely based on Monticello ... The differences are in the fact that Cypress packages do not have ancestry, author names or version numbers:
  - the ancestry and author is recorded and managed by git (or svn or hg)
  - there is only one version of a package in a fieltree repo, so there is no
     need to sort packages by version number to fine the "latest version"

So Cypress is more of a "take all of the really good parts of Monticello while dropping the bits that are handled better  by git" than it is a complete _rewrite_ ...

 

I have created a version of Metacello that uses Monticello OR Cypress and I expect to eventually (in the next several months --- it doesn't take that long, but I've got other things on my plate) to have a version of Metacello that uses Monticello AND/OR Cypress again I think that smooth transitions (that may take a long time) are better supported in this fashion than to draw a line in the sand and force the usage of one OR the other ...

As long as Cypress behaves as a MC repository and fits into the same API, I really don't see where is the difficulty. As we are saying now, adding a type of repository / emulating the MC overall API is no real difficulty in itself and isolates one from all the project management issues (because that means Metacello, ConfigurationOf and BaselineOf just keep working as usual).

There are api differences between MC and Cypress ... MC has ancestry and package numbers while Cypress does not ... but I don;t think that these are major items ... I have yet to try to merge MC and Cypress in the same Metacello code base and I have yet to look closely at the requirements for  Cypress Package Browser tool, but I know that the Cypress Package Browser will be much simpler ... But the goal for Metacello is that individual developers will be able to choose the repository/package model that best fits their work flow and portability needs ...

We need to get the MC repository API to evolve a bit, in part to handle the "save multiple packages, do one commit without using package dependencies." required now, and also to expose more of the repository organisation (remotes, branches) for the GUI above. And remove some of the things saying "I'm MC trying to do Metacello's job".

yes I think this is the area where a Metallo Project Browser comes into play. The technology for committing the dirty packages in a a ConfigurationOf has been around for quite awhile, but a tool that unifies the "multi-package" commit for ConfigurationOf and BaselineOf hasn't popped out of the wood work ...

What is a bit annoying, really, is this talk of rewriting everything where some of the pieces (the project browsing you're talkin about, for example) is already there in the image, just not exposed.

Well I am annoyed as well .. I would say that perhaps we are in screaming agreement ... I've said that I don't think it's a lot of work to do the project-centric tool. but a project-centric tool that encompasses both ConfigurationOf and BaselineOf will expose the features that "are already there"

I'm trying to code a little something for that... stay tuned for a screenshot soon.

Cool:)

Dale

2016-11-07 15:30 GMT+01:00 Thierry Goubier <[hidden email]>:

Thierry, If I'm not mistaken, Esteban is referring to the fact that in FileTree we are still using Monticello to do the load of the packages and even when we are running metadataless, we end creating fake meta data to simulate an mcz ... you and I have had conversations about ways to eliminate this "requirement" because it is meaningless in a git context ...

Yes, this I understood. I do believe that what I suggested at one point (have the ability to compare versions with an 'isAncestorOf') would be very nice for that transition (work in mcz as well as on git with/without metadata).

I would like to listen to your ideas about this topic, but I am not sure it is possible to achieve that compatibility. In fact we tried to do it for Iceberg and at some point we decided to abort it.

On one side, trying to re-create monticello sequential file numbers in git is simply not possible, at least in a reliable way. On the other side, loading the graph of package versions and dependencies is really slow for big repositories (such as pharo-core), once we removed that requirement Iceberg got like 100x faster.

I agree that the git-oriented tools should not go ot of their may to conform to the MC model .. as you say it just does not fit ... but many of the standard operations that are performed can be performed on both git and mcz repositories without forcing both of the models to share identical implementations ... in image tools should distinguish between mcz and git projects and dispatch the standard operations to the appropriate repository --- and this si the spot where the handling changes based upon the repository model ...

Dale

Hi Dale,

2016-11-08 2:03 GMT+01:00 Dale Henrichs <[hidden email]>:

[ snip ... ]

yes I think this is the area where a Metallo Project Browser comes into play. The technology for committing the dirty packages in a a ConfigurationOf has been around for quite awhile, but a tool that unifies the "multi-package" commit for ConfigurationOf and BaselineOf hasn't popped out of the wood work ...

I'm having some issues with the scripting approach of Metacello, for that. Just manipulating a baseline in the Metacello registry (locking it) seems to be like writing a compiler code generation item: write a Metacello script ;) (Metacello>>#lock contains interesting code).

I think I agree:) ... that is why i am talking about a tool ... developers shouldn't have to pass around workspaces to load and manipulate projects .. a project tool can partially automate the process, but I believe that there needs to be a "load spec" object that can be passed around and massaged by code instead of requiring a user to edit a Smalltalk workspace ... it's what I do in tODE ... the only times I have to write Smalltalk load scripts is when I work in Pharo :)

I will repeat that I don't think Pharo should do exactly what I've done with tODE, but a Metacello Project Browser and some sort of "load spec" object that is shared amongst a cluster of images is something that needs to be seriously considered ...

 

What is a bit annoying, really, is this talk of rewriting everything where some of the pieces (the project browsing you're talkin about, for example) is already there in the image, just not exposed.

Well I am annoyed as well .. I would say that perhaps we are in screaming agreement ... I've said that I don't think it's a lot of work to do the project-centric tool. but a project-centric tool that encompasses both ConfigurationOf and BaselineOf will expose the features that "are already there"

I'm trying to code a little something for that... stay tuned for a screenshot soon.

Cool:)

And here it is! I'm still fighting with the Metacello lock / unlock thing.

 

If Pharo was more really modular in configurations and baselines, then I would see more things into the registry and I could group packages based on that (instead of having to prepare a ad-hoc classification for my my browser).

Yes this is headed in the right direction ... I am in Argentina this week so it isn't easy for me to do much collaboration, but I would love to help you with your lock problem next week ... For example ... you do not want to directly expose the Metacello registry as there are "hybrid projects" where a ConfigurationOf causes a BaselineOf to be loaded, so the BaselineOf needs to take precedence (I have code for resolving this) and I strongly suggest that the Project Browser not be built-into the Class Browser, but be a separate tool like the CataLogBrowser ... but this is just a suggestion --- a separate tool dedicated only to loaded Projects and unloaded Projects of interest allows one to get a very quick overview of what is in the project or not and when version skew pops up, one can easily see the affected projects ... embedding this kind of information in the code browser can make it hard to get the overview and if you choose to live with version skew can be annoying to ignore .... again these are just my thoughts ..

Regardless I think you are definitely headed in the right direction and it isn't all that much work ... yet:)

Dale

On 7 November 2016 at 14:28, stepharo <[hidden email]> wrote:

[ ... ]

And this one I don't understand. A smooth, git / iceberg oriented transition over Monticello/Metacello is perfectly doable... As Dale explained. A nice Iceberg gui reworking / making git usable is perfect.

But why make the transition so hard? You get Stef angry on a Sunday morning because he can't find things anymore... even if he is a strong proponent of the strategy he complains about ;)

No my point was not that.
My point is that it is important to pay attention and not to add more noise than necessary. Let us take the time and move alltogether.

If you want to get somewhere with this story, you don't want to wait till everything will be ready.
Transition will never start unless you force users to enter the minefield and let them clear the mines for you. Step by step. Many will blow themselves up, while some will manage to pass unhurt.. 
Because else, it will be always a minefield between you and the destination of your journey :)

I think that at the early stages of the transition you have to support both approaches ... when the new tools are in place and stabilized then one can consider ... the transition has already started so this is not a case where you need to force folks to change, but a case where you need to support the folks who choose to change ... it can be relatively low cost to keep the old tools around for quite awhile ... I would think ...

Dale