Metacello --- How to make sure that we will be able to load... in 3 years from now

Hi all

I would like that we have a discussion about how do we put in place an infrastructure to make sure that we can get
metacello configuration working in 3/5 years from now.

I think that there are two scenarios:

MyProject
======
in my project I used metacello and I publish a config and if it breaks in the future this is my responsibility


PublishedMyProjectIn
=============
When I publish my project (for now I copy the configuration file from my repository to the one of metacello)
in the metacelloPharo1.0 project, we could automatically pulled up all the necessary files and copy them in the
1.0 repository. This way we can shield and make sure that a project can be loaded.

I would like to have
        MetacelloMetaRepository1.0
        MetacelloMetaRepository1.1
        MetacelloMetaRepository1.2
        ...

what do you think?
Stef


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

El mar, 23-02-2010 a las 11:14 +0100, stephane ducasse escribió:
I was thinking about this problem yesterday in the light of the packages
moved by Lukas.
We have a problem with the packages referenced by a ConfigurationOfXXX.
The can disappear anytime. The repository can disappear anytime.

So I thought that we need a dedicated hosting server where:

- We have all the ConfigurationOfXXX (in the MetacelloRepository repo,
just as now)
- We copy there *all* the packages that conform a giver
ConfigurationOfXXX>>versionXX: from the original repo to this dedicated
repo.
- There is *no way* to delete anything. This is a *write only, read
always, delete never* repository.

So maybe we need a modified squeaksource image there that disables
delete/rename/move packages. Or maybe the metasource from Torsten (is
that the name of the new app designed for Metacello) can add this
requirement. Besides, the app can scan all the ConfigurationOfXXX and
check for what ones are affected by a deletion of a package before
permiting delete of the package. This way we can delete packages but
also see what that will affect and correctly promptly.

The point is, we need an independent, mostly never breaking metacello
repository.

Other idea will be to use google or github versioned repositories and
from them build the published repo, so that when a package is deleted by
accident or by mistake, it can be restored quickly. But I like more the
previous version.

What do you think?

--
Miguel Cobá
http://miguel.leugim.com.mx


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Miguel Enrique Cobá Martinez wrote:
Sidenote: SqueakMap actually caches all URLs used for versions and if
the original URL fails an SHA check it falls back on the cached file on
the SqueakMap server.

In that way it was meant to handle original URLs going stale or
disappearing.

regards, Göran


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

El mar, 23-02-2010 a las 14:12 +0100, Göran Krampe escribió:
>
> Sidenote: SqueakMap actually caches all URLs used for versions and if
> the original URL fails an SHA check it falls back on the cached file on
> the SqueakMap server.
>
> In that way it was meant to handle original URLs going stale or
> disappearing.
>

Does that mean that everything pushed to squeakmap can never be deleted?
If that isn't true, then the problem is just pushed one step further.

> regards, Göran
>

Thanks
--
Miguel Cobá
http://miguel.leugim.com.mx


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi, I am very interested in the whole configuration management, metacello, monticello, public/private repository, project, library/package, object, project, environment, release, version stuff ... some clarity around is badly needed.

My first contact with the whole Squeak/Pharo packages and versioning was very confusing with SqueakSource, SqueakMap, Sake/Packages, Package Universe, Monticello.

Why not look at other distributed version control systems like Darcs, Bazaar, GIT ... and see what could work for smalltalk?

and with that I did not mean to "use" those other DVCS but maybe take ideas from them :)

> and with that I did not mean to "use" those other DVCS but maybe take ideas
> from them :)

This is not a problem any DVCS tackles. Rather take ideas of package
management systems like aptitude, rpm, portage, fink, ... As you see
in all such system, they ship the code (binary or source) together
with the package definition. None of these mechanisms refer to the VCS
where the code is developed, but they include all the dependencies
they need.

I was thinking more along of something of a SAR file that would
include the Metacello configuration and all the referenced mcz files
in one file. Dragging and dropping that file onto the image (or using
some GUI) would bootstrap Metacello, execute the Metacello definition,
and if necessary download and pull in other SAR files from a well
known place. That's how Linux works quite successful for over a
decade.

Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

neat!


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

I like the notion of a secondary repository for packages. It is absolutely necessary to protect us from a catastrophic failure with any one of the SqueakSource repositories.

Perhaps Gofer can be provided with a secondary repository location in the case when the primary repository fails.

With this kind of approach, the metacello configurations can continue to reference the primary development repository, in this way we might just be able to have our cake and eat it too:) The Pharo secondary repository would protect users from repository volatility and package volatility ... Once a version is published in Pharo you can count of being able to load that version over time.

This secondary repository could be maintained by periodically scanning the Pharo configurations and copying the referenced packages from the primary repository to a known secondary repository (this repository could just be done with Apache file access since a fancy UI for the secondary repository isn't essential).

Having code and processes in place means that individuals can reuse the code and process to maintain their own secondary repository for their own production environment.

I have code that does most of what is needed, as I use this process at GemStone for archiving the configurations and mcz packages in our company source code control system.

Dale
----- "Göran Krampe" <[hidden email]> wrote:

| Miguel Enrique Cobá Martinez wrote:
| > El mar, 23-02-2010 a las 11:14 +0100, stephane ducasse escribió:
| > I was thinking about this problem yesterday in the light of the
| packages
| > moved by Lukas.
| > We have a problem with the packages referenced by a
| ConfigurationOfXXX.
| > The can disappear anytime. The repository can disappear anytime.
| >
| > So I thought that we need a dedicated hosting server where:
| >
| > - We have all the ConfigurationOfXXX (in the MetacelloRepository
| repo,
| > just as now)
| > - We copy there *all* the packages that conform a giver
| > ConfigurationOfXXX>>versionXX: from the original repo to this
| dedicated
| > repo.
| > - There is *no way* to delete anything. This is a *write only, read
| > always, delete never* repository.
|
| Sidenote: SqueakMap actually caches all URLs used for versions and if
| the original URL fails an SHA check it falls back on the cached file
| on
| the SqueakMap server.
|
| In that way it was meant to handle original URLs going stale or
| disappearing.
|
| regards, Göran
|
|
| _______________________________________________
| Pharo-project mailing list
| [hidden email]
| http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

> Perhaps Gofer can be provided with a secondary repository location in the case when the primary repository fails.

Yes, it can. You just specify multiple repositories and it will use
them all. If you tell Gofer to #disableRepositoryErrors then it will
silently ignore repositories that are not available.

Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

----- "Lukas Renggli" <[hidden email]> wrote:

| > Perhaps Gofer can be provided with a secondary repository location
| in the case when the primary repository fails.
|
| Yes, it can. You just specify multiple repositories and it will use
| them all. If you tell Gofer to #disableRepositoryErrors then it will
| silently ignore repositories that are not available.

Lukas,

The only thing that worries about including such repository all of the time, is the cost of building the cache of repository when resolving repositories ... the secondary repository would get pretty large over time and most of the time (only in the case of errors) would it need to be scanned.

I suppose Metacello could handle load errors in the code that calls Gofer and do a retry using the secondary repository...the secondary repository _could_ be specified as part of the configuration, so this probably isn't a bad idea...

Thanks,

Dale

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

What if use redudancy features of Internet itself? As you know the
Internet was designed to withstand a nuclear attack, with a lot of
redudant features, which we can explore to harden the Smalltalk
infracture as well.

In the similar ways as guyss like Google do. You know, they have always
a single entry, http://www.google.com, but actually they direct your
search to 1000s and milions of their servers behind. A mini version of
Google can be use for Squeak or PharoSource as well.

As I can see the Monticello repositories are basically the stateless
WebDAV enabled Apache web servers. So having a mirror is a first thing
to do. To still have a single entry point like
http://www.squeaksource.com), we can exploit the redundancy feature of
DNS: round robin entries. That is, you enter two entries in DNS zone,
with the same name but different IP addresses, say:

  squeaksource.com

  www IN A 87.54.32.108
  www IN A 212.18.45.87

This way in the case of failure of one server, the another one will
responds. If both are working, DNS will return one of them in roun robin
fashion.

This will work for WebDAV part of the story, but we still need to tackle
of the statefull web application of SqueakSource... hmm, a bit more
thinking is needed therefore...


Best regards
Janko


On 23. 02. 2010 19:48, Dale Henrichs wrote:
--
Janko Mivšek
Svetovalec za informatiko
Eranova d.o.o.
Ljubljana, Slovenija
www.eranova.si
tel:  01 514 22 55
faks: 01 514 22 56
gsm: 031 674 565

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi Dale,

I believe the original problem is that you do not know upfront about  
where the actual files will be. In this case, perhaps the easiest  
solution is to provide some scripting facility to provide a repository  
mapping that should could be modified at loading time, and not be  
stored in the actual configuration.

Suppose you have something like this:

ConfigurationOfXXX>>baselineX: spec
        <version: 'X-baseline'>
        spec package: 'SomePackage' with: [ spec repository: 'SOME_URL' ].
        ...

ConfigurationOfXXX>>versionX: spec
        <version: 'X' imports: #('X-baseline')>
        spec package: 'SomePackage' with: 'SomePackage.tg.1'.
        ...


Maybe a solution would be that when loading you could say:

(ConfigurationOfXXX project version: 'X')
        withRepositoryMapping: 'SOME_URL' -> 'SOME_NEW_URL';
        withRepositoryMapping: '...' -> '...';
        load

What do you think?

In this way, the original configuration specifies everything, and the  
URL works either as the actual URL or as an alias that can be mapped  
to something else later on when you know the actual URL. Actually,  
this can already be accomplished at engine level. I believe a simple  
scripting like above would just make it practical. What do you think?

Cheers,
Doru


On 23 Feb 2010, at 21:18, Dale Henrichs wrote:

--
www.tudorgirba.com

"Every thing has its own flow."




_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Doru,

That is a good idea. It might also make sense from a Pharo perspective to have a global fallback defined. I am expecting that the mechanisms that we come up with will be useful for individual projects as well as Pharo overall, so building in flexibility at load time (instead of embedded in the configuration) is a good idea.

To summarize my takeaway on this issue:

  - A pharo backup or secondary repository is a good solution to ensure
    that configurations and packages are "guaranteed" to survive over long periods
    of time regardless of the state of the primary project repositories

  - The secondary repository will be maintained by running a scraper script that
    scans the Metacello configurations that are critical to Pharo.

  - The script will copy the packages referenced by the configurations into the
    secondary repository. The configurations will also be copied into the
    repository.

  - Whether or not the configuration copies are modified to reference the secondary
    repository directly or not is still undefined.

  - This issue will be managed as a separate collection of packages to keep the core
    of Metacello clean.

  - The initial short term goal will be to provide a mechanism to do a "one-shot"
    checkpoint of configurations and packages (no configuration modification) so
    that a Pharo 1.0 repository of all of the important build artifacts can be
    archived in the event that the longer term goals aren't met in time. This
    initial short term goal will be based upon code already in use at GemStone to
    perform a similar function.

  - The second phase will be to work out the details of using the repository in
    real life conditions ... and at this point there are a number of possibilities
    under consideration.

  - More details when we get this far:)

If I've missed something important, I'm sure you'll let me know:)

Dale
----- "Tudor Girba" <[hidden email]> wrote:

| Hi Dale,
|
| I believe the original problem is that you do not know upfront about
|
| where the actual files will be. In this case, perhaps the easiest  
| solution is to provide some scripting facility to provide a repository
|  
| mapping that should could be modified at loading time, and not be  
| stored in the actual configuration.
|
| Suppose you have something like this:
|
| ConfigurationOfXXX>>baselineX: spec
| <version: 'X-baseline'>
| spec package: 'SomePackage' with: [ spec repository: 'SOME_URL' ].
| ...
|
| ConfigurationOfXXX>>versionX: spec
| <version: 'X' imports: #('X-baseline')>
| spec package: 'SomePackage' with: 'SomePackage.tg.1'.
| ...
|
|
| Maybe a solution would be that when loading you could say:
|
| (ConfigurationOfXXX project version: 'X')
| withRepositoryMapping: 'SOME_URL' -> 'SOME_NEW_URL';
| withRepositoryMapping: '...' -> '...';
| load
|
| What do you think?
|
| In this way, the original configuration specifies everything, and the
|
| URL works either as the actual URL or as an alias that can be mapped
|
| to something else later on when you know the actual URL. Actually,  
| this can already be accomplished at engine level. I believe a simple
|
| scripting like above would just make it practical. What do you think?
|
| Cheers,
| Doru
|
|
| On 23 Feb 2010, at 21:18, Dale Henrichs wrote:
|
| >
| > ----- "Lukas Renggli" <[hidden email]> wrote:
| >
| > | > Perhaps Gofer can be provided with a secondary repository
| location
| > | in the case when the primary repository fails.
| > |
| > | Yes, it can. You just specify multiple repositories and it will
| use
| > | them all. If you tell Gofer to #disableRepositoryErrors then it
| will
| > | silently ignore repositories that are not available.
| >
| > Lukas,
| >
| > The only thing that worries about including such repository all of
|
| > the time, is the cost of building the cache of repository when  
| > resolving repositories ... the secondary repository would get pretty
|  
| > large over time and most of the time (only in the case of errors)  
| > would it need to be scanned.
| >
| > I suppose Metacello could handle load errors in the code that calls
|
| > Gofer and do a retry using the secondary repository...the secondary
|
| > repository _could_ be specified as part of the configuration, so  
| > this probably isn't a bad idea...
| >
| > Thanks,
| >
| > Dale
| >
| > _______________________________________________
| > Pharo-project mailing list
| > [hidden email]
| > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
|
| --
| www.tudorgirba.com
|
| "Every thing has its own flow."

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project