Packages, Packages, Packages

On 12/12/09 4:02 PM, "Chris Cunnington" <[hidden email]> wrote:

One of the differences between Seaside 2.8.3 and Seaside 3.0 is that the packages must be loaded in a specific order. You cannot install them piecemeal.. That's why builder.seaside.st <http://builder.seaside.st>  was created. It looks as though Dale's abstracting this problem and solving it with Metacello. I think that we should look at this project very carefully. I think Miguel is right to say SM and Universes are not really in our future. 

@Colin

I remember reading Edgar saying he was using Monticello for delivering files. And you said he should be using sar. You say it's not a package management system, but a package versioning system. As usual, your posts make clear - starkly clear - distinctions, but I think you're missing something. The intent of what you created is irrelevant. How people use the tool shows its actual value to the community. If that's not how you envisioned it, then that doesn't matter. If I want to use a butter knife as a screwdriver to fix the toaster, and it works, then that's all that matters. 

Whether you admit it or not, Monticello is used as a package management system. And yes, that kind of sucks. The day I don't have to go to SqueakSource to get the proper path name for a project I want to install, and I can just do it from my image like the SMPL or Universes, then that will be a good day. 

Chris 

It’s why I wish get rid of all SqueakMap, Universes, Installer,ScriptLoader, for two years now.
We could improve Monticello.
I could put any into a “package” and load into regular SqueakSource and use as it was .sar.
But the way I also remember years of people fighting me as I always like .sar .

Metacello is the answer ?

Edgar

I found my own solution to this problem. Simple as smalltalk is.

Create an MC package in your project, named
MyProject-loader

and tell users to load this package if they want to install the project.
The rest of stuff about what load , in what order and using what means
is solved easily by putting a code into the

MyProjectLoader class>>initialize.

Keep it simply stupid.

2009/12/12 Edgar J. De Cleene <[hidden email]>:


--
Best regards,
Igor Stasenko AKA sig.

On Sat, Dec 12, 2009 at 12:39 PM, Igor Stasenko <[hidden email]> wrote:
I don't know towards who was this offense directed but a "real"
package management code
isn't stupid nor its intent to build makes the people stupid.

About your example, it has a lot of shortcommings that the package
manager solve.

It must be changed every time that a new package in your dependencies
change. E.g. if
your package OwnPackage dependes on PackageA version 2 and PackageB
version 3 you somehow
hardcode those versions on your initialize method.
If fow any reason, the mantainer of PackageA found a bug and fix it on
version 3, suddently your
package is using the wrong version. If you happen to be on holidays or
unreachable, the package won't
be updated to use the new correct version 3 for PackageA.
If a million people install your package before you update your
configuration, a million people will be using
the wrong versions.

What a package manager (as Metacello does) is to depende on PackageA
(no matter what version is currently,
although you can also specify a given version or a version greater or
lower than a given version).
This way no matter how many new versions the mantainer of PackageA
does, your Package doesn't need
to be updated each time and can always work.

Cheers,

Miguel Cobá
http://miguel.leugim.com.mx

2009/12/12 Miguel Cobá <[hidden email]>:
I meant no offense at all.
http://en.wikipedia.org/wiki/KISS_principle

Look, same applied to <put your preferable> dependency management tool.
There is virtually nothing which you can't solve using code, which
executed directly and doing what you, as maintainer, it should do.
So, i don't see any shortcomings of such approach comparing to some
dependency management tool.
A tool also written by someone and contains code which does something.
What makes you assume that code written by
maintainer is automatically worse than code written for that tool?


> What a package manager (as Metacello does) is to depende on PackageA
> (no matter what version is currently,
> although you can also specify a given version or a version greater or
> lower than a given version).
> This way no matter how many new versions the mantainer of PackageA
> does, your Package doesn't need
> to be updated each time and can always work.
>

It is all up to you, whether update it or not. I don't see anything,
which i can't do by simply implementing it in my code,
putting my own knowledge and preferences how i think this should be
done best for installing my software.

> Cheers,
>
> Miguel Cobá
> http://miguel.leugim.com.mx
>
>



--
Best regards,
Igor Stasenko AKA sig.

> It is all up to you, whether update it or not. I don't see anything,
> which i can't do by simply implementing it in my code,
> putting my own knowledge and preferences how i think this should be
> done best for installing my software.
>


Ok, and that is precisely the mindshare that keeps squeak relegated to its
current "selfish, self-satisfied, proud, no friends" niche.
Lets have an open mind to new tools, squeak can't remain reinventing everything
from scratch. That "non-invented-here" syndrome is mortal.

2009/12/12 Miguel Cobá <[hidden email]>:
You are taking my words too literally.
Im not opposed to using existing tools. I just wanted to point out
that simpler is better.
If a simple installer script works in 99% of cases for me, then why i
need to care about using something else?
>From now, i don't see much need in a complex build scripts.
Most of installation configs i seen were a plain list of packages,
denoting the order in which they should be loaded.
This is not something, which requires installing advanced frameworks,
other than installer.
And of course, if i would need something more complex for installation
of my project,
then i may consider to use different tool and use it in my code.

>>> Cheers,
>>>
>>> Miguel Cobá
>>> http://miguel.leugim.com.mx
>>>
>> --
>> Best regards,
>> Igor Stasenko AKA sig.
>>


--
Best regards,
Igor Stasenko AKA sig.

Igor Stasenko wrote:
> Im not opposed to using existing tools. I just wanted to point out
> that simpler is better.
> If a simple installer script works in 99% of cases for me, then why i
> need to care about using something else?

You're confusing an installation script with a package management
system. A package *includes* an installation script; but it also
includes dependency and other information that is utilized by the
package management system. In unix terms, the former is the makefile,
the latter is the rpm with its spec file. You're saying you don't need
an rpm and in many cases you're right. But in many other cases you're not.

The biggest issue with Metacello I'm seeing so far is that it muddles
the installation script with dependency management and metadata. Very
troublesome because it means you can't even *look* at a Metacello
configuration from Squeak 3.8 or ealier. Extremely disturbing
considering that in order to do dependency analysis one must run
untrusted code to obtain the dependencies (so forget about server-side
dependency analysis). And unfortunate because it means that Metacello
has hardwired assumptions about its interactions with the rest of the
environment (Gofer, OB etc) making it implausible to replace its
underpinnings with something much simpler like Installer. To be honest,
Metacello has been somewhat of a disappointment so far.

Cheers,
   - andreas

> The biggest issue with Metacello I'm seeing so far is that it muddles the
> installation script with dependency management and metadata.

But that is the same with the .spec file from a RPM package. They have both
the metadata about the package (packager, date, release number, version, etc),
the install scripts (pre/post install/uninstall scripts for things to
configure in the host
system). I can't see the problem here.

Very
> troublesome because it means you can't even *look* at a Metacello
> configuration from Squeak 3.8 or ealier.

Why would you want to do that. 3.8 is dead, must be dead and should be kill.
Let the past behind. Let your childs live their lives and go on. You
can't oversee
them *your* entire life. Squeak can't remain trying to get backwards
compatibility
forever. That way squeak will never make the foundation changes that needs.
If only one person uses 3.8 (or any old version, 3.8 is just the
example) but 100
are waiting to get a new feature in trunk that is non-backwards compatible with
some random old version you're screwing 100 people by caring about 1.
And don't say that productions systems depends on those old versions.
If that were
the case we would still be using fortran or cobol for programing Excel
or OO.org Calc.
The systems can change, it only the people that don't want to change.


 Extremely disturbing considering
> that in order to do dependency analysis one must run untrusted code to
> obtain the dependencies (so forget about server-side dependency analysis).
> And unfortunate because it means that Metacello has hardwired assumptions
> about its interactions with the rest of the environment (Gofer, OB etc)

Yes, but that is the compromise. You choose your tools and build some new tool.
Maybe Installer isn't the right tool for building Metacello. Maybe
because Gofer didn't exist
Metacello didn't existe before.
Remember, new knowledge leads to even more new knowledge. Tools are multipliers
of the ideas and producers of new ideas not imagined before.

> making it implausible to replace its underpinnings with something much
> simpler like Installer. To be honest, Metacello has been somewhat of a
> disappointment so far.
>
> Cheers,
>  - andreas
>
>
>

Cheers,
Miguel Cobá
http://miguel.leugim.com.mx

Miguel Cobá wrote:
>> The biggest issue with Metacello I'm seeing so far is that it muddles the
>> installation script with dependency management and metadata.
>
> But that is the same with the .spec file from a RPM package. They have both
> the metadata about the package (packager, date, release number, version, etc),
> the install scripts (pre/post install/uninstall scripts for things to
> configure in the host system). I can't see the problem here.

You can parse the RPM spec and do a good bit of analysis without running
the install or any other scripts. Yes, at some point there are scripts
involved in the installation, but I don't need to execute a shell script
*before* I can find out that this package (for example) has a build
requirement of kdebase3-devel. Tell me how I find out anything about a
Metacello package without running it.

>> Very
>> troublesome because it means you can't even *look* at a Metacello
>> configuration from Squeak 3.8 or ealier.
>
> Why would you want to do that. 3.8 is dead, must be dead and should be kill.

The point is not about 3.8. The point is that because Metacello insists
on using Smalltalk code, the package definitions are inherently tied to
very specific language and environment versions. If you use a language
construct in your package that's new, no older version will be able to
read it. If your package uses a method that was removed it won't load
into a newer version. One of the main points about package management
systems is that they are supposed to be robust - Metacello seems very
brittle because it inherently depends on the environment that the
description was created in. Five years from now, you will not be able to
analyze the package configurations you created today. Because
*something* will have changed and break that code and as a consequence
the entire package.

Perhaps so, but Metacello feels bloated. It just doesn't feel right to
use that much code for something that simple.

Cheers,
   - Andreas

On 2009-12-12, at 10:02 AM, Chris Cunnington <[hidden email]> wrote:

> The intent of what you created is irrelevant. How people use the tool shows its actual value to the community. If that's not how you envisioned it, then that doesn't matter. If I want to use a butter knife as a screwdriver to fix the toaster, and it works, then that's all that matters.

Sure, you can do whatever you want with my work! That's why I release it under the MIT license, so others can come up with novel uses for it. But this isn't using a butter knife to tighten a screw, it's using a sponge to tighten a screw. If you somehow manage to get the screw tightened, it's not thanks to the sponge.

Monticello doesn't manage packages. It loads packages, yes. But package *management* requires more than that. The folks who are supposedly using Monticello to manage packages are in fact managing them by hand, and using MC to load them. That's fine, a package management system may well use MC to do the actual loading of packages. But let's not confusing loading with management.

If you look at SqueakMap and Package Universes, they *do* provide package management. SqueakMaps lacks dependency metadata, and Package Universes lacks a web interface. They may in adequate for the task at hand, but they're at least attempts to solve the problem facing us.

Colin

On Sat, Dec 12, 2009 at 06:01:55PM -0600, Miguel Cob? wrote:
>
> Why would you want to do that. 3.8 is dead, must be dead and should be kill.
> Let the past behind. Let your childs live their lives and go on. You
> can't oversee them *your* entire life. Squeak can't remain trying to
> get backwards compatibility forever. That way squeak will never make the
> foundation changes that needs.

<rant>
  This is complete utter bullshit. Failing to make a reasonable attempt
  at backward compatibility is lazy and sloppy, nothing more.
 
  Norton's third law of innovation:
    Change is good, stupid is bad. Don't confuse the two.
</rant>

Ok I feel better now. There are some rather significant forks based on
Squeak 3.8, and any package management system that fails to accomodate
this level of variation is in trouble. SqueakMap fell apart when Squeak
3.9 came along for this very reason, and we can reasonably expect that
future variations on Squeak will introduce similar incompatibilities.
This is a good thing, and our tools do need to be flexible enough to
support it.

Dave

On 2009-12-12, at 3:29 PM, Andreas Raab wrote:

> The biggest issue with Metacello I'm seeing so far is that it muddles the installation script with dependency management and metadata. Very troublesome because it means you can't even *look* at a Metacello configuration from Squeak 3.8 or earlier. Extremely disturbing considering that in order to do dependency analysis one must run untrusted code to obtain the dependencies (so forget about server-side dependency analysis). And unfortunate because it means that Metacello has hardwired assumptions about its interactions with the rest of the environment (Gofer, OB etc) making it implausible to replace its underpinnings with something much simpler like Installer. To be honest, Metacello has been somewhat of a disappointment so far.

This was my reaction to Metacello as well. Nevertheless, I've played with using Metacello to specify configurations of OmniBrowser 2.0 and Monticello 2.1. I find that the model Metacello uses is quite nice - I was able to capture all the details of the relationships between the dozens of packages involved, and use that data to do some fairly nifty things with Mason. It was incredibly frustrating to get Mason to do the dependency analysis and give me back the results - essentially I had to get it to do a fake load and then examine what would have been loaded.

So, yeah, the implementation of Metacello leaves a lot to be desired, but the abstract model it's based on is excellent.

Colin

Colin Putney wrote:
> On 2009-12-12, at 3:29 PM, Andreas Raab wrote:
>
>> The biggest issue with Metacello I'm seeing so far is that it muddles the installation script with dependency management and metadata. Very troublesome because it means you can't even *look* at a Metacello configuration from Squeak 3.8 or earlier. Extremely disturbing considering that in order to do dependency analysis one must run untrusted code to obtain the dependencies (so forget about server-side dependency analysis). And unfortunate because it means that Metacello has hardwired assumptions about its interactions with the rest of the environment (Gofer, OB etc) making it implausible to replace its underpinnings with something much simpler like Installer. To be honest, Metacello has been somewhat of a disappointment so far.
>
> This was my reaction to Metacello as well. Nevertheless, I've played with using Metacello to specify configurations of OmniBrowser 2.0 and Monticello 2.1. I find that the model Metacello uses is quite nice - I was able to capture all the details of the relationships between the dozens of packages involved, and use that data to do some fairly nifty things with Mason. It was incredibly frustrating to get Mason to do the dependency analysis and give me back the results - essentially I had to get it to do a fake load and then examine what would have been loaded.

Would you mind pointing me to the configurations of OB 2.0 and MC 2.1?
It would be interesting to look at an example in context.

Cheers,
   - Andreas

On 2009-12-12, at 9:03 PM, Andreas Raab wrote:

> Colin Putney wrote:
>> On 2009-12-12, at 3:29 PM, Andreas Raab wrote:
>>> The biggest issue with Metacello I'm seeing so far is that it muddles the installation script with dependency management and metadata. Very troublesome because it means you can't even *look* at a Metacello configuration from Squeak 3.8 or earlier. Extremely disturbing considering that in order to do dependency analysis one must run untrusted code to obtain the dependencies (so forget about server-side dependency analysis). And unfortunate because it means that Metacello has hardwired assumptions about its interactions with the rest of the environment (Gofer, OB etc) making it implausible to replace its underpinnings with something much simpler like Installer. To be honest, Metacello has been somewhat of a disappointment so far.
>> This was my reaction to Metacello as well. Nevertheless, I've played with using Metacello to specify configurations of OmniBrowser 2.0 and Monticello 2.1. I find that the model Metacello uses is quite nice - I was able to capture all the details of the relationships between the dozens of packages involved, and use that data to do some fairly nifty things with Mason. It was incredibly frustrating to get Mason to do the dependency analysis and give me back the results - essentially I had to get it to do a fake load and then examine what would have been loaded.
>
> Would you mind pointing me to the configurations of OB 2.0 and MC 2.1? It would be interesting to look at an example in context.

The OmniBrowser configuration is here:

        http://source.wiresong.ca/ob/OB-Dev-Metacello-cwp.2.mcz

It's a little out of date with respect to Metacello; it hasn't been updated to use the latest version of the specification API.


The Monticello 2.1 configuration is here:

        http://source.wiresong.ca/mc/ConfigurationOfMonticello2-cwp.6.mcz

This one is up-to-date. I use it with Mason to run the test suites in isolation. I have a Mason product that finds all the test packages, and loads each one into a virgin trunk image along with all its dependencies. Then it runs the tests and writes the results out to a file. If there are hidden or unspecified dependencies, the tests will fail horribly. :-)

Colin

On Sat, Dec 12, 2009 at 6:56 PM, Andreas Raab <[hidden email]> wrote:

> You can parse the RPM spec and do a good bit of analysis without running the
> install or any other scripts. Yes, at some point there are scripts involved
> in the installation, but I don't need to execute a shell script *before* I
> can find out that this package (for example) has a build requirement of
> kdebase3-devel. Tell me how I find out anything about a Metacello package
> without running it.

But that is not true, in fact the repositories are prepared on package
upload so that
every download is first an update and then an install/upgrade.
The update just synchronizes the client db (a file with the latest
package list and
package dependencies and "provides" hints for the dependencies solver)
with the updated
server list.
Then goes ahead and do a dependency solving algorithm. Doesn't download any
package to do this, it uses the meta information got from the spec file uploaded
and concentrated in this db and does its job.
The same can be made in squeak with metacello. On upload time, the
server software
in charge of the repository can analyze the configuration of the
package and generate
a new db list with the apropriate meta information.
When a cliente image updates its own lists get this new db and then proceed to
solve the dependency graph. When a solution is found, then downloads the
appropriate package versions for the package installed and all its dependencies.

Complex, maybe, for the developer of the client and server software.
But for the user is just a simple do it in a workspace. For example, from a core
image, just open a workspace and put (pseudo code):

Metacello update;
 install: 'Seaside'.

And this will grow the core image to a full image with lots of
packages installed
automatically.


The same applies to formats that grow over time without a version number
indicating what version is this format. Just open a new excel file in
a very old excel
install. Nobody spect it to work. The same for trying to run a
Makefile from 2010 in
a unix from 1980.
 If
> your package uses a method that was removed it won't load into a newer
> version. One of the main points about package management systems is that
> they are supposed to be robust - Metacello seems very brittle because it
> inherently depends on the environment that the description was created in.
> Five years from now, you will not be able to analyze the package
> configurations you created today. Because *something* will have changed and
> break that code and as a consequence the entire package.

Granted, but also, but it is the same that happens right now with squeak map and
squeak source abandoned packages. They no longer work in squeak or
pharo. They need
modification to run in newer images. That is something that we accept.
I don't believe in
the idea of coding ones and forever. That is imposible. Even the trunk
right now is better
that the squeak version it started from. That surely will render
several packages broken.
But it is good, because the image is more healthy. And the price for
the health is to
invest some time in adapting old code to the new environment.


Well, then we'll wait for the next marvelous thing that
squeak/smalltalk is going
to provide to show the world how dumb they are for using something that works
now instead of waiting for the lightweight, fast, non-bloated, well
engineered tool you'll
have tomorrow, or the next month, or ...


>
> Cheers,
>  - Andreas
>
>

Miguel Cobá
http://miguel.leugim.com.mx

>>
>> Would you mind pointing me to the configurations of OB 2.0 and MC 2.1? It would be interesting to look at an example in context.


You can also take a look on the project that was created to host the
configurations for several packages already:

MCHttpRepository
        location: 'http://www.squeaksource.com/MetacelloRepository'
        user: ''
        password: ''

there are few projects right now because the project was started a
couple of days ago.
Also, the configurations are being refined everyday but you can see
working examples of using
metacello.

Cheers,
Miguel Cobá
http://miguel.leugim.com.mx

On Sat, Dec 12, 2009 at 9:55 PM, David T. Lewis <[hidden email]> wrote:
I think that that is the real problem with the solution provided by Keith.
He tries to solve the problem for every fork. As if the forks care
about its changes
being compatible with the project they forked from.
Keith mail after mail tried to convince people that a change must be/should be
loaded in every fork existing and future. Utopia. That don't happens.
If I choose Pharo and the software fix that I made works in pharo, my problem is
solved. I couldn't care less if croquet, etoys, olpc or whatever can use it.
The same they don't give a dime if their changes works on squeak.
As an old discussion in this list pointed, the etoys project long ago
forget squeak.
Nevertheless, squeak is worried if their changes could somehow
break/work/load/compile
in squeak. :(

SqueakMap fell apart when Squeak
> 3.9 came along for this very reason, and we can reasonably expect that
> future variations on Squeak will introduce similar incompatibilities.
> This is a good thing, and our tools do need to be flexible enough to
> support it.

The only real example of code that works unmodified in newer hosting
environment is
in the zSeries Operating systems from IBM.
They say (I don't know and maybe will never could verify because I
don't have the millions
to buy a mainframe from them) is that you can run software from the
60s in the 2010 machines
without a single line modified. That is impressive but they charge
accordingly also.
We aren't so well funded nor have a real practical motivation to
verify that every new version
of squeak or pharo can run _every_ old version of _every_ package that
some time ran on
squeak. If someone want to keep using that old software they have two options:

- remain part of the every day smaller group of users (and what that
implies for bug fixing, etc)
- load the old software in the new images and correct the errors found

I choose number 2 because is what happend in the nature. The groups
are stronger than the
individuals.

>
> Dave
>
>
>

Miguel Cobá wrote:
What is not true? That the client doesn't have to run (untrusted) code
to find out about dependencies? Are you saying that rpm runs code that
some joe created with the package just to find out about the
dependencies? That would be news to me.

> The same can be made in squeak with metacello. On upload time, the
> server software in charge of the repository can analyze the configuration
> of the package and generate a new db list with the apropriate meta information.

And that just goes back to my point about why it's a problem to require
  untrusted code to run to do that. You are willing to run arbitrary
code on your server *just* to be able to get to the dependency
information? The day where someone doesn't like you they'll upload some
configuration that nukes your server, runs it as a spam bot or whatever.
The desire to do server-side analysis is precisely why requiring code to
express the dependencies is a bad idea.

> When a cliente image updates its own lists get this new db and then proceed to
> solve the dependency graph. When a solution is found, then downloads the
> appropriate package versions for the package installed and all its dependencies.

If this actually worked, it would certainly be a major step up from the
current situation. Because it would mean a client can look at the
information of the package without running that code. A 3.8 image could
read that information and tell the user that this package only works in
3.9 or later. Which is a major improvement compared to just breaking.

>> Five years from now, you will not be able to analyze the package
>> configurations you created today. Because *something* will have changed and
>> break that code and as a consequence the entire package.
>
> Granted, but also, but it is the same that happens right now with squeak map and
> squeak source abandoned packages. They no longer work in squeak or
> pharo. They need modification to run in newer images. That is something
> that we accept.

No, "we" don't accept this. You do. Don't speak for others. And this is
about the meta information, not the actual package contents. If
Monticello would have taken the same stance it would've never gotten
where it is today. MC is what it is precisely because it works in
*every* Squeak version under the sun, and precisely because it has an
explicit, declarative model for its contents (instead of requiring code
to run to produce that model). As a consequence, MC is for example
capable of parsing traits definitions even in systems that don't have
traits. That ensures great robustness, and a level of interoperability
without which it wouldn't have become the SCM of choice for Squeak.

>> Perhaps so, but Metacello feels bloated. It just doesn't feel right to use
>> that much code for something that simple.
>
> Well, then we'll wait for the next marvelous thing that squeak/smalltalk
> is going to provide to show the world how dumb they are for
> using something that works now instead of waiting for the lightweight,
> fast, non-bloated, well engineered tool you'll have tomorrow,
> or the next month, or ...

Aren't you mistaking a few things here? This discussion started with me
laying out the options of the *existing* solutions and what would need
to be done to make one of them work. Then you came along selling the
shiny new stuff.

Cheers,
   - Andreas

2009/12/13 Andreas Raab <[hidden email]>:
Personally, i don't see the uttermost need of having dependency
information on server.
Code, that runs in specific environment, where you want to install a
project is the place where you can
do dependency analysis 100% correctly, and this is why the code there
can and should be run.
What is the purpose of having it on server side? Display nice web page?
Cmon.. you just need to click and load that project. Nothing else.




--
Best regards,
Igor Stasenko AKA sig.