Packages, Packages, Packages

Folks -

Here is the long overdue post about packages and how to manage them. One
of the things that comes directly out of Edgars and my desire to unload
a few more packages, is the question of what to do with them. That
raises the question of how to deal with external packages in general. So
here we go with some thoughts on both.

We have various package management systems that all fall short in some
respect. First, there is Monticello, our source code management system.
Monticello works reasonably well for maintaining individual source
packages but is awfully lacking in two core areas: Managing dependencies
and being findable. MCs "builtin" dependency mechanism is just awful;
configurations work somewhat better but neither address "higher level"
dependencies such as "this package depends on Seaside 2.8" (the only way
to express this is by listing all of Seaside's packages explicitly).
Secondly, MC is generally not indexable by Google which is a fatal flaw
from my point of view (because how the hell are you going to find a
package if you can't google for it?)

Our second candidate is Squeakmap. Of all the solutions, Squeakmap is
currently the only one that is indexable by Google; a major advantage.
However, Squeakmap doesn't have dependencies, rendering it difficult to
use for non-experts. Squeakmap also has the disadvantage that it's
pretty dated by now and has been in need of a serious face lift since
its inception :-) (that said I do like it and would rather like to see
it revived)

The third candidate I'm aware of is Universes. Universes is the only
candidate here which has dependencies that actually work reasonably
well. In addition, universes can be used off-line so it would be
feasible to ship a "standard library" with Squeak for people to download
and install (a little bit like Python's site-packages). The downside of
Universes is that they're not indexable and their user interface is by
far the most awful and unusable of all the proposals.

I think that's all the serious contenders at this point (if I'm missing
something please let me know). From my perspective, I don't care which
solution we choose as long as it fits the bill of:
* Being indexable by Google
* Reasonable dependency management
* Having a usable UI
Since none of the solutions above currently fits all the requirements
we'll have to fix some. That could mean adding dependencies to
SqueakMap; it could mean giving Universes a reasonable UI; it could mean
fixing dependencies in Monticello. I really don't care either way but I
think we'll have to figure something out here. Is anyone out there
interested in or perhaps even working on addressing any of these issues?

The second big issue is how we deal with "core" vs. "extended" packages
in Squeak. As we move towards a smaller kernel, we should know which
packages are core and which ones are not. And when we remove packages
that are currently part of the image, we need a place to put them.

In addition, we really need to start thinking about the packages that a
"standard" Squeak should contain. Personally, I feel that one of the
best decisions David and I made in Croquet was having the small base
image be tucked away in the distribution so that people who want to
build something small have a starting point but with the "standard"
build being a fully featured set of packages that contained everything
we wanted to show.

I think the approach is perfectly applicable to Squeak - we need to move
the "core" image aggressively forward; every release should make it
significantly smaller. And while we're doing this we should have a
"standard" image (not necessarily a kitchen-sink image with *everything*
thrown in) but the packages that we think your average Squeak user would
be interested in seeing.

If people agree with the direction of aggressively shrinking the core
image as well as figuring out what the default image should contain then
our next steps should be:
* Figure out where to place unloaded packages (we could leave those in
trunk if we wanted to but I'm not certain that's the right place;
perhaps Edgar's Ladrillos could work?)
* Figure out a build process that we can apply to the core image to
build the default image automatically
* Discuss and decide which packages should be part of the standard
Squeak version
* Unload some more packages!

So far my thoughts on the issue. Feedback is very welcome, I think we
need to move forward on this set of issues before the next release goes out.

Cheers,
   - Andreas

Just two thoughts: 

Chris Cunnington wrote:
> You've outlined the technical merits of the different systems, but I
> think that the community has chosen Monticello. My experience looking
> for an RFB that worked demonstrated to me that it's the only system
> people take seriously for updating. A sudden change in direction to SM
> or Universes is going to jar the largest number of people, I would imagine.

Without implying any judgment here this would mean that we really have
to fix dependencies in Monticello for the style of use we're after here.
By which I mean that in order to express "this package requires Seaside
and Tweak and Croquet" we can't expect people to list all the 500 MC
packages that make up Seaside, Tweak and Croquet.

> What about parcels? Elliot was saying some extremely compelling things
> about them recently. Is that a non-starter or too far in the future?

I'm specifically trying to avoid starting yet another package management
system. Our problem isn't that we have none, our problem is that we have
too many and that none of them works for the situation that we're
looking at. So I'm in strong favor of taking one of the existing
solutions and just fix them.

Cheers,
   - Andreas

On Fri, Dec 11, 2009 at 11:14 PM, Andreas Raab <[hidden email]> wrote:


Maybe more attention to the real uses of the technology
outside the "core" squeak is needed here.
Since some months ago, Dale Henrichs has made available Metacello
that doesn't tries to "fix" monticello, but to build on top of the
limitations of it
by treating monticello as what does better: to load a single version of a given
package in the image. Then it adds the apropriate "package" dependencies in
a higher level. No need to feature creep monticello but use it as a sound base
for solving the issues you mentioned.

>From my perspective, I don't care which
> solution we choose as long as it fits the bill of:
> * Being indexable by Google
> * Reasonable dependency management
> * Having a usable UI

I think that you are mixing responsibilities here. The same way that
Git in the FOSS
world doesn't need to be "google indexed" to be useful, the tool
needed by squeak
doesn't need to have all of these capabilities. Different tools can
fulfill the requeriments
you mention.

You can have
- monticello to manage package source code and conflict managment.
- metacello to manage package dependencies
- "some tool" to serve as a web hub (a la github) to store the
metainformation for metacello
to work.

> Since none of the solutions above currently fits all the requirements we'll
> have to fix some. That could mean adding dependencies to SqueakMap; it could
> mean giving Universes a reasonable UI; it could mean fixing dependencies in
> Monticello. I really don't care either way but I think we'll have to figure
> something out here. Is anyone out there interested in or perhaps even
> working on addressing any of these issues?

Please, let behind that "squeak for all" mentality. The right tool for
the right job.
One tool (squeak or any other) can't solve all the problems.

And respect to squeak map/universes and "species" please, please, they are
in this almost 2010, plain prehistoric. Don't let them live out of
their time (they were
useful I am sure, ten year ago) by adding more features to some old codebase.
Seaside is a marvelous tool, and can be used to build something more "user
attractive" (again, a la github).

>
> The second big issue is how we deal with "core" vs. "extended" packages in
> Squeak. As we move towards a smaller kernel, we should know which packages
> are core and which ones are not. And when we remove packages that are
> currently part of the image, we need a place to put them.

Discussend long time ago in Pharo. Some useful ideas can be gathered
from there,
even if the project name produce some bad feelings.

>
> In addition, we really need to start thinking about the packages that a
> "standard" Squeak should contain. Personally, I feel that one of the best
> decisions David and I made in Croquet was having the small base image be
> tucked away in the distribution so that people who want to build something
> small have a starting point but with the "standard" build being a fully
> featured set of packages that contained everything we wanted to show.

Same here.

>
> I think the approach is perfectly applicable to Squeak - we need to move the
> "core" image aggressively forward; every release should make it
> significantly smaller. And while we're doing this we should have a
> "standard" image (not necessarily a kitchen-sink image with *everything*
> thrown in) but the packages that we think your average Squeak user would be
> interested in seeing.

Same here.

I will risk my own permanence in the squeak mailing list because I
could be banned
by being labeled as troll or pharo gruppie. This is an honest
question. Why don't you
consider or at least follow the pharo discussion, as I see it now,
Squeak is walking
the same path pharo has already left behind. Same discussions about future,
about core vs  full distributions, about external packages, about
package management,
heck even about the today discussion about the image update number!

So the question is, has ever been considered to simply build the bridge between
communities and to use the PharoCore image as the base for Squeak?

I want to express my respect for the work made in Squeak (but also the work made
in Pharo) and overall the work of the people integrating changes from
one project
in the other no matter where the change originated. This is amazing.

But I am more than disappointed to see this discussion taking place
here as if the
world were only Squeak and the knowledge of the outside events and ideas weren't
worth even the time to having a look in them.
I am active in pharo, but always read the squeak list because I want
to be informed
with the activities, goals, technology and overall, people involved.
This by itself is
very valuable to me.

>
> So far my thoughts on the issue. Feedback is very welcome, I think we need
> to move forward on this set of issues before the next release goes out.

What release. This means that the 3.11 (that millions of times were
stated as being
only about the process and not about a new squeak release) is over and squeak
is heading to the next (4.0) release?

This doesn't align with the stated goals for squeak. Has the board a
new roadmap for
squeak?

>
> Cheers,
>  - Andreas
>

Very respectful and wanting to avoid a flame war,
--
Miguel Cobá
http://miguel.leugim.com.mx

On 2009-12-11, at 9:29 PM, Chris Cunnington wrote:

> You've outlined the technical merits of the different systems, but I think that the community has chosen Monticello. My experience looking for an RFB that worked demonstrated to me that it's the only system people take seriously for updating. A sudden change in direction to SM or Universes is going to jar the largest number of people, I would imagine.

No no no no no!

Monticello is *not* a package management system. It's a package versioning system. These are very different things. Consider the analogues in the Linux world. Monticello is a versioning system like git. What we need is a package management system like apt.

Colin

Miguel Cobá wrote:
> Maybe more attention to the real uses of the technology
> outside the "core" squeak is needed here.
> Since some months ago, Dale Henrichs has made available Metacello
> that doesn't tries to "fix" monticello, but to build on top of the
> limitations of it by treating monticello as what does better:
> to load a single version of a given package in the image.
> Then it adds the apropriate "package" dependencies in
> a higher level. No need to feature creep monticello but use
> it as a sound base for solving the issues you mentioned.

I do not understand which problem domain Metacello is addressing and
how. If you have more information, this would be very welcome.

I'm not claiming it can. I'm laying out a couple of issues that we need
to address, and list the current options as far as I  understand them.
The only thing I'm strongly against is to end up with another dependency
on a tool that hasn't been written yet and that by the end of the day
will the job just as incomplete as the alternatives.

>> The second big issue is how we deal with "core" vs. "extended" packages in
>> Squeak. As we move towards a smaller kernel, we should know which packages
>> are core and which ones are not. And when we remove packages that are
>> currently part of the image, we need a place to put them.
>
> Discussend long time ago in Pharo. Some useful ideas can be gathered
> from there, even if the project name produce some bad feelings.

Can you summarize the results of that discussion?

> So the question is, has ever been considered to simply build the bridge between
> communities and to use the PharoCore image as the base for Squeak?

This wouldn't work for the same reasons that it wouldn't work to use a
Squeak-trunk image as the basis for Pharo. You should propose that at
some point just to see what kind of reaction you get ;-)

Cheers,
   - Andreas

On Sat, Dec 12, 2009 at 1:43 AM, Andreas Raab <[hidden email]> wrote:
http://www.google.com.mx/search?hl=en&source=hp&q=metacello&btnG=Google+Search&aq=f&oq=

http://gemstonesoup.wordpress.com/2009/08/25/metacello-package-management-for-monticello/

Quoting from the blog post:

"
Metacello: Package Management for Monticello

August 25, 2009 in Deployment, Metacello, porting

Rate This

Quantcast

[1]

Metacello is a package management system for Monticello. A package
management system is…

…a collection of tools to automate the process of installing,
upgrading, configuring, and removing software packages from a
computer. Distributions of Linux and other Unix-like systems typically
consist of hundreds or even thousands of distinct software packages;
in the former case a package management system is nice, in the latter
case it is essential.

Packages are distributions of software and metadata such as the
software’s full name, description of its purpose, version number,
vendor, checksum, and a list of dependencies necessary for the
software to run properly. Upon installation, metadata is stored in a
local package database.

A package management system provides a consistent method of installing software…

Last April I finally became fed up with the lack of a decent package
management system for Monticello and decided to write my own from
scratch. I wanted a package management system for Monticello that was
consistent with the important features of Monticello:

    * Declarative modeling. A Metacello project has named versions
consisting of lists of explicit Monticello package versions.
Dependencies are explicitly expressed in terms of named versions of
required projects. A required project is a reference to another
Metacello project.
    * Distributed repositories. Metacello project metadata is
represented as instance methods in a class therefore the Metacello
project metadata is stored in a Monticello package. As a result, it is
easy for distributed groups of developers to collaborate on ad hoc
projects.
    * Optimistic development. With Monticello-based packages,
concurrent updates to the project metadata can be easily managed.
Parallel versions of the metadata can be merged just like parallel
versions of the code base itself.

Additionally, the following three points are important considerations
for Metacello:

    * Cross platform operation. Metacello must run on all platforms
that support Monticello: currently Squeak, Pharo and GLASS.
    * Conditional Monticello package loading. For projects that are
expected to run on multiple platforms, it is essential that
platform-specific Monticello packages can be conditionally loaded. At
the moment, conditional loading is specified based upon the following
attributes:
          o #common. Code that is common across all platforms.
          o #squeakCommon. Code that is common to Squeak and Pharo.
          o #squeak. Code that is specific to Squeak.
          o #pharo. Code that is specific to Pharo.
          o #gemstone. Code that is specific to GemStone.

      It should be possible to inject project-specific attributes, so
code that is dependent upon attributes not covered by the standard
list can be conditionally loaded. For example, in GLASS, different
Monticello package versions are loaded based on which version of
GemStone/S is running (i.e., version 2.0 versus 3.0).
    * Compatible with MC2. It must be possible to manage Metacello
projects that are based on alternate Distributed Source Code
Management systems like Monticello2.
    * MIT license."

I think that is not "another tool" but it is created from a vendor (Gemstone)
 to address a real problem.
So, it will get more attention than a altruist-generated tool (because of
costumer pressure). This is good for all.
Also, in GLASS, Metacello is used for loading the next version 3 of Seaside
and all of dependencies.
Also, Mariano Martínez Peck et al have started to create
configurations for all the
packages that are part of the dev distribution of PharoCore.
This will allow an apt-get behavior in installing packages.


Well, core will always tend to be smaller, right now, PharoCore 1.0 rc1 is 11 MB
in size. All the other packages are being extracted to own
squeaksource repositories.

Also, a list of packages to be included in PharoCore and a list of
packages to be
included in dev and web distributions are available in the wiki part
of the Pharo
project.


>
>> So the question is, has ever been considered to simply build the bridge
>> between
>> communities and to use the PharoCore image as the base for Squeak?
>
> This wouldn't work for the same reasons that it wouldn't work to use a
> Squeak-trunk image as the basis for Pharo. You should propose that at some
> point just to see what kind of reaction you get ;-)
>


Well, but it is only now that squeak has reached the point where it
could be considered
a core part of other distributions. Pharo has advantage here. It has
several months
being a small core image that could be used as the infrastructure for
other forks, as
sometimes ago was proposed in the squeak list.
So, yes, maybe squeak trunk is approaching to a core image with everything added
as external packages, but it remains some steps more to be there: packages
tested as working, release policy, release process, documentation for new users
and what to expect when updating, how to use the core squeak, a new
updated page,
update documentation, tutorials, a book, promotion, blogs, articles,
the resolution about
the license, etc.

> Cheers,
>  - Andreas
>
>

Cheers,
--
Miguel Cobá
http://miguel.leugim.com.mx

On 2009-12-11, at 9:14 PM, Andreas Raab wrote:

> * Figure out a build process that we can apply to the core image to build the default image automatically

I might be able to help with this one. I've written a build tool (called Mason) that could be used to automatically build a core image into a standard release image. The feature that differentiates Mason from most other build tools is that it doesn't have to be loaded into the image it's creating. Instead, it generates chunk-format scripts and runs them in a base image using OSProcess. I've been using it to build SARs, development images, HTML documentation, etc. I also use it to run SUnit test suites against various base images, and do inter-package dependency testing.

The specification of what to build uses a protocol similar to Seaside's Canvas protocol, to build a tree of targets. Here's a short example:

specifyOn: targets
        targets directory
                name: 'MasonExample';
                with:
                        [targets file
                                name: 'README';
                                content: self readMeContents.
                        targets image
                                baseImage: self squeakTrunkImage;
                                name: 'example';
                                with: [targets version name: 'Filesystem-cwp.27' repository: self repository]].

This creates a directory called "MasonExample." Inside is a file called README, and an image (with changes) called example.image. That example.image is based on the Squeak Trunk Image and has Filesystem-cwp.27 loaded into it.

I'll tidy things up for release tonight or tomorrow.

Colin

Andreas Raab wrote:

> I think that's all the serious contenders at this point (if I'm missing
> something please let me know). From my perspective, I don't care which
> solution we choose as long as it fits the bill of:
> * Being indexable by Google
> * Reasonable dependency management

My experience with RPMs, RPMs+Synaptic, APT, Portage and many others is
that reasonable dependency management is sometimes unreasonable. When
that happens, what will save you is if "being indexable by Google" means
the capability to find out which package has a particular file. In our
case, what we need badly right now is, given the name of some missing
class, figure out what to load into our image to fix that.

Of course, given the kinds of names that Squeak packages tend to have,
"being indexable by Google" also helps if it means being able to type
"squeak multiple windows" and have the first hit be "Areithfa Ffenestri"
http://wiki.squeak.org/squeak/3862

-- Jecel

2009/12/12 Miguel Cobá <[hidden email]>:
I am clueless, why people eagerly jumping into the boat in one case,
but prefer to stay away from it in another.
Something irrational in this..

The Sake and Bob [1] projects were announced more than a half year
earlier than Metacello, but there are few people who ever took time to
evaluate it (including me).
But sure, why we would need to ever look at it, if we got a new shiny toy?

I have nothing against Metacello project , neither beloved with
Sake/Bob solution(s). I know very little about them.
But if i would need a packaging tool, i'd first consider alternatives
& look for existing tools and evaluate them before jumping with closed
eyes.

[1] http://wiki.squeak.org/squeak/5953

--
Best regards,
Igor Stasenko AKA sig.

2009/12/12 Jecel Assumpcao Jr <[hidden email]>:
+100. We need just a simple web server which can give us the plain
simple answer - the list of packages which including class with given
name.
This is all what i need most of the times (when loading stuff using
MC), because i can easily solve the rest using existing tools.
So if we could have it, the need of having & developing/adopting
highly advanced dependency management tools will be not an issue.
Keep it simply stupid.

Moreover, by having such global registry, where people could register
their packages, we also can deal with another issue: class names
clashing. Developer, before publishing the package could simply check
if given class is already used by someone else, and choose another
name.

> Of course, given the kinds of names that Squeak packages tend to have,
> "being indexable by Google" also helps if it means being able to type
> "squeak multiple windows" and have the first hit be "Areithfa Ffenestri"
> http://wiki.squeak.org/squeak/3862
>
> -- Jecel
>


--
Best regards,
Igor Stasenko AKA sig.

At 8:14 PM +0200 12/12/09, Igor Stasenko apparently wrote:
Thx for bringing that up, I cannot understand it either.

Look here for some further info I've saved over time re: Sake/Bob etc.
It seems to me that Keith had always talked sense regarding the way forward.
Maybe Andreas will take a look at Sake and friends now.

<http://squeaktipsandtrickswatch.blogspot.com/>

Ken G. Brown

On Fri, Dec 11, 2009 at 11:43:21PM -0800, Andreas Raab wrote:
Project home with good summary description:
  http://code.google.com/p/metacello/
Another good reference:
  http://gemstonesoup.wordpress.com/2009/08/25/metacello-package-management-for-monticello/
Much discussion here:
  http://groups.google.com/group/monticello-dev/topics

I have not personally used it, but Metacello seems to be well
thought out, with documented project objectives and a competent and
considerate developer. It is MIT licensed, and support for Squeak
is an explicit design objective. I think this is worth serious
consideration.

>From the project page:

Metacello is a package management system for Monticello that is
consistent with the important features of Monticello:

 * Declarative modeling. A Metacello project has named versions
   consisting of lists of explicit Monticello package versions.
   Dependencies are explicitly expressed in terms of named versions
   of required projects. A required project is a reference to
   another Metacello project.
 * Distributed repositories. Metacello project metadata is represented
   as instance methods in a class therefore the Metacello project
   metadata is stored in a Monticello package. As a result, it is
   easy for distributed groups of developers to collaborate on ad hoc
   projects.
 * Optimistic development. With Monticello-based packages, concurrent
   updates to the project metadata can be easily managed. Parallel
   versions of the metadata can be merged just like parallel versions
   of the code base itself.

Additionally, the following three points are important considerations
for Metacello:

 * Cross platform operation. Metacello must run on all platforms that
   support Monticello: currently Squeak, Pharo and GLASS.
 * Conditional Monticello package loading. For projects that are
   expected to run on multiple platforms, it is essential that
   platform-specific Monticello packages can be conditionally loaded.

2009/12/12 David T. Lewis <[hidden email]>:
Full agree.

On 2009-12-12, at 10:14 AM, Igor Stasenko wrote:

> The Sake and Bob [1] projects were announced more than a half year
> earlier than Metacello, but there are few people who ever took time to
> evaluate it (including me).
> But sure, why we would need to ever look at it, if we got a new shiny toy?

I don't know if Dale looked into Sake and Bob, but they wouldn't have fulfilled his needs in any case. Metacello needs to run on Gemstone as well as Squeak, which Sake and Bob don't.

Before writing Mason, I did look into Sake and Bob. Sake doesn't meet one of my core needs, which is that the image being built does *not* need to have the build tool loaded. This provides a lot more flexibility in terms of what can be built, and Sake doesn't provide that.

Colin

>
> I am clueless, why people eagerly jumping into the boat in one case,
> but prefer to stay away from it in another.
> Something irrational in this..

I'm not jumping blindly.
But even after several people bashing Keith about the _documentation_
for using it
(and the sade comment from himself about not open sourcing Bob) that is a
closed way for many.
It is responsibility of the creator of the software to build the documentation.
And not, the freaking squeak wiki isn't a good documentation. Tutorials,
blog posts, real use scenarios are most than needed. The dry, one after
other list of functionality that is not inmediatly obvious how to use and
_where_ and _when_ to apply can't get many followers.

>
> The Sake and Bob [1] projects were announced more than a half year
> earlier than Metacello, but there are few people who ever took time to
> evaluate it (including me).
> But sure, why we would need to ever look at it, if we got a new shiny toy?
>

That is not true, but Dale cared to made documentation, blog posts and
indeed to promote its own tool and that gave him an advantage an several
followers.
Maybe Sake/Bob is genial, but we'll never know and, with time, maybe
Metacello could get the same functionality also. Why, because there are
more people interested in it just by the simple fact that it has digerible
documentation.


Cheers,
Miguel Cobá
http://miguel.leugim.com.mx

> Before writing Mason, I did look into Sake and Bob. Sake doesn't meet one of my core needs, which is that the image being built does *not* need to have the build tool loaded. This provides a lot more flexibility in terms of what can be built, and Sake doesn't provide that.
>
> Colin
>
>
>

But that is a project decision. The case for pharo, they could simply
choose to include the packages to load more packages in the core image
and walk ahead. Isn't necessarily a problem to not have a pristine
only kernel image. At least in my production servers I can have
*those* packages installed, the same way that a minimal Debian server
must have apt-get/aptitude installed in order to be *real* useful.
In this internet age, is very valuable to download everything with
just a command in a remote server. If that means to have some more
packages in the core image, that is ok with me.

Cheers,
Miguel Cobá
http://miguel.leugim.com.mx

On 12/12/09 7:44 PM, "Miguel Cobá" <[hidden email]> wrote:


I don't know if Mason could do the trick, but as I said in the K era, prefer
Wine with Maria instead Sake and Bob :=)

And Colin have all my faith


Edgar

On 2009-12-12, at 1:44 PM, Miguel Cobá wrote:

The problem is that you not only need the builder in your image, you need all of its dependencies. If the builder relies on Monticello, you need that loaded. Depending on how the packages you're loading are developed, you might need also Monticello 2, or Metacello. You also need the base image to be compatible with those dependencies. That's what led Keith to develop LevelPlayingField - a way to bring various base images up to the level of compatibility required by Sake and its dependencies.

I chose to take a different route, and have the builder live in it's own image, and produce a separate image with absolutely minimal dependencies. Mason can, for example, build a functional image based on Pavel's minimal image. To get that base image to run Sake, you'd have to build it back up to something resembling Squeak 3.9... which kind of defeats the purpose.

Now, if you don't mind having the builder loaded in the image, then I'm sure Sake is adequate. If it meets your requirements, by all means use it. But if you want to know why I wrote Mason rather than just using Sake, this is why.

Colin