[squeak-dev] Perl is to CPAN as Squeak is to (what)?

Since I promised the folks in #squeak I would start this thread...

It's been my observation for quite a while that Squeak is missing
the equivalent of what Perl has in the CPAN.

First, let me describe the CPAN:

Anyone can get a CPAN ID for free.  Once they have a CPAN ID, they upload
packages to a central server, which are then indexed and replicated to 500+
places worldwide, usually within an hour.

A package has a name and a version number.  Version numbers are floating point
numbers, with "larger = better".  If the version number contains an
underscore, it's considered "better, but only beta" and not automatically
offered as an upgrade (see below).

In general, package names are *not* registered, meaning that if I upload
Foo-3.4, and you upload Foo-3.5, users around the world will be asked if they
want to upgrade my package with yours.  By default, this allows cooperating
team members to upgrade new releases of a cooperative project, as long as they
work together on the version numbers.  This also allows the
long-gone-developer's code to be updated in-absentia.  Although this has the
possibility for abuse, in the 15 years of the CPAN it has only been abused a
few times, and the CPAN gods can quickly remedy the situation.

For more managed care, a package name or prefix can be *registered*, which
means only a small group of individuals can upload things in that prefix.
Again, this is rare, but solves the wildcard issue.  If a package is
abandoned, the CPAN gods can transfer the registration or deregister it.

Any CPAN package can declare that it is dependent on another CPAN package, or
even particular versions of core-installed packages.  The CPAN installer can
be told to automatically or manually install all dependencies (or even ignore
them, but that's rarely wise).

Speaking of which, the core Perl distro has had a CPAN installer for many
years (CPAN.pm), and even a newer one (CPANPLUS.pm) has been included in
recent releases.  As long as the CPAN index can be read, and the packages
downloaded, anyone can write additional CPAN installers too, but they probably
won't go into the core.

As a result of the above rules:

* CPAN installation is easy
* CPAN publishing is easy
* The CPAN is vital and thriving - see search.cpan.org/recent

Notice about 50 uploads a *day*.  And everything *works* in an
automated way, except for the "CPAN gods" exceptions listed above.

Now.  Squeak has been around only half as long as Perl.  But that's about
how long Perl was around before we came up with the CPAN, so it's about
time Squeak had the same thing, I think.

Here's what I know (and I could be wrong):

Squeakmap - *everything* is "registered", so cooperation is hard
Squeaksource - no index is made, so you have to find things yourself
Universes - like Squeakmap
Sake/Packages - single point of update, making publishing hard

And except for Squeakmap, there's none of these that have been in all of the
recent Squeak releases for installation, so we have a bootstrapping problem.
Even installing Universes is hard from Squeakmap. :)

And all of these systems are missing dependency and automatic upgrade
management.

Can we discuss something like the CPAN for Squeak?

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[hidden email]> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

+1

On Jun 26, 2008, at 9:35 PM, Randal L. Schwartz wrote:

>
> Since I promised the folks in #squeak I would start this thread...
>
> It's been my observation for quite a while that Squeak is missing
> the equivalent of what Perl has in the CPAN.

Yes, that is indeed something that is missing, and I think we need it  
badly.

I agree that not registering is better.

> As a result of the above rules:
>
> * CPAN installation is easy
> * CPAN publishing is easy

Lowering this barrier in Squeak is important

Merik

 > Can we discuss something like the CPAN for Squeak?

I like the idea. From a practical point of view it seems that relaxing
the rules at SqueakMap such that "registered" access becomes an option
seems the most realistic approach. In addition, one might consider
creating a breed of SM and Squeaksource by having SM automatically index
SqueakSource projects.

Cheers,
   - Andreas

Randal L. Schwartz wrote:

I would like to speak up for Package Universe.

I have always disliked SqueakMap and I like Package Universe.  The
problem with SqueakMap is that there is no dependency information.  I
can't just select a package in SqueakMap and file it in and expect it
to work.  I can do that with Package Universe.  With SqueakMap, I have
to read comments and try to figure out what else needs to be filed in.
 I have found PU to be much easier to use than SM.  I haven't
published anything with either system, so by "to use" I mean "to get
packages so I can run them" not "to publish packages".

Of course, SM could be fixed.  I can't see what would be hard, so I
wonder why it hasn't been done.

A "feature" of Squeak that might differentiate it from perl is that
there are many versions of Squeak.  Packages that work with one image
might not work with another.  So, if you are using 3.8 then you do not
necessarily want the latest version of some package if that package
only works in 3.9 or 3.10.   PU addresses this problem by having a
different Universe for each version of the image, though I think there
is currently a Universe only for 3.9 and 3.10.  Even if the code for
PU could run in earlier images, it isn't useful until someone builds a
Universe for it.

This could be handled in SqueakMap by saying that you only load the
versions of packages that have been registered to work with your
image.

In theory, PU is much different from CPAN because it limits who is
allowed to maintain a package.  In practice, lots of people are able
to do it.  Existing Universes have had a large set of people who can
change them, and pretty much anybody who wanted to do it, could.  So,
in practice that aspect of CPAN isn't much different from Universes.

CPAN seems to be distributed, whereas each Universe is on a single
server. Plus, Universes (as with SqueakMap) do not actually store the
source code, but just have pointers to it.  Most users can't tell the
difference, though.

-Ralph

>>>>> "Ralph" == Ralph Johnson <[hidden email]> writes:

Ralph> A "feature" of Squeak that might differentiate it from perl is that
Ralph> there are many versions of Squeak.  Packages that work with one image
Ralph> might not work with another.  So, if you are using 3.8 then you do not
Ralph> necessarily want the latest version of some package if that package
Ralph> only works in 3.9 or 3.10.   PU addresses this problem by having a
Ralph> different Universe for each version of the image, though I think there
Ralph> is currently a Universe only for 3.9 and 3.10.  Even if the code for
Ralph> PU could run in earlier images, it isn't useful until someone builds a
Ralph> Universe for it.

It would be nice if it would work like SM where SM at least says "this
might break", and lets you try anyway.

I think the UI for PU is also pretty "early", to put it nicely.  At least
I can *search* in SM!

Maybe PU is the closest to CPAN we have.  So the question is, do we keep
mutating PU into what I'm looking for?  Or do we first need to agree that the
CPAN might be a pretty darn good model?

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[hidden email]> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

CPAN is a killer feature for perl.

I've done projects in perl simply because of CPAN.
I found libraries etc I needed there quickly and easily,
no hunting around and said, good enough and banged out
the work.

I would love to see a CPAN for Squeak....

On Thu, Jun 26, 2008 at 4:26 PM, Randal L. Schwartz
<[hidden email]> wrote:

> Maybe PU is the closest to CPAN we have.  So the question is, do we keep
> mutating PU into what I'm looking for?  Or do we first need to agree that the
> CPAN might be a pretty darn good model?

It might be easier to add dependency information to SqueakMap.  I
don't have an opinion about that.  I just wanted to point out the
features of PU that made me prefer it.  I'd be happy if every Squeak
packaging system had those features.  And if there was a single system
that had the best of all possible features.

CPAN is probably a good model.  However, a model that works well for
one language does not always work well for another.  How would you
think a CPAN-like system would handle the fact that people are using
3.6, 3.7, 3.8, 3.9 and 3.10 and that some packages work for all of
these, but some packages are specialized for a particular image?

-Ralph

>>>>> "Ralph" == Ralph Johnson <[hidden email]> writes:

Ralph> CPAN is probably a good model.  However, a model that works well for
Ralph> one language does not always work well for another.  How would you
Ralph> think a CPAN-like system would handle the fact that people are using
Ralph> 3.6, 3.7, 3.8, 3.9 and 3.10 and that some packages work for all of
Ralph> these, but some packages are specialized for a particular image?

Well, that's why I'm liking Universes now. :)

We do need something that says package X works with squeak Y and package X
depends on package Z.  so it's two dimensions, not just one.

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[hidden email]> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

On Thu, Jun 26, 2008 at 02:49:09PM -0700, Randal L. Schwartz wrote:
SqueakMap already does this: packages are marked as being stable
in certain image versions. Universes has no such mechanism

--
Matthew Fulmer -- http://mtfulmer.wordpress.com/

Am 26.06.2008 um 23:08 schrieb Ralph Johnson:

> Universes (as with SqueakMap) do not actually store the
> source code, but just have pointers to it.  Most users can't tell the
> difference, though.


Actually, SqueakMap nowadays caches packages, because URLs got stale  
too often.

I agree opening up is a relatively cheap and promising way to make  
contributing easier.

- Bert -

But you can search also in PackageUniverse.
I agree with Ralph. I gave up trying somethig on SM because it broke  
on me.

Stef

On Jun 26, 2008, at 11:26 PM, Randal L. Schwartz wrote:

I have a naive question
why Squeaksource/MC + config map does not work.

You can build map of components and sveral people can work on the same  
project?

Stef


On Jun 27, 2008, at 1:07 AM, Bert Freudenberg wrote:

I do not understand your question. An MC configuration is simply a  
list of several MC package versions, to be loaded together, plus a  
list of repositories to find the versions. It's an alternative to the  
MC dependency mechanism, which for our needs was too inflexible. It's  
also TSTTCPW so it might lack a great deal of sophistication.

- Bert -

Am 27.06.2008 um 09:10 schrieb stephane ducasse:

Hi all!

Being the principal developer of SM I thought I should "chip in", note
though that I have NOT read this thread and I am going to Mallorca
tomorrow. :)

Mmm, anyone can register on SM. One of the primary ideas with SM was to
not only keep track of packages but also of developers.

SM does have a very simplistic upload facility but it was mostly added for
hosting simple scripts.

And of course, SM lacks signatures etc. And there is no replication of
packages themselves - except for the server keeping a server side cache
(not very well known), but it was on the other hand designed more like a
"freshmeat" than a "cpan" - the map is downloaded and cached locally
inside your image.

This was btw also a primary idea - the fact that the whole map is a true
object model that you can easily use with normal Smalltalk. I still find
that aspect quite nice, even though the monolithic nature is something I
have wanted to move away from for several years.

> A package has a name and a version number.  Version numbers are floating
> point
> numbers, with "larger = better".  If the version number contains an
> underscore, it's considered "better, but only beta" and not automatically
> offered as an upgrade (see below).

SM keeps track of releases and each release has both an automatic version
number and a manual one. The automatic one is not that good (doesn't
support branching that good actually).

> In general, package names are *not* registered, meaning that if I upload
> Foo-3.4, and you upload Foo-3.5, users around the world will be asked if
> they
> want to upgrade my package with yours.  By default, this allows
> cooperating
> team members to upgrade new releases of a cooperative project, as long as
> they
> work together on the version numbers.

SM instead has a concept of co-maintainership - a maintainer can add other
developers as co-maintainers and they can more or less do the same stuff
as the maintainer can.

SM also has the ability to transfer packages. Hmmm, can't recall right now
if that is offered in the UI.

>  This also allows the
> long-gone-developer's code to be updated in-absentia.  Although this has
> the
> possibility for abuse, in the 15 years of the CPAN it has only been abused
> a
> few times, and the CPAN gods can quickly remedy the situation.

If someone wants to co-maintain it is IMHO quite easy to contact the
maintainer and get added as a co-maintainer. If the maintainer does not
answer - you can always talk to the SM "gods" = being me more or less.

But I very seldom get such calls so I am not sure this is a BIG problem
(yet).

> For more managed care, a package name or prefix can be *registered*, which
> means only a small group of individuals can upload things in that prefix.
> Again, this is rare, but solves the wildcard issue.  If a package is
> abandoned, the CPAN gods can transfer the registration or deregister it.

Again, transfers are easily done in SM too. This is also why it has a
separate Authors-field, in order to give credit to the original authors
despite the current maintainers.

> Any CPAN package can declare that it is dependent on another CPAN package,
> or
> even particular versions of core-installed packages.  The CPAN installer
> can
> be told to automatically or manually install all dependencies (or even
> ignore
> them, but that's rarely wise).

There is no dependency management in SM today. I started one model,
another guy did one too (can't recall his name) - and I asked him if he
could just add it and then take responsibility for maintaining it -
because I didn't quite grok the code - but there was no such interest at
the time.

I have repeatedly asked for help in adapting SM and adding dependencies,
but so far only a very few individuals have expressed interest in helping
out. Please, please, please raise your hand! :)

It would be great to:

- Clean the model, make it distributed and also make it "bidirectional" so
that client images can also "upload" model changes and thus open up for
Morphic UIs for maintaining packages (and not just the web UI).
- Smack a new Seaside good lookin UI on top of the model.
- Add dependencies to the model.
- Make it aware of Installer, Universes and Squeaksource
- Add free text indexing. In Gjallar we already have nice code for using
Swish-e.

...and of course there are tons of other things we could do.

[SNIP]
> Here's what I know (and I could be wrong):
>
> Squeakmap - *everything* is "registered", so cooperation is hard

Mmmm, not sure what the above means.

Hehe, of course. Ever since I created SM we *have* been discussing it AND
making things happen. When SS came around I expressed fears that it would
"compete" with SM. People didn't really agree - but I think we can clearly
see it today. Universes came about in order to attack lack of dependencies
in SM and I have repeatedly argued for merging Universes/SM but without
really getting any traction for that idea.

These days I don't have THAT much time to share - but IF there were a few
individuals standing up - I would definitely help out making something
happen. It just takes 2 raised arms. Or perhaps even just one. :)

regards, Göran

Hi again!

Ok, read the thread and wanted to add a few things:

Dependencies
============
Adding dependency management to SM would not be hard. There have been one
successful attempt (though, don't ask me about the code) and there have
been several proposed models. My model may have been too "complex", but
eventually I came to the point where I said "Let's take ANY model".

Why it has not been done? Well, because only one person did it, and that
person didn't have time/energy/whatever to pull it all the way (not me).

I perhaps ended up trying to fix "too many things" in one go - so I got
stuck in "analysis/design paralysis". :)

Patches
=======
I personally would like to keep the "map" separate from the "format" and
the actual SCM/package tools. So SM handles many different formats, not
just MC. For patches I would like Deltas (as in DeltaStreams) to replace
ChangeSets.


There is one aspect of SM that I think people may miss:

Since SM can install .st-files .cs-files, mcz-files etc etc - there is
nothing really stopping us from adding new "kinds" of packages. For
example, I have toyed with the idea of adding a "Universe" package. The
package content itself would be a complete Universe map. Thus we could
have a group of co-maintainers of that Universe and they could make new
releases of it. The Universe itself would be simply a maintained "subset"
of SM packagereleases that work in a given environment - say a specific
image or whatever.

Distribution
============
The new model I have sketched on for SM is to go "distributed" like
apt-get etc. The concept of storing partial map information typically
using "dumb servers" and having the SM client code in the image "merge
them together". This would enable private SM catalogs, company wide SM
catalogs etc etc.

It would also enable "pushing" changes up to the map instead of going
through a centralized web UI. Thus much easier to make a Morphic UI (or
whatever) to *maintain* entries in SM.


Anyway, tons of ideas, tons of good things we can do. Just one thing
missing, and it is *always* the same thing:

 ** Noone actually steps up and offers to help in doing the work on SM to
make it happen!**

Except for Brian who did a really nice workover on the package loader UI,
and a few more things. :)

regards, Göran

El 6/27/08 9:19 AM, "Göran Krampe" <[hidden email]> escribió:

> I have repeatedly argued for merging Universes/SM but without
> really getting any traction for that idea.


+1

I always like Universes/SM  merge , but as usual time is to short.
Web interface is HttpView2 ?
If so, I  think a single way to way is to have some like

http://map1.squeakfoundation.org/sm/packages3dot8
http://map1.squeakfoundation.org/sm/packages3dot9
http://map1.squeakfoundation.org/sm/packages3dot10
http://map1.squeakfoundation.org/sm/packages3dot11
http://map1.squeakfoundation.org/sm/packagesSpoon
http://map1.squeakfoundation.org/sm/packagesCroquet
http://map1.squeakfoundation.org/sm/packagesOLPC
etc

I wish polish and have SqueakLightII out so too busy, but could help
testing.

Edgar

Randal L. Schwartz wrote:

> Since I promised the folks in #squeak I would start this thread...
>
> It's been my observation for quite a while that Squeak is missing
> the equivalent of what Perl has in the CPAN.

Yes.

*snip CPAN description*

> Any CPAN package can declare that it is dependent on another CPAN package, or
> even particular versions of core-installed packages.  The CPAN installer can
> be told to automatically or manually install all dependencies (or even ignore
> them, but that's rarely wise).

Updating the CPAN installer itself can be a bit of a PITA, though. I
have one server at work where updating CPAN is simply not working.

*snip*

I think It would be a start to agree on one of the four above mentioned
systems, like CST has it with the public store (whatever problems STORE
itself might have aside).

The absence of something like STORE/CPAN that is what is driving me
towards CST.

The existence of CPAN is keeping me with Perl ;)

>
> The absence of something like STORE/CPAN that is what is driving me
> towards CST.
>
> The existence of CPAN is keeping me with Perl ;)
The existence of sake/packages to enable me to consistently build and
debug the building of the production image that I want is helping to
keep me sane! (after attempting to achieve what I needed with Universes
drove me mad for a couple of weeks)

The last thing we need is yet another package management system, when we
have hardly started using the features of the existing ones.

Oh yes it is all very well to go and drum up political support for
various ideas and complain that it will not handle the needs of millions
of coders, but the reality is that it is a lot of hot air! (see next
paragraph)

The [hidden email] list was recently re-rejuvenated
to address this very issue, and as far as I am aware we had 1 more
member subscribe to it as a result (and one member left). Hardly a sign
of a community desparate to actually contribute to make things better
for all, do you think?

Furthermore, the goals chosen by the current informal release team is to
work in ways which will support the disparate communities that are
emerging based upon squeak. (olpc etoys, sophie, croquet etc) Part of
this involves championing tools that have been designed to actually work
in all squeak derivatives.

Unfortunatelty none of these "other" communities have shown much
interest as yet. The invitation for volunteers to be a package-czar for
any version of squeak is still very much open.

==

Since the full list of actual dependencies relies upon use-ontext rather
than publication-context, having a central repository as the sole place
to manage dependencies is going to be useful but only to a limited extent.

I would personally prefer SqueakMap without dependencies, or at least if
it had them sake/packages would enable the defaults to be overridden.

I think we could re-write SqueakMap to be a front end manager of
Sake/Packages. If we did this then the dependencies could be managed to
resolve to the correct data entries. But having said that, who has the
time to actually do the work.

For those highlighting of lack of documentation for Sake/Packages , it
is true I have had very very little time since writing it... however it
DOES have class comments, and I have posted numerous examples and emails
in the past 5 months or so, so you could do worse than search the mail
archives for my posts.

best regards

Keith