Package Universes and Squeakmap

I have been using package universes for a few days, and have discovered
some of its limitations. I have also been wondering how best to
integrate its use with SqueakMap.

We already have SqueakMap as the master repository and Lex, the creator
of package universes does not seem to think that simply tagging items in
SqueakMap according to what works with what and what doesnt would work
very well, all things considered. To a certain extent I now agree with him.

I initially thought that having Universes separate from SqueakMap was a
bad idea. However, after using universes I have now changed my mind and
think that is a good idea. I imagine SqueakMap to be the Model, and
Universes to be the View, with the community acting as the controller so
to speak.

One of the limitations of Universes is that it does not have much in the
way of integration with Squeakmap or Monticello, and so it looks to be
attempting to make SqueakMap obsolete, but it is somewhat of a poor
replacement. At present it is totally agnostic to the package provider,
it simply loads a package from a url based upon the file's extension.

One of the positive things about SqueakMap is that it caches the
packages, it also has more information on it than the universe. I think
that it too would benefit from some feedback features, and more
openness, and some indication of dependencies, but perhaps it makes
sense not to have these as hard and fast rules. For example, Seaside
specifies Kom, but some users may prefer Swazoo. This flexibility seems
difficult to support prescriptively on the macro level, but could be
supported in local mini/sub-universes, such as a "Swazoo users seaside
universe".

Perhaps SqueakMap could allow anyone to upload a new version, but
reserve "blessing" of releases to the owner/maintainers, since this is
how a public squeaksource repository works by "social" default.

One Universe limitation is that when using a Universe package, unlike an
installer script which integrates with the native monticello tools, the
package is loaded, but the Monticello browser is not informed of the
repository details.

This means that you have to manually inform the Monticello Browser of
the required repository before a monticello package with dependencies
referred to in a package universe will work. For example when Chris
posted  'Magma Server' he would have had the Monticello Browser properly
configured on posting the package to the Universe and it will work for
him but it will not work for users out of the box who do not have the
Monticello repositories configured.

So thinking about it this it would not be a problem at all if Package
Universes replaced:

    http://www.squeaksource.com/Magma/MagmaServerLoader-cmm.25.mcz

with:

    Installer squeaksource project: 'MagmaTester'; install:
'MagmaServerLoader-cmm.25'.

So why not have Package Universes call installer for SqueakMap items too.

    Installer squeakmap install: 'MyPackage(1.0)'.

(I will fix installer to fall back onto using the http interface
#websqueakmap if SMSPackageLoader is not present)

By doing this we can uphold the use of SqueakMap as the place to publish
released packages, with Universes being the place to define dependencies
within a locally defined context/scope/use scenarios.

If the "publish to squeakmap" button could be fixed on squeaksource the
picture would be complete.

This approach also allows mantis fixes and change sets to be included in
the Universes without any extra work.

thoughts feedback?

best regards

Keith







I think that having universes, together with sub-universes and
personal-local universes

2007/7/26, Keith Hodges <[hidden email]>:
> One of the positive things about SqueakMap is that it caches the
> packages, it also has more information on it than the universe. I think
> that it too would benefit from some feedback features, and more
> openness, and some indication of dependencies, but perhaps it makes
> sense not to have these as hard and fast rules. For example, Seaside
> specifies Kom, but some users may prefer Swazoo. This flexibility seems
> difficult to support prescriptively on the macro level, but could be
> supported in local mini/sub-universes, such as a "Swazoo users seaside
> universe".


This is already supported by Universe but I don't know if it works:

Seaside
  requires: WebServer

Swazoo
  provides: WebServer

Kom
  provides: WebServer

The 2 fields exists but I do not know if they are used by the system or not.



I don't think Universe should follow Monticello dependencies. The
developer must put everything to the Universe and it should work. Have
a look at OmniBrowser, Nile, Magritte... in the universe.


--
Damien Cassou

"Damien Cassou" <[hidden email]> writes:
It does not, but it should, if we need that kind of dependency now.

-Lex

Lex Spoon wrote: Implementation that makes use of "provides" is in verision of universes
in http://www.squeaksource.com/Installer Also support for monticello
configurations is included.

Keith

Keith Hodges <[hidden email]> writes:
> > It does not, but it should, if we need that kind of dependency now.
> >
> > -Lex
> >
> Implementation that makes use of "provides" is in verision of
> universes in http://www.squeaksource.com/Installer Also support for
> monticello configurations is included.

First, that's cool that you looked at the problem.  Did you modify
UGlobalInstaller, or did you make a replacement for it, or what?

Second, doesn't that mean the above version is incompatible with the
standard universes toolkit?  This means it is problematic to have that
version, if it is, in the development universe.  Packages in the same
universe should work nicely with each other.


-Lex

 
>> Implementation that makes use of "provides" is in verision of
>> universes in http://www.squeaksource.com/Installer Also support for
>> monticello configurations is included.
>>    
>
> First, that's cool that you looked at the problem.  Did you modify
> UGlobalInstaller, or did you make a replacement for it, or what?
>
>  
Monticello already has a mechaism for mapping file extensions to reader
classes. So I simply implemented the method below to use this mechanism.
I also ensured that the mcm reader was compatiable with this invocation.
The same scheme should also work for Installer.

In Monticello1.5 ;-) There is also a reader for files of extension
'.mcs' , if you give a change-set the extension .mcs instead of .cs then
Monticello will load it, rather than the standard .cs reader. This will
be useful when monticello supports proper atomic loading. The rationale
for the new extension is simply that if you switch to using monticell ot
load .cs files then any non-basic cs files, with preambles etc dont
work. Having .mcs explicitly defined avoids any confusion.
 

UIMonticello-installFileNamed: filename

    | reader |

    Smalltalk at: #MCReader ifPresent: [ :top |

        reader := top readerClassForFileNamed: filename
       
    ].
    reader ifNil: [ MczInstaller installFileNamed: filename ].
   
    ^ reader loadVersionFile: filename

> Second, doesn't that mean the above version is incompatible with the
> standard universes toolkit?  This means it is problematic to have that
> version, if it is, in the development universe.  Packages in the same
> universe should work nicely with each other.
>  
The latest Magma beta's are being distributed as a monticello
configuration. Chris and I agreed that maintaining dependencies within
basic monticello without MonticelloConfigurations was too much of a
headache.

So if you want to load it with Universe then this little fix is needed
to the Universes client.

I just put a couple of mods into a Universe in the Installer repo, in
the hope that you might pick up on the desirable changes at some point.

best regards

Keith

Regarding Package Universes I have two points I have noticed using the
3.10 Universe:

1. Packages should have information such as purpose and maintainer.
   At the moment there seems to be no such information. Instead it is
repeated for every version of the package.
2. It would be nice if version information would contain release date
and comment. As there is no
    common release number system the release date would be an
interesting information.

Regards
Andreas

+1.  It would be nice if there was a spot that said who actually wrote it even if I have to type it in.

"Jason Johnson" <[hidden email]> writes:
> +1.  It would be nice if there was a spot that said who actually wrote it
> even if I have to type it in.

There is a spot for home page, which should give all the information
that is relevant.  Of course, you can also list the author name in the
description if you want -- if there even is a single identifiable
author.

Anyway, I plan to add a link to the SqueakMap entry.  SqueakMap seems
like a more appropriate place to add all the bits of meta-data you
might ever want, including authorship and time stamps.



> > 1. Packages should have information such as purpose and maintainer.
> >    At the moment there seems to be no such information. Instead it is
> > repeated for every version of the package.

The maintainer and description can and do change asn time goes on, so
it seems better to relist them for each version.

Most of the time, anyway, you are only interested in the most recent
version of a package.


Lex

"Jason Johnson" <[hidden email]> writes:
> Aren't universes and Monticello configurations in the same space?  If so, do
> you guys have any interest in just switching to Universes for your package
> management so we don't have to maintain so many different ways of doing the
> same thing?  The software can stay simpler (and therefor easier to maintain)
> if each package (e.g. Universes) don't have to support every possible
> format.


I have always had the same question.  With a universes approach, you
would set up a package universe that includes the blessed versions of
each individual Monticello package.  Then you would have a virtual
package that depends on all of the individual ones.  Installing that
virtual package would get you the latest blessed version of each
individual package.


In theory it is less general.  However, I find micro-managing
individual package versions to be too fine-grained.  Plus, if your
individual packages are actually meaningful on their own, other
projects will have their own dependencies on the individual packages,
and you really want to avoid the headache of solving jigsaw puzzles
like "A 1.2 depends on B version 1.5, but C 1.2 depends on B version
1.4, so maybe I can stick with C 1.1, which depends on..." etc. etc.


Anyway, if upstream authors use MCM's, then either PU needs to support
MCM's or one of the universe maintainers needs to post the individual
packages.  I would lean towards the latter as the first thing to try,
but maybe the former is useful as well.  It is worth looking into.


-Lex

> Anyway, if upstream authors use MCM's, then either PU needs to support
> MCM's or one of the universe maintainers needs to post the individual
> packages.  I would lean towards the latter as the first thing to try,
> but maybe the former is useful as well.  It is worth looking into.
>
>
> -Lex
In the development universe the package Magma Tester beta, r40Beta.3 is
posted as a Monticello Configuration Map (.mcm)

If you load this into a 3.10 image, it will load
MonticelloConfigurations, loading this depends upon a file in which will
patch Monticello1 (trivially), and it will patch Universes (almost
trivially) in order that mcm's will work.

So, in order to post your mcm package all you need to do is to add
MonticelloConfigurations as a dependency.

cheers

Keith

On 8/18/07, Keith Hodges <[hidden email]> wrote:
>
> In the development universe the package Magma Tester beta, r40Beta.3 is
> posted as a Monticello Configuration Map (.mcm)

I still don't understand why.  Why does Magma Tester beta need to be
an MCM instead of a universe?  What is the MCM doing that the universe
can't?

This seems to me like patching apt-get to accept RPM configurations to
describe dependencies.  I would personally rather see a patch that can
read MCM and convert it to Universe configuration.

Jason Johnson wrote: It specifies a specific, i.e. exact set of packages that are loaded in a
specific order. It can be managed in your mc repository together with
the rest of your files, and it represents a complete deliverable. Not really. Monticello can load many file types. It is perfectly possible for monticello to define its own new file types and new file formats.

For correct usage you say to the reader superclass, "find me a reader for this file" and it supplies one, then you use it. If Universes goes around this mechanism, (which it does, which is not monticello's public interface) then it is broken. The patch in this case is only a fix to make "apt-get" work as it should, not to do anything unusual.

regards

Keith

On 17 Aug 2007 17:33:15 -0400, Lex Spoon <[hidden email]> wrote:

> Anyway, I plan to add a link to the SqueakMap entry.  SqueakMap seems
> like a more appropriate place to add all the bits of meta-data you
> might ever want, including authorship and time stamps.

Would it be possible to use the SqueakMap entry as a pointer to the
source?  Some package maintainers have stopped updating their SM entry
and just use PU.  This is a shame, since SM is the nearest thing we
have to a catalog of the entire Squeak universe.  But I can see that
people don't want to keep two systems in synch.  If PU could refer to
the SM entry then perhaps they could just update their SM entry and
get the PU entry updated for free.

> > > 1. Packages should have information such as purpose and maintainer.
> > >    At the moment there seems to be no such information. Instead it is
> > > repeated for every version of the package.
>
> The maintainer and description can and do change asn time goes on, so
> it seems better to relist them for each version.

But people tend to not bother copying the description, so it can be
hard to find.

> Most of the time, anyway, you are only interested in the most recent
> version of a package.

One thing that is annoying about PU is all the old versions.  I wish
maintainers would delete the old versions when they put in the new.

-Ralph

On 8/19/07, Ralph Johnson <[hidden email]> wrote:
> On 17 Aug 2007 17:33:15 -0400, Lex Spoon <[hidden email]> wrote:
>
> One thing that is annoying about PU is all the old versions.  I wish
> maintainers would delete the old versions when they put in the new.
>
> -Ralph

I think the old versions have to stay so that PU knows how to update
the old ones, no?

On Sat, Aug 18, 2007 at 10:41:49PM -0500, Ralph Johnson wrote:
This sounds right to me. I maintain a number of packages, and it's more
than enough work to keep the SqueakMap entries up to date and functional
(the latter being a separate issue). If someone wants to maintain (for
example) a specialized Package Universe that includes TimeZoneDatabase,
it's a good thing if they can do it without my intervention, and it's
a good thing for me to be able to update the SqueakMap entry with a new
release of TimeZoneDatabase without needing to know about all of the
possible dependent Package Universes.

> If PU could refer to
> the SM entry then perhaps they could just update their SM entry and
> get the PU entry updated for free.

The idea of having the PU entry be updated for free when SM is updated
should not apply to source, because the PU entry for a Universe has to
point to a known-good SM release version, not to the latest and greatest
version (of course). That said, it would certainly be nice if there was
an easy way for a user of an older Package Universe to see that there
is (for example) a more recent version of TimeZoneDatabase than the one
that is marked as known-good for that particular Universe.

Dave

Jason Johnson writes:

I've also had instances (not PU-related) where an upgrade to the latest
version broke something badly, so I like the ability to choose a slightly
older version (and perhaps more stable). Just my $0.03

The Viking

"Ralph Johnson" <[hidden email]> writes:

I would like there to be an easy way to post to both indexes in
addition to others.  That's why I emailed the SqueakSource maintainer
with code that posts a release to a package universe, in the same way
SqueakSource used to do this to SqueakMap.  I got no reply, and last I
heard the SqueakMap button didn't work any longer.  Bummer.  The code
is on the wiki, for people interested in setting up a "post update"
code snippet for their own personal use.

For your concrete suggestion, I do not see how it would help.  You
still have to explicitly post to two services, only now instead of
posting a simple URL to the package universe -- the same URL you'd
paste to SqueakMap -- you'd post a SqueakMap version number.  Isn't it
just as easy to paste the URL into both tools?


Anyway, there is a better way to get SqueakMap holding more content:
have it pull package versions from popular package universes.  At one
swoop this would make SqueakMap hold strictly more content than
package universes.  Plus, the presense of a package in particular
package universes could cause the SqueakMap version to have tags
added.



> > > > 1. Packages should have information such as purpose and maintainer.
> > > >    At the moment there seems to be no such information. Instead it is
> > > > repeated for every version of the package.
> >
> > The maintainer and description can and do change asn time goes on, so
> > it seems better to relist them for each version.
>
> But people tend to not bother copying the description, so it can be
> hard to find.

That should not be an issue in practice.  Can you point me to a
package where this happened?

Maybe the UI needs improving, but there is already a "new package
version" button in the universe editor.  If you use it, the tool will
automatically copy forward all the fields from the previous release of
the package.



> > Most of the time, anyway, you are only interested in the most recent
> > version of a package.
>
> One thing that is annoying about PU is all the old versions.  I wish
> maintainers would delete the old versions when they put in the new.

Well, whenever you release 3.10, it will get cleaned out in the
process of releasing the 3.10 stable universe.  :)

I am not sure it is a worthwhile expendature of time for individual
contributors to delete old versions.  They can do so, if they like,
however.  The only potential problem is the one mentioned by the
Viking, that users sometimes *want* to use an old version.  So if it
is really a problem, I believe it would be more helpful to have a
weekly cleaner script that removes all but the most recent two
versions.

Given Damien's new UI, though, it seems like less of an issue
nowadays.


Thanks for the comments everyone,

Lex Spoon

Keith Hodges <[hidden email]> writes:
A package universe does that for you, too.  It has an exact list of
package versions that are current, including dependency information.

Have you considered what happens if you load two different mcm's that
request different versions of the same Monticello package?  If you
instead listed the individual mcz's, then you can use a package
universe to sort out the mess.


-Lex