[squeak-dev] Squeak Package Management

> SqueakMap: Packages are out-of-date, because the security model
> does not allow anyone to maintain the package; also, there are
> no dependencies
>
> Universes addressed the dependency issue, but still does not
> allow widely distributed maintenance of the packages. So far,
> it is kept mostly up-to-date, due to a few motivated
> individuals.
>
> Sake/Packages is an attempt to address both of the above
> concerns. It as yet has no UI. Unlike in SM or Universes, the
> entire distribution is versioned as a whole, rather than the
> individual packages, so editing one package means making a new
> distribution version. This makes it easy to specify exactly what
> version of the database your image is up-to-date with, but does
> not scale if there are many contributions.

> On Thu, Jun 26, 2008 at 9:28 PM, Matthew Fulmer <[hidden email]>  
> wrote:
>> SqueakMap: Packages are out-of-date, because the security model
>> does not allow anyone to maintain the package; also, there are
>> no dependencies
>>
>> Universes addressed the dependency issue, but still does not
>> allow widely distributed maintenance of the packages. So far,
>> it is kept mostly up-to-date, due to a few motivated
>> individuals.
>>
>> Sake/Packages is an attempt to address both of the above
>> concerns. It as yet has no UI. Unlike in SM or Universes, the
>> entire distribution is versioned as a whole, rather than the
>> individual packages, so editing one package means making a new
>> distribution version. This makes it easy to specify exactly what
>> version of the database your image is up-to-date with, but does
>> not scale if there are many contributions.
>
>
> I don't see the point of having completely separate tools/websites for
> code management (SqueakSource/MC) and package management (all others).
> I want to configure an image as a user, then decide that I want to
> contribute on such package, and upgrade it to the devel snapshot with
> the same tool.
>
> Package management and code version control are really the same thing,
> just at different levels of granularity.
>
> As a developer, you want the bleeding edge version of the code most of
> the time, but also need flexibility (load older versions, branches,
> bypass dependancies). You do need dependancies if only for the
> documentation value (it's always a pain when code is split in several
> packages and you have to checkout each of them manually).
>
> As a user, you want no hassles, ie. a complete and up-to-date package
> set with dependancies between stable blessed versions that work. Also,
> feature-oriented rather than code-oriented packages, ie. Seaside
> instead of all the dozen actual class categories.
>
> Of course these are just two points somewhere in a continuum: paranoid
> users will only use bare major release images, power-users will want
> more recent packages at the price of more frequent updates and less
> stability. Even developers are only developers for a subset of their
> image and users for all the rest.
>
> So for me, SqueakSource with practical dependancies and the
> possibility to make actual releases would be the best. Maybe the
> dependancy/release handling can be done by merging SM  or
> Sake-Packages metadata into MC snapshots, but I do believe that it
> should be a single integrated application.
>
> --
> Damien Pollet
> type less, do more [ | ] http://people.untyped.org/damien.pollet
>
>

> Matthew wrote:
>> Sake/Packages is an attempt to address both of the above
>> concerns. It as yet has no UI. Unlike in SM or Universes, the
>> entire distribution is versioned as a whole, rather than the
>>
> This is a feature. If you want an old version of the s/p package  
> map, then everything is as it was on that past date. (if certain  
> criteria are adhered to)
>> individual packages, so editing one package means making a new
>> distribution version. This makes it easy to specify exactly what
>> version of the database your image is up-to-date with, but does
>> not scale if there are many contributions.
>>
> Currently s/p uses mcz files to store its package definitions as a  
> body of code.
>
> I do not know of any squeak project that currently has scalability  
> issues with too many contributors to the code repo. s/p is a body of  
> code like any other.
>
> If a scalability problem was to occur which I doubt very much  
> indeed, there are numerous solutions.
>
> 1. split the packages up into smaller granularity, i.e.  
> WebDevelopment packages, Dev Tools packages
>
> 2. adopt an editorial policy informally - i.e. load scripts only  
> load the latest "approved" version.
> (seaside users do this frequently, loading Lukas or Phillippe's  
> versions in preference to unknown contributors.)
>
> 3. adopt an editorial policy formally - close the repo to a small  
> number of approved editors.
>
> 4. spilt the package up, so that some packages load further package  
> definitions as needed.
>
> 5. adopt a different server from the basic mcz server, such as Magma  
> with more intelligent actions/merging capbability.
>
> 6. write a front end tool for publishing versions with some form of  
> process management, aka, Gjallar / map.squeak.org / SqueakMap3
>
> 7. Split the package definitions master repo up into smaller repo's  
> arbitrarily as needed. e.g. Database packages, Seaside, Kernel,  
> etc.  Use sake/tasks to auto generate combinations for specific  
> squeak images based upon metadata. i.e. collect all definitions with  
> the flag #worksInCroquet set into PackagesCroquet10
>
> So yes it is scalable, because the actual nuts and bolts of the  
> implementation of sake/packages is fairly orthogonal to how it is  
> used. If the usage pattern turns out not to be scalable then we  
> change the usage pattern.
>
> i.e. Sake/Package definitons could be distributed via DeltaStreams  
> rather than Monticello.
>
> Randal wrote:
>> Notice about 50 uploads a *day*.  And everything *works* in an
>> automated way, except for the "CPAN gods" exceptions listed above
> As I said before if the current solution cannot cope with 50 uploads  
> a day, then it is not the fault of s/p, this is a server issue or a  
> usage patterns issue. Squeaksource is not the only mcz server out  
> there, and it is not the only solution for distributing/managing  
> code. mcz can store diffs, magma might make a more intelligent back  
> end and mc2 is out there somewhere too. This issue is orthogonal to  
> s/p.
>
> The s/p model is intended to be like that of wikipedia etc. relying  
> upon open editorial policies. They cope with scalability issues  
> somehow.
>
> In addition those 50 uploads a day is quite high bandwidth. But I  
> think that this would consist of people writing their packages, and  
> every time they bump a single version, they have to republish so  
> that their users can keep up.
>
> Not so with s/p.
>
> s/p allows publishers to publish 'NON-SPECIFIC' "the latest version  
> that I have in my repo whatever it is". So package writers do not  
> have to constantly worry about publishing their work for their  
> bleeding edge users. This is a wonderful feature, you just publish  
> ONCE even at the beginning of your project, you dont even need to  
> wait until it is ready for release!
>
> Users or publishers can write definitions that are more specific  
> such as "the latest version that is authored by 'the master coder'. "
>
> or even more specific such as "the latest version that is published  
> on SqueakMap."
>
> It's only when users find that a specific version is needed for a  
> specific image that the definition needs to be updated with a  
> "SPECIFIC" definition.  "Squeak3.10 definitely works with version  
> 522 of Seaside2.8."
>
> So...
>
> A developer can publish a package, generating 100 versions as he  
> goes, as it reaches some level of maturity the users add the  
> definition that says version 100 definitely works in Squeak1.1.
>
> Beta users who want the latest all the time, use
>
> Packages beta load: 'ThePackageName'.
>
> Normal users what the definitions that have been conformed to work  
> BY OTHER USERS. They use..
>
> Packages current load: 'ThePackageName'.
>
> So... I doubt very much that the bandwidth for s/p will be anything  
> like CPAN for the same amount of business.
>> Sake/Packages - single point of update, making publishing hard
> Single point of update, making management easy! And putting the  
> power in the USERS hands, not the publishers.
>
> The publisher will not have used the package in the specific images  
> that the users have tested it in, so actually the Users collectively  
> have more knowledge about the use of the package than the publisher.  
> This model allows the user to publish his knowledge in the  
> appropriate place. i.e A fix for Squeak 3.7 does not need to  
> contaminate everyone elses definitions.
>
> I would like to  know how CPAN publishes version 23 of the database  
> driver for use on this version of Linux, with this Database loaded  
> and this specialised file system. I beleive the database driver  
> publishes his work and it is the USERS which decide how it will be  
> used. S/P is primarily a means by which USERS share "What Works"  
> information. SqueakMap and SqueakSource is where publishers can  
> publish "What is Available".
>
> Randal wrote:
>> recent Squeak releases for installation, so we have a bootstrapping  
>> problem.
>>
> I dont think s/p has a bootstrapping problem, since if your image is  
> capable of loading code, s/p is only code and can be loaded any  
> which way.  I surmse that s/p will have negligable bootstrapping  
> problems when compared with the SqueakMap problems of the past.  
> Furthermore LevelPlayingField is present to help with any  
> bootstrapping problems we may have for any package in the future.
>
> Randal wrote:
>> And all of these systems are missing dependency and automatic upgrade
>> management.
>>
> s/p models dependencies to whatever depth you wish.
>
> s/p has a rudimentary upgrade capability AND package UNLOAD  
> capability...
>
> If I want to unload Seaside it will unload in a sensible order every  
> package that has been loaded which depended upon Seaside, prior to  
> unloading Seaside (well that is the theory).
>
> Damien Wrote:
>
>> I don't see the point of having completely separate tools/websites  
>> for
>> code management (SqueakSource/MC) and package management (all  
>> others).
>> I want to configure an image as a user, then decide that I want to
>> contribute on such package, and upgrade it to the devel snapshot with
>> the same tool.
>>
>> Package management and code version control are really the same  
>> thing,
>> just at different levels of granularity.
>>
> Amen Brother!
>
> If you fit the paradigm to the task it makes things much easier I  
> think.
>
> Let a coder use code to determine what and how his code is loaded  
> into his image, and it will be a lot simpler than....
>
> Using a web front end to a database of meta data, describing code  
> files that are to be replicated all over the world for users that  
> are still using 1980's modems. etc etc.
>
> 90% of software engineering is solving the right problem. The  
> problem is actually very simple,
>
> "collect the necessary data, what works where and with what  
> dependencies."
>
> s/p collects that data in an executable form (and all of the  
> metadata is optional anyway), I dont see how it can get much simpler  
> than that.
>
> Randal wrote:
>> * CPAN installation is easy
>> * CPAN publishing is easy
> s/p publishing is probably easier than CPAN, since the package  
> writer only has to publish once ever, and it is easy to browse every  
> use case of a particular package...
>
> load:  Installer install: 'PacakgesAllVersions'.
>
> To find all versions of squeak which can load versions of seaside,  
> do "implementers of #Seaside."
> To find all packages that are dependant upon Seaside..
>
> Packages current Seaside allDependants.
>
>> Lowering this barrier in Squeak is important
>>
>> Merik
>
> It may not be perfect but it is lowered to the point of actually  
> being usable.
>
> Ralph worte:
>> A "feature" of Squeak that might differentiate it from perl is that
>> there are many versions of Squeak.  Packages that work with one image
>> might not work with another.  So, if you are using 3.8 then you do  
>> not
>> necessarily want the latest version of some package if that package
>> only works in 3.9 or 3.10.   PU addresses this problem by having a
>> different Universe for each version of the image, though I think  
>> there
>> is currently a Universe only for 3.9 and 3.10.  Even if the code for
>> PU could run in earlier images, it isn't useful until someone  
>> builds a
>> Universe for it.
> Got it exactly. CPAN does not address the real problems that we  
> actually face. i.e "WHAT works WHERE." as seen by package users.  
> And this is exactly why it is that Users of a particular image that  
> need to manage the package definitions for that image. The package  
> writers are often behind the enlightened users who are putting their  
> package to new and varied uses in different contexts.
>
>> Maybe PU is the closest to CPAN we have.  So the question is, do we  
>> keep
>> mutating PU into what I'm looking for?
> No, because PU will be no use in an image that doesnt have any  
> Network code, doesnt have XML parser, doesnt have Morphic, so if you  
> do improve PU someone will have to develop another solution for the  
> more stripped down image. If future squeaks are based upon more  
> stripped down images then we end up with two solutions to the same  
> problem.
>> Or do we first need to agree that the
>> CPAN might be a pretty darn good model?
>>
> Its not a good model because it is solving a problem that we don't  
> currently have. We are not distributing tar files for users around  
> the world that are using modems to access the internet at 1200 baud,  
> for use in a language with no reflexive capabilities. i.e. unix bash/
> csh etc
>
> (btw: Our packages do not have to be restricted to code, they could  
> include objects, projects, or anything loadable or creatable by  
> code, this includes mantis fixes.)
>
> Our problem is not with publishing packages, our problem is enabling  
> users to load what THEY want into their images and to share their  
> feedback as to what works with other users in a way that can be used.
>
> Ralph wrote:
>> CPAN is probably a good model.  However, a model that works well for
>> one language does not always work well for another.  How would you
>> think a CPAN-like system would handle the fact that people are using
>> 3.6, 3.7, 3.8, 3.9 and 3.10 and that some packages work for all of
>> these, but some packages are specialized for a particular image?
>>
> Exactly, that's why CPAN is not a darn good model for this problem.  
> I can only envisage that it would be a darn big complicated  
> unmanageable pain. The proposal is made with blinkers on, since  
> Randal refused to even look at s/p because of his concerns over  
> "scalability", which as I have described above is a complete red-
> herring-issue.
>
> Randal wrote:
>> We do need something that says package X works with squeak Y and  
>> package X
>> depends on package Z.  so it's two dimensions, not just one.
>>
> Yes we do... and which omnipotent package publisher knows the answer  
> to that question in every context? Answer none, problem is solvable  
> with a user feedback mechanism, not a package publishing mechanism.
>
> Merik wrote:
>> And what about patches?
>>
>> When an Seaside website needs a patch of an Integer method, for  
>> example.
> Yes indeed , some images need patching before a particular package  
> will load. Sake/Packages handles this, it can load patches from  
> mantis using Installer ensureFix: 1234.  OR It can specify a  
> dependency upon the Sake/Tasks load mantis fix task, which documents  
> the fixes loaded as it goes.
>
> so Randal why you wont look at Sake/Packages is beyond me.
>
> regards
>
> Keith
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of Chris
> Muller
> Sent: 27 June 2008 6:57 PM
> To: The general-purpose Squeak developers list
> Subject: Re: [squeak-dev] Squeak Package Management
>
>
> I think one thing that would go a long way toward making things easier
> is for the standard packaging tools (Monticello, Monticello
> Configurations, Universes, SqueakMap) to be delivered in **stable,
> working conditions** in base Squeaks downloaded from squeak.org.
>
> The packaging systems are how Squeak users take the base environment
> and enrich it with other packages.  Therefore, it is crucial that
> these systems at least be able to LOAD things.  Unfortunately, this
> did not occur for a time in public (squeak.org) Squeak releases.
>
> But please help me understand something:
>
> > SqueakMap: Packages are out-of-date, because the security model
> > does not allow anyone to maintain the package; also, there are
> > no dependencies
>
> As far as being out-of-date that's a human issue, not a software one.
> SqueakMap has a list of "Co-maintainers:".  So what is the security
> model limitation you are referring to?
>
> Also, dependencies are not really needed with SqueakMap because it is
> superceded by SAR processing.  A SAR can do anything, like check if a
> certain dependent package is loaded, ask if you want a newer version,
> etc.  For Magma, its as simple as loading all of the dependent
> packages automatically from within the SAR script.
>
> SAR allows something to be deployed exactly as the creator wants it to
> be.  The install script can pop up starting screens, documentation,
> etc.
>
> I'll add, SAR works in 3.7, 3.8, 3.9, (3.10?), Croquet, etc.  Do the
> other packaging systems?
>
> > Universes addressed the dependency issue, but still does not
> > allow widely distributed maintenance of the packages. So far,
> > it is kept mostly up-to-date, due to a few motivated
> > individuals.
>
> By "addressed" do you mean it put a UI on it?  I did that too by
> simply adding a button to Monticello Configurations called, "Build
> SAR".  You make your configuration and, in one click, you get a SAR
> file deployable on SqueakMap.
>
> Universes tries to make a "model" out of the dependencies is that
> right?  It portends to keep a record of "what works with what else"
> but that question is really ambiguous and impossible to answer
> universally or correctly isn't it?  Besides, running a successful
> *Test Suite* is my indication that packages are still working after
> I've loaded another package, not the official declaration by a package
> tool "works with XYZ".
>
> Also, does Universes support, say, calling out to an external Internet
> site during package installation to grab latest resource files like
> SAR does?
>
> My other gripe about Universes is the UI; it puts the software in
> charge of the user, forcing me expand and scroll a tree, manually
> hunting for a package whose name I already know.  Isn't there a way to
> *search* for a package?
>
> > Sake/Packages is an attempt to address both of the above
> > concerns. It as yet has no UI. Unlike in SM or Universes, the
> > entire distribution is versioned as a whole, rather than the
> > individual packages, so editing one package means making a new
> > distribution version. This makes it easy to specify exactly what
> > version of the database your image is up-to-date with, but does
> > not scale if there are many contributions.
>
> But "many contributions" would simply be accumulated before versioing
> up an entire distribution wouldn't it?  Just like Magma's process
> today; I post multiple individual packages to squeaksource multiple
> times before posting another Configuration..?
>

> Also, dependencies are not really needed with SqueakMap because it is
> superceded by SAR processing.  A SAR can do anything, like check if a
> certain dependent package is loaded, ask if you want a newer version,
> etc.  For Magma, its as simple as loading all of the dependent
> packages automatically from within the SAR script.
>
> SAR allows something to be deployed exactly as the creator wants it to
> be.  The install script can pop up starting screens, documentation,
> etc.
>
> I'll add, SAR works in 3.7, 3.8, 3.9, (3.10?), Croquet, etc.  Do the
> other packaging systems?