[squeak-dev] Organising and Documenting Micro-Universes in Squeaksource

Dear All,

I have long been frustrated that there is no place in squeaksource for
documenting what is in a repository, and offering feedback as to what
works where. We have several solutions on a macro-universal-scale,
namely, universes, squeakmap, and sake/packages.

I have started to use Sake/Packages as a kind of micro "universe" for my
Seaside "Client" UI & Backend Component Framework. Having seen how
useful this is I thought that this might be useful for pier, and of
course seaside and other projects.

For an example of this, have a look at squeaksource/Jetsam at the
project Client-Packages

In www.squeaksource.com/Jetsam there is a package Client-Packages.
For pier you would have a package in source.renggli.ch/pier called
Pier-Packages.
For seaside29 you would have Seaside29-Packages etc etc.

This contains a class/classes which document all of the packages, and
the dependencies that are available for the project, together with
metadata, and package comments.

Some have reservations about Sake/Packages because they say it is not
declaritive. It is if you want it to be. 99% of the package declarations
already present are declarative. Exceptions are usually weird cases like
the unload script for Sake itself.

Having seen recently how useful this idea is, I will be looking at the
following improvements to the idea.

1) how best to integrate these micro-universes in to the main
Packages-Library
2) how to declare alternative definitions for pharo and other images
3) how to make the micro universes light-weight, so that they may be
available for documentation without depending upon Sake/Packages being
loaded, (hmm, perhaps they could be published as traits!).
4) Bob support - Sake/Packages will also be able to generate load
scripts (for Bob or anyone who is interested), so that "Bob the Builder"
can build complex images without having ANY of the following loaded
monticello/sake/packages/universes/squeakmap, Installer will be the only
dependency.

Keith

On Thu, Mar 19, 2009 at 22:46, Keith Hodges <[hidden email]> wrote:
> I have long been frustrated that there is no place in squeaksource for
> documenting what is in a repository, and offering feedback as to what
> works where.

Yes, and the MC tools should support that IMHO.
BTW I'd also add: no nice support for named/tagged branches (I mean,
in a big project there should be released/working/devel/in-progress
branches, or at least documented sequences of snapshots for stable
versions, and we have
package-developerUniversallyKnownToNotRegularlyBreakThings.biggestNumber.mcz)

> I have started to use Sake/Packages as a kind of micro "universe" for my
> Seaside "Client" UI & Backend Component Framework. Having seen how
> useful this is I thought that this might be useful for pier, and of
> course seaside and other projects.

Yup, I like the idea.

> Some have reservations about Sake/Packages because they say it is not
> declaritive.

Huh… it's a rule system, right ? The alternative I know about being
scripts that explicitly load snapshots in sequence, I fail to see
where sake is not declarative.

>From my discussions on the subject, problems are more about the
perception of what level of predictability and control one would get
by using sake. Questions like "how do I know what will get installed
if I pull this one" or "how do I install this, but with this
particular version of that", and so on. Maybe it's a problem of
polishing the DSL, of documentation, maybe of tools, I'm not sure;
that could be investigated a bit more.

For me it's clear that automatic dependancy management is the way to
go, though the squeak/pharo community does not have resources or
organization like debian's, so I think we need a more decentralized
and agile way of managing dependancies, detecting and resolving
incoherencies, etc.

--
Damien Pollet
type less, do more [ | ] http://people.untyped.org/damien.pollet

Damien Pollet wrote:
> On Thu, Mar 19, 2009 at 22:46, Keith Hodges <[hidden email]> wrote:
>  
>> I have long been frustrated that there is no place in squeaksource for
>> documenting what is in a repository, and offering feedback as to what
>> works where.
>>    
>
> Yes, and the MC tools should support that IMHO.
>  
It was one of the first ideas I was going to add to MC, even just to
have a simple text file explaining what is going on in a repo.

I put a lot of effort into MC to bring the forks together, and make it
maintained and able to move forward with one voice. That failed, so my
motivation for hacking MC anymore is pretty low.
> BTW I'd also add: no nice support for named/tagged branches (I mean,
> in a big project there should be released/working/devel/in-progress
> branches, or at least documented sequences of snapshots for stable
> versions, and we have
> package-developerUniversallyKnownToNotRegularlyBreakThings.biggestNumber.mcz)
>  
My brain struggles with branches. Perhaps not having them is a good idea?
>> I have started to use Sake/Packages as a kind of micro "universe" for my
>> Seaside "Client" UI & Backend Component Framework. Having seen how
>> useful this is I thought that this might be useful for pier, and of
>> course seaside and other projects.
>>    
>
> Yup, I like the idea.
Great!
>> Some have reservations about Sake/Packages because they say it is not
>> declaritive.
>>    
>
> Huh… it's a rule system, right ? The alternative I know about being
> scripts that explicitly load snapshots in sequence, I fail to see
> where sake is not declarative.
>  
Its because the rules to load: a package can be blocks executing an
arbitrary script. In normal use the #defaultAction is to process the
given "declared" url to find the package to load.
> From my discussions on the subject, problems are more about the
> perception of what level of predictability and control one would get
> by using sake. Questions like "how do I know what will get installed
> if I pull this one"
(Package named: 'SomePackage) what explore.

This could be improved to ask Installer exactly what it will install,
perhaps -

(Package named: 'SomePackage) whatUsingDryRun explore.

Also with DeltaStreams integration with Installer, we might have the
ability to load into a "Delta/SystemEditor" rather than actually in to
the image.
>  or "how do I install this, but with this
> particular version of that", and so on. Maybe it's a problem of
>  
Interesting point. I cant think of how to do that. Unless you use sake
to generate a script or a task data scructure, and then you manually
edit it.

How about merging two dependency trees like so:

(Packages current named: 'Seaside) using: (Packages beta named: 'Pier)
> polishing the DSL, of documentation, maybe of tools, I'm not sure;
> that could be investigated a bit more.
>  
Can never get enough Documentation!
> For me it's clear that automatic dependancy management is the way to
> go, though the squeak/pharo community does not have resources or
> organization like debian's, so I think we need a more decentralized
> and agile way of managing dependancies, detecting and resolving
> incoherencies, etc.
>  
Bob might be able to do something like that in the future. He is sitting
there monitoring a number of repositories, he could periodically process
contributions and derive dependencies because he has some knowledge of
the big picture.

Keith

Hi!

Keith Hodges wrote:
> Also with DeltaStreams integration with Installer, we might have the
> ability to load into a "Delta/SystemEditor" rather than actually in to
> the image.

Ah! There's my queue! :)

Yes, although I only read casually what you are talking about I do agree
with the general notions expressed. And yes, hopefully we will soon have
Deltas in a useful form.

Quick status again, followed with some questions:

- Lots of tests are now green, still lots of tests not green but most of
them are "wishful thinking"-tests and not really broken tests.
- I am refactoring and cleaning right now, as a pre-cursor to adding the
new file format.
- I am also just beginning to try out Tirade as a file format. For more
info on Tirade, see my first article:

    http://goran.krampe.se/blog/Squeak/Tirade.rdoc

So in a relatively short timeperiod I hope we have code to load and save
Deltas as xxx.d (Delta Tirade format) and xxx.dgz (Gzipped). Would these
two file extensions work? Or should we use convention *.d and *.d.gz?

Next step IMHO is to write a Dual Change Sorter-ish tool for Deltas - I
would also like to hear advice on how to build that in order to:

- Be compatible with as many Squeak forks as possible.
- Get a clean nice simple tool.

Just copy plain dual change sorter? Use Toolbuilder? Use MC codebase?
Omnibase?

The end goal would be to fully replace Dual Change Sorter - which means
that I would also like to have some support for loading a cs as a Delta
and to save a Delta as a plain cs.

...and as always, if you like to join us in moving Deltastreams forward,
just holler!

regards, Göran

Hi goran

I did not get at all why you need a new format.
Without self.
I think that if you want adoption of deltastream do not mix them with  
a file format.

Stef

On Mar 20, 2009, at 9:42 AM, Göran Krampe wrote:

Hi!

Stéphane Ducasse wrote:
> Hi goran
>
> I did not get at all why you need a new format.
> Without self.
> I think that if you want adoption of deltastream do not mix them with  
> a file format.

I am sorry, perhaps you haven't followed the reasoning, a quick recap:

1. Deltas are not bound to a specific format on disk. You can store them
in any way you like. They are "self contained object graphs" in the
image and can trivially be serialized.

2. Matthew made a format called "Interleaved ChangeSet" which combines
binary data with the chunk format meaning that a Delta can be stored
both as a changeset (loosing lots of info) and a Delta embedded inside
it. This means an old image can "file it in" just like a cs.

3. I think the Interleaved format is a bit too complex and while it is
"cool" I don't think the use case is *that* important (being backward
compatible for images that don't know what a Delta is). It is probably
simpler to include easy conversion delta2cs and cs2delta. These
conversions are not lossless however.

4. Since I wanted a readable, simple, fast, editable, and secure format
for Deltas - I came up with Tirade. Chunk format is readable, simple and
editable but it is not fast nor secure. It is also less "declarative" in
nature IMHO.

Tirade is well defined, small and simple. The Tirade package is just a
few classes and it is intended to work in "all Squeaks" - no
dependencies on any advanced stuff.

Tirade is a "data format" - it is NOT "Smalltalk code", it just happens
to match a lot of syntax and concepts. It is "merely" a sequence of
keyword messages with "data" as arguments (String, Symbol, Arrays,
Associations and Numbers - nested however you like). That is it. No
assignment, no variables, no globals, no expressions etc etc. It is NOT
Smalltalk. Adding "self" to it has no purpose, well, the ONLY purpose
would be to make it "look" like Smalltalk.

Now... this is repeating stuff I have already written elsewhere - but
there ya go. :)

regards, Göran

Igor Stasenko wrote: You mean the exact situation that many of us have been in for the past
3+ years!

At a lower level, you have to have some kind of buffer package. Thats
what Kernel-Extensions was for, it was the buffer covering the multitude
of sins in the unmaintained Kernel. This can also be achieved with more
intelligent installation tools that can install things according to
rules "if this class is missing then apply this fix/changeset/package".
(btw coincidentally (or not) Sake implements such rules in it's ClassTasks)

That's what "Installer mantis ensureFix:" is all about, its a dumb way
of acheiving the same goal. The next level up in intelligence is
Installer-Scripts (which includes the current implementation of
LevelPlayingField). It  can install things specific to product/image
version. (Pharo's ScriptLoader is somewhat equivalent, for pharo only.)

If everyone who is maintaining packages A,B,C,Y that depends upon X,
manages their fixes to X in a co-ordinated manner with tools such as
Installer-Scripts, or Sake/Packages, this documents all use cases in
detail. Then the future maintainer of X has simply to look at all of the
package load definitions, and the list of fixes to his package that they
require, his work is practically done for him.

Please note that was also the explicit intention and purpose behind
allowing Sake/Packages to be non-declarative if need be. If a user has
to add a script to get a package to work in their specific context, then
Sake/Packages is providing a knowledge capture service for the developer
of X, showing exactly what is needed to get X working in all the
contexts in which it has been applied.

At a higher level, you need to consider architecting your code to be
modular and to have interface/protocol definitions, and at the highest
level you can consider doing interface/protocol negotiations, either
statically (at package load time, or dynamically at run time.

Statically is easiest -  if they have version 1 of the database use
interface A... etc, all of which Sake/Packages can do. For example I
have written a number of modules for seaside28, called the "Client
Framework", these will need to be reimplemented for Seaside29, but
should still provide the same API.

Dynamically is doable, but there is so much irrational feeling against
use of #respondsTo: #askFor:, Null, and other facilities for making this
work on a generic basis. For example Null needs to be in the kernel,
because it must be maintained in sync with the kernel and IDE, otherwise
the Kernel/IDE developers keep doing things which ruin Null's day. (stef
please take note - I know you hate Null, but it is not intended to be a
replacement for nil, it is most useful as a replacement for an entire
missing interface on a macro scale)

Going the other way, there is also help for maintaining a single package
that can be deployed in muliple different contexts. The new PackageInfo
in MC1.5 has more facilities for exporting a package, and deploying it
in different contexts. For example, you can mark methods as "squeak
only", "gemstone", "VW only" for example.

So while you code and maintain "MyPackage" in your repository, when you
save "MyPackage.squeak" it will filter out any vw only methods, or pharo
only methods. When you save MyPackage.vw it will not include classes in
MyPackage-Squeak.  MyPackage.impl filters out the tests, and
MyPackage.test saves the tests only.


...btw Igor in response to your other email about splitting up MC.

MC1.5 has already been organised into categories for potential splitting
up. For example separation of Model and GUI. The tests have already
departed. The Monticello-Orphanage is intended to be unloadable.

I was planning to split out the PasswordManager into a  system project
"System-Passwords".

Keith