[squeak-dev] Smalltalk images considered harmful

Etoys was being considered to get into Debian. Now it may be rejected,  
because an image file is not "transparent enough" (see below). It was  
suggested to discuss this issue on the debian-devel list.

Do any of you have ideas how to respond? Are there perhaps other  
Debian packages that have a similar issue of accountability?

And how hard would it actually be to bootstrap a fresh Squeak image  
from sources nowadays?

- Bert -

Begin forwarded message:

My response would be to point out that an image is an executable that
runs on top of a Squeak VM. Just like Java, where .java files get
compiled into .class files and packaged into .jar files; just like any
OS where .c files get compiled and packaged into ELF binaries; just like
Python where .py files get compiled into .pyc files and packaged in zip
files; Squeak compiles .st or .cs files and packages them into an image
file.

It is *not* correct to say that a Squeak image cannot be compiled from
source code. We do this all the time. It is correct that it is difficult
to compile it *entirely* from source code but this is just as true for
other executable files (the ELF headers are not generated from source
code either!).

If the amount to which a system can be recreated from source code is a
concern to the Debian people I think we can work this out (though maybe
not immediately). But I'd be more than a little puzzled if they'd be
making such artificial distinctions.

Cheers,
   - Andreas

Bert Freudenberg wrote:

Bert Freudenberg wrote: How about the Squeak FUSE plugin, say its a filesystem in a file.

Keith

This is a variation of the "human readable files" argument. So once
again I claim that no such thing exists other than printed listings
(which then are not machine readable, at least not much in practice).

The image file is our source: the preferred form for editing programs.
It is far nicer when coupled with the .sources and .changes files, of
course, but we can get by without these. You do need the proper
application for dealing with the image - the VM.

Now we can dump out the image as a huge XML file and we should have no
problems reading that back in. We can also develop a more refined system
like the Transporter tool in Self. But would the result be human
readable? Only in the sense that it could be printed and a person could
read it over the phone to some other person. Nobody would understand it
(unless it was a trivial "3+4" image) nor want to make any changes to it
in this format. So I would argue that this would not be a "source file"
by any reasonable definition even if it did play nice with cvs and vi.
And having pointed text editors to my share of .sources and .changes
files from various Smalltalk I would argue that these aren't quite
sources for Squeak either (though they are from GNU Smalltalk and Little
Smalltalk).

Our sources are the .image files. It isn't what people are used to, but
that doesn't make it less true.

-- Jecel

2008/5/22 Bert Freudenberg <[hidden email]>: Of course its auditable, easy enough for one who know how to use
squeak tools: browser, inspectors and debugger.
For those who think that image (object memory state) is something
which can be expressed by source code, they are totally losing the
point.

>> - Does Debian (including the various teams that might have to take
>>  a look at your packages) want to have easy access to the
>>  differences between upstream version, one Debian revision and
>>  another? Do squeak session images provide this in a way that
>>  is acceptable to Debian?
>>

Compare the Debian installation CD with squeak image.
They are both images. They contain both: source code and binaries.
They contain non-trivial structural organization (directories/files,
some in compessed form, links e.t.c ) , which can't be expressed in
form of some source code.
So, what is the point in looking at differences between two Debian vanilla CDs?
The only thing which you can state is, that they equal or they not.
Then, when you get deeper, you can say what packages image contains,
what is their versions.
But we have nearly same model for squeak.
--
Best regards,
Igor Stasenko AKA sig.

>>>>> "Jecel" == Jecel Assumpcao <[hidden email]> writes:

Jecel> Our sources are the .image files. It isn't what people are used to, but
Jecel> that doesn't make it less true.

I would also presume that by this rule, PDFs aren't allowed anywhere in the
distro either.  After all, they're not "human readable", but with the right
tools, they're perfectly clear.  It just happens that the tool to read a
.image is in fact the image itself, running on a suitable VM. :)

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[hidden email]> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

2008/5/22 Randal L. Schwartz <[hidden email]>:
Yeah, clearly, any digital media is not human readable without proper tools.
So, i fail to see the point in their reasoning.

> --
> Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
> <[hidden email]> <URL:http://www.stonehenge.com/merlyn/>
> Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
> See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion
>
>



--
Best regards,
Igor Stasenko AKA sig.

On 21-May-08, at 2:34 PM, Bert Freudenberg wrote:

> Etoys was being considered to get into Debian. Now it may be  
> rejected, because an image file is not "transparent enough" (see  
> below). It was suggested to discuss this issue on the debian-devel  
> list.
>
> Do any of you have ideas how to respond? Are there perhaps other  
> Debian packages that have a similar issue of accountability?

I think it's important to establish what level of transparency Debian  
is looking for. Apparently they want to be able to compare different  
versions of the package and see what has changed. In reviewing those  
changes, what would they be looking for? Objectionable content?  
Malware? Copyright violations? Material that carries a license  
incompatible with the GPL? Insecure code? Bugs? Spelling mistakes?  
Depending on what they want to know we might be able to provide a way  
to provide that information.

Also, Andreas is right - it's probably possible to distribute Etoys  
such that it can be "built" by filing "source code" into a base image  
of some kind. If the base image didn't include Etoys itself (as  
distinct from Squeak) that would provide the kind of transparency that  
Debian is used to, at least as far as Etoys is concerned. That would  
make the debian version be different from the upstream distribution,  
which would presumably still be based on images.

Or it could be done in reverse - Etoys would be distributed as an  
image, but would have a way of spitting out textual representations  
that could be fed into the debian process. That might mean more than  
just filing out Smalltalk code. It could also mean spitting out files  
that represent other types of media: images, sounds, hierarchies of  
Morphs, whatever is deemed important.

Colin

>>>>> "Colin" == Colin Putney <[hidden email]> writes:

Colin> Also, Andreas is right - it's probably possible to distribute Etoys
Colin> such that it can be "built" by filing "source code" into a base image
Colin> of some kind. If the base image didn't include Etoys itself (as
Colin> distinct from Squeak) that would provide the kind of transparency that
Colin> Debian is used to, at least as far as Etoys is concerned. That would
Colin> make the debian version be different from the upstream distribution,
Colin> which would presumably still be based on images.

Would they accept a KernelImage (http://www.comtalk.eu/Squeak/98)
and the rest of it as filein's?

Of course, even then, it means someone on the squeak release team would have
to tear down an image and build it back up before releasing the "official"
squeak, or we'd have to live with the idea that Debian is distributing a
slightly different image.

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[hidden email]> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

I think that the distribution of Squeak on Debian should be this way:

1- VM.
   A suitable current stable version of the squeak vm should be part of
the distro. This is source code in the "classical" way that it is
converted to an executable file shipped with debian or downloaded from
the debian repositories/mirrors. This is already happening.

2. An independent repository/location/url (squeak.org) should provide a
standard location (a la debian repo) from several images (minimal,
Damian's, FunSqueak, etc), with a sensible name format, that should be
downloaded (at the debian user responsability) at the postinstall step
of installing squeakvm. This maybe can show the user the list (with
short and full descriptions) of the goal of each image and the user
should be given the chance of download them. All of then if he so wants.
This can be a little like the ruby gems, where you download and install
the gem program from debian and download the particular gems from other
sources. Or maybe like the old sun java installation using make-jpkg. Or
maybe like the bcm43xx drivers instalation from pre 2.4.25 kernels,
where you install one part (the driver) from debian repos and the other
part (the firmware for the wireless card) from a public site unrelated
to debian.

So, I don't think that should be good to try to do all the debian way,
because (as ruby's gems, as Perl's CPAN, as TeX CTAN) there is no way
that debian could maintain all the code is in the web.

What do you think?

2008/5/22 Miguel Enrique Cobá Martínez <[hidden email]>: 2. or simple note where to download squeak image(s).

we can even reserve a web page, say squeak.org/debian

>



--
Best regards,
Igor Stasenko AKA sig.

On Thursday 22 May 2008 4:28:29 am Jecel Assumpcao Jr wrote:
> Our sources are the .image files. It isn't what people are used to, but
> that doesn't make it less true.
From the tone of the email, it looks as if Debian maintainer is just trying to
understand images and figure out how to handle images in their current
version control process.  Text-based tools like texteditor, cmp, diff, patch
etc.used in version control will not work with images. A Squeak image is not
a 'source' but records the entire state of a virtual machine. In a way, it is
like firmware to configure wireless devices or codecs required to interpret
certain audio/video files. VM mimics a virtual device and image is
its 'firmware'.

The issues are:

1. How does one compare two images A and B are equal?
2. If A and B are not equal, how to extract the patch which produced B from A?
3. How does one compare two patch set to see if they are equal?
4. Given an image A and a series of patches P1..PN, can different people apply
them in a sequence to produce the 'same' image, B.
5. Is an image file specific to an architecture or not (answer: not)

I remember 1) was discussed in this mailing list sometime back but don't
recollect a solution emerging from the discussions.

Subbu

2008/5/22 K. K. Subramaniam <[hidden email]>:
> On Thursday 22 May 2008 4:28:29 am Jecel Assumpcao Jr wrote:
>> Our sources are the .image files. It isn't what people are used to, but
>> that doesn't make it less true.
> From the tone of the email, it looks as if Debian maintainer is just trying to
> understand images and figure out how to handle images in their current
> version control process.

Well, from that point, i think we should make it clear , that version
control of squeak images are totally on under squeak community control
(release team(s), to be precise). If they find this inappropriate,
then we have no other choice but not include images in distro and let
users download them by own.



--
Best regards,
Igor Stasenko AKA sig.

Igor Stasenko wrote:
They want to apply fixes of their own sometimes.  This is what caused
the recent OpenSSL fiasco, but also has advantages (e.g. patches that go
in GCC packages are very safe, and the Debian package maintainer for GNU
Smalltalk also applied this kind of patch when portability problems were
found).

>> 1. How does one compare two images A and B are equal?
>> 2. If A and B are not equal, how to extract the patch which produced B from A?
>> 3. How does one compare two patch set to see if they are equal?
>> 4. Given an image A and a series of patches P1..PN, can different people apply
>> them in a sequence to produce the 'same' image, B.
>> 5. Is an image file specific to an architecture or not (answer: not)

That's exactly the point.

Images might be the "preferred form for modification", and even the
"preferred form for distribution" of complete applications (as opposed
to packages).  But they are a long way from being the "preferred form
for maintenance", and that's what the Debian maintainers need.

Paolo

Hi Keith,

>> Do any of you have ideas how to respond? Are there perhaps other
>> Debian packages that have a similar issue of accountability?
>>
>> And how hard would it actually be to bootstrap a fresh Squeak image
>> from sources nowadays?

Not sure if this is any use - but in Dolphin I take the basic image and I
call it with a command line parameter that indicates a root package to load
from the file system (so called .pac files) - the loader then starts loading
the package and all its dependencies checking for any errors that occur (if
they do it fails startup) - it then begines a deployment routine to generate
a .exe

I am wondering if the Debian guys might go for a super slim image in sqeuak
which they have "blessed" and then you have a loader that reads in .mcz files
to build up the rest of the more complete system. As mcz files are just zips
that people could extract and diff - this might overcome there nervousness
about future versions being difficult to compare for changes. On load completion
you save a "development" image which can be used over and over again (as
we have now).

Again, not sure if this is what the trouble is - but it might work.

Tim

As a long-time Debian and squeak user--though I haven't done much
squeaking lately--I'd like to see the 2 get together.  I have some
late-night thoughts.

One possible analogy is with systems such as virus and spam scanners
that get regular updates from "upstream" (that is, upstream from
Debian), sometimes directly from upstream.  The image, or possibly
change sets might be seen as analogous.

By the way, Debian does actually package at lots of pieces from
repositories, include those for Perl, Python, TeX, and R.

Another analogy is to packages that have significant amounts of
non-verbal data: star charts, representations of the solar system,
music, maps.  And many package have at least some graphical material.
The main concern with these I was aware of was that some were just too
big.

Thomas's message (appended below) seems to raise several distinct
concerns.

The first is auditability in the licensing sense.  Debian needs to be
sure that the license of everything is known, and that the licenses are
compatible with DFSG (Debian Free Software guidelines), as well as being
compatible with each other.  At one level it seems to me this could be
addressed by a license that says "everything in the image is under this
license."  At another level, software systems frequently turn out, on
inspection, to have parts with licenses that are either unclear or
unfree.  By unclear I mean it's not clear what, if any license exists,
and who claims ownership or copyright.  Sometimes the system includes
components with licenses that don't get along (BSD and GNU?).  So I'm
not sure if the blanket assertion is sufficient.  So Debian might want
to take a closer look at the parts.

At the level of detailed check, things are harder.  One advantage of
text is that you can start at the top and go to the bottom, and known
you've examined it all.  In an image, there's always the possibility
that you'll miss something (e.g., a game shows a copyrighted image only
after an obscure series of steps), or that parts (e.g., the Welcoming
text, other help, or some components) really might have someone who
could pop up and claim them.

In principle one could enumerate all the objects in an image; maybe that
would provide some comfort.  One can even imagine a tool the identifies
all text and audio-video like objects and the context in which they
occur.

I think it has always been pretty clear that stuff going into squeak was
under a very permissive license, so that might allay some of the
concerns.


A second concern behind the desire for "source" is that people be able
to inspect, modify and adapt what they have.  I think the focus on text
may be a bit misplaced here; that is simply an easy way to realize that.
I don't know that Debian policy, or the more fundamental documents like
Debian's "social contract" require it.  I believe it is explicitly
acknowledged that the editability is the key concern.  For example, a
postscript file is text, but distributing only postscript is not
considered OK: one needs to distribute the material that generated the
postscript.  Someone else brought up pdf's.  I think in Debian the idea
is that if you have binary package with pdf's, you can get the source
package, and the source package will have the raw materials that made
the pdf, and that one can modify to create your own pdf.

Generally, the squeak image is also the most natural form for modifying
itself.  In fact, the whole source vs binary package distinction clearly
arose for compiled languages (though not all Debian packages consist of
programs that get run through a compiler), and is an awkward fit for
smalltalk.

I think there may be some difficulties getting non-smalltalkers to wrap
their heads around this self-referential quality of an image.  But the
whole system was clearly designed to be inspectable and malleable, and
so seems to me fundamentally consistent with the DFSG's concerns that
people be able to understand and modify the software they use.  (The
self-referential quality is not unique: C compilers are written in C,
and most higher level-languages systems consist significantly of code
written in the higher-level language).

A final concern is with the differences between versions.  First of all,
I'm not sure where that's coming from: it doesn't seem as central to the
core Debian principles.  Of course, it does have practical significance.
The last time I checked, most of the evolution of images consisted of
taking in change sets, or change set like things, so that seems pretty
auditable.  I'm not exactly sure what the dominant tools for managing
updates are now, so perhaps life is not so simple.

There are other considerations that I don't think were raised in the
message from Thomas, but seem worth mentioning.

First, there seem to be a lot of kinds of images floating around, and
being proposed: minimal, beginner, developer, web developer, MVC, Tweak,
Morphic, ...  Should they all be packaged?  Just some or one?  None,
with the message "go and pick one"?  (the last doesn't seem too
friendly, but is one way around the concern with images).

Second, images alone are not a good way for existing users to get
updates, for the simple reason that using a new one will mean tossing
out everything you've done.  (Yes, I know there are ways to move stuff
between images, but keeping current with an update stream seems easier.)
This is a bit different from the typical Debian package.  If I get a new
compiler, I generally don't lose my source code.  Debian has gone to
great pains to make it easy to keep your customizations when you update
other packages (e.g., apache, spamassassin).  The only analogous
strategy I can see would be to have the squeak launcher script
automatically export your changes to a new image, and that's got all
kinds of problems.

Third, Debian divides software into a "main" section and "non-free" (or
maybe other?).  My understanding is that squeak is in main.  Software
can't go in main if it depends on stuff outside of main.  This might be
problematic if some kind of "get the image yourself" method is followed.
However, this issue only affects what section of the archive the package
is in.

Finally, I am not a Debian Developer or an expert on Debian policy.  I
am not even exactly awake!  I know I've gone on a bit, but I hope it's
some help.

It probably would be useful to take the to debian-devel soon.

Ross Boylan.

Yeah, I know: top-posting :)

On Wed, 2008-05-21 at 23:34 +0200, Bert Freudenberg wrote:

El 5/22/08 12:26 AM, "Randal L. Schwartz" <[hidden email]> escribió:

> Of course, even then, it means someone on the squeak release team would have
> to tear down an image and build it back up before releasing the "official"
> squeak, or we'd have to live with the idea that Debian is distributing a
> slightly different image.


I remember you this is exact my point of view.
We must go to kernel and grow from kernel (or Spoon).
And I remember all I put as "tentative" 3.11 for all could view and discuss
, see http://tech.groups.yahoo.com/group/squeak/message/127740
I such image, we don't have Etoys and could load from packages.

If Debian people still complain, the .mcz could be converted to "human
readable" .st or .cs.

Edgar

On Thursday 22 May 2008 3:22:33 am Andreas Raab wrote:

> It is *not* correct to say that a Squeak image cannot be compiled from
> source code. We do this all the time. It is correct that it is difficult
> to compile it *entirely* from source code but this is just as true for
> other executable files (the ELF headers are not generated from source
> code either!)
The alarmist subject line seems to trigger defensive responses. Debian just
seeks is a prototypical VM (with full sources) that takes a prototypical
image from which all other VMs and images may be generated through patches
and file-ins. This is no different from previous quests for a 'minimal' image
from which other projects like Etoys, Croquet can be built. The prototypical
image does not contain any machine-specific code and may be treated like, say
a WAV file

I see conformance Debian guidelines as a positive for the future of Squeak. We
may finally leave behind the annoying MNUs during file-ins. Just imagine
being able to replicate Alan Kay's famous demo image with a simple "apt-get
install squeak-demo" :-).

Subbu

On Wed, 2008-05-21 at 23:34 +0200, Bert Freudenberg wrote: I think there are two issues in his mail. One is the lack of
understanding how squeak works. It'd be kind to spend some words
to make a few issues clear. I would try to figure out if there
are lisp distributions that bring an image with them.

The main concern they have (and a few mentioned in response to
you) is to have a history of the package changes and they need
to have a chance to tweak the package if there are problems.

One thing we already have: a distribution image with a number
that precisely determines an image. Maybe there is some lack
of description what is in it exactly. The shrinking initiatives
will help this a lot.

To build the image from source isn't IMHO quite necessary. A debian
maintainer never changes the source of the original package. The
packages are just patched. I do this on my webserver deployment.

I have stages

- squeak (the downloaded unmodified source)
- bootstrap (patches like Delay, Semaphores, etc)
- base (core packages)
- app (deployable unit)

Every stage is invoked with a startup script that patches the image
and saves it to the next stage. That is exactly what a debian
maintainer will do. Collecting some patch scripts (Smalltalk can
patch itself :) ) and combine them with original source to a debian
package.

We have an upstream which is kind of detailed. AFAIK the upstream
version changes are changesets. So it would be possible to save every
single version bump as changeset and in the debian changelog (which
is needed to produce versioned package anyway).

In my opinion all we need is a script the does one version upgrade
to an image from the commandline. I just don't know if there are
updates of images like etoy that can be done by simply applying a
changeset.

If it is necessary to produce text based diffs this can be produced
on message level. If they need the source it should be possible to
have script that applies the .changes to the .sources and then there
are sources as well.

My 2 cents,

Norbert

On Thursday 22 May 2008 1:15:35 pm Igor Stasenko wrote: Debian does not dictate upstream versions. Their team builds programs from
sources on more than 40 platforms and need to make sure that these versions
are in sync, so that Squeak 3.10 means Squeak 3.10 on all their platforms.
Since image is not a text file, they must be curious about how to 'build' it
on different platforms in a consistent way. You could make out a case that
images, like font files, don't contain any machine-specific or platform
specific instructions and therefore need not be compiled across different
architectures.

If (When?) Squeak image is accepted by Debian, then we can expect to read
active essays from small 50MB distros or Live CDs.

Subbu