VM packaging for Cog transition

 
Bert and others have raised the question of how best to package the
Cog VM and traditional interpreter VM for the next round of VM releases
in December. Cog is a significant advance, and the Squeak board would
like this to be fully supported for the Squeak 4.2 release; clearly it
is a priority for Pharo as well. Meanwhile we have a responsibility for
Etoys, Scratch, OLPC etc to provide VMs that do not break those systems.

So - how best to do this?

For unix VMs, Bert has suggested this:

> Maybe we need a "squeak-interp" and "squeak-cog" binary, with "squeak"
> symlinking to either one (using the alternatives system) or "squeak"
> being a script choosing the right VM based on the image version. But
> we should present a coherent story to the packagers. As well as to
> Squeak users.

Personally I am inclined to think it would be best for this next release
not to automate the selection of VM, but instead have users make an
explicit selection. I say this for two reasons:

1) From a marketing point of view, it is good if users can directly experience
the improvement from Cog. So the message might be to first run in the standard
way (squeak-interp) to get a very cool system that is quite fast, then activate
turbo mode (squeak-cog) to get a system that is very noticeably 3X faster.

2) From a technical and support point of view, there may be various cases
where it will be necessary to say "sorry, you will need to run squeak-interp
if you want to do that." When that happens, it would be best if we do not need
to explain e.g. how to edit a shell script to force it to run squeak-interp.

The above is just my $.02 to get the discussion started. What do others think?

Dave

 
I'm of the mind that Apple takes at the moment. Try the action and then deal with the
failure case.   I'd just ship with the Cog VM turned on.  If people have problems with it
they can pursue how to turn it off.   Otherwise you'll not get people to migrate.

On 2010-11-07, at 12:27 PM, David T. Lewis wrote:
--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================

On 7 November 2010 23:22, John M McIntosh
<[hidden email]> wrote:
>
> I'm of the mind that Apple takes at the moment. Try the action and then deal with the
> failure case.   I'd just ship with the Cog VM turned on.  If people have problems with it
> they can pursue how to turn it off.   Otherwise you'll not get people to migrate.
>

+1
If people like to use older VM(s), they can always download them.

> On 2010-11-07, at 12:27 PM, David T. Lewis wrote:
>


--
Best regards,
Igor Stasenko AKA sig.

 
On 11/7/2010 12:27 PM, David T. Lewis wrote:
> The above is just my $.02 to get the discussion started. What do others think?

I don't really have any skin in the game since this is a Unix specific
issue (i.e., the "squeakvm" dependency) that doesn't really apply on Mac
or Windows. That said, I'll add my $.02 by saying that in our Linux
production systems we do exactly what you were proposing, i.e., we have
fallback VMs installed side-by-side with the latest and symlink the
desired VM. That makes it very easy to switch the default *and* allows
you to be explicit where you want to (i.e., by executing from the
linked-to location).

Cheers,
   - Andreas

 
On Sun, Nov 07, 2010 at 07:29:56PM -0800, Andreas Raab wrote:
Well that is Bert's suggestion exactly, and it is consistent with John's
point of view as well, so it sounds like this is the right thing to do.

Dave

On 08.11.2010, at 04:55, David T. Lewis wrote:

So we would ship two VMs. One is the old interpreter. The other is cog, or the stack interpreter, depending on the machine's architecture.

The two new ones should be in one source archive, IMHO. At build time either cog or stack VM would be selected depending on the machine's architecture. What should the name be for those binaries? Just use "squeakvm-cog" even if it's the Stack VM because it can run the same images?

Would we drop development on the old interpreter? If plugins were compatible between it and the newer VMs (are they?) it should not be much effort to keep it alive.

What about 64 bit images? We don't need extra sources for those VMs, since it's just a build flag. But we need a naming scheme for these too.

- Bert -

 
On Mon, Nov 08, 2010 at 04:03:13AM +0200, Igor Stasenko wrote:
To be clear, the intent is to provide both Cog and the interpreter VM.
The question is how best to package this such that we deliver the benefits
of Cog, but also ensure that the system still works when a child opens
an older Etoys image after their teacher has installed the new VM. In
the near term at least, we also want to have a straightforward way to
answer the question "I tried to do XXX and it does not work, what should
I do?" in cases where XXX does not yet work on Cog.

Dave
 

 
On 11/8/2010 3:35 AM, David T. Lewis wrote:
> To be clear, the intent is to provide both Cog and the interpreter VM.

Agree. I think the first step is to unify the support code. I.e., when
it's all said and done we should have a structure roughly along the
lines of:

platforms/
        Cross/
        Unix/
        Mac/
        Win/
src-interp/
src-jit/

etc. and the *only* difference should be in the src-interp vs. src-jit
directories. In which case we can decide what to build based on the
selecting the appropriate interp/jit source directory.

Cheers,
   - Andreas

 
On Mon, Nov 08, 2010 at 12:31:35PM +0100, Bert Freudenberg wrote:
Yes it should be exactly as you say, but I think we should be cautious
about setting expectations. It may take a while to get this accomplished,
and in the December time frame the important thing is to deliver solid
working Cog and interpreter VMs. If we can merge all the code bases etc
that is great, but let's not make it a precondition for the December VM
and Squeak 4.2 release cycle. We have all seen what happens when a release
team gets caught up in unrealistic development objectives, so let's not
go there ;)

> At build time either
> cog or stack VM would be selected depending on the machine's architecture.
> What should the name be for those binaries? Just use "squeakvm-cog" even
> if it's the Stack VM because it can run the same images?

Someone recently submitted a magic file with good names (Subbu? I can't
find the link). I think the names were squeak-cog, squeak-stack,
squeak-interp or somthing similar. In any case, "interp" can refer to the
traditional interpreter, "stack" can refer to the portable stack interpreter
positioned as a replacement for "interp", and "cog" can refer to the high
performance VM.

> Would we drop development on the old interpreter? If plugins were compatible
> between it and the newer VMs (are they?) it should not be much effort to
> keep it alive.

It's no real trouble to keep it alive and I intend to do so.

The plugins should definitely be identical. The C code generator should
also be the same (I'm currently fumbling my ameteurish way through adoption
of Eliot's enhancements into VMMaker trunk, but it's obvious that the end
result should be the same). Hopefully we will get the old interpreter merged
with Eliot's version (more work than I might have expected) but even if we
can't do that short term, maintaining the old interpreter is easy and I'm
quite happy to continue doing so as needed. Likewise for VMMakerTool, some
folks may not need to use it but it is little or no work to maintain it,
and I use it myself on a regular basis so keeping it alive is no problem.

> What about 64 bit images? We don't need extra sources for those VMs,
> since it's just a build flag. But we need a naming scheme for these too.

This is something that is only in the trunk VMMaker. I am assuming that
it would be good to merge this with the Cog VMMaker, although it is a fair
amount of work with no direct benefit to the Cog VM so I should defer
to Eliot on this. The changes are straightforward, but a lot of code gets
touched and this is an example of something that may take some time and
effort to merge.

Dave
 

 
On Mon, Nov 08, 2010 at 11:43:50AM -0800, Andreas Raab wrote:
To check my understanding, you are suggesting that the src-interp and
src-jit directories would be for the generated sources, as distinct
from the platform support code under platforms/...  and that the structure
of the platforms/... tree would remain unchanged. Is that right?

Dave

 
On 11/8/2010 5:39 PM, David T. Lewis wrote:
Exactly. Most importantly the platforms support code would be
*identical* across the variants and thus eliminate further proliferation
of various bits of C code.

Cheers,
   - Andreas

On 09.11.2010, at 03:44, Andreas Raab wrote:

I'd love to have this structure. Making the platform names more uniform is good. Though I'd retain the space in "Mac OS" because many files in that subtree contain spaces, and it's a visible reminder of that.

Shouldn't we have a single folder for generated plugins? Also, I always felt that "src" was misleading. So how about this:

platforms/
        Cross/
        Unix/
        Mac OS/
        Win/
        iOS/
generated/
        interp/
        jit/
        stack/
        plugins/

However, one reason to put the generated sources inside the platform folders was because often the platform code lagged behind, and only the single maintainer could fix it. A better solution to that would be opening up VM development. As I mentioned before I'd like us to move to a DCVS. If we intend to change the repository structure this would be an excellent time.

- Bert -

 
On Tuesday 09 Nov 2010 3:55:53 pm Bert Freudenberg wrote:
> Shouldn't we have a single folder for generated plugins?
IMHO, plugins should be capable of being built anywhere, even outside the VM
sources. Plugins, by definition, use arch-specific or os-specific features and
the 'upper half' could be squirted into platform directory as necessary.

> Also, I always
> felt that "src" was misleading. So how about this:
>
> platforms/
> Cross/
> Unix/
> Mac OS/
> Win/
> iOS/
I prefer <arch>/<platform>/<os> structure (e.g. x86/pc/Win32, WinCE, Win64,
linux, Mac OS, arm/davinci/linux,..). It is much easier to re-use code and
development tools this way.

> generated/
> interp/
> jit/
> stack/
> plugins/
I would vote to skip generated/ and Cross/ directories and leave all these file
at top level. Apart from C code, there could also be auto-generated
configuration files, Makefiles etc. Why treat C code specially?

> However, one reason to put the generated sources inside the platform
> folders was because often the platform code lagged behind, and only the
> single maintainer could fix it. A better solution to that would be opening
> up VM development.
+1

> As I mentioned before I'd like us to move to a DCVS. If
> we intend to change the repository structure this would be an excellent
> time.
A distributed version manager like git would be useful only for hand-coded
files. What about blobs like images, changesets and sources?

Subbu

 
On Tue, Nov 9, 2010 at 8:56 AM, K. K. Subramaniam <[hidden email]> wrote:

> A distributed version manager like git would be useful only for hand-coded
> files. What about blobs like images, changesets and sources?

How do you figure? Git or Mercurial can handle binary files as well as
subversion.

Colin

 
On Monday 08 Nov 2010 5:01:35 pm Bert Freudenberg wrote:
> The two new ones should be in one source archive, IMHO. At build time
> either cog or stack VM would be selected depending on the machine's
> architecture. What should the name be for those binaries? Just use
> "squeakvm-cog" even if it's the Stack VM because it can run the same
> images?
For some time, we need to build both VMs and select it at run-time in the
launcher script. This shouldn't be a big issue on Linux. See

http://lists.squeakfoundation.org/pipermail/vm-dev/2010-October/005619.html

Subbu

On 09.11.2010, at 18:00, Colin Putney wrote:

We're talking about the VM source tree. So far there are no images in there.

- Bert -

 
On Tuesday 09 Nov 2010 7:00:51 am David T. Lewis wrote:
> Someone recently submitted a magic file with good names (Subbu? I can't
> find the link).
you mean
   http://lists.squeakfoundation.org/pipermail/vm-dev/2010-October/005619.html

The archive doesn't seem to contain the attachment, so I am attaching it
again.

Subbu

On 09.11.2010, at 18:13, K. K. Subramaniam wrote:

That is agreed upon. There will be an interpreter, plus a VM capable of running a cog image.

I was asking how to name that cog/stack vm binary. I don't see a reason to have both cog and stack VMs on one architecture officially. On x86 it would be cog, on everything else stack VM for now. The question is, should that difference be reflected in the binary name? Is it desirable to have both cog and stack VMs installed next to each other? Would people be confused if there is no "cog" binary on their architecture?

- Bert -

 
On 11/9/2010 2:25 AM, Bert Freudenberg wrote:
I don't think these issues are related at all. A particular release will
still bundle a certain version of the platforms tree with a certain
version of the generated code. However, there is no requirement for the
generated code to live inside the platforms directory. To make it one
notch more complicated, our internal structure for VM building actually
puts the various toolchains side-by-side with platforms and
src/generated, i.e.,


cygwinbuild/  "<- Windows Cygwin build"
macbuild/     "<- Mac X-Code build"
mgwbuild/     "<- Windows MingW build"
mscbuild/     "<- Windows Visual C++ build"
unixbuild/    "<- Unix gcc build"
platforms/
src/

This has some advantages for us since we can check in and build VMs from
the same (checked-in) src/ and platforms/ tree in the various
environments, but I'm not sure we need this for the Squeak VM in general.

Cheers,
   - Andreas

 
On Tuesday 09 Nov 2010 10:30:17 pm Colin Putney wrote:
> On Tue, Nov 9, 2010 at 8:56 AM, K. K. Subramaniam <[hidden email]>
wrote:
> > A distributed version manager like git would be useful only for
> > hand-coded files. What about blobs like images, changesets and sources?
>
> How do you figure? Git or Mercurial can handle binary files as well as
> subversion.
Yes. but the history gets quite big with image blobs.

Bert's reply is ingenious ;-). Generated files are intermediate files, not true
source files. True "sources" are actually Slang code in VMMaker image and these
should be in version control.

When bootstrapping VM on a new platform, a sub-set of these intermediate files
along with hand-coded source files are compiled generate a proto-vm. The proto-
vm runs VMMaker image to translate all Slang code into C so that the regular
VM and plugins can now be built and the proto-vm can be discarded.

The VMMaker image is a critical part of bootstrap and should be put under
version control.

Subbu