Fwd: [squeak-dev] Teleplace Cog VMs are now available

>  
> ---------- Forwarded message ----------
> From: Eliot Miranda <[hidden email]>
> Date: 2010/6/20
> Subject: [squeak-dev] Teleplace Cog VMs are now available
> To: The general-purpose Squeak developers list
> <[hidden email]>, etoys dev
> <[hidden email]>, Pharo Development
> <[hidden email]>, Squeak Virtual Machine
> Development Discussion <[hidden email]>
>
> Hi All,
>
> it gives me great pleasure to announce that the Teleplace Cog VMs are
> now available.  Huge thanks to all at Teleplace who have given me the
> opportunity to build Cog and release it as open source, been willing
> guinea pigs braving its bugs, and providing indispensable
> participation in getting Cog to its current state.  Huge thanks are
> also due to the original Back To The Future team whose VMMaker Cog
> extends to write the VM, and to Peter Deutsch from whom I've taken
> many ideas.
>
> This release contains two VMs.  The Stack VM, is a cross-platform
> interpreter that uses context-to-stack mapping to achieve modest
> performance gains.  The Cog VM is a just-in-time compiler that
> currently supports only x86 that builds upon the Stack VM to achieve
> substantial performance improvements.  The release is in the form of a
> Monticello package containing the VMMaker source and a tarball
> containing the platform sources, the generated sources and a Squeak
> 4.1 image containing the VMMaker sources.  Download both at
>
> http://ftp.squeak.org/Cog/VMMaker-oscog.11.mcz
>
> http://ftp.squeak.org/Cog/OpenSourceCog.tar.gz
>
> Cog VMs:
>
> The Cog VMs are Squeak/Croquet VMs that run closure
> Squeak/Croquet/Pharo/Cuis images. The VMs support existing plugin
> source but will require plugins to be recompiled as the VM_PROXY_MAJOR
> plugin api has been extended.
>
> This release contains two distinct VMs, the StackInterpreter and the
> Cogit.  The StackInterpreter is a fully-portable plug-in replacement
> for the current closure Squeak VMs and images.  The Stack VM uses
> context-to-stack mapping and a somewhat improved garbage collector to
> achieve modest but useful performance gains in the 10% to 15% range.
> The StackInterpreter is intended to supersede the Squeak VM on
> platforms where the Cogit cannot be used.  The Cogit extends the
> StackInterpreter with a just-in-time compiler that uses aggressive
> inline caching techniques to deliver substantial performance gains in
> the 3x to 15x range, depending on benchmark.  The Cogit currently
> supports only x86 and the floating-point primitives and parts of the
> platform support code depend on SSE2.  I hope members of the community
> will attempt to port it, e.g. to ARM, PowerPC and x86-64.  The Cogit
> (excuse the pun) is so named because it is both an interpreter and a
> JIT, choosing not to generate machine code for large methods,
> interpreting them instead, the default policy being not to JIT methods
> with more than 60 literals.
>
> The Cog VM requires a few minor image changes all in
> image/NecessaryImageChangesForCogToWork.1.cs.  The JIT's machine-code
> SmallInteger primitives insist on a SmallInteger receiver so the
> primitives in LargePositiveInteger = ~= bitAnd: bitOr: butShift: and
> bitXor: cannot be used and these methods must be deleted.  The Cogit
> inlines the address of the Character instance table, Smalltalk
> specialObjectsArray at: 25, into the machine-code at: primitive for
> faster ByteString>>at: and so the table cannot be rebuilt in
> SmalltalkImage>>recreateSpecialObjectsArray.  The new version
> preserves the existing table.  Both VMs maintain floats in platform
> order to ease implementation of machine code floating-point
> primitives, and hence internally are in little-endian order instead of
> big-endian in current Squeak images.  While the VMs convert float
> order automatically on load they do require special accessing
> primitives Float>>basicAt: & Float>>basicAt:put: that undo the
> reversal and answer Float contents in big-endian order so that e.g.
> Float>>hash is unchanged.  The methods assume these primitives can
> fail, allowing the code to be used on current Squeak VMs.
>
> The image/VMMaker-Squeak4.1.image is a Squeak 4.1 image, runnable with
> the current Squeak VMs, that contains these changes, and can hence
> also be run with a Cog VM.  But beware, once an image has been saved
> on Cog it cannot be run by an existing Squeak VM, because existing VMs
> cannot undo the Float order change.
>
> Platform Subsystem:
>
> Most of the platform subsystem is unchanged but there are some
> important changes that need description.  The biggest change is the
> heartbeat and the clock in platforms/unix/vm/sqUnixHeartbeat.c and
> platforms/win32/vm/sqWin32Heartbeat.c.  The Cog VMs avoid the slow and
> variable interruptCheckCounter, folding the event check into the stack
> overflow check on frame build.  The heartbeat, typically 500Hz or
> 1KHz, changes the stackLimit to a value that will always fail.  On the
> next frame building send the VM will enter stack overflow handling
> that, as a side effect, will also check for events.  This is more
> efficient than the update of interruptCheckCounter and much more
> regular.  If one is running code that executes long-running primitives
> (e.g. large integer arithmetic) the counter approach will result in
> too low an interrupt check frequency, and conversely if one is running
> normal code the interrupt check frequency can be very high.
>
> The heartbeat also maintains a 64-bit microsecond clock, UTC
> microseconds from 1901, from which the backward-compatible millisecond
> and second clocks are derived.  Primitives exist to answer UTC
> microseconds and local microseconds.  Updating the clock in the
> heartbeat results in a 1 or 2 millisecond resolution but avoids the
> cost of accessing the OS time on every prim tie which we've found
> important for performance at Teleplace.  The 64-bit microsecond clocks
> provide a unified time basis and eliminate wrapping (for the next
> 54,000 years at least).  I hope community images will move to these
> clocks.  It's worked well in VisualWorks.
>
> Another significant change is in the external semaphore table support
> code.  This is now lock-free at the cost of having to specify a
> maximum number of external semaphores at start-up (default 256).  The
> support code for the lock-free data structures are processor-specific
> and is currently implemented only for x86 and gcc-compatible
> compilers; see platforms/Cross/vm/{sqAtomicOps.h,sqMemoryFence.h}.
>
> There is also improved crash reporting code that prints a primitive
> log and a C backtrace in addition to the Smalltalk backtrace.  See
> platforms/Mac OS/vm/sqMacMain.c, platforms/unix/vm/sqUnixMain.c,
> platforms/win32/vm/sqWin32Intel.c &
> platforms/win32/vm/sqWin32Backtrace.c.
>
> Finally there is support for the QVMProfiler, a pc-sampling profiler
> for profiling at the VM level.  See
> platforms/unix/vm/sqUnixVMProfile.c and
> platforms/win32/vm/sqWin32VMProfile.c.  The profiler itself is in the
> VMMaker image described below in Qwaq-VMProfiling.
>
> There are also changes to do with Teleplace-specific extensions to the
> HostWindowPlugin but these are not essential to Cog.
>
> VMMaker and Slang:
>
> The image/VMMaker-Squeak4.1.image Squeak 4.1 image contains the
> complete Cog VMMaker with necessary support code for simulation. This
> image was used to generate the sources in the src and stacksrc
> directories.
>
> Cog's VMMaker is substantially revised and extended from the current
> VMMaker.  It supports multiple classes, not just Interpreter and
> superclasses, because both context-to-stack mapping and the Cogit are
> too complex to write monolithically.  Classes can specify
> ancilliaryClasses and ancilliaryStructClasses, such as
> CoInterpreterStackPage, CogMethod and CogAbstractInstruction.  The
> Monticello package version is included in the header of all generated
> files and constitutes the version stamp for generated code.  Code is
> generated in sorted order so that minor changes in the Smalltalk
> source produce correspondingly minor changes in the generated code.
> The gnuification step is built-in to VMMaker.  No effort has been made
> to maintain 64-bit compatibility.  Apologies, this was unaffordable.
>
> The VMMaker generates a single source tree used by all platforms.
> Instead of deciding at generation time whether to use the Interpreter
> struct the generated code depends on the SQ_USE_GLOBAL_STRUCT define
> which can be overridden in platform makefiles.  All plugins live in
> src/plugins and platform makefiles along with plugins.int and
> plugins.ext files in the build subdirectories decide which plugins are
> built as external or internal.  The VM Generation Workspace from
> Workspace.text workspace contains dots to generate the sources.  We no
> longer use the VMMakerTool since there should be nothing
> platform-specific in the generated sources (if we add ports to other
> ISAs all their source can be included and selected as required by the
> platform makefiles).
>
> Since the Cogit generates x86 machine code simulation is much more
> complex.  There is a support plugin,
> platforms/Cross/plugins/BochsIA32Plugin that depends on a large
> simulation of the x86 family implemented in C++ (see
> processors/IA32/bochs) and on Alien.  I use the simulator frequently
> (but note that I haven't had time to build a working version for
> Squeak 4.1).  I have tested Cog simulation in this image, running on
> the image/VMMaker-Squeak4.1.image itself.  The VM Simulation Workspace
> in the VMMaker image contains an example doit that starts the
> simulator. Be patient, even on a fast machine unhibernating the Squeak
> display background image takes nearly a minute.  Native fonts do not
> (yet) simulate correctly, but the system runs.  But note that I have
> only attempted to build and run the simulator on Mac OS X.  I expect
> Bochs can be built on linux and win32 but I have not tried.  By the
> way, I've not described how to run the Bochs simulator on the current
> Squeak VM.  That's because the plugin depends on the heartbeat to
> break out of simulation occasionally via a new interpreterProxy entry
> point setInterruptCheckChain.  As this isn't supported by the current
> Squeak VMs the plugin would require modification.  So to simulate
> first build either of the Cog VMs and then run the simulation with it.
>
> There are a number of unpublished changes to the base other than those
> in NecessaryImageChangesForCogToWork.1.cs.  This is partly laziness on
> my part, partly avoiding publishing things in advance of Cog.  These
> changes are better motivated once Cog is in use.  There are changes to
> the "translated primitives" (see implementors of translatedPrimitives)
> which replace messages with method tags for generation directives.
> The Cog VMMaker uses Object>>perform:with:with:with:with: &
> Object>>perform:with:with:with:with:with: during simulation, and
> Collection>>#fold: & SquenceableCollection>>#copyUpThrough: during
> generation.  Object>>inline: and Object var:declareC:, which are
> mispackaged in Kernel in Squeak 4.1 are obsolete (method tags being
> used instead) and have been removed. I have changed Integer>>hex and
> Integer>>hex8 back to their original semantics as of 3.8.  Backward
> compatibility is important and one can easily add new selectors if one
> wants different functionality.  VMMaker was here first ;)
>
> Tarball:
>
> The top-level directories in the tarball are
>
> src
>
> the tree for the Cog generated sources including all plugins
>
> stacksrc/vm
>
> the directory containing the Stack VM source (plugins can be taken
> from above)
>
> platforms
>
> the usual svn platform tree but including Cog specific changes such as
> the heartbeat
>
> processors
>
> the tree containing simulation support code, i.e. the bochs C++ x86
> simulation library, along with a potential ARM, PowerPC & MIPS
> simulator, Skeye.
>
> image
>
> the Cog-prepared Squeak 4.1 VMMaker image
>
> scripts
>
> some svn scripts to revert unchanged plugins that haven't really
> changed
>
> cygwinbuild
>
> the win32 build directory
>
> winbuild
>
> the old win32 build directory for minnow gcc 2.95.  Not entirely
> obsolete as the cygwin build as yet fails to generate a functional
> FFIPlugin
>
> macbuild
>
> the CoreVM.xcodeproj and support build projects for Mac OS X 10.5 or
> better
>
> unixbuild
>
> the build directory for linux
>
> Building Cog:
>
> Each build directory above contains a HowToBuild file that describes
> building in more detail.  The build directories only contain Cogit
> makefiles.  f you want to build a Stack VM you're on your own but this
> is very close to the existing Squeak VM build.
>
> Status:
>
> The Cogit VM has been our sole VM at Teleplace for nearly a year.  We
> do occasionally find bugs and there are almost certainly areas of
> functionality that we have not touched (for example I know that
> co-routining does not yet work).  If you find a bug please try and
> create a reproducible test case and let me know.  I can't promise to
> take a look or fix it but I am motivated to do so and will try my best
> as time allows.  Better still if you find and fix bugs be sure to let
> me know.
>
> License (MIT):
>
> All contributions from Teleplace in this release are
>
> Copyright (c) 2010 Teleplace, Inc.
>
> Permission is hereby granted, free of charge, to any person obtaining
> a copy
>
> of this software and associated documentation files (the "Software"),
> to deal
>
> in the Software without restriction, including without limitation the
> rights
>
> to use, copy, modify, merge, publish, distribute, sublicense, and/or
> sell
>
> copies of the Software, and to permit persons to whom the Software is
>
> furnished to do so, subject to the following conditions:
>
> The above copyright notice and this permission notice shall be
> included in
>
> all copies or substantial portions of the Software.
>
> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> EXPRESS OR
>
> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> MERCHANTABILITY,
>
> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
> SHALL THE
>
> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
>
> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> ARISING FROM,
>
> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
> IN
>
> THE SOFTWARE.
>
> Eliot Miranda
>
> June 2010
>
> --
> =================================================
> Germán S. Arduino <gsa @ arsol.net> Twitter: garduino
> Arduino Software & Web Hosting http://www.arduinosoftware.com
> PasswordsPro http://www.passwordspro.com
> =================================================
>
>
>
>