RFC: Unix 3.11.3-2116 VM

 
Unix VM hackers,

I've published a preview of the 3.11.3 Unix VM for the amusement of  
alpha testers everywhere.  Lots of things are bound to be broken and/
or forgotten and/or incomplete and/or begging for improvement.  I'm  
not quite finished tweaking but comments and suggestions are  
definitely welcome at this stage!

Grab sources from the trunk of Subversion, or look in

   http://squeakvm.org/unix/alpha

for archives of sources and binaries for i386 (Linux, NetBSD, FreeBSD  
and Solaris) and powerpc (Darwin).  (Files ending in '.sh' are self-
extracting archives; run as shell script and follow instructions.)

The noticeable changes since 3.10 are:

0. Plugins have been regenerated from latest releases

At least for those on squeaksource.  Any hiding in more obscure  
places might still be stale.

1. New build process based on CMake

See README in the source archive or platforms/unix/README.CMake in  
the repository.

2. Different plugin and FFI library search strategy

The default plugin directory is now the VM binary directory.  This is  
correct for installed VMs where the binary lives alongside the  
plugins in /usr/.../lib/squeak/version/squeakvm.  A launcher script  
installed as /usr/.../bin/squeak figures out where the VM is (either  
installed or running from a build directory) and adds a -plugins  
option appropriately.  The script also does the pulse audio OSS  
kludge if necessary.  (Another script, squeak.sh, is intended to  
launch Squeak from a menu.  It's begging for improvement.)

SQUEAK_PLUGINS (or the argument to -plugins) can be set to a colon-
separated list of places to look for plugins.  (Dave can debug one  
plugin at a time using 'squeakvm -plugins bld/%n:/usr/lib/squeak/
<version>'.)

Libraries are searched only by whatever mechanisms dlopen() uses.  
Applications that used to rely on the VM searching every nook and  
cranny for FFI callouts may have to set LD_LIBRARY_PATH, or whatever  
local equivalent is provided, to ensure their external functions'  
libraries are found.

3. New numbering scheme

The VM is now numbered X.Y.Z-R where X.Y.Z are the version of VMMaker  
used to generate the Interpreter and plugins, and R is the Subversion  
repository revision of the Unix support code.  (The archives are  
named X.Y.Z.R, just because rpmbuild is grumpy otherwise.)

Cheers,
Ian

 
Hi Ian,

 

On 31.08.2009, at 11:08, Ian Piumarta wrote:

> Unix VM hackers,
>
> I've published a preview of the 3.11.3 Unix VM for the amusement of  
> alpha testers everywhere.  Lots of things are bound to be broken and/
> or forgotten and/or incomplete and/or begging for improvement.  I'm  
> not quite finished tweaking but comments and suggestions are  
> definitely welcome at this stage!

Looks nice :) Is the compiling faster too? It sure feels like ...

What's the equivalent of configure --without-gl now? I need that to  
build for OLPC, my compiling machine has GL installed, but the XO not,  
so I explicitly need to disable it.

Could we define SUGAR by default? The additions are minimal and it's  
just easier to use one VM for all.

Do we depend on libffi even on x86 Linux now? From my log:

-- SqueakFFIPrims: any-libffi
-- checking for module 'libffi'
--   package 'libffi' not found
-- Looking for ffi.h
-- Looking for ffi.h - not found
-- Looking for ffi/ffi.h
-- Looking for ffi/ffi.h - not found
!! SqueakFFIPrims disabled

And it was not built indeed (this is on Fedora 10).

But it ran Etoys under Sugar on first try (I only patched in the  
#define SUGAR) ... yay!

- Bert -

 
Hi Eliot,

On Aug 31, 2009, at 10:49 AM, Eliot Miranda wrote:

> Does the build still create symbolic links

I don't create any symlinks, either during the build or during the  
install.  There are still two in the unix/doc directory.  I assume  
these are irrelevant.

> At Qwaq we [...] keep the Unix vm in a gzipped tar file [...]  
> unpacking the tar, and doing a diff -r, which is tedious.

For a small, tax-deductible donation I'm sure I could figure out how  
to do without symlinks.

Oh, wait.  I already did.  There goes my fund-raising plan.

Cheers,
Ian

 

On 31.08.2009, at 11:08, Ian Piumarta wrote:

> Unix VM hackers,
>
> I've published a preview of the 3.11.3 Unix VM for the amusement of  
> alpha testers everywhere.  Lots of things are bound to be broken and/
> or forgotten and/or incomplete and/or begging for improvement.  I'm  
> not quite finished tweaking but comments and suggestions are  
> definitely welcome at this stage!

Noticed the "PATH=/usr/bin:/bin" in squeak.in - is it kosher to mess  
with the user's path?

Also, I think the initial assignments for "me", "bindir", and  
"prefix", need to quote their arguments to properly deal with spaces  
in path names.

- Bert -

 
Hi Bert,

> Looks nice :) Is the compiling faster too? It sure feels like ...

There is no more libtool script wrapped around every invocation of  
the compiler and linker.  Whether it's faster or not probably depends  
to a significant degree on how fast the libtool script ran on your  
machine.  OTOH, there are more recursive invocations of make than  
there used to be just to traverse the build tree.  For me it's a lot  
faster compiling from scratch but a low slower (relative to a  
fraction of a second ;) to check all the dependencies when everything  
is already up to date.

> What's the equivalent of configure --without-gl now?

You can turn off any plugin using '--without-PluginName' so '--
without-B3DAcceleratorPlugin' should do what you want.  But I think  
the runtime GL dependency might be attached to vm-dpy-X11 at the  
moment, which is wrong.  I'll fix it so that either B3D carries the  
dependency and/or special-case '--without-opengl' to remove both B3D  
and the GL dependency in X11.

> Could we define SUGAR by default? The additions are minimal and  
> it's just easier to use one VM for all.

Absolutely.  Is config.h good enough or do you need a -D in the  
compiler flags?

> Do we depend on libffi even on x86 Linux now?

Shouldn't do.

> From my log:
> -- SqueakFFIPrims: any-libffi

This is likely a bug in the FFI config logic.  On linux/x86 the  
plugin should not be trying to use the any-libffi glue.

> But it ran Etoys under Sugar on first try (I only patched in the  
> #define SUGAR) ... yay!

Indeed! :)

Cheers,
Ian

 
On Aug 31, 2009, at 11:40 AM, Bert Freudenberg wrote:

> Noticed the "PATH=/usr/bin:/bin" in squeak.in - is it kosher to  
> mess with the user's path?

It's less important to mess with the path in a script that will never  
run as root.  I could remove the line.

> Also, I think the initial assignments for "me", "bindir", and  
> "prefix", need to quote their arguments to properly deal with  
> spaces in path names.

Absolutely right.  I've been trying hard to remember to do this  
recently, often not succeeding.

Cheers,
Ian

 

On 31.08.2009, at 20:49, Ian Piumarta wrote:

I only tried from scratch and it felt faster.

>> What's the equivalent of configure --without-gl now?
>
> You can turn off any plugin using '--without-PluginName' so '--
> without-B3DAcceleratorPlugin' should do what you want.  But I think  
> the runtime GL dependency might be attached to vm-dpy-X11 at the  
> moment, which is wrong.  I'll fix it so that either B3D carries the  
> dependency and/or special-case '--without-opengl' to remove both B3D  
> and the GL dependency in X11.

Thanks.

>> Could we define SUGAR by default? The additions are minimal and  
>> it's just easier to use one VM for all.
>
> Absolutely.  Is config.h good enough or do you need a -D in the  
> compiler flags?

I don't care, I just want it on.

>> Do we depend on libffi even on x86 Linux now?
>
> Shouldn't do.
>
>> From my log:
>> -- SqueakFFIPrims: any-libffi
>
> This is likely a bug in the FFI config logic.  On linux/x86 the  
> plugin should not be trying to use the any-libffi glue.


Should I send any debug info? Not sure where to look.

- Bert -

 
On Aug 31, 2009, at 11:59 AM, Bert Freudenberg wrote:
> Should I send any debug info? Not sure where to look.

Don't worry about that (yet): I will try to reproduce the problem.

Cheers,
Ian

 

On 31.08.2009, at 21:06, Ian Piumarta wrote:

> On Aug 31, 2009, at 11:59 AM, Bert Freudenberg wrote:
>> Should I send any debug info? Not sure where to look.
>
> Don't worry about that (yet): I will try to reproduce the problem.



One more thing: is the plugin naming scheme (so.* instead of *.so)  
intentional?

- Bert -

 
On Aug 31, 2009, at 1:41 PM, Bert Freudenberg wrote:

> One more thing: is the plugin naming scheme (so.* instead of *.so)  
> intentional?

Yes.  It prevents plugins from shadowing libraries loaded via FFI.

Cheers,
Ian

 
On Aug 31, 2009, at 2:08 AM, Ian Piumarta wrote:

>   http://squeakvm.org/unix/alpha

is now at 3.11.3-2122 fixing the issues raised by Bert.

Cheers,
Ian

 
Begin forwarded message:

>



Can we add this to the VM? Helps debugging frozen images. Fully  
patched file attached below.

Andreas, is this the patch you are still using? Or did you hook up  
printAllStacks() instead?

- Bert -

 
On Tue, Sep 01, 2009 at 03:53:22PM +0200, Bert Freudenberg wrote:
One caution here, it is common for the SIGUSR1 and SIGUSR2 to be used
by third party libraries that someone might want to run with Squeak.
For example, the commercial c-tree isam libraries hook both these signals
(just something I happen to have at hand right now as an example), and
I would not be surprised if various other sound, video, or database
libraries do the same.  If we want to hook the SIGUSR1 signal it would
be best to have this as a settable option.

I did not look at this carefully; apologies if this is already included
in the patch.

Dave

 
On Sep 1, 2009, at 6:53 AM, Bert Freudenberg wrote:

>> kill -USR1 12345
> Can we add this to the VM? Helps debugging frozen images. Fully  
> patched file attached below.

The Unix VM used to do exactly this.  It still does, but the stack  
printing was subsumed by emergencyDump() (which, if enabled, prints a  
stack and saves an image file after SIGHUP or SIGQUIT).  This also  
avoids a conflict of interest with the framebuffer driver which needs  
to use a SIGUSR.

Is enabling IMAGE_DUMP in sqUnixMain.c sufficient for what you need?  
We could also have an option that names an image dump file, disabling  
the dump if no name is given but still printing the stack.

BTW: it is extremely dangerous to print the stack asynchronously from  
within an interrupt handler.  In the Unix VM, SIGHUP sets a flag that  
is checked and prints the stack from ioProcessEvents (which is safe)  
and SIGQUIT prints it asynchronously from the handler.

Cheers,
Ian

>
> Andreas, is this the patch you are still using? Or did you hook up  
> printAllStacks() instead?
>
> - Bert -
>
> <sqUnixMain.c.gz>

 

On 01.09.2009, at 17:28, Ian Piumarta wrote:


Ah, I had not noticed this code before. Cool :) Yes, enabling this  
would be great.

An option to toggle it would be nice, too, though I wouldn't mind  
either way.

- Bert -

 
Bert Freudenberg wrote:
> Can we add this to the VM? Helps debugging frozen images. Fully patched
> file attached below.
>
> Andreas, is this the patch you are still using? Or did you hook up
> printAllStacks() instead?

The latter. We use printAllStacks() now. And yes, it's *extremely* handy
if your server runs at 100% CPU for no apparent reason.

Cheers,
   - Andreas

 

On Sep 1, 2009, at 9:28 AM, Ian Piumarta wrote:

> Is enabling IMAGE_DUMP in sqUnixMain.c sufficient for what you  
> need?  We could also have an option that names an image dump file,  
> disabling the dump if no name is given but still printing the stack.

I was looking at this code earlier this year.  I couldn't convince  
myself that the resulting image would actually be usable.  If the VM  
just dumps the image, then won't you (at the very least) lose all  
your file handles including the handle to the changes file?  And if  
you've lost your handle to the changes file then the image won't, for  
any practical purposes be usable.

Am I missing something?

David

 
Ian Piumarta wrote:
> Is enabling IMAGE_DUMP in sqUnixMain.c sufficient for what you need?  We
> could also have an option that names an image dump file, disabling the
> dump if no name is given but still printing the stack.

Saving the image can be useful but only when it's decoupled from dumping
the call stack. Dumping the stack is a non-destructive action so it can
be done to investigate the status of the image. I have seen situations
where images were at 100% but where I found that the activity actually
didn't cause any harm, so no emergency action was necessary. Think of
the stack dump as purely informative: You want to find out what's going
on without actually shooting the image. Whatever does that works for me.

> BTW: it is extremely dangerous to print the stack asynchronously from
> within an interrupt handler.  In the Unix VM, SIGHUP sets a flag that is
> checked and prints the stack from ioProcessEvents (which is safe) and
> SIGQUIT prints it asynchronously from the handler.

Interesting. We've never had any issues with that.

Cheers,
   - Andreas

 
David Farber wrote:
The simulator. It can be used to inspect the contents of an image file
regardless of whether you run it or it. Useful for post-mortem analysis.

Cheers,
   - Andreas