Alien support in standard VM?

Hi,

would it be a problem to package Alien support with the VMs by
default? Are there any reasons for not doing it? Can I help making
this happen?

I think delivering standard images with Alien right away would be a
strong plus, but of course the VMs need to support that.

Best,

Michael

Actually, I think Michael's question was more on an immediate basis. As it stands there is a significant number of patches that one needs to scrape from the four corners of the universe before one can even begin to build the Alien FFI plugin. It would be helpful if this could be folded into VMMaker so that the plugin can be built out of the box.

Cheers,
 - Andreas

Eliot Miranda wrote:

 

------------------------------------------------------------------------

On Wed, Sep 2, 2009 at 9:08 AM, Michael Haupt <[hidden email] <mailto:[hidden email]>> wrote:


   Hi,

   would it be a problem to package Alien support with the VMs by
   default? Are there any reasons for not doing it? Can I help making
   this happen?

   I think delivering standard images with Alien right away would be a
   strong plus, but of course the VMs need to support that.


Two things need to happen.
1. Alien's data manipulation facilities need to be separated from Alien's very limited call facilities.
2. FFI's marshalling needs to be extended to marshal Aliens.

Alien's call support only works on very simple ABIs such as typical IA32 (x86) ones.  As soon as one confronts a machine with a register-based calling convention the simple approach falls apart for many cases (structure arguments, floating-point).

Further on I want to build an ABI interface compiler above the JIT that can efficiently marshal arbitrary calls in arbitrary ABIs, essentially reimplementing libffi.

So Michael, it would help if you would extract the data manipulation parts of the IA32ABI plugin into e.g. Alienlugin and rewrite all the data manipulation methods to use it.  Then shoe-horn Alien marshalling into the FFIPlugin.


Why reimplement libffi instead of use it directly?

First, libffi only does call-outs, not call-backs.  Alien's callback architecture is quite nice; one gets a pointer to a register struct (null on x86) and the stack pointer at the point of callback, wrapped up in Aliens and extracts the arguments there-from.  But this needs to be wrapped up in some platform-specific abstractions to present the programmer with a portable interface.

Second, libffi does nothing to help translating Smalltalk objects to C equivalents (mapping String instances to to null-terminated char * or wchar_t * arrays, etc, etc).

Third, libffi's architecture involves building up an AST of a call signature, passing the AST to a compiler which answers some marshalling thunk, and then using the marshallng thunk to make a call.  The AST and the marshalling thunk are C objects that one has to associate with e.g. a Smalltalk compiled method and arrange that if the compiled method is GCed the marshalling thunk is reclaimed.

Fourth, the marshalling thunk itself looks to be interpreted, and to apply it one must call the marshaling machinery with the thunk and a pointer to the vector of arguments, to have the call made.  This is a little bit too much intervening machinery.

I'd rather have the JIT build the marshalling code directly into the machine code generated for an external call method, translating some RTL sequence that describes the marshalling operations in a platform-specific manner, and write the call spec to RTL compiler in Smalltalk and have it live in the image.

libffi makes a great reference for the ABI semantics, and can also be a source of tests.
 

   Best,

   Michael


cheers!
Eliot

I believe I had all the VMMaker changes in the alien vmmaker MC changesets. For the macintosh I have those in my current build image and currently I am merging in the production Vmmaker source.   Obviously it would be good if the alien vmmaker stuff was folded into the production vmmaker.

Also there was some confusion about the earlier changes for vm callback submitted by qwaq a few years back.
I have for example in my PRIVATE sqVirtualMachine.h (did I mention PRIVATE)  this code to expose the
callbackEnter:  callbackID: addGCRoot  removeGCRoot

If these are to be replaced by Alien then someone should clarify that since Mar on Squeak is exploiting the callbackEnter:
to do Cocoa controller callbacks on os-x via the Objective-C plugin SqueakProxy logic

#if VM_PROXY_MINOR > 7
       sqInt  (*internalIsImmutable)(sqInt oop);
       sqInt  (*internalIsMutable)(sqInt oop);
       sqInt  (*primitiveFailFor)(sqInt code);
       sqInt  (*classAlien)(void);
       sqInt *(*getStackPointer)(void);
       sqInt  (*sendInvokeCallbackStackRegistersJmpbuf)(sqInt thunkPtrAsInt, sqInt stackPtrAsInt, sqInt regsPtrAsInt, sqInt jmpBufPtrAsInt);
       sqInt  (*reestablishContextPriorToCallback)(sqInt callbackContext);
       sqInt  (*classUnsafeAlien)(void);
       /* New methods for proxy version 1.8 */
       
       /* callbackEnter: Re-enter the interpreter loop for a callback.
       Arguments:
       callbackID: Pointer to a location receiving the callback ID
       used in callbackLeave
       Returns: True if successful, false otherwise */
       sqInt (*callbackEnter)(sqInt *callbackID);
       
       /* callbackLeave: Leave the interpreter from a previous callback
       Arguments:
       callbackID: The ID of the callback received from callbackEnter()
       Returns: True if succcessful, false otherwise. */
       sqInt (*callbackLeave)(sqInt  callbackID);
       
       /* addGCRoot: Add a variable location to the garbage collector.
       The contents of the variable location will be updated accordingly.
       Arguments:
       varLoc: Pointer to the variable location
       Returns: True if successful, false otherwise. */
       sqInt (*addGCRoot)(sqInt *varLoc);
       
       /* removeGCRoot: Remove a variable location from the garbage collector.
       Arguments:
       varLoc: Pointer to the variable location
       Returns: True if successful, false otherwise.
       */
       sqInt (*removeGCRoot)(sqInt *varLoc);
#endif

On 2-Sep-09, at 7:00 PM, Eliot Miranda wrote:

On Wed, Sep 2, 2009 at 9:54 AM, Andreas Raab <[hidden email]> wrote:

Actually, I think Michael's question was more on an immediate basis. As it stands there is a significant number of patches that one needs to scrape from the four corners of the universe before one can even begin to build the Alien FFI plugin. It would be helpful if this could be folded into VMMaker so that the plugin can be built out of the box.

Right.  But before it can go into the VM we need to throw away the x86-specificities.  And that means splitting the data manipulation facilities from the ABI machinery.  I can do that, but Michael volunteered :)

I think the pragma issue has already been solved right?  There was a commit for the compiler yesterday that did that.


Cheers,
 - Andreas

Eliot Miranda wrote:


------------------------------------------------------------------------




On Wed, Sep 2, 2009 at 9:08 AM, Michael Haupt <[hidden email] <mailto:[hidden email]>> wrote:


  Hi,

  would it be a problem to package Alien support with the VMs by
  default? Are there any reasons for not doing it? Can I help making
  this happen?

  I think delivering standard images with Alien right away would be a
  strong plus, but of course the VMs need to support that.


Two things need to happen.
1. Alien's data manipulation facilities need to be separated from Alien's very limited call facilities.
2. FFI's marshalling needs to be extended to marshal Aliens.

Alien's call support only works on very simple ABIs such as typical IA32 (x86) ones.  As soon as one confronts a machine with a register-based calling convention the simple approach falls apart for many cases (structure arguments, floating-point).

Further on I want to build an ABI interface compiler above the JIT that can efficiently marshal arbitrary calls in arbitrary ABIs, essentially reimplementing libffi.

So Michael, it would help if you would extract the data manipulation parts of the IA32ABI plugin into e.g. Alienlugin and rewrite all the data manipulation methods to use it.  Then shoe-horn Alien marshalling into the FFIPlugin.


Why reimplement libffi instead of use it directly?

First, libffi only does call-outs, not call-backs.  Alien's callback architecture is quite nice; one gets a pointer to a register struct (null on x86) and the stack pointer at the point of callback, wrapped up in Aliens and extracts the arguments there-from.  But this needs to be wrapped up in some platform-specific abstractions to present the programmer with a portable interface.

Second, libffi does nothing to help translating Smalltalk objects to C equivalents (mapping String instances to to null-terminated char * or wchar_t * arrays, etc, etc).

Third, libffi's architecture involves building up an AST of a call signature, passing the AST to a compiler which answers some marshalling thunk, and then using the marshallng thunk to make a call.  The AST and the marshalling thunk are C objects that one has to associate with e.g. a Smalltalk compiled method and arrange that if the compiled method is GCed the marshalling thunk is reclaimed.

Fourth, the marshalling thunk itself looks to be interpreted, and to apply it one must call the marshaling machinery with the thunk and a pointer to the vector of arguments, to have the call made.  This is a little bit too much intervening machinery.

I'd rather have the JIT build the marshalling code directly into the machine code generated for an external call method, translating some RTL sequence that describes the marshalling operations in a platform-specific manner, and write the call spec to RTL compiler in Smalltalk and have it live in the image.

libffi makes a great reference for the ABI semantics, and can also be a source of tests.


  Best,

  Michael


cheers!
Eliot

--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================