Smalltalk › Squeak › Squeak VM

Alien and FFI integration

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

2 messages Options

Andreas.Raab

Alien and FFI integration

[Sorry for posting this again; I used the wrong email address the first
time]

Hi Eliot -

[cc: vm-dev to see if anyone has some good ideas] I was thinking about
how one would move between FFI and Alien more freely and one thing that
seemed interesting is to have the compiler simply generate Alien call
code for FFI specs. E.g., when the compiler encounters, say:

system: aString
<apicall: ulong 'system' (char*) module: 'libc'>
^self externalCallFailed

it would generate something like here:

system: aString
AlienCall ifNil:[
"Register this method to be flushed on shutdown"
Alien registerMethod: thisContext method.
"Lookup the call if it isn't cached"
AlienCall := Alien lookup: 'system' inLibrary: 'libc'.
"Result can use a cached Alien since it never changes"
AlienResult := Alien new: 4.
].
"Arg1 needs to be reloaded every time we call"
Arg1 := Alien fromString: aString.
AlienCall primFFICallResult: AlienResult with: Arg1.
Arg1 free. "or however one frees those"
"XXXX: Would need some success indication here"
success ifFalse:[^AlienResult asUnsignedLong].

"Non-primitive code follows here"
^self externalCallFailed "aka: (thisContext sender method
literalAt: 1) reportLastError"

The "globals" in the above (AlienCall, AlienResult, AlienArg1) are all
method literals to cache the information (have you ever considered
adding a <static: foo> directive to the compiler to mark temp foo as a
temp stored in the literal frame?).

There are two advantages and a major disadvantage to this approach. The
good news is that this would allow one to continue to use existing FFI
specs with Alien (i.e., ease migration). Plus, some optimizations (like
caching lookups and args) can be done by the compiler now and don't
require people to write code themselves.

The obvious disadvantage is that you need to change the code to adjust
for changes in the ABI. Which sucks. I really wish Alien could take a
descriptor and deal with the ABI details itself. I wonder if it's
feasible to have subclasses of Alien function calls that implement the
different ABIs and pass enough information along that the ABI dependent
parts could be done there?

Cheers,
- Andreas

Eliot Miranda-2

Re: Alien and FFI integration

Hi Andreas,

what I really think about this is that call-outs are the weakest part of Alien. The callout facilities are essentially a hack that works on IA32, could work on ARM or PowerPC at a pinch, and will fail horribly in general on x86-64 (um EMT64 & AMD64). e.g. the AMD64 calling convention splits struct contents across available registers as needed so e.g. a struct with one float, four longs and a float will get passed in fp arg reg 0 & fp arg reg 1 (for first and last fields) and in int arg regs 0 through 3 (for the second through the fifth fields).

Yes, one can ask the programmer to work this out and use a low level interface but one will get rightly a rude answer to the question. But the only real answer is to keep call specs and have the image compile them into something that matches the ABI.

For a while when David Simmons and I have discussed this we have agreed that the right way to do it is have an ABI compiler framework in the image whose job it is to compile code to make the callout on first use. When we've discussed it we've suggested that the ABI compiler produces some kind of RTL language that the JOT consumes and turns into proper machine code to actually make the callout. But as you noted on Monday, the ABI compiler could actually produce machine code. Alien already does this for producing the callback thunks for callback support. The only issues here are how to plumb in the machine code to the machine code the JIT generates for methods and (for me) how to avoid the ABI compiler mushrooming into a full blown native compiler with too much duplication.

On Sun, Mar 1, 2009 at 11:56 PM, Andreas Raab <[hidden email]> wrote:

Hi Eliot -

[cc: vm-dev to see if anyone has some good ideas] I was thinking about how one would move between FFI and Alien more freely and one thing that seemed interesting is to have the compiler simply generate Alien call code for FFI specs. E.g., when the compiler encounters, say:

system: aString
<apicall: ulong 'system' (char*) module: 'libc'>
^self externalCallFailed

it would generate something like here:

system: aString
AlienCall ifNil:[
"Register this method to be flushed on shutdown"
Alien registerMethod: thisContext method.
"Lookup the call if it isn't cached"
AlienCall := Alien lookup: 'system' inLibrary: 'libc'.
"Result can use a cached Alien since it never changes"
AlienResult := Alien new: 4.
].
"Arg1 needs to be reloaded every time we call"
Arg1 := Alien fromString: aString.
AlienCall primFFICallResult: AlienResult with: Arg1.
Arg1 free. "or however one frees those"
"XXXX: Would need some success indication here"
success ifFalse:[^AlienResult asUnsignedLong].

"Non-primitive code follows here"
^self externalCallFailed "aka: (thisContext sender method literalAt: 1) reportLastError"

The "globals" in the above (AlienCall, AlienResult, AlienArg1) are all method literals to cache the information (have you ever considered adding a <static: foo> directive to the compiler to mark temp foo as a temp stored in the literal frame?).

There are two advantages and a major disadvantage to this approach. The good news is that this would allow one to continue to use existing FFI specs with Alien (i.e., ease migration). Plus, some optimizations (like caching lookups and args) can be done by the compiler now and don't require people to write code themselves.

The obvious disadvantage is that you need to change the code to adjust for changes in the ABI. Which sucks. I really wish Alien could take a descriptor and deal with the ABI details itself. I wonder if it's feasible to have subclasses of Alien function calls that implement the different ABIs and pass enough information along that the ABI dependent parts could be done there?

Cheers,
- Andreas