Cog/FFI Return Values

For kicks, I tried running the GemStone tools in a Cog VM. It *almost* works. Apparently this sort of call:

> apiGciPerform: anOopType a: aString a: anArray a: anInteger
>
>       <apicall: OopType64 'GciPerform' (OopType64 char* ulong long) >
>       ^self externalCallFailed

... works differently now. In the old (non-Cog) VM, it return an instance of OopType64. In Cog, it returns a SmallInteger.

The function itself returns a uint64_t, so apparently the OopType64 class (a subclass of ExternalStructure) is being used to trick the FFI into automatically wrapping the return value in the way we want.

1. Is this trick no longer supported/legit (assuming it ever was)?

2. Is there a better way to accomplish the same thing, or do we just have to do the wrapping ourselves now?

I'm pretty new to the FFI, so forgive me if the premise is wrong. All I know is that it doesn't work the way it used to. :)

--
Ken Treis
Miriam Technologies, Inc.

ken

it would be good to enter a bug entry on the cog vm bug tracker.

Stef

On Mar 15, 2011, at 11:19 PM, Ken Treis wrote:

> For kicks, I tried running the GemStone tools in a Cog VM. It *almost* works. Apparently this sort of call:
>
>> apiGciPerform: anOopType a: aString a: anArray a: anInteger
>>
>>      <apicall: OopType64 'GciPerform' (OopType64 char* ulong long) >
>>      ^self externalCallFailed
>
> ... works differently now. In the old (non-Cog) VM, it return an instance of OopType64. In Cog, it returns a SmallInteger.
>
> The function itself returns a uint64_t, so apparently the OopType64 class (a subclass of ExternalStructure) is being used to trick the FFI into automatically wrapping the return value in the way we want.
>
> 1. Is this trick no longer supported/legit (assuming it ever was)?
> 2. Is there a better way to accomplish the same thing, or do we just have to do the wrapping ourselves now?
>
> I'm pretty new to the FFI, so forgive me if the premise is wrong. All I know is that it doesn't work the way it used to. :)
>
> --
> Ken Treis
> Miriam Technologies, Inc.
>
>

On Mar 15, 2011, at 3:36 PM, Eliot Miranda wrote:

> I expect that if the standard FFI plugin answers an instance of OopType64 then you've found a bug in the ReentrantFFIPlugin.  If you're brave you'll have a look at both plugins and try and verify that that's the case.  But I have to say that the FFI plugins are not simple.

Brave, yes. Smart, no. Here's what it looks like to my untrained eye:

1. When the ExternalFunction is compiled, the return argType is not flagged as a structure. So the primitives, old and new, see this as a regular non-struct callout, but with return class = OopType64. So far, so good.

2. The old plugin had this one-slot "wrapping" capability at the bottom of #ffiCreateReturn: (which is called only for non-pointer, non-structure functions):

       ffiRetClass := interpreterProxy popRemappableOop.
       ffiRetClass == interpreterProxy nilObject ifTrue:[
               "Just return oop"
               ^interpreterProxy push: retOop].
       "Otherwise create an instance of external structure and store the return oop"
       interpreterProxy pushRemappableOop: retOop.
       retOop := interpreterProxy instantiateClass: ffiRetClass indexableSize: 0.
       oop := interpreterProxy popRemappableOop.
       interpreterProxy storePointer: 0 ofObject: retOop withValue: oop.
       ^interpreterProxy push: retOop.

3. There's some very similar code in FFIPlugin>>ffiCreateReturnStruct and ReentrantIA32FFIPlugin>>ffiReturnStruct:ofType:in: -- the main difference is a memcpy().

Am I on the right track? If so, how would you like to see the code structured to get this feature restored? We could add a new wrap-if-necessary return method, or we could try to reuse #ffiReturnStruct:ofType:in: if we guarded the memcpy() with a flag check for FFIFlagStructure.

Or maybe there's a cleaner solution that's beyond my hazy view at the moment.

--
Ken Treis
Miriam Technologies, Inc.

See attached for changes vs VMMaker-oscog.40. I added a couple of simple test methods to the preamble and used them in my test cases.

The push/popRemappableOop stuff is over my head. I basically cloned the code quoted below, but I had to introduce a temporary variable. I'm guessing that it doesn't need the same treatment, but no doubt you can clean that up for me if needs be.

With this change, GemTools work in Cog for me. Thanks for the guidance and encouragement.

Ken

On Mar 16, 2011, at 4:45 PM, Eliot Miranda wrote:

On Wed, Mar 16, 2011 at 4:25 PM, Ken Treis <[hidden email]> wrote:

On Mar 15, 2011, at 3:36 PM, Eliot Miranda wrote:

> I expect that if the standard FFI plugin answers an instance of OopType64 then you've found a bug in the ReentrantFFIPlugin.  If you're brave you'll have a look at both plugins and try and verify that that's the case.  But I have to say that the FFI plugins are not simple.

Brave, yes. Smart, no. Here's what it looks like to my untrained eye:

1. When the ExternalFunction is compiled, the return argType is not flagged as a structure. So the primitives, old and new, see this as a regular non-struct callout, but with return class = OopType64. So far, so good.

2. The old plugin had this one-slot "wrapping" capability at the bottom of #ffiCreateReturn: (which is called only for non-pointer, non-structure functions):

       ffiRetClass := interpreterProxy popRemappableOop.
       ffiRetClass == interpreterProxy nilObject ifTrue:[
               "Just return oop"
               ^interpreterProxy push: retOop].
       "Otherwise create an instance of external structure and store the return oop"
       interpreterProxy pushRemappableOop: retOop.
       retOop := interpreterProxy instantiateClass: ffiRetClass indexableSize: 0.
       oop := interpreterProxy popRemappableOop.
       interpreterProxy storePointer: 0 ofObject: retOop withValue: oop.
       ^interpreterProxy push: retOop.

3. There's some very similar code in FFIPlugin>>ffiCreateReturnStruct and ReentrantIA32FFIPlugin>>ffiReturnStruct:ofType:in: -- the main difference is a memcpy().

Am I on the right track? If so, how would you like to see the code structured to get this feature restored? We could add a new wrap-if-necessary return method, or we could try to reuse #ffiReturnStruct:ofType:in: if we guarded the memcpy() with a flag check for FFIFlagStructure.

If you're happy producing a fix I'll take whatever works, a test case and the I can always muck it up if I disapprove :)

 

Or maybe there's a cleaner solution that's beyond my hazy view at the moment.

I think you're on the right track.  I probably misread when I wrote my first cut.  Apologies.

best

Eliot

 

--
Ken Treis
Miriam Technologies, Inc.

--

Ken Treis

Miriam Technologies, Inc.