Smalltalk › Squeak › Squeak VM

Alien/FFI issue with large struct passed as value and how to load vmmaker?

‹ Previous Topic Next Topic ›
 
classic Classic list List threaded Threaded
♦
♦
 6 messages Options
Holger Freyther
Reply | Threaded
Open this post in threaded view
♦
♦
|

Alien/FFI issue with large struct passed as value and how to load vmmaker?

Holger Freyther
 
Hi,

Sorry if this sounds ignorant but I am not sure how to do this with Opensmalltalk VM. How do I get a working VMMaker loaded into a Pharo image these days? It was quite simple in the pharo-vm days but now I am not even sure where to start looking?



I want to debug a FFI problem I see with Ben's clang-c bindings. I get a callback from C with three arguments and one of them is of type CXCursor (it needs to be passed on the stack):


struct CXCursor {
        int
        enum
        void *ptr[3];
};

It's represented as FFIExternalStructure subclass: #CXCursor. In the case of the callback the handle is an instance of FFIExternalStructureReferenceHandle with an Alien embedded into it.



I am trying to understand if the bug is on handling the callback when the CXCursor is created or when trying to push it onto the stack when making a call.


The memory on the stack is turned into an Alien. And Alien>>#referenceStructAt:length: doesn't return a ByteArray but the FFIExternalStructureReferenceHandle with the Alien in it.


At the same time I think FFIPlugin>>#ffiPushStructureContentsOf: doesn't seem to deal with FFIExternalStructureReferencehandle (or handle being an Alien).


Or do I chase it from the wrong end?

cheers
        holger
Ben Coman
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Alien/FFI issue with large struct passed as value and how to load vmmaker?

Ben Coman
 


On Wed, 3 Apr 2019 at 22:54, Holger Freyther <[hidden email]> wrote:
 
Hi,

Sorry if this sounds ignorant but I am not sure how to do this with Opensmalltalk VM. How do I get a working VMMaker loaded into a Pharo image these days? It was quite simple in the pharo-vm days but now I am not even sure where to start looking?

You haven't said what you've tried.  This looks updated recently... http://wiki.squeak.org/squeak/vmmaker

btw, I see an experimental git repo for VMMaker ...
but this is not what the production VMs are built from. 
 

I want to debug a FFI problem I see with Ben's clang-c bindings. I get a callback from C with three arguments and one of them is of type CXCursor (it needs to be passed on the stack):

You are referring this page then...?  
Which part of it are you up to?


struct CXCursor {
        int
        enum
        void *ptr[3];
};

It's represented as FFIExternalStructure subclass: #CXCursor.

For more context for interested readers, CxCursor definition is shown here...
 

In the case of the callback the handle is an instance of FFIExternalStructureReferenceHandle with an Alien embedded into it.

Wow, those posts were back in 2016. How time flies. Its vague, but I don't remember ever seeing class FFIExternalStructureReferenceHandle or Alien anywhere.
I'll have to freshly re-follow the posts myself to review.

 
I am trying to understand if the bug is on handling the callback when the CXCursor is created or when trying to push it onto the stack when making a call.

The memory on the stack is turned into an Alien. And Alien>>#referenceStructAt:length: doesn't return a ByteArray but the FFIExternalStructureReferenceHandle with the Alien in it.

At the same time I think FFIPlugin>>#ffiPushStructureContentsOf: doesn't seem to deal with FFIExternalStructureReferencehandle (or handle being an Alien).

I'm not familiar with that part of it.  Hopefully someone else can chime in.
btw, did you do the whole series of post individually, or bulk load the code from... http://smalltalkhub.com/#!/~BenComan/LibclangPractice

cheers -ben
 
Eliot Miranda-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Alien/FFI issue with large struct passed as value and how to load vmmaker?

Eliot Miranda-2
In reply to this post by Holger Freyther
 
Hi Holger,

On Wed, Apr 3, 2019 at 7:36 AM Holger Freyther <[hidden email]> wrote:
 
Hi,

Sorry if this sounds ignorant but I am not sure how to do this with Opensmalltalk VM. How do I get a working VMMaker loaded into a Pharo image these days? It was quite simple in the pharo-vm days but now I am not even sure where to start looking?



I want to debug a FFI problem I see with Ben's clang-c bindings. I get a callback from C with three arguments and one of them is of type CXCursor (it needs to be passed on the stack):


struct CXCursor {
        int
        enum
        void *ptr[3];
};

It's represented as FFIExternalStructure subclass: #CXCursor. In the case of the callback the handle is an instance of FFIExternalStructureReferenceHandle with an Alien embedded into it.

What is the C signature of the callback function, and what is the source of the marshaling method in the relevant Callback subclass's signatures protocol?
Also, what's the callout's signature and what is the Smalltalk code for the call?

I am trying to understand if the bug is on handling the callback when the CXCursor is created or when trying to push it onto the stack when making a call.


The memory on the stack is turned into an Alien. And Alien>>#referenceStructAt:length: doesn't return a ByteArray but the FFIExternalStructureReferenceHandle with the Alien in it.


At the same time I think FFIPlugin>>#ffiPushStructureContentsOf: doesn't seem to deal with FFIExternalStructureReferencehandle (or handle being an Alien).


Or do I chase it from the wrong end?

Without the code I can't help, hence my questions above.

_,,,^..^,,,_
best, Eliot
Holger Freyther
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Alien/FFI issue with large struct passed as value and how to load vmmaker?

Holger Freyther
 


> On 3. Apr 2019, at 23:50, Eliot Miranda <[hidden email]> wrote:
>
> Hi Holger,
>


> It's represented as FFIExternalStructure subclass: #CXCursor. In the case of the callback the handle is an instance of FFIExternalStructureReferenceHandle with an Alien embedded into it.
>
> What is the C signature of the callback function, and what is the source of the marshaling method in the relevant Callback subclass's signatures protocol?
> Also, what's the callout's signature and what is the Smalltalk code for the call?


C declaration:

typedef enum CXChildVisitResult(* CXCursorVisitor) (CXCursor cursor, CXCursor parent, CXClientData client_data);

and passed to the c function below.

unsigned clang_visitChildren (CXCursor parent, CXCursorVisitor visitor,  CXClientData client_data );


Smalltalk Source:

        acceptCallbackFn :=
           FFICallback
                signature:  #( CXChildVisitResult (
                                                                CXCursor cursor,
                                                                CXCursor parent,
                                                                CXString client_data))


CXChildVisitResult is an enum, CXCursor/CXString are FFIExternalStructure subclasses.



Smalltalk code:


        Libclang clang_visitChildren__parentCursor: rootCursor
                                   visitorCallback: acceptCallbackFn
                                        clientData: rootClientData.

>>#clang_visitChildren__parentCursor: parent
                  visitorCallback: visitor
                       clientData: client_data
        ^ self ffiCall: #( uint clang_visitChildren(
          CXCursor parent,
                                                                CXCursorVisitor visitor,
                                  CXClientData client_data))



> Or do I chase it from the wrong end?
>
> Without the code I can't help, hence my questions above.

The full code is in Ben's LibclangPractice (http://smalltalkhub.com/#!/~BenComan/LibclangPractice). I have set a breakpoint in LibclangTest>>#visitChildrenCallbackReturning: to look at the cursor variable and call it's methods. I think my  Pharo PR (https://github.com/pharo-project/pharo/pull/3136/files#diff-8f4b31166c3a817dd8ad1f0518ae633a) is fixing the callback handling in Unified-FFI.


I think I have stumbled into three separate bugs and I just noticed that they are Pharo specific. Unified-FFI doesn't seem to exist for Squeak.


1st) an Alien ending inside a handle of a FFIExternalStructure sub-instance. I have made a PR for it and I think it is reasonable to turn the Alien into an ExternalAddress early.

2nd) Pushing a FFIExternalStructure sub-instance with a FFIExternalStructureReferenceHandle in it to the stack doesn't work. Squeak+ThreadedFFIPlugin simply don't know what a FFIExternalStructureReferenceHandle valueOOP is.


3rd) Returning a struct from the callback doesn't to work either. That's a UnifiedFFI bug as well:

In FFICallbackParameterTests>>#testPassingStructureInTheStack
       
        | param |
        callback := FFICallback
                signature: #(int (FFITestStructureSmallIntFloatStructure a))
                block: [ :a |
                        self assert: a x equals: 2.0.
                        self assert: a y equals: 3. ..

change the "int" to "FFITestStructureSmallIntFloatStructure" and the first assertion will fail. The value of "a x" becomes 3. I have not debugged this.


I think only 2nd) is relevant to this list. Support for FFIExternalStructureReferenceHandle will add one level of indirection.


Eliot Miranda-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Alien/FFI issue with large struct passed as value and how to load vmmaker?

Eliot Miranda-2
 
Hi Holger,

On Thu, Apr 4, 2019 at 10:04 AM Holger Freyther <[hidden email]> wrote:

> On 3. Apr 2019, at 23:50, Eliot Miranda <[hidden email]> wrote:
>
> Hi Holger,

> It's represented as FFIExternalStructure subclass: #CXCursor. In the case of the callback the handle is an instance of FFIExternalStructureReferenceHandle with an Alien embedded into it.
>
> What is the C signature of the callback function, and what is the source of the marshaling method in the relevant Callback subclass's signatures protocol?
> Also, what's the callout's signature and what is the Smalltalk code for the call?


C declaration:

typedef enum CXChildVisitResult(* CXCursorVisitor) (CXCursor cursor, CXCursor parent, CXClientData client_data);

and passed to the c function below.

unsigned clang_visitChildren    (CXCursor parent, CXCursorVisitor visitor,  CXClientData client_data );


Smalltalk Source:

        acceptCallbackFn :=
           FFICallback
                signature:  #( CXChildVisitResult (
                                                                CXCursor cursor,
                                                                CXCursor parent,
                                                                CXString client_data))

somewhere in the Callback hierarchy there should me a marshaling method for the platform you're on that matches that signature.  Fo example, in Squeak if I'm calling, say, sort, then there's a callback such as

Callback
signature:  #(int (*)(const void *, const void *))
block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign].

to marshal the callback's incoming arguments each platform needs a suitable marshaling method that the Callback machinery matches to the signature.  Here they are from Squeak:

Callback methods for signatures
voidstarvoidstarRetint: callbackContext regs: regsAlien
<signature: #(int (*)(const void *, const void *))>
self subclassResponsibility

CallbackForARM32 methods for signatures
voidstarvoidstarRetint: callbackContext regs: regsAlien
<signature: #(int (*)(const void *, const void *))>
^callbackContext wordResult:
(block
value: (Alien forPointer: (regsAlien unsignedLongAt: 1))
value: (Alien forPointer: (regsAlien unsignedLongAt: 5)))

CallbackForWin64X64 methods for signatures
voidstarvoidstarRetint: callbackContext regs: regsAlien
<signature: #(int (*)(const void *, const void *))>
^callbackContext wordResult:
(block
value: (Alien forPointer: (regsAlien unsignedLongLongAt: 1))
value: (Alien forPointer: (regsAlien unsignedLongLongAt: 9)))

CallbackForX64 methods for signatures
voidstarvoidstarRetint: callbackContext regs: regsAlien
<signature: #(int (*)(const void *, const void *))>
^callbackContext wordResult:
(block
value: (Alien forPointer: (regsAlien unsignedLongLongAt: 1))
value: (Alien forPointer: (regsAlien unsignedLongLongAt: 9)))

What are is method for your platform?


CXChildVisitResult is an enum, CXCursor/CXString are FFIExternalStructure subclasses.



Smalltalk code:


        Libclang clang_visitChildren__parentCursor: rootCursor
                                   visitorCallback: acceptCallbackFn
                                        clientData: rootClientData.

>>#clang_visitChildren__parentCursor: parent
                  visitorCallback: visitor
                       clientData: client_data
        ^ self ffiCall: #( uint clang_visitChildren(
                                                                CXCursor parent,
                                                                CXCursorVisitor visitor,
                                                                CXClientData client_data))



> Or do I chase it from the wrong end?
>
> Without the code I can't help, hence my questions above.

The full code is in Ben's LibclangPractice (http://smalltalkhub.com/#!/~BenComan/LibclangPractice). I have set a breakpoint in LibclangTest>>#visitChildrenCallbackReturning: to look at the cursor variable and call it's methods. I think my  Pharo PR (https://github.com/pharo-project/pharo/pull/3136/files#diff-8f4b31166c3a817dd8ad1f0518ae633a) is fixing the callback handling in Unified-FFI.


I think I have stumbled into three separate bugs and I just noticed that they are Pharo specific. Unified-FFI doesn't seem to exist for Squeak.


1st) an Alien ending inside a handle of a FFIExternalStructure sub-instance. I have made a PR for it and I think it is reasonable to turn the Alien into an ExternalAddress early.

2nd) Pushing a FFIExternalStructure sub-instance with a FFIExternalStructureReferenceHandle in it to the stack doesn't work. Squeak+ThreadedFFIPlugin simply don't know what a FFIExternalStructureReferenceHandle valueOOP is.


3rd) Returning a struct from the callback doesn't to work either. That's a UnifiedFFI bug as well:

In FFICallbackParameterTests>>#testPassingStructureInTheStack

        | param |
        callback := FFICallback
                signature: #(int (FFITestStructureSmallIntFloatStructure a))
                block: [ :a |
                        self assert: a x equals: 2.0.
                        self assert: a y equals: 3. ..

change the "int" to "FFITestStructureSmallIntFloatStructure" and the first assertion will fail. The value of "a x" becomes 3. I have not debugged this.


I think only 2nd) is relevant to this list. Support for FFIExternalStructureReferenceHandle will add one level of indirection.




--
_,,,^..^,,,_
best, Eliot
Eliot Miranda-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Alien/FFI issue with large struct passed as value and how to load vmmaker?

Eliot Miranda-2
 


On Thu, Apr 4, 2019 at 11:43 AM Eliot Miranda <[hidden email]> wrote:
Hi Holger,

On Thu, Apr 4, 2019 at 10:04 AM Holger Freyther <[hidden email]> wrote:

> On 3. Apr 2019, at 23:50, Eliot Miranda <[hidden email]> wrote:
>
> Hi Holger,

> It's represented as FFIExternalStructure subclass: #CXCursor. In the case of the callback the handle is an instance of FFIExternalStructureReferenceHandle with an Alien embedded into it.
>
> What is the C signature of the callback function, and what is the source of the marshaling method in the relevant Callback subclass's signatures protocol?
> Also, what's the callout's signature and what is the Smalltalk code for the call?


C declaration:

typedef enum CXChildVisitResult(* CXCursorVisitor) (CXCursor cursor, CXCursor parent, CXClientData client_data);

and passed to the c function below.

unsigned clang_visitChildren    (CXCursor parent, CXCursorVisitor visitor,  CXClientData client_data );


Smalltalk Source:

        acceptCallbackFn :=
           FFICallback
                signature:  #( CXChildVisitResult (
                                                                CXCursor cursor,
                                                                CXCursor parent,
                                                                CXString client_data))

somewhere in the Callback hierarchy there should me a marshaling method for the platform you're on that matches that signature.  Fo example, in Squeak if I'm calling, say, sort, then there's a callback such as

Callback
signature:  #(int (*)(const void *, const void *))
block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign].

to marshal the callback's incoming arguments each platform needs a suitable marshaling method that the Callback machinery matches to the signature.  Here they are from Squeak:

Callback methods for signatures
voidstarvoidstarRetint: callbackContext regs: regsAlien
<signature: #(int (*)(const void *, const void *))>
self subclassResponsibility

CallbackForARM32 methods for signatures
voidstarvoidstarRetint: callbackContext regs: regsAlien
<signature: #(int (*)(const void *, const void *))>
^callbackContext wordResult:
(block
value: (Alien forPointer: (regsAlien unsignedLongAt: 1))
value: (Alien forPointer: (regsAlien unsignedLongAt: 5)))

CallbackForWin64X64 methods for signatures
voidstarvoidstarRetint: callbackContext regs: regsAlien
<signature: #(int (*)(const void *, const void *))>
^callbackContext wordResult:
(block
value: (Alien forPointer: (regsAlien unsignedLongLongAt: 1))
value: (Alien forPointer: (regsAlien unsignedLongLongAt: 9)))

CallbackForX64 methods for signatures
voidstarvoidstarRetint: callbackContext regs: regsAlien
<signature: #(int (*)(const void *, const void *))>
^callbackContext wordResult:
(block
value: (Alien forPointer: (regsAlien unsignedLongLongAt: 1))
value: (Alien forPointer: (regsAlien unsignedLongLongAt: 9)))

Missed one (finger trouble)

CallbackForIA32 methods for signatures
voidstarvoidstarRetint: callbackContext sp: spAlien
<signature: #(int (*)(const void *, const void *))>
^callbackContext wordResult:
(block
value: (Alien forPointer: (spAlien unsignedLongAt: 1))
value: (Alien forPointer: (spAlien unsignedLongAt: 5)))


What are is method for your platform?


CXChildVisitResult is an enum, CXCursor/CXString are FFIExternalStructure subclasses.



Smalltalk code:


        Libclang clang_visitChildren__parentCursor: rootCursor
                                   visitorCallback: acceptCallbackFn
                                        clientData: rootClientData.

>>#clang_visitChildren__parentCursor: parent
                  visitorCallback: visitor
                       clientData: client_data
        ^ self ffiCall: #( uint clang_visitChildren(
                                                                CXCursor parent,
                                                                CXCursorVisitor visitor,
                                                                CXClientData client_data))



> Or do I chase it from the wrong end?
>
> Without the code I can't help, hence my questions above.

The full code is in Ben's LibclangPractice (http://smalltalkhub.com/#!/~BenComan/LibclangPractice). I have set a breakpoint in LibclangTest>>#visitChildrenCallbackReturning: to look at the cursor variable and call it's methods. I think my  Pharo PR (https://github.com/pharo-project/pharo/pull/3136/files#diff-8f4b31166c3a817dd8ad1f0518ae633a) is fixing the callback handling in Unified-FFI.


I think I have stumbled into three separate bugs and I just noticed that they are Pharo specific. Unified-FFI doesn't seem to exist for Squeak.


1st) an Alien ending inside a handle of a FFIExternalStructure sub-instance. I have made a PR for it and I think it is reasonable to turn the Alien into an ExternalAddress early.

2nd) Pushing a FFIExternalStructure sub-instance with a FFIExternalStructureReferenceHandle in it to the stack doesn't work. Squeak+ThreadedFFIPlugin simply don't know what a FFIExternalStructureReferenceHandle valueOOP is.


3rd) Returning a struct from the callback doesn't to work either. That's a UnifiedFFI bug as well:

In FFICallbackParameterTests>>#testPassingStructureInTheStack

        | param |
        callback := FFICallback
                signature: #(int (FFITestStructureSmallIntFloatStructure a))
                block: [ :a |
                        self assert: a x equals: 2.0.
                        self assert: a y equals: 3. ..

change the "int" to "FFITestStructureSmallIntFloatStructure" and the first assertion will fail. The value of "a x" becomes 3. I have not debugged this.


I think only 2nd) is relevant to this list. Support for FFIExternalStructureReferenceHandle will add one level of indirection.




--
_,,,^..^,,,_
best, Eliot


--
_,,,^..^,,,_
best, Eliot
Free forum by Nabble Edit this page