Best way for FFI in Pharo

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Best way for FFI in Pharo

Udo Schneider
All,

what's the best way (at the moment & in the future) to call external
libraries from Pharo. I'm a bit confused at the moment will all the
stuff going around FFI, NativeBoost ...

Is there maybe an abstraction layer above all those native-call
mechanisms somewhere?

Best Regards,

Udo


Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Stéphane Ducasse
Hi Udo

indeed this is a bit the mess :).
As soon as athens1.0 is out (planned for the end of the month). We will ask igor to clean the system. Our idea
is to provide one API where we can plug the two infrastructures
        NativeBoost (we are writing a documentation too but I should pair write with igor so it goes slowly :(
        With nativeBoost you can simply invoke C functions. For example the cairo binding is a good example to show how it works. The chapter is on the pharobooks inria gforge svn, I can send it to you if you want.

        FFI (when esteban arrives I would like to write the other chapter) and makes sure that FFI works also like a charm
        on all the platforms.

Stef
                 



> All,
>
> what's the best way (at the moment & in the future) to call external libraries from Pharo. I'm a bit confused at the moment will all the stuff going around FFI, NativeBoost ...
>
> Is there maybe an abstraction layer above all those native-call mechanisms somewhere?
>
> Best Regards,
>
> Udo
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

fstephany
I'm also a bit lost between all the different options (plugins,
NativeBoost, FFI, Alien).

Thanks for the clarification !

Will this interface handle callbacks ?


On 08/01/12 04:37, Stéphane Ducasse wrote:

> Hi Udo
>
> indeed this is a bit the mess :).
> As soon as athens1.0 is out (planned for the end of the month). We will ask igor to clean the system. Our idea
> is to provide one API where we can plug the two infrastructures
> NativeBoost (we are writing a documentation too but I should pair write with igor so it goes slowly :(
> With nativeBoost you can simply invoke C functions. For example the cairo binding is a good example to show how it works. The chapter is on the pharobooks inria gforge svn, I can send it to you if you want.
>
> FFI (when esteban arrives I would like to write the other chapter) and makes sure that FFI works also like a charm
> on all the platforms.
>
> Stef
>
>
>
>
>> All,
>>
>> what's the best way (at the moment&  in the future) to call external libraries from Pharo. I'm a bit confused at the moment will all the stuff going around FFI, NativeBoost ...
>>
>> Is there maybe an abstraction layer above all those native-call mechanisms somewhere?
>>
>> Best Regards,
>>
>> Udo
>>
>>
>
>

--
http://tulipemoutarde.be
BE: +32 (0)65 709 131
CA: +1 778 558 3225

Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

ncalexan
fstephany wrote
I'm also a bit lost between all the different options (plugins,
NativeBoost, FFI, Alien).
I also found this confusing, so let me tell my recent experience and my conclusion.

I have written bindings to the SDL game programming library for use with Pharo.  SDL is cross-platform, and I want to support Mac (most of all), Windows (one must), and Linux (if I must).  As far as I can tell, Alien is not cross-platform and not maintained, so I have not spent time investigating it.

Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package which parses the relevant C header files and then writes either an NB or FFI callout class tree.  I need to define a few structures, make a class pool with some constants, and then define lots of callouts, many of which take references to structures (like 'foo(struct x*)').

I was able to make both work, more or less, on my Macbook Pro, using stock Pharo 1.3 images and Igor's NBCog.app.

I found NB to be a nicer programmer experience but I found that the FFI just works.  There are a lot of gratuitous differences, but:

* NB requires a VM compiled with a separate plugin (that I built from source with no trouble -- Igor has done splendid work with CMakeVMMaker).
* NB should be faster than the FFI at pretty much everything, but I can't say since I haven't measured.
* NB has useful primitives for determining if your external objects are still valid and for determining the current operating system (I intend to use these even if I don't use NB).
* NB has very few tests and examples and essentially no documentation.
* NB has a simple conceptual model, but it is still not as simple as the FFI model.
* NB uses variableByteClasses to great advantage -- this is very cool.

* NB makes it easy to crash your image, since you are evaluating native code in image.
* NB is still, to my eye, raw and untested on Mac.  I had problems with stack alignments in bizarre places, and NB's interaction with garbage collection basically ended my attempts to use it.
* NB is basically impossible to debug if you aren't very strong with x86 assembly (I'm not even strong with x86 assembly), and the stack manipulations rendered GDB pretty much useless for me.
* NB does not integrate well with the Pharo tools -- I think the MethodTrailers are screwing up references, implementors, and source views, but I haven't investigated.
* NB does not seem to support indirections very easily -- I struggled to understand how to interface to foreign C functions with specs like 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually gave up.

I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings crash horribly for me, and although I was able to fix them, it I just don't hack enough x86 assembly to figure out all the tricks Igor is doing.  The thought of making it all work on 3 platforms, and fix all the tool integration, was just too much for me.

In conclusion, I prefer the FFI.  It is old, cross platform, well tested, somewhat documented, and reliable.  I think Igor has done some amazing work and I do not want to bash his project (and certainly not his code -- the man's a magician) but the fancy features are not necessary for my project.

fstephany wrote
Will this interface handle callbacks ?
I do not need callbacks so cannot speak to this issue.

Yours,
Nick Alexander
Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Stéphane Ducasse
thanks for the feedback.
We will come back you soon :)
Because we should get FFI and NativeBoost fully working :).

Stef

On Jan 8, 2012, at 7:29 PM, ncalexan wrote:

>
> fstephany wrote
>>
>> I'm also a bit lost between all the different options (plugins,
>> NativeBoost, FFI, Alien).
>>
>
> I also found this confusing, so let me tell my recent experience and my
> conclusion.
>
> I have written bindings to the SDL game programming library for use with
> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
> not cross-platform and not maintained, so I have not spent time
> investigating it.
>
> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
> which parses the relevant C header files and then writes either an NB or FFI
> callout class tree.  I need to define a few structures, make a class pool
> with some constants, and then define lots of callouts, many of which take
> references to structures (like 'foo(struct x*)').
>
> I was able to make both work, more or less, on my Macbook Pro, using stock
> Pharo 1.3 images and Igor's NBCog.app.
>
> I found NB to be a nicer programmer experience but I found that the FFI just
> works.  There are a lot of gratuitous differences, but:
>
> * NB requires a VM compiled with a separate plugin (that I built from source
> with no trouble -- Igor has done splendid work with CMakeVMMaker).
> * NB should be faster than the FFI at pretty much everything, but I can't
> say since I haven't measured.
> * NB has useful primitives for determining if your external objects are
> still valid and for determining the current operating system (I intend to
> use these even if I don't use NB).
> * NB has very few tests and examples and essentially no documentation.
> * NB has a simple conceptual model, but it is still not as simple as the FFI
> model.
> * NB uses variableByteClasses to great advantage -- this is very cool.
>
> * NB makes it easy to crash your image, since you are evaluating native code
> in image.
> * NB is still, to my eye, raw and untested on Mac.  I had problems with
> stack alignments in bizarre places, and NB's interaction with garbage
> collection basically ended my attempts to use it.
> * NB is basically impossible to debug if you aren't very strong with x86
> assembly (I'm not even strong with x86 assembly), and the stack
> manipulations rendered GDB pretty much useless for me.
> * NB does not integrate well with the Pharo tools -- I think the
> MethodTrailers are screwing up references, implementors, and source views,
> but I haven't investigated.
> * NB does not seem to support indirections very easily -- I struggled to
> understand how to interface to foreign C functions with specs like
> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
> gave up.
>
> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
> crash horribly for me, and although I was able to fix them, it I just don't
> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
> thought of making it all work on 3 platforms, and fix all the tool
> integration, was just too much for me.
>
> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
> somewhat documented, and reliable.  I think Igor has done some amazing work
> and I do not want to bash his project (and certainly not his code -- the
> man's a magician) but the fancy features are not necessary for my project.
>
>
> fstephany wrote
>>
>> Will this interface handle callbacks ?
>>
>
> I do not need callbacks so cannot speak to this issue.
>
> Yours,
> Nick Alexander
>
> --
> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>


Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Schwab,Wilhelm K
Stef,

Absent the NB experience, +1 to the comments below.  FFI has pretty much "just worked" for me.  We need double arrays.  Callbacks would be great.  One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0  It makes a LOT more sense to have a class per library, and that class knows the name to use.  That's all the more true when one considers code such as ODBC that can run on multiple platforms with different names.  #moduleName can test the OS and answer the correct name.

Bill


________________________________________
From: [hidden email] [[hidden email]] on behalf of Stéphane Ducasse [[hidden email]]
Sent: Sunday, January 08, 2012 2:16 PM
To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo

thanks for the feedback.
We will come back you soon :)
Because we should get FFI and NativeBoost fully working :).

Stef

On Jan 8, 2012, at 7:29 PM, ncalexan wrote:

>
> fstephany wrote
>>
>> I'm also a bit lost between all the different options (plugins,
>> NativeBoost, FFI, Alien).
>>
>
> I also found this confusing, so let me tell my recent experience and my
> conclusion.
>
> I have written bindings to the SDL game programming library for use with
> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
> not cross-platform and not maintained, so I have not spent time
> investigating it.
>
> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
> which parses the relevant C header files and then writes either an NB or FFI
> callout class tree.  I need to define a few structures, make a class pool
> with some constants, and then define lots of callouts, many of which take
> references to structures (like 'foo(struct x*)').
>
> I was able to make both work, more or less, on my Macbook Pro, using stock
> Pharo 1.3 images and Igor's NBCog.app.
>
> I found NB to be a nicer programmer experience but I found that the FFI just
> works.  There are a lot of gratuitous differences, but:
>
> * NB requires a VM compiled with a separate plugin (that I built from source
> with no trouble -- Igor has done splendid work with CMakeVMMaker).
> * NB should be faster than the FFI at pretty much everything, but I can't
> say since I haven't measured.
> * NB has useful primitives for determining if your external objects are
> still valid and for determining the current operating system (I intend to
> use these even if I don't use NB).
> * NB has very few tests and examples and essentially no documentation.
> * NB has a simple conceptual model, but it is still not as simple as the FFI
> model.
> * NB uses variableByteClasses to great advantage -- this is very cool.
>
> * NB makes it easy to crash your image, since you are evaluating native code
> in image.
> * NB is still, to my eye, raw and untested on Mac.  I had problems with
> stack alignments in bizarre places, and NB's interaction with garbage
> collection basically ended my attempts to use it.
> * NB is basically impossible to debug if you aren't very strong with x86
> assembly (I'm not even strong with x86 assembly), and the stack
> manipulations rendered GDB pretty much useless for me.
> * NB does not integrate well with the Pharo tools -- I think the
> MethodTrailers are screwing up references, implementors, and source views,
> but I haven't investigated.
> * NB does not seem to support indirections very easily -- I struggled to
> understand how to interface to foreign C functions with specs like
> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
> gave up.
>
> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
> crash horribly for me, and although I was able to fix them, it I just don't
> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
> thought of making it all work on 3 platforms, and fix all the tool
> integration, was just too much for me.
>
> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
> somewhat documented, and reliable.  I think Igor has done some amazing work
> and I do not want to bash his project (and certainly not his code -- the
> man's a magician) but the fancy features are not necessary for my project.
>
>
> fstephany wrote
>>
>> Will this interface handle callbacks ?
>>
>
> I do not need callbacks so cannot speak to this issue.
>
> Yours,
> Nick Alexander
>
> --
> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>



Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Stéphane Ducasse
can you give an example that I understand
"One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0 "

Tx

On Jan 8, 2012, at 9:18 PM, Schwab,Wilhelm K wrote:

> Stef,
>
> Absent the NB experience, +1 to the comments below.  FFI has pretty much "just worked" for me.  We need double arrays.  Callbacks would be great.  One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0  It makes a LOT more sense to have a class per library, and that class knows the name to use.  That's all the more true when one considers code such as ODBC that can run on multiple platforms with different names.  #moduleName can test the OS and answer the correct name.
>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Stéphane Ducasse [[hidden email]]
> Sent: Sunday, January 08, 2012 2:16 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>
> thanks for the feedback.
> We will come back you soon :)
> Because we should get FFI and NativeBoost fully working :).
>
> Stef
>
> On Jan 8, 2012, at 7:29 PM, ncalexan wrote:
>
>>
>> fstephany wrote
>>>
>>> I'm also a bit lost between all the different options (plugins,
>>> NativeBoost, FFI, Alien).
>>>
>>
>> I also found this confusing, so let me tell my recent experience and my
>> conclusion.
>>
>> I have written bindings to the SDL game programming library for use with
>> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
>> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
>> not cross-platform and not maintained, so I have not spent time
>> investigating it.
>>
>> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
>> which parses the relevant C header files and then writes either an NB or FFI
>> callout class tree.  I need to define a few structures, make a class pool
>> with some constants, and then define lots of callouts, many of which take
>> references to structures (like 'foo(struct x*)').
>>
>> I was able to make both work, more or less, on my Macbook Pro, using stock
>> Pharo 1.3 images and Igor's NBCog.app.
>>
>> I found NB to be a nicer programmer experience but I found that the FFI just
>> works.  There are a lot of gratuitous differences, but:
>>
>> * NB requires a VM compiled with a separate plugin (that I built from source
>> with no trouble -- Igor has done splendid work with CMakeVMMaker).
>> * NB should be faster than the FFI at pretty much everything, but I can't
>> say since I haven't measured.
>> * NB has useful primitives for determining if your external objects are
>> still valid and for determining the current operating system (I intend to
>> use these even if I don't use NB).
>> * NB has very few tests and examples and essentially no documentation.
>> * NB has a simple conceptual model, but it is still not as simple as the FFI
>> model.
>> * NB uses variableByteClasses to great advantage -- this is very cool.
>>
>> * NB makes it easy to crash your image, since you are evaluating native code
>> in image.
>> * NB is still, to my eye, raw and untested on Mac.  I had problems with
>> stack alignments in bizarre places, and NB's interaction with garbage
>> collection basically ended my attempts to use it.
>> * NB is basically impossible to debug if you aren't very strong with x86
>> assembly (I'm not even strong with x86 assembly), and the stack
>> manipulations rendered GDB pretty much useless for me.
>> * NB does not integrate well with the Pharo tools -- I think the
>> MethodTrailers are screwing up references, implementors, and source views,
>> but I haven't investigated.
>> * NB does not seem to support indirections very easily -- I struggled to
>> understand how to interface to foreign C functions with specs like
>> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
>> gave up.
>>
>> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
>> crash horribly for me, and although I was able to fix them, it I just don't
>> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
>> thought of making it all work on 3 platforms, and fix all the tool
>> integration, was just too much for me.
>>
>> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
>> somewhat documented, and reliable.  I think Igor has done some amazing work
>> and I do not want to bash his project (and certainly not his code -- the
>> man's a magician) but the fancy features are not necessary for my project.
>>
>>
>> fstephany wrote
>>>
>>> Will this interface handle callbacks ?
>>>
>>
>> I do not need callbacks so cannot speak to this issue.
>>
>> Yours,
>> Nick Alexander
>>
>> --
>> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Mariano Martinez Peck


On Sun, Jan 8, 2012 at 9:25 PM, Stéphane Ducasse <[hidden email]> wrote:
can you give an example that I understand
"One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0 "


Bill:  that already exists in FFI.
Stef: what Bill says is that if you bind to a specific library you have to put its library name in each method that calls a ffi function. Example of DBX:

apiErrorType: handle number: err
    "int odbx_error_type( odbx_t*, int )"
    <cdecl: long 'odbx_error_type' (ulong long) module: 'opendbx'>
    ^ self externalCallFailed

----

apiInitialize: handle backend: backend host: host port: port
    "long odbx_init(odbx_t**, char*, char*, char*)"
    <cdecl: long 'odbx_init' (ulong* char* char* char*) module: 'opendbx'>
    ^self externalCallFailed

---

xxx

---

Notice the "module: 'opendbx"
So...if now the library is renamed or whatever, you have to change all methods. But I don't think this is a real big deal. There are much worst things.

Finaly, I copy paste an answer from Andreas from a previous thread:


The Right Way to do this is to have a subclass of ExternalLibrary and implement the class-side method #moduleName along the lines of:

MyLibrary class>>moduleName
 "Answer the module name to use for this library"
 Smalltalk platformName = 'Win32' ifTrue:[^'MyLibrary32.dll'].
 Smalltalk platformName = 'unix' ifTrue:[
    "check various locations and versions"
    #('/usr/lib/libMyLibrary.so'
      '/usr/lib/libMyLibrary.so.1'
      '/usr/lib/libMyLibrary.so.2'
      '/usr/share/libMyLibrary.so.1'
      '/usr/share/libMyLibrary.so.2'
) do:[:location|
         (FileDirectory fileExists: location) ifTrue:[^location].
    ].
    ^self error:'MyLibrary is not installed'
  ].

 
Tx

On Jan 8, 2012, at 9:18 PM, Schwab,Wilhelm K wrote:

> Stef,
>
> Absent the NB experience, +1 to the comments below.  FFI has pretty much "just worked" for me.  We need double arrays.  Callbacks would be great.  One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0  It makes a LOT more sense to have a class per library, and that class knows the name to use.  That's all the more true when one considers code such as ODBC that can run on multiple platforms with different names.  #moduleName can test the OS and answer the correct name.
>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Stéphane Ducasse [[hidden email]]
> Sent: Sunday, January 08, 2012 2:16 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>
> thanks for the feedback.
> We will come back you soon :)
> Because we should get FFI and NativeBoost fully working :).
>
> Stef
>
> On Jan 8, 2012, at 7:29 PM, ncalexan wrote:
>
>>
>> fstephany wrote
>>>
>>> I'm also a bit lost between all the different options (plugins,
>>> NativeBoost, FFI, Alien).
>>>
>>
>> I also found this confusing, so let me tell my recent experience and my
>> conclusion.
>>
>> I have written bindings to the SDL game programming library for use with
>> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
>> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
>> not cross-platform and not maintained, so I have not spent time
>> investigating it.
>>
>> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
>> which parses the relevant C header files and then writes either an NB or FFI
>> callout class tree.  I need to define a few structures, make a class pool
>> with some constants, and then define lots of callouts, many of which take
>> references to structures (like 'foo(struct x*)').
>>
>> I was able to make both work, more or less, on my Macbook Pro, using stock
>> Pharo 1.3 images and Igor's NBCog.app.
>>
>> I found NB to be a nicer programmer experience but I found that the FFI just
>> works.  There are a lot of gratuitous differences, but:
>>
>> * NB requires a VM compiled with a separate plugin (that I built from source
>> with no trouble -- Igor has done splendid work with CMakeVMMaker).
>> * NB should be faster than the FFI at pretty much everything, but I can't
>> say since I haven't measured.
>> * NB has useful primitives for determining if your external objects are
>> still valid and for determining the current operating system (I intend to
>> use these even if I don't use NB).
>> * NB has very few tests and examples and essentially no documentation.
>> * NB has a simple conceptual model, but it is still not as simple as the FFI
>> model.
>> * NB uses variableByteClasses to great advantage -- this is very cool.
>>
>> * NB makes it easy to crash your image, since you are evaluating native code
>> in image.
>> * NB is still, to my eye, raw and untested on Mac.  I had problems with
>> stack alignments in bizarre places, and NB's interaction with garbage
>> collection basically ended my attempts to use it.
>> * NB is basically impossible to debug if you aren't very strong with x86
>> assembly (I'm not even strong with x86 assembly), and the stack
>> manipulations rendered GDB pretty much useless for me.
>> * NB does not integrate well with the Pharo tools -- I think the
>> MethodTrailers are screwing up references, implementors, and source views,
>> but I haven't investigated.
>> * NB does not seem to support indirections very easily -- I struggled to
>> understand how to interface to foreign C functions with specs like
>> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
>> gave up.
>>
>> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
>> crash horribly for me, and although I was able to fix them, it I just don't
>> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
>> thought of making it all work on 3 platforms, and fix all the tool
>> integration, was just too much for me.
>>
>> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
>> somewhat documented, and reliable.  I think Igor has done some amazing work
>> and I do not want to bash his project (and certainly not his code -- the
>> man's a magician) but the fancy features are not necessary for my project.
>>
>>
>> fstephany wrote
>>>
>>> Will this interface handle callbacks ?
>>>
>>
>> I do not need callbacks so cannot speak to this issue.
>>
>> Yours,
>> Nick Alexander
>>
>> --
>> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>
>
>
>





--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Schwab,Wilhelm K
Agreed, except for thinking that changing all of the module names is not a big deal :)   An external library subclass is the way to go.  I know this already exists in FFI, but the evolving chapter on Spock does not mention it - it should not only mention it, but recommend it, IMHO.

Stef, does it make sense yet?





From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 3:52 PM
To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 9:25 PM, Stéphane Ducasse <[hidden email]> wrote:
can you give an example that I understand
"One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0 "


Bill:  that already exists in FFI.
Stef: what Bill says is that if you bind to a specific library you have to put its library name in each method that calls a ffi function. Example of DBX:

apiErrorType: handle number: err
    "int odbx_error_type( odbx_t*, int )"
    <cdecl: long 'odbx_error_type' (ulong long) module: 'opendbx'>
    ^ self externalCallFailed

----

apiInitialize: handle backend: backend host: host port: port
    "long odbx_init(odbx_t**, char*, char*, char*)"
    <cdecl: long 'odbx_init' (ulong* char* char* char*) module: 'opendbx'>
    ^self externalCallFailed

---

xxx

---

Notice the "module: 'opendbx"
So...if now the library is renamed or whatever, you have to change all methods. But I don't think this is a real big deal. There are much worst things.

Finaly, I copy paste an answer from Andreas from a previous thread:


The Right Way to do this is to have a subclass of ExternalLibrary and implement the class-side method #moduleName along the lines of:

MyLibrary class>>moduleName
 "Answer the module name to use for this library"
 Smalltalk platformName = 'Win32' ifTrue:[^'MyLibrary32.dll'].
 Smalltalk platformName = 'unix' ifTrue:[
    "check various locations and versions"
    #('/usr/lib/libMyLibrary.so'
      '/usr/lib/libMyLibrary.so.1'
      '/usr/lib/libMyLibrary.so.2'
      '/usr/share/libMyLibrary.so.1'
      '/usr/share/libMyLibrary.so.2'
) do:[:location|
         (FileDirectory fileExists: location) ifTrue:[^location].
    ].
    ^self error:'MyLibrary is not installed'
  ].

 
Tx

On Jan 8, 2012, at 9:18 PM, Schwab,Wilhelm K wrote:

> Stef,
>
> Absent the NB experience, +1 to the comments below.  FFI has pretty much "just worked" for me.  We need double arrays.  Callbacks would be great.  One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0  It makes a LOT more sense to have a class per library, and that class knows the name to use.  That's all the more true when one considers code such as ODBC that can run on multiple platforms with different names.  #moduleName can test the OS and answer the correct name.
>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Stéphane Ducasse [[hidden email]]
> Sent: Sunday, January 08, 2012 2:16 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>
> thanks for the feedback.
> We will come back you soon :)
> Because we should get FFI and NativeBoost fully working :).
>
> Stef
>
> On Jan 8, 2012, at 7:29 PM, ncalexan wrote:
>
>>
>> fstephany wrote
>>>
>>> I'm also a bit lost between all the different options (plugins,
>>> NativeBoost, FFI, Alien).
>>>
>>
>> I also found this confusing, so let me tell my recent experience and my
>> conclusion.
>>
>> I have written bindings to the SDL game programming library for use with
>> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
>> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
>> not cross-platform and not maintained, so I have not spent time
>> investigating it.
>>
>> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
>> which parses the relevant C header files and then writes either an NB or FFI
>> callout class tree.  I need to define a few structures, make a class pool
>> with some constants, and then define lots of callouts, many of which take
>> references to structures (like 'foo(struct x*)').
>>
>> I was able to make both work, more or less, on my Macbook Pro, using stock
>> Pharo 1.3 images and Igor's NBCog.app.
>>
>> I found NB to be a nicer programmer experience but I found that the FFI just
>> works.  There are a lot of gratuitous differences, but:
>>
>> * NB requires a VM compiled with a separate plugin (that I built from source
>> with no trouble -- Igor has done splendid work with CMakeVMMaker).
>> * NB should be faster than the FFI at pretty much everything, but I can't
>> say since I haven't measured.
>> * NB has useful primitives for determining if your external objects are
>> still valid and for determining the current operating system (I intend to
>> use these even if I don't use NB).
>> * NB has very few tests and examples and essentially no documentation.
>> * NB has a simple conceptual model, but it is still not as simple as the FFI
>> model.
>> * NB uses variableByteClasses to great advantage -- this is very cool.
>>
>> * NB makes it easy to crash your image, since you are evaluating native code
>> in image.
>> * NB is still, to my eye, raw and untested on Mac.  I had problems with
>> stack alignments in bizarre places, and NB's interaction with garbage
>> collection basically ended my attempts to use it.
>> * NB is basically impossible to debug if you aren't very strong with x86
>> assembly (I'm not even strong with x86 assembly), and the stack
>> manipulations rendered GDB pretty much useless for me.
>> * NB does not integrate well with the Pharo tools -- I think the
>> MethodTrailers are screwing up references, implementors, and source views,
>> but I haven't investigated.
>> * NB does not seem to support indirections very easily -- I struggled to
>> understand how to interface to foreign C functions with specs like
>> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
>> gave up.
>>
>> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
>> crash horribly for me, and although I was able to fix them, it I just don't
>> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
>> thought of making it all work on 3 platforms, and fix all the tool
>> integration, was just too much for me.
>>
>> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
>> somewhat documented, and reliable.  I think Igor has done some amazing work
>> and I do not want to bash his project (and certainly not his code -- the
>> man's a magician) but the fancy features are not necessary for my project.
>>
>>
>> fstephany wrote
>>>
>>> Will this interface handle callbacks ?
>>>
>>
>> I do not need callbacks so cannot speak to this issue.
>>
>> Yours,
>> Nick Alexander
>>
>> --
>> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>
>
>
>





--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Stéphane Ducasse
In reply to this post by Mariano Martinez Peck
tx!
Now I understand what you mean.

On Jan 8, 2012, at 9:52 PM, Mariano Martinez Peck wrote:

>
>
> On Sun, Jan 8, 2012 at 9:25 PM, Stéphane Ducasse <[hidden email]> wrote:
> can you give an example that I understand
> "One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0 "
>
>
> Bill:  that already exists in FFI.
> Stef: what Bill says is that if you bind to a specific library you have to put its library name in each method that calls a ffi function. Example of DBX:
>
> apiErrorType: handle number: err
>     "int odbx_error_type( odbx_t*, int )"
>     <cdecl: long 'odbx_error_type' (ulong long) module: 'opendbx'>
>     ^ self externalCallFailed
>
> ----
>
> apiInitialize: handle backend: backend host: host port: port
>     "long odbx_init(odbx_t**, char*, char*, char*)"
>     <cdecl: long 'odbx_init' (ulong* char* char* char*) module: 'opendbx'>
>     ^self externalCallFailed
>
> ---
>
> xxx
>
> ---
>
> Notice the "module: 'opendbx"
> So...if now the library is renamed or whatever, you have to change all methods. But I don't think this is a real big deal. There are much worst things.
>
> Finaly, I copy paste an answer from Andreas from a previous thread:
>
>
> The Right Way to do this is to have a subclass of ExternalLibrary and implement the class-side method #moduleName along the lines of:
>
> MyLibrary class>>moduleName
>  "Answer the module name to use for this library"
>  Smalltalk platformName = 'Win32' ifTrue:[^'MyLibrary32.dll'].
>  Smalltalk platformName = 'unix' ifTrue:[
>     "check various locations and versions"
>     #('/usr/lib/libMyLibrary.so'
>       '/usr/lib/libMyLibrary.so.1'
>       '/usr/lib/libMyLibrary.so.2'
>       '/usr/share/libMyLibrary.so.1'
>       '/usr/share/libMyLibrary.so.2'
> ) do:[:location|
>          (FileDirectory fileExists: location) ifTrue:[^location].
>     ].
>     ^self error:'MyLibrary is not installed'
>   ].
>
>  
> Tx
>
> On Jan 8, 2012, at 9:18 PM, Schwab,Wilhelm K wrote:
>
> > Stef,
> >
> > Absent the NB experience, +1 to the comments below.  FFI has pretty much "just worked" for me.  We need double arrays.  Callbacks would be great.  One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0  It makes a LOT more sense to have a class per library, and that class knows the name to use.  That's all the more true when one considers code such as ODBC that can run on multiple platforms with different names.  #moduleName can test the OS and answer the correct name.
> >
> > Bill
> >
> >
> > ________________________________________
> > From: [hidden email] [[hidden email]] on behalf of Stéphane Ducasse [[hidden email]]
> > Sent: Sunday, January 08, 2012 2:16 PM
> > To: [hidden email]
> > Subject: Re: [Pharo-project] Best way for FFI in Pharo
> >
> > thanks for the feedback.
> > We will come back you soon :)
> > Because we should get FFI and NativeBoost fully working :).
> >
> > Stef
> >
> > On Jan 8, 2012, at 7:29 PM, ncalexan wrote:
> >
> >>
> >> fstephany wrote
> >>>
> >>> I'm also a bit lost between all the different options (plugins,
> >>> NativeBoost, FFI, Alien).
> >>>
> >>
> >> I also found this confusing, so let me tell my recent experience and my
> >> conclusion.
> >>
> >> I have written bindings to the SDL game programming library for use with
> >> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
> >> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
> >> not cross-platform and not maintained, so I have not spent time
> >> investigating it.
> >>
> >> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
> >> which parses the relevant C header files and then writes either an NB or FFI
> >> callout class tree.  I need to define a few structures, make a class pool
> >> with some constants, and then define lots of callouts, many of which take
> >> references to structures (like 'foo(struct x*)').
> >>
> >> I was able to make both work, more or less, on my Macbook Pro, using stock
> >> Pharo 1.3 images and Igor's NBCog.app.
> >>
> >> I found NB to be a nicer programmer experience but I found that the FFI just
> >> works.  There are a lot of gratuitous differences, but:
> >>
> >> * NB requires a VM compiled with a separate plugin (that I built from source
> >> with no trouble -- Igor has done splendid work with CMakeVMMaker).
> >> * NB should be faster than the FFI at pretty much everything, but I can't
> >> say since I haven't measured.
> >> * NB has useful primitives for determining if your external objects are
> >> still valid and for determining the current operating system (I intend to
> >> use these even if I don't use NB).
> >> * NB has very few tests and examples and essentially no documentation.
> >> * NB has a simple conceptual model, but it is still not as simple as the FFI
> >> model.
> >> * NB uses variableByteClasses to great advantage -- this is very cool.
> >>
> >> * NB makes it easy to crash your image, since you are evaluating native code
> >> in image.
> >> * NB is still, to my eye, raw and untested on Mac.  I had problems with
> >> stack alignments in bizarre places, and NB's interaction with garbage
> >> collection basically ended my attempts to use it.
> >> * NB is basically impossible to debug if you aren't very strong with x86
> >> assembly (I'm not even strong with x86 assembly), and the stack
> >> manipulations rendered GDB pretty much useless for me.
> >> * NB does not integrate well with the Pharo tools -- I think the
> >> MethodTrailers are screwing up references, implementors, and source views,
> >> but I haven't investigated.
> >> * NB does not seem to support indirections very easily -- I struggled to
> >> understand how to interface to foreign C functions with specs like
> >> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
> >> gave up.
> >>
> >> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
> >> crash horribly for me, and although I was able to fix them, it I just don't
> >> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
> >> thought of making it all work on 3 platforms, and fix all the tool
> >> integration, was just too much for me.
> >>
> >> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
> >> somewhat documented, and reliable.  I think Igor has done some amazing work
> >> and I do not want to bash his project (and certainly not his code -- the
> >> man's a magician) but the fancy features are not necessary for my project.
> >>
> >>
> >> fstephany wrote
> >>>
> >>> Will this interface handle callbacks ?
> >>>
> >>
> >> I do not need callbacks so cannot speak to this issue.
> >>
> >> Yours,
> >> Nick Alexander
> >>
> >> --
> >> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
> >> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
> >>
> >
> >
> >
>
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>


Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Mariano Martinez Peck
In reply to this post by Schwab,Wilhelm K


On Sun, Jan 8, 2012 at 10:13 PM, Schwab,Wilhelm K <[hidden email]> wrote:
Agreed, except for thinking that changing all of the module names is not a big deal :) 

Why not? how many smalltalk lines of code do you think it could take to automate that? ;)

 
  An external library subclass is the way to go.  I know this already exists in FFI, but the evolving chapter on Spock does not mention it - it should not only mention it, but recommend it, IMHO.

Stef, does it make sense yet?





From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 3:52 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 9:25 PM, Stéphane Ducasse <[hidden email]> wrote:
can you give an example that I understand
"One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0 "


Bill:  that already exists in FFI.
Stef: what Bill says is that if you bind to a specific library you have to put its library name in each method that calls a ffi function. Example of DBX:

apiErrorType: handle number: err
    "int odbx_error_type( odbx_t*, int )"
    <cdecl: long 'odbx_error_type' (ulong long) module: 'opendbx'>
    ^ self externalCallFailed

----

apiInitialize: handle backend: backend host: host port: port
    "long odbx_init(odbx_t**, char*, char*, char*)"
    <cdecl: long 'odbx_init' (ulong* char* char* char*) module: 'opendbx'>
    ^self externalCallFailed

---

xxx

---

Notice the "module: 'opendbx"
So...if now the library is renamed or whatever, you have to change all methods. But I don't think this is a real big deal. There are much worst things.

Finaly, I copy paste an answer from Andreas from a previous thread:


The Right Way to do this is to have a subclass of ExternalLibrary and implement the class-side method #moduleName along the lines of:

MyLibrary class>>moduleName
 "Answer the module name to use for this library"
 Smalltalk platformName = 'Win32' ifTrue:[^'MyLibrary32.dll'].
 Smalltalk platformName = 'unix' ifTrue:[
    "check various locations and versions"
    #('/usr/lib/libMyLibrary.so'
      '/usr/lib/libMyLibrary.so.1'
      '/usr/lib/libMyLibrary.so.2'
      '/usr/share/libMyLibrary.so.1'
      '/usr/share/libMyLibrary.so.2'
) do:[:location|
         (FileDirectory fileExists: location) ifTrue:[^location].
    ].
    ^self error:'MyLibrary is not installed'
  ].

 
Tx

On Jan 8, 2012, at 9:18 PM, Schwab,Wilhelm K wrote:

> Stef,
>
> Absent the NB experience, +1 to the comments below.  FFI has pretty much "just worked" for me.  We need double arrays.  Callbacks would be great.  One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0  It makes a LOT more sense to have a class per library, and that class knows the name to use.  That's all the more true when one considers code such as ODBC that can run on multiple platforms with different names.  #moduleName can test the OS and answer the correct name.
>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Stéphane Ducasse [[hidden email]]
> Sent: Sunday, January 08, 2012 2:16 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>
> thanks for the feedback.
> We will come back you soon :)
> Because we should get FFI and NativeBoost fully working :).
>
> Stef
>
> On Jan 8, 2012, at 7:29 PM, ncalexan wrote:
>
>>
>> fstephany wrote
>>>
>>> I'm also a bit lost between all the different options (plugins,
>>> NativeBoost, FFI, Alien).
>>>
>>
>> I also found this confusing, so let me tell my recent experience and my
>> conclusion.
>>
>> I have written bindings to the SDL game programming library for use with
>> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
>> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
>> not cross-platform and not maintained, so I have not spent time
>> investigating it.
>>
>> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
>> which parses the relevant C header files and then writes either an NB or FFI
>> callout class tree.  I need to define a few structures, make a class pool
>> with some constants, and then define lots of callouts, many of which take
>> references to structures (like 'foo(struct x*)').
>>
>> I was able to make both work, more or less, on my Macbook Pro, using stock
>> Pharo 1.3 images and Igor's NBCog.app.
>>
>> I found NB to be a nicer programmer experience but I found that the FFI just
>> works.  There are a lot of gratuitous differences, but:
>>
>> * NB requires a VM compiled with a separate plugin (that I built from source
>> with no trouble -- Igor has done splendid work with CMakeVMMaker).
>> * NB should be faster than the FFI at pretty much everything, but I can't
>> say since I haven't measured.
>> * NB has useful primitives for determining if your external objects are
>> still valid and for determining the current operating system (I intend to
>> use these even if I don't use NB).
>> * NB has very few tests and examples and essentially no documentation.
>> * NB has a simple conceptual model, but it is still not as simple as the FFI
>> model.
>> * NB uses variableByteClasses to great advantage -- this is very cool.
>>
>> * NB makes it easy to crash your image, since you are evaluating native code
>> in image.
>> * NB is still, to my eye, raw and untested on Mac.  I had problems with
>> stack alignments in bizarre places, and NB's interaction with garbage
>> collection basically ended my attempts to use it.
>> * NB is basically impossible to debug if you aren't very strong with x86
>> assembly (I'm not even strong with x86 assembly), and the stack
>> manipulations rendered GDB pretty much useless for me.
>> * NB does not integrate well with the Pharo tools -- I think the
>> MethodTrailers are screwing up references, implementors, and source views,
>> but I haven't investigated.
>> * NB does not seem to support indirections very easily -- I struggled to
>> understand how to interface to foreign C functions with specs like
>> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
>> gave up.
>>
>> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
>> crash horribly for me, and although I was able to fix them, it I just don't
>> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
>> thought of making it all work on 3 platforms, and fix all the tool
>> integration, was just too much for me.
>>
>> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
>> somewhat documented, and reliable.  I think Igor has done some amazing work
>> and I do not want to bash his project (and certainly not his code -- the
>> man's a magician) but the fancy features are not necessary for my project.
>>
>>
>> fstephany wrote
>>>
>>> Will this interface handle callbacks ?
>>>
>>
>> I do not need callbacks so cannot speak to this issue.
>>
>> Yours,
>> Nick Alexander
>>
>> --
>> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>
>
>
>





--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Schwab,Wilhelm K
Why do it (automated or not) 2000 times when it can be done once?  Think of the wasted time and change log bloat.  Would you like to do it every time you move from Windows to Linux and back?  Say it once.  The superiority of #moduleName is clear.




From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 4:29 PM
To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 10:13 PM, Schwab,Wilhelm K <[hidden email]> wrote:
Agreed, except for thinking that changing all of the module names is not a big deal :) 

Why not? how many smalltalk lines of code do you think it could take to automate that? ;)

 
  An external library subclass is the way to go.  I know this already exists in FFI, but the evolving chapter on Spock does not mention it - it should not only mention it, but recommend it, IMHO.

Stef, does it make sense yet?





From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 3:52 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 9:25 PM, Stéphane Ducasse <[hidden email]> wrote:
can you give an example that I understand
"One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0 "


Bill:  that already exists in FFI.
Stef: what Bill says is that if you bind to a specific library you have to put its library name in each method that calls a ffi function. Example of DBX:

apiErrorType: handle number: err
    "int odbx_error_type( odbx_t*, int )"
    <cdecl: long 'odbx_error_type' (ulong long) module: 'opendbx'>
    ^ self externalCallFailed

----

apiInitialize: handle backend: backend host: host port: port
    "long odbx_init(odbx_t**, char*, char*, char*)"
    <cdecl: long 'odbx_init' (ulong* char* char* char*) module: 'opendbx'>
    ^self externalCallFailed

---

xxx

---

Notice the "module: 'opendbx"
So...if now the library is renamed or whatever, you have to change all methods. But I don't think this is a real big deal. There are much worst things.

Finaly, I copy paste an answer from Andreas from a previous thread:


The Right Way to do this is to have a subclass of ExternalLibrary and implement the class-side method #moduleName along the lines of:

MyLibrary class>>moduleName
 "Answer the module name to use for this library"
 Smalltalk platformName = 'Win32' ifTrue:[^'MyLibrary32.dll'].
 Smalltalk platformName = 'unix' ifTrue:[
    "check various locations and versions"
    #('/usr/lib/libMyLibrary.so'
      '/usr/lib/libMyLibrary.so.1'
      '/usr/lib/libMyLibrary.so.2'
      '/usr/share/libMyLibrary.so.1'
      '/usr/share/libMyLibrary.so.2'
) do:[:location|
         (FileDirectory fileExists: location) ifTrue:[^location].
    ].
    ^self error:'MyLibrary is not installed'
  ].

 
Tx

On Jan 8, 2012, at 9:18 PM, Schwab,Wilhelm K wrote:

> Stef,
>
> Absent the NB experience, +1 to the comments below.  FFI has pretty much "just worked" for me.  We need double arrays.  Callbacks would be great.  One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0  It makes a LOT more sense to have a class per library, and that class knows the name to use.  That's all the more true when one considers code such as ODBC that can run on multiple platforms with different names.  #moduleName can test the OS and answer the correct name.
>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Stéphane Ducasse [[hidden email]]
> Sent: Sunday, January 08, 2012 2:16 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>
> thanks for the feedback.
> We will come back you soon :)
> Because we should get FFI and NativeBoost fully working :).
>
> Stef
>
> On Jan 8, 2012, at 7:29 PM, ncalexan wrote:
>
>>
>> fstephany wrote
>>>
>>> I'm also a bit lost between all the different options (plugins,
>>> NativeBoost, FFI, Alien).
>>>
>>
>> I also found this confusing, so let me tell my recent experience and my
>> conclusion.
>>
>> I have written bindings to the SDL game programming library for use with
>> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
>> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
>> not cross-platform and not maintained, so I have not spent time
>> investigating it.
>>
>> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
>> which parses the relevant C header files and then writes either an NB or FFI
>> callout class tree.  I need to define a few structures, make a class pool
>> with some constants, and then define lots of callouts, many of which take
>> references to structures (like 'foo(struct x*)').
>>
>> I was able to make both work, more or less, on my Macbook Pro, using stock
>> Pharo 1.3 images and Igor's NBCog.app.
>>
>> I found NB to be a nicer programmer experience but I found that the FFI just
>> works.  There are a lot of gratuitous differences, but:
>>
>> * NB requires a VM compiled with a separate plugin (that I built from source
>> with no trouble -- Igor has done splendid work with CMakeVMMaker).
>> * NB should be faster than the FFI at pretty much everything, but I can't
>> say since I haven't measured.
>> * NB has useful primitives for determining if your external objects are
>> still valid and for determining the current operating system (I intend to
>> use these even if I don't use NB).
>> * NB has very few tests and examples and essentially no documentation.
>> * NB has a simple conceptual model, but it is still not as simple as the FFI
>> model.
>> * NB uses variableByteClasses to great advantage -- this is very cool.
>>
>> * NB makes it easy to crash your image, since you are evaluating native code
>> in image.
>> * NB is still, to my eye, raw and untested on Mac.  I had problems with
>> stack alignments in bizarre places, and NB's interaction with garbage
>> collection basically ended my attempts to use it.
>> * NB is basically impossible to debug if you aren't very strong with x86
>> assembly (I'm not even strong with x86 assembly), and the stack
>> manipulations rendered GDB pretty much useless for me.
>> * NB does not integrate well with the Pharo tools -- I think the
>> MethodTrailers are screwing up references, implementors, and source views,
>> but I haven't investigated.
>> * NB does not seem to support indirections very easily -- I struggled to
>> understand how to interface to foreign C functions with specs like
>> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
>> gave up.
>>
>> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
>> crash horribly for me, and although I was able to fix them, it I just don't
>> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
>> thought of making it all work on 3 platforms, and fix all the tool
>> integration, was just too much for me.
>>
>> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
>> somewhat documented, and reliable.  I think Igor has done some amazing work
>> and I do not want to bash his project (and certainly not his code -- the
>> man's a magician) but the fancy features are not necessary for my project.
>>
>>
>> fstephany wrote
>>>
>>> Will this interface handle callbacks ?
>>>
>>
>> I do not need callbacks so cannot speak to this issue.
>>
>> Yours,
>> Nick Alexander
>>
>> --
>> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>
>
>
>





--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Mariano Martinez Peck


On Sun, Jan 8, 2012 at 11:26 PM, Schwab,Wilhelm K <[hidden email]> wrote:
Why do it (automated or not) 2000 times when it can be done once? 

I don't understand. Why should I need to change the module name so frequently?  I have been working with squeakDBX since 4 years and we never needed to change it.
 
Think of the wasted time and change log bloat.  Would you like to do it every time you move from Windows to Linux and back? 

I don't understand why you say you should change that when moving from one OS to another one. I think you may have design issue, not FFI. With DBXTalk we have the superclass PharoOpenDBX, with has all the openDBX api with methods ^  self subclassResponsability. That's to mark the API. Then we have 2 subclasses, OpenDBXUnix and OpenDBXWindows. And then OpenDBXMac which is subclass from the Unix one but doesn't change or add anything so far. 
Then we have a Current (singleton) stored in a class variable and a class side initialize that does

ffiImplementationForOS
    | platformName |
    platformName := Smalltalk os platformName.
    platformName = 'Win32' ifTrue:[ ^OpenDBXWin32 basicNew initialize].
    platformName = 'unix' ifTrue:[ ^OpenDBXUnix basicNew initialize].
    platformName = 'Mac OS' ifTrue:[ ^OpenDBXMacOSX basicNew initialize].
    self error: 'Cannot identify platform'.

And that's all. We never need to change anything.  The only thing you should care is if moving the same image from one OS to another one then you have to reinitialize this class variable.  Or...you could add it to startUp list if you want.

Cheers
 
Say it once.  The superiority of #moduleName is clear.




From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 4:29 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 10:13 PM, Schwab,Wilhelm K <[hidden email]> wrote:
Agreed, except for thinking that changing all of the module names is not a big deal :) 

Why not? how many smalltalk lines of code do you think it could take to automate that? ;)

 
  An external library subclass is the way to go.  I know this already exists in FFI, but the evolving chapter on Spock does not mention it - it should not only mention it, but recommend it, IMHO.

Stef, does it make sense yet?





From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 3:52 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 9:25 PM, Stéphane Ducasse <[hidden email]> wrote:
can you give an example that I understand
"One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0 "


Bill:  that already exists in FFI.
Stef: what Bill says is that if you bind to a specific library you have to put its library name in each method that calls a ffi function. Example of DBX:

apiErrorType: handle number: err
    "int odbx_error_type( odbx_t*, int )"
    <cdecl: long 'odbx_error_type' (ulong long) module: 'opendbx'>
    ^ self externalCallFailed

----

apiInitialize: handle backend: backend host: host port: port
    "long odbx_init(odbx_t**, char*, char*, char*)"
    <cdecl: long 'odbx_init' (ulong* char* char* char*) module: 'opendbx'>
    ^self externalCallFailed

---

xxx

---

Notice the "module: 'opendbx"
So...if now the library is renamed or whatever, you have to change all methods. But I don't think this is a real big deal. There are much worst things.

Finaly, I copy paste an answer from Andreas from a previous thread:


The Right Way to do this is to have a subclass of ExternalLibrary and implement the class-side method #moduleName along the lines of:

MyLibrary class>>moduleName
 "Answer the module name to use for this library"
 Smalltalk platformName = 'Win32' ifTrue:[^'MyLibrary32.dll'].
 Smalltalk platformName = 'unix' ifTrue:[
    "check various locations and versions"
    #('/usr/lib/libMyLibrary.so'
      '/usr/lib/libMyLibrary.so.1'
      '/usr/lib/libMyLibrary.so.2'
      '/usr/share/libMyLibrary.so.1'
      '/usr/share/libMyLibrary.so.2'
) do:[:location|
         (FileDirectory fileExists: location) ifTrue:[^location].
    ].
    ^self error:'MyLibrary is not installed'
  ].

 
Tx

On Jan 8, 2012, at 9:18 PM, Schwab,Wilhelm K wrote:

> Stef,
>
> Absent the NB experience, +1 to the comments below.  FFI has pretty much "just worked" for me.  We need double arrays.  Callbacks would be great.  One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0  It makes a LOT more sense to have a class per library, and that class knows the name to use.  That's all the more true when one considers code such as ODBC that can run on multiple platforms with different names.  #moduleName can test the OS and answer the correct name.
>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Stéphane Ducasse [[hidden email]]
> Sent: Sunday, January 08, 2012 2:16 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>
> thanks for the feedback.
> We will come back you soon :)
> Because we should get FFI and NativeBoost fully working :).
>
> Stef
>
> On Jan 8, 2012, at 7:29 PM, ncalexan wrote:
>
>>
>> fstephany wrote
>>>
>>> I'm also a bit lost between all the different options (plugins,
>>> NativeBoost, FFI, Alien).
>>>
>>
>> I also found this confusing, so let me tell my recent experience and my
>> conclusion.
>>
>> I have written bindings to the SDL game programming library for use with
>> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
>> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
>> not cross-platform and not maintained, so I have not spent time
>> investigating it.
>>
>> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
>> which parses the relevant C header files and then writes either an NB or FFI
>> callout class tree.  I need to define a few structures, make a class pool
>> with some constants, and then define lots of callouts, many of which take
>> references to structures (like 'foo(struct x*)').
>>
>> I was able to make both work, more or less, on my Macbook Pro, using stock
>> Pharo 1.3 images and Igor's NBCog.app.
>>
>> I found NB to be a nicer programmer experience but I found that the FFI just
>> works.  There are a lot of gratuitous differences, but:
>>
>> * NB requires a VM compiled with a separate plugin (that I built from source
>> with no trouble -- Igor has done splendid work with CMakeVMMaker).
>> * NB should be faster than the FFI at pretty much everything, but I can't
>> say since I haven't measured.
>> * NB has useful primitives for determining if your external objects are
>> still valid and for determining the current operating system (I intend to
>> use these even if I don't use NB).
>> * NB has very few tests and examples and essentially no documentation.
>> * NB has a simple conceptual model, but it is still not as simple as the FFI
>> model.
>> * NB uses variableByteClasses to great advantage -- this is very cool.
>>
>> * NB makes it easy to crash your image, since you are evaluating native code
>> in image.
>> * NB is still, to my eye, raw and untested on Mac.  I had problems with
>> stack alignments in bizarre places, and NB's interaction with garbage
>> collection basically ended my attempts to use it.
>> * NB is basically impossible to debug if you aren't very strong with x86
>> assembly (I'm not even strong with x86 assembly), and the stack
>> manipulations rendered GDB pretty much useless for me.
>> * NB does not integrate well with the Pharo tools -- I think the
>> MethodTrailers are screwing up references, implementors, and source views,
>> but I haven't investigated.
>> * NB does not seem to support indirections very easily -- I struggled to
>> understand how to interface to foreign C functions with specs like
>> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
>> gave up.
>>
>> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
>> crash horribly for me, and although I was able to fix them, it I just don't
>> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
>> thought of making it all work on 3 platforms, and fix all the tool
>> integration, was just too much for me.
>>
>> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
>> somewhat documented, and reliable.  I think Igor has done some amazing work
>> and I do not want to bash his project (and certainly not his code -- the
>> man's a magician) but the fancy features are not necessary for my project.
>>
>>
>> fstephany wrote
>>>
>>> Will this interface handle callbacks ?
>>>
>>
>> I do not need callbacks so cannot speak to this issue.
>>
>> Yours,
>> Nick Alexander
>>
>> --
>> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>
>
>
>





--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Schwab,Wilhelm K
I am just the messenger: the library names are different on the various platforms, and the names can change over time (especially on Linux).



From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 5:33 PM
To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 11:26 PM, Schwab,Wilhelm K <[hidden email]> wrote:
Why do it (automated or not) 2000 times when it can be done once? 

I don't understand. Why should I need to change the module name so frequently?  I have been working with squeakDBX since 4 years and we never needed to change it.
 
Think of the wasted time and change log bloat.  Would you like to do it every time you move from Windows to Linux and back? 

I don't understand why you say you should change that when moving from one OS to another one. I think you may have design issue, not FFI. With DBXTalk we have the superclass PharoOpenDBX, with has all the openDBX api with methods ^  self subclassResponsability. That's to mark the API. Then we have 2 subclasses, OpenDBXUnix and OpenDBXWindows. And then OpenDBXMac which is subclass from the Unix one but doesn't change or add anything so far. 
Then we have a Current (singleton) stored in a class variable and a class side initialize that does

ffiImplementationForOS
    | platformName |
    platformName := Smalltalk os platformName.
    platformName = 'Win32' ifTrue:[ ^OpenDBXWin32 basicNew initialize].
    platformName = 'unix' ifTrue:[ ^OpenDBXUnix basicNew initialize].
    platformName = 'Mac OS' ifTrue:[ ^OpenDBXMacOSX basicNew initialize].
    self error: 'Cannot identify platform'.

And that's all. We never need to change anything.  The only thing you should care is if moving the same image from one OS to another one then you have to reinitialize this class variable.  Or...you could add it to startUp list if you want.

Cheers
 
Say it once.  The superiority of #moduleName is clear.




From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 4:29 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 10:13 PM, Schwab,Wilhelm K <[hidden email]> wrote:
Agreed, except for thinking that changing all of the module names is not a big deal :) 

Why not? how many smalltalk lines of code do you think it could take to automate that? ;)

 
  An external library subclass is the way to go.  I know this already exists in FFI, but the evolving chapter on Spock does not mention it - it should not only mention it, but recommend it, IMHO.

Stef, does it make sense yet?





From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 3:52 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 9:25 PM, Stéphane Ducasse <[hidden email]> wrote:
can you give an example that I understand
"One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0 "


Bill:  that already exists in FFI.
Stef: what Bill says is that if you bind to a specific library you have to put its library name in each method that calls a ffi function. Example of DBX:

apiErrorType: handle number: err
    "int odbx_error_type( odbx_t*, int )"
    <cdecl: long 'odbx_error_type' (ulong long) module: 'opendbx'>
    ^ self externalCallFailed

----

apiInitialize: handle backend: backend host: host port: port
    "long odbx_init(odbx_t**, char*, char*, char*)"
    <cdecl: long 'odbx_init' (ulong* char* char* char*) module: 'opendbx'>
    ^self externalCallFailed

---

xxx

---

Notice the "module: 'opendbx"
So...if now the library is renamed or whatever, you have to change all methods. But I don't think this is a real big deal. There are much worst things.

Finaly, I copy paste an answer from Andreas from a previous thread:


The Right Way to do this is to have a subclass of ExternalLibrary and implement the class-side method #moduleName along the lines of:

MyLibrary class>>moduleName
 "Answer the module name to use for this library"
 Smalltalk platformName = 'Win32' ifTrue:[^'MyLibrary32.dll'].
 Smalltalk platformName = 'unix' ifTrue:[
    "check various locations and versions"
    #('/usr/lib/libMyLibrary.so'
      '/usr/lib/libMyLibrary.so.1'
      '/usr/lib/libMyLibrary.so.2'
      '/usr/share/libMyLibrary.so.1'
      '/usr/share/libMyLibrary.so.2'
) do:[:location|
         (FileDirectory fileExists: location) ifTrue:[^location].
    ].
    ^self error:'MyLibrary is not installed'
  ].

 
Tx

On Jan 8, 2012, at 9:18 PM, Schwab,Wilhelm K wrote:

> Stef,
>
> Absent the NB experience, +1 to the comments below.  FFI has pretty much "just worked" for me.  We need double arrays.  Callbacks would be great.  One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0  It makes a LOT more sense to have a class per library, and that class knows the name to use.  That's all the more true when one considers code such as ODBC that can run on multiple platforms with different names.  #moduleName can test the OS and answer the correct name.
>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Stéphane Ducasse [[hidden email]]
> Sent: Sunday, January 08, 2012 2:16 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>
> thanks for the feedback.
> We will come back you soon :)
> Because we should get FFI and NativeBoost fully working :).
>
> Stef
>
> On Jan 8, 2012, at 7:29 PM, ncalexan wrote:
>
>>
>> fstephany wrote
>>>
>>> I'm also a bit lost between all the different options (plugins,
>>> NativeBoost, FFI, Alien).
>>>
>>
>> I also found this confusing, so let me tell my recent experience and my
>> conclusion.
>>
>> I have written bindings to the SDL game programming library for use with
>> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
>> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
>> not cross-platform and not maintained, so I have not spent time
>> investigating it.
>>
>> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
>> which parses the relevant C header files and then writes either an NB or FFI
>> callout class tree.  I need to define a few structures, make a class pool
>> with some constants, and then define lots of callouts, many of which take
>> references to structures (like 'foo(struct x*)').
>>
>> I was able to make both work, more or less, on my Macbook Pro, using stock
>> Pharo 1.3 images and Igor's NBCog.app.
>>
>> I found NB to be a nicer programmer experience but I found that the FFI just
>> works.  There are a lot of gratuitous differences, but:
>>
>> * NB requires a VM compiled with a separate plugin (that I built from source
>> with no trouble -- Igor has done splendid work with CMakeVMMaker).
>> * NB should be faster than the FFI at pretty much everything, but I can't
>> say since I haven't measured.
>> * NB has useful primitives for determining if your external objects are
>> still valid and for determining the current operating system (I intend to
>> use these even if I don't use NB).
>> * NB has very few tests and examples and essentially no documentation.
>> * NB has a simple conceptual model, but it is still not as simple as the FFI
>> model.
>> * NB uses variableByteClasses to great advantage -- this is very cool.
>>
>> * NB makes it easy to crash your image, since you are evaluating native code
>> in image.
>> * NB is still, to my eye, raw and untested on Mac.  I had problems with
>> stack alignments in bizarre places, and NB's interaction with garbage
>> collection basically ended my attempts to use it.
>> * NB is basically impossible to debug if you aren't very strong with x86
>> assembly (I'm not even strong with x86 assembly), and the stack
>> manipulations rendered GDB pretty much useless for me.
>> * NB does not integrate well with the Pharo tools -- I think the
>> MethodTrailers are screwing up references, implementors, and source views,
>> but I haven't investigated.
>> * NB does not seem to support indirections very easily -- I struggled to
>> understand how to interface to foreign C functions with specs like
>> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
>> gave up.
>>
>> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
>> crash horribly for me, and although I was able to fix them, it I just don't
>> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
>> thought of making it all work on 3 platforms, and fix all the tool
>> integration, was just too much for me.
>>
>> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
>> somewhat documented, and reliable.  I think Igor has done some amazing work
>> and I do not want to bash his project (and certainly not his code -- the
>> man's a magician) but the fancy features are not necessary for my project.
>>
>>
>> fstephany wrote
>>>
>>> Will this interface handle callbacks ?
>>>
>>
>> I do not need callbacks so cannot speak to this issue.
>>
>> Yours,
>> Nick Alexander
>>
>> --
>> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>
>
>
>





--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Mariano Martinez Peck


On Sun, Jan 8, 2012 at 11:53 PM, Schwab,Wilhelm K <[hidden email]> wrote:
I am just the messenger: the library names are different on the various platforms,

Yes, but that's not a problem with the solution I have just mentioned. No need to change anything.
 
and the names can change over time (especially on Linux).


yes, but that's why I meant this is not that frequent at all. How many times it change? once a year?  
And that's why I recommended having a small script to change it.
So again, I don't see this like a critical problem of FFI.
 


From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 5:33 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 11:26 PM, Schwab,Wilhelm K <[hidden email]> wrote:
Why do it (automated or not) 2000 times when it can be done once? 

I don't understand. Why should I need to change the module name so frequently?  I have been working with squeakDBX since 4 years and we never needed to change it.
 
Think of the wasted time and change log bloat.  Would you like to do it every time you move from Windows to Linux and back? 

I don't understand why you say you should change that when moving from one OS to another one. I think you may have design issue, not FFI. With DBXTalk we have the superclass PharoOpenDBX, with has all the openDBX api with methods ^  self subclassResponsability. That's to mark the API. Then we have 2 subclasses, OpenDBXUnix and OpenDBXWindows. And then OpenDBXMac which is subclass from the Unix one but doesn't change or add anything so far. 
Then we have a Current (singleton) stored in a class variable and a class side initialize that does

ffiImplementationForOS
    | platformName |
    platformName := Smalltalk os platformName.
    platformName = 'Win32' ifTrue:[ ^OpenDBXWin32 basicNew initialize].
    platformName = 'unix' ifTrue:[ ^OpenDBXUnix basicNew initialize].
    platformName = 'Mac OS' ifTrue:[ ^OpenDBXMacOSX basicNew initialize].
    self error: 'Cannot identify platform'.

And that's all. We never need to change anything.  The only thing you should care is if moving the same image from one OS to another one then you have to reinitialize this class variable.  Or...you could add it to startUp list if you want.

Cheers
 
Say it once.  The superiority of #moduleName is clear.




From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 4:29 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 10:13 PM, Schwab,Wilhelm K <[hidden email]> wrote:
Agreed, except for thinking that changing all of the module names is not a big deal :) 

Why not? how many smalltalk lines of code do you think it could take to automate that? ;)

 
  An external library subclass is the way to go.  I know this already exists in FFI, but the evolving chapter on Spock does not mention it - it should not only mention it, but recommend it, IMHO.

Stef, does it make sense yet?





From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 3:52 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 9:25 PM, Stéphane Ducasse <[hidden email]> wrote:
can you give an example that I understand
"One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0 "


Bill:  that already exists in FFI.
Stef: what Bill says is that if you bind to a specific library you have to put its library name in each method that calls a ffi function. Example of DBX:

apiErrorType: handle number: err
    "int odbx_error_type( odbx_t*, int )"
    <cdecl: long 'odbx_error_type' (ulong long) module: 'opendbx'>
    ^ self externalCallFailed

----

apiInitialize: handle backend: backend host: host port: port
    "long odbx_init(odbx_t**, char*, char*, char*)"
    <cdecl: long 'odbx_init' (ulong* char* char* char*) module: 'opendbx'>
    ^self externalCallFailed

---

xxx

---

Notice the "module: 'opendbx"
So...if now the library is renamed or whatever, you have to change all methods. But I don't think this is a real big deal. There are much worst things.

Finaly, I copy paste an answer from Andreas from a previous thread:


The Right Way to do this is to have a subclass of ExternalLibrary and implement the class-side method #moduleName along the lines of:

MyLibrary class>>moduleName
 "Answer the module name to use for this library"
 Smalltalk platformName = 'Win32' ifTrue:[^'MyLibrary32.dll'].
 Smalltalk platformName = 'unix' ifTrue:[
    "check various locations and versions"
    #('/usr/lib/libMyLibrary.so'
      '/usr/lib/libMyLibrary.so.1'
      '/usr/lib/libMyLibrary.so.2'
      '/usr/share/libMyLibrary.so.1'
      '/usr/share/libMyLibrary.so.2'
) do:[:location|
         (FileDirectory fileExists: location) ifTrue:[^location].
    ].
    ^self error:'MyLibrary is not installed'
  ].

 
Tx

On Jan 8, 2012, at 9:18 PM, Schwab,Wilhelm K wrote:

> Stef,
>
> Absent the NB experience, +1 to the comments below.  FFI has pretty much "just worked" for me.  We need double arrays.  Callbacks would be great.  One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0  It makes a LOT more sense to have a class per library, and that class knows the name to use.  That's all the more true when one considers code such as ODBC that can run on multiple platforms with different names.  #moduleName can test the OS and answer the correct name.
>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Stéphane Ducasse [[hidden email]]
> Sent: Sunday, January 08, 2012 2:16 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>
> thanks for the feedback.
> We will come back you soon :)
> Because we should get FFI and NativeBoost fully working :).
>
> Stef
>
> On Jan 8, 2012, at 7:29 PM, ncalexan wrote:
>
>>
>> fstephany wrote
>>>
>>> I'm also a bit lost between all the different options (plugins,
>>> NativeBoost, FFI, Alien).
>>>
>>
>> I also found this confusing, so let me tell my recent experience and my
>> conclusion.
>>
>> I have written bindings to the SDL game programming library for use with
>> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
>> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
>> not cross-platform and not maintained, so I have not spent time
>> investigating it.
>>
>> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
>> which parses the relevant C header files and then writes either an NB or FFI
>> callout class tree.  I need to define a few structures, make a class pool
>> with some constants, and then define lots of callouts, many of which take
>> references to structures (like 'foo(struct x*)').
>>
>> I was able to make both work, more or less, on my Macbook Pro, using stock
>> Pharo 1.3 images and Igor's NBCog.app.
>>
>> I found NB to be a nicer programmer experience but I found that the FFI just
>> works.  There are a lot of gratuitous differences, but:
>>
>> * NB requires a VM compiled with a separate plugin (that I built from source
>> with no trouble -- Igor has done splendid work with CMakeVMMaker).
>> * NB should be faster than the FFI at pretty much everything, but I can't
>> say since I haven't measured.
>> * NB has useful primitives for determining if your external objects are
>> still valid and for determining the current operating system (I intend to
>> use these even if I don't use NB).
>> * NB has very few tests and examples and essentially no documentation.
>> * NB has a simple conceptual model, but it is still not as simple as the FFI
>> model.
>> * NB uses variableByteClasses to great advantage -- this is very cool.
>>
>> * NB makes it easy to crash your image, since you are evaluating native code
>> in image.
>> * NB is still, to my eye, raw and untested on Mac.  I had problems with
>> stack alignments in bizarre places, and NB's interaction with garbage
>> collection basically ended my attempts to use it.
>> * NB is basically impossible to debug if you aren't very strong with x86
>> assembly (I'm not even strong with x86 assembly), and the stack
>> manipulations rendered GDB pretty much useless for me.
>> * NB does not integrate well with the Pharo tools -- I think the
>> MethodTrailers are screwing up references, implementors, and source views,
>> but I haven't investigated.
>> * NB does not seem to support indirections very easily -- I struggled to
>> understand how to interface to foreign C functions with specs like
>> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
>> gave up.
>>
>> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
>> crash horribly for me, and although I was able to fix them, it I just don't
>> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
>> thought of making it all work on 3 platforms, and fix all the tool
>> integration, was just too much for me.
>>
>> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
>> somewhat documented, and reliable.  I think Igor has done some amazing work
>> and I do not want to bash his project (and certainly not his code -- the
>> man's a magician) but the fancy features are not necessary for my project.
>>
>>
>> fstephany wrote
>>>
>>> Will this interface handle callbacks ?
>>>
>>
>> I do not need callbacks so cannot speak to this issue.
>>
>> Yours,
>> Nick Alexander
>>
>> --
>> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>
>
>
>





--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Schwab,Wilhelm K
It's not a problem at all, because the correct solution is supported - it just needs better exposure in the Spock chapter.



\

From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 5:59 PM
To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 11:53 PM, Schwab,Wilhelm K <[hidden email]> wrote:
I am just the messenger: the library names are different on the various platforms,

Yes, but that's not a problem with the solution I have just mentioned. No need to change anything.
 
and the names can change over time (especially on Linux).


yes, but that's why I meant this is not that frequent at all. How many times it change? once a year?  
And that's why I recommended having a small script to change it.
So again, I don't see this like a critical problem of FFI.
 


From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 5:33 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 11:26 PM, Schwab,Wilhelm K <[hidden email]> wrote:
Why do it (automated or not) 2000 times when it can be done once? 

I don't understand. Why should I need to change the module name so frequently?  I have been working with squeakDBX since 4 years and we never needed to change it.
 
Think of the wasted time and change log bloat.  Would you like to do it every time you move from Windows to Linux and back? 

I don't understand why you say you should change that when moving from one OS to another one. I think you may have design issue, not FFI. With DBXTalk we have the superclass PharoOpenDBX, with has all the openDBX api with methods ^  self subclassResponsability. That's to mark the API. Then we have 2 subclasses, OpenDBXUnix and OpenDBXWindows. And then OpenDBXMac which is subclass from the Unix one but doesn't change or add anything so far. 
Then we have a Current (singleton) stored in a class variable and a class side initialize that does

ffiImplementationForOS
    | platformName |
    platformName := Smalltalk os platformName.
    platformName = 'Win32' ifTrue:[ ^OpenDBXWin32 basicNew initialize].
    platformName = 'unix' ifTrue:[ ^OpenDBXUnix basicNew initialize].
    platformName = 'Mac OS' ifTrue:[ ^OpenDBXMacOSX basicNew initialize].
    self error: 'Cannot identify platform'.

And that's all. We never need to change anything.  The only thing you should care is if moving the same image from one OS to another one then you have to reinitialize this class variable.  Or...you could add it to startUp list if you want.

Cheers
 
Say it once.  The superiority of #moduleName is clear.




From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 4:29 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 10:13 PM, Schwab,Wilhelm K <[hidden email]> wrote:
Agreed, except for thinking that changing all of the module names is not a big deal :) 

Why not? how many smalltalk lines of code do you think it could take to automate that? ;)

 
  An external library subclass is the way to go.  I know this already exists in FFI, but the evolving chapter on Spock does not mention it - it should not only mention it, but recommend it, IMHO.

Stef, does it make sense yet?





From: [hidden email] [[hidden email]] on behalf of Mariano Martinez Peck [[hidden email]]
Sent: Sunday, January 08, 2012 3:52 PM

To: [hidden email]
Subject: Re: [Pharo-project] Best way for FFI in Pharo



On Sun, Jan 8, 2012 at 9:25 PM, Stéphane Ducasse <[hidden email]> wrote:
can you give an example that I understand
"One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0 "


Bill:  that already exists in FFI.
Stef: what Bill says is that if you bind to a specific library you have to put its library name in each method that calls a ffi function. Example of DBX:

apiErrorType: handle number: err
    "int odbx_error_type( odbx_t*, int )"
    <cdecl: long 'odbx_error_type' (ulong long) module: 'opendbx'>
    ^ self externalCallFailed

----

apiInitialize: handle backend: backend host: host port: port
    "long odbx_init(odbx_t**, char*, char*, char*)"
    <cdecl: long 'odbx_init' (ulong* char* char* char*) module: 'opendbx'>
    ^self externalCallFailed

---

xxx

---

Notice the "module: 'opendbx"
So...if now the library is renamed or whatever, you have to change all methods. But I don't think this is a real big deal. There are much worst things.

Finaly, I copy paste an answer from Andreas from a previous thread:


The Right Way to do this is to have a subclass of ExternalLibrary and implement the class-side method #moduleName along the lines of:

MyLibrary class>>moduleName
 "Answer the module name to use for this library"
 Smalltalk platformName = 'Win32' ifTrue:[^'MyLibrary32.dll'].
 Smalltalk platformName = 'unix' ifTrue:[
    "check various locations and versions"
    #('/usr/lib/libMyLibrary.so'
      '/usr/lib/libMyLibrary.so.1'
      '/usr/lib/libMyLibrary.so.2'
      '/usr/share/libMyLibrary.so.1'
      '/usr/share/libMyLibrary.so.2'
) do:[:location|
         (FileDirectory fileExists: location) ifTrue:[^location].
    ].
    ^self error:'MyLibrary is not installed'
  ].

 
Tx

On Jan 8, 2012, at 9:18 PM, Schwab,Wilhelm K wrote:

> Stef,
>
> Absent the NB experience, +1 to the comments below.  FFI has pretty much "just worked" for me.  We need double arrays.  Callbacks would be great.  One thing I strongly recommend is to favor #moduleName over pragmas listing the library - imagine changing 2000+ GSL calls if the library name changes =:0  It makes a LOT more sense to have a class per library, and that class knows the name to use.  That's all the more true when one considers code such as ODBC that can run on multiple platforms with different names.  #moduleName can test the OS and answer the correct name.
>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Stéphane Ducasse [[hidden email]]
> Sent: Sunday, January 08, 2012 2:16 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>
> thanks for the feedback.
> We will come back you soon :)
> Because we should get FFI and NativeBoost fully working :).
>
> Stef
>
> On Jan 8, 2012, at 7:29 PM, ncalexan wrote:
>
>>
>> fstephany wrote
>>>
>>> I'm also a bit lost between all the different options (plugins,
>>> NativeBoost, FFI, Alien).
>>>
>>
>> I also found this confusing, so let me tell my recent experience and my
>> conclusion.
>>
>> I have written bindings to the SDL game programming library for use with
>> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
>> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
>> not cross-platform and not maintained, so I have not spent time
>> investigating it.
>>
>> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
>> which parses the relevant C header files and then writes either an NB or FFI
>> callout class tree.  I need to define a few structures, make a class pool
>> with some constants, and then define lots of callouts, many of which take
>> references to structures (like 'foo(struct x*)').
>>
>> I was able to make both work, more or less, on my Macbook Pro, using stock
>> Pharo 1.3 images and Igor's NBCog.app.
>>
>> I found NB to be a nicer programmer experience but I found that the FFI just
>> works.  There are a lot of gratuitous differences, but:
>>
>> * NB requires a VM compiled with a separate plugin (that I built from source
>> with no trouble -- Igor has done splendid work with CMakeVMMaker).
>> * NB should be faster than the FFI at pretty much everything, but I can't
>> say since I haven't measured.
>> * NB has useful primitives for determining if your external objects are
>> still valid and for determining the current operating system (I intend to
>> use these even if I don't use NB).
>> * NB has very few tests and examples and essentially no documentation.
>> * NB has a simple conceptual model, but it is still not as simple as the FFI
>> model.
>> * NB uses variableByteClasses to great advantage -- this is very cool.
>>
>> * NB makes it easy to crash your image, since you are evaluating native code
>> in image.
>> * NB is still, to my eye, raw and untested on Mac.  I had problems with
>> stack alignments in bizarre places, and NB's interaction with garbage
>> collection basically ended my attempts to use it.
>> * NB is basically impossible to debug if you aren't very strong with x86
>> assembly (I'm not even strong with x86 assembly), and the stack
>> manipulations rendered GDB pretty much useless for me.
>> * NB does not integrate well with the Pharo tools -- I think the
>> MethodTrailers are screwing up references, implementors, and source views,
>> but I haven't investigated.
>> * NB does not seem to support indirections very easily -- I struggled to
>> understand how to interface to foreign C functions with specs like
>> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
>> gave up.
>>
>> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
>> crash horribly for me, and although I was able to fix them, it I just don't
>> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
>> thought of making it all work on 3 platforms, and fix all the tool
>> integration, was just too much for me.
>>
>> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
>> somewhat documented, and reliable.  I think Igor has done some amazing work
>> and I do not want to bash his project (and certainly not his code -- the
>> man's a magician) but the fancy features are not necessary for my project.
>>
>>
>> fstephany wrote
>>>
>>> Will this interface handle callbacks ?
>>>
>>
>> I do not need callbacks so cannot speak to this issue.
>>
>> Yours,
>> Nick Alexander
>>
>> --
>> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>
>
>
>





--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

drush66
In reply to this post by Udo Schneider
So, am I getting this right: if I need callbacks, then I need NativeBoost?

Can those callbacks be used in situations where it is necessary to deliver very high number of them quickly and efficiently?

I am looking at libev - event library used to deliver high I/O traffic through evented interface, and it depends heavily on callbacks to deliver each event.
 
Are there prebuilt VM-s with NB?
Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

vonbecmann
see 
https://ci.lille.inria.fr/pharo/view/NativeBoost/

On Fri, Jan 20, 2012 at 12:31 PM, drush66 <[hidden email]> wrote:


Are there prebuilt VM-s with NB?

-----
http://www.cloud208.com/
--
View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4313534.html
Sent from the Pharo Smalltalk mailing list archive at Nabble.com.


Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Igor Stasenko
In reply to this post by ncalexan
Thanks, a lot for your first-hand experience.

On 8 January 2012 20:29, ncalexan <[hidden email]> wrote:

>
> fstephany wrote
>>
>> I'm also a bit lost between all the different options (plugins,
>> NativeBoost, FFI, Alien).
>>
>
> I also found this confusing, so let me tell my recent experience and my
> conclusion.
>
> I have written bindings to the SDL game programming library for use with
> Pharo.  SDL is cross-platform, and I want to support Mac (most of all),
> Windows (one must), and Linux (if I must).  As far as I can tell, Alien is
> not cross-platform and not maintained, so I have not spent time
> investigating it.
>
> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs package
> which parses the relevant C header files and then writes either an NB or FFI
> callout class tree.  I need to define a few structures, make a class pool
> with some constants, and then define lots of callouts, many of which take
> references to structures (like 'foo(struct x*)').

Do you have it available somewhere?

>
> I was able to make both work, more or less, on my Macbook Pro, using stock
> Pharo 1.3 images and Igor's NBCog.app.
>
> I found NB to be a nicer programmer experience but I found that the FFI just
> works.  There are a lot of gratuitous differences, but:
>
> * NB requires a VM compiled with a separate plugin (that I built from source
> with no trouble -- Igor has done splendid work with CMakeVMMaker).
> * NB should be faster than the FFI at pretty much everything, but I can't
> say since I haven't measured.
> * NB has useful primitives for determining if your external objects are
> still valid and for determining the current operating system (I intend to
> use these even if I don't use NB).
> * NB has very few tests and examples and essentially no documentation.

:) Yes, i should start writing it. I was waiting till project will
mature enough & stabilize a bit.
and of course i am lazy.

> * NB has a simple conceptual model, but it is still not as simple as the FFI
> model.

yep. a bit more concepts, because a bit more features.

> * NB uses variableByteClasses to great advantage -- this is very cool.
>
> * NB makes it easy to crash your image, since you are evaluating native code
> in image.

this statement can be applied to any FFInterface, if you declare function
with wrong types/arguments or pass wrong argument values. :)
I spent last 3 days rewriting aligment interface, and hacking opengl
glue code, and guess what:
it never crashed on me.
And yes, i saving image every time before going to run the freshly
hacked assembly code :)

> * NB is still, to my eye, raw and untested on Mac.  I had problems with
> stack alignments in bizarre places, and NB's interaction with garbage
> collection basically ended my attempts to use it.

need more data, data pls. I refined a stack-aligned interface recently, and
don't have any problems with alignment.. but sure, i maybe lack of
real-world experience ;)

> * NB is basically impossible to debug if you aren't very strong with x86
> assembly (I'm not even strong with x86 assembly), and the stack
> manipulations rendered GDB pretty much useless for me.

well, you don't have to go there. Usually i takes less time to grok
assembler code in image
to find an error than trying to understand what happens in GDB.. in
text mode... ohhhhhh :)

> * NB does not integrate well with the Pharo tools -- I think the
> MethodTrailers are screwing up references, implementors, and source views,
> but I haven't investigated.

Hmm.. there should be everything fine there. Trailers are not
stripping any interpreter-specific state,
just adds own on top of it.

> * NB does not seem to support indirections very easily -- I struggled to
> understand how to interface to foreign C functions with specs like
> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and eventually
> gave up.

consider following:

struct x k;
struct x * l = &k;
struct x ** m = &l;

"struct x" foo (k);   << push &k, because all structs are passed by reference.
"struct x * " foo (l); << push value of l = &k
"struct x **" foo (m);  << push value of m = &l

&l obviously <> &k
becasue &k points to first byte of your structure, while &l points to
first byte of _pointer_ to that structure.

usually a ** used in cases when function wants to store result on
success. It is like say:
 give me a memory location where i should store a pointer to that
structure, so you can use it later.

Unfortunately there is no way to automate that. It means that if you
owning a structure,
myStruct = NBExternalStructureSubclass new.

passing it as argument to functions which takes struct x** may be dangerous.

So, in case if function under the question wants to return a pointer
to struct back to you, you can do just following:

- change the function declaration to following:

 #( void foo ( NBExternalAddress * address ) )  "this stands for
void foo(struct **x) "

in this case NB will not push the value of address , but a place in
memory where this pointer is stored.
Function then writes a new address value there. and so you can use it
later by simply:

addr := NBExternalAddress new.
self foo: addr.
mystruct := NBExternalStructure fromPointer: addr.

and now you can access the values in struct:
mystruct x = 5 ifTrue: [...]

If function may want not only to give you address back, but also may
want to read your struct fields by occasion,
then you have no choice but to use external memory:

myStruct := MyExternalStruct new.
myStruct x: 4; y 5; ... etc..

externalSpace := NativeBoost allocate: myStruct class instanceSize.
NativeBoost memcpy: myStruct to: externalSpace size: myStruct class
instanceSize.

now, if you pass externalSpace in same way:

self foo: externalSpace

it will push an address to that address, where copy of your structure lies ;)

Now, i would like to know how you solving this with FFI.

>
> I wanted to help Igor with the NB project, but the NB-OpenGL Mac bindings
> crash horribly for me, and although I was able to fix them, it I just don't
> hack enough x86 assembly to figure out all the tricks Igor is doing.  The
> thought of making it all work on 3 platforms, and fix all the tool
> integration, was just too much for me.
>

Now it should work fine, so no worries :)
Just remember: Communication is the key.
I am not dead, and i willing to help anyone who using NativeBoost,
So, please, send me feedback, troublesome code, or even whole image(s)
and do not hesitate to contact with me.
Because it is two way road: i helping you to make it work, you helping
me to get rid of defects and misconceptions and missing features. :)

> In conclusion, I prefer the FFI.  It is old, cross platform, well tested,
> somewhat documented, and reliable.  I think Igor has done some amazing work
> and I do not want to bash his project (and certainly not his code -- the
> man's a magician) but the fancy features are not necessary for my project.
>
>
> fstephany wrote
>>
>> Will this interface handle callbacks ?
>>
>
> I do not need callbacks so cannot speak to this issue.
>
> Yours,
> Nick Alexander
>
> --
> View this message in context: http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>



--
Best regards,
Igor Stasenko.

Reply | Threaded
Open this post in threaded view
|

Re: Best way for FFI in Pharo

Igor Stasenko
In reply to this post by Schwab,Wilhelm K
On 9 January 2012 02:26, Schwab,Wilhelm K <[hidden email]> wrote:
> It's not a problem at all, because the correct solution is supported - it
> just needs better exposure in the Spock chapter.
>
Sure it is there.
Lately , when Spock chapter was under writing we decided to add a
convenience interface, so you can type less to define a callout.

Here how it usually looks like today:

CGMainDisplayID
        <primitive: #primitiveNativeCall module: #NativeBoostPlugin>

        "Returns the display ID of the main display."
        ^  self nbCall: #( CGDirectDisplayID CGMainDisplayID (  void ) )
module: self ApplicationServices


but you can also omit #module: keyword, if you implement
#nbLibraryNameOrHandle in your class.
so you can leave it with just:
        ^  self nbCall: #( CGDirectDisplayID CGMainDisplayID (  void ) )

Also, ultimately you can override
#nbGetSymbolAddress: aName module: aModuleNameOrHandle
to redefine completely semantics where and how your class should look
for given symbol (function) and obtain a pointer to it.

>
> \
> ________________________________
> From: [hidden email]
> [[hidden email]] on behalf of Mariano Martinez
> Peck [[hidden email]]
> Sent: Sunday, January 08, 2012 5:59 PM
>
> To: [hidden email]
> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>
>
>
> On Sun, Jan 8, 2012 at 11:53 PM, Schwab,Wilhelm K <[hidden email]>
> wrote:
>>
>> I am just the messenger: the library names are different on the various
>> platforms,
>
>
> Yes, but that's not a problem with the solution I have just mentioned. No
> need to change anything.
>
>>
>> and the names can change over time (especially on Linux).
>>
>
> yes, but that's why I meant this is not that frequent at all. How many times
> it change? once a year?
> And that's why I recommended having a small script to change it.
> So again, I don't see this like a critical problem of FFI.
>
>>
>>
>> ________________________________
>> From: [hidden email]
>> [[hidden email]] on behalf of Mariano Martinez
>> Peck [[hidden email]]
>> Sent: Sunday, January 08, 2012 5:33 PM
>>
>> To: [hidden email]
>> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>>
>>
>>
>> On Sun, Jan 8, 2012 at 11:26 PM, Schwab,Wilhelm K <[hidden email]>
>> wrote:
>>>
>>> Why do it (automated or not) 2000 times when it can be done once?
>>
>>
>> I don't understand. Why should I need to change the module name so
>> frequently?  I have been working with squeakDBX since 4 years and we never
>> needed to change it.
>>
>>>
>>> Think of the wasted time and change log bloat.  Would you like to do it
>>> every time you move from Windows to Linux and back?
>>
>>
>> I don't understand why you say you should change that when moving from one
>> OS to another one. I think you may have design issue, not FFI. With DBXTalk
>> we have the superclass PharoOpenDBX, with has all the openDBX api with
>> methods ^  self subclassResponsability. That's to mark the API. Then we have
>> 2 subclasses, OpenDBXUnix and OpenDBXWindows. And then OpenDBXMac which is
>> subclass from the Unix one but doesn't change or add anything so far.
>> Then we have a Current (singleton) stored in a class variable and a class
>> side initialize that does
>>
>> ffiImplementationForOS
>>     | platformName |
>>     platformName := Smalltalk os platformName.
>>     platformName = 'Win32' ifTrue:[ ^OpenDBXWin32 basicNew initialize].
>>     platformName = 'unix' ifTrue:[ ^OpenDBXUnix basicNew initialize].
>>     platformName = 'Mac OS' ifTrue:[ ^OpenDBXMacOSX basicNew initialize].
>>     self error: 'Cannot identify platform'.
>>
>> And that's all. We never need to change anything.  The only thing you
>> should care is if moving the same image from one OS to another one then you
>> have to reinitialize this class variable.  Or...you could add it to startUp
>> list if you want.
>>
>> Cheers
>>
>>>
>>> Say it once.  The superiority of #moduleName is clear.
>>>
>>>
>>>
>>> ________________________________
>>> From: [hidden email]
>>> [[hidden email]] on behalf of Mariano Martinez
>>> Peck [[hidden email]]
>>> Sent: Sunday, January 08, 2012 4:29 PM
>>>
>>> To: [hidden email]
>>> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>>>
>>>
>>>
>>> On Sun, Jan 8, 2012 at 10:13 PM, Schwab,Wilhelm K <[hidden email]>
>>> wrote:
>>>>
>>>> Agreed, except for thinking that changing all of the module names is not
>>>> a big deal :)
>>>
>>>
>>> Why not? how many smalltalk lines of code do you think it could take to
>>> automate that? ;)
>>>
>>>
>>>>
>>>>   An external library subclass is the way to go.  I know this already
>>>> exists in FFI, but the evolving chapter on Spock does not mention it - it
>>>> should not only mention it, but recommend it, IMHO.
>>>>
>>>> Stef, does it make sense yet?
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: [hidden email]
>>>> [[hidden email]] on behalf of Mariano Martinez
>>>> Peck [[hidden email]]
>>>> Sent: Sunday, January 08, 2012 3:52 PM
>>>>
>>>> To: [hidden email]
>>>> Subject: Re: [Pharo-project] Best way for FFI in Pharo
>>>>
>>>>
>>>>
>>>> On Sun, Jan 8, 2012 at 9:25 PM, Stéphane Ducasse
>>>> <[hidden email]> wrote:
>>>>>
>>>>> can you give an example that I understand
>>>>> "One thing I strongly recommend is to favor #moduleName over pragmas
>>>>> listing the library - imagine changing 2000+ GSL calls if the library name
>>>>> changes =:0 "
>>>>>
>>>>
>>>> Bill:  that already exists in FFI.
>>>> Stef: what Bill says is that if you bind to a specific library you have
>>>> to put its library name in each method that calls a ffi function. Example of
>>>> DBX:
>>>>
>>>> apiErrorType: handle number: err
>>>>     "int odbx_error_type( odbx_t*, int )"
>>>>     <cdecl: long 'odbx_error_type' (ulong long) module: 'opendbx'>
>>>>     ^ self externalCallFailed
>>>>
>>>> ----
>>>>
>>>> apiInitialize: handle backend: backend host: host port: port
>>>>     "long odbx_init(odbx_t**, char*, char*, char*)"
>>>>     <cdecl: long 'odbx_init' (ulong* char* char* char*) module:
>>>> 'opendbx'>
>>>>     ^self externalCallFailed
>>>>
>>>> ---
>>>>
>>>> xxx
>>>>
>>>> ---
>>>>
>>>> Notice the "module: 'opendbx"
>>>> So...if now the library is renamed or whatever, you have to change all
>>>> methods. But I don't think this is a real big deal. There are much worst
>>>> things.
>>>>
>>>> Finaly, I copy paste an answer from Andreas from a previous thread:
>>>>
>>>>
>>>> The Right Way to do this is to have a subclass of ExternalLibrary and
>>>> implement the class-side method #moduleName along the lines of:
>>>>
>>>> MyLibrary class>>moduleName
>>>>  "Answer the module name to use for this library"
>>>>  Smalltalk platformName = 'Win32' ifTrue:[^'MyLibrary32.dll'].
>>>>  Smalltalk platformName = 'unix' ifTrue:[
>>>>     "check various locations and versions"
>>>>     #('/usr/lib/libMyLibrary.so'
>>>>       '/usr/lib/libMyLibrary.so.1'
>>>>       '/usr/lib/libMyLibrary.so.2'
>>>>       '/usr/share/libMyLibrary.so.1'
>>>>       '/usr/share/libMyLibrary.so.2'
>>>> ) do:[:location|
>>>>          (FileDirectory fileExists: location) ifTrue:[^location].
>>>>     ].
>>>>     ^self error:'MyLibrary is not installed'
>>>>   ].
>>>>
>>>>
>>>>>
>>>>> Tx
>>>>>
>>>>> On Jan 8, 2012, at 9:18 PM, Schwab,Wilhelm K wrote:
>>>>>
>>>>> > Stef,
>>>>> >
>>>>> > Absent the NB experience, +1 to the comments below.  FFI has pretty
>>>>> > much "just worked" for me.  We need double arrays.  Callbacks would be
>>>>> > great.  One thing I strongly recommend is to favor #moduleName over pragmas
>>>>> > listing the library - imagine changing 2000+ GSL calls if the library name
>>>>> > changes =:0  It makes a LOT more sense to have a class per library, and that
>>>>> > class knows the name to use.  That's all the more true when one considers
>>>>> > code such as ODBC that can run on multiple platforms with different names.
>>>>> >  #moduleName can test the OS and answer the correct name.
>>>>> >
>>>>> > Bill
>>>>> >
>>>>> >
>>>>> > ________________________________________
>>>>> > From: [hidden email]
>>>>> > [[hidden email]] on behalf of Stéphane Ducasse
>>>>> > [[hidden email]]
>>>>> > Sent: Sunday, January 08, 2012 2:16 PM
>>>>> > To: [hidden email]
>>>>> > Subject: Re: [Pharo-project] Best way for FFI in Pharo
>>>>> >
>>>>> > thanks for the feedback.
>>>>> > We will come back you soon :)
>>>>> > Because we should get FFI and NativeBoost fully working :).
>>>>> >
>>>>> > Stef
>>>>> >
>>>>> > On Jan 8, 2012, at 7:29 PM, ncalexan wrote:
>>>>> >
>>>>> >>
>>>>> >> fstephany wrote
>>>>> >>>
>>>>> >>> I'm also a bit lost between all the different options (plugins,
>>>>> >>> NativeBoost, FFI, Alien).
>>>>> >>>
>>>>> >>
>>>>> >> I also found this confusing, so let me tell my recent experience and
>>>>> >> my
>>>>> >> conclusion.
>>>>> >>
>>>>> >> I have written bindings to the SDL game programming library for use
>>>>> >> with
>>>>> >> Pharo.  SDL is cross-platform, and I want to support Mac (most of
>>>>> >> all),
>>>>> >> Windows (one must), and Linux (if I must).  As far as I can tell,
>>>>> >> Alien is
>>>>> >> not cross-platform and not maintained, so I have not spent time
>>>>> >> investigating it.
>>>>> >>
>>>>> >> Following Igor Stasenko's OpenGLSpecs package, I wrote an SDLSpecs
>>>>> >> package
>>>>> >> which parses the relevant C header files and then writes either an
>>>>> >> NB or FFI
>>>>> >> callout class tree.  I need to define a few structures, make a class
>>>>> >> pool
>>>>> >> with some constants, and then define lots of callouts, many of which
>>>>> >> take
>>>>> >> references to structures (like 'foo(struct x*)').
>>>>> >>
>>>>> >> I was able to make both work, more or less, on my Macbook Pro, using
>>>>> >> stock
>>>>> >> Pharo 1.3 images and Igor's NBCog.app.
>>>>> >>
>>>>> >> I found NB to be a nicer programmer experience but I found that the
>>>>> >> FFI just
>>>>> >> works.  There are a lot of gratuitous differences, but:
>>>>> >>
>>>>> >> * NB requires a VM compiled with a separate plugin (that I built
>>>>> >> from source
>>>>> >> with no trouble -- Igor has done splendid work with CMakeVMMaker).
>>>>> >> * NB should be faster than the FFI at pretty much everything, but I
>>>>> >> can't
>>>>> >> say since I haven't measured.
>>>>> >> * NB has useful primitives for determining if your external objects
>>>>> >> are
>>>>> >> still valid and for determining the current operating system (I
>>>>> >> intend to
>>>>> >> use these even if I don't use NB).
>>>>> >> * NB has very few tests and examples and essentially no
>>>>> >> documentation.
>>>>> >> * NB has a simple conceptual model, but it is still not as simple as
>>>>> >> the FFI
>>>>> >> model.
>>>>> >> * NB uses variableByteClasses to great advantage -- this is very
>>>>> >> cool.
>>>>> >>
>>>>> >> * NB makes it easy to crash your image, since you are evaluating
>>>>> >> native code
>>>>> >> in image.
>>>>> >> * NB is still, to my eye, raw and untested on Mac.  I had problems
>>>>> >> with
>>>>> >> stack alignments in bizarre places, and NB's interaction with
>>>>> >> garbage
>>>>> >> collection basically ended my attempts to use it.
>>>>> >> * NB is basically impossible to debug if you aren't very strong with
>>>>> >> x86
>>>>> >> assembly (I'm not even strong with x86 assembly), and the stack
>>>>> >> manipulations rendered GDB pretty much useless for me.
>>>>> >> * NB does not integrate well with the Pharo tools -- I think the
>>>>> >> MethodTrailers are screwing up references, implementors, and source
>>>>> >> views,
>>>>> >> but I haven't investigated.
>>>>> >> * NB does not seem to support indirections very easily -- I
>>>>> >> struggled to
>>>>> >> understand how to interface to foreign C functions with specs like
>>>>> >> 'foo(struct x)', 'foo(struct x *)' and 'foo(struct x **)', and
>>>>> >> eventually
>>>>> >> gave up.
>>>>> >>
>>>>> >> I wanted to help Igor with the NB project, but the NB-OpenGL Mac
>>>>> >> bindings
>>>>> >> crash horribly for me, and although I was able to fix them, it I
>>>>> >> just don't
>>>>> >> hack enough x86 assembly to figure out all the tricks Igor is doing.
>>>>> >>  The
>>>>> >> thought of making it all work on 3 platforms, and fix all the tool
>>>>> >> integration, was just too much for me.
>>>>> >>
>>>>> >> In conclusion, I prefer the FFI.  It is old, cross platform, well
>>>>> >> tested,
>>>>> >> somewhat documented, and reliable.  I think Igor has done some
>>>>> >> amazing work
>>>>> >> and I do not want to bash his project (and certainly not his code --
>>>>> >> the
>>>>> >> man's a magician) but the fancy features are not necessary for my
>>>>> >> project.
>>>>> >>
>>>>> >>
>>>>> >> fstephany wrote
>>>>> >>>
>>>>> >>> Will this interface handle callbacks ?
>>>>> >>>
>>>>> >>
>>>>> >> I do not need callbacks so cannot speak to this issue.
>>>>> >>
>>>>> >> Yours,
>>>>> >> Nick Alexander
>>>>> >>
>>>>> >> --
>>>>> >> View this message in context:
>>>>> >> http://forum.world.st/Best-way-for-FFI-in-Pharo-tp4275467p4276356.html
>>>>> >> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Mariano
>>>> http://marianopeck.wordpress.com
>>>>
>>>
>>>
>>>
>>> --
>>> Mariano
>>> http://marianopeck.wordpress.com
>>>
>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com
>>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>



--
Best regards,
Igor Stasenko.