Usability of Squeak FFI for anything serious...

I'm trying to see how hard it is to port my HDF5 work from VW to Squeak.
It was not a pleasure in VW, because that already raised some questions despite that there is a somehow detailed DLLCC manual:

https://stackoverflow.com/questions/49544642/why-cant-i-pass-an-uninterpretedbytes-to-a-void-thru-dll-c-connect
https://stackoverflow.com/questions/49564024/isnt-pointer-type-checking-disabled-in-dll-c-connect-and-is-that-ok

But in Squeak, it sounds like hard++:

https://stackoverflow.com/questions/49782651/how-one-aligns-structure-fields-in-squeak-ffi
https://stackoverflow.com/questions/49783126/how-one-defines-a-union-type-in-squeak-ffi
https://stackoverflow.com/questions/49783443/how-one-defines-a-fixed-size-array-member-in-a-struct-in-squeak-ffi
https://stackoverflow.com/questions/49783882/how-one-deals-with-typedefs-in-squeak-ffi
https://stackoverflow.com/questions/49784253/how-one-deals-with-multiple-pointer-level-like-char-in-squeak-ffi
https://stackoverflow.com/questions/49784522/how-one-supports-both-32-and-64-bits-target-in-squeak-ffi

Either I'm blind, or the very light documentation that I found lacks those details
http://wiki.squeak.org/squeak/2426

I think that I could throw a few more questions (dealing with enums...) but I don't want to excede some unknown quota and start irritating a touchy SO moderator ;)

I'm afraid that there is no answer but "you're on your own"...

I'm open to studying Alien or UnifiedFFI of Pharo.

Or I can try and improve FFI.

At this stage, my trajectory has already been quite perturbed, and despite my knowledge of rocket science, I can see that I'm not going to put HDF5 on the intended orbit :(

Thoughts?

Hi Nicolas,

On Wed, Apr 11, 2018 at 2:23 PM, Nicolas Cellier <[hidden email]> wrote:

I'm trying to see how hard it is to port my HDF5 work from VW to Squeak.
It was not a pleasure in VW, because that already raised some questions despite that there is a somehow detailed DLLCC manual:

https://stackoverflow.com/questions/49544642/why-cant-i-pass-an-uninterpretedbytes-to-a-void-thru-dll-c-connect
https://stackoverflow.com/questions/49564024/isnt-pointer-type-checking-disabled-in-dll-c-connect-and-is-that-ok

What are you actually trying to do?  Can you post the C definitions?  I'm very familiar with both the SqueakFFI and DLLCC (as of 7.4.1) and the basic differences are to do with DLLCC's better cross-platform support, in that it auto-redefines typedefs on load.  But other than that I think the SqueakFFI is either just as good, or in some cases better (e.g. DLLCC has a bug in that it is the actual parameter that carries detailed type information for a struct, whereas in Andreas' SqueakFFI it is, correctly, the formal parameter).

 

But in Squeak, it sounds like hard++:

https://stackoverflow.com/questions/49782651/how-one-aligns-structure-fields-in-squeak-ffi
https://stackoverflow.com/questions/49783126/how-one-defines-a-union-type-in-squeak-ffi
https://stackoverflow.com/questions/49783443/how-one-defines-a-fixed-size-array-member-in-a-struct-in-squeak-ffi
https://stackoverflow.com/questions/49783882/how-one-deals-with-typedefs-in-squeak-ffi
https://stackoverflow.com/questions/49784253/how-one-deals-with-multiple-pointer-level-like-char-in-squeak-ffi
https://stackoverflow.com/questions/49784522/how-one-supports-both-32-and-64-bits-target-in-squeak-ffi

Either I'm blind, or the very light documentation that I found lacks those details
http://wiki.squeak.org/squeak/2426

I think that I could throw a few more questions (dealing with enums...) but I don't want to excede some unknown quota and start irritating a touchy SO moderator ;)

I'm afraid that there is no answer but "you're on your own"...

I'm open to studying Alien or UnifiedFFI of Pharo.

Or I can try and improve FFI.

At this stage, my trajectory has already been quite perturbed, and despite my knowledge of rocket science, I can see that I'm not going to put HDF5 on the intended orbit :(

Thoughts?

--

_,,,^..^,,,_

best, Eliot

2018-04-12 3:10 GMT+02:00 Eliot Miranda <[hidden email]>:

Hi Nicolas,

On Wed, Apr 11, 2018 at 2:23 PM, Nicolas Cellier <[hidden email]> wrote:

I'm trying to see how hard it is to port my HDF5 work from VW to Squeak.
It was not a pleasure in VW, because that already raised some questions despite that there is a somehow detailed DLLCC manual:

https://stackoverflow.com/questions/49544642/why-cant-i-pass-an-uninterpretedbytes-to-a-void-thru-dll-c-connect
https://stackoverflow.com/questions/49564024/isnt-pointer-type-checking-disabled-in-dll-c-connect-and-is-that-ok

What are you actually trying to do?  Can you post the C definitions?  I'm very familiar with both the SqueakFFI and DLLCC (as of 7.4.1) and the basic differences are to do with DLLCC's better cross-platform support, in that it auto-redefines typedefs on load.  But other than that I think the SqueakFFI is either just as good, or in some cases better (e.g. DLLCC has a bug in that it is the actual parameter that carries detailed type information for a struct, whereas in Andreas' SqueakFFI it is, correctly, the formal parameter).

Hi Eliot,

the library is huge (HDF5).

All the interfacing code is available at cincom public store (HDF5 bundle)

To get an idea, you can inspect a clone of HDF5 library at https://github.com/live-clones/hdf5/tree/develop/src

The main structures/enum/typedefs/prototypes are in the H5*public.h files

The problems are all exposed in SO questions, but I can try and priotitize.

The essential barrier is that those headers use almost exclusively typedef'ed types.

Library is too huge for manual subsitution; this must be mechanized.

I would prefer some kind of support in FFI rather than having to hack own fragile ad-hoc pre-processor.

Then I have a problem because of alignment of struct fields which is compact in Squeak FFI rather than conforming to platform conventions.

Same here, manually hacking with field size is a no go, this must be mechanized.

Then some unions are used too and there is no support at all in FFI.

Then definitions like size_t are dependent on machine word size (32 or 64 bits), and there is no support for that in FFI.
I'll have to duplicate some work for supporting both 32 and 64 bits Squeak, or explore new hacks...

But there is no support and no documentation (how to).

Then while digging in source coden I did not see any double pointer support in struct fields.

The example I provide on SO shows what I would qualify as a bug.

I don't know yet whether I'll need those, but it's not good.

Maybe I can shoot each problem with more or less ugly workarounds.
At the end i can declare void * and deal with offset directly, but that's both:

- a lot of work

- a good recipe for generating very hard to find bugs - crashes if lucky, silent corruption if not

 (dealing with offsets is like writing in assembler instead of C)

Given the size of the library, the probability of not creating such fault in manual translation is 0%

Plus, there are a few evolutions from one version of the library to the other...

That's my definition of hard++

 

But in Squeak, it sounds like hard++:

https://stackoverflow.com/questions/49782651/how-one-aligns-structure-fields-in-squeak-ffi
https://stackoverflow.com/questions/49783126/how-one-defines-a-union-type-in-squeak-ffi
https://stackoverflow.com/questions/49783443/how-one-defines-a-fixed-size-array-member-in-a-struct-in-squeak-ffi
https://stackoverflow.com/questions/49783882/how-one-deals-with-typedefs-in-squeak-ffi
https://stackoverflow.com/questions/49784253/how-one-deals-with-multiple-pointer-level-like-char-in-squeak-ffi
https://stackoverflow.com/questions/49784522/how-one-supports-both-32-and-64-bits-target-in-squeak-ffi

Either I'm blind, or the very light documentation that I found lacks those details
http://wiki.squeak.org/squeak/2426

I think that I could throw a few more questions (dealing with enums...) but I don't want to excede some unknown quota and start irritating a touchy SO moderator ;)

I'm afraid that there is no answer but "you're on your own"...

I'm open to studying Alien or UnifiedFFI of Pharo.

Or I can try and improve FFI.

At this stage, my trajectory has already been quite perturbed, and despite my knowledge of rocket science, I can see that I'm not going to put HDF5 on the intended orbit :(

Thoughts?

--

_,,,^..^,,,_

best, Eliot

On 12 April 2018 at 16:00, Nicolas Cellier <[hidden email]> wrote:

2018-04-12 3:10 GMT+02:00 Eliot Miranda <[hidden email]>:

Hi Nicolas,

On Wed, Apr 11, 2018 at 2:23 PM, Nicolas Cellier <[hidden email]> wrote:

I'm trying to see how hard it is to port my HDF5 work from VW to Squeak.
It was not a pleasure in VW, because that already raised some questions despite that there is a somehow detailed DLLCC manual:

https://stackoverflow.com/questions/49544642/why-cant-i-pass-an-uninterpretedbytes-to-a-void-thru-dll-c-connect
https://stackoverflow.com/questions/49564024/isnt-pointer-type-checking-disabled-in-dll-c-connect-and-is-that-ok

What are you actually trying to do?  Can you post the C definitions?  I'm very familiar with both the SqueakFFI and DLLCC (as of 7.4.1) and the basic differences are to do with DLLCC's better cross-platform support, in that it auto-redefines typedefs on load.  But other than that I think the SqueakFFI is either just as good, or in some cases better (e.g. DLLCC has a bug in that it is the actual parameter that carries detailed type information for a struct, whereas in Andreas' SqueakFFI it is, correctly, the formal parameter).

Hi Eliot,

the library is huge (HDF5).

All the interfacing code is available at cincom public store (HDF5 bundle)

To get an idea, you can inspect a clone of HDF5 library at https://github.com/live-clones/hdf5/tree/develop/src

The main structures/enum/typedefs/prototypes are in the H5*public.h files

The problems are all exposed in SO questions, but I can try and priotitize.

The essential barrier is that those headers use almost exclusively typedef'ed types.

Library is too huge for manual subsitution; this must be mechanized.

I would prefer some kind of support in FFI rather than having to hack own fragile ad-hoc pre-processor.

Maybe more of a distraction than a help, but browsing around because its an interesting topic, 

some discoveries...

* https://eli.thegreenplace.net/2012/12/17/dumping-a-c-objects-memory-layout-with-clang

* https://github.com/rpav/c2ffi

* http://blog.llvm.org/2018/03/dragonffi-ffijit-for-c-language-using.html

* https://github.com/marianopeck/FFICHeaderExtractor

cheers -ben

 

Then I have a problem because of alignment of struct fields which is compact in Squeak FFI rather than conforming to platform conventions.

Same here, manually hacking with field size is a no go, this must be mechanized.

Then some unions are used too and there is no support at all in FFI.

Then definitions like size_t are dependent on machine word size (32 or 64 bits), and there is no support for that in FFI.
I'll have to duplicate some work for supporting both 32 and 64 bits Squeak, or explore new hacks...

But there is no support and no documentation (how to).

Then while digging in source coden I did not see any double pointer support in struct fields.

The example I provide on SO shows what I would qualify as a bug.

I don't know yet whether I'll need those, but it's not good.

Maybe I can shoot each problem with more or less ugly workarounds.
At the end i can declare void * and deal with offset directly, but that's both:

- a lot of work

- a good recipe for generating very hard to find bugs - crashes if lucky, silent corruption if not

 (dealing with offsets is like writing in assembler instead of C)

Given the size of the library, the probability of not creating such fault in manual translation is 0%

Plus, there are a few evolutions from one version of the library to the other...

That's my definition of hard++

 

But in Squeak, it sounds like hard++:

https://stackoverflow.com/questions/49782651/how-one-aligns-structure-fields-in-squeak-ffi
https://stackoverflow.com/questions/49783126/how-one-defines-a-union-type-in-squeak-ffi
https://stackoverflow.com/questions/49783443/how-one-defines-a-fixed-size-array-member-in-a-struct-in-squeak-ffi
https://stackoverflow.com/questions/49783882/how-one-deals-with-typedefs-in-squeak-ffi
https://stackoverflow.com/questions/49784253/how-one-deals-with-multiple-pointer-level-like-char-in-squeak-ffi
https://stackoverflow.com/questions/49784522/how-one-supports-both-32-and-64-bits-target-in-squeak-ffi

Either I'm blind, or the very light documentation that I found lacks those details
http://wiki.squeak.org/squeak/2426

I think that I could throw a few more questions (dealing with enums...) but I don't want to excede some unknown quota and start irritating a touchy SO moderator ;)

I'm afraid that there is no answer but "you're on your own"...

I'm open to studying Alien or UnifiedFFI of Pharo.

Or I can try and improve FFI.

At this stage, my trajectory has already been quite perturbed, and despite my knowledge of rocket science, I can see that I'm not going to put HDF5 on the intended orbit :(

Thoughts?

--

_,,,^..^,,,_

best, Eliot

Nicolas,

I've had to deal with a variety of libraries (OpenGL, databases, etc) and have yet to run into a situation where I can't make things work with vanilla FFI (with one caveat: I haven't needed callback support which has been discussed several times here and on vmdev and solutions exists, I just haven't needed them to date.)  But this has required writing a variety of preprocessors (for static APIs) or bridges (for dynamic) as I'm not aware of any tooling to universally deal with headers/bridging (FFI doesn't help you with header files etc. so you will have to deal with things like unions in your preprocessor). 32/64-bit is pretty easily dealt with provided you check Smalltalk>>wordSize and adjust as needed (i.e. it can be dealt with statically or dynamically depending on your needs.)  The way to think of FFI is as just providing the lowest level plumbing materials that allows the image to communicate with the outside world.  (i.e. pipes, elbows, etc.. but you have to assemble it and supply the liquid)  I'm not saying that this is ideal... it would be great if one could point to an arbitrary API and say 'FFI, figure it out' but thats not what exists.  What we have currently does the job but does require some work.

Just my $0.02,

Phil

On Thu, Apr 12, 2018, 4:00 AM Nicolas Cellier <[hidden email]> wrote:

2018-04-12 3:10 GMT+02:00 Eliot Miranda <[hidden email]>:

Hi Nicolas,

On Wed, Apr 11, 2018 at 2:23 PM, Nicolas Cellier <[hidden email]> wrote:

I'm trying to see how hard it is to port my HDF5 work from VW to Squeak.
It was not a pleasure in VW, because that already raised some questions despite that there is a somehow detailed DLLCC manual:

https://stackoverflow.com/questions/49544642/why-cant-i-pass-an-uninterpretedbytes-to-a-void-thru-dll-c-connect
https://stackoverflow.com/questions/49564024/isnt-pointer-type-checking-disabled-in-dll-c-connect-and-is-that-ok

What are you actually trying to do?  Can you post the C definitions?  I'm very familiar with both the SqueakFFI and DLLCC (as of 7.4.1) and the basic differences are to do with DLLCC's better cross-platform support, in that it auto-redefines typedefs on load.  But other than that I think the SqueakFFI is either just as good, or in some cases better (e.g. DLLCC has a bug in that it is the actual parameter that carries detailed type information for a struct, whereas in Andreas' SqueakFFI it is, correctly, the formal parameter).

Hi Eliot,

the library is huge (HDF5).

All the interfacing code is available at cincom public store (HDF5 bundle)

To get an idea, you can inspect a clone of HDF5 library at https://github.com/live-clones/hdf5/tree/develop/src

The main structures/enum/typedefs/prototypes are in the H5*public.h files

The problems are all exposed in SO questions, but I can try and priotitize.

The essential barrier is that those headers use almost exclusively typedef'ed types.

Library is too huge for manual subsitution; this must be mechanized.

I would prefer some kind of support in FFI rather than having to hack own fragile ad-hoc pre-processor.

Then I have a problem because of alignment of struct fields which is compact in Squeak FFI rather than conforming to platform conventions.

Same here, manually hacking with field size is a no go, this must be mechanized.

Then some unions are used too and there is no support at all in FFI.

Then definitions like size_t are dependent on machine word size (32 or 64 bits), and there is no support for that in FFI.
I'll have to duplicate some work for supporting both 32 and 64 bits Squeak, or explore new hacks...

But there is no support and no documentation (how to).

Then while digging in source coden I did not see any double pointer support in struct fields.

The example I provide on SO shows what I would qualify as a bug.

I don't know yet whether I'll need those, but it's not good.

Maybe I can shoot each problem with more or less ugly workarounds.
At the end i can declare void * and deal with offset directly, but that's both:

- a lot of work

- a good recipe for generating very hard to find bugs - crashes if lucky, silent corruption if not

 (dealing with offsets is like writing in assembler instead of C)

Given the size of the library, the probability of not creating such fault in manual translation is 0%

Plus, there are a few evolutions from one version of the library to the other...

That's my definition of hard++

 

But in Squeak, it sounds like hard++:

https://stackoverflow.com/questions/49782651/how-one-aligns-structure-fields-in-squeak-ffi
https://stackoverflow.com/questions/49783126/how-one-defines-a-union-type-in-squeak-ffi
https://stackoverflow.com/questions/49783443/how-one-defines-a-fixed-size-array-member-in-a-struct-in-squeak-ffi
https://stackoverflow.com/questions/49783882/how-one-deals-with-typedefs-in-squeak-ffi
https://stackoverflow.com/questions/49784253/how-one-deals-with-multiple-pointer-level-like-char-in-squeak-ffi
https://stackoverflow.com/questions/49784522/how-one-supports-both-32-and-64-bits-target-in-squeak-ffi

Either I'm blind, or the very light documentation that I found lacks those details
http://wiki.squeak.org/squeak/2426

I think that I could throw a few more questions (dealing with enums...) but I don't want to excede some unknown quota and start irritating a touchy SO moderator ;)

I'm afraid that there is no answer but "you're on your own"...

I'm open to studying Alien or UnifiedFFI of Pharo.

Or I can try and improve FFI.

At this stage, my trajectory has already been quite perturbed, and despite my knowledge of rocket science, I can see that I'm not going to put HDF5 on the intended orbit :(

Thoughts?

--

_,,,^..^,,,_

best, Eliot

There are at least two problems:

- documentation (lack of)

- limitations (unions, pointer on pointer on ...)

- automation (I said at least two, not at most)

For the first problem, I opened the SO issues. Not ideal, but it's a start.

For the second, I'll see if I can do something...

For the third, I did import some .h header with VW DLLCC, and I'm in the process of translating to other dialects.
It's easier because Smalltalk syntax is order of magnitudes smaller than C syntax.

Hi Nicolas,

On Apr 13, 2018, at 7:01 AM, Nicolas Cellier <[hidden email]> wrote:

So I started to improve the Squeak FFI:

-1) I corrected pointer arithmetic on 64 bits http://source.squeak.org/FFI/FFI-Kernel-nice.48.diff

-2) I prepared the possibility of having automated layout spec updates http://source.squeak.org/FFI/FFI-Kernel-nice.49.diff

Note that the FFI plugin by itself did not change.

The next steps should be to provide support for native platform alignment of fields and unions.

I'll focus on these when i can.

I shall join you as soon as I am able.  Right now I have a pressing task, but within a few weeks I should be able to help.

2018-04-12 23:27 GMT+02:00 Denis Kudriashov <[hidden email]>:

Hi

There is project TalkFFI from Ciprian Teodorov.

2018-04-12 22:50 GMT+02:00 Bert Freudenberg <[hidden email]>:

On 12 April 2018 at 22:34, Nicolas Cellier <[hidden email]> wrote:

There are at least two problems:

- documentation (lack of)

- limitations (unions, pointer on pointer on ...)

- automation (I said at least two, not at most)

For the first problem, I opened the SO issues. Not ideal, but it's a start.

For the second, I'll see if I can do something...

For the third, I did import some .h header with VW DLLCC, and I'm in the process of translating to other dialects.
It's easier because Smalltalk syntax is order of magnitudes smaller than C syntax.

​I always thought it should be fairly straightforward to make a FFI bindings generator using SWIG (http://www.swig.org/).

SWIG parses C headers and outputs XML. Starting from that XML should be way easier than trying to parse C ourselves.

​- Bert -​