FFI beginner question

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

FFI beginner question

Richard O'Keefe
I am developing a Smalltalk interface to an existing C library which I
intend to make generally available.  Naturally I am doing this in my own
Smalltalk system first, where it's unsurprisingly easy for me.  But when
I have it working, I'd like to make a Pharo port available.

I have never used the Foreign Function Interface in Pharo before and
don't even know where to start.

What should I read first?
Is there a model project to imitate?

Reply | Threaded
Open this post in threaded view
|

Re: FFI beginner question

Brainstorms
I'm also interested in this...  

Any plans to draft a Pharo booklet on this subject?

-Ted




--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Reply | Threaded
Open this post in threaded view
|

Re: FFI beginner question

eftomi
Hi,


Best wishes,
Tomaz


 
Any plans to draft a Pharo booklet on this subject?
 
-Ted

Reply | Threaded
Open this post in threaded view
|

Re: FFI beginner question

Richard O'Keefe
Thank you for that.  I note that some sections have nothing in them.
Now that I have
read that, what is the next thing to read?  In particular, how do I
connect C numeric
types introduced by the library to FFI?

On Mon, 23 Sep 2019 at 16:45, Tomaž Turk <[hidden email]> wrote:

>
> Hi,
>
> The book draft is here: https://files.pharo.org/books-pdfs/booklet-uFFI/UFFIDRAFT.pdf.
>
> Best wishes,
> Tomaz
>
>
>
> Any plans to draft a Pharo booklet on this subject?
>
> -Ted
>

Reply | Threaded
Open this post in threaded view
|

Re: FFI beginner question

Guillermo Polito
Just more detail into it:

 - the source code of the booklet, written in Pillar, resides in here:

 - I’ve been in the last weeks doing a pass on it (see branch version2 https://github.com/SquareBracketAssociates/Booklet-uFFI/tree/version2)
   I started adding examples, and enhancing the explanations.
   I got particularly blocked when explaining marshalling where I saw several issues to address / features to add before continuing documenting

I have some issues written down to fix in FFI soon:
  1 - unify vocabulary: module and library in the API (see #ffiCall:module: #macModuleName #ffiLibraryName, #ffiLibrary…)
  2 - some of the names above are misleading (#ffiLibraryName returning a library object and not a name for example)
  3 - extend literal object support to floats
  4 - add a strict (but safe) mode where types for literals are mandatory (otherwise marshalling can go wrong :))

I’ll create issues for these in the next hours ^^.

El 23 sept 2019, a las 13:38, Richard O'Keefe <[hidden email]> escribió:

how do I
connect C numeric
types introduced by the library to FFI?

Hi Richard, what do you mean? Do you have an example? What’s the signature of your C function look like?

Guille
Reply | Threaded
Open this post in threaded view
|

Re: FFI beginner question

Guillermo Polito
See


Of course, if somebody wants to help, you’re welcome to :)

Guille

El 23 sept 2019, a las 13:59, Guillermo Polito <[hidden email]> escribió:

Just more detail into it:

 - the source code of the booklet, written in Pillar, resides in here:

 - I’ve been in the last weeks doing a pass on it (see branch version2 https://github.com/SquareBracketAssociates/Booklet-uFFI/tree/version2)
   I started adding examples, and enhancing the explanations.
   I got particularly blocked when explaining marshalling where I saw several issues to address / features to add before continuing documenting

I have some issues written down to fix in FFI soon:
  1 - unify vocabulary: module and library in the API (see #ffiCall:module: #macModuleName #ffiLibraryName, #ffiLibrary…)
  2 - some of the names above are misleading (#ffiLibraryName returning a library object and not a name for example)
  3 - extend literal object support to floats
  4 - add a strict (but safe) mode where types for literals are mandatory (otherwise marshalling can go wrong :))

I’ll create issues for these in the next hours ^^.

El 23 sept 2019, a las 13:38, Richard O'Keefe <[hidden email]> escribió:

how do I
connect C numeric
types introduced by the library to FFI?

Hi Richard, what do you mean? Do you have an example? What’s the signature of your C function look like?

Guille

Reply | Threaded
Open this post in threaded view
|

Re: FFI beginner question

Brainstorms
Guillermo,

I'm interested in helping, but at this point, I think I'd be most helpful
working at improving documentation (mainly editing) rather than working on
Pharo code itself.  (I'd like to work toward that, though.)  

I'm still a newbie with Pharo, but I am a good writer/editor.  And I expect
that working with Pharo documentation would be another means of increasing
my knowledge of the Pharo ecosystem -- so that's additional incentive for
me.

I gather that the PDF books are written using Pillar, which I know nothing
about.  Are there resources & guides for this tool/format that would help me
learn how to make & edit these kinds of documents?

Also, I've never contributed to an open source project; Pharo seems to be a
good place to start doing so.  I see that most of the documentation, web
pages, booklets, etc. are in English so there's the advantage that English
is my first language (and I actually paid attention in school  :^).  I'm
also aware, from experience, that Documentation is rarely the first choice
for developers to apply their time & enthusiasm...

-Ted




--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Reply | Threaded
Open this post in threaded view
|

Re: FFI beginner question

Richard O'Keefe
In reply to this post by Guillermo Polito
What I want to do is to create a binding for the SoftPosit library.
A typical function has an interface like
        quire16_t q16_fdp_add(quire16_t, posit16_t, posit16_t);
where
        typedef struct { uint16_t v; } posit16_t;
        typedef struct { uint64_t v[2]; } quire16_t;
A "quire" is a dot product accumulator.
A "posit" is a number.
{quire,posit}{8,16,32} are defined, and a number of other types.

Reply | Threaded
Open this post in threaded view
|

Pumping FFI documentation [WAS] FFI beginner question

Guillermo Polito
In reply to this post by Brainstorms
Hi Ted,

I split this in a separate thread to avoid noise :)

El 23 sept 2019, a las 23:14, Brainstorms <[hidden email]> escribió:

Guillermo,

I'm interested in helping, but at this point, I think I'd be most helpful
working at improving documentation (mainly editing) rather than working on
Pharo code itself.  (I'd like to work toward that, though.)  

I’ve been doing a pass on the structure, and I was thinking on a rough structure as follows:
 1) Intro to FFI (callouts, function and library lookup, intro to value marshalling)
 2) Marshalling (sending arguments, literal arguments, more on marshalling, basic C types: ints, floats, pointers and how they are transformed to pharo objects and vice-versa…)
 3) Complex types: strings, unions, arrays, opaque types
 4) Derived types on the Pharo side: How to design nice classes with all this
 5) Callbacks
 6) Memory management

I did already a pass on 1), and I got blocked in 2), though I want to release a version of it this week.

If you’re up for it, there are several things we can do:
 - review the english :)
 - give feedback on what is missing, what is not understandable, what can be explained better
 - testing the examples?


I'm still a newbie with Pharo, but I am a good writer/editor.  And I expect
that working with Pharo documentation would be another means of increasing
my knowledge of the Pharo ecosystem -- so that's additional incentive for
me.

Cool :)

I gather that the PDF books are written using Pillar, which I know nothing
about.  Are there resources & guides for this tool/format that would help me
learn how to make & edit these kinds of documents?

Pillar is a markup syntax (from Pier’s CMS, if you know it).

Pillar comes with a document model, parser and generators to html, pdf (through latex), and others…
In Pillar’s readme there are the installation instructions + usage.

If you check the travis file in the ffi booklet repository


You’ll see it is built with pillar 7.4.1. In other words

# install pillar
$ cd pillar && ./scripts/build.sh && cd ..

# go into the booklet repository and build the pdf
$ ./pillar/build/pillar build pdf

Although you’ll need a mostly up-to-date latex version (latexmk required, plus several other packages, check Pillar’s readme)

Also, I've never contributed to an open source project; Pharo seems to be a
good place to start doing so.  I see that most of the documentation, web
pages, booklets, etc. are in English so there's the advantage that English
is my first language (and I actually paid attention in school  :^).  I'm
also aware, from experience, that Documentation is rarely the first choice
for developers to apply their time & enthusiasm…

And it’s super important nevertheless ^^.

Guille
Reply | Threaded
Open this post in threaded view
|

Re: FFI beginner question

Guillermo Polito
In reply to this post by Richard O'Keefe

> El 24 sept 2019, a las 4:35, Richard O'Keefe <[hidden email]> escribió:
>
> What I want to do is to create a binding for the SoftPosit library.
> A typical function has an interface like
>        quire16_t q16_fdp_add(quire16_t, posit16_t, posit16_t);
> where
>        typedef struct { uint16_t v; } posit16_t;
>        typedef struct { uint64_t v[2]; } quire16_t;
> A "quire" is a dot product accumulator.
> A "posit" is a number.
> {quire,posit}{8,16,32} are defined, and a number of other types.

Ok, so IIUC, what you need is:

1) define type aliases (e.g., typedef posit16_t uint_t)
2) define a couple of structs based on those types

I see in the booklet no documentation about type aliases so far, but in the latest pharo version there are a couple of examples you can look at.
Type aliases can be defined as class variables in the class using the type, or in a shared pool, if you mean to use those types in several places.
Take a look at FT2Types class side for an example, where the type FT_Error is defined

FT2Types class>>initialize
        ...
        FT_Error := 'long’.
        …

and then used in a callout like this

FT2Face >> ffiGetKerningLeft: left_glyph right: right_glyph result: akerning
        self ffiCall: #(FT_Error FT_Get_Kerning(self, FT_UInt left_glyph, FT_UInt right_glyph, 2, FT_Vector  *akerning ))

Then, to use those defined types in a struct definition, you only need to make those type aliases (class vars) available in your struct, and use them in the struct field definition
See for example FTMatrix

FTMatrix class >> fieldsDesc
        ^ #(    
                FT_Fixed   xx
                FT_Fixed xy
                FT_Fixed   yx
                FT_Fixed yy
        )

Hope this helps,
Guille
Reply | Threaded
Open this post in threaded view
|

Re: Pumping FFI documentation [WAS] FFI beginner question

eftomi
In reply to this post by Guillermo Polito
Hi Guille,

Thanks for your efforts, we really appreciate it!

I was intrigued by Stephan and tried to prepare something for the book, like
a subtitle/small chapter "Exotic types", but then I noticed that you already
put some additional content to the book on GitHub since the pdf at pharo.org
was created last year - the text that cleared some of my challenges.

I went through UFFI complexities over and over with the debugger, so I have
relatively good picture of what is going on when a call is "converted" to
bytecode, however I'm constantly finding new pearls inside. UFFI is really
great and heavy package!

Anyways, I don't think that I can add anything substantial to the content of
the book, however I can help with the feedback and suggestions, I can also
test the examples if you need.

As for the "missing pieces & understandable", I will fire right away :-)
Most of my FFI experiences are coming from fiddling with the "mysterious"
Win32WideString implementation or wchar_t*, char16_t*, char32_t* … family of
wide strings on Windows. It seems to me that Win32WideString class is only
half made, or maybe it was made before UFFI -> I'm just guessing, however
it's a subclass of ExternalObject and not the subclass of FFIExternalObject.
When you pass it through UFFI as an argument, everything is OK, however when
it is returned from a function call it is not automagically treated as a
proper Win32WideString - you have to "manually" convert the data at the
ExternalAddress. And now my suggestions on the book :-)

- I hardly found (on the net, Pharo books ...) any explanation on
ExternalObject's handle, which is quite often used in FFIExternalObjects,
too; what's the difference between a handle and a pointer, and what exactly
the various implementations of #fromHandle: are supposed to do. #fromHandle
is now mentioned in the book, based on  this discussion
<http://forum.world.st/UFFI-correct-way-to-cast-void-td5067008.html>   about
casting, however if you are dealing with an "exotic type" then it would be
nice to have some guidelines about the purpuse of it. To illustrate, in
Win32WideString the handle is implemented as a ByteArray with the data
itself, in other objects that I know of it's normally an (external)
address/pointer, huh?

- When the UFFI call ends, and the return argument's value is put onto the
stack right after the #send:invokeWithArguments:, what happens with that
value - is there anything that can be done with it - can the "casting" be
performed "automatically"? Of course one can always prepare a "wrapper" to
do that, however … I found FFIExternalType>>#emitReturnArgument:context:
where "loader" gets onto the scene, but this leads to heavy lifting with
bytecodes, is I understand :-)

- In relation to that, there is a bit vague explanation in the book about
objects under the title External Objects ("... But often, frameworks will
allocate structures, pointers, etc. which actually represents ”an object”
(not in the same sense as a Smalltalk object, but can be interpreted like
one) ..."). This "interpretation" is a kind of mysterious :-) A couple of
suggestions about possible directions would be nice to have.

- As far as I can see, the FFIExternalValueHolder is not mentioned in the
book yet. It is nicely described in its comment, however the missing
information is that the type (the resulting subclass of
FFIExternalValueHolder) should be defined as a class variable - without that
the signature parser cannot find it. The purpose of FFIExternalValueHolder
is similar to FFIExternalType class>>#newBuffer.

I hope you find these suggestions helpful.

Best wishes,
Tomaz



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Reply | Threaded
Open this post in threaded view
|

Re: Pumping FFI documentation [WAS] FFI beginner question

Brainstorms
In reply to this post by Guillermo Polito
Hi Guillermo,

I obviously gave the impression that I was focused on the UFFI
code/documentation, but actually I'm up for helping in this way for any/all
elements of Pharo, not just this one.

Reviewing & editing (for grammar, clarification, spelling & punctuation, et
al.) is definitely something I can help with.  That would apply to PDF
booklets, code comments, web pages, etc.  The English in Pharo is generally
very good, yet there are places here & there that could use.. 'polishing',
mostly to make Pharo's English prose look as professional as the tool it
represents, of course.  :^)

So the question is (given that you have a fairly high & wide view of
things), what elements could best use my talents & efforts?  If you want to
start with UFFI, that's fine with me.  It *is* a subject of interest for me,
and I have a long-term project idea that will require a good understanding
of it.  (I automate instrumentation for data acquisition & process control;
we started using Lua as a scripting language to allow our end-users to
script the operation of compiled apps we produce.  There's a potential to
also use Pharo to do this.  I think Lua's well-designed FFI will make
"swapping out" Lua for Pharo achievable.)

I'll dig into Pillar/Pier, install everything, and get the process
working...  Thanks for the detailed how-to for setting these up!  It will
likely be next week at the earliest that I'll have that in place.

I'll also study the latest rev of the UFFI booklet to pick up the details of
how it works in Pharo, and run through the examples.  I'll make notes of my
impressions & questions for feedback "as a newcomer" (that won't be hard!)

-Ted



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Reply | Threaded
Open this post in threaded view
|

Re: Pumping FFI documentation [WAS] FFI beginner question

Ben Coman
On Wed, 25 Sep 2019 at 08:35, Brainstorms <[hidden email]> wrote:
I automate instrumentation for data acquisition & process control;
we started using Lua as a scripting language to allow our end-users to
script the operation of compiled apps we produce.  There's a potential to
also use Pharo to do this.  I think Lua's well-designed FFI will make
"swapping out" Lua for Pharo achievable.

Cool!  I'd be interested to track what you do in this space.
(I'm an electrical engineer and always thought Pharo would make a great SCADA system.

Btw, I haven't managed to bubble playing-with-this to the top of my queue, but it may interest you... 

cheers -ben