UnifiedFFI Docs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

UnifiedFFI Docs

tblanchard
Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

Stephan Eggermont-3
Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

tblanchard
Wonderful!  Thanks.

> On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:
>
> On 17-10-17 23:06, Todd Blanchard wrote:
>> Anyone know what happened to this?
>> https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf
>
> https://github.com/SquareBracketAssociates/Booklet-uFFI
>
> has a link to a bintray pdf download
>
> Stephan
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

Stephane Ducasse-3
Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]> wrote:

> Wonderful!  Thanks.
>
>> On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:
>>
>> On 17-10-17 23:06, Todd Blanchard wrote:
>>> Anyone know what happened to this?
>>> https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf
>>
>> https://github.com/SquareBracketAssociates/Booklet-uFFI
>>
>> has a link to a bintray pdf download
>>
>> Stephan
>>
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

kilon.alios
Sure the documentation could be better, that is definetly important, but is already good enough and UFFI is a very technical subject much more suited to a mailing list . Its not physical possible to cover the massive potential of UFFI.  

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]> wrote:
Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]> wrote:
> Wonderful!  Thanks.
>
>> On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:
>>
>> On 17-10-17 23:06, Todd Blanchard wrote:
>>> Anyone know what happened to this?
>>> https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf
>>
>> https://github.com/SquareBracketAssociates/Booklet-uFFI
>>
>> has a link to a bintray pdf download
>>
>> Stephan
>>
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

tblanchard
I'm working through Ben's great blog post about playing with libclang and I am puzzled by something.

invalidateSessionData
   handle atAllPut: 0. 

zero;s out the handle.  Cool.  However, 

handle isNull 

does not return true despite it being used in the getString method as 

^ handle isNull
         ifTrue: ['external memory invalidated by session restart']
         ifFalse:[LibClang clang_getCString__cxString: self].

Looks like there should be an isNull on ByteArray that returns true if all bytes are zero but it isn't  there.  Was it dropped for some reason?



On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]> wrote:

Sure the documentation could be better, that is definetly important, but is already good enough and UFFI is a very technical subject much more suited to a mailing list . Its not physical possible to cover the massive potential of UFFI.  

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]> wrote:
Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]> wrote:
> Wonderful!  Thanks.
>
>> On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:
>>
>> On 17-10-17 23:06, Todd Blanchard wrote:
>>> Anyone know what happened to this?
>>> https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf
>>
>> https://github.com/SquareBracketAssociates/Booklet-uFFI
>>
>> has a link to a bintray pdf download
>>
>> Stephan
>>
>>
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

Nicolas Cellier


2017-10-18 19:05 GMT+02:00 Todd Blanchard <[hidden email]>:
I'm working through Ben's great blog post about playing with libclang and I am puzzled by something.

invalidateSessionData
   handle atAllPut: 0. 

zero;s out the handle.  Cool.  However, 

handle isNull 

does not return true despite it being used in the getString method as 

^ handle isNull
         ifTrue: ['external memory invalidated by session restart']
         ifFalse:[LibClang clang_getCString__cxString: self].

Looks like there should be an isNull on ByteArray that returns true if all bytes are zero but it isn't  there.  Was it dropped for some reason?



In original FFI, a ByteArray could not represent a pointer, so it answered:
ByteArray>>isNull
    "Answer false since only external addresses can be null"
    ^false

There was an ExternalAddress subclass for this purpose where isNull was defined:
ExternalAddress>>isNull
    "Answer true if I am a null pointer"
    1 to: self size do:[:i| (self at: i) = 0 ifFalse:[^false]].
    ^true

and there was also a #beNull to nullify the address...


What happens now with UnifiedFFI, is the old FFI API supported (and thus the example correct), what about Alien, are questions that you must inquire, but once upon a time, it worked.

Should isNull be defined as you suggest in ByteArray? If the semantic is #isNullPointer, then I don't think so. If it's a more general message, then maybe...

What is the class of handle?



On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]> wrote:

Sure the documentation could be better, that is definetly important, but is already good enough and UFFI is a very technical subject much more suited to a mailing list . Its not physical possible to cover the massive potential of UFFI.  

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]> wrote:
Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]> wrote:
> Wonderful!  Thanks.
>
>> On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:
>>
>> On 17-10-17 23:06, Todd Blanchard wrote:
>>> Anyone know what happened to this?
>>> https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf
>>
>> https://github.com/SquareBracketAssociates/Booklet-uFFI
>>
>> has a link to a bintray pdf download
>>
>> Stephan
>>
>>
>>
>
>



Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

tblanchard

On Oct 18, 2017, at 1:21 PM, Nicolas Cellier <[hidden email]> wrote:

<snip>

Should isNull be defined as you suggest in ByteArray? If the semantic is #isNullPointer, then I don't think so. If it's a more general message, then maybe...

What is the class of handle?

It is ByteArray.  I agree that ByteArray responding to isNull is weird.

The struct the memory maps is struct { void* data, int flags } so I can work around this with something like 

self data ifNil: [.....]

I have just encountered another problem.

Ben creates a type specification as a class variable to model void*[3]

VoidPointer3 := FFITypeArray ofType: 'void*' size: 3.

which produces the following accessor for data

(FFITypeArray ofType: #FFIVoid size: 3) fromHandle: (handle copyFrom: OFFSET_DATA to: OFFSET_DATA + 23)

This fails because the native size of void is undefined even though it is a void* being mapped.

I will continue to see if I can figure out why it generates this spec - definitely a broken assumption somewhere in here.

-Todd Blanchard

Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

Ben Coman
In reply to this post by tblanchard


On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:
I'm working through Ben's great blog post about playing with libclang and I am puzzled by something.

Thx Todd. Knowing someone is looking at it encourages me to expand it.  
I'm interested in looking at your specific questions, but it might not be until next week after this stint of long work days.

cheers -ben

 

invalidateSessionData
   handle atAllPut: 0. 

zero;s out the handle.  Cool.  However, 

handle isNull 

does not return true despite it being used in the getString method as 

^ handle isNull
         ifTrue: ['external memory invalidated by session restart']
         ifFalse:[LibClang clang_getCString__cxString: self].

Looks like there should be an isNull on ByteArray that returns true if all bytes are zero but it isn't  there.  Was it dropped for some reason?



On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]> wrote:

Sure the documentation could be better, that is definetly important, but is already good enough and UFFI is a very technical subject much more suited to a mailing list . Its not physical possible to cover the massive potential of UFFI.  

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]> wrote:
Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]> wrote:
> Wonderful!  Thanks.
>
>> On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:
>>
>> On 17-10-17 23:06, Todd Blanchard wrote:
>>> Anyone know what happened to this?
>>> https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf
>>
>> https://github.com/SquareBracketAssociates/Booklet-uFFI
>>
>> has a link to a bintray pdf download
>>
>> Stephan
>>
>>
>>
>
>



Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

tblanchard
That’s great - it’s been kind of magical but a couple things have changed since you wrote it and some bugs have crept into ffi.

Right now I’m finding the generated accessor for the VoidPointer3 is actually generating a void 3 accessor and that doesn’t work. I spent all day yesterday tracking it to the accessor generating code.

Sent from the road

On Oct 18, 2017, at 22:29, Ben Coman <[hidden email]> wrote:



On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:
I'm working through Ben's great blog post about playing with libclang and I am puzzled by something.

Thx Todd. Knowing someone is looking at it encourages me to expand it.  
I'm interested in looking at your specific questions, but it might not be until next week after this stint of long work days.

cheers -ben

 

invalidateSessionData
   handle atAllPut: 0. 

zero;s out the handle.  Cool.  However, 

handle isNull 

does not return true despite it being used in the getString method as 

^ handle isNull
         ifTrue: ['external memory invalidated by session restart']
         ifFalse:[LibClang clang_getCString__cxString: self].

Looks like there should be an isNull on ByteArray that returns true if all bytes are zero but it isn't  there.  Was it dropped for some reason?



On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]> wrote:

Sure the documentation could be better, that is definetly important, but is already good enough and UFFI is a very technical subject much more suited to a mailing list . Its not physical possible to cover the massive potential of UFFI.  

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]> wrote:
Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]> wrote:
> Wonderful!  Thanks.
>
>> On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:
>>
>> On 17-10-17 23:06, Todd Blanchard wrote:
>>> Anyone know what happened to this?
>>> https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf
>>
>> https://github.com/SquareBracketAssociates/Booklet-uFFI
>>
>> has a link to a bintray pdf download
>>
>> Stephan
>>
>>
>>
>
>



Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

tblanchard
I have found the problem with VoidPointer3 generating accessors.

No idea how to contribute back a fix but this is what I've found.

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream 
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer ifTrue: [self externalTypeWithArity printString] ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

Recalling that we are trying to come up with an accessor that can pull out a void*[3], this produces

'(FFITypeArray ofType: #FFIVoid size: 3)'

which produces an error as FFIVoid's size is undefined.  In general, this will be wrong for any pointer type and will probably get the size calculation wrong.

Doing a little digging I find that 

FFITypeArray>>ofType: aType size: aSize 

delegates resolution of aType to FFIExternalType>>resolveType:aType and this can take all kinds of different things including the native type name.

It would be better if this generated the native name for pointers.

(FFITypeArray ofType: 'void*' size: 3)

So I changed it to:

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream 
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer 
ifTrue: [self externalTypeWithArity printString] 
ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

and this seems to work fine.

Onwards...

On Oct 19, 2017, at 7:18 AM, Todd Blanchard <[hidden email]> wrote:

That’s great - it’s been kind of magical but a couple things have changed since you wrote it and some bugs have crept into ffi.

Right now I’m finding the generated accessor for the VoidPointer3 is actually generating a void 3 accessor and that doesn’t work. I spent all day yesterday tracking it to the accessor generating code.

Sent from the road

On Oct 18, 2017, at 22:29, Ben Coman <[hidden email]> wrote:



On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:
I'm working through Ben's great blog post about playing with libclang and I am puzzled by something.

Thx Todd. Knowing someone is looking at it encourages me to expand it.  
I'm interested in looking at your specific questions, but it might not be until next week after this stint of long work days.

cheers -ben

 

invalidateSessionData
   handle atAllPut: 0. 

zero;s out the handle.  Cool.  However, 

handle isNull 

does not return true despite it being used in the getString method as 

^ handle isNull
         ifTrue: ['external memory invalidated by session restart']
         ifFalse:[LibClang clang_getCString__cxString: self].

Looks like there should be an isNull on ByteArray that returns true if all bytes are zero but it isn't  there.  Was it dropped for some reason?



On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]> wrote:

Sure the documentation could be better, that is definetly important, but is already good enough and UFFI is a very technical subject much more suited to a mailing list . Its not physical possible to cover the massive potential of UFFI.  

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]> wrote:
Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]> wrote:
> Wonderful!  Thanks.
>
>> On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:
>>
>> On 17-10-17 23:06, Todd Blanchard wrote:
>>> Anyone know what happened to this?
>>> https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf
>>
>> https://github.com/SquareBracketAssociates/Booklet-uFFI
>>
>> has a link to a bintray pdf download
>>
>> Stephan
>>
>>
>>
>
>




Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

Stephane Ducasse-3
Esteban will probably reply to this thread


On Thu, Oct 19, 2017 at 10:34 PM, Todd Blanchard <[hidden email]> wrote:

> I have found the problem with VoidPointer3 generating accessors.
>
> No idea how to contribute back a fix but this is what I've found.
>
> FFITypeArrayType>>annonymousClassCreator
> ^ String streamContents: [ :stream |
> stream
> nextPutAll: '(FFITypeArray ofType: ';
> print: (self objectClass type isPointer ifTrue: [self externalTypeWithArity
> printString] ifFalse: ['#',self objectClass type class]);
> nextPutAll: ' size: ';
> print: self objectClass numberOfElements;
> nextPutAll: ')' ]
>
> Recalling that we are trying to come up with an accessor that can pull out a
> void*[3], this produces
>
> '(FFITypeArray ofType: #FFIVoid size: 3)'
>
> which produces an error as FFIVoid's size is undefined.  In general, this
> will be wrong for any pointer type and will probably get the size
> calculation wrong.
>
> Doing a little digging I find that
>
> FFITypeArray>>ofType: aType size: aSize
>
> delegates resolution of aType to FFIExternalType>>resolveType:aType and this
> can take all kinds of different things including the native type name.
>
> It would be better if this generated the native name for pointers.
>
> (FFITypeArray ofType: 'void*' size: 3)
>
> So I changed it to:
>
> FFITypeArrayType>>annonymousClassCreator
> ^ String streamContents: [ :stream |
> stream
> nextPutAll: '(FFITypeArray ofType: ';
> print: (self objectClass type isPointer
> ifTrue: [self externalTypeWithArity printString]
> ifFalse: ['#',self objectClass type class]);
> nextPutAll: ' size: ';
> print: self objectClass numberOfElements;
> nextPutAll: ')' ]
>
> and this seems to work fine.
>
> Onwards...
>
> On Oct 19, 2017, at 7:18 AM, Todd Blanchard <[hidden email]> wrote:
>
> That’s great - it’s been kind of magical but a couple things have changed
> since you wrote it and some bugs have crept into ffi.
>
> Right now I’m finding the generated accessor for the VoidPointer3 is
> actually generating a void 3 accessor and that doesn’t work. I spent all day
> yesterday tracking it to the accessor generating code.
>
> Sent from the road
>
> On Oct 18, 2017, at 22:29, Ben Coman <[hidden email]> wrote:
>
>
>
> On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:
>>
>> I'm working through Ben's great blog post about playing with libclang and
>> I am puzzled by something.
>
>
> Thx Todd. Knowing someone is looking at it encourages me to expand it.
> I'm interested in looking at your specific questions, but it might not be
> until next week after this stint of long work days.
>
> cheers -ben
>
>
>>
>>
>> invalidateSessionData
>>    handle atAllPut: 0.
>>
>> zero;s out the handle.  Cool.  However,
>>
>> handle isNull
>>
>> does not return true despite it being used in the getString method as
>>
>> ^ handle isNull
>>          ifTrue: ['external memory invalidated by session restart']
>>          ifFalse:[LibClang clang_getCString__cxString: self].
>>
>> Looks like there should be an isNull on ByteArray that returns true if all
>> bytes are zero but it isn't  there.  Was it dropped for some reason?
>>
>>
>>
>> On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]>
>> wrote:
>>
>> Sure the documentation could be better, that is definetly important, but
>> is already good enough and UFFI is a very technical subject much more suited
>> to a mailing list . Its not physical possible to cover the massive potential
>> of UFFI.
>>
>> On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]>
>> wrote:
>>>
>>> Please do not hesitate to do Pull Requests.
>>> Luc told me that he wants to do a pass on it and Esteban promises that to
>>> me
>>> but he is super busy.
>>>
>>> Stef
>>>
>>>
>>> On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]>
>>> wrote:
>>> > Wonderful!  Thanks.
>>> >
>>> >> On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:
>>> >>
>>> >> On 17-10-17 23:06, Todd Blanchard wrote:
>>> >>> Anyone know what happened to this?
>>> >>>
>>> >>> https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf
>>> >>
>>> >> https://github.com/SquareBracketAssociates/Booklet-uFFI
>>> >>
>>> >> has a link to a bintray pdf download
>>> >>
>>> >> Stephan
>>> >>
>>> >>
>>> >>
>>> >
>>> >
>>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

EstebanLM
Hi,

> On 21 Oct 2017, at 21:15, Stephane Ducasse <[hidden email]> wrote:
>
> Esteban will probably reply to this thread

yes, I will :)

>
>
> On Thu, Oct 19, 2017 at 10:34 PM, Todd Blanchard <[hidden email]> wrote:
>> I have found the problem with VoidPointer3 generating accessors.

can you report that on fogbugz ?
(along with your solution)

thanks!
Esteban

>>
>> No idea how to contribute back a fix but this is what I've found.
>>
>> FFITypeArrayType>>annonymousClassCreator
>> ^ String streamContents: [ :stream |
>> stream
>> nextPutAll: '(FFITypeArray ofType: ';
>> print: (self objectClass type isPointer ifTrue: [self externalTypeWithArity
>> printString] ifFalse: ['#',self objectClass type class]);
>> nextPutAll: ' size: ';
>> print: self objectClass numberOfElements;
>> nextPutAll: ')' ]
>>
>> Recalling that we are trying to come up with an accessor that can pull out a
>> void*[3], this produces
>>
>> '(FFITypeArray ofType: #FFIVoid size: 3)'
>>
>> which produces an error as FFIVoid's size is undefined.  In general, this
>> will be wrong for any pointer type and will probably get the size
>> calculation wrong.
>>
>> Doing a little digging I find that
>>
>> FFITypeArray>>ofType: aType size: aSize
>>
>> delegates resolution of aType to FFIExternalType>>resolveType:aType and this
>> can take all kinds of different things including the native type name.
>>
>> It would be better if this generated the native name for pointers.
>>
>> (FFITypeArray ofType: 'void*' size: 3)
>>
>> So I changed it to:
>>
>> FFITypeArrayType>>annonymousClassCreator
>> ^ String streamContents: [ :stream |
>> stream
>> nextPutAll: '(FFITypeArray ofType: ';
>> print: (self objectClass type isPointer
>> ifTrue: [self externalTypeWithArity printString]
>> ifFalse: ['#',self objectClass type class]);
>> nextPutAll: ' size: ';
>> print: self objectClass numberOfElements;
>> nextPutAll: ')' ]
>>
>> and this seems to work fine.
>>
>> Onwards...
>>
>> On Oct 19, 2017, at 7:18 AM, Todd Blanchard <[hidden email]> wrote:
>>
>> That’s great - it’s been kind of magical but a couple things have changed
>> since you wrote it and some bugs have crept into ffi.
>>
>> Right now I’m finding the generated accessor for the VoidPointer3 is
>> actually generating a void 3 accessor and that doesn’t work. I spent all day
>> yesterday tracking it to the accessor generating code.
>>
>> Sent from the road
>>
>> On Oct 18, 2017, at 22:29, Ben Coman <[hidden email]> wrote:
>>
>>
>>
>> On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:
>>>
>>> I'm working through Ben's great blog post about playing with libclang and
>>> I am puzzled by something.
>>
>>
>> Thx Todd. Knowing someone is looking at it encourages me to expand it.
>> I'm interested in looking at your specific questions, but it might not be
>> until next week after this stint of long work days.
>>
>> cheers -ben
>>
>>
>>>
>>>
>>> invalidateSessionData
>>>   handle atAllPut: 0.
>>>
>>> zero;s out the handle.  Cool.  However,
>>>
>>> handle isNull
>>>
>>> does not return true despite it being used in the getString method as
>>>
>>> ^ handle isNull
>>>         ifTrue: ['external memory invalidated by session restart']
>>>         ifFalse:[LibClang clang_getCString__cxString: self].
>>>
>>> Looks like there should be an isNull on ByteArray that returns true if all
>>> bytes are zero but it isn't  there.  Was it dropped for some reason?
>>>
>>>
>>>
>>> On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]>
>>> wrote:
>>>
>>> Sure the documentation could be better, that is definetly important, but
>>> is already good enough and UFFI is a very technical subject much more suited
>>> to a mailing list . Its not physical possible to cover the massive potential
>>> of UFFI.
>>>
>>> On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]>
>>> wrote:
>>>>
>>>> Please do not hesitate to do Pull Requests.
>>>> Luc told me that he wants to do a pass on it and Esteban promises that to
>>>> me
>>>> but he is super busy.
>>>>
>>>> Stef
>>>>
>>>>
>>>> On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]>
>>>> wrote:
>>>>> Wonderful!  Thanks.
>>>>>
>>>>>> On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:
>>>>>>
>>>>>> On 17-10-17 23:06, Todd Blanchard wrote:
>>>>>>> Anyone know what happened to this?
>>>>>>>
>>>>>>> https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf
>>>>>>
>>>>>> https://github.com/SquareBracketAssociates/Booklet-uFFI
>>>>>>
>>>>>> has a link to a bintray pdf download
>>>>>>
>>>>>> Stephan
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>


Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

tblanchard
https://pharo.fogbugz.com/f/cases/20574/FFITypeArray-annonymousClassCreator-doesn-t-handle-pointer-types-correctly

There you go.

On Oct 22, 2017, at 2:42 AM, Esteban Lorenzano <[hidden email]> wrote:

Hi,

On 21 Oct 2017, at 21:15, Stephane Ducasse <[hidden email]> wrote:

Esteban will probably reply to this thread

yes, I will :)



On Thu, Oct 19, 2017 at 10:34 PM, Todd Blanchard <[hidden email]> wrote:
I have found the problem with VoidPointer3 generating accessors.

can you report that on fogbugz ?
(along with your solution)

thanks!
Esteban


No idea how to contribute back a fix but this is what I've found.

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer ifTrue: [self externalTypeWithArity
printString] ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

Recalling that we are trying to come up with an accessor that can pull out a
void*[3], this produces

'(FFITypeArray ofType: #FFIVoid size: 3)'

which produces an error as FFIVoid's size is undefined.  In general, this
will be wrong for any pointer type and will probably get the size
calculation wrong.

Doing a little digging I find that

FFITypeArray>>ofType: aType size: aSize

delegates resolution of aType to FFIExternalType>>resolveType:aType and this
can take all kinds of different things including the native type name.

It would be better if this generated the native name for pointers.

(FFITypeArray ofType: 'void*' size: 3)

So I changed it to:

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer
ifTrue: [self externalTypeWithArity printString]
ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

and this seems to work fine.

Onwards...

On Oct 19, 2017, at 7:18 AM, Todd Blanchard <[hidden email]> wrote:

That’s great - it’s been kind of magical but a couple things have changed
since you wrote it and some bugs have crept into ffi.

Right now I’m finding the generated accessor for the VoidPointer3 is
actually generating a void 3 accessor and that doesn’t work. I spent all day
yesterday tracking it to the accessor generating code.

Sent from the road

On Oct 18, 2017, at 22:29, Ben Coman <[hidden email]> wrote:



On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:

I'm working through Ben's great blog post about playing with libclang and
I am puzzled by something.


Thx Todd. Knowing someone is looking at it encourages me to expand it.
I'm interested in looking at your specific questions, but it might not be
until next week after this stint of long work days.

cheers -ben




invalidateSessionData
 handle atAllPut: 0.

zero;s out the handle.  Cool.  However,

handle isNull

does not return true despite it being used in the getString method as

^ handle isNull
       ifTrue: ['external memory invalidated by session restart']
       ifFalse:[LibClang clang_getCString__cxString: self].

Looks like there should be an isNull on ByteArray that returns true if all
bytes are zero but it isn't  there.  Was it dropped for some reason?



On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]>
wrote:

Sure the documentation could be better, that is definetly important, but
is already good enough and UFFI is a very technical subject much more suited
to a mailing list . Its not physical possible to cover the massive potential
of UFFI.

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]>
wrote:

Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to
me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]>
wrote:
Wonderful!  Thanks.

On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:

On 17-10-17 23:06, Todd Blanchard wrote:
Anyone know what happened to this?

https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf

https://github.com/SquareBracketAssociates/Booklet-uFFI

has a link to a bintray pdf download

Stephan













Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

tblanchard
In reply to this post by EstebanLM
Hey Esteban,

Now that you have outed yourself as a maintainer of FFI, I have a question. :-)

I want to call the libclang function:

/**
 * \brief Parse the given source file and the translation unit corresponding
 * to that file.
 *
 * This routine is the main entry point for the Clang C API, providing the
 * ability to parse a source file into a translation unit that can then be
 * queried by other functions in the API. This routine accepts a set of
 * command-line arguments so that the compilation can be configured in the same
 * way that the compiler is configured on the command line.
 *
 * \param CIdx The index object with which the translation unit will be
 * associated.
 *
 * \param source_filename The name of the source file to load, or NULL if the
 * source file is included in \c command_line_args.
 *
 * \param command_line_args The command-line arguments that would be
 * passed to the \c clang executable if it were being invoked out-of-process.
 * These command-line options will be parsed and will affect how the translation
 * unit is parsed. Note that the following options are ignored: '-c',
 * '-emit-ast', '-fsyntax-only' (which is the default), and '-o \<output file>'.
 *
 * \param num_command_line_args The number of command-line arguments in
 * \c command_line_args.
 *
 * \param unsaved_files the files that have not yet been saved to disk
 * but may be required for parsing, including the contents of
 * those files.  The contents and name of these files (as specified by
 * CXUnsavedFile) are copied when necessary, so the client only needs to
 * guarantee their validity until the call to this function returns.
 *
 * \param num_unsaved_files the number of unsaved file entries in \p
 * unsaved_files.
 *
 * \param options A bitmask of options that affects how the translation unit
 * is managed but not its compilation. This should be a bitwise OR of the
 * CXTranslationUnit_XXX flags.
 *
 * \param[out] out_TU A non-NULL pointer to store the created
 * \c CXTranslationUnit, describing the parsed code and containing any
 * diagnostics produced by the compiler.
 *
 * \returns Zero on success, otherwise returns an error code.
 */
enum CXErrorCode
clang_parseTranslationUnit2(CXIndex CIdx,
                            const char *source_filename,
                            const char *const *command_line_args,
                            int num_command_line_args,
                            struct CXUnsavedFile *unsaved_files,
                            unsigned num_unsaved_files,
                            unsigned options,
                            CXTranslationUnit *out_TU);

The parts I'm not clear on are turning an array or ordered collection of strings into an argv/argc, turning an array of CXUnsavedFile's (FFIExternalStruct) into an array of pointers and count, and getting data back in an output argument as CXTranslationUnit which is an opaque pointer to a C++ object.

Any tips on these mappings would be great.  I've currently guessed it as:

clang_parseTranslationUnit__cxIndex: cxIndex source: sourceFile arguments: argv unsavedFiles:unsaved options: opts

        | argc unsaved_files num_unsaved_files options error out_TU |
        argc := argv size.
        unsaved_files := unsaved ifNil: [ #() ] ifNotNil: [ unsaved ].
        num_unsaved_files := unsaved_files ifNil: [ 0 ] ifNotNil: [unsaved_files size].
        options := opts sum.
       
        error := self ffiCall:  #(CXErrorCode clang_parseTranslationUnit2(CXIndex cxIndex,
                           String sourceFile,
                           String *argv,
                           uint argc,
                           CXUnsavedFile *unsaved_files,
                           uint num_unsaved_files,
                           CXTranslationUnitFlag options,
                                                                  CXTranslationUnit *out_TU)).
                                                               
        (error = CXError_Success) ifTrue: [ ^out_TU ].
        self error: error

Thanks.


> On Oct 22, 2017, at 2:42 AM, Esteban Lorenzano <[hidden email]> wrote:
>
> Hi,
>
>> On 21 Oct 2017, at 21:15, Stephane Ducasse <[hidden email]> wrote:
>>
>> Esteban will probably reply to this thread
>
> yes, I will :)
>
>>
>>
>> On Thu, Oct 19, 2017 at 10:34 PM, Todd Blanchard <[hidden email]> wrote:
>>> I have found the problem with VoidPointer3 generating accessors.
>
> can you report that on fogbugz ?
> (along with your solution)
>
> thanks!
> Esteban
>
>>>
>>> No idea how to contribute back a fix but this is what I've found.
>>>
>>> FFITypeArrayType>>annonymousClassCreator
>>> ^ String streamContents: [ :stream |
>>> stream
>>> nextPutAll: '(FFITypeArray ofType: ';
>>> print: (self objectClass type isPointer ifTrue: [self externalTypeWithArity
>>> printString] ifFalse: ['#',self objectClass type class]);
>>> nextPutAll: ' size: ';
>>> print: self objectClass numberOfElements;
>>> nextPutAll: ')' ]
>>>
>>> Recalling that we are trying to come up with an accessor that can pull out a
>>> void*[3], this produces
>>>
>>> '(FFITypeArray ofType: #FFIVoid size: 3)'
>>>
>>> which produces an error as FFIVoid's size is undefined.  In general, this
>>> will be wrong for any pointer type and will probably get the size
>>> calculation wrong.
>>>
>>> Doing a little digging I find that
>>>
>>> FFITypeArray>>ofType: aType size: aSize
>>>
>>> delegates resolution of aType to FFIExternalType>>resolveType:aType and this
>>> can take all kinds of different things including the native type name.
>>>
>>> It would be better if this generated the native name for pointers.
>>>
>>> (FFITypeArray ofType: 'void*' size: 3)
>>>
>>> So I changed it to:
>>>
>>> FFITypeArrayType>>annonymousClassCreator
>>> ^ String streamContents: [ :stream |
>>> stream
>>> nextPutAll: '(FFITypeArray ofType: ';
>>> print: (self objectClass type isPointer
>>> ifTrue: [self externalTypeWithArity printString]
>>> ifFalse: ['#',self objectClass type class]);
>>> nextPutAll: ' size: ';
>>> print: self objectClass numberOfElements;
>>> nextPutAll: ')' ]
>>>
>>> and this seems to work fine.
>>>
>>> Onwards...
>>>
>>> On Oct 19, 2017, at 7:18 AM, Todd Blanchard <[hidden email]> wrote:
>>>
>>> That’s great - it’s been kind of magical but a couple things have changed
>>> since you wrote it and some bugs have crept into ffi.
>>>
>>> Right now I’m finding the generated accessor for the VoidPointer3 is
>>> actually generating a void 3 accessor and that doesn’t work. I spent all day
>>> yesterday tracking it to the accessor generating code.
>>>
>>> Sent from the road
>>>
>>> On Oct 18, 2017, at 22:29, Ben Coman <[hidden email]> wrote:
>>>
>>>
>>>
>>> On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:
>>>>
>>>> I'm working through Ben's great blog post about playing with libclang and
>>>> I am puzzled by something.
>>>
>>>
>>> Thx Todd. Knowing someone is looking at it encourages me to expand it.
>>> I'm interested in looking at your specific questions, but it might not be
>>> until next week after this stint of long work days.
>>>
>>> cheers -ben
>>>
>>>
>>>>
>>>>
>>>> invalidateSessionData
>>>>  handle atAllPut: 0.
>>>>
>>>> zero;s out the handle.  Cool.  However,
>>>>
>>>> handle isNull
>>>>
>>>> does not return true despite it being used in the getString method as
>>>>
>>>> ^ handle isNull
>>>>        ifTrue: ['external memory invalidated by session restart']
>>>>        ifFalse:[LibClang clang_getCString__cxString: self].
>>>>
>>>> Looks like there should be an isNull on ByteArray that returns true if all
>>>> bytes are zero but it isn't  there.  Was it dropped for some reason?
>>>>
>>>>
>>>>
>>>> On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]>
>>>> wrote:
>>>>
>>>> Sure the documentation could be better, that is definetly important, but
>>>> is already good enough and UFFI is a very technical subject much more suited
>>>> to a mailing list . Its not physical possible to cover the massive potential
>>>> of UFFI.
>>>>
>>>> On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]>
>>>> wrote:
>>>>>
>>>>> Please do not hesitate to do Pull Requests.
>>>>> Luc told me that he wants to do a pass on it and Esteban promises that to
>>>>> me
>>>>> but he is super busy.
>>>>>
>>>>> Stef
>>>>>
>>>>>
>>>>> On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]>
>>>>> wrote:
>>>>>> Wonderful!  Thanks.
>>>>>>
>>>>>>> On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:
>>>>>>>
>>>>>>> On 17-10-17 23:06, Todd Blanchard wrote:
>>>>>>>> Anyone know what happened to this?
>>>>>>>>
>>>>>>>> https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf
>>>>>>>
>>>>>>> https://github.com/SquareBracketAssociates/Booklet-uFFI
>>>>>>>
>>>>>>> has a link to a bintray pdf download
>>>>>>>
>>>>>>> Stephan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

EstebanLM
:)


it was using parseTranslationUnit (no 2)… there you can find how is being used (I just converted it to tonel format, but older versions are in filetree)

Esteban

On 24 Oct 2017, at 19:17, Todd Blanchard <[hidden email]> wrote:

Hey Esteban,

Now that you have outed yourself as a maintainer of FFI, I have a question. :-)

I want to call the libclang function:

/**
* \brief Parse the given source file and the translation unit corresponding
* to that file.
*
* This routine is the main entry point for the Clang C API, providing the
* ability to parse a source file into a translation unit that can then be
* queried by other functions in the API. This routine accepts a set of
* command-line arguments so that the compilation can be configured in the same
* way that the compiler is configured on the command line.
*
* \param CIdx The index object with which the translation unit will be
* associated.
*
* \param source_filename The name of the source file to load, or NULL if the
* source file is included in \c command_line_args.
*
* \param command_line_args The command-line arguments that would be
* passed to the \c clang executable if it were being invoked out-of-process.
* These command-line options will be parsed and will affect how the translation
* unit is parsed. Note that the following options are ignored: '-c',
* '-emit-ast', '-fsyntax-only' (which is the default), and '-o \<output file>'.
*
* \param num_command_line_args The number of command-line arguments in
* \c command_line_args.
*
* \param unsaved_files the files that have not yet been saved to disk
* but may be required for parsing, including the contents of
* those files.  The contents and name of these files (as specified by
* CXUnsavedFile) are copied when necessary, so the client only needs to
* guarantee their validity until the call to this function returns.
*
* \param num_unsaved_files the number of unsaved file entries in \p
* unsaved_files.
*
* \param options A bitmask of options that affects how the translation unit
* is managed but not its compilation. This should be a bitwise OR of the
* CXTranslationUnit_XXX flags.
*
* \param[out] out_TU A non-NULL pointer to store the created
* \c CXTranslationUnit, describing the parsed code and containing any
* diagnostics produced by the compiler.
*
* \returns Zero on success, otherwise returns an error code.
*/
enum CXErrorCode
clang_parseTranslationUnit2(CXIndex CIdx,
                           const char *source_filename,
                           const char *const *command_line_args,
                           int num_command_line_args,
                           struct CXUnsavedFile *unsaved_files,
                           unsigned num_unsaved_files,
                           unsigned options,
                           CXTranslationUnit *out_TU);

The parts I'm not clear on are turning an array or ordered collection of strings into an argv/argc, turning an array of CXUnsavedFile's (FFIExternalStruct) into an array of pointers and count, and getting data back in an output argument as CXTranslationUnit which is an opaque pointer to a C++ object.

Any tips on these mappings would be great.  I've currently guessed it as:

clang_parseTranslationUnit__cxIndex: cxIndex source: sourceFile arguments: argv unsavedFiles:unsaved options: opts

| argc unsaved_files num_unsaved_files options error out_TU |
argc := argv size.
unsaved_files := unsaved ifNil: [ #() ] ifNotNil: [ unsaved ].
num_unsaved_files := unsaved_files ifNil: [ 0 ] ifNotNil: [unsaved_files size].
options := opts sum.

error := self ffiCall:  #(CXErrorCode clang_parseTranslationUnit2(CXIndex cxIndex,
                          String sourceFile,
                          String *argv,
                          uint argc,
                          CXUnsavedFile *unsaved_files,
                          uint num_unsaved_files,
                          CXTranslationUnitFlag options,
 CXTranslationUnit *out_TU)).

(error = CXError_Success) ifTrue: [ ^out_TU ].
self error: error

Thanks.


On Oct 22, 2017, at 2:42 AM, Esteban Lorenzano <[hidden email]> wrote:

Hi,

On 21 Oct 2017, at 21:15, Stephane Ducasse <[hidden email]> wrote:

Esteban will probably reply to this thread

yes, I will :)



On Thu, Oct 19, 2017 at 10:34 PM, Todd Blanchard <[hidden email]> wrote:
I have found the problem with VoidPointer3 generating accessors.

can you report that on fogbugz ?
(along with your solution)

thanks!
Esteban


No idea how to contribute back a fix but this is what I've found.

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer ifTrue: [self externalTypeWithArity
printString] ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

Recalling that we are trying to come up with an accessor that can pull out a
void*[3], this produces

'(FFITypeArray ofType: #FFIVoid size: 3)'

which produces an error as FFIVoid's size is undefined.  In general, this
will be wrong for any pointer type and will probably get the size
calculation wrong.

Doing a little digging I find that

FFITypeArray>>ofType: aType size: aSize

delegates resolution of aType to FFIExternalType>>resolveType:aType and this
can take all kinds of different things including the native type name.

It would be better if this generated the native name for pointers.

(FFITypeArray ofType: 'void*' size: 3)

So I changed it to:

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer
ifTrue: [self externalTypeWithArity printString]
ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

and this seems to work fine.

Onwards...

On Oct 19, 2017, at 7:18 AM, Todd Blanchard <[hidden email]> wrote:

That’s great - it’s been kind of magical but a couple things have changed
since you wrote it and some bugs have crept into ffi.

Right now I’m finding the generated accessor for the VoidPointer3 is
actually generating a void 3 accessor and that doesn’t work. I spent all day
yesterday tracking it to the accessor generating code.

Sent from the road

On Oct 18, 2017, at 22:29, Ben Coman <[hidden email]> wrote:



On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:

I'm working through Ben's great blog post about playing with libclang and
I am puzzled by something.


Thx Todd. Knowing someone is looking at it encourages me to expand it.
I'm interested in looking at your specific questions, but it might not be
until next week after this stint of long work days.

cheers -ben




invalidateSessionData
handle atAllPut: 0.

zero;s out the handle.  Cool.  However,

handle isNull

does not return true despite it being used in the getString method as

^ handle isNull
      ifTrue: ['external memory invalidated by session restart']
      ifFalse:[LibClang clang_getCString__cxString: self].

Looks like there should be an isNull on ByteArray that returns true if all
bytes are zero but it isn't  there.  Was it dropped for some reason?



On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]>
wrote:

Sure the documentation could be better, that is definetly important, but
is already good enough and UFFI is a very technical subject much more suited
to a mailing list . Its not physical possible to cover the massive potential
of UFFI.

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]>
wrote:

Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to
me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]>
wrote:
Wonderful!  Thanks.

On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:

On 17-10-17 23:06, Todd Blanchard wrote:
Anyone know what happened to this?

https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf

https://github.com/SquareBracketAssociates/Booklet-uFFI

has a link to a bintray pdf download

Stephan














Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

tblanchard
I did not - had no idea it existed.

Is if fair to say that the 32 bit warning in the readme does not apply to Pharo 64 bit?  

On Oct 25, 2017, at 7:35 AM, Esteban Lorenzano <[hidden email]> wrote:

:)


it was using parseTranslationUnit (no 2)… there you can find how is being used (I just converted it to tonel format, but older versions are in filetree)

Esteban

On 24 Oct 2017, at 19:17, Todd Blanchard <[hidden email]> wrote:

Hey Esteban,

Now that you have outed yourself as a maintainer of FFI, I have a question. :-)

I want to call the libclang function:

/**
* \brief Parse the given source file and the translation unit corresponding
* to that file.
*
* This routine is the main entry point for the Clang C API, providing the
* ability to parse a source file into a translation unit that can then be
* queried by other functions in the API. This routine accepts a set of
* command-line arguments so that the compilation can be configured in the same
* way that the compiler is configured on the command line.
*
* \param CIdx The index object with which the translation unit will be
* associated.
*
* \param source_filename The name of the source file to load, or NULL if the
* source file is included in \c command_line_args.
*
* \param command_line_args The command-line arguments that would be
* passed to the \c clang executable if it were being invoked out-of-process.
* These command-line options will be parsed and will affect how the translation
* unit is parsed. Note that the following options are ignored: '-c',
* '-emit-ast', '-fsyntax-only' (which is the default), and '-o \<output file>'.
*
* \param num_command_line_args The number of command-line arguments in
* \c command_line_args.
*
* \param unsaved_files the files that have not yet been saved to disk
* but may be required for parsing, including the contents of
* those files.  The contents and name of these files (as specified by
* CXUnsavedFile) are copied when necessary, so the client only needs to
* guarantee their validity until the call to this function returns.
*
* \param num_unsaved_files the number of unsaved file entries in \p
* unsaved_files.
*
* \param options A bitmask of options that affects how the translation unit
* is managed but not its compilation. This should be a bitwise OR of the
* CXTranslationUnit_XXX flags.
*
* \param[out] out_TU A non-NULL pointer to store the created
* \c CXTranslationUnit, describing the parsed code and containing any
* diagnostics produced by the compiler.
*
* \returns Zero on success, otherwise returns an error code.
*/
enum CXErrorCode
clang_parseTranslationUnit2(CXIndex CIdx,
                           const char *source_filename,
                           const char *const *command_line_args,
                           int num_command_line_args,
                           struct CXUnsavedFile *unsaved_files,
                           unsigned num_unsaved_files,
                           unsigned options,
                           CXTranslationUnit *out_TU);

The parts I'm not clear on are turning an array or ordered collection of strings into an argv/argc, turning an array of CXUnsavedFile's (FFIExternalStruct) into an array of pointers and count, and getting data back in an output argument as CXTranslationUnit which is an opaque pointer to a C++ object.

Any tips on these mappings would be great.  I've currently guessed it as:

clang_parseTranslationUnit__cxIndex: cxIndex source: sourceFile arguments: argv unsavedFiles:unsaved options: opts

| argc unsaved_files num_unsaved_files options error out_TU |
argc := argv size.
unsaved_files := unsaved ifNil: [ #() ] ifNotNil: [ unsaved ].
num_unsaved_files := unsaved_files ifNil: [ 0 ] ifNotNil: [unsaved_files size].
options := opts sum.

error := self ffiCall:  #(CXErrorCode clang_parseTranslationUnit2(CXIndex cxIndex,
                          String sourceFile,
                          String *argv,
                          uint argc,
                          CXUnsavedFile *unsaved_files,
                          uint num_unsaved_files,
                          CXTranslationUnitFlag options,
 CXTranslationUnit *out_TU)).

(error = CXError_Success) ifTrue: [ ^out_TU ].
self error: error

Thanks.


On Oct 22, 2017, at 2:42 AM, Esteban Lorenzano <[hidden email]> wrote:

Hi,

On 21 Oct 2017, at 21:15, Stephane Ducasse <[hidden email]> wrote:

Esteban will probably reply to this thread

yes, I will :)



On Thu, Oct 19, 2017 at 10:34 PM, Todd Blanchard <[hidden email]> wrote:
I have found the problem with VoidPointer3 generating accessors.

can you report that on fogbugz ?
(along with your solution)

thanks!
Esteban


No idea how to contribute back a fix but this is what I've found.

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer ifTrue: [self externalTypeWithArity
printString] ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

Recalling that we are trying to come up with an accessor that can pull out a
void*[3], this produces

'(FFITypeArray ofType: #FFIVoid size: 3)'

which produces an error as FFIVoid's size is undefined.  In general, this
will be wrong for any pointer type and will probably get the size
calculation wrong.

Doing a little digging I find that

FFITypeArray>>ofType: aType size: aSize

delegates resolution of aType to FFIExternalType>>resolveType:aType and this
can take all kinds of different things including the native type name.

It would be better if this generated the native name for pointers.

(FFITypeArray ofType: 'void*' size: 3)

So I changed it to:

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer
ifTrue: [self externalTypeWithArity printString]
ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

and this seems to work fine.

Onwards...

On Oct 19, 2017, at 7:18 AM, Todd Blanchard <[hidden email]> wrote:

That’s great - it’s been kind of magical but a couple things have changed
since you wrote it and some bugs have crept into ffi.

Right now I’m finding the generated accessor for the VoidPointer3 is
actually generating a void 3 accessor and that doesn’t work. I spent all day
yesterday tracking it to the accessor generating code.

Sent from the road

On Oct 18, 2017, at 22:29, Ben Coman <[hidden email]> wrote:



On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:

I'm working through Ben's great blog post about playing with libclang and
I am puzzled by something.


Thx Todd. Knowing someone is looking at it encourages me to expand it.
I'm interested in looking at your specific questions, but it might not be
until next week after this stint of long work days.

cheers -ben




invalidateSessionData
handle atAllPut: 0.

zero;s out the handle.  Cool.  However,

handle isNull

does not return true despite it being used in the getString method as

^ handle isNull
      ifTrue: ['external memory invalidated by session restart']
      ifFalse:[LibClang clang_getCString__cxString: self].

Looks like there should be an isNull on ByteArray that returns true if all
bytes are zero but it isn't  there.  Was it dropped for some reason?



On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]>
wrote:

Sure the documentation could be better, that is definetly important, but
is already good enough and UFFI is a very technical subject much more suited
to a mailing list . Its not physical possible to cover the massive potential
of UFFI.

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]>
wrote:

Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to
me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]>
wrote:
Wonderful!  Thanks.

On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:

On 17-10-17 23:06, Todd Blanchard wrote:
Anyone know what happened to this?

https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf

https://github.com/SquareBracketAssociates/Booklet-uFFI

has a link to a bintray pdf download

Stephan















Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

tblanchard
In reply to this post by EstebanLM
OK, new question.

How do I file this in?  I was able to load tonel from iceberg.  I added the repo and it gave me a choice of packages to load and I loaded them using the iceberg browser.

I added this to the iceberg browser and there are no packages listed and I have absolutely no clue how to get this code into my image.

I'm about 80 hours into trying to get my head around FFI and really getting discouraged at how many walls I'm hitting trying to get a foreign library interface done.

On Oct 25, 2017, at 7:35 AM, Esteban Lorenzano <[hidden email]> wrote:

:)


it was using parseTranslationUnit (no 2)… there you can find how is being used (I just converted it to tonel format, but older versions are in filetree)

Esteban

On 24 Oct 2017, at 19:17, Todd Blanchard <[hidden email]> wrote:

Hey Esteban,

Now that you have outed yourself as a maintainer of FFI, I have a question. :-)

I want to call the libclang function:

/**
* \brief Parse the given source file and the translation unit corresponding
* to that file.
*
* This routine is the main entry point for the Clang C API, providing the
* ability to parse a source file into a translation unit that can then be
* queried by other functions in the API. This routine accepts a set of
* command-line arguments so that the compilation can be configured in the same
* way that the compiler is configured on the command line.
*
* \param CIdx The index object with which the translation unit will be
* associated.
*
* \param source_filename The name of the source file to load, or NULL if the
* source file is included in \c command_line_args.
*
* \param command_line_args The command-line arguments that would be
* passed to the \c clang executable if it were being invoked out-of-process.
* These command-line options will be parsed and will affect how the translation
* unit is parsed. Note that the following options are ignored: '-c',
* '-emit-ast', '-fsyntax-only' (which is the default), and '-o \<output file>'.
*
* \param num_command_line_args The number of command-line arguments in
* \c command_line_args.
*
* \param unsaved_files the files that have not yet been saved to disk
* but may be required for parsing, including the contents of
* those files.  The contents and name of these files (as specified by
* CXUnsavedFile) are copied when necessary, so the client only needs to
* guarantee their validity until the call to this function returns.
*
* \param num_unsaved_files the number of unsaved file entries in \p
* unsaved_files.
*
* \param options A bitmask of options that affects how the translation unit
* is managed but not its compilation. This should be a bitwise OR of the
* CXTranslationUnit_XXX flags.
*
* \param[out] out_TU A non-NULL pointer to store the created
* \c CXTranslationUnit, describing the parsed code and containing any
* diagnostics produced by the compiler.
*
* \returns Zero on success, otherwise returns an error code.
*/
enum CXErrorCode
clang_parseTranslationUnit2(CXIndex CIdx,
                           const char *source_filename,
                           const char *const *command_line_args,
                           int num_command_line_args,
                           struct CXUnsavedFile *unsaved_files,
                           unsigned num_unsaved_files,
                           unsigned options,
                           CXTranslationUnit *out_TU);

The parts I'm not clear on are turning an array or ordered collection of strings into an argv/argc, turning an array of CXUnsavedFile's (FFIExternalStruct) into an array of pointers and count, and getting data back in an output argument as CXTranslationUnit which is an opaque pointer to a C++ object.

Any tips on these mappings would be great.  I've currently guessed it as:

clang_parseTranslationUnit__cxIndex: cxIndex source: sourceFile arguments: argv unsavedFiles:unsaved options: opts

| argc unsaved_files num_unsaved_files options error out_TU |
argc := argv size.
unsaved_files := unsaved ifNil: [ #() ] ifNotNil: [ unsaved ].
num_unsaved_files := unsaved_files ifNil: [ 0 ] ifNotNil: [unsaved_files size].
options := opts sum.

error := self ffiCall:  #(CXErrorCode clang_parseTranslationUnit2(CXIndex cxIndex,
                          String sourceFile,
                          String *argv,
                          uint argc,
                          CXUnsavedFile *unsaved_files,
                          uint num_unsaved_files,
                          CXTranslationUnitFlag options,
 CXTranslationUnit *out_TU)).

(error = CXError_Success) ifTrue: [ ^out_TU ].
self error: error

Thanks.


On Oct 22, 2017, at 2:42 AM, Esteban Lorenzano <[hidden email]> wrote:

Hi,

On 21 Oct 2017, at 21:15, Stephane Ducasse <[hidden email]> wrote:

Esteban will probably reply to this thread

yes, I will :)



On Thu, Oct 19, 2017 at 10:34 PM, Todd Blanchard <[hidden email]> wrote:
I have found the problem with VoidPointer3 generating accessors.

can you report that on fogbugz ?
(along with your solution)

thanks!
Esteban


No idea how to contribute back a fix but this is what I've found.

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer ifTrue: [self externalTypeWithArity
printString] ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

Recalling that we are trying to come up with an accessor that can pull out a
void*[3], this produces

'(FFITypeArray ofType: #FFIVoid size: 3)'

which produces an error as FFIVoid's size is undefined.  In general, this
will be wrong for any pointer type and will probably get the size
calculation wrong.

Doing a little digging I find that

FFITypeArray>>ofType: aType size: aSize

delegates resolution of aType to FFIExternalType>>resolveType:aType and this
can take all kinds of different things including the native type name.

It would be better if this generated the native name for pointers.

(FFITypeArray ofType: 'void*' size: 3)

So I changed it to:

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer
ifTrue: [self externalTypeWithArity printString]
ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

and this seems to work fine.

Onwards...

On Oct 19, 2017, at 7:18 AM, Todd Blanchard <[hidden email]> wrote:

That’s great - it’s been kind of magical but a couple things have changed
since you wrote it and some bugs have crept into ffi.

Right now I’m finding the generated accessor for the VoidPointer3 is
actually generating a void 3 accessor and that doesn’t work. I spent all day
yesterday tracking it to the accessor generating code.

Sent from the road

On Oct 18, 2017, at 22:29, Ben Coman <[hidden email]> wrote:



On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:

I'm working through Ben's great blog post about playing with libclang and
I am puzzled by something.


Thx Todd. Knowing someone is looking at it encourages me to expand it.
I'm interested in looking at your specific questions, but it might not be
until next week after this stint of long work days.

cheers -ben




invalidateSessionData
handle atAllPut: 0.

zero;s out the handle.  Cool.  However,

handle isNull

does not return true despite it being used in the getString method as

^ handle isNull
      ifTrue: ['external memory invalidated by session restart']
      ifFalse:[LibClang clang_getCString__cxString: self].

Looks like there should be an isNull on ByteArray that returns true if all
bytes are zero but it isn't  there.  Was it dropped for some reason?



On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]>
wrote:

Sure the documentation could be better, that is definetly important, but
is already good enough and UFFI is a very technical subject much more suited
to a mailing list . Its not physical possible to cover the massive potential
of UFFI.

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]>
wrote:

Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to
me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]>
wrote:
Wonderful!  Thanks.

On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:

On 17-10-17 23:06, Todd Blanchard wrote:
Anyone know what happened to this?

https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf

https://github.com/SquareBracketAssociates/Booklet-uFFI

has a link to a bintray pdf download

Stephan















Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

EstebanLM
In reply to this post by tblanchard


On 25 Oct 2017, at 19:58, Todd Blanchard <[hidden email]> wrote:

I did not - had no idea it existed.

Is if fair to say that the 32 bit warning in the readme does not apply to Pharo 64 bit?  

is not. I never tested it on 64bit :) 
in theory, it should work out of the box, but since is not tested, warning remains.

Esteban


On Oct 25, 2017, at 7:35 AM, Esteban Lorenzano <[hidden email]> wrote:

:)


it was using parseTranslationUnit (no 2)… there you can find how is being used (I just converted it to tonel format, but older versions are in filetree)

Esteban

On 24 Oct 2017, at 19:17, Todd Blanchard <[hidden email]> wrote:

Hey Esteban,

Now that you have outed yourself as a maintainer of FFI, I have a question. :-)

I want to call the libclang function:

/**
* \brief Parse the given source file and the translation unit corresponding
* to that file.
*
* This routine is the main entry point for the Clang C API, providing the
* ability to parse a source file into a translation unit that can then be
* queried by other functions in the API. This routine accepts a set of
* command-line arguments so that the compilation can be configured in the same
* way that the compiler is configured on the command line.
*
* \param CIdx The index object with which the translation unit will be
* associated.
*
* \param source_filename The name of the source file to load, or NULL if the
* source file is included in \c command_line_args.
*
* \param command_line_args The command-line arguments that would be
* passed to the \c clang executable if it were being invoked out-of-process.
* These command-line options will be parsed and will affect how the translation
* unit is parsed. Note that the following options are ignored: '-c',
* '-emit-ast', '-fsyntax-only' (which is the default), and '-o \<output file>'.
*
* \param num_command_line_args The number of command-line arguments in
* \c command_line_args.
*
* \param unsaved_files the files that have not yet been saved to disk
* but may be required for parsing, including the contents of
* those files.  The contents and name of these files (as specified by
* CXUnsavedFile) are copied when necessary, so the client only needs to
* guarantee their validity until the call to this function returns.
*
* \param num_unsaved_files the number of unsaved file entries in \p
* unsaved_files.
*
* \param options A bitmask of options that affects how the translation unit
* is managed but not its compilation. This should be a bitwise OR of the
* CXTranslationUnit_XXX flags.
*
* \param[out] out_TU A non-NULL pointer to store the created
* \c CXTranslationUnit, describing the parsed code and containing any
* diagnostics produced by the compiler.
*
* \returns Zero on success, otherwise returns an error code.
*/
enum CXErrorCode
clang_parseTranslationUnit2(CXIndex CIdx,
                           const char *source_filename,
                           const char *const *command_line_args,
                           int num_command_line_args,
                           struct CXUnsavedFile *unsaved_files,
                           unsigned num_unsaved_files,
                           unsigned options,
                           CXTranslationUnit *out_TU);

The parts I'm not clear on are turning an array or ordered collection of strings into an argv/argc, turning an array of CXUnsavedFile's (FFIExternalStruct) into an array of pointers and count, and getting data back in an output argument as CXTranslationUnit which is an opaque pointer to a C++ object.

Any tips on these mappings would be great.  I've currently guessed it as:

clang_parseTranslationUnit__cxIndex: cxIndex source: sourceFile arguments: argv unsavedFiles:unsaved options: opts

| argc unsaved_files num_unsaved_files options error out_TU |
argc := argv size.
unsaved_files := unsaved ifNil: [ #() ] ifNotNil: [ unsaved ].
num_unsaved_files := unsaved_files ifNil: [ 0 ] ifNotNil: [unsaved_files size].
options := opts sum.

error := self ffiCall:  #(CXErrorCode clang_parseTranslationUnit2(CXIndex cxIndex,
                          String sourceFile,
                          String *argv,
                          uint argc,
                          CXUnsavedFile *unsaved_files,
                          uint num_unsaved_files,
                          CXTranslationUnitFlag options,
 CXTranslationUnit *out_TU)).

(error = CXError_Success) ifTrue: [ ^out_TU ].
self error: error

Thanks.


On Oct 22, 2017, at 2:42 AM, Esteban Lorenzano <[hidden email]> wrote:

Hi,

On 21 Oct 2017, at 21:15, Stephane Ducasse <[hidden email]> wrote:

Esteban will probably reply to this thread

yes, I will :)



On Thu, Oct 19, 2017 at 10:34 PM, Todd Blanchard <[hidden email]> wrote:
I have found the problem with VoidPointer3 generating accessors.

can you report that on fogbugz ?
(along with your solution)

thanks!
Esteban


No idea how to contribute back a fix but this is what I've found.

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer ifTrue: [self externalTypeWithArity
printString] ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

Recalling that we are trying to come up with an accessor that can pull out a
void*[3], this produces

'(FFITypeArray ofType: #FFIVoid size: 3)'

which produces an error as FFIVoid's size is undefined.  In general, this
will be wrong for any pointer type and will probably get the size
calculation wrong.

Doing a little digging I find that

FFITypeArray>>ofType: aType size: aSize

delegates resolution of aType to FFIExternalType>>resolveType:aType and this
can take all kinds of different things including the native type name.

It would be better if this generated the native name for pointers.

(FFITypeArray ofType: 'void*' size: 3)

So I changed it to:

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer
ifTrue: [self externalTypeWithArity printString]
ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

and this seems to work fine.

Onwards...

On Oct 19, 2017, at 7:18 AM, Todd Blanchard <[hidden email]> wrote:

That’s great - it’s been kind of magical but a couple things have changed
since you wrote it and some bugs have crept into ffi.

Right now I’m finding the generated accessor for the VoidPointer3 is
actually generating a void 3 accessor and that doesn’t work. I spent all day
yesterday tracking it to the accessor generating code.

Sent from the road

On Oct 18, 2017, at 22:29, Ben Coman <[hidden email]> wrote:



On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:

I'm working through Ben's great blog post about playing with libclang and
I am puzzled by something.


Thx Todd. Knowing someone is looking at it encourages me to expand it.
I'm interested in looking at your specific questions, but it might not be
until next week after this stint of long work days.

cheers -ben




invalidateSessionData
handle atAllPut: 0.

zero;s out the handle.  Cool.  However,

handle isNull

does not return true despite it being used in the getString method as

^ handle isNull
      ifTrue: ['external memory invalidated by session restart']
      ifFalse:[LibClang clang_getCString__cxString: self].

Looks like there should be an isNull on ByteArray that returns true if all
bytes are zero but it isn't  there.  Was it dropped for some reason?



On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]>
wrote:

Sure the documentation could be better, that is definetly important, but
is already good enough and UFFI is a very technical subject much more suited
to a mailing list . Its not physical possible to cover the massive potential
of UFFI.

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]>
wrote:

Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to
me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]>
wrote:
Wonderful!  Thanks.

On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:

On 17-10-17 23:06, Todd Blanchard wrote:
Anyone know what happened to this?

https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf

https://github.com/SquareBracketAssociates/Booklet-uFFI

has a link to a bintray pdf download

Stephan
















Reply | Threaded
Open this post in threaded view
|

Re: UnifiedFFI Docs

EstebanLM
In reply to this post by tblanchard


On 25 Oct 2017, at 21:39, Todd Blanchard <[hidden email]> wrote:

OK, new question.

How do I file this in?  I was able to load tonel from iceberg.  I added the repo and it gave me a choice of packages to load and I loaded them using the iceberg browser.

I added this to the iceberg browser and there are no packages listed and I have absolutely no clue how to get this code into my image.

I'm about 80 hours into trying to get my head around FFI and really getting discouraged at how many walls I'm hitting trying to get a foreign library interface done.

in Pharo7, you do nothing, is there by default. 
you do not “file in”, you add an iceberg project and then you load the baseline, as any other project.

about the walls: you didn’t chose a simple library to learn FFI, you chose one of the hardest libraries that exist. 
What you are doing is like trying to learn to be a doctor through open-heart operations :)

Esteban


On Oct 25, 2017, at 7:35 AM, Esteban Lorenzano <[hidden email]> wrote:

:)


it was using parseTranslationUnit (no 2)… there you can find how is being used (I just converted it to tonel format, but older versions are in filetree)

Esteban

On 24 Oct 2017, at 19:17, Todd Blanchard <[hidden email]> wrote:

Hey Esteban,

Now that you have outed yourself as a maintainer of FFI, I have a question. :-)

I want to call the libclang function:

/**
* \brief Parse the given source file and the translation unit corresponding
* to that file.
*
* This routine is the main entry point for the Clang C API, providing the
* ability to parse a source file into a translation unit that can then be
* queried by other functions in the API. This routine accepts a set of
* command-line arguments so that the compilation can be configured in the same
* way that the compiler is configured on the command line.
*
* \param CIdx The index object with which the translation unit will be
* associated.
*
* \param source_filename The name of the source file to load, or NULL if the
* source file is included in \c command_line_args.
*
* \param command_line_args The command-line arguments that would be
* passed to the \c clang executable if it were being invoked out-of-process.
* These command-line options will be parsed and will affect how the translation
* unit is parsed. Note that the following options are ignored: '-c',
* '-emit-ast', '-fsyntax-only' (which is the default), and '-o \<output file>'.
*
* \param num_command_line_args The number of command-line arguments in
* \c command_line_args.
*
* \param unsaved_files the files that have not yet been saved to disk
* but may be required for parsing, including the contents of
* those files.  The contents and name of these files (as specified by
* CXUnsavedFile) are copied when necessary, so the client only needs to
* guarantee their validity until the call to this function returns.
*
* \param num_unsaved_files the number of unsaved file entries in \p
* unsaved_files.
*
* \param options A bitmask of options that affects how the translation unit
* is managed but not its compilation. This should be a bitwise OR of the
* CXTranslationUnit_XXX flags.
*
* \param[out] out_TU A non-NULL pointer to store the created
* \c CXTranslationUnit, describing the parsed code and containing any
* diagnostics produced by the compiler.
*
* \returns Zero on success, otherwise returns an error code.
*/
enum CXErrorCode
clang_parseTranslationUnit2(CXIndex CIdx,
                           const char *source_filename,
                           const char *const *command_line_args,
                           int num_command_line_args,
                           struct CXUnsavedFile *unsaved_files,
                           unsigned num_unsaved_files,
                           unsigned options,
                           CXTranslationUnit *out_TU);

The parts I'm not clear on are turning an array or ordered collection of strings into an argv/argc, turning an array of CXUnsavedFile's (FFIExternalStruct) into an array of pointers and count, and getting data back in an output argument as CXTranslationUnit which is an opaque pointer to a C++ object.

Any tips on these mappings would be great.  I've currently guessed it as:

clang_parseTranslationUnit__cxIndex: cxIndex source: sourceFile arguments: argv unsavedFiles:unsaved options: opts

| argc unsaved_files num_unsaved_files options error out_TU |
argc := argv size.
unsaved_files := unsaved ifNil: [ #() ] ifNotNil: [ unsaved ].
num_unsaved_files := unsaved_files ifNil: [ 0 ] ifNotNil: [unsaved_files size].
options := opts sum.

error := self ffiCall:  #(CXErrorCode clang_parseTranslationUnit2(CXIndex cxIndex,
                          String sourceFile,
                          String *argv,
                          uint argc,
                          CXUnsavedFile *unsaved_files,
                          uint num_unsaved_files,
                          CXTranslationUnitFlag options,
 CXTranslationUnit *out_TU)).

(error = CXError_Success) ifTrue: [ ^out_TU ].
self error: error

Thanks.


On Oct 22, 2017, at 2:42 AM, Esteban Lorenzano <[hidden email]> wrote:

Hi,

On 21 Oct 2017, at 21:15, Stephane Ducasse <[hidden email]> wrote:

Esteban will probably reply to this thread

yes, I will :)



On Thu, Oct 19, 2017 at 10:34 PM, Todd Blanchard <[hidden email]> wrote:
I have found the problem with VoidPointer3 generating accessors.

can you report that on fogbugz ?
(along with your solution)

thanks!
Esteban


No idea how to contribute back a fix but this is what I've found.

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer ifTrue: [self externalTypeWithArity
printString] ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

Recalling that we are trying to come up with an accessor that can pull out a
void*[3], this produces

'(FFITypeArray ofType: #FFIVoid size: 3)'

which produces an error as FFIVoid's size is undefined.  In general, this
will be wrong for any pointer type and will probably get the size
calculation wrong.

Doing a little digging I find that

FFITypeArray>>ofType: aType size: aSize

delegates resolution of aType to FFIExternalType>>resolveType:aType and this
can take all kinds of different things including the native type name.

It would be better if this generated the native name for pointers.

(FFITypeArray ofType: 'void*' size: 3)

So I changed it to:

FFITypeArrayType>>annonymousClassCreator
^ String streamContents: [ :stream |
stream
nextPutAll: '(FFITypeArray ofType: ';
print: (self objectClass type isPointer
ifTrue: [self externalTypeWithArity printString]
ifFalse: ['#',self objectClass type class]);
nextPutAll: ' size: ';
print: self objectClass numberOfElements;
nextPutAll: ')' ]

and this seems to work fine.

Onwards...

On Oct 19, 2017, at 7:18 AM, Todd Blanchard <[hidden email]> wrote:

That’s great - it’s been kind of magical but a couple things have changed
since you wrote it and some bugs have crept into ffi.

Right now I’m finding the generated accessor for the VoidPointer3 is
actually generating a void 3 accessor and that doesn’t work. I spent all day
yesterday tracking it to the accessor generating code.

Sent from the road

On Oct 18, 2017, at 22:29, Ben Coman <[hidden email]> wrote:



On Thu, Oct 19, 2017 at 1:05 AM, Todd Blanchard <[hidden email]> wrote:

I'm working through Ben's great blog post about playing with libclang and
I am puzzled by something.


Thx Todd. Knowing someone is looking at it encourages me to expand it.
I'm interested in looking at your specific questions, but it might not be
until next week after this stint of long work days.

cheers -ben




invalidateSessionData
handle atAllPut: 0.

zero;s out the handle.  Cool.  However,

handle isNull

does not return true despite it being used in the getString method as

^ handle isNull
      ifTrue: ['external memory invalidated by session restart']
      ifFalse:[LibClang clang_getCString__cxString: self].

Looks like there should be an isNull on ByteArray that returns true if all
bytes are zero but it isn't  there.  Was it dropped for some reason?



On Oct 18, 2017, at 2:58 AM, Dimitris Chloupis <[hidden email]>
wrote:

Sure the documentation could be better, that is definetly important, but
is already good enough and UFFI is a very technical subject much more suited
to a mailing list . Its not physical possible to cover the massive potential
of UFFI.

On Wed, Oct 18, 2017 at 9:32 AM Stephane Ducasse <[hidden email]>
wrote:

Please do not hesitate to do Pull Requests.
Luc told me that he wants to do a pass on it and Esteban promises that to
me
but he is super busy.

Stef


On Wed, Oct 18, 2017 at 5:54 AM, Todd Blanchard <[hidden email]>
wrote:
Wonderful!  Thanks.

On Oct 17, 2017, at 3:45 PM, stephan <[hidden email]> wrote:

On 17-10-17 23:06, Todd Blanchard wrote:
Anyone know what happened to this?

https://ci.inria.fr/pharo-contribution/view/Books/job/PharoBookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/UnifiedFFI.pdf

https://github.com/SquareBracketAssociates/Booklet-uFFI

has a link to a bintray pdf download

Stephan