uFFI ExternalAddress challenges

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

uFFI ExternalAddress challenges

eftomi
Dear all,

I'm struggling with passing an ExternalAddress to a c function call with uFFI. Firstly, I get the ExternalAddress from this method:

myFFI>>create: aString
    ^ self
        ffiCall: #(void * CreateObject (String aString) ) 

[in c:
void* CreateObject(char* szProgId)]

The method that uses the resulting ExternalAddress is defined as:

myFFI>>ask: anExternalAddress
    ^ self
        ffiCall: #(void CallMethod (void * anExternalAddress) )

[in c:
void CallMethod(void* myObj)]

In playground I have

| w |
w := myFFI create: 'Word.Application'.
myFFI ask: w .

w gets a "nice", properly looking external address, however the last line crushes Pharo, its window gets closed.

I went through the uFFI book, however I cannot find the answer.

I'm on Windows 10 x64, Pharo 7.0.4 32-bit.

Best wishes,
Tomaz
Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

Guillermo Polito
Hi Thomas, 

Can you share more about the definition of your CallMethod function in C?
From the uFFI point of view the binding looks ok…

How are you opening Pharo? From the command line or the pharo launcher?
Are you on cygwin/mingw? One other possibility (since you’re playing with C libraries) is to launch Pharo inside a gdb/lldb so you can check the reason behind the error when it crashes.

Keep us posted,
Guille

El 10 sept 2019, a las 19:53, Tomaž Turk <[hidden email]> escribió:

Dear all,

I'm struggling with passing an ExternalAddress to a c function call with uFFI. Firstly, I get the ExternalAddress from this method:

myFFI>>create: aString
    ^ self 
        ffiCall: #(void * CreateObject (String aString) ) 

[in c:
void* CreateObject(char* szProgId)]

The method that uses the resulting ExternalAddress is defined as:

myFFI>>ask: anExternalAddress
    ^ self 
        ffiCall: #(void CallMethod (void * anExternalAddress) )

[in c:
void CallMethod(void* myObj)]

In playground I have

| w |
w := myFFI create: 'Word.Application'.
myFFI ask: w .

w gets a "nice", properly looking external address, however the last line crushes Pharo, its window gets closed.

I went through the uFFI book, however I cannot find the answer.

I'm on Windows 10 x64, Pharo 7.0.4 32-bit.

Best wishes,
Tomaz

Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
Hi, thanks for your quick reply. I found the culprint while writing this email :-)

Generally, what I'm trying to do is to make a very basic package to create and communicate with COM components on Windows, mostly as an exercise :-) I found this library: http://disphelper.sourceforge.net/ as a great start, so you don't have to fiddle with OLE/COM headaches.

The problematic function in C is defined as:

__declspec(dllexport) void CallMethod(void* myObj) {
    ...
}

It takes a pointer to a COM object, which was previously returned by 

__declspec(dllexport) void* CreateObject(char* szProgId) {
...
   IDispatch * myObj = NULL
...
   return &myObj;
}

So, CreateObject() creates COM object and returns a pointer to it, CallMethod() takes this pointer and tries to communicate with it. 

My error was in returning a pointer to a pointer ... since IDispatch * myObj = NULL was actually hidden in a macro and I missed its true definition, huh. 

I'm running Pharo from PharoLauncher directly on Win10, without cygwin/mingw. For DLL I use Visual Studio 2019. I was not sure whether the pointer survives from one FFI call to another, or how Pharo loads/unloads DLLs - but it works as expected.

Thanks again and best wishes,
Tomaz

------ Original Message ------
From: "Guillermo Polito" <[hidden email]>
To: "Tomaž Turk" <[hidden email]>; "Any question about pharo is welcome" <[hidden email]>
Sent: 11. 09. 2019 10:29:34
Subject: Re: [Pharo-users] uFFI ExternalAddress challenges

Hi Thomas, 

Can you share more about the definition of your CallMethod function in C?
From the uFFI point of view the binding looks ok…

How are you opening Pharo? From the command line or the pharo launcher?
Are you on cygwin/mingw? One other possibility (since you’re playing with C libraries) is to launch Pharo inside a gdb/lldb so you can check the reason behind the error when it crashes.

Keep us posted,
Guille

Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

Guillermo Polito


El 11 sept 2019, a las 11:07, Tomaž Turk <[hidden email]> escribió:

Hi, thanks for your quick reply.

no problem ;)

I found the culprint while writing this email :-)

Cool!


Generally, what I'm trying to do is to make a very basic package to create and communicate with COM components on Windows, mostly as an exercise :-) I found this library: http://disphelper.sourceforge.net/ as a great start, so you don't have to fiddle with OLE/COM headaches.

The problematic function in C is defined as:

__declspec(dllexport) void CallMethod(void* myObj) {
    ...
}

It takes a pointer to a COM object, which was previously returned by 

__declspec(dllexport) void* CreateObject(char* szProgId) {
...
   IDispatch * myObj = NULL
...
   return &myObj;
}

So, CreateObject() creates COM object and returns a pointer to it, CallMethod() takes this pointer and tries to communicate with it. 

My error was in returning a pointer to a pointer ... since IDispatch * myObj = NULL was actually hidden in a macro and I missed its true definition, huh. 

Yes, looking at the code above it seems strange returning the &myObj, being myObj a local variable.
Because that means that you would be returning a pointer to the stack, from a frame that already returned...


I was not sure whether the pointer survives from one FFI call to another

Well, the pointer survives, the area of memory pointed by that pointer will be recycled with subsequent calls in general.

, or how Pharo loads/unloads DLLs - but it works as expected.

In general Pharo will load the library on usage, so you don’t have to care about it.
I’m sure the library is not unloaded, unless it is done explicitly (you can check VirtualMachine >> unloadModule:)


Thanks again and best wishes,
Tomaz

Cheers!

Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
>  Well, the pointer survives, the area of memory pointed by that pointer will be recycled with subsequent calls in general.

I'll pay special attention to that, but it's probably a matter of properly creating and releasing COM objects.


I have another question - in FFI call, how would one send a wide string, as in 

void CallMethod(void * COMObj, wchar_t * MethodName)

I might have additional questions about UTF-8 and other code page nuances ...

Best wishes,
Tomaz
Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

Guillermo Polito


El 11 sept 2019, a las 11:43, Tomaž Turk <[hidden email]> escribió:

>  Well, the pointer survives, the area of memory pointed by that pointer will be recycled with subsequent calls in general.

I'll pay special attention to that, but it's probably a matter of properly creating and releasing COM objects.


I have another question - in FFI call, how would one send a wide string, as in 

void CallMethod(void * COMObj, wchar_t * MethodName)

I might have additional questions about UTF-8 and other code page nuances …

:D Windows is particularly different than other platforms.
In *nixes most APIs require a char* with the encoded bytes, which we can encode in Pharo doing something like

‘my string’ utf8Encoded => a byte array with the encoded string

But for windows, I think the best is to use their encoding APIs.
You’ll find the image already has some support for that, that we use to access environment variables and the working directory string.

Check the class Win32WideString and its class comment ;)


Best wishes,
Tomaz

Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
Excellent, thanks!

Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
In reply to this post by eftomi
> We are interested in it :)
> If I recall well Pablo got one version somewhere.

Me too :-)  Unfortunately, I only have time for 'weekend' projects, and any help is appreciated :-)

Regarding the uFFI calls, is it possible to pass Win32WideString in a similar fashion as a String - or where to look to implement that?

Best wishes,
Tomaz
Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

Torsten Bergmann
From your snippets it looks like you want to do COM Automation on Windows.

Maybe you should have a look at: https://github.com/tesonep/pharo-com

Bye
T.
 
Gesendet: Donnerstag, 12. September 2019 um 11:08 Uhr
Von: "Tomaž Turk" <[hidden email]>
An: "Any question about pharo is welcome" <[hidden email]>
Betreff: Re: [Pharo-users] uFFI ExternalAddress challenges
> We are interested in it :)
> If I recall well Pablo got one version somewhere.
 
Me too :-)  Unfortunately, I only have time for 'weekend' projects, and any help is appreciated :-)
 
Regarding the uFFI calls, is it possible to pass Win32WideString in a similar fashion as a String - or where to look to implement that?
 
Best wishes,
Tomaz
Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

Ben Coman


On Thu, 12 Sep 2019 at 18:08, Torsten Bergmann <[hidden email]> wrote:
From your snippets it looks like you want to do COM Automation on Windows.

Maybe you should have a look at: https://github.com/tesonep/pharo-com

Does that by chance include an implementation of "MS-CFB" Microsoft Compound File Binary 

I was wanting to write a utility to parse headers of MSG files saved from Outlook,
to process a for a project email archive by bulk renaming each MSG file with its transmit-date & subject.

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
In reply to this post by Torsten Bergmann
> Maybe you should have a look at: https://github.com/tesonep/pharo-com

Thanks for the link, I'll check it out - that's probably the work Steph has mentioned.

Best wishes,
Tomaz
Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
In reply to this post by Torsten Bergmann
OK, I'm proceeding with the library that I mentioned (http://disphelper.sourceforge.net/) and I got nice results. Pablo's package goes directly into the core of OLE/COM automation and it would be too hard to for me to study it and continue with it at this stage. 

An example below loads Word for Windows, makes it visible, creates a new document and writes some text:

| w |
COMEngineAlpha initializeCOM .
w := COMEngineAlpha createObjectByName: 'Word.Application'.
COMEngineAlpha callObject: w setProperty: '.Visible = %b' withInteger: 1.
COMEngineAlpha callObject: w method: '.Documents.Add'.
COMEngineAlpha callObject: w method: '.Selection.TypeText(%S)' withString: 'This is a message to appear in Word document.'.
COMEngineAlpha safeReleaseObject: w.
COMEngineAlpha uninitializeCOM.

However, I'd like to marshall a WideString through uFFI so that I won't loose UTF-8 support, of course the plain (String aString) signature doesn't work ("Cannot coerce arguments"). Is there anything I can do myself (i.e. without poking into Pharo virtual machine complexity) to solve this challenge?

Best wishes,
Tomaz

------ Original Message ------
From: "Torsten Bergmann" <[hidden email]>
Cc: "Any question about pharo is welcome" <[hidden email]>
Sent: 12. 09. 2019 12:07:34
Subject: Aw: Re: [Pharo-users] uFFI ExternalAddress challenges

From your snippets it looks like you want to do COM Automation on Windows.

Maybe you should have a look at: https://github.com/tesonep/pharo-com

Bye
T.
 
Gesendet: Donnerstag, 12. September 2019 um 11:08 Uhr
Von: "Tomaž Turk" <[hidden email]>
An: "Any question about pharo is welcome" <[hidden email]>
Betreff: Re: [Pharo-users] uFFI ExternalAddress challenges
> We are interested in it :)
> If I recall well Pablo got one version somewhere.
 
Me too :-)  Unfortunately, I only have time for 'weekend' projects, and any help is appreciated :-)
 
Regarding the uFFI calls, is it possible to pass Win32WideString in a similar fashion as a String - or where to look to implement that?
 
Best wishes,
Tomaz
Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
A rudimentary disphelper library "approach" is working OK, I succeeded to
create a connection to SQL Server through ADODB, too, which is rather cool,
since I've been looking for this functionality for too long.

I re-checked Pablo's pharo-com package and it is rather comprehensive - it
successfully creates COM servers and more. However, when trying to invoke
writing to COM server properties, as in:

wordObj propertyNamed: 'Visible' put: true.

the call fails with:

Primitive failed: primitive #integerAt:size:signed: in ExternalAddress

Problematic method uses a primitive

<primitive: 'primitiveFFIIntegerAt' module:'SqueakFFIPrims'>

which probably doesn't exist. Does anybody know about the purpose/intent and
the history of this primitive in 'SqueakFFIPrims'?

I think it would be a pitty that we leave Pablo's work unfinished.



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
It seems that there's nothing wrong with the primitive, but with Win32Variant
creation in ExternalStructure class>>fromHandle:, specifically with basicNew
and Win32Variant class as receiver.

In what circumstances the debugger/inspector shows 'error printing' message
(on the result of basicNew)?

Sorry for newbie questions :-)

Best wishes,
Tomaz



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

Guillermo Polito
In reply to this post by eftomi
Hi Tomaz,

Have you checked Win32WideString in Pharo7/8 and its users?
You can do

aWindowsWideString := ‘a wide string’ asWin32WideString.

And then check the senders of Win32WideString, you will find you can declare ffi signatures like this

removeEnvironmentVariable: nameString

^ self ffiCall: #( int SetEnvironmentVariableW ( Win32WideString nameString, 0 ) )

Isn’t this working for you?
If not, I’d like to have more details on why it does not work, so we can figure out a solution :)

Cheers,
Guille


El 13 sept 2019, a las 15:22, Tomaž Turk <[hidden email]> escribió:

OK, I'm proceeding with the library that I mentioned (http://disphelper.sourceforge.net/) and I got nice results. Pablo's package goes directly into the core of OLE/COM automation and it would be too hard to for me to study it and continue with it at this stage. 

An example below loads Word for Windows, makes it visible, creates a new document and writes some text:

| w |
COMEngineAlpha initializeCOM .
w := COMEngineAlpha createObjectByName: 'Word.Application'.
COMEngineAlpha callObject: w setProperty: '.Visible = %b' withInteger: 1.
COMEngineAlpha callObject: w method: '.Documents.Add'.
COMEngineAlpha callObject: w method: '.Selection.TypeText(%S)' withString: 'This is a message to appear in Word document.'.
COMEngineAlpha safeReleaseObject: w.
COMEngineAlpha uninitializeCOM.

However, I'd like to marshall a WideString through uFFI so that I won't loose UTF-8 support, of course the plain (String aString) signature doesn't work ("Cannot coerce arguments"). Is there anything I can do myself (i.e. without poking into Pharo virtual machine complexity) to solve this challenge?

Best wishes,
Tomaz

------ Original Message ------
From: "Torsten Bergmann" <[hidden email]>
Cc: "Any question about pharo is welcome" <[hidden email]>
Sent: 12. 09. 2019 12:07:34
Subject: Aw: Re: [Pharo-users] uFFI ExternalAddress challenges

From your snippets it looks like you want to do COM Automation on Windows.

Maybe you should have a look at: https://github.com/tesonep/pharo-com

Bye
T.
 
Gesendet: Donnerstag, 12. September 2019 um 11:08 Uhr
Von: "Tomaž Turk" <[hidden email]>
An: "Any question about pharo is welcome" <[hidden email]>
Betreff: Re: [Pharo-users] uFFI ExternalAddress challenges
> We are interested in it <emoticon_smiley.png>
> If I recall well Pablo got one version somewhere.
 
Me too <emoticon_smiley.png>  Unfortunately, I only have time for 'weekend' projects, and any help is appreciated <emoticon_smiley.png>
 
Regarding the uFFI calls, is it possible to pass Win32WideString in a similar fashion as a String - or where to look to implement that?
 
Best wishes,
Tomaz

Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
OK, great, I'll try it out. 

Thanks,
Tomaz

------ Original Message ------
From: "Guillermo Polito" <[hidden email]>
To: "Tomaž Turk" <[hidden email]>
Sent: 17.9.2019 11:56:21
Subject: Re: [Pharo-users] uFFI ExternalAddress challenges

I don’t know if there is support for that (I don’t have the time to check it in detail now).
But, as a workaround, you may try

 - return a void*
    
  myFunction
    ^self ffiCall: #(void* myFunction ( void ) ) 

  - and then do the transformation yourself using Win32WideString>>#fromHandle:
  
Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

Stephan Eggermont-3
Tomaž Turk <[hidden email]> wrote:
> OK, great, I'll try it out.


Hi Tomaž,

What you are discovering here would be very useful as a section/chapter in
the uFFI booklet. Just posting it as text here will do if you don’t feel up
to translating to Pillar.

Cheers
  Stephan



Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
Hi,

> I don’t know if there is support for that (I don’t have the time to check it in detail now).
> But, as a workaround, you may try
 > - return a void*
 >    
 >  myFunction
 >    ^self ffiCall: #(void* myFunction ( void ) ) 
>   - and then do the transformation yourself using Win32WideString>>#fromHandle:

I checked, Win32WideString class>>#asExternalTypeOn: is defined, and it creates FFIExternalObjectType(Win32WideString) nicely with

^self ffiCall: #(Win32WideString myFunction ( void ) ) 

I also tried the other approach:

^self ffiCall: #(void* myFunction ( void ) ) 

and the transformation Win32WideString>>#fromHandle 

Interestingly, in both cases I get 


however I cannot pinpoint the reason for this. 

> What you are discovering here would be very useful as a section/chapter in
> the uFFI booklet.

Thanks for the invitation, yes, it would be nice to participate, I'll try to prepare something when I get to the bottom of the challenge :-) 

Thanks and best wishes,
Tomaz


Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
Hi, after some more investigation the easiest way to get Win32WideString from
external module is to pass it as a parameter, like in:

^self ffiCall: #( void myFunction ( Win32WideString aWin32WideString ) )

and the external module can access it directly to do r/w.

There are other possibilities as well - like adapting the method
Win32WideString>>#fromHandle: to translate from ExternalData to
Win32WideString, creating new ExternalType, or even creating a new
FFIWin32WideString. I wonder what would be the best approach :-)

Best wishes,
Tomaz



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Reply | Threaded
Open this post in threaded view
|

Re: uFFI ExternalAddress challenges

eftomi
Hi,

as a wrap-up for this forum thread, I found an elegant way to take a result
as C wchar_t* from an FFI call on Windows, based on the idea of
ExternalData>>#fromCString. If anybody needs it:
https://github.com/eftomi/pharo-uFFI-readWin32WideString.

Best wishes,
Tomaz



--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html