Smalltalk › Pharo › Pharo Smalltalk Users

Accessing DLLs in Smalltalk/V

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

13 messages Options

Bob Williams

Accessing DLLs in Smalltalk/V

I found an old Smalltalk/V imagine from 2000 and tracked down how I was able add calls to Windows DLLs and to add my own. It was in the class DynamicLinkLibrary and its subclasses, e. g., a call to the GDIDLL to draw a Bezier spline (which I believe I added) was:

bezierTo: hDC lpPoints: anLPPOINT count: anInt

<api: PolyBezierTo ulong struct ulong boolean>

^self invalidArgument

I added by own DLL to do matrix transformations that was written in ASSEMBLER to optimized the floating point processor pipelining (a start operation every cycle while it took 3 cycles to complete) and it performed about 15% better.

I have not found anything similar in Pharo, but it must if external libraries are used. NativeBoost seems to come close, but I cannot find any documentation. Can someone point me to some documentation, code, anything? It is greatly appreciated.

kilon.alios

Re: Accessing DLLs in Smalltalk/V

The PharoForTheEnterprise book has a chapter on Nativeboost.

You can find the book here in pdf format

https://ci.inria.fr/pharo-contribution/job/PharoForTheEnterprise/lastSuccessfulBuild/artifact/

also you can click on the folder to download only the Nativeboost chapter, but I advice to download the whole book since it has tons of useful info.

On Thu, May 29, 2014 at 5:15 PM, Bob Williams <[hidden email]> wrote:

I found an old Smalltalk/V imagine from 2000 and tracked down how I was able add calls to Windows DLLs and to add my own. It was in the class DynamicLinkLibrary and its subclasses, e. g., a call to the GDIDLL to draw a Bezier spline (which I believe I added) was:

bezierTo: hDC lpPoints: anLPPOINT count: anInt

<api: PolyBezierTo ulong struct ulong boolean>

^self invalidArgument

I added by own DLL to do matrix transformations that was written in ASSEMBLER to optimized the floating point processor pipelining (a start operation every cycle while it took 3 cycles to complete) and it performed about 15% better.

I have not found anything similar in Pharo, but it must if external libraries are used. NativeBoost seems to come close, but I cannot find any documentation. Can someone point me to some documentation, code, anything? It is greatly appreciated.

bw

stepharo

Re: Accessing DLLs in Smalltalk/V

In reply to this post by Bob Williams

For nativeBoost have a look at the tutorial to use NB to bind X11

https://ci.inria.fr/pharo-contribution/job/PharoForTheEnterprise/lastSuccessfulBuild/artifact/NativeBoost/NativeBoost.pier.html

I'm sorry but if igor does not write documentation, it takes far too much energy from me to understand
and write something. It would be so much better if people with the knowledge would write.

Stef

On 29/5/14 16:15, Bob Williams wrote:

I found an old Smalltalk/V imagine from 2000 and tracked down how I was able add calls to Windows DLLs and to add my own. It was in the class DynamicLinkLibrary and its subclasses, e. g., a call to the GDIDLL to draw a Bezier spline (which I believe I added) was:

bezierTo: hDC lpPoints: anLPPOINT count: anInt

<api: PolyBezierTo ulong struct ulong boolean>

^self invalidArgument

I added by own DLL to do matrix transformations that was written in ASSEMBLER to optimized the floating point processor pipelining (a start operation every cycle while it took 3 cycles to complete) and it performed about 15% better.

I have not found anything similar in Pharo, but it must if external libraries are used. NativeBoost seems to come close, but I cannot find any documentation. Can someone point me to some documentation, code, anything? It is greatly appreciated.

bw

Clément Béra

Re: Accessing DLLs in Smalltalk/V

In reply to this post by Bob Williams

Hello,

To do what you want, there are 3 options:

- using NativeBoost

- using FFI from the VM

- compiling your dll as part of the vm (VM plugin)

NativeBoost is clearly the best option because it is the fastest, it is mostly implemented in the image so you can look at the code and it is by default in the image.

FFI from the VM typically manages much better callbacks, so if you plan to extensively use callbacks you should look at it, by loading the package of Aliens (It does not rely any more on the old implementation of aliens but still have the same image side API and library). Alternatively you can improve NativeBoost so it manages better call back, this is "easy" as it is mostly implemented in the image.

Compiling your dll as a VM plugin is usually not the best option, but it can be useful in some situation (uncommon OS or uncommon hardware for example).

I explained how to compile and bind a dll with NativeBoost here (subsection Implementing the FFI calls) :

http://clementbera.wordpress.com/2013/06/19/optimizing-pharo-to-c-speed-with-nativeboost-ffi/

It is a very basic but very simple example.

Additional examples can be found in the class NBBasicExamples in the package NativeBoost-Examples.

There are also tests (For example NBFFICallbackTests>>#benchQSort) that you can look at, they work and show how to use NativeBoost.

Else there is the chapter in Pharo for the entrerprise. Other people will give you a link.

2014-05-29 16:15 GMT+02:00 Bob Williams <[hidden email]>:

I found an old Smalltalk/V imagine from 2000 and tracked down how I was able add calls to Windows DLLs and to add my own. It was in the class DynamicLinkLibrary and its subclasses, e. g., a call to the GDIDLL to draw a Bezier spline (which I believe I added) was:

bezierTo: hDC lpPoints: anLPPOINT count: anInt

<api: PolyBezierTo ulong struct ulong boolean>

^self invalidArgument

I added by own DLL to do matrix transformations that was written in ASSEMBLER to optimized the floating point processor pipelining (a start operation every cycle while it took 3 cycles to complete) and it performed about 15% better.

I have not found anything similar in Pharo, but it must if external libraries are used. NativeBoost seems to come close, but I cannot find any documentation. Can someone point me to some documentation, code, anything? It is greatly appreciated.

bw

Joachim Geidel

Re: Accessing DLLs in Smalltalk/V

Hello Clément,

Am 29.05.2014 um 17:23 schrieb Clément Bera <[hidden email]>:

NativeBoost is clearly the best option because it is the fastest, it is mostly implemented in the image so you can look at the code and it is by default in the image.

FFI from the VM typically manages much better callbacks, so if you plan to extensively use callbacks you should look at it, by loading the package of Aliens (It does not rely any more on the old implementation of aliens but still have the same image side API and library). Alternatively you can improve NativeBoost so it manages better call back, this is "easy" as it is mostly implemented in the image.

Could you explain why FFI is better at managing callbacks? I have used NativeBoost or implementing JNIPort, and callbacks were one of the easier parts of the work I had to do. As far as I know, both FFI and NativeBoost don’t support callbacks from threads other than the thread in which the Smalltalk code is executed, but that’s a limitation of both of them.

As to improving NativeBoost: That’s not „easy“ at all. Some parts of it are quite cryptic if you are not familiar with x86 assembler code, the documentation is still sparse, and I am under the impression that there are some unfinished parts. It is practically impossible to add something to NativeBoost if you are not Igor, and just want to use it for your own project as opposed to spending many hours trying to figure out how NativeBoost works. I was somewhat lucky that I found everything I needed in the archives of the pharo-dev and pharo-users mailing list., but i wouldn’t call this documentation.

I explained how to compile and bind a dll with NativeBoost here (subsection Implementing the FFI calls) :

http://clementbera.wordpress.com/2013/06/19/optimizing-pharo-to-c-speed-with-nativeboost-ffi/
It is a very basic but very simple example.

Yes, this page helped, but as you wrote: basic and simple example - the more involved stuff remains undocumented. When working on JNIPort, I struggled with simple things like de-referencing a pointer to a pointer to a struct etc. - I didn’t find documentation for this, and this is a basic feature in my opinion.

Best regards,

Joachim Geidel

Additional examples can be found in the class NBBasicExamples in the package NativeBoost-Examples.
There are also tests (For example NBFFICallbackTests>>#benchQSort) that you can look at, they work and show how to use NativeBoost.

Else there is the chapter in Pharo for the entrerprise. Other people will give you a link.

stepharo

Re: Accessing DLLs in Smalltalk/V

Hi joachim

could you spend some time and helping us to write a documentation on nativeboost?
Else it will always stay the same. I cannot systematically try to write documentation on things I do not have expertise
because it is killing me.

For igor everything is obvious :) but we spent some afternoons improving the x11 tutorial.

Stef

On 30/5/14 08:50, Joachim Geidel wrote:

Hello Clément,

Am 29.05.2014 um 17:23 schrieb Clément Bera <[hidden email]>:

NativeBoost is clearly the best option because it is the fastest, it is mostly implemented in the image so you can look at the code and it is by default in the image.

FFI from the VM typically manages much better callbacks, so if you plan to extensively use callbacks you should look at it, by loading the package of Aliens (It does not rely any more on the old implementation of aliens but still have the same image side API and library). Alternatively you can improve NativeBoost so it manages better call back, this is "easy" as it is mostly implemented in the image.

Could you explain why FFI is better at managing callbacks? I have used NativeBoost or implementing JNIPort, and callbacks were one of the easier parts of the work I had to do. As far as I know, both FFI and NativeBoost don’t support callbacks from threads other than the thread in which the Smalltalk code is executed, but that’s a limitation of both of them.

As to improving NativeBoost: That’s not „easy“ at all. Some parts of it are quite cryptic if you are not familiar with x86 assembler code, the documentation is still sparse, and I am under the impression that there are some unfinished parts. It is practically impossible to add something to NativeBoost if you are not Igor, and just want to use it for your own project as opposed to spending many hours trying to figure out how NativeBoost works. I was somewhat lucky that I found everything I needed in the archives of the pharo-dev and pharo-users mailing list., but i wouldn’t call this documentation.

I explained how to compile and bind a dll with NativeBoost here (subsection Implementing the FFI calls) :

http://clementbera.wordpress.com/2013/06/19/optimizing-pharo-to-c-speed-with-nativeboost-ffi/

It is a very basic but very simple example.

Yes, this page helped, but as you wrote: basic and simple example - the more involved stuff remains undocumented. When working on JNIPort, I struggled with simple things like de-referencing a pointer to a pointer to a struct etc. - I didn’t find documentation for this, and this is a basic feature in my opinion.

Best regards,

Joachim Geidel

Additional examples can be found in the class NBBasicExamples in the package NativeBoost-Examples.

There are also tests (For example NBFFICallbackTests>>#benchQSort) that you can look at, they work and show how to use NativeBoost.

Else there is the chapter in Pharo for the entrerprise. Other people will give you a link.

kilon.alios

Re: Accessing DLLs in Smalltalk/V

he has not provided documentation for Nativeboost that's true, but he never left a question unanswered either ;)

I think I will come around eventually to adding documentation to Nativeboost.

Imagine every Pharoer adding 10 lines of documentation every day. Say 100 people . That's 33 pages per day , 12167 pages per year. Thats 30 books of the size of PBE per year. An insane amount of documentation.

On Fri, May 30, 2014 at 10:57 AM, stepharo <[hidden email]> wrote:

Hi joachim

could you spend some time and helping us to write a documentation on nativeboost?
Else it will always stay the same. I cannot systematically try to write documentation on things I do not have expertise
because it is killing me.

For igor everything is obvious :) but we spent some afternoons improving the x11 tutorial.

Stef

On 30/5/14 08:50, Joachim Geidel wrote:

Hello Clément,

Am 29.05.2014 um 17:23 schrieb Clément Bera <[hidden email]>:

NativeBoost is clearly the best option because it is the fastest, it is mostly implemented in the image so you can look at the code and it is by default in the image.

FFI from the VM typically manages much better callbacks, so if you plan to extensively use callbacks you should look at it, by loading the package of Aliens (It does not rely any more on the old implementation of aliens but still have the same image side API and library). Alternatively you can improve NativeBoost so it manages better call back, this is "easy" as it is mostly implemented in the image.

Could you explain why FFI is better at managing callbacks? I have used NativeBoost or implementing JNIPort, and callbacks were one of the easier parts of the work I had to do. As far as I know, both FFI and NativeBoost don’t support callbacks from threads other than the thread in which the Smalltalk code is executed, but that’s a limitation of both of them.

As to improving NativeBoost: That’s not „easy“ at all. Some parts of it are quite cryptic if you are not familiar with x86 assembler code, the documentation is still sparse, and I am under the impression that there are some unfinished parts. It is practically impossible to add something to NativeBoost if you are not Igor, and just want to use it for your own project as opposed to spending many hours trying to figure out how NativeBoost works. I was somewhat lucky that I found everything I needed in the archives of the pharo-dev and pharo-users mailing list., but i wouldn’t call this documentation.

I explained how to compile and bind a dll with NativeBoost here (subsection Implementing the FFI calls) :

http://clementbera.wordpress.com/2013/06/19/optimizing-pharo-to-c-speed-with-nativeboost-ffi/

It is a very basic but very simple example.

Yes, this page helped, but as you wrote: basic and simple example - the more involved stuff remains undocumented. When working on JNIPort, I struggled with simple things like de-referencing a pointer to a pointer to a struct etc. - I didn’t find documentation for this, and this is a basic feature in my opinion.

Best regards,

Joachim Geidel

Additional examples can be found in the class NBBasicExamples in the package NativeBoost-Examples.

There are also tests (For example NBFFICallbackTests>>#benchQSort) that you can look at, they work and show how to use NativeBoost.

Else there is the chapter in Pharo for the entrerprise. Other people will give you a link.

Joachim Geidel

Re: Accessing DLLs in Smalltalk/V

Hi all,

Am 30.05.2014 um 10:30 schrieb kilon alios <[hidden email]>:

he has not provided documentation for Nativeboost that's true, but he never left a question unanswered either ;)

just to make sure that my message isn’t misunderstood: I am not blaming Igor for not having written a book about NativeBoost (yet), and I am very thankful that he wrote NativeBoost in the first place. I know how hard it is to maintain a piece of software over an extended period of time. You have to work on a full-time job, earn money, have some remainders of a life etc. It’s sometimes easier for academics as they may have some more freedom what they work on (but not necessarily so). If Chris Uppal had not written the excellent documentation of JNIPort for Dolphin before I took over maintenance, the JNIPort documentation would be in the same state as the NativeBoost docs. I simply don’t have enough spare time for it.

I think I will come around eventually to adding documentation to Nativeboost.

Excellent, thanks in advance! I can help in proof reading, but don’t expect too much - see above.

Imagine every Pharoer adding 10 lines of documentation every day. Say 100 people . That's 33 pages per day , 12167 pages per year. Thats 30 books of the size of PBE per year. An insane amount of documentation.

If it were so easy… To write NativeBoost docs, you have to understand what you are writing about in the first place, and you have to have a plan - chapter structure, consistent examples, ideally everything explained in the context of a typical problem. The current tutorial in already very good in this respect. But that’s hard and takes time, and it can’t be split up easily among more than three people - which accidentally seems to be the approximate number of knowledgeable NativeBoost experts on this planet who actually could actually do it.

On Fri, May 30, 2014 at 10:57 AM, stepharo <[hidden email]> wrote:

Hi joachim

could you spend some time and helping us to write a documentation on nativeboost?
Else it will always stay the same. I cannot systematically try to write documentation on things I do not have expertise
because it is killing me.

Same for me, I am just a NativeBoost amateur, and unfortunately my time budget is severely limited. I would prefer spending my time on enhancing JNIPort (callbacks are currently too complicated). But I can help in proofreading.

Joachim

kilon.alios

Re: Accessing DLLs in Smalltalk/V

"just to make sure that my message isn’t misunderstood: I am not blaming Igor for not having written a book about NativeBoost (yet), and I am very thankful that he wrote NativeBoost in the first place. I know how hard it is to maintain a piece of software over an extended period of time. You have to work on a full-time job, earn money, have some remainders of a life etc. It’s sometimes easier for academics as they may have some more freedom what they work on (but not necessarily so). If Chris Uppal had not written the excellent documentation of JNIPort for Dolphin before I took over maintenance, the JNIPort documentation would be in the same state as the NativeBoost docs. I simply don’t have enough spare time for it.

good but I was not replying to you, I was replying to Stephan.

"Excellent, thanks in advance! I can help in proof reading, but don’t expect too much - see above."

I never expect anything from anyone, even when I complain about it ;). I am already porting PBE to Pharo 3 and doing video tutorials on Pharo 3 so my hands are full too.

"If it were so easy… To write NativeBoost docs, you have to understand what you are writing about in the first place, and you have to have a plan - chapter structure, consistent examples, ideally everything explained in the context of a typical problem. The current tutorial in already very good in this respect. But that’s hard and takes time, and it can’t be split up easily among more than three people - which accidentally seems to be the approximate number of knowledgeable NativeBoost experts on this planet who actually could actually do it."

yeah its not that easy, that's true, its easier, write a line a day, Your philosophy about well structured documentation is of course ideal but this comes down to Well Structure but little documentation vs loads but messy documentation. Which in that case I pick the second option with my eyes closed. Which you think most people will pick ? Plus most documentation is very messy anyway. I am a lawyer as professional and I deal with messy documentation from day to day basis, its not that I don't hate i, I do, but the alternative of no documentation is simply not an option to me. And if you think about it messy documentation still beats google search hands down. So never underestimate the immense value of messy documentation.

The idea that Nativeboost itself is a tricky subject, well sure you need to understand how C works and be able to tolerate all the manual management mess, but I can't say beyond that is that hard. I suspect however that the problem is that not that many people use Nativeboost in the first place.

On Fri, May 30, 2014 at 12:40 PM, Joachim Geidel <[hidden email]> wrote:

Hi all,

Am 30.05.2014 um 10:30 schrieb kilon alios <[hidden email]>:

he has not provided documentation for Nativeboost that's true, but he never left a question unanswered either ;)

just to make sure that my message isn’t misunderstood: I am not blaming Igor for not having written a book about NativeBoost (yet), and I am very thankful that he wrote NativeBoost in the first place. I know how hard it is to maintain a piece of software over an extended period of time. You have to work on a full-time job, earn money, have some remainders of a life etc. It’s sometimes easier for academics as they may have some more freedom what they work on (but not necessarily so). If Chris Uppal had not written the excellent documentation of JNIPort for Dolphin before I took over maintenance, the JNIPort documentation would be in the same state as the NativeBoost docs. I simply don’t have enough spare time for it.

I think I will come around eventually to adding documentation to Nativeboost.

Excellent, thanks in advance! I can help in proof reading, but don’t expect too much - see above.

Imagine every Pharoer adding 10 lines of documentation every day. Say 100 people . That's 33 pages per day , 12167 pages per year. Thats 30 books of the size of PBE per year. An insane amount of documentation.

If it were so easy… To write NativeBoost docs, you have to understand what you are writing about in the first place, and you have to have a plan - chapter structure, consistent examples, ideally everything explained in the context of a typical problem. The current tutorial in already very good in this respect. But that’s hard and takes time, and it can’t be split up easily among more than three people - which accidentally seems to be the approximate number of knowledgeable NativeBoost experts on this planet who actually could actually do it.

On Fri, May 30, 2014 at 10:57 AM, stepharo <[hidden email]> wrote:

Hi joachim

could you spend some time and helping us to write a documentation on nativeboost?
Else it will always stay the same. I cannot systematically try to write documentation on things I do not have expertise
because it is killing me.

Same for me, I am just a NativeBoost amateur, and unfortunately my time budget is severely limited. I would prefer spending my time on enhancing JNIPort (callbacks are currently too complicated). But I can help in proofreading.

Joachim

Sean P. DeNigris

Re: Accessing DLLs in Smalltalk/V

Administrator

In reply to this post by kilon.alios

kilon alios wrote

An insane amount of documentation

<rant>
"insane" may be the key term here. We could be inventing executable active essays, which is the kind of magic our live environment makes possible, instead of writing and rewriting paper docs that quickly go stale. Non-executable docs are a duplication smell and maintenance hassle. Text-only is one of the least-effective ways to communicate. I'd love to see more lively kernel style docs</rant>

Cheers,
Sean

kilon.alios

Re: Accessing DLLs in Smalltalk/V

great point , I think that Pillar already does this but its not exactly hassle free. I have to say I really like Pillar, very simple syntax, it generates pdf and hmtl ouput as well latex. Still will be an extra effort to bring it inside Pharo in a form of an editor. But frankly if you want to make your own documentation with Pillar is already very easy and the final result looks very good.

On Fri, May 30, 2014 at 6:02 PM, Sean P. DeNigris <[hidden email]> wrote:

kilon alios wrote
> An insane amount of documentation

<rant>
"insane" may be the key term here. We could be inventing executable active
essays, which is the kind of magic our live environment makes possible,
instead of writing and rewriting paper docs that quickly go stale.
Non-executable docs are a duplication smell and maintenance hassle.
Text-only is one of the least-effective ways to communicate. I'd love to see
more lively kernel style docs</rant>

-----
Cheers,
Sean
--
View this message in context: http://forum.world.st/Accessing-DLLs-in-Smalltalk-V-tp4760856p4761006.html
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

Markus Fritsche-4

Re: Accessing DLLs in Smalltalk/V

In reply to this post by Joachim Geidel

On 30.05.2014 08:50, Joachim Geidel wrote:
> Yes, this page helped, but as you wrote: basic and simple example -
> the more involved stuff remains undocumented. When working on JNIPort,
> I struggled with simple things like de-referencing a pointer to a
> pointer to a struct etc. - I didn’t find documentation for this, and
> this is a basic feature in my opinion.
Looking through blog posts and the mailing list archive, I didn't find
it so hard to work with (yet), but I am also just dealing with opaque
types. One thing I'd wish to learn more about is proper resource
management - the library I am using is constructing a tree of connected
structs, and I haven't found a recipe how to deal with that (integrating
with the pharo vm garbage collector).

<thread_hijack>Silly question: where can I find information about
finalization et al in Pharo/ Squeak? My google searches didn't yield any
results that where easy to understand.</thread_hijack>

Offray

Re: Accessing DLLs in Smalltalk/V

In reply to this post by kilon.alios

Hi,

About active essays and writing documentation inside Pharo, I'm trying
to bring some experience on writing using a tree/outliner like interface
for it. Questions are in another thread, but I hope to help with some
kind of "dynabooklets", for the moment using pandoc's markdown, wich
seems better known and suited for academical writing, but may be at some
point it can share some Pillar insights.

Cheers,

Offray

On 05/30/2014 11:12 AM, kilon alios wrote:

> great point , I think that Pillar already does this but its not exactly hassle
> free. I have to say I really like Pillar, very simple syntax, it generates pdf
> and hmtl ouput as well latex. Still will be an extra effort to bring it inside
> Pharo in a form of an editor. But frankly if you want to make your own
> documentation with Pillar is already very easy and the final result looks very
> good.
>
>
> On Fri, May 30, 2014 at 6:02 PM, Sean P. DeNigris <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> kilon alios wrote
> > An insane amount of documentation
>
> <rant>
> "insane" may be the key term here. We could be inventing executable active
> essays, which is the kind of magic our live environment makes possible,
> instead of writing and rewriting paper docs that quickly go stale.
> Non-executable docs are a duplication smell and maintenance hassle.
> Text-only is one of the least-effective ways to communicate. I'd love to see
> more lively kernel style docs</rant>
>
>
>
> -----
> Cheers,
> Sean
> --
> View this message in context:
> http://forum.world.st/Accessing-DLLs-in-Smalltalk-V-tp4760856p4761006.html
> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
>
>