Smalltalk › Pharo › Pharo Smalltalk Developers

NativeBoost examples (was: Re: DoubleArray)

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

12 messages Options

Igor Stasenko

NativeBoost examples (was: Re: DoubleArray)

On 23 September 2010 02:00, Schwab,Wilhelm K <[hidden email]> wrote:
> Sig,
>
> Fair enough :) The most recent thing I recall on NativeBoost is Stef asking for what I assume must be a Mac plugin; I thought Linux would be left out too, but it appears not, though it is not in my current vm. Beyond some list traffic, I found this:
>

The work for supporting Linux and Mac is in progress.

> http://code.google.com/p/nativeboost/wiki/NativeBoost
>
> Is there other documentation somewhere? Build instructions?

You can find all existing documentation, installation instructions on
wiki pages at the above site. There is no other place for it.
http://code.google.com/p/nativeboost/wiki/Installation - installation
instructions.

> You have a block copy, which is good. What about indexing, such as getting floats or doubles out of external memory?

This is just a piece of cake. ;)

readDoubleFrom: address
" This method loads the double from given external address.
an address can be an instance of NBExternalAddress, or
simple ByteArray with at least 8 bytes long, which holds a double
floating value"

<primitive: 'primitiveNativeCall' module: 'NativeBoostPlugin'>

"We are using a pseudo-function prototype and supplying own assembler code,
instead of making a call to external function.
In given example, an address argument , after coercion is pushed on stack. "

^ NBFFICallout

cdecl: #( double ( void * address) )
" A pseudo-function takes 1 argument, and returns double value.
Under cdecl call convention, all floating point return types is
returned in fp(0) CPU register"

emitCall: [:gen | | asm |
asm := gen asm.

"Here , we expecting that an address value is already pushed on stack"

asm pop: asm EAX; "load an address value into EAX register by
popping a stack"
fld: (asm EAX ptr64). "load a floating point value from memory, at
base address, held in EAX register into fp(0) register,
we are using #ptr64, to indicate that memory operand size is 64bits long"

" return value set, we are done. A code generator will take care
for emitting code, which converts
a double floating point value into smalltalk object. "
]

I just uploaded this example and also how to write doubles from/to memory.
See at SqS/NativeBoost - NativeBoost-Examples package, class NBBasicExamples.

> I am very comfortable adding functions to a library for integer and float/double calculations on "large" arrays and calling them via FFI; plugins are another story. Are there examples to follow?

Tell me, what examples you would like to see. I will gladly provide them.

--
Best regards,
Igor Stasenko AKA sig.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Schwab,Wilhelm K

Re: NativeBoost examples (was: Re: DoubleArray)

Sig,

The examples that make sense depend on what NativeBoost wants to be when it grows up. Is it a full competitor to FFI and/or Alien? If so, then it would be nice to see a rich set of calls to functions with different parameter types; ideally it would be in the form of something useful that gets added to Pharo using NativeBoost. That does a couple of things, providing both proof of concept and examples in one shot.

With tongue firmly in cheek, looking at your indexing example, my stack might pop before the machine's :) However, it looks like it should settle down quickly enough, and things like that go on in the vm all the time; you are arguably just brining it into the image. Where would #readDoubleFrom: be in the class hiearchy? One of the reasons I asked about indexing an external array is that FFI is not good at such things. If you can provide ways to copy, fill, etc. blocks of memory and do it quickly, you gain ground, assuming you want to compete with FFI.

Do callbacks, and you gain a lot of ground. Make calls on separate threads, and you gain a LOT of ground. That assumes that you can make it simple enough for a mildly above-average C programmer to understand how to use and ideally extend it. I will look at your example package on Squeak Source.

One thing that drives me nuts with the Linux vm is that it looks for things where it wants to look, and gives no feedback about what it tried to do or even that it failed (except that handle is still zero). Your callout pragmas take a module name; is there any way to see inside of that or to provide a full path to the library? It is nice to be able to confirm loading of a library w/o having to call something, since there won't always be something "safe" (free of side effects) to call as a test.

At the risk of seeming difficult, everything is academic until we have a Linux plugin or at least can build same. Alien has that problem.

Bill

________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Igor Stasenko [[hidden email]]
Sent: Wednesday, September 22, 2010 10:05 PM
To: [hidden email]
Subject: [Pharo-project] NativeBoost examples (was: Re: DoubleArray)

On 23 September 2010 02:00, Schwab,Wilhelm K <[hidden email]> wrote:
> Sig,
>
> Fair enough :) The most recent thing I recall on NativeBoost is Stef asking for what I assume must be a Mac plugin; I thought Linux would be left out too, but it appears not, though it is not in my current vm. Beyond some list traffic, I found this:
>

The work for supporting Linux and Mac is in progress.

> http://code.google.com/p/nativeboost/wiki/NativeBoost
>
> Is there other documentation somewhere? Build instructions?

You can find all existing documentation, installation instructions on
wiki pages at the above site. There is no other place for it.
http://code.google.com/p/nativeboost/wiki/Installation - installation
instructions.

> You have a block copy, which is good. What about indexing, such as getting floats or doubles out of external memory?

This is just a piece of cake. ;)

readDoubleFrom: address
" This method loads the double from given external address.
an address can be an instance of NBExternalAddress, or
simple ByteArray with at least 8 bytes long, which holds a double
floating value"

<primitive: 'primitiveNativeCall' module: 'NativeBoostPlugin'>

"We are using a pseudo-function prototype and supplying own assembler code,
instead of making a call to external function.
In given example, an address argument , after coercion is pushed on stack. "

^ NBFFICallout

cdecl: #( double ( void * address) )
" A pseudo-function takes 1 argument, and returns double value.
Under cdecl call convention, all floating point return types is
returned in fp(0) CPU register"

emitCall: [:gen | | asm |
asm := gen asm.

"Here , we expecting that an address value is already pushed on stack"

asm pop: asm EAX; "load an address value into EAX register by
popping a stack"
fld: (asm EAX ptr64). "load a floating point value from memory, at
base address, held in EAX register into fp(0) register,
we are using #ptr64, to indicate that memory operand size is 64bits long"

" return value set, we are done. A code generator will take care
for emitting code, which converts
a double floating point value into smalltalk object. "
]

I just uploaded this example and also how to write doubles from/to memory.
See at SqS/NativeBoost - NativeBoost-Examples package, class NBBasicExamples.

> I am very comfortable adding functions to a library for integer and float/double calculations on "large" arrays and calling them via FFI; plugins are another story. Are there examples to follow?

Tell me, what examples you would like to see. I will gladly provide them.

--
Best regards,
Igor Stasenko AKA sig.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Igor Stasenko

Re: NativeBoost examples (was: Re: DoubleArray)

On 23 September 2010 06:11, Schwab,Wilhelm K <[hidden email]> wrote:
> Sig,
>
> The examples that make sense depend on what NativeBoost wants to be when it grows up. Is it a full competitor to FFI and/or Alien? If so, then it would be nice to see a rich set of calls to functions with different parameter types; ideally it would be in the form of something useful that gets added to Pharo using NativeBoost. That does a couple of things, providing both proof of concept and examples in one shot.
>
One major 'proof of concept' kinda, is NBOpenGL package, which
provides a full OpenGL bindings by using NativeBoost :)
But sure thing, basic examples is a must. That's why i asked you ,
whant you wanna see.

I have no idea, what can be userful to pharoers , since i can do
anything and its completely trivial for me to implement,
as an author of this project ;)
So, without some feedback, i have no idea, if something in my code
looks confusing, or hard to use and needs
deeper explanation(s) and examples.

> With tongue firmly in cheek, looking at your indexing example, my stack might pop before the machine's :) However, it looks like it should settle down quickly enough, and things like that go on in the vm all the time; you are arguably just brining it into the image.

Indeed. Arguably. You putting your code into an image, instead of extra plugin.
So, once your code is debugged and works well, you are done, and don't
have to worry with shipping a custom VM/plugin
with your product. The development and deployment of new primitives is
much faster if you using NB.

> Where would #readDoubleFrom: be in the class hiearchy? One of the reasons I asked about indexing an external array is that FFI is not good at such things. If you can provide ways to copy, fill, etc. blocks of memory and do it quickly, you gain ground, assuming you want to compete with FFI.

In this particular example, you can do a usual way - make a
DoubleArray class, which is a subclass
of ByteArray , then implement #at: and #at:put: using nativeboost.
Implement various stuff to fully cover Collection protocol,
implement any extra stuff like blob operations (multiplication,
addition etc) if you want.

And you don't have to compete with FFI, since you can implement all
primitives in NativeBoost without using any external functions.
You can use any kind of optimizations in native code, which are not
available in C, like using SSE/SIMD instructions etc.
Even more, you can detect a CPU type, and depending on that, generate
a most efficient code for your little class/library.

>
> Do callbacks, and you gain a lot of ground. Make calls on separate threads, and you gain a LOT of ground. That assumes that you can make it simple enough for a mildly above-average C programmer to understand how to use and ideally extend it. I will look at your example package on Squeak Source.

Callbacks is there. See NBFFICallbackTests>>benchQSort for example.

>
> One thing that drives me nuts with the Linux vm is that it looks for things where it wants to look, and gives no feedback about what it tried to do or even that it failed (except that handle is still zero). Your callout pragmas take a module name; is there any way to see inside of that or to provide a full path to the library? It is nice to be able to confirm loading of a library w/o having to call something, since there won't always be something "safe" (free of side effects) to call as a test.
>

Wait, there is no callout pragmas..
All stuff which starts from

^ NBFFICallout
cdecl: #( double ( void * address) ) ...

is smalltalk code, and you can debug it and see what it does, and how
it loading a module and retrieving an external function
pointer from it.
You can customize this procedure in any way you want (by subclassing
NBFFICallout), you can even customize the way how you making a call.

For instance, in NBOpenGL, a generated code does not making a direct
call to previously discovered function address.
It using an sub-instance of NBOpenGLAPIBase, which is a
variable-sized object, holding a function pointers,
and it reads a corresponding pointer from this object, and only then
making a call.

So, there are plenty various tricks which can be done without much
hassle. You just need to step away from rigid patterns,
imposed by C compler, linker and all stuff around that, and you will
see, that you can do miracles in smalltalk + native code :)

> At the risk of seeming difficult, everything is academic until we have a Linux plugin or at least can build same. Alien has that problem.

I'm working on that.

>
> Bill
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] On Behalf Of Igor Stasenko [[hidden email]]
> Sent: Wednesday, September 22, 2010 10:05 PM
> To: [hidden email]
> Subject: [Pharo-project] NativeBoost examples (was: Re: DoubleArray)
>
> On 23 September 2010 02:00, Schwab,Wilhelm K <[hidden email]> wrote:
>> Sig,
>>
>> Fair enough :) The most recent thing I recall on NativeBoost is Stef asking for what I assume must be a Mac plugin; I thought Linux would be left out too, but it appears not, though it is not in my current vm. Beyond some list traffic, I found this:
>>
>
> The work for supporting Linux and Mac is in progress.
>
>> http://code.google.com/p/nativeboost/wiki/NativeBoost
>>
>> Is there other documentation somewhere? Build instructions?
>
> You can find all existing documentation, installation instructions on
> wiki pages at the above site. There is no other place for it.
> http://code.google.com/p/nativeboost/wiki/Installation - installation
> instructions.
>
>> You have a block copy, which is good. What about indexing, such as getting floats or doubles out of external memory?
>
> This is just a piece of cake. ;)
>
> readDoubleFrom: address
> " This method loads the double from given external address.
> an address can be an instance of NBExternalAddress, or
> simple ByteArray with at least 8 bytes long, which holds a double
> floating value"
>
> <primitive: 'primitiveNativeCall' module: 'NativeBoostPlugin'>
>
> "We are using a pseudo-function prototype and supplying own assembler code,
> instead of making a call to external function.
> In given example, an address argument , after coercion is pushed on stack. "
>
> ^ NBFFICallout
>
> cdecl: #( double ( void * address) )
> " A pseudo-function takes 1 argument, and returns double value.
> Under cdecl call convention, all floating point return types is
> returned in fp(0) CPU register"
>
> emitCall: [:gen | | asm |
> asm := gen asm.
>
> "Here , we expecting that an address value is already pushed on stack"
>
> asm pop: asm EAX; "load an address value into EAX register by
> popping a stack"
> fld: (asm EAX ptr64). "load a floating point value from memory, at
> base address, held in EAX register into fp(0) register,
> we are using #ptr64, to indicate that memory operand size is 64bits long"
>
> " return value set, we are done. A code generator will take care
> for emitting code, which converts
> a double floating point value into smalltalk object. "
> ]
>
> I just uploaded this example and also how to write doubles from/to memory.
> See at SqS/NativeBoost - NativeBoost-Examples package, class NBBasicExamples.
>
>> I am very comfortable adding functions to a library for integer and float/double calculations on "large" arrays and calling them via FFI; plugins are another story. Are there examples to follow?
>
> Tell me, what examples you would like to see. I will gladly provide them.
>
>>
>> Bill
>>
>>
>> ________________________________________
>> From: [hidden email] [[hidden email]] On Behalf Of Igor Stasenko [[hidden email]]
>> Sent: Wednesday, September 22, 2010 6:20 PM
>> To: [hidden email]
>> Subject: Re: [Pharo-project] DoubleArray
>>
>> On 23 September 2010 00:37, Schwab,Wilhelm K <[hidden email]> wrote:
>>> Henry,
>>>
>>> I have been getting away with #doubleAt: and #doubleAt:put:, and otherwise doing pretty much what you said, though I just use a byte array to hold the data. It seems to work; for FFI, I end up passing void pointers instead of double pointers, which I don't like doing.
>>>
>>> Your point about using ordinary arrays makes sense to a point, though even w/o external interfacing, there is still a use for DoubleArray just as there is a use for String vs. an array of characters.
>>>
>>
>> The main reason, why they are not here, that VM needs an additional
>> set of primitives, which working with them.
>>
>> <sellsman hat on>
>> I can only say that with NativeBoost, you could be able to implement
>> it quite easily in image and ship it in image,
>> without a need to change VM or use external plugin.
>> <sellsman hat off>
>>
>>> Bill
>>>
>>>
>>> ________________________________________
>>> From: [hidden email] [[hidden email]] On Behalf Of Henrik Sperre Johansen [[hidden email]]
>>> Sent: Wednesday, September 22, 2010 5:23 PM
>>> To: [hidden email]
>>> Subject: Re: [Pharo-project] DoubleArray
>>>
>>> On 22.09.2010 23:09, Schwab,Wilhelm K wrote:
>>>> Squeak, deliberately, does not have a double array class. Is that because there is some horrible reason it can't work, or because it won't work on certain platforms?
>>> If the goal is to pass it as a parameter to an external function, one
>>> way would be to subclass WordArray, storing(and reading) each Float in
>>> two elements, and have the endianness decided by Smalltalk endianness.
>>> Probably want a startUp method for the class to swap instances
>>> endianness if the platform has changed as well.
>>>> Put another way, should we add such a class, if only via an external heap or whatever would be needed to make it reliable?
>>>>
>>>> Bill
>>> More likely because no one has actually had a need for one.
>>> It's definitely not _needed_ in base, where normal arrays of floats
>>> would do the same job just as well, if not better.
>>> IMHO, if someone makes one, it so should probably be part of a package
>>> related to external interfacing.
>>>
>>> Cheers,
>>> Henry
>>>
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>
>>
>>
>> --
>> Best regards,
>> Igor Stasenko AKA sig.
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>
>
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

--
Best regards,
Igor Stasenko AKA sig.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Stéphane Ducasse

Re: NativeBoost examples (was: Re: DoubleArray)

In reply to this post by Schwab,Wilhelm K

igor

May be a layer for people that are not fluent with assembly would help.
Make the path easy from FFI writing.
Get some patterns so that people can learn by following patterns.

Stef

On Sep 23, 2010, at 3:07 PM, Igor Stasenko wrote:

> On 23 September 2010 06:11, Schwab,Wilhelm K <[hidden email]> wrote:
>> Sig,
>>
>> The examples that make sense depend on what NativeBoost wants to be when it grows up. Is it a full competitor to FFI and/or Alien? If so, then it would be nice to see a rich set of calls to functions with different parameter types; ideally it would be in the form of something useful that gets added to Pharo using NativeBoost. That does a couple of things, providing both proof of concept and examples in one shot.
>>
> One major 'proof of concept' kinda, is NBOpenGL package, which
> provides a full OpenGL bindings by using NativeBoost :)
> But sure thing, basic examples is a must. That's why i asked you ,
> whant you wanna see.
>
> I have no idea, what can be userful to pharoers , since i can do
> anything and its completely trivial for me to implement,
> as an author of this project ;)
> So, without some feedback, i have no idea, if something in my code
> looks confusing, or hard to use and needs
> deeper explanation(s) and examples.
>
>> With tongue firmly in cheek, looking at your indexing example, my stack might pop before the machine's :) However, it looks like it should settle down quickly enough, and things like that go on in the vm all the time; you are arguably just brining it into the image.
>
> Indeed. Arguably. You putting your code into an image, instead of extra plugin.
> So, once your code is debugged and works well, you are done, and don't
> have to worry with shipping a custom VM/plugin
> with your product. The development and deployment of new primitives is
> much faster if you using NB.
>
>> Where would #readDoubleFrom: be in the class hiearchy? One of the reasons I asked about indexing an external array is that FFI is not good at such things. If you can provide ways to copy, fill, etc. blocks of memory and do it quickly, you gain ground, assuming you want to compete with FFI.
>
> In this particular example, you can do a usual way - make a
> DoubleArray class, which is a subclass
> of ByteArray , then implement #at: and #at:put: using nativeboost.
> Implement various stuff to fully cover Collection protocol,
> implement any extra stuff like blob operations (multiplication,
> addition etc) if you want.
>
> And you don't have to compete with FFI, since you can implement all
> primitives in NativeBoost without using any external functions.
> You can use any kind of optimizations in native code, which are not
> available in C, like using SSE/SIMD instructions etc.
> Even more, you can detect a CPU type, and depending on that, generate
> a most efficient code for your little class/library.
>
>>
>> Do callbacks, and you gain a lot of ground. Make calls on separate threads, and you gain a LOT of ground. That assumes that you can make it simple enough for a mildly above-average C programmer to understand how to use and ideally extend it. I will look at your example package on Squeak Source.
>
> Callbacks is there. See NBFFICallbackTests>>benchQSort for example.
>
>>
>> One thing that drives me nuts with the Linux vm is that it looks for things where it wants to look, and gives no feedback about what it tried to do or even that it failed (except that handle is still zero). Your callout pragmas take a module name; is there any way to see inside of that or to provide a full path to the library? It is nice to be able to confirm loading of a library w/o having to call something, since there won't always be something "safe" (free of side effects) to call as a test.
>>
>
> Wait, there is no callout pragmas..
> All stuff which starts from
>
> ^ NBFFICallout
> cdecl: #( double ( void * address) ) ...
>
> is smalltalk code, and you can debug it and see what it does, and how
> it loading a module and retrieving an external function
> pointer from it.
> You can customize this procedure in any way you want (by subclassing
> NBFFICallout), you can even customize the way how you making a call.
>
> For instance, in NBOpenGL, a generated code does not making a direct
> call to previously discovered function address.
> It using an sub-instance of NBOpenGLAPIBase, which is a
> variable-sized object, holding a function pointers,
> and it reads a corresponding pointer from this object, and only then
> making a call.
>
> So, there are plenty various tricks which can be done without much
> hassle. You just need to step away from rigid patterns,
> imposed by C compler, linker and all stuff around that, and you will
> see, that you can do miracles in smalltalk + native code :)
>
>> At the risk of seeming difficult, everything is academic until we have a Linux plugin or at least can build same. Alien has that problem.
>
> I'm working on that.
>
>>
>> Bill
>>
>>
>> ________________________________________
>> From: [hidden email] [[hidden email]] On Behalf Of Igor Stasenko [[hidden email]]
>> Sent: Wednesday, September 22, 2010 10:05 PM
>> To: [hidden email]
>> Subject: [Pharo-project] NativeBoost examples (was: Re: DoubleArray)
>>
>> On 23 September 2010 02:00, Schwab,Wilhelm K <[hidden email]> wrote:
>>> Sig,
>>>
>>> Fair enough :) The most recent thing I recall on NativeBoost is Stef asking for what I assume must be a Mac plugin; I thought Linux would be left out too, but it appears not, though it is not in my current vm. Beyond some list traffic, I found this:
>>>
>>
>> The work for supporting Linux and Mac is in progress.
>>
>>> http://code.google.com/p/nativeboost/wiki/NativeBoost
>>>
>>> Is there other documentation somewhere? Build instructions?
>>
>> You can find all existing documentation, installation instructions on
>> wiki pages at the above site. There is no other place for it.
>> http://code.google.com/p/nativeboost/wiki/Installation - installation
>> instructions.
>>
>>> You have a block copy, which is good. What about indexing, such as getting floats or doubles out of external memory?
>>
>> This is just a piece of cake. ;)
>>
>> readDoubleFrom: address
>> " This method loads the double from given external address.
>> an address can be an instance of NBExternalAddress, or
>> simple ByteArray with at least 8 bytes long, which holds a double
>> floating value"
>>
>> <primitive: 'primitiveNativeCall' module: 'NativeBoostPlugin'>
>>
>> "We are using a pseudo-function prototype and supplying own assembler code,
>> instead of making a call to external function.
>> In given example, an address argument , after coercion is pushed on stack. "
>>
>> ^ NBFFICallout
>>
>> cdecl: #( double ( void * address) )
>> " A pseudo-function takes 1 argument, and returns double value.
>> Under cdecl call convention, all floating point return types is
>> returned in fp(0) CPU register"
>>
>> emitCall: [:gen | | asm |
>> asm := gen asm.
>>
>> "Here , we expecting that an address value is already pushed on stack"
>>
>> asm pop: asm EAX; "load an address value into EAX register by
>> popping a stack"
>> fld: (asm EAX ptr64). "load a floating point value from memory, at
>> base address, held in EAX register into fp(0) register,
>> we are using #ptr64, to indicate that memory operand size is 64bits long"
>>
>> " return value set, we are done. A code generator will take care
>> for emitting code, which converts
>> a double floating point value into smalltalk object. "
>> ]
>>
>> I just uploaded this example and also how to write doubles from/to memory.
>> See at SqS/NativeBoost - NativeBoost-Examples package, class NBBasicExamples.
>>
>>> I am very comfortable adding functions to a library for integer and float/double calculations on "large" arrays and calling them via FFI; plugins are another story. Are there examples to follow?
>>
>> Tell me, what examples you would like to see. I will gladly provide them.
>>
>>>
>>> Bill
>>>
>>>
>>> ________________________________________
>>> From: [hidden email] [[hidden email]] On Behalf Of Igor Stasenko [[hidden email]]
>>> Sent: Wednesday, September 22, 2010 6:20 PM
>>> To: [hidden email]
>>> Subject: Re: [Pharo-project] DoubleArray
>>>
>>> On 23 September 2010 00:37, Schwab,Wilhelm K <[hidden email]> wrote:
>>>> Henry,
>>>>
>>>> I have been getting away with #doubleAt: and #doubleAt:put:, and otherwise doing pretty much what you said, though I just use a byte array to hold the data. It seems to work; for FFI, I end up passing void pointers instead of double pointers, which I don't like doing.
>>>>
>>>> Your point about using ordinary arrays makes sense to a point, though even w/o external interfacing, there is still a use for DoubleArray just as there is a use for String vs. an array of characters.
>>>>
>>>
>>> The main reason, why they are not here, that VM needs an additional
>>> set of primitives, which working with them.
>>>
>>> <sellsman hat on>
>>> I can only say that with NativeBoost, you could be able to implement
>>> it quite easily in image and ship it in image,
>>> without a need to change VM or use external plugin.
>>> <sellsman hat off>
>>>
>>>> Bill
>>>>
>>>>
>>>> ________________________________________
>>>> From: [hidden email] [[hidden email]] On Behalf Of Henrik Sperre Johansen [[hidden email]]
>>>> Sent: Wednesday, September 22, 2010 5:23 PM
>>>> To: [hidden email]
>>>> Subject: Re: [Pharo-project] DoubleArray
>>>>
>>>> On 22.09.2010 23:09, Schwab,Wilhelm K wrote:
>>>>> Squeak, deliberately, does not have a double array class. Is that because there is some horrible reason it can't work, or because it won't work on certain platforms?
>>>> If the goal is to pass it as a parameter to an external function, one
>>>> way would be to subclass WordArray, storing(and reading) each Float in
>>>> two elements, and have the endianness decided by Smalltalk endianness.
>>>> Probably want a startUp method for the class to swap instances
>>>> endianness if the platform has changed as well.
>>>>> Put another way, should we add such a class, if only via an external heap or whatever would be needed to make it reliable?
>>>>>
>>>>> Bill
>>>> More likely because no one has actually had a need for one.
>>>> It's definitely not _needed_ in base, where normal arrays of floats
>>>> would do the same job just as well, if not better.
>>>> IMHO, if someone makes one, it so should probably be part of a package
>>>> related to external interfacing.
>>>>
>>>> Cheers,
>>>> Henry
>>>>
>>>>
>>>> _______________________________________________
>>>> Pharo-project mailing list
>>>> [hidden email]
>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>>
>>>> _______________________________________________
>>>> Pharo-project mailing list
>>>> [hidden email]
>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Igor Stasenko AKA sig.
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>
>>
>>
>> --
>> Best regards,
>> Igor Stasenko AKA sig.
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>
>
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Henrik Sperre Johansen

Re: NativeBoost examples (was: Re: DoubleArray)

On Sep 23, 2010, at 4:03 07PM, Stéphane Ducasse wrote:

igor

May be a layer for people that are not fluent with assembly would help.
Make the path easy from FFI writing.

It is already, RTM :)

http://code.google.com/p/nativeboost/wiki/NBFnSpecParser

Get some patterns so that people can learn by following patterns.

Stef

Cheers,

Henry

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Igor Stasenko

Re: NativeBoost examples (was: Re: DoubleArray)

2010/9/23 Henrik Johansen <[hidden email]>:

>
> On Sep 23, 2010, at 4:03 07PM, Stéphane Ducasse wrote:
>
> igor
>
> May be a layer for people that are not fluent with assembly would help.
> Make the path easy from FFI writing.
>
> It is already, RTM :)
> http://code.google.com/p/nativeboost/wiki/NBFnSpecParser
>
> Get some patterns so that people can learn by following patterns.
>

Oh, yeah.. see. I was showing an example how to implement own
primitive using NativeBoost.

Of course, an FFI API is there, and doing all the magic under the
hood, so you dont have to know any assemler voodo:

releaseDC: hdc
<primitive: #primitiveNativeCall module: #NativeBoostPlugin>

^ NBFFICallout stdcall: #(

int ReleaseDC (
HWND self , " handle to window"
HDC hdc " handle to DC "
))
module: #user32

the above is an FFI call to external function ReleaseDC, which can be
found in 'user32.dll' windoze module.

> Stef
>
> Cheers,
> Henry
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

--
Best regards,
Igor Stasenko AKA sig.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Igor Stasenko

Re: NativeBoost examples (was: Re: DoubleArray)

On 23 September 2010 18:02, Igor Stasenko <[hidden email]> wrote:

> 2010/9/23 Henrik Johansen <[hidden email]>:
>>
>> On Sep 23, 2010, at 4:03 07PM, Stéphane Ducasse wrote:
>>
>> igor
>>
>> May be a layer for people that are not fluent with assembly would help.
>> Make the path easy from FFI writing.
>>
>> It is already, RTM :)
>> http://code.google.com/p/nativeboost/wiki/NBFnSpecParser
>>
>> Get some patterns so that people can learn by following patterns.
>>
>
> Oh, yeah.. see. I was showing an example how to implement own
> primitive using NativeBoost.
>
> Of course, an FFI API is there, and doing all the magic under the
> hood, so you dont have to know any assemler voodo:
>
> releaseDC: hdc
> <primitive: #primitiveNativeCall module: #NativeBoostPlugin>
>
> ^ NBFFICallout stdcall: #(
>
> int ReleaseDC (
> HWND self , " handle to window"
> HDC hdc " handle to DC "
> ))
> module: #user32
>
> the above is an FFI call to external function ReleaseDC, which can be
> found in 'user32.dll' windoze module.
>

One of Bill's concerns is to be able to customize an external library
search algorithm.
And i can easily extend this api for custom-loading a library:

Suppose, that if you passing a block into #module: argument, instead
of string/symbol,
then this block should answer a ready-to-use external library handle.
So, then above example can be rewritten as:

releaseDC: hdc
<primitive: #primitiveNativeCall module: #NativeBoostPlugin>

^ NBFFICallout stdcall: #(

int ReleaseDC (
HWND self , " handle to window"
HDC hdc " handle to DC " ))
module: [ self whereTheHeckMyModule ]

And you are free to implement #whereTheHeckMyModule in any way you like :)

>> Stef
>>
>> Cheers,
>> Henry
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>
>
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>

--
Best regards,
Igor Stasenko AKA sig.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Igor Stasenko

Re: NativeBoost examples (was: Re: DoubleArray)

On 23 September 2010 18:21, Igor Stasenko <[hidden email]> wrote:

> On 23 September 2010 18:02, Igor Stasenko <[hidden email]> wrote:
>> 2010/9/23 Henrik Johansen <[hidden email]>:
>>>
>>> On Sep 23, 2010, at 4:03 07PM, Stéphane Ducasse wrote:
>>>
>>> igor
>>>
>>> May be a layer for people that are not fluent with assembly would help.
>>> Make the path easy from FFI writing.
>>>
>>> It is already, RTM :)
>>> http://code.google.com/p/nativeboost/wiki/NBFnSpecParser
>>>
>>> Get some patterns so that people can learn by following patterns.
>>>
>>
>> Oh, yeah.. see. I was showing an example how to implement own
>> primitive using NativeBoost.
>>
>> Of course, an FFI API is there, and doing all the magic under the
>> hood, so you dont have to know any assemler voodo:
>>
>> releaseDC: hdc
>> <primitive: #primitiveNativeCall module: #NativeBoostPlugin>
>>
>> ^ NBFFICallout stdcall: #(
>>
>> int ReleaseDC (
>> HWND self , " handle to window"
>> HDC hdc " handle to DC "
>> ))
>> module: #user32
>>
>> the above is an FFI call to external function ReleaseDC, which can be
>> found in 'user32.dll' windoze module.
>>
>
> One of Bill's concerns is to be able to customize an external library
> search algorithm.
> And i can easily extend this api for custom-loading a library:
>
> Suppose, that if you passing a block into #module: argument, instead
> of string/symbol,
> then this block should answer a ready-to-use external library handle.
> So, then above example can be rewritten as:
>
> releaseDC: hdc
> <primitive: #primitiveNativeCall module: #NativeBoostPlugin>
>
> ^ NBFFICallout stdcall: #(
>
> int ReleaseDC (
> HWND self , " handle to window"
> HDC hdc " handle to DC " ))
> module: [ self whereTheHeckMyModule ]
>

oh, that's a bit overengineering...
come-on.. this is just a smalltalk code.. so it could be just:

module: self whereTheHeckMyModule

so, callout expects either a module name (then it using a default
search algorithm),
or a ready for use module handle, then it is up to you to supply it.

>
> And you are free to implement #whereTheHeckMyModule in any way you like :)
>
>
>>> Stef
>>>
>>> Cheers,
>>> Henry
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>
>>
>>
>> --
>> Best regards,
>> Igor Stasenko AKA sig.
>>
>
>
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>

--
Best regards,
Igor Stasenko AKA sig.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Schwab,Wilhelm K

Re: NativeBoost examples (was: Re: DoubleArray)

In reply to this post by Igor Stasenko

Sig,

I certainly don't want to stir up a war you might be trying to avoid, but it does strike me that you offer an alternative to FFI. That's ok, because Alien clearly does that, so we might as well have a third choice.

Please make a lot of noise when you have a Linux plugin. I will give some thought to what might make a useful project/demo, but OpenGL should certainly have its points in both directions. If it does something cool, so much the better. Long term, it would be nice to have a retained mode 3D system. Croquet's TObject hierarchy seemed to be heading in the right direction, but they resisted the idea of putting a 3D world in a 2D application; I'm thinking of 3D modeling, games, etc. A way to create a window, add some objects, lights and a camera and rotate, move them would be wonderful to have. Alice is/was close, but it required a lot of turning off Wonderland (end user) features that I felt should have been added if desired rather than taken away. It also required the mathematically aware to think like layman, which was a problem. Prior to finding QCad, I designed a few electronics enclosures by putting "boxes on shelves" and flying around it with the help of Alice.

There is a grant application I need to be working on (should be doing so right now, in fact, right Stef?<g>). That has me thinking about Fourier Transforms, graphs and statistics. I have a fairly pathetic DOUBLEArray that seems to work. Having a complete set of double, float, [s]dword, [s]word arrays would be nice for many uses. When 1.1.1 comes along, I need to redo some FFI code to benefit from the selector fixes. I might be able to stall that long enough to try NB, but it really just needs to have some things renamed. My PLplot interface could be redone or not, but so far I see no reason to bother. If it ends up needing callbacks, it would be a good candidate. A different interface really needs work because it screams for callbacks and would be a wonderful showpiece for NB. That's a big job, but a good goal. There is little to no time pressure (what I have works as far as it goes), but a big payoff if I can make it work with callbacks.

From a marketing perspective, if you don't yet have a cool 3D demo, that might be your best use of time. Either way, I need to slowly study what you have and begin to experiement toward the above mentiond rewrite once you have Linux locked down.

If you do have a cool 3D demo, why isn't it linked from your NativeBoost web site? :) Seriously, add a page of links to projects, or just one page that links to the SqueakSource pages of interest. The easier you make it for someone to see that you can do all of this stuff, the more likley you are to create converts.

Have you looked at Ocean? It is being built with Alien. When you can do Windows, Linux and Mac, there might be an argument to switch. I don't control the project, but it seems reasonble to at least consider NB for it.

I hope that's of some help. I look forward to trying it.

Bill

________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Igor Stasenko [[hidden email]]
Sent: Thursday, September 23, 2010 9:07 AM
To: [hidden email]
Subject: Re: [Pharo-project] NativeBoost examples (was: Re: DoubleArray)

On 23 September 2010 06:11, Schwab,Wilhelm K <[hidden email]> wrote:
> Sig,
>
> The examples that make sense depend on what NativeBoost wants to be when it grows up. Is it a full competitor to FFI and/or Alien? If so, then it would be nice to see a rich set of calls to functions with different parameter types; ideally it would be in the form of something useful that gets added to Pharo using NativeBoost. That does a couple of things, providing both proof of concept and examples in one shot.
>
One major 'proof of concept' kinda, is NBOpenGL package, which
provides a full OpenGL bindings by using NativeBoost :)
But sure thing, basic examples is a must. That's why i asked you ,
whant you wanna see.

I have no idea, what can be userful to pharoers , since i can do
anything and its completely trivial for me to implement,
as an author of this project ;)
So, without some feedback, i have no idea, if something in my code
looks confusing, or hard to use and needs
deeper explanation(s) and examples.

> With tongue firmly in cheek, looking at your indexing example, my stack might pop before the machine's :) However, it looks like it should settle down quickly enough, and things like that go on in the vm all the time; you are arguably just brining it into the image.

Indeed. Arguably. You putting your code into an image, instead of extra plugin.
So, once your code is debugged and works well, you are done, and don't
have to worry with shipping a custom VM/plugin
with your product. The development and deployment of new primitives is
much faster if you using NB.

> Where would #readDoubleFrom: be in the class hiearchy? One of the reasons I asked about indexing an external array is that FFI is not good at such things. If you can provide ways to copy, fill, etc. blocks of memory and do it quickly, you gain ground, assuming you want to compete with FFI.

In this particular example, you can do a usual way - make a
DoubleArray class, which is a subclass
of ByteArray , then implement #at: and #at:put: using nativeboost.
Implement various stuff to fully cover Collection protocol,
implement any extra stuff like blob operations (multiplication,
addition etc) if you want.

And you don't have to compete with FFI, since you can implement all
primitives in NativeBoost without using any external functions.
You can use any kind of optimizations in native code, which are not
available in C, like using SSE/SIMD instructions etc.
Even more, you can detect a CPU type, and depending on that, generate
a most efficient code for your little class/library.

>
> Do callbacks, and you gain a lot of ground. Make calls on separate threads, and you gain a LOT of ground. That assumes that you can make it simple enough for a mildly above-average C programmer to understand how to use and ideally extend it. I will look at your example package on Squeak Source.

Callbacks is there. See NBFFICallbackTests>>benchQSort for example.

>
> One thing that drives me nuts with the Linux vm is that it looks for things where it wants to look, and gives no feedback about what it tried to do or even that it failed (except that handle is still zero). Your callout pragmas take a module name; is there any way to see inside of that or to provide a full path to the library? It is nice to be able to confirm loading of a library w/o having to call something, since there won't always be something "safe" (free of side effects) to call as a test.
>

Wait, there is no callout pragmas..
All stuff which starts from

^ NBFFICallout
cdecl: #( double ( void * address) ) ...

is smalltalk code, and you can debug it and see what it does, and how
it loading a module and retrieving an external function
pointer from it.
You can customize this procedure in any way you want (by subclassing
NBFFICallout), you can even customize the way how you making a call.

For instance, in NBOpenGL, a generated code does not making a direct
call to previously discovered function address.
It using an sub-instance of NBOpenGLAPIBase, which is a
variable-sized object, holding a function pointers,
and it reads a corresponding pointer from this object, and only then
making a call.

So, there are plenty various tricks which can be done without much
hassle. You just need to step away from rigid patterns,
imposed by C compler, linker and all stuff around that, and you will
see, that you can do miracles in smalltalk + native code :)

> At the risk of seeming difficult, everything is academic until we have a Linux plugin or at least can build same. Alien has that problem.

I'm working on that.

Schwab,Wilhelm K

Re: NativeBoost examples (was: Re: DoubleArray)

In reply to this post by Igor Stasenko

Sig,

You say "Of course, an FFI API is there" but it's not always clear what is and isn't working. You appear to be further along that I thought was the case. I *really* enjoy being wrong in that direction :)

And yes, being able to enter ASM is good - it would be REALLY bad if we had to do that for everything. Add some questions (valid or FUD, I don't know yet) about the ease of use of Alien, and it gets confusing.

Bill

________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Igor Stasenko [[hidden email]]
Sent: Thursday, September 23, 2010 11:02 AM
To: [hidden email]
Subject: Re: [Pharo-project] NativeBoost examples (was: Re: DoubleArray)

2010/9/23 Henrik Johansen <[hidden email]>:

Schwab,Wilhelm K

Re: NativeBoost examples (was: Re: DoubleArray)

In reply to this post by Igor Stasenko

NICE!!!!!

________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Igor Stasenko [[hidden email]]
Sent: Thursday, September 23, 2010 11:21 AM
To: [hidden email]
Subject: Re: [Pharo-project] NativeBoost examples (was: Re: DoubleArray)

On 23 September 2010 18:02, Igor Stasenko <[hidden email]> wrote:

Schwab,Wilhelm K

Re: NativeBoost examples (was: Re: DoubleArray)

In reply to this post by Igor Stasenko

Slightly overbuilt or not, we can bang out the details over time. The point is that Sig is bringing things into the image where it can be seen and changed vs. locked away in a compiled binary that often does not describe where/how it gets into trouble.

________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Igor Stasenko [[hidden email]]
Sent: Thursday, September 23, 2010 11:30 AM
To: [hidden email]
Subject: Re: [Pharo-project] NativeBoost examples (was: Re: DoubleArray)

On 23 September 2010 18:21, Igor Stasenko <[hidden email]> wrote: