FFI related questions

> Igor Stasenko wrote:
>>
>> The difference between FloatArray and Array of floats (or array of
>> arbitrary objects)
>> is in storage format. While first storing a number of native
>> floating-point values, edigible for C,
>> while second stores references to other objects (oop-s).
>
> Indeed and this means that we can change values in C land when they
> are contained in said Arrays (ByteArray, FloatArray, IntegerArray, etc)
> but not when sent down as single values, which strikes me as kinda
> inconsistent, at least from an FFI usage point of view.
> A bit OT, speaking of FloatArray, it changes the Floats you put into it
> into single precision. So currently it is impossible to pass to C an array
> of double precision Floats. Shouldn't the image also have a DoubleArray for
> that purpose?
>

>>> Another thing that isn't clear to me is if its possible to have FFI
>>> create
>>> Smalltalk objects
>>> in behalf of a requesting client. In Java's JNI for instance one can say:
>>>
>>>  jni->NewFloatArray(length of the array);
>
>
>> in that case, i think, you'd better create a VM plugin, which having
>> an access to something similar to 'jni'
>> object, which in squeak is called 'interpreterProxy'.
>
> If i understood correctly, writing a plugin with the sole purpose
> of giving access to interpreterProxy seems overkill. I would like to
> have this as standard functionality provided by FFI (and i am willing
> to implement it if you (guys) see it as valuable.
> If on the other hand you meant writing a plugin and completely bypassing
> FFI, i would rather not do it until i exhaust all possibilities with FFI
> because, modulo the issues we are talking about here, it serves my needs
> excellently.
>
>> Thinking about it, i came to interesting idea, that really, we could
>> pass a pointer to interpreterProxy
>> variable to FFI function, so, potentially, it could make use of VM API
>> functions to create objects, or access existing ones.
>
> Yeap, plus have a bunch of utility routines for creating arrays.
> There is some code in FFI thats along those lines, FFI can create
> Strings (because it knows how to copy the data from a null terminated
> C char array, see ffiReturnCStringFrom).
> Symmetrically, we can extend FFI to create Arrays of all possible basic
> types for us, let the client fill those appropriately, and then let
> FFI return the oop up to the image as usual.
>
>> I don't know if that will be ever used, since writing a plugin looks
>> much more appropriate in that case.
>
> In my scenarios it would be useful, hopefully its generally useful :-)
>
> To sum up my thoughts/propositions:
>  - extend FFI with a bunch of functions that create Arrays of predefined
> size
>   of all possible C types
>  - extend FFI to provide access to the VM (via interpreterProxy) for all
>   possible unforeseen usages and needs that aren't served by FFI's existing
>   functionality
>  - (not directly FFI related) create DoubleArray object in image.
>
> I hope all these are technically feasible...
>
> Cheers
>  - Vagelis
>
>

> 2010/1/5 vagy <[hidden email]>:
>> Igor Stasenko wrote:
>>>
>>> The difference between FloatArray and Array of floats (or array of
>>> arbitrary objects)
>>> is in storage format. While first storing a number of native
>>> floating-point values, edigible for C,
>>> while second stores references to other objects (oop-s).
>>
>> Indeed and this means that we can change values in C land when they
>> are contained in said Arrays (ByteArray, FloatArray, IntegerArray, etc)
>> but not when sent down as single values, which strikes me as kinda
>> inconsistent, at least from an FFI usage point of view.
>> A bit OT, speaking of FloatArray, it changes the Floats you put into it
>> into single precision. So currently it is impossible to pass to C an  
>> array
>> of double precision Floats. Shouldn't the image also have a DoubleArray  
>> for
>> that purpose?
>>
>
> See how I did this in Smallapack-Collections
> http://www.squeaksource.com/Smallapack.html
> I created an ExternalArray and various subclasses.
>
> Nicolas

> You're right it [DoubleArray]'s nontrivial, at least for the current
> object model.  Many platforms have 8 byte alignment requirements for
> doubles and our oops are only 4 byte aligned. This is the main reason
> why there isn't a double array - since garbage collector will move
> objects around there is absolutely no guarantee that the double array
> will be 8 byte aligned.  And having a double array that the code in
> the VM needs to shift to align properly would completely defeat the
> purpose of having it in the first place. This may change with the move
> to 64 bit or perhaps with some of Eliot's ideas about changing the
> object format, but it's nothing that is going to happen in the short
> term.

> Igor Stasenko wrote:
>>
>> The difference between FloatArray and Array of floats (or array of
>> arbitrary objects)
>> is in storage format. While first storing a number of native
>> floating-point values, edigible for C,
>> while second stores references to other objects (oop-s).
>
> Indeed and this means that we can change values in C land when they
> are contained in said Arrays (ByteArray, FloatArray, IntegerArray, etc)
> but not when sent down as single values, which strikes me as kinda
> inconsistent, at least from an FFI usage point of view.
> A bit OT, speaking of FloatArray, it changes the Floats you put into it
> into single precision. So currently it is impossible to pass to C an array
> of double precision Floats. Shouldn't the image also have a DoubleArray for
> that purpose?
>

>>> Another thing that isn't clear to me is if its possible to have FFI
>>> create
>>> Smalltalk objects
>>> in behalf of a requesting client. In Java's JNI for instance one can say:
>>>
>>>  jni->NewFloatArray(length of the array);
>
>
>> in that case, i think, you'd better create a VM plugin, which having
>> an access to something similar to 'jni'
>> object, which in squeak is called 'interpreterProxy'.
>
> If i understood correctly, writing a plugin with the sole purpose
> of giving access to interpreterProxy seems overkill. I would like to
> have this as standard functionality provided by FFI (and i am willing
> to implement it if you (guys) see it as valuable.
> If on the other hand you meant writing a plugin and completely bypassing
> FFI, i would rather not do it until i exhaust all possibilities with FFI
> because, modulo the issues we are talking about here, it serves my needs
> excellently.
>

>> Thinking about it, i came to interesting idea, that really, we could
>> pass a pointer to interpreterProxy
>> variable to FFI function, so, potentially, it could make use of VM API
>> functions to create objects, or access existing ones.
>
> Yeap, plus have a bunch of utility routines for creating arrays.
> There is some code in FFI thats along those lines, FFI can create
> Strings (because it knows how to copy the data from a null terminated
> C char array, see ffiReturnCStringFrom).
> Symmetrically, we can extend FFI to create Arrays of all possible basic
> types for us, let the client fill those appropriately, and then let
> FFI return the oop up to the image as usual.

>
>> I don't know if that will be ever used, since writing a plugin looks
>> much more appropriate in that case.
>
> In my scenarios it would be useful, hopefully its generally useful :-)
>
> To sum up my thoughts/propositions:
>  - extend FFI with a bunch of functions that create Arrays of predefined
> size
>   of all possible C types
>  - extend FFI to provide access to the VM (via interpreterProxy) for all
>   possible unforeseen usages and needs that aren't served by FFI's existing
>   functionality
>  - (not directly FFI related) create DoubleArray object in image.
>
> I hope all these are technically feasible...
>
> Cheers
>  - Vagelis
>
>

> vagy wrote:
>> Thanx for the tip. I saw your code, though i was thinking more
>> along the lines of having a standard DoubleArray implementation in the  
>> image
>> as we have for 32bit Floats. I just saw that FloatArray is implemented  
>> in
>> terms of variableWordSubclass but there is no variableDoubleWordSubclass
>> which, at first glance, may make the whole issue of having a standard
>> DoubleArray non trivial. I think we need the opinion of a VM expert  
>> here.
>> In any way, what do you think about having a standard DoubleArray?
>
> You're right it's nontrivial, at least for the current object model.  
> Many platforms have 8 byte alignment requirements for doubles and our  
> oops are only 4 byte aligned. This is the main reason why there isn't a  
> double array - since garbage collector will move objects around there is  
> absolutely no guarantee that the double array will be 8 byte aligned.  
> And having a double array that the code in the VM needs to shift to  
> align properly would completely defeat the purpose of having it in the  
> first place. This may change with the move to 64 bit or perhaps with  
> some of Eliot's ideas about changing the object format, but it's nothing  
> that is going to happen in the short term.
>
> Cheers,
>    - Andreas

> On Tue, 05 Jan 2010 22:13:28 +0200, Andreas Raab <[hidden email]>
> wrote:
>
>> vagy wrote:
>>>
>>> Thanx for the tip. I saw your code, though i was thinking more
>>> along the lines of having a standard DoubleArray implementation in the
>>> image
>>> as we have for 32bit Floats. I just saw that FloatArray is implemented in
>>> terms of variableWordSubclass but there is no variableDoubleWordSubclass
>>> which, at first glance, may make the whole issue of having a standard
>>> DoubleArray non trivial. I think we need the opinion of a VM expert here.
>>> In any way, what do you think about having a standard DoubleArray?
>>
>> You're right it's nontrivial, at least for the current object model. Many
>> platforms have 8 byte alignment requirements for doubles and our oops are
>> only 4 byte aligned. This is the main reason why there isn't a double array
>> - since garbage collector will move objects around there is absolutely no
>> guarantee that the double array will be 8 byte aligned. And having a double
>> array that the code in the VM needs to shift to align properly would
>> completely defeat the purpose of having it in the first place. This may
>> change with the move to 64 bit or perhaps with some of Eliot's ideas about
>> changing the object format, but it's nothing that is going to happen in the
>> short term.
>>
>> Cheers,
>>   - Andreas
>
> I see, thanx for the info. Well then i'll put DoubleArrays on hold for now
> and craft something special for my project's use case.
> Did you take a look at the other issues i raised about FFI? (i can't paste a
> link to the relevant message at the moment because the message archive
> server
> is down). But I would definitely like a word from you before going on with
> it.
> In a nutshell:
> - Being able to change single values (and not only the values contained in
> Arrays)
>  when passed down to C through pointer arguments
>  (for instance if you pass down an Integer through a long* argument, FFI
> tells
>  you that you cant treat an Integer like a pointer, when the intended usage
> is
>  treating it like in C, i.e. change the value of the pointed memory)
> - Have FFI helper functions (that a C client can link against) that create
>  whatever Smalltalk Arrays FFI is able to handle (this is inspired from
> Java's JNI)
> - Expose interpreterProxy through FFI for advanced uses (sic!) :-)
>  (Though i don't completely understand the consequences of that and it feels
> a little
>  bit dangerous)
>

> 2010/1/6 vagy <[hidden email]>:
>> I see, thanx for the info. Well then i'll put DoubleArrays on hold for now
>> and craft something special for my project's use case.
>> Did you take a look at the other issues i raised about FFI? (i can't paste a
>> link to the relevant message at the moment because the message archive
>> server
>> is down). But I would definitely like a word from you before going on with
>> it.
>> In a nutshell:
>> - Being able to change single values (and not only the values contained in
>> Arrays)
>>  when passed down to C through pointer arguments
>>  (for instance if you pass down an Integer through a long* argument, FFI
>> tells
>>  you that you cant treat an Integer like a pointer, when the intended usage
>> is
>>  treating it like in C, i.e. change the value of the pointed memory)
>> - Have FFI helper functions (that a C client can link against) that create
>>  whatever Smalltalk Arrays FFI is able to handle (this is inspired from
>> Java's JNI)
>> - Expose interpreterProxy through FFI for advanced uses (sic!) :-)
>>  (Though i don't completely understand the consequences of that and it feels
>> a little
>>  bit dangerous)
>>
>
> Not a bit more dangerous than FFI itself :)
>
> By changing a module-loading logic a bit (in sqNamedPrims.c -
> findAndLoadModule() function)
> to attempt to call the setInterpreter() function if it available, even
> for FFI module.

> 2010/1/6 vagy <[hidden email]>:
>> On Tue, 05 Jan 2010 22:13:28 +0200, Andreas Raab <[hidden email]>
>> wrote:
>>
>>> vagy wrote:
>>>>
>>>> Thanx for the tip. I saw your code, though i was thinking more
>>>> along the lines of having a standard DoubleArray implementation in the
>>>> image
>>>> as we have for 32bit Floats. I just saw that FloatArray is  
>>>> implemented in
>>>> terms of variableWordSubclass but there is no  
>>>> variableDoubleWordSubclass
>>>> which, at first glance, may make the whole issue of having a standard
>>>> DoubleArray non trivial. I think we need the opinion of a VM expert  
>>>> here.
>>>> In any way, what do you think about having a standard DoubleArray?
>>>
>>> You're right it's nontrivial, at least for the current object model.  
>>> Many
>>> platforms have 8 byte alignment requirements for doubles and our oops  
>>> are
>>> only 4 byte aligned. This is the main reason why there isn't a double  
>>> array
>>> - since garbage collector will move objects around there is absolutely  
>>> no
>>> guarantee that the double array will be 8 byte aligned. And having a  
>>> double
>>> array that the code in the VM needs to shift to align properly would
>>> completely defeat the purpose of having it in the first place. This may
>>> change with the move to 64 bit or perhaps with some of Eliot's ideas  
>>> about
>>> changing the object format, but it's nothing that is going to happen  
>>> in the
>>> short term.
>>>
>>> Cheers,
>>>   - Andreas
>>
>> I see, thanx for the info. Well then i'll put DoubleArrays on hold for  
>> now
>> and craft something special for my project's use case.
>> Did you take a look at the other issues i raised about FFI? (i can't  
>> paste a
>> link to the relevant message at the moment because the message archive
>> server
>> is down). But I would definitely like a word from you before going on  
>> with
>> it.
>> In a nutshell:
>> - Being able to change single values (and not only the values contained  
>> in
>> Arrays)
>>  when passed down to C through pointer arguments
>>  (for instance if you pass down an Integer through a long* argument, FFI
>> tells
>>  you that you cant treat an Integer like a pointer, when the intended  
>> usage
>> is
>>  treating it like in C, i.e. change the value of the pointed memory)
>> - Have FFI helper functions (that a C client can link against) that  
>> create
>>  whatever Smalltalk Arrays FFI is able to handle (this is inspired from
>> Java's JNI)
>> - Expose interpreterProxy through FFI for advanced uses (sic!) :-)
>>  (Though i don't completely understand the consequences of that and it  
>> feels
>> a little
>>  bit dangerous)
>>
>
> Not a bit more dangerous than FFI itself :)
>
> By changing a module-loading logic a bit (in sqNamedPrims.c -
> findAndLoadModule() function)
> to attempt to call the setInterpreter() function if it available, even
> for FFI module.
>
> Then all you have to do in your module is to define that function, and
> you can use it to
> store pointer to VM API, which will let you create squeak objects and
> have fun :)
>
>> Cheers
>>  - Vagelis
>>
>