FFI question: array of structures as parameter

> All,
>
> I'm trying to call a function in an external library which takes an array of
> structures as one of its parameters. I can see how to define a subclass of
> ExternalStructure, and how that creates convenient field accessors which
> read/write an underlying ByteArray. I can see how to use such an
> ExternalStructure to pass the struct, or a pointer to it, to a function in
> the library. Is there a similar object I can create which allows me to
> read/write an array of these structs (i.e. does the proper marshaling
> to/from a ByteArray)?
>
> I got the i/f working by rolling my own -- directly filling up a ByteArray
> with the right number of instances of the serialized-by-hand structure,
> creating an ExternalData instance with said ByteArray as the handle, and
> passing this as the parameter. But this doesn't allow me to use the field
> accessor methods created by ExternalStructure, so it feels like I'm missing
> something.
>
> For concreteness, the API looks something like this:
>
> typedef struct
> {
>     uint32_t someId;
>     int32_t  someInt;
> } fooStruct;
>
> int32_t apiFunction
> (
>  const  fooStruct *foo_list,
>  uint32_t foo_list_size
>  );
>
> Any suggestions?
>
> Thanks,
> Doug
>
>
>

> AFAIK, you'll have to use void * in API because arrays of structures
> are not supported.
> I mean FFI is not able to perform any type verification on array of
> structures...
>
> I created this class in Smallapack:
>
> ArrayedCollection subclass: #ExternalArray
> instanceVariableNames: 'handle size data'
> classVariableNames: 'EndianCache'
> poolDictionaries: ''
> category: 'Smallapack-Collection'
>
> - handle holds the pointer (ExternalAddress),
> - size the number of elements,
> - data is the object to be passed to the foreign function call (it
> holds address + type information).
> And here is how I had to initialize data:
>
> resetExternalData
> "Reset the ExternalData object used as FFI argument.
> This is cached in an inst-var instead of being recreated at each call.
> However, this must be reset whenever handle is changed"
> | cType |
> cType := self class type.
> data := cType isStructureType
> ifTrue:
> ["Arrays of structure cannot carry type checking in FFI"
> ExternalData fromHandle: handle type: ExternalType void]
> ifFalse:
> ["Atomic types"
> ExternalData fromHandle: handle type: self class type]
>
> Nicolas
>
> 2012/6/10 Douglas McPherson <[hidden email]>:
>> All,
>>
>> I'm trying to call a function in an external library which takes an array of
>> structures as one of its parameters. I can see how to define a subclass of
>> ExternalStructure, and how that creates convenient field accessors which
>> read/write an underlying ByteArray. I can see how to use such an
>> ExternalStructure to pass the struct, or a pointer to it, to a function in
>> the library. Is there a similar object I can create which allows me to
>> read/write an array of these structs (i.e. does the proper marshaling
>> to/from a ByteArray)?
>>
>> I got the i/f working by rolling my own -- directly filling up a ByteArray
>> with the right number of instances of the serialized-by-hand structure,
>> creating an ExternalData instance with said ByteArray as the handle, and
>> passing this as the parameter. But this doesn't allow me to use the field
>> accessor methods created by ExternalStructure, so it feels like I'm missing
>> something.
>>
>> For concreteness, the API looks something like this:
>>
>> typedef struct
>> {
>> uint32_t someId;
>> int32_t  someInt;
>> } fooStruct;
>>
>> int32_t apiFunction
>> (
>> const  fooStruct *foo_list,
>> uint32_t foo_list_size
>> );
>>
>> Any suggestions?
>>
>> Thanks,
>> Doug
>>
>>
>>
>

> Awesome. I took at a look at ExternalArray and its subclasses. This is
> exactly what I was looking for! In fact I think the implementation of
> ExternalDoubleComplexArray does the exact type of thing I need, which is
> provide the ability to marshal an array of ExternalStructure to/from an
> underlying byte array.
>
> I've never looked at Smallapack till now. Do you think that ExternalArray
> etc should be made part of FFI, since it has uses beyond just Smallapack?
> Are there caveats? I notice that many methods have a comment like "Use with
> caution ! Might result in segmentation fault". Is this because of
> verification limitations, alignment issues, or are there other concerns as
> well?
>

> Nice, nice :)
>
> Thanks,
> Doug
>
>
>
> On Jun 10, 2012, at 13:00 , Nicolas Cellier wrote:
>
> AFAIK, you'll have to use void * in API because arrays of structures
>
> are not supported.
>
> I mean FFI is not able to perform any type verification on array of
>
> structures...
>
>
> I created this class in Smallapack:
>
>
> ArrayedCollection subclass: #ExternalArray
>
> instanceVariableNames: 'handle size data'
>
> classVariableNames: 'EndianCache'
>
> poolDictionaries: ''
>
> category: 'Smallapack-Collection'
>
>
> - handle holds the pointer (ExternalAddress),
>
> - size the number of elements,
>
> - data is the object to be passed to the foreign function call (it
>
> holds address + type information).
>
> And here is how I had to initialize data:
>
>
> resetExternalData
>
> "Reset the ExternalData object used as FFI argument.
>
> This is cached in an inst-var instead of being recreated at each call.
>
> However, this must be reset whenever handle is changed"
>
> | cType |
>
> cType := self class type.
>
> data := cType isStructureType
>
> ifTrue:
>
> ["Arrays of structure cannot carry type checking in FFI"
>
> ExternalData fromHandle: handle type: ExternalType void]
>
> ifFalse:
>
> ["Atomic types"
>
> ExternalData fromHandle: handle type: self class type]
>
>
> Nicolas
>
>
> 2012/6/10 Douglas McPherson <[hidden email]>:
>
> All,
>
>
> I'm trying to call a function in an external library which takes an array of
>
> structures as one of its parameters. I can see how to define a subclass of
>
> ExternalStructure, and how that creates convenient field accessors which
>
> read/write an underlying ByteArray. I can see how to use such an
>
> ExternalStructure to pass the struct, or a pointer to it, to a function in
>
> the library. Is there a similar object I can create which allows me to
>
> read/write an array of these structs (i.e. does the proper marshaling
>
> to/from a ByteArray)?
>
>
> I got the i/f working by rolling my own -- directly filling up a ByteArray
>
> with the right number of instances of the serialized-by-hand structure,
>
> creating an ExternalData instance with said ByteArray as the handle, and
>
> passing this as the parameter. But this doesn't allow me to use the field
>
> accessor methods created by ExternalStructure, so it feels like I'm missing
>
> something.
>
>
> For concreteness, the API looks something like this:
>
>
> typedef struct
>
> {
>
> uint32_t someId;
>
> int32_t  someInt;
>
> } fooStruct;
>
>
> int32_t apiFunction
>
> (
>
> const  fooStruct *foo_list,
>
> uint32_t foo_list_size
>
> );
>
>
> Any suggestions?
>
>
> Thanks,
>
> Doug
>
>
>
>
>
>
>
>
>

> 2012/6/12 Douglas McPherson <[hidden email]>:
>> Awesome. I took at a look at ExternalArray and its subclasses. This is
>> exactly what I was looking for! In fact I think the implementation of
>> ExternalDoubleComplexArray does the exact type of thing I need, which is
>> provide the ability to marshal an array of ExternalStructure to/from an
>> underlying byte array.
>>
>> I've never looked at Smallapack till now. Do you think that ExternalArray
>> etc should be made part of FFI, since it has uses beyond just Smallapack?
>> Are there caveats? I notice that many methods have a comment like "Use with
>> caution ! Might result in segmentation fault". Is this because of
>> verification limitations, alignment issues, or are there other concerns as
>> well?
>>
>
> Part of FFI? I never thought of it, but why not...
>
> Caveats? obviously, passing an unbounded pointer as parameter to an
> external function is not going to be fool proof any time soon...
> VM isn't going to enforce anything, it's all under programmer responsibility...
> Table overflow generally hang the VM quickly when happening in
> VM-managed memory, but there is no guaranty, you can as well save a
> corrupted image.
> Table overflow in external heap might get un-noticed a bit longer, so
> testing and debugging is not that easy.
> Another source of error is miss-definition of external function
> interface (I'm not aware of any signature verification, except maybe
> the @20 suffix decoration of M$ stdcall...).
> For the case of Lapack, there are many functions with many parameters,
> some defining the array bounds straightly, other more indirectly...
> This kind of library is exposed to both copy/paste errors and
> combinatorial complexity errors (like when a function works with a
> matrix transposed or not, with different bounds interpretation when it
> is transposed, or with different bounds expectations for optional
> outputs depending on the requested options, different work array
> bounds expectations etc...). Due to work array bounds requirement
> complexity, it's even not excluded that lapack parameter checking has
> flaws (I think there were some corrections related to this in lapack
> 3.3, you can dig in http://www.netlib.org/lapack/release_notes.html).
> I tried my best, and also generated automatically some low level FFi
> calls, but coverage is not very high so far...
>
> However, using bounded-garbage-collectable-memory - ByteArray would
> make an ExternalArray quite safe for in-image-handling.
> Except two problems...
> 1) The main problems I had with Smallapack was because ByteArray are 4
> bytes-aligned, while some external function expect 8-byte alignment
> for array of double... (I think this is related to usage of SS2/SSE3,
> but not totally sure).
> 2) The second problem is that I can't pass the address of a sub-array with FFI
> For example, {double *data=...; memmove(data+2,data,5*8);} is a FFI
> call you can do with external heap pointer, but not with a ByteArray.
> IMHO, this would deserve an extension to FFI, because emulating low
> level operations with bounded memory is a nice property. I used this
> kind of trick to handle real or imaginary part of a complex matrix
> (with a strand of 2), but this also apply to other sub-matrices (a
> row, a column, a sub/super diagonal).
>