FFI: FFI-Kernel-nice.118.mcz

Hi Nicolas,

> I'd like to simply write (MyStruct new: 25) to get an array of 25 MyStruct.

> I DO NOT want to see the ExternalData, it's an implementation detail blurring the intentions.

Then you would just hide what I just proposed behind #new: in your struct. :-) I just argued to not put 

#allocate: in ExternalType.

However, you want to treat MyStruct to be able to answer an array of itself. This is not okay. Well, it might be C-style, but not Smalltalk-style. We should try to find a connection between C-world and Smalltalk-world. Would you do that with "Morph new: 25"? Expect an array of 25 morphs?

If you want to have an array, ExternalData is the closes we have at the moment. It can directly work like Collection.

So, #new: (or similar) is a thing you want to implement in ExternalData, not MyStruct.

Well ... if you insist on creating an array of MyStruct through MyStruct .... I would at least not use #new: but rather #newArray: ... 

> I want to see/handle my domain specific "objects" DIRECTLY.

Sure. But having an array of your domain specific objects is handled via ExternalData. ExternalData is like Collection (or Array).

Best,

Marcel

Am 22.06.2020 10:21:23 schrieb Nicolas Cellier <[hidden email]>:

Hi Marcel,

The idea is to manipulate the alias or structures types as if they were classes.

I already can write MyStruct new, and pass that to an external function call.

I'd like to simply write (MyStruct new: 25) to get an array of 25 MyStruct.

I DO NOT want to see the ExternalData, it's an implementation detail blurring the intentions.

I want to see/handle my domain specific "objects" DIRECTLY.

That's what I manipulate in function signatures, etc...

Since external types are not exactly classes, i prefer to have another vocabulary in place of new:.

In VW DLLCC, the selectors malloc malloc: calloc calloc: gcMalloc gcMalloc: are sent to the CType.

Unlike us, DLLCC does not allocate in objectMemory, always on the heap.

So having the C vocabulary is logical for them.

Allocating in ObjectMemory is one essential feature of Squeak FFI.

That's why I prefer not use C vocabulary.

#allocate: is something in between.

I know that it is already used on the low level for ExternalAddress, which is unfortunate.

Le lun. 22 juin 2020 à 09:31, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas.

I like the idea of having #allocate: ... but maybe not in ExternalType:

- The term "allocate:" is somewhat reserved for heap memory from the SqueakFFI perspective; See ExternalAddress class >> #allocate:.

- The counterpart #free is missing ... which means that, maybe, both #allocate: and #free should be implemented in ExternalData, not ExternalType

What about such a this programming interface:

data := ExternalData

   fromHandle: nil

   type: MyStruct externalType asPointerType.

data allocate.

myStruct := data asExternalStructure.

You could also copy it back to object memory:

myStruct := data getExternalStructure. "get/pull/fetch ...."

data free.

You are proposing a way to start off in object memory. Maybe like this:

data := ExternalData

   fromHandle: nil "maybe optional?"

   type: MyStruct externalType asPointerType.

data allocateLocal.

myStruct := data asExternalStructure.

Then we don't need to call #free. See how #allocate and #allocateLocal in ExternalData could mirror #new and #externalNew of ExternalStructure?

There is really an interesting releationship between ExternalStructure (incl. ExternalUnion/ExternalTypeAlias/ExternalPackedStructure) and ExternalData. Can you see it? ^__^

Arrays could be similar:

data := ExternalData type: Externaltype int32_t asPointerType.

data size: 20.

data allocate. "or #allocateLocal"

Now you would have an array of 20 integers. :-) Note that #allocate: with argument could inline that #size: call.

Best,

Marcel

Am 21.06.2020 13:20:44 schrieb [hidden email] <[hidden email]>:

Nicolas Cellier uploaded a new version of FFI-Kernel to project FFI:
http://source.squeak.org/FFI/FFI-Kernel-nice.118.mcz

==================== Summary ====================

Name: FFI-Kernel-nice.118
Author: nice
Time: 21 June 2020, 1:20:34.369348 pm
UUID: a8c48997-991d-4e32-9b67-6d93fc0dabe0
Ancestors: FFI-Kernel-mt.117

Add the ability to allocate: an ExternalType or an ExternalStructure/Alias.

It's equivalent to new: but we want a specific idiom, C types are not exactly classes.

Add the ability to compare aliases (at least for equality)

=============== Diff against FFI-Kernel-mt.117 ===============

Item was added:
+ ----- Method: ExternalStructure class>>allocate: (in category 'instance creation') -----
+ allocate: anInteger
+ "Create an ExternalData with enough room for storing an array of size anInteger of such structure"
+ ^self externalType allocate: anInteger!

Item was added:
+ ----- Method: ExternalType>>allocate: (in category 'external data') -----
+ allocate: anInteger
+ "Allocate space for containing an array of size anInteger of this dataType"
+
+ | handle |
+ handle := ByteArray new: self byteSize * anInteger.
+ ^(ExternalData fromHandle: handle type: self) size: anInteger!

Item was added:
+ ----- Method: ExternalTypeAlias>>= (in category 'comparing') -----
+ = anExternalTypeAlias
+ ^self class = anExternalTypeAlias class and: [self value = anExternalTypeAlias value]!

Item was added:
+ ----- Method: ExternalTypeAlias>>hash (in category 'comparing') -----
+ hash
+ ^self class hash hashMultiply bitXor: self value hash!

> We're interfacing C.

From within Smalltalk. Let's not forget that. We have to find trade-offs.

> In C, I can simply have MyStruct foo; or MyStruct foo[10];

Then add #newArray: to ExternalStructure, not #new:. To not confuse C and Smalltalk terminology more than we must. Yet, I would not want to write an FFI bridge such a way. But both ways should be possible. :-)

> ExternalData? The name does not speak as Collection.

Could be renamed, but maybe not. Because it could remain a transperent helper.

> I do not want to manipulate ExternalData by myself. I do not want to see it.

Then hide the use of ExternalData  through ExternalStructure. But favor #newArray: over #new:. As argued before.

> I do want to handle my domain objects DIRECTLY, without intermediary (or at least have the illusion of).

That's where we agree on. :-) 

> So direct manipulation of MyStruct without any intermediary is much more appealing/convenient.

Ehm ... sure. But MyStruct and array of MyStruct should not both be handled via the same MyStruct class in Squeak FFI. Your ExternalArray could be a simple subclass of ExternalData. If you like such a name better. Implementation wouldn't change at all. :-)

Best,

Marcel

Am 22.06.2020 10:55:14 schrieb Nicolas Cellier <[hidden email]>:

Le lun. 22 juin 2020 à 10:30, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas,

> I'd like to simply write (MyStruct new: 25) to get an array of 25 MyStruct.

> I DO NOT want to see the ExternalData, it's an implementation detail blurring the intentions.

Then you would just hide what I just proposed behind #new: in your struct. :-) I just argued to not put 

#allocate: in ExternalType.

However, you want to treat MyStruct to be able to answer an array of itself. This is not okay. Well, it might be C-style, but not Smalltalk-style. We should try to find a connection between C-world and Smalltalk-world. Would you do that with "Morph new: 25"? Expect an array of 25 morphs?

If you want to have an array, ExternalData is the closes we have at the moment. It can directly work like Collection.

So, #new: (or similar) is a thing you want to implement in ExternalData, not MyStruct.

Well ... if you insist on creating an array of MyStruct through MyStruct .... I would at least not use #new: but rather #newArray: ... 

In your mind, ExternalData <=> Array.

(Array of: MyStruct) new: 10 <=> (ExternalData type: MyStruct) new: 10.

That's what you tend toward: add some of Collection protocol to ExternalData.

If you have a look at Smallapack ExternalArray, that's somehow what I tried to achieve years ago...

have Collection protocol for arrays external objects.

My POV is different.

We're interfacing C.

In C, I can simply have MyStruct foo; or MyStruct foo[10];

So direct manipulation of MyStruct without any intermediary is much more appealing/convenient.

ExternalData? The name does not speak as Collection.

In fact it's a hurdle. It's verbiage. It's too low level, like all the external*

I do not see the ExternalFunction, can live without EternalAddress, etc...

I do not want to manipulate ExternalData by myself. I do not want to see it.

I do not want to have to tell an ExternalData to create an array of the type I need for myself.

I do want to handle my domain objects DIRECTLY, without intermediary (or at least have the illusion of).

See it?

> I want to see/handle my domain specific "objects" DIRECTLY.

Sure. But having an array of your domain specific objects is handled via ExternalData. ExternalData is like Collection (or Array).

Best,

Marcel

Am 22.06.2020 10:21:23 schrieb Nicolas Cellier <[hidden email]>:

Hi Marcel,

The idea is to manipulate the alias or structures types as if they were classes.

I already can write MyStruct new, and pass that to an external function call.

I'd like to simply write (MyStruct new: 25) to get an array of 25 MyStruct.

I DO NOT want to see the ExternalData, it's an implementation detail blurring the intentions.

I want to see/handle my domain specific "objects" DIRECTLY.

That's what I manipulate in function signatures, etc...

Since external types are not exactly classes, i prefer to have another vocabulary in place of new:.

In VW DLLCC, the selectors malloc malloc: calloc calloc: gcMalloc gcMalloc: are sent to the CType.

Unlike us, DLLCC does not allocate in objectMemory, always on the heap.

So having the C vocabulary is logical for them.

Allocating in ObjectMemory is one essential feature of Squeak FFI.

That's why I prefer not use C vocabulary.

#allocate: is something in between.

I know that it is already used on the low level for ExternalAddress, which is unfortunate.

Le lun. 22 juin 2020 à 09:31, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas.

I like the idea of having #allocate: ... but maybe not in ExternalType:

- The term "allocate:" is somewhat reserved for heap memory from the SqueakFFI perspective; See ExternalAddress class >> #allocate:.

- The counterpart #free is missing ... which means that, maybe, both #allocate: and #free should be implemented in ExternalData, not ExternalType

What about such a this programming interface:

data := ExternalData

   fromHandle: nil

   type: MyStruct externalType asPointerType.

data allocate.

myStruct := data asExternalStructure.

You could also copy it back to object memory:

myStruct := data getExternalStructure. "get/pull/fetch ...."

data free.

You are proposing a way to start off in object memory. Maybe like this:

data := ExternalData

   fromHandle: nil "maybe optional?"

   type: MyStruct externalType asPointerType.

data allocateLocal.

myStruct := data asExternalStructure.

Then we don't need to call #free. See how #allocate and #allocateLocal in ExternalData could mirror #new and #externalNew of ExternalStructure?

There is really an interesting releationship between ExternalStructure (incl. ExternalUnion/ExternalTypeAlias/ExternalPackedStructure) and ExternalData. Can you see it? ^__^

Arrays could be similar:

data := ExternalData type: Externaltype int32_t asPointerType.

data size: 20.

data allocate. "or #allocateLocal"

Now you would have an array of 20 integers. :-) Note that #allocate: with argument could inline that #size: call.

Best,

Marcel

Am 21.06.2020 13:20:44 schrieb [hidden email] <[hidden email]>:

Nicolas Cellier uploaded a new version of FFI-Kernel to project FFI:
http://source.squeak.org/FFI/FFI-Kernel-nice.118.mcz

==================== Summary ====================

Name: FFI-Kernel-nice.118
Author: nice
Time: 21 June 2020, 1:20:34.369348 pm
UUID: a8c48997-991d-4e32-9b67-6d93fc0dabe0
Ancestors: FFI-Kernel-mt.117

Add the ability to allocate: an ExternalType or an ExternalStructure/Alias.

It's equivalent to new: but we want a specific idiom, C types are not exactly classes.

Add the ability to compare aliases (at least for equality)

=============== Diff against FFI-Kernel-mt.117 ===============

Item was added:
+ ----- Method: ExternalStructure class>>allocate: (in category 'instance creation') -----
+ allocate: anInteger
+ "Create an ExternalData with enough room for storing an array of size anInteger of such structure"
+ ^self externalType allocate: anInteger!

Item was added:
+ ----- Method: ExternalType>>allocate: (in category 'external data') -----
+ allocate: anInteger
+ "Allocate space for containing an array of size anInteger of this dataType"
+
+ | handle |
+ handle := ByteArray new: self byteSize * anInteger.
+ ^(ExternalData fromHandle: handle type: self) size: anInteger!

Item was added:
+ ----- Method: ExternalTypeAlias>>= (in category 'comparing') -----
+ = anExternalTypeAlias
+ ^self class = anExternalTypeAlias class and: [self value = anExternalTypeAlias value]!

Item was added:
+ ----- Method: ExternalTypeAlias>>hash (in category 'comparing') -----
+ hash
+ ^self class hash hashMultiply bitXor: self value hash!

> For the generalization to other atomic types, we indeed need to go through the ExternalType and implement allocate: there too.

Not needed. You can access ExternalType >> #byteSize from within ExternalData. It's public.

Best,

Marcel

Am 22.06.2020 12:08:47 schrieb Nicolas Cellier <[hidden email]>:

Le lun. 22 juin 2020 à 11:05, Marcel Taeumel <[hidden email]> a écrit :

> We're interfacing C.

From within Smalltalk. Let's not forget that. We have to find trade-offs.

> In C, I can simply have MyStruct foo; or MyStruct foo[10];

Then add #newArray: to ExternalStructure, not #new:. To not confuse C and Smalltalk terminology more than we must. Yet, I would not want to write an FFI bridge such a way. But both ways should be possible. :-)

> ExternalData? The name does not speak as Collection.

Could be renamed, but maybe not. Because it could remain a transperent helper.

> I do not want to manipulate ExternalData by myself. I do not want to see it.

Then hide the use of ExternalData  through ExternalStructure. But favor #newArray: over #new:. As argued before.

exactly, that's why I have used #allocate:, not new:, those types are not exactly classes, though MyStruct is a class...

> I do want to handle my domain objects DIRECTLY, without intermediary (or at least have the illusion of).

That's where we agree on. :-) 

> So direct manipulation of MyStruct without any intermediary is much more appealing/convenient.

Ehm ... sure. But MyStruct and array of MyStruct should not both be handled via the same MyStruct class in Squeak FFI. Your ExternalArray could be a simple subclass of ExternalData. If you like such a name better. Implementation wouldn't change at all. :-)

Absolutely!

I want to create those ExternalData. It's just that I want to ask my object to create those, I don't want to ask ExternalData to create my objects.

For the generalization to other atomic types, we indeed need to go through the ExternalType and implement allocate: there too.

That's somehow how it is done in DLLCC, and the API is convenient IMO.

Best,

Marcel

Am 22.06.2020 10:55:14 schrieb Nicolas Cellier <[hidden email]>:

Le lun. 22 juin 2020 à 10:30, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas,

> I'd like to simply write (MyStruct new: 25) to get an array of 25 MyStruct.

> I DO NOT want to see the ExternalData, it's an implementation detail blurring the intentions.

Then you would just hide what I just proposed behind #new: in your struct. :-) I just argued to not put 

#allocate: in ExternalType.

However, you want to treat MyStruct to be able to answer an array of itself. This is not okay. Well, it might be C-style, but not Smalltalk-style. We should try to find a connection between C-world and Smalltalk-world. Would you do that with "Morph new: 25"? Expect an array of 25 morphs?

If you want to have an array, ExternalData is the closes we have at the moment. It can directly work like Collection.

So, #new: (or similar) is a thing you want to implement in ExternalData, not MyStruct.

Well ... if you insist on creating an array of MyStruct through MyStruct .... I would at least not use #new: but rather #newArray: ... 

In your mind, ExternalData <=> Array.

(Array of: MyStruct) new: 10 <=> (ExternalData type: MyStruct) new: 10.

That's what you tend toward: add some of Collection protocol to ExternalData.

If you have a look at Smallapack ExternalArray, that's somehow what I tried to achieve years ago...

have Collection protocol for arrays external objects.

My POV is different.

We're interfacing C.

In C, I can simply have MyStruct foo; or MyStruct foo[10];

So direct manipulation of MyStruct without any intermediary is much more appealing/convenient.

ExternalData? The name does not speak as Collection.

In fact it's a hurdle. It's verbiage. It's too low level, like all the external*

I do not see the ExternalFunction, can live without EternalAddress, etc...

I do not want to manipulate ExternalData by myself. I do not want to see it.

I do not want to have to tell an ExternalData to create an array of the type I need for myself.

I do want to handle my domain objects DIRECTLY, without intermediary (or at least have the illusion of).

See it?

> I want to see/handle my domain specific "objects" DIRECTLY.

Sure. But having an array of your domain specific objects is handled via ExternalData. ExternalData is like Collection (or Array).

Best,

Marcel

Am 22.06.2020 10:21:23 schrieb Nicolas Cellier <[hidden email]>:

Hi Marcel,

The idea is to manipulate the alias or structures types as if they were classes.

I already can write MyStruct new, and pass that to an external function call.

I'd like to simply write (MyStruct new: 25) to get an array of 25 MyStruct.

I DO NOT want to see the ExternalData, it's an implementation detail blurring the intentions.

I want to see/handle my domain specific "objects" DIRECTLY.

That's what I manipulate in function signatures, etc...

Since external types are not exactly classes, i prefer to have another vocabulary in place of new:.

In VW DLLCC, the selectors malloc malloc: calloc calloc: gcMalloc gcMalloc: are sent to the CType.

Unlike us, DLLCC does not allocate in objectMemory, always on the heap.

So having the C vocabulary is logical for them.

Allocating in ObjectMemory is one essential feature of Squeak FFI.

That's why I prefer not use C vocabulary.

#allocate: is something in between.

I know that it is already used on the low level for ExternalAddress, which is unfortunate.

Le lun. 22 juin 2020 à 09:31, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas.

I like the idea of having #allocate: ... but maybe not in ExternalType:

- The term "allocate:" is somewhat reserved for heap memory from the SqueakFFI perspective; See ExternalAddress class >> #allocate:.

- The counterpart #free is missing ... which means that, maybe, both #allocate: and #free should be implemented in ExternalData, not ExternalType

What about such a this programming interface:

data := ExternalData

   fromHandle: nil

   type: MyStruct externalType asPointerType.

data allocate.

myStruct := data asExternalStructure.

You could also copy it back to object memory:

myStruct := data getExternalStructure. "get/pull/fetch ...."

data free.

You are proposing a way to start off in object memory. Maybe like this:

data := ExternalData

   fromHandle: nil "maybe optional?"

   type: MyStruct externalType asPointerType.

data allocateLocal.

myStruct := data asExternalStructure.

Then we don't need to call #free. See how #allocate and #allocateLocal in ExternalData could mirror #new and #externalNew of ExternalStructure?

There is really an interesting releationship between ExternalStructure (incl. ExternalUnion/ExternalTypeAlias/ExternalPackedStructure) and ExternalData. Can you see it? ^__^

Arrays could be similar:

data := ExternalData type: Externaltype int32_t asPointerType.

data size: 20.

data allocate. "or #allocateLocal"

Now you would have an array of 20 integers. :-) Note that #allocate: with argument could inline that #size: call.

Best,

Marcel

Am 21.06.2020 13:20:44 schrieb [hidden email] <[hidden email]>:

Nicolas Cellier uploaded a new version of FFI-Kernel to project FFI:
http://source.squeak.org/FFI/FFI-Kernel-nice.118.mcz

==================== Summary ====================

Name: FFI-Kernel-nice.118
Author: nice
Time: 21 June 2020, 1:20:34.369348 pm
UUID: a8c48997-991d-4e32-9b67-6d93fc0dabe0
Ancestors: FFI-Kernel-mt.117

Add the ability to allocate: an ExternalType or an ExternalStructure/Alias.

It's equivalent to new: but we want a specific idiom, C types are not exactly classes.

Add the ability to compare aliases (at least for equality)

=============== Diff against FFI-Kernel-mt.117 ===============

Item was added:
+ ----- Method: ExternalStructure class>>allocate: (in category 'instance creation') -----
+ allocate: anInteger
+ "Create an ExternalData with enough room for storing an array of size anInteger of such structure"
+ ^self externalType allocate: anInteger!

Item was added:
+ ----- Method: ExternalType>>allocate: (in category 'external data') -----
+ allocate: anInteger
+ "Allocate space for containing an array of size anInteger of this dataType"
+
+ | handle |
+ handle := ByteArray new: self byteSize * anInteger.
+ ^(ExternalData fromHandle: handle type: self) size: anInteger!

Item was added:
+ ----- Method: ExternalTypeAlias>>= (in category 'comparing') -----
+ = anExternalTypeAlias
+ ^self class = anExternalTypeAlias class and: [self value = anExternalTypeAlias value]!

Item was added:
+ ----- Method: ExternalTypeAlias>>hash (in category 'comparing') -----
+ hash
+ ^self class hash hashMultiply bitXor: self value hash!

Le lun. 22 juin 2020 à 12:11, Marcel Taeumel <[hidden email]> a écrit :

> For the generalization to other atomic types, we indeed need to go through the ExternalType and implement allocate: there too.

Not needed. You can access ExternalType >> #byteSize from within ExternalData. It's public.

Best,

Marcel

Hi Marcel,

you constantly bring me to a lower level API ...

I know that byteSize is public, but I don't want to bother with that.

I want to handle an array of N things, not an ExternalData withHandle: (ByteArray new: thing byteSize * N) type: thing.

I want (thing allocate: N), that's all I need...

I've used similar constructs provided by VW DLLCC, and believe me, it's convenient!

Beside, think of it, an ExternalData has to provide both memory AND type right?

    ExternalData fromHandle: (ByteArray new: type byteSize * anInteger) type: type.

Who knows better about its own size than the type itself?

Why scatter the responsibilities that could be gathered in the type?

Where would this snippet go?