Re: [squeak-dev] FFI ExternalTypeAlias

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

Nicolas Cellier
 
Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :
Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel

Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:

Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)



Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

marcel.taeumel
 
Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Best,
Marcel

Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :

> Hi Marcel,
> thanks for your vote. Anyone else?
>
> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> écrit :
>
>> Hi Nicolas, hi all! :-)
>>
>> > Type alias are subclasses of ExternalStructure.
>>
>> And with it, they get their own instance of ExternalType, which is
>> managed in the classVar StructTypes. Let's call them struct type.
>>
>> > Now we have another mean by adding a class side method in ExternalType.
>>
>> That kind of aliasing is for compile-time type referencing only -- such
>> as in FFI calls (i.e. pragmas) or struct-field definitions
>> (i.e. #fields). (And soon if I have the first version of
>> FFI-Callback to show you :)
>>
>> Because those aliases become fully transparent after compiling the method
>> for the FFI call into an instance of ExternalLibraryFunction and after
>> compiling the field specs of external structures into compiledSpec for
>> struct types.
>>
>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',
>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'
>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI
>> vocabulary. Nothing more. Nothing less.
>>
>> Having that, please do not use class-side methods in ExternalType to
>> alias a (complex) struct type. Use them only for renaming atomic types.
>> Please. We may want to add checks to enforce that.
>>
>> I suppose that this discussion focuses on type aliases that are
>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>> thus get their own struct type. So, let's ignore this other mechanism for
>> now.
>>
>> > This has one advantage: type safety. You cannot pass a Foo to a
>> function expecting a Bar, even if they both alias int...
>>
>> Yes! So, we are now talking about type aliases to atomic types. The
>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>> this:
>>
>> (...forgive my "creative" representation of WordArray here...)
>>
>> compiledSpec: 0A 04 00 01
>> referentClass: Foo
>>
>> How does the atomic type for 'char' look like?
>>
>> compiledSpec: 0A 04 00 01
>> referentClass: nil
>>
>> Consequently, argument coercing will fail if you pass a 'char' when a
>> 'Foo' is expected. Because the referentClass does not match --- even if the
>> compiledSpec does match.
>>
>> > But this has one major drawback: with current FFI, you cannot pass a
>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>> an integer type...
>>
>> That's correct. If the origin of such an argument is from within the
>> image, you have to wrap it. But when returned from another call ... well
>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>
>> > FFI plugin is clever enough to recognize an atomic type alias and
>> simply return a Smalltalk Integer...
>> > So it's not Bar all the way down, I have to manually wrap Bar...
>> > This is not consistent.
>>
>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>> because it does so for argument coercing.
>>
>> Now I understand why the code generation of struct-field accessors was so
>> apparently broken for me. I changed that so that having an alias type as
>> part of a larger struct will automatically wrap that for you into the alias
>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>> ... Ha ha. ;-) #ffiLongVsInt.
>>
>> It should be "Bar all the way down". Definitely.
>>
>> > If we return an int, we must accept an int, otherwise, we cannot chain
>> FFI calls...
>>
>> Well, not for type aliases. I would like keep the path of type safety
>> here, you mentioned above.
>>
>> Just fix the FFI plugin to respect referentClass when packaging the
>> return value.
>>
>> > I could use the lightweight alias instead, [...]
>>
>> Please don't. See above. Those "lightweight alias" are for renaming
>> atomic types only. Let's keep it simple and try to address you concern here
>> with struct types and ExternalTypeAlias. :-)
>>
>> > there was another usage where it was convenient to use
>> ExternalTypeAlias: enum.
>> > Indeed, I simply defined all enum symbols as class side method.
>>
>> I like that! It reads like a next step for ExternalPool. Once you have
>> extracted the constant values from header files, and once you have those
>> values for your platform in classVars in your external pool, use that pool
>> to define enums through ExternalStructure.
>>
>> Like this:
>>
>> ExternalTypeAlias subclass: #MyEnumBar
>> instanceVariableNames: ''
>> classVariableNames: ''
>> poolDictionaries: 'HDF5Pool'
>> category: 'HDF5'
>>
>> MyEnumBar class >> #poolFields
>> "
>> self defineFields.
>> "
>> ^ #(
>> (beg HDF5_BEG)
>> (mid HDF5_MID)
>> (end HDF5_END)
>> )
>>
>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>> external-pool definitions. See class comment in ExternalPool to get started.
>>
>> Note that #poolFields would translate to class-side methods as you
>> suggested.
>>
>> > If I use a simpler alias, where are those ids going to?
>>
>> Please don't. See above. :-)
>>
>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>> inputs/outputs behave consistently.
>>
>> Absolutely!
>>
>> > - should a function expecting an ExternalTypeAlias of atomic type
>> accept an immediate value of compatible type?
>>
>> Considering type safety, I think not.
>>
>> Well ... since this is a convenience issue and thus not about saving
>> run-time cost ... What about adding a callback into the image to react on
>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>> stored in ExternalLibraryFunction since this is accessible to the plugin
>> anyway?
>>
>>
>>
>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>
>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>> FFI-Callback. But ignore me here :)
>>
>> > - should a function returning an ExternalTypeAlias of atomic type
>> instantiate the Alias?
>>
>> Yes, please! See above.
>>
>> If you want to trade type safety in for a performance gain, just use a
>> "lightweight alias" as explained above. And package that return value
>> manually.
>>
>> Best,
>> Marcel
>>
>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:
>> Hi all,
>> following the question of Marcel about usage of ExternalTypeAlias:
>>
>> Type alias are subclasses of ExternalStructure.
>> I used them in HDF5 because there was no other means to define an alias
>> at the time.
>> Now we have another mean by adding a class side method in ExternalType (I
>> would have rather imagined the usage of some annotation to get a
>> declarative style)
>>
>> This has one advantage: type safety. You cannot pass a Foo to a function
>> expecting a Bar, even if they both alias int...
>>
>> But this has one major drawback: with current FFI, you cannot pass a
>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>> an integer type...
>>
>> Thus, you have to wrap every Bar argument into a Bar instance...
>> Gasp, this is going too far...
>>
>> What about functions returning such alias?
>> Function returning struct by value allocate a struct, but it's not the
>> case here, FFI plugin is clever enough to recognize an atomic type alias
>> and simply return a Smalltalk Integer...
>> So it's not Bar all the way down, I have to manually wrap Bar...
>> This is not consistent.
>> If we return an int, we must accept an int, otherwise, we cannot chain
>> FFI calls...
>>
>> I could use the lightweight alias instead, but there was another usage
>> where it was convenient to use ExternalTypeAlias: enum.
>> Indeed, I simply defined all enum symbols as class side method. For
>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>> This way, I normally can pass a Bar mid to an external function.
>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>
>> If I use a simpler alias, where are those ids going to?
>>
>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>> inputs/outputs behave consistently.
>> The choice is this one:
>> - should a function expecting an ExternalTypeAlias of atomic type accept
>> an immediate value of compatible type? (the permissive implementation)
>> - should a function returning an ExternalTypeAlias of atomic type
>> instantiate the Alias? (the typesafe implementation, with higher runtime
>> cost due to pressure on GC)
>>
>>
>>
>>
Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel

Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:


Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)








Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

Nicolas Cellier
 


Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :
 
Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...
The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).
This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,
Marcel

Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :

> Hi Marcel,
> thanks for your vote. Anyone else?
>
> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> écrit :
>
>> Hi Nicolas, hi all! :-)
>>
>> > Type alias are subclasses of ExternalStructure.
>>
>> And with it, they get their own instance of ExternalType, which is
>> managed in the classVar StructTypes. Let's call them struct type.
>>
>> > Now we have another mean by adding a class side method in ExternalType.
>>
>> That kind of aliasing is for compile-time type referencing only -- such
>> as in FFI calls (i.e. pragmas) or struct-field definitions
>> (i.e. #fields). (And soon if I have the first version of
>> FFI-Callback to show you :)
>>
>> Because those aliases become fully transparent after compiling the method
>> for the FFI call into an instance of ExternalLibraryFunction and after
>> compiling the field specs of external structures into compiledSpec for
>> struct types.
>>
>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',
>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'
>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI
>> vocabulary. Nothing more. Nothing less.
>>
>> Having that, please do not use class-side methods in ExternalType to
>> alias a (complex) struct type. Use them only for renaming atomic types.
>> Please. We may want to add checks to enforce that.
>>
>> I suppose that this discussion focuses on type aliases that are
>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>> thus get their own struct type. So, let's ignore this other mechanism for
>> now.
>>
>> > This has one advantage: type safety. You cannot pass a Foo to a
>> function expecting a Bar, even if they both alias int...
>>
>> Yes! So, we are now talking about type aliases to atomic types. The
>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>> this:
>>
>> (...forgive my "creative" representation of WordArray here...)
>>
>> compiledSpec: 0A 04 00 01
>> referentClass: Foo
>>
>> How does the atomic type for 'char' look like?
>>
>> compiledSpec: 0A 04 00 01
>> referentClass: nil
>>
>> Consequently, argument coercing will fail if you pass a 'char' when a
>> 'Foo' is expected. Because the referentClass does not match --- even if the
>> compiledSpec does match.
>>
>> > But this has one major drawback: with current FFI, you cannot pass a
>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>> an integer type...
>>
>> That's correct. If the origin of such an argument is from within the
>> image, you have to wrap it. But when returned from another call ... well
>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>
>> > FFI plugin is clever enough to recognize an atomic type alias and
>> simply return a Smalltalk Integer...
>> > So it's not Bar all the way down, I have to manually wrap Bar...
>> > This is not consistent.
>>
>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>> because it does so for argument coercing.
>>
>> Now I understand why the code generation of struct-field accessors was so
>> apparently broken for me. I changed that so that having an alias type as
>> part of a larger struct will automatically wrap that for you into the alias
>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>> ... Ha ha. ;-) #ffiLongVsInt.
>>
>> It should be "Bar all the way down". Definitely.
>>
>> > If we return an int, we must accept an int, otherwise, we cannot chain
>> FFI calls...
>>
>> Well, not for type aliases. I would like keep the path of type safety
>> here, you mentioned above.
>>
>> Just fix the FFI plugin to respect referentClass when packaging the
>> return value.
>>
>> > I could use the lightweight alias instead, [...]
>>
>> Please don't. See above. Those "lightweight alias" are for renaming
>> atomic types only. Let's keep it simple and try to address you concern here
>> with struct types and ExternalTypeAlias. :-)
>>
>> > there was another usage where it was convenient to use
>> ExternalTypeAlias: enum.
>> > Indeed, I simply defined all enum symbols as class side method.
>>
>> I like that! It reads like a next step for ExternalPool. Once you have
>> extracted the constant values from header files, and once you have those
>> values for your platform in classVars in your external pool, use that pool
>> to define enums through ExternalStructure.
>>
>> Like this:
>>
>> ExternalTypeAlias subclass: #MyEnumBar
>> instanceVariableNames: ''
>> classVariableNames: ''
>> poolDictionaries: 'HDF5Pool'
>> category: 'HDF5'
>>
>> MyEnumBar class >> #poolFields
>> "
>> self defineFields.
>> "
>> ^ #(
>> (beg HDF5_BEG)
>> (mid HDF5_MID)
>> (end HDF5_END)
>> )
>>
>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>> external-pool definitions. See class comment in ExternalPool to get started.
>>
>> Note that #poolFields would translate to class-side methods as you
>> suggested.
>>
>> > If I use a simpler alias, where are those ids going to?
>>
>> Please don't. See above. :-)
>>
>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>> inputs/outputs behave consistently.
>>
>> Absolutely!
>>
>> > - should a function expecting an ExternalTypeAlias of atomic type
>> accept an immediate value of compatible type?
>>
>> Considering type safety, I think not.
>>
>> Well ... since this is a convenience issue and thus not about saving
>> run-time cost ... What about adding a callback into the image to react on
>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>> stored in ExternalLibraryFunction since this is accessible to the plugin
>> anyway?
>>
>>
>>
>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>
>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>> FFI-Callback. But ignore me here :)
>>
>> > - should a function returning an ExternalTypeAlias of atomic type
>> instantiate the Alias?
>>
>> Yes, please! See above.
>>
>> If you want to trade type safety in for a performance gain, just use a
>> "lightweight alias" as explained above. And package that return value
>> manually.
>>
>> Best,
>> Marcel
>>
>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:
>> Hi all,
>> following the question of Marcel about usage of ExternalTypeAlias:
>>
>> Type alias are subclasses of ExternalStructure.
>> I used them in HDF5 because there was no other means to define an alias
>> at the time.
>> Now we have another mean by adding a class side method in ExternalType (I
>> would have rather imagined the usage of some annotation to get a
>> declarative style)
>>
>> This has one advantage: type safety. You cannot pass a Foo to a function
>> expecting a Bar, even if they both alias int...
>>
>> But this has one major drawback: with current FFI, you cannot pass a
>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>> an integer type...
>>
>> Thus, you have to wrap every Bar argument into a Bar instance...
>> Gasp, this is going too far...
>>
>> What about functions returning such alias?
>> Function returning struct by value allocate a struct, but it's not the
>> case here, FFI plugin is clever enough to recognize an atomic type alias
>> and simply return a Smalltalk Integer...
>> So it's not Bar all the way down, I have to manually wrap Bar...
>> This is not consistent.
>> If we return an int, we must accept an int, otherwise, we cannot chain
>> FFI calls...
>>
>> I could use the lightweight alias instead, but there was another usage
>> where it was convenient to use ExternalTypeAlias: enum.
>> Indeed, I simply defined all enum symbols as class side method. For
>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>> This way, I normally can pass a Bar mid to an external function.
>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>
>> If I use a simpler alias, where are those ids going to?
>>
>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>> inputs/outputs behave consistently.
>> The choice is this one:
>> - should a function expecting an ExternalTypeAlias of atomic type accept
>> an immediate value of compatible type? (the permissive implementation)
>> - should a function returning an ExternalTypeAlias of atomic type
>> instantiate the Alias? (the typesafe implementation, with higher runtime
>> cost due to pressure on GC)
>>
>>
>>
>>
Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel

Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:


Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)








Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

marcel.taeumel
 
> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.

We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,
Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...
The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>


Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :
 






Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...
The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).
This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,
Marcel



Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :


Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel


Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:



Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)














Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

marcel.taeumel
 
Hi Nicolas.

Maybe one more thought on

ExternalData<handle, type>

There is no need to have

ExternalData<1312301580, int>

because it can just be 1312301580.

On the other hand:

ExternalData<#[ 12 34 56 78 ], int>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is already in Squeak's object memory.

Finally:

ExternalData<@00F3A3EE, int*>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is in external memory.

If you want to make an array out of it, make use of the "size" field in ExternalData.

If you want to make a pointer array out of it, change the type to int**

ExternalData<@00F3A3EE, int**>

So, what does this mean then:

ExternalData<#[ 12 34 56 78 ], int*>

Well, it tells you that there is a pointer to an int stored in a byte array. This comes very handy if you think about type aliases to pointer types such as "typedef int* INT_P". Because then it would be

ExternalData<#[ 12 34 56 78 ], INT_P>

To where this pointer points to? Probably garbage. I just made it up for this example. :-D

Best,
Marcel

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:

> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.

We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,
Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...
The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>


Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :
 






Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...
The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).
This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,
Marcel



Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :


Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel


Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:



Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)














Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

marcel.taeumel
 
If you want to make a pointer array out of it, change the type to int**

Correction: If you happen to now, that this external address points to an array of pointers to int, then you would use int* and not int**. You cannot just "decide" what the data behind that external address should be. .. you know what I mean. ;-)

best,
Marcel

Am 19.06.2020 17:47:36 schrieb Marcel Taeumel <[hidden email]>:

Hi Nicolas.

Maybe one more thought on

ExternalData<handle, type>

There is no need to have

ExternalData<1312301580, int>

because it can just be 1312301580.

On the other hand:

ExternalData<#[ 12 34 56 78 ], int>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is already in Squeak's object memory.

Finally:

ExternalData<@00F3A3EE, int*>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is in external memory.

If you want to make an array out of it, make use of the "size" field in ExternalData.

If you want to make a pointer array out of it, change the type to int**

ExternalData<@00F3A3EE, int**>

So, what does this mean then:

ExternalData<#[ 12 34 56 78 ], int*>

Well, it tells you that there is a pointer to an int stored in a byte array. This comes very handy if you think about type aliases to pointer types such as "typedef int* INT_P". Because then it would be

ExternalData<#[ 12 34 56 78 ], INT_P>

To where this pointer points to? Probably garbage. I just made it up for this example. :-D

Best,
Marcel

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:

> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.

We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,
Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...
The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>


Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :
 






Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...
The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).
This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,
Marcel



Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :


Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel


Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:



Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)














Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

marcel.taeumel
 
then you would use int* and not int**

My bad: "  then you would use int** and not int*  " Sorry for the noise -.-"

Am 19.06.2020 17:52:21 schrieb Marcel Taeumel <[hidden email]>:

If you want to make a pointer array out of it, change the type to int**

Correction: If you happen to now, that this external address points to an array of pointers to int, then you would use int* and not int**. You cannot just "decide" what the data behind that external address should be. .. you know what I mean. ;-)

best,
Marcel

Am 19.06.2020 17:47:36 schrieb Marcel Taeumel <[hidden email]>:

Hi Nicolas.

Maybe one more thought on

ExternalData<handle, type>

There is no need to have

ExternalData<1312301580, int>

because it can just be 1312301580.

On the other hand:

ExternalData<#[ 12 34 56 78 ], int>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is already in Squeak's object memory.

Finally:

ExternalData<@00F3A3EE, int*>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is in external memory.

If you want to make an array out of it, make use of the "size" field in ExternalData.

If you want to make a pointer array out of it, change the type to int**

ExternalData<@00F3A3EE, int**>

So, what does this mean then:

ExternalData<#[ 12 34 56 78 ], int*>

Well, it tells you that there is a pointer to an int stored in a byte array. This comes very handy if you think about type aliases to pointer types such as "typedef int* INT_P". Because then it would be

ExternalData<#[ 12 34 56 78 ], INT_P>

To where this pointer points to? Probably garbage. I just made it up for this example. :-D

Best,
Marcel

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:

> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.

We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,
Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...
The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>


Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :
 






Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...
The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).
This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,
Marcel



Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :


Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel


Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:



Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)














Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

marcel.taeumel
 
So, what about this:

ExternalData<@00F3A3EE, int>

I think it could mean that you want to interpret the very pointer address itself as an integer. That is, you do not want to follow to where it points to.

Then what about this:

ExternalData<@00F3A3EE, double>

Ha! An error, I suppose. Unless ... given 8-byte pointers, you can try to read a double out of it. :-D Just kidding. It should be an error.

.
.
.

> > (On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)
> Let me disagree here...

The examples I just wrote down in the previous messages let me realize why you did disagree. :-D

I hope this helps you with improving argument coercing and return value packaging in the FFI plugin. ^__^

Best,
Marcel

Am 19.06.2020 17:53:21 schrieb Marcel Taeumel <[hidden email]>:

then you would use int* and not int**

My bad: "  then you would use int** and not int*  " Sorry for the noise -.-"

Am 19.06.2020 17:52:21 schrieb Marcel Taeumel <[hidden email]>:

If you want to make a pointer array out of it, change the type to int**

Correction: If you happen to now, that this external address points to an array of pointers to int, then you would use int* and not int**. You cannot just "decide" what the data behind that external address should be. .. you know what I mean. ;-)

best,
Marcel

Am 19.06.2020 17:47:36 schrieb Marcel Taeumel <[hidden email]>:

Hi Nicolas.

Maybe one more thought on

ExternalData<handle, type>

There is no need to have

ExternalData<1312301580, int>

because it can just be 1312301580.

On the other hand:

ExternalData<#[ 12 34 56 78 ], int>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is already in Squeak's object memory.

Finally:

ExternalData<@00F3A3EE, int*>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is in external memory.

If you want to make an array out of it, make use of the "size" field in ExternalData.

If you want to make a pointer array out of it, change the type to int**

ExternalData<@00F3A3EE, int**>

So, what does this mean then:

ExternalData<#[ 12 34 56 78 ], int*>

Well, it tells you that there is a pointer to an int stored in a byte array. This comes very handy if you think about type aliases to pointer types such as "typedef int* INT_P". Because then it would be

ExternalData<#[ 12 34 56 78 ], INT_P>

To where this pointer points to? Probably garbage. I just made it up for this example. :-D

Best,
Marcel

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:

> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.

We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,
Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...
The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>


Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :
 






Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...
The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).
This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,
Marcel



Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :


Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel


Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:



Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)














Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

Nicolas Cellier
 
Well, my mental model is this one:

ExternalData needs to refer to an area of memory, so as to be able to act as a pointer (single element or array does not really matter).
If this memory area lies in external heap, then we need an ExternalAddress as handle.
If this memory area lies in object memory, it must be a ByteArray.
We cannot handle pointers to object memory, because object memory is relocatable...
Well, unless we pin the object - that why we pass the ByteArray itself, not its address..

This is independent of how we interpret the contents.
IMO, it's more useful to avoid the pointer indirection in type, because we cannot handle double pointer currently...
Of course, the only pointers that the memory area could contain are pointers to external heap (or eventually pinned objects).

I'd like to have a 3rd possibility: pointing to a sub-region of a ByteArray (ByteArray + offset).
That may be very useful for emulating.
An example in Smallapack: pass a pointer to the imaginary part of a complex array.
Since real and imaginary parts are interleaved, just offsetting the pointer and using a stride of 2 elements does the trick when one wants to access real or imaginary part
(most lapack/blas routines are equipped to handle simple strides).
This is currently possible with ExternalAddress, but NOT when matrix data lies in ObjectMemory.
Another example would be if you wanted to implement an interpreter for a low level language... (emulate a processor, whatever).



Le ven. 19 juin 2020 à 18:04, Marcel Taeumel <[hidden email]> a écrit :
 
So, what about this:

ExternalData<@00F3A3EE, int>

I think it could mean that you want to interpret the very pointer address itself as an integer. That is, you do not want to follow to where it points to.

Then what about this:

ExternalData<@00F3A3EE, double>

Ha! An error, I suppose. Unless ... given 8-byte pointers, you can try to read a double out of it. :-D Just kidding. It should be an error.

.
.
.

> > (On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)
> Let me disagree here...

The examples I just wrote down in the previous messages let me realize why you did disagree. :-D

I hope this helps you with improving argument coercing and return value packaging in the FFI plugin. ^__^

Best,
Marcel

Am 19.06.2020 17:53:21 schrieb Marcel Taeumel <[hidden email]>:

then you would use int* and not int**

My bad: "  then you would use int** and not int*  " Sorry for the noise -.-"

Am 19.06.2020 17:52:21 schrieb Marcel Taeumel <[hidden email]>:

If you want to make a pointer array out of it, change the type to int**

Correction: If you happen to now, that this external address points to an array of pointers to int, then you would use int* and not int**. You cannot just "decide" what the data behind that external address should be. .. you know what I mean. ;-)

best,
Marcel

Am 19.06.2020 17:47:36 schrieb Marcel Taeumel <[hidden email]>:

Hi Nicolas.

Maybe one more thought on

ExternalData<handle, type>

There is no need to have

ExternalData<1312301580, int>

because it can just be 1312301580.

On the other hand:

ExternalData<#[ 12 34 56 78 ], int>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is already in Squeak's object memory.

Finally:

ExternalData<@00F3A3EE, int*>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is in external memory.

If you want to make an array out of it, make use of the "size" field in ExternalData.

If you want to make a pointer array out of it, change the type to int**

ExternalData<@00F3A3EE, int**>

So, what does this mean then:

ExternalData<#[ 12 34 56 78 ], int*>

Well, it tells you that there is a pointer to an int stored in a byte array. This comes very handy if you think about type aliases to pointer types such as "typedef int* INT_P". Because then it would be

ExternalData<#[ 12 34 56 78 ], INT_P>

To where this pointer points to? Probably garbage. I just made it up for this example. :-D

Best,
Marcel

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:

> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.

We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,
Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...
The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>


Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :
 






Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...
The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).
This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,
Marcel



Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :


Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel


Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:



Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)














Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

marcel.taeumel
 
Hi Nicolas,

Well, my mental model is this one:

I see. I agree. :-)

IMO, it's more useful to avoid the pointer indirection in type, because we
> cannot handle double pointer currently...

Please don't close that door. It's still open. I would love to follow, for example, int** in ExternalData via consecutive sends to #at:. ^__^

Best,
Marcel

Am 19.06.2020 18:52:02 schrieb Nicolas Cellier <[hidden email]>:

Well, my mental model is this one:

ExternalData needs to refer to an area of memory, so as to be able to act
as a pointer (single element or array does not really matter).
If this memory area lies in external heap, then we need an ExternalAddress
as handle.
If this memory area lies in object memory, it must be a ByteArray.
We cannot handle pointers to object memory, because object memory is
relocatable...
Well, unless we pin the object - that why we pass the ByteArray itself, not
its address..

This is independent of how we interpret the contents.
IMO, it's more useful to avoid the pointer indirection in type, because we
cannot handle double pointer currently...
Of course, the only pointers that the memory area could contain are
pointers to external heap (or eventually pinned objects).

I'd like to have a 3rd possibility: pointing to a sub-region of a ByteArray
(ByteArray + offset).
That may be very useful for emulating.
An example in Smallapack: pass a pointer to the imaginary part of a complex
array.
Since real and imaginary parts are interleaved, just offsetting the pointer
and using a stride of 2 elements does the trick when one wants to access
real or imaginary part
(most lapack/blas routines are equipped to handle simple strides).
This is currently possible with ExternalAddress, but NOT when matrix data
lies in ObjectMemory.
Another example would be if you wanted to implement an interpreter for a
low level language... (emulate a processor, whatever).



Le ven. 19 juin 2020 à 18:04, Marcel Taeumel a
écrit :

>
> So, what about this:
>
> ExternalData<@00f3a3ee, int="">
>
> I think it could mean that you want to interpret the very pointer address
> itself as an integer. That is, you do not want to follow to where it points
> to.
>
> Then what about this:
>
> ExternalData<@00f3a3ee, double="">
>
> Ha! An error, I suppose. Unless ... given 8-byte pointers, you can try to
> read a double out of it. :-D Just kidding. It should be an error.
>
> .
> .
> .
>
> > > (On a side note: I think that ExternalData should always have a
> pointer type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
> > Let me disagree here...
>
> The examples I just wrote down in the previous messages let me realize why
> you did disagree. :-D
>
> I hope this helps you with improving argument coercing and return value
> packaging in the FFI plugin. ^__^
>
> Best,
> Marcel
>
> Am 19.06.2020 17:53:21 schrieb Marcel Taeumel :
> > then you would use int* and not int**
>
> My bad: " then you would use int** and not int* " Sorry for the noise
> -.-"
>
> Am 19.06.2020 17:52:21 schrieb Marcel Taeumel :
> > If you want to make a pointer array out of it, change the type to int**
>
> Correction: If you happen to now, that this external address points to an
> array of pointers to int, then you would use int* and not int**. You cannot
> just "decide" what the data behind that external address should be. .. you
> know what I mean. ;-)
>
> best,
> Marcel
>
> Am 19.06.2020 17:47:36 schrieb Marcel Taeumel :
> Hi Nicolas.
>
> Maybe one more thought on
>
> ExternalData
>
> There is no need to have
>
> ExternalData<1312301580, int="">
>
> because it can just be 1312301580.
>
> On the other hand:
>
> ExternalData<#[ 12="" 34="" 56="" 78="" ],="" int="">
>
> Tells you that (1) this data wants to be interpreted as signed int and (2)
> this data is already in Squeak's object memory.
>
> Finally:
>
> ExternalData<@00f3a3ee, int*="">
>
> Tells you that (1) this data wants to be interpreted as signed int and (2)
> this data is in external memory.
>
> If you want to make an array out of it, make use of the "size" field in
> ExternalData.
>
> If you want to make a pointer array out of it, change the type to int**
>
> ExternalData<@00f3a3ee, int**="">
>
> So, what does this mean then:
>
> ExternalData<#[ 12="" 34="" 56="" 78="" ],="" int*="">
>
> Well, it tells you that there is a pointer to an int stored in a byte
> array. This comes very handy if you think about type aliases to pointer
> types such as "typedef int* INT_P". Because then it would be
>
> ExternalData<#[ 12="" 34="" 56="" 78="" ],="" int_p="">
>
> To where this pointer points to? Probably garbage. I just made it up for
> this example. :-D
>
> Best,
> Marcel
>
> Am 19.06.2020 17:15:23 schrieb Marcel Taeumel :
> > The type should better describe the contents of the handle (be it an
> > ExternalAddress or a zone of object memory (ByteArray)).
> > Typically, if you have an external (global) variable of type foo (extern
> > foo my_var;) then the natural type is foo.
> > Having foo*, would mean that we handle an array of foo*, or a pointer to
> > foo* (foo**).
> > This way, we make no difference between type surrogates
> > (ExternalStruct/ExternalTypeAlias) and other atomic cases.
>
> We have the "size" field in ExternalData encode whether "int*" points to a
> single or multiple things. No need to treat a global variable sitting in
> external memory any different here. ... IMO :-)
>
> I know, that this is my personal mental model for Squeak FFI to treat
> pointer types as things in external memory and non-pointer types as things
> in Squeak's object memory. The latter includes ByteArray and Integer and
> Float and also IntegerArray etc. Just not ExternalAddress.
>
> This basically comes from the fact that, until now, the FFI plugin gave me
> a ByteArray in return when I told it to be "Foo", not "Foo*". So,
> non-pointer type means ByteArray. Pointer-type means ExternalAddress.
>
> Type aliases are not really different here. Each alias as a corresponding
> struct type, which has again a non-pointer variant and a pointer variant.
>
> Type aliases to pointer types are just special because they store the
> pointer address in a byte array. The corresponding non-pointer type will do
> fine for such aliases. Their pointer type is not relevant in general, I
> suppose.
>
> > Having foo*, would mean that we handle an array of foo*, or a pointer to foo*
> (foo**).
>
> Well, I have a simple idea on how to represent foo** (and other
> dimensions) without the FFI plugin needing to know. ... unless you are
> concerned about type safety for that, too. Then we would really need to
> encode that in the compiledSpec.
>
> If not, ExternalData with foo** type will just answer ExternalData with
> foo* when asked via "at: 1" for example. :-)
>
> > or an Alien as we may want to support that too.
>
> I don't think that's necessary. Alien has its own call-out mechanism
> through the IA32ABI plugin and also a different layout/usage of ByteArray.
>
> Best,
> Marcel
>
> Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <>
> [hidden email]>:
> Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
> écrit :
>
> >
> > > Err, I move this thread to Opensmalltalk VM dev, because it is not
> > Squeak specific!
> >
> > Absolutely. :-)
> >
> > (On a side note: I think that ExternalData should always have a pointer
> > type in its "type" field unless "handle" is a ByteArray instead of
> > ExternalAddress)
> >
> > Let me disagree here...
> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.
>
> (On another side note: I also think that "handle" in ExternalData should
> > never ever be an atomic Smalltalk object (Integer or Float) but always
> > either ByteArray or ExternalAddress. ... which makes it different from
> what
> > "handle" can be in ExternalStructs that represent type aliases. :-)
> >
> > Yes!
> It could be an ExternalAddress, or a zone of object memory (ByteArray) or
> an Alien as we may want to support that too.
> I'd be inclined to support doubleByte, word, and doubleWord arrays if the
> size of atomic type matches
> (Typically passing a DoubleByteArray, WordArray, a Float32Array, a
> Float64Array to a short */int */float*/double*).
>
> Best,
> > Marcel
> >
> > Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> > [hidden email]>:
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak
> > specific!
> >
> >
> >
> > Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> > [hidden email]> a écrit :
> >
> > > Hi Marcel,
> > > thanks for your vote. Anyone else?
> > >
> > > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > > écrit :
> > >
> > >> Hi Nicolas, hi all! :-)
> > >>
> > >> > Type alias are subclasses of ExternalStructure.
> > >>
> > >> And with it, they get their own instance of ExternalType, which is
> > >> managed in the classVar StructTypes. Let's call them struct type.
> > >>
> > >> > Now we have another mean by adding a class side method in
> > ExternalType.
> > >>
> > >> That kind of aliasing is for compile-time type referencing only --
> such
> > >> as in FFI calls (i.e. pragmas) or struct-field definitions
> > >> (i.e. #fields). (And soon if I have the first version of
> > >> FFI-Callback to show you :)
> > >>
> > >> Because those aliases become fully transparent after compiling the
> > method
> > >> for the FFI call into an instance of ExternalLibraryFunction and
> after
> > >> compiling the field specs of external structures into compiledSpec
> for
> > >> struct types.
> > >>
> > >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> > 'long',
> > >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> > 'int32_t'
> > >> also becomes a 'long'. So, it just translates C vocabulary into
> Squeak
> > FFI
> > >> vocabulary. Nothing more. Nothing less.
> > >>
> > >> Having that, please do not use class-side methods in ExternalType to
> > >> alias a (complex) struct type. Use them only for renaming atomic
> types.
> > >> Please. We may want to add checks to enforce that.
> > >>
> > >> I suppose that this discussion focuses on type aliases that are
> > >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> > and
> > >> thus get their own struct type. So, let's ignore this other mechanism
> > for
> > >> now.
> > >>
> > >> > This has one advantage: type safety. You cannot pass a Foo to a
> > >> function expecting a Bar, even if they both alias int...
> > >>
> > >> Yes! So, we are now talking about type aliases to atomic types. The
> > >> struct type you get when aliasing a 'char' as Foo, for example, looks
> > like
> > >> this:
> > >>
> > >> (...forgive my "creative" representation of WordArray here...)
> > >>
> > >> compiledSpec: 0A 04 00 01
> > >> referentClass: Foo
> > >>
> > >> How does the atomic type for 'char' look like?
> > >>
> > >> compiledSpec: 0A 04 00 01
> > >> referentClass: nil
> > >>
> > >> Consequently, argument coercing will fail if you pass a 'char' when a
> > >> 'Foo' is expected. Because the referentClass does not match --- even
> if
> > the
> > >> compiledSpec does match.
> > >>
> > >> > But this has one major drawback: with current FFI, you cannot pass
> a
> > >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias
> > for
> > >> an integer type...
> > >>
> > >> That's correct. If the origin of such an argument is from within the
> > >> image, you have to wrap it. But when returned from another call ...
> > well
> > >> ... that wrapping should have happened in the plugin ... But read on!
> > :-)
> > >>
> > >> > FFI plugin is clever enough to recognize an atomic type alias and
> > >> simply return a Smalltalk Integer...
> > >> > So it's not Bar all the way down, I have to manually wrap Bar...
> > >> > This is not consistent.
> > >>
> > >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> > >> because it does so for argument coercing.
> > >>
> > >> Now I understand why the code generation of struct-field accessors
> was
> > so
> > >> apparently broken for me. I changed that so that having an alias type
> > as
> > >> part of a larger struct will automatically wrap that for you into the
> > alias
> > >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> > 'long'
> > >> ... Ha ha. ;-) #ffiLongVsInt.
> > >>
> > >> It should be "Bar all the way down". Definitely.
> > >>
> > >> > If we return an int, we must accept an int, otherwise, we cannot
> > chain
> > >> FFI calls...
> > >>
> > >> Well, not for type aliases. I would like keep the path of type safety
> > >> here, you mentioned above.
> > >>
> > >> Just fix the FFI plugin to respect referentClass when packaging the
> > >> return value.
> > >>
> > >> > I could use the lightweight alias instead, [...]
> > >>
> > >> Please don't. See above. Those "lightweight alias" are for renaming
> > >> atomic types only. Let's keep it simple and try to address you
> concern
> > here
> > >> with struct types and ExternalTypeAlias. :-)
> > >>
> > >> > there was another usage where it was convenient to use
> > >> ExternalTypeAlias: enum.
> > >> > Indeed, I simply defined all enum symbols as class side method.
> > >>
> > >> I like that! It reads like a next step for ExternalPool. Once you
> have
> > >> extracted the constant values from header files, and once you have
> > those
> > >> values for your platform in classVars in your external pool, use that
> > pool
> > >> to define enums through ExternalStructure.
> > >>
> > >> Like this:
> > >>
> > >> ExternalTypeAlias subclass: #MyEnumBar
> > >> instanceVariableNames: ''
> > >> classVariableNames: ''
> > >> poolDictionaries: 'HDF5Pool'
> > >> category: 'HDF5'
> > >>
> > >> MyEnumBar class >> #poolFields
> > >> "
> > >> self defineFields.
> > >> "
> > >> ^ #(
> > >> (beg HDF5_BEG)
> > >> (mid HDF5_MID)
> > >> (end HDF5_END)
> > >> )
> > >>
> > >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> > >> external-pool definitions. See class comment in ExternalPool to get
> > started.
> > >>
> > >> Note that #poolFields would translate to class-side methods as you
> > >> suggested.
> > >>
> > >> > If I use a simpler alias, where are those ids going to?
> > >>
> > >> Please don't. See above. :-)
> > >>
> > >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> > >> inputs/outputs behave consistently.
> > >>
> > >> Absolutely!
> > >>
> > >> > - should a function expecting an ExternalTypeAlias of atomic type
> > >> accept an immediate value of compatible type?
> > >>
> > >> Considering type safety, I think not.
> > >>
> > >> Well ... since this is a convenience issue and thus not about saving
> > >> run-time cost ... What about adding a callback into the image to
> react
> > on
> > >> FFIErrorCoercionFailed? Maybe the selector for such a callback could
> be
> > >> stored in ExternalLibraryFunction since this is accessible to the
> > plugin
> > >> anyway?
> > >>
> > >>
> > >>
> > >> ... would it be possible? Like that #doesNotUnderstand: callback?
> > >>
> > >> I would love to do ad-hoc packaging of arguments. (Also thinking
> about
> > >> FFI-Callback. But ignore me here :)
> > >>
> > >> > - should a function returning an ExternalTypeAlias of atomic type
> > >> instantiate the Alias?
> > >>
> > >> Yes, please! See above.
> > >>
> > >> If you want to trade type safety in for a performance gain, just use
> a
> > >> "lightweight alias" as explained above. And package that return value
> > >> manually.
> > >>
> > >> Best,
> > >> Marcel
> > >>
> > >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> > >> [hidden email]>:
> > >> Hi all,
> > >> following the question of Marcel about usage of ExternalTypeAlias:
> > >>
> > >> Type alias are subclasses of ExternalStructure.
> > >> I used them in HDF5 because there was no other means to define an
> alias
> > >> at the time.
> > >> Now we have another mean by adding a class side method in
> ExternalType
> > (I
> > >> would have rather imagined the usage of some annotation to get a
> > >> declarative style)
> > >>
> > >> This has one advantage: type safety. You cannot pass a Foo to a
> > function
> > >> expecting a Bar, even if they both alias int...
> > >>
> > >> But this has one major drawback: with current FFI, you cannot pass a
> > >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias
> > for
> > >> an integer type...
> > >>
> > >> Thus, you have to wrap every Bar argument into a Bar instance...
> > >> Gasp, this is going too far...
> > >>
> > >> What about functions returning such alias?
> > >> Function returning struct by value allocate a struct, but it's not
> the
> > >> case here, FFI plugin is clever enough to recognize an atomic type
> > alias
> > >> and simply return a Smalltalk Integer...
> > >> So it's not Bar all the way down, I have to manually wrap Bar...
> > >> This is not consistent.
> > >> If we return an int, we must accept an int, otherwise, we cannot
> chain
> > >> FFI calls...
> > >>
> > >> I could use the lightweight alias instead, but there was another
> usage
> > >> where it was convenient to use ExternalTypeAlias: enum.
> > >> Indeed, I simply defined all enum symbols as class side method. For
> > >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> > Bar
> > >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> > >> This way, I normally can pass a Bar mid to an external function.
> > >> (currently, we must defne mid ^Bar new value: 1; yourself)
> > >>
> > >> If I use a simpler alias, where are those ids going to?
> > >>
> > >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> > >> inputs/outputs behave consistently.
> > >> The choice is this one:
> > >> - should a function expecting an ExternalTypeAlias of atomic type
> > accept
> > >> an immediate value of compatible type? (the permissive
> implementation)
> > >> - should a function returning an ExternalTypeAlias of atomic type
> > >> instantiate the Alias? (the typesafe implementation, with higher
> > runtime
> > >> cost due to pressure on GC)
> > >>
> > >>
> > >>
> > >>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak
> > specific!
> >
> >
> >
> > Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> > [hidden email]> a écrit :
> >
> >> Hi Marcel,
> >> thanks for your vote. Anyone else?
> >>
> >> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> >> écrit :
> >>
> >>>
> >>> Hi Nicolas, hi all! :-)
> >>>
> >>> > Type alias are subclasses of ExternalStructure.
> >>>
> >>> And with it, they get their own instance of ExternalType, which is
> >>> managed in the classVar StructTypes. Let's call them struct type.
> >>>
> >>> > Now we have another mean by adding a class side method in
> ExternalType.
> >>>
> >>> That kind of aliasing is for compile-time type referencing only --
> such
> >>> as in FFI calls (i.e. pragmas) or struct-field definitions
> >>> (i.e. #fields). (And soon if I have the first version of
> >>> FFI-Callback to show you :)
> >>>
> >>> Because those aliases become fully transparent after compiling the
> >>> method for the FFI call into an instance of ExternalLibraryFunction
> and
> >>> after compiling the field specs of external structures into
> compiledSpec
> >>> for struct types.
> >>>
> >>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> >>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> >>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary
> into
> >>> Squeak FFI vocabulary. Nothing more. Nothing less.
> >>>
> >>> Having that, please do not use class-side methods in ExternalType to
> >>> alias a (complex) struct type. Use them only for renaming atomic
> types.
> >>> Please. We may want to add checks to enforce that.
> >>>
> >>> I suppose that this discussion focuses on type aliases that are
> >>> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >>> thus get their own struct type. So, let's ignore this other mechanism
> for
> >>> now.
> >>>
> >>> > This has one advantage: type safety. You cannot pass a Foo to a
> >>> function expecting a Bar, even if they both alias int...
> >>>
> >>> Yes! So, we are now talking about type aliases to atomic types. The
> >>> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >>> this:
> >>>
> >>> (...forgive my "creative" representation of WordArray here...)
> >>>
> >>> compiledSpec: 0A 04 00 01
> >>> referentClass: Foo
> >>>
> >>> How does the atomic type for 'char' look like?
> >>>
> >>> compiledSpec: 0A 04 00 01
> >>> referentClass: nil
> >>>
> >>> Consequently, argument coercing will fail if you pass a 'char' when a
> >>> 'Foo' is expected. Because the referentClass does not match --- even
> if the
> >>> compiledSpec does match.
> >>>
> >>> > But this has one major drawback: with current FFI, you cannot pass a
> >>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias for
> >>> an integer type...
> >>>
> >>> That's correct. If the origin of such an argument is from within the
> >>> image, you have to wrap it. But when returned from another call ...
> well
> >>> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>>
> >>> > FFI plugin is clever enough to recognize an atomic type alias and
> >>> simply return a Smalltalk Integer...
> >>> > So it's not Bar all the way down, I have to manually wrap Bar...
> >>> > This is not consistent.
> >>>
> >>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >>> because it does so for argument coercing.
> >>>
> >>> Now I understand why the code generation of struct-field accessors was
> >>> so apparently broken for me. I changed that so that having an alias
> type as
> >>> part of a larger struct will automatically wrap that for you into the
> alias
> >>> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >>> ... Ha ha. ;-) #ffiLongVsInt.
> >>>
> >>> It should be "Bar all the way down". Definitely.
> >>>
> >>> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >>> FFI calls...
> >>>
> >>> Well, not for type aliases. I would like keep the path of type safety
> >>> here, you mentioned above.
> >>>
> >>> Just fix the FFI plugin to respect referentClass when packaging the
> >>> return value.
> >>>
> >>> > I could use the lightweight alias instead, [...]
> >>>
> >>> Please don't. See above. Those "lightweight alias" are for renaming
> >>> atomic types only. Let's keep it simple and try to address you concern
> here
> >>> with struct types and ExternalTypeAlias. :-)
> >>>
> >>> > there was another usage where it was convenient to use
> >>> ExternalTypeAlias: enum.
> >>> > Indeed, I simply defined all enum symbols as class side method.
> >>>
> >>> I like that! It reads like a next step for ExternalPool. Once you have
> >>> extracted the constant values from header files, and once you have
> those
> >>> values for your platform in classVars in your external pool, use that
> pool
> >>> to define enums through ExternalStructure.
> >>>
> >>> Like this:
> >>>
> >>> ExternalTypeAlias subclass: #MyEnumBar
> >>> instanceVariableNames: ''
> >>> classVariableNames: ''
> >>> poolDictionaries: 'HDF5Pool'
> >>> category: 'HDF5'
> >>>
> >>> MyEnumBar class >> #poolFields
> >>> "
> >>> self defineFields.
> >>> "
> >>> ^ #(
> >>> (beg HDF5_BEG)
> >>> (mid HDF5_MID)
> >>> (end HDF5_END)
> >>> )
> >>>
> >>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >>> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>>
> >>> Note that #poolFields would translate to class-side methods as you
> >>> suggested.
> >>>
> >>> > If I use a simpler alias, where are those ids going to?
> >>>
> >>> Please don't. See above. :-)
> >>>
> >>> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >>> inputs/outputs behave consistently.
> >>>
> >>> Absolutely!
> >>>
> >>> > - should a function expecting an ExternalTypeAlias of atomic type
> >>> accept an immediate value of compatible type?
> >>>
> >>> Considering type safety, I think not.
> >>>
> >>> Well ... since this is a convenience issue and thus not about saving
> >>> run-time cost ... What about adding a callback into the image to react
> on
> >>> FFIErrorCoercionFailed? Maybe the selector for such a callback could
> be
> >>> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >>> anyway?
> >>>
> >>>
> >>>
> >>> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>>
> >>> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >>> FFI-Callback. But ignore me here :)
> >>>
> >>> > - should a function returning an ExternalTypeAlias of atomic type
> >>> instantiate the Alias?
> >>>
> >>> Yes, please! See above.
> >>>
> >>> If you want to trade type safety in for a performance gain, just use a
> >>> "lightweight alias" as explained above. And package that return value
> >>> manually.
> >>>
> >>> Best,
> >>> Marcel
> >>>
> >>>
> >>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >>> [hidden email]>:
> >>>
> >>> Hi all,
> >>> following the question of Marcel about usage of ExternalTypeAlias:
> >>>
> >>> Type alias are subclasses of ExternalStructure.
> >>> I used them in HDF5 because there was no other means to define an
> alias
> >>> at the time.
> >>> Now we have another mean by adding a class side method in ExternalType
> >>> (I would have rather imagined the usage of some annotation to get a
> >>> declarative style)
> >>>
> >>> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >>> expecting a Bar, even if they both alias int...
> >>>
> >>> But this has one major drawback: with current FFI, you cannot pass a
> >>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias for
> >>> an integer type...
> >>>
> >>> Thus, you have to wrap every Bar argument into a Bar instance...
> >>> Gasp, this is going too far...
> >>>
> >>> What about functions returning such alias?
> >>> Function returning struct by value allocate a struct, but it's not the
> >>> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >>> and simply return a Smalltalk Integer...
> >>> So it's not Bar all the way down, I have to manually wrap Bar...
> >>> This is not consistent.
> >>> If we return an int, we must accept an int, otherwise, we cannot chain
> >>> FFI calls...
> >>>
> >>> I could use the lightweight alias instead, but there was another usage
> >>> where it was convenient to use ExternalTypeAlias: enum.
> >>> Indeed, I simply defined all enum symbols as class side method. For
> >>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >>> This way, I normally can pass a Bar mid to an external function.
> >>> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>>
> >>> If I use a simpler alias, where are those ids going to?
> >>>
> >>> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >>> inputs/outputs behave consistently.
> >>> The choice is this one:
> >>> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >>> an immediate value of compatible type? (the permissive implementation)
> >>> - should a function returning an ExternalTypeAlias of atomic type
> >>> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >>> cost due to pressure on GC)
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >
>
>
> Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
> écrit :
>
>>
>>
>>
>>
>>
>>
>>
>> > Err, I move this thread to Opensmalltalk VM dev, because it is not
>> Squeak specific!
>>
>> Absolutely. :-)
>>
>> (On a side note: I think that ExternalData should always have a pointer
>> type in its "type" field unless "handle" is a ByteArray instead of
>> ExternalAddress)
>>
>> Let me disagree here...
> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.
>
> (On another side note: I also think that "handle" in ExternalData should
>> never ever be an atomic Smalltalk object (Integer or Float) but always
>> either ByteArray or ExternalAddress. ... which makes it different from what
>> "handle" can be in ExternalStructs that represent type aliases. :-)
>>
>> Yes!
> It could be an ExternalAddress, or a zone of object memory (ByteArray) or
> an Alien as we may want to support that too.
> I'd be inclined to support doubleByte, word, and doubleWord arrays if the
> size of atomic type matches
> (Typically passing a DoubleByteArray, WordArray, a Float32Array, a
> Float64Array to a short */int */float*/double*).
>
> Best,
>> Marcel
>>
>>
>>
>> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
>> [hidden email]>:
>> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
>>
>> specific!
>>
>>
>>
>>
>>
>>
>>
>> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
>> [hidden email]> a écrit :
>>
>>
>>
>> > Hi Marcel,
>>
>> > thanks for your vote. Anyone else?
>>
>> >
>>
>> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>>
>> > écrit :
>>
>> >
>>
>> >> Hi Nicolas, hi all! :-)
>>
>> >>
>>
>> >> > Type alias are subclasses of ExternalStructure.
>>
>> >>
>>
>> >> And with it, they get their own instance of ExternalType, which is
>>
>> >> managed in the classVar StructTypes. Let's call them struct type.
>>
>> >>
>>
>> >> > Now we have another mean by adding a class side method in
>> ExternalType.
>>
>> >>
>>
>> >> That kind of aliasing is for compile-time type referencing only --
>> such
>>
>> >> as in FFI calls (i.e. pragmas) or struct-field definitions
>>
>> >> (i.e. #fields). (And soon if I have the first version of
>>
>> >> FFI-Callback to show you :)
>>
>> >>
>>
>> >> Because those aliases become fully transparent after compiling the
>> method
>>
>> >> for the FFI call into an instance of ExternalLibraryFunction and after
>>
>> >> compiling the field specs of external structures into compiledSpec for
>>
>> >> struct types.
>>
>> >>
>>
>> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>> 'long',
>>
>> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>> 'int32_t'
>>
>> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
>> FFI
>>
>> >> vocabulary. Nothing more. Nothing less.
>>
>> >>
>>
>> >> Having that, please do not use class-side methods in ExternalType to
>>
>> >> alias a (complex) struct type. Use them only for renaming atomic
>> types.
>>
>> >> Please. We may want to add checks to enforce that.
>>
>> >>
>>
>> >> I suppose that this discussion focuses on type aliases that are
>>
>> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
>> and
>>
>> >> thus get their own struct type. So, let's ignore this other mechanism
>> for
>>
>> >> now.
>>
>> >>
>>
>> >> > This has one advantage: type safety. You cannot pass a Foo to a
>>
>> >> function expecting a Bar, even if they both alias int...
>>
>> >>
>>
>> >> Yes! So, we are now talking about type aliases to atomic types. The
>>
>> >> struct type you get when aliasing a 'char' as Foo, for example, looks
>> like
>>
>> >> this:
>>
>> >>
>>
>> >> (...forgive my "creative" representation of WordArray here...)
>>
>> >>
>>
>> >> compiledSpec: 0A 04 00 01
>>
>> >> referentClass: Foo
>>
>> >>
>>
>> >> How does the atomic type for 'char' look like?
>>
>> >>
>>
>> >> compiledSpec: 0A 04 00 01
>>
>> >> referentClass: nil
>>
>> >>
>>
>> >> Consequently, argument coercing will fail if you pass a 'char' when a
>>
>> >> 'Foo' is expected. Because the referentClass does not match --- even
>> if the
>>
>> >> compiledSpec does match.
>>
>> >>
>>
>> >> > But this has one major drawback: with current FFI, you cannot pass a
>>
>> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
>> alias for
>>
>> >> an integer type...
>>
>> >>
>>
>> >> That's correct. If the origin of such an argument is from within the
>>
>> >> image, you have to wrap it. But when returned from another call ...
>> well
>>
>> >> ... that wrapping should have happened in the plugin ... But read on!
>> :-)
>>
>> >>
>>
>> >> > FFI plugin is clever enough to recognize an atomic type alias and
>>
>> >> simply return a Smalltalk Integer...
>>
>> >> > So it's not Bar all the way down, I have to manually wrap Bar...
>>
>> >> > This is not consistent.
>>
>> >>
>>
>> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>
>> >> because it does so for argument coercing.
>>
>> >>
>>
>> >> Now I understand why the code generation of struct-field accessors was
>> so
>>
>> >> apparently broken for me. I changed that so that having an alias type
>> as
>>
>> >> part of a larger struct will automatically wrap that for you into the
>> alias
>>
>> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
>> 'long'
>>
>> >> ... Ha ha. ;-) #ffiLongVsInt.
>>
>> >>
>>
>> >> It should be "Bar all the way down". Definitely.
>>
>> >>
>>
>> >> > If we return an int, we must accept an int, otherwise, we cannot
>> chain
>>
>> >> FFI calls...
>>
>> >>
>>
>> >> Well, not for type aliases. I would like keep the path of type safety
>>
>> >> here, you mentioned above.
>>
>> >>
>>
>> >> Just fix the FFI plugin to respect referentClass when packaging the
>>
>> >> return value.
>>
>> >>
>>
>> >> > I could use the lightweight alias instead, [...]
>>
>> >>
>>
>> >> Please don't. See above. Those "lightweight alias" are for renaming
>>
>> >> atomic types only. Let's keep it simple and try to address you concern
>> here
>>
>> >> with struct types and ExternalTypeAlias. :-)
>>
>> >>
>>
>> >> > there was another usage where it was convenient to use
>>
>> >> ExternalTypeAlias: enum.
>>
>> >> > Indeed, I simply defined all enum symbols as class side method.
>>
>> >>
>>
>> >> I like that! It reads like a next step for ExternalPool. Once you have
>>
>> >> extracted the constant values from header files, and once you have
>> those
>>
>> >> values for your platform in classVars in your external pool, use that
>> pool
>>
>> >> to define enums through ExternalStructure.
>>
>> >>
>>
>> >> Like this:
>>
>> >>
>>
>> >> ExternalTypeAlias subclass: #MyEnumBar
>>
>> >> instanceVariableNames: ''
>>
>> >> classVariableNames: ''
>>
>> >> poolDictionaries: 'HDF5Pool'
>>
>> >> category: 'HDF5'
>>
>> >>
>>
>> >> MyEnumBar class >> #poolFields
>>
>> >> "
>>
>> >> self defineFields.
>>
>> >> "
>>
>> >> ^ #(
>>
>> >> (beg HDF5_BEG)
>>
>> >> (mid HDF5_MID)
>>
>> >> (end HDF5_END)
>>
>> >> )
>>
>> >>
>>
>> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>
>> >> external-pool definitions. See class comment in ExternalPool to get
>> started.
>>
>> >>
>>
>> >> Note that #poolFields would translate to class-side methods as you
>>
>> >> suggested.
>>
>> >>
>>
>> >> > If I use a simpler alias, where are those ids going to?
>>
>> >>
>>
>> >> Please don't. See above. :-)
>>
>> >>
>>
>> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>
>> >> inputs/outputs behave consistently.
>>
>> >>
>>
>> >> Absolutely!
>>
>> >>
>>
>> >> > - should a function expecting an ExternalTypeAlias of atomic type
>>
>> >> accept an immediate value of compatible type?
>>
>> >>
>>
>> >> Considering type safety, I think not.
>>
>> >>
>>
>> >> Well ... since this is a convenience issue and thus not about saving
>>
>> >> run-time cost ... What about adding a callback into the image to react
>> on
>>
>> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could
>> be
>>
>> >> stored in ExternalLibraryFunction since this is accessible to the
>> plugin
>>
>> >> anyway?
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >> ... would it be possible? Like that #doesNotUnderstand: callback?
>>
>> >>
>>
>> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>
>> >> FFI-Callback. But ignore me here :)
>>
>> >>
>>
>> >> > - should a function returning an ExternalTypeAlias of atomic type
>>
>> >> instantiate the Alias?
>>
>> >>
>>
>> >> Yes, please! See above.
>>
>> >>
>>
>> >> If you want to trade type safety in for a performance gain, just use a
>>
>> >> "lightweight alias" as explained above. And package that return value
>>
>> >> manually.
>>
>> >>
>>
>> >> Best,
>>
>> >> Marcel
>>
>> >>
>>
>> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> >> [hidden email]>:
>>
>> >> Hi all,
>>
>> >> following the question of Marcel about usage of ExternalTypeAlias:
>>
>> >>
>>
>> >> Type alias are subclasses of ExternalStructure.
>>
>> >> I used them in HDF5 because there was no other means to define an
>> alias
>>
>> >> at the time.
>>
>> >> Now we have another mean by adding a class side method in ExternalType
>> (I
>>
>> >> would have rather imagined the usage of some annotation to get a
>>
>> >> declarative style)
>>
>> >>
>>
>> >> This has one advantage: type safety. You cannot pass a Foo to a
>> function
>>
>> >> expecting a Bar, even if they both alias int...
>>
>> >>
>>
>> >> But this has one major drawback: with current FFI, you cannot pass a
>>
>> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
>> alias for
>>
>> >> an integer type...
>>
>> >>
>>
>> >> Thus, you have to wrap every Bar argument into a Bar instance...
>>
>> >> Gasp, this is going too far...
>>
>> >>
>>
>> >> What about functions returning such alias?
>>
>> >> Function returning struct by value allocate a struct, but it's not the
>>
>> >> case here, FFI plugin is clever enough to recognize an atomic type
>> alias
>>
>> >> and simply return a Smalltalk Integer...
>>
>> >> So it's not Bar all the way down, I have to manually wrap Bar...
>>
>> >> This is not consistent.
>>
>> >> If we return an int, we must accept an int, otherwise, we cannot chain
>>
>> >> FFI calls...
>>
>> >>
>>
>> >> I could use the lightweight alias instead, but there was another usage
>>
>> >> where it was convenient to use ExternalTypeAlias: enum.
>>
>> >> Indeed, I simply defined all enum symbols as class side method. For
>>
>> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
>> Bar
>>
>> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>
>> >> This way, I normally can pass a Bar mid to an external function.
>>
>> >> (currently, we must defne mid ^Bar new value: 1; yourself)
>>
>> >>
>>
>> >> If I use a simpler alias, where are those ids going to?
>>
>> >>
>>
>> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>
>> >> inputs/outputs behave consistently.
>>
>> >> The choice is this one:
>>
>> >> - should a function expecting an ExternalTypeAlias of atomic type
>> accept
>>
>> >> an immediate value of compatible type? (the permissive implementation)
>>
>> >> - should a function returning an ExternalTypeAlias of atomic type
>>
>> >> instantiate the Alias? (the typesafe implementation, with higher
>> runtime
>>
>> >> cost due to pressure on GC)
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
>> specific!
>>
>>
>>
>> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
>> [hidden email]> a écrit :
>>
>>> Hi Marcel,
>>> thanks for your vote. Anyone else?
>>>
>>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>>> écrit :
>>>
>>>>
>>>>
>>>> Hi Nicolas, hi all! :-)
>>>>
>>>> > Type alias are subclasses of ExternalStructure.
>>>>
>>>> And with it, they get their own instance of ExternalType, which is
>>>> managed in the classVar StructTypes. Let's call them struct type.
>>>>
>>>> > Now we have another mean by adding a class side method in
>>>> ExternalType.
>>>>
>>>> That kind of aliasing is for compile-time type referencing only -- such
>>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>>> (i.e. #fields). (And soon if I have the first version of
>>>> FFI-Callback to show you :)
>>>>
>>>> Because those aliases become fully transparent after compiling the
>>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>>> after compiling the field specs of external structures into compiledSpec
>>>> for struct types.
>>>>
>>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>>
>>>> Having that, please do not use class-side methods in ExternalType to
>>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>>> Please. We may want to add checks to enforce that.
>>>>
>>>> I suppose that this discussion focuses on type aliases that are
>>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>>> thus get their own struct type. So, let's ignore this other mechanism for
>>>> now.
>>>>
>>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>>> function expecting a Bar, even if they both alias int...
>>>>
>>>> Yes! So, we are now talking about type aliases to atomic types. The
>>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>>> this:
>>>>
>>>> (...forgive my "creative" representation of WordArray here...)
>>>>
>>>> compiledSpec: 0A 04 00 01
>>>> referentClass: Foo
>>>>
>>>> How does the atomic type for 'char' look like?
>>>>
>>>> compiledSpec: 0A 04 00 01
>>>> referentClass: nil
>>>>
>>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>>> compiledSpec does match.
>>>>
>>>> > But this has one major drawback: with current FFI, you cannot pass a
>>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>>> an integer type...
>>>>
>>>> That's correct. If the origin of such an argument is from within the
>>>> image, you have to wrap it. But when returned from another call ... well
>>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>>
>>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>>> simply return a Smalltalk Integer...
>>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>>> > This is not consistent.
>>>>
>>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>>> because it does so for argument coercing.
>>>>
>>>> Now I understand why the code generation of struct-field accessors was
>>>> so apparently broken for me. I changed that so that having an alias type as
>>>> part of a larger struct will automatically wrap that for you into the alias
>>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>>
>>>> It should be "Bar all the way down". Definitely.
>>>>
>>>> > If we return an int, we must accept an int, otherwise, we cannot
>>>> chain FFI calls...
>>>>
>>>> Well, not for type aliases. I would like keep the path of type safety
>>>> here, you mentioned above.
>>>>
>>>> Just fix the FFI plugin to respect referentClass when packaging the
>>>> return value.
>>>>
>>>> > I could use the lightweight alias instead, [...]
>>>>
>>>> Please don't. See above. Those "lightweight alias" are for renaming
>>>> atomic types only. Let's keep it simple and try to address you concern here
>>>> with struct types and ExternalTypeAlias. :-)
>>>>
>>>> > there was another usage where it was convenient to use
>>>> ExternalTypeAlias: enum.
>>>> > Indeed, I simply defined all enum symbols as class side method.
>>>>
>>>> I like that! It reads like a next step for ExternalPool. Once you have
>>>> extracted the constant values from header files, and once you have those
>>>> values for your platform in classVars in your external pool, use that pool
>>>> to define enums through ExternalStructure.
>>>>
>>>> Like this:
>>>>
>>>> ExternalTypeAlias subclass: #MyEnumBar
>>>> instanceVariableNames: ''
>>>> classVariableNames: ''
>>>> poolDictionaries: 'HDF5Pool'
>>>> category: 'HDF5'
>>>>
>>>> MyEnumBar class >> #poolFields
>>>> "
>>>> self defineFields.
>>>> "
>>>> ^ #(
>>>> (beg HDF5_BEG)
>>>> (mid HDF5_MID)
>>>> (end HDF5_END)
>>>> )
>>>>
>>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>>
>>>> Note that #poolFields would translate to class-side methods as you
>>>> suggested.
>>>>
>>>> > If I use a simpler alias, where are those ids going to?
>>>>
>>>> Please don't. See above. :-)
>>>>
>>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>>> inputs/outputs behave consistently.
>>>>
>>>> Absolutely!
>>>>
>>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>>> accept an immediate value of compatible type?
>>>>
>>>> Considering type safety, I think not.
>>>>
>>>> Well ... since this is a convenience issue and thus not about saving
>>>> run-time cost ... What about adding a callback into the image to react on
>>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>>> anyway?
>>>>
>>>>
>>>>
>>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>>
>>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>>> FFI-Callback. But ignore me here :)
>>>>
>>>> > - should a function returning an ExternalTypeAlias of atomic type
>>>> instantiate the Alias?
>>>>
>>>> Yes, please! See above.
>>>>
>>>> If you want to trade type safety in for a performance gain, just use a
>>>> "lightweight alias" as explained above. And package that return value
>>>> manually.
>>>>
>>>> Best,
>>>> Marcel
>>>>
>>>>
>>>>
>>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>>> [hidden email]>:
>>>>
>>>>
>>>> Hi all,
>>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>>
>>>> Type alias are subclasses of ExternalStructure.
>>>> I used them in HDF5 because there was no other means to define an alias
>>>> at the time.
>>>> Now we have another mean by adding a class side method in ExternalType
>>>> (I would have rather imagined the usage of some annotation to get a
>>>> declarative style)
>>>>
>>>> This has one advantage: type safety. You cannot pass a Foo to a
>>>> function expecting a Bar, even if they both alias int...
>>>>
>>>> But this has one major drawback: with current FFI, you cannot pass a
>>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>>> an integer type...
>>>>
>>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>>> Gasp, this is going too far...
>>>>
>>>> What about functions returning such alias?
>>>> Function returning struct by value allocate a struct, but it's not the
>>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>>> and simply return a Smalltalk Integer...
>>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>>> This is not consistent.
>>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>>> FFI calls...
>>>>
>>>> I could use the lightweight alias instead, but there was another usage
>>>> where it was convenient to use ExternalTypeAlias: enum.
>>>> Indeed, I simply defined all enum symbols as class side method. For
>>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>>> This way, I normally can pass a Bar mid to an external function.
>>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>>
>>>> If I use a simpler alias, where are those ids going to?
>>>>
>>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>>> inputs/outputs behave consistently.
>>>> The choice is this one:
>>>> - should a function expecting an ExternalTypeAlias of atomic type
>>>> accept an immediate value of compatible type? (the permissive
>>>> implementation)
>>>> - should a function returning an ExternalTypeAlias of atomic type
>>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>>> cost due to pressure on GC)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
Well, my mental model is this one:

ExternalData needs to refer to an area of memory, so as to be able to act as a pointer (single element or array does not really matter).
If this memory area lies in external heap, then we need an ExternalAddress as handle.
If this memory area lies in object memory, it must be a ByteArray.
We cannot handle pointers to object memory, because object memory is relocatable...
Well, unless we pin the object - that why we pass the ByteArray itself, not its address..

This is independent of how we interpret the contents.
IMO, it's more useful to avoid the pointer indirection in type, because we cannot handle double pointer currently...
Of course, the only pointers that the memory area could contain are pointers to external heap (or eventually pinned objects).

I'd like to have a 3rd possibility: pointing to a sub-region of a ByteArray (ByteArray + offset).
That may be very useful for emulating.
An example in Smallapack: pass a pointer to the imaginary part of a complex array.
Since real and imaginary parts are interleaved, just offsetting the pointer and using a stride of 2 elements does the trick when one wants to access real or imaginary part
(most lapack/blas routines are equipped to handle simple strides).
This is currently possible with ExternalAddress, but NOT when matrix data lies in ObjectMemory.
Another example would be if you wanted to implement an interpreter for a low level language... (emulate a processor, whatever).



Le ven. 19 juin 2020 à 18:04, Marcel Taeumel <[hidden email]> a écrit :
 

So, what about this:

ExternalData<@00F3A3EE, int>

I think it could mean that you want to interpret the very pointer address itself as an integer. That is, you do not want to follow to where it points to.

Then what about this:

ExternalData<@00F3A3EE, double>

Ha! An error, I suppose. Unless ... given 8-byte pointers, you can try to read a double out of it. :-D Just kidding. It should be an error.

.
.
.

> > (On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)
> Let me disagree here...

The examples I just wrote down in the previous messages let me realize why you did disagree. :-D

I hope this helps you with improving argument coercing and return value packaging in the FFI plugin. ^__^

Best,
Marcel

Am 19.06.2020 17:53:21 schrieb Marcel Taeumel <[hidden email]>:


then you would use int* and not int**

My bad: "  then you would use int** and not int*  " Sorry for the noise -.-"

Am 19.06.2020 17:52:21 schrieb Marcel Taeumel <[hidden email]>:


If you want to make a pointer array out of it, change the type to int**

Correction: If you happen to now, that this external address points to an array of pointers to int, then you would use int* and not int**. You cannot just "decide" what the data behind that external address should be. .. you know what I mean. ;-)

best,
Marcel

Am 19.06.2020 17:47:36 schrieb Marcel Taeumel <[hidden email]>:


Hi Nicolas.

Maybe one more thought on

ExternalData<handle, type>

There is no need to have

ExternalData<1312301580, int>

because it can just be 1312301580.

On the other hand:

ExternalData<#[ 12 34 56 78 ], int>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is already in Squeak's object memory.

Finally:

ExternalData<@00F3A3EE, int*>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is in external memory.

If you want to make an array out of it, make use of the "size" field in ExternalData.

If you want to make a pointer array out of it, change the type to int**

ExternalData<@00F3A3EE, int**>

So, what does this mean then:

ExternalData<#[ 12 34 56 78 ], int*>

Well, it tells you that there is a pointer to an int stored in a byte array. This comes very handy if you think about type aliases to pointer types such as "typedef int* INT_P". Because then it would be

ExternalData<#[ 12 34 56 78 ], INT_P>

To where this pointer points to? Probably garbage. I just made it up for this example. :-D

Best,
Marcel

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:







> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.



We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,
Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a

écrit :



>

> > Err, I move this thread to Opensmalltalk VM dev, because it is not

> Squeak specific!

>

> Absolutely. :-)

>

> (On a side note: I think that ExternalData should always have a pointer

> type in its "type" field unless "handle" is a ByteArray instead of

> ExternalAddress)

>

> Let me disagree here...

The type should better describe the contents of the handle (be it an

ExternalAddress or a zone of object memory (ByteArray)).

Typically, if you have an external (global) variable of type foo (extern

foo my_var;) then the natural type is foo.

Having foo*, would mean that we handle an array of foo*, or a pointer to

foo* (foo**).

This way, we make no difference between type surrogates

(ExternalStruct/ExternalTypeAlias) and other atomic cases.



(On another side note: I also think that "handle" in ExternalData should

> never ever be an atomic Smalltalk object (Integer or Float) but always

> either ByteArray or ExternalAddress. ... which makes it different from what

> "handle" can be in ExternalStructs that represent type aliases. :-)

>

> Yes!

It could be an ExternalAddress, or a zone of object memory (ByteArray) or

an Alien as we may want to support that too.

I'd be inclined to support doubleByte, word, and doubleWord arrays if the

size of atomic type matches

(Typically passing a DoubleByteArray, WordArray, a Float32Array, a

Float64Array to a short */int */float*/double*).



Best,

> Marcel

>

> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:

> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

> specific!

>

>

>

> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>

> [hidden email]> a écrit :

>

> > Hi Marcel,

> > thanks for your vote. Anyone else?

> >

> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> > écrit :

> >

> >> Hi Nicolas, hi all! :-)

> >>

> >> > Type alias are subclasses of ExternalStructure.

> >>

> >> And with it, they get their own instance of ExternalType, which is

> >> managed in the classVar StructTypes. Let's call them struct type.

> >>

> >> > Now we have another mean by adding a class side method in

> ExternalType.

> >>

> >> That kind of aliasing is for compile-time type referencing only -- such

> >> as in FFI calls (i.e. pragmas) or struct-field definitions

> >> (i.e. #fields). (And soon if I have the first version of

> >> FFI-Callback to show you :)

> >>

> >> Because those aliases become fully transparent after compiling the

> method

> >> for the FFI call into an instance of ExternalLibraryFunction and after

> >> compiling the field specs of external structures into compiledSpec for

> >> struct types.

> >>

> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a

> 'long',

> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And

> 'int32_t'

> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak

> FFI

> >> vocabulary. Nothing more. Nothing less.

> >>

> >> Having that, please do not use class-side methods in ExternalType to

> >> alias a (complex) struct type. Use them only for renaming atomic types.

> >> Please. We may want to add checks to enforce that.

> >>

> >> I suppose that this discussion focuses on type aliases that are

> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)

> and

> >> thus get their own struct type. So, let's ignore this other mechanism

> for

> >> now.

> >>

> >> > This has one advantage: type safety. You cannot pass a Foo to a

> >> function expecting a Bar, even if they both alias int...

> >>

> >> Yes! So, we are now talking about type aliases to atomic types. The

> >> struct type you get when aliasing a 'char' as Foo, for example, looks

> like

> >> this:

> >>

> >> (...forgive my "creative" representation of WordArray here...)

> >>

> >> compiledSpec: 0A 04 00 01

> >> referentClass: Foo

> >>

> >> How does the atomic type for 'char' look like?

> >>

> >> compiledSpec: 0A 04 00 01

> >> referentClass: nil

> >>

> >> Consequently, argument coercing will fail if you pass a 'char' when a

> >> 'Foo' is expected. Because the referentClass does not match --- even if

> the

> >> compiledSpec does match.

> >>

> >> > But this has one major drawback: with current FFI, you cannot pass a

> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias

> for

> >> an integer type...

> >>

> >> That's correct. If the origin of such an argument is from within the

> >> image, you have to wrap it. But when returned from another call ...

> well

> >> ... that wrapping should have happened in the plugin ... But read on!

> :-)

> >>

> >> > FFI plugin is clever enough to recognize an atomic type alias and

> >> simply return a Smalltalk Integer...

> >> > So it's not Bar all the way down, I have to manually wrap Bar...

> >> > This is not consistent.

> >>

> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

> >> because it does so for argument coercing.

> >>

> >> Now I understand why the code generation of struct-field accessors was

> so

> >> apparently broken for me. I changed that so that having an alias type

> as

> >> part of a larger struct will automatically wrap that for you into the

> alias

> >> structure. :-) Even if you are just aliasing a plain 'int' ... or

> 'long'

> >> ... Ha ha. ;-) #ffiLongVsInt.

> >>

> >> It should be "Bar all the way down". Definitely.

> >>

> >> > If we return an int, we must accept an int, otherwise, we cannot

> chain

> >> FFI calls...

> >>

> >> Well, not for type aliases. I would like keep the path of type safety

> >> here, you mentioned above.

> >>

> >> Just fix the FFI plugin to respect referentClass when packaging the

> >> return value.

> >>

> >> > I could use the lightweight alias instead, [...]

> >>

> >> Please don't. See above. Those "lightweight alias" are for renaming

> >> atomic types only. Let's keep it simple and try to address you concern

> here

> >> with struct types and ExternalTypeAlias. :-)

> >>

> >> > there was another usage where it was convenient to use

> >> ExternalTypeAlias: enum.

> >> > Indeed, I simply defined all enum symbols as class side method.

> >>

> >> I like that! It reads like a next step for ExternalPool. Once you have

> >> extracted the constant values from header files, and once you have

> those

> >> values for your platform in classVars in your external pool, use that

> pool

> >> to define enums through ExternalStructure.

> >>

> >> Like this:

> >>

> >> ExternalTypeAlias subclass: #MyEnumBar

> >> instanceVariableNames: ''

> >> classVariableNames: ''

> >> poolDictionaries: 'HDF5Pool'

> >> category: 'HDF5'

> >>

> >> MyEnumBar class >> #poolFields

> >> "

> >> self defineFields.

> >> "

> >> ^ #(

> >> (beg HDF5_BEG)

> >> (mid HDF5_MID)

> >> (end HDF5_END)

> >> )

> >>

> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

> >> external-pool definitions. See class comment in ExternalPool to get

> started.

> >>

> >> Note that #poolFields would translate to class-side methods as you

> >> suggested.

> >>

> >> > If I use a simpler alias, where are those ids going to?

> >>

> >> Please don't. See above. :-)

> >>

> >> > IMO we cannot keep status quo in the plugin, we gotta to make the

> >> inputs/outputs behave consistently.

> >>

> >> Absolutely!

> >>

> >> > - should a function expecting an ExternalTypeAlias of atomic type

> >> accept an immediate value of compatible type?

> >>

> >> Considering type safety, I think not.

> >>

> >> Well ... since this is a convenience issue and thus not about saving

> >> run-time cost ... What about adding a callback into the image to react

> on

> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

> >> stored in ExternalLibraryFunction since this is accessible to the

> plugin

> >> anyway?

> >>

> >>

> >>

> >> ... would it be possible? Like that #doesNotUnderstand: callback?

> >>

> >> I would love to do ad-hoc packaging of arguments. (Also thinking about

> >> FFI-Callback. But ignore me here :)

> >>

> >> > - should a function returning an ExternalTypeAlias of atomic type

> >> instantiate the Alias?

> >>

> >> Yes, please! See above.

> >>

> >> If you want to trade type safety in for a performance gain, just use a

> >> "lightweight alias" as explained above. And package that return value

> >> manually.

> >>

> >> Best,

> >> Marcel

> >>

> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>

> >> [hidden email]>:

> >> Hi all,

> >> following the question of Marcel about usage of ExternalTypeAlias:

> >>

> >> Type alias are subclasses of ExternalStructure.

> >> I used them in HDF5 because there was no other means to define an alias

> >> at the time.

> >> Now we have another mean by adding a class side method in ExternalType

> (I

> >> would have rather imagined the usage of some annotation to get a

> >> declarative style)

> >>

> >> This has one advantage: type safety. You cannot pass a Foo to a

> function

> >> expecting a Bar, even if they both alias int...

> >>

> >> But this has one major drawback: with current FFI, you cannot pass a

> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias

> for

> >> an integer type...

> >>

> >> Thus, you have to wrap every Bar argument into a Bar instance...

> >> Gasp, this is going too far...

> >>

> >> What about functions returning such alias?

> >> Function returning struct by value allocate a struct, but it's not the

> >> case here, FFI plugin is clever enough to recognize an atomic type

> alias

> >> and simply return a Smalltalk Integer...

> >> So it's not Bar all the way down, I have to manually wrap Bar...

> >> This is not consistent.

> >> If we return an int, we must accept an int, otherwise, we cannot chain

> >> FFI calls...

> >>

> >> I could use the lightweight alias instead, but there was another usage

> >> where it was convenient to use ExternalTypeAlias: enum.

> >> Indeed, I simply defined all enum symbols as class side method. For

> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type

> Bar

> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

> >> This way, I normally can pass a Bar mid to an external function.

> >> (currently, we must defne mid ^Bar new value: 1; yourself)

> >>

> >> If I use a simpler alias, where are those ids going to?

> >>

> >> IMO we cannot keep statu quo in the plugin, we gotta to make the

> >> inputs/outputs behave consistently.

> >> The choice is this one:

> >> - should a function expecting an ExternalTypeAlias of atomic type

> accept

> >> an immediate value of compatible type? (the permissive implementation)

> >> - should a function returning an ExternalTypeAlias of atomic type

> >> instantiate the Alias? (the typesafe implementation, with higher

> runtime

> >> cost due to pressure on GC)

> >>

> >>

> >>

> >>

> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

> specific!

>

>

>

> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :

>

>> Hi Marcel,

>> thanks for your vote. Anyone else?

>>

>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

>> écrit :

>>

>>>

>>> Hi Nicolas, hi all! :-)

>>>

>>> > Type alias are subclasses of ExternalStructure.

>>>

>>> And with it, they get their own instance of ExternalType, which is

>>> managed in the classVar StructTypes. Let's call them struct type.

>>>

>>> > Now we have another mean by adding a class side method in ExternalType.

>>>

>>> That kind of aliasing is for compile-time type referencing only -- such

>>> as in FFI calls (i.e. pragmas) or struct-field definitions

>>> (i.e. #fields). (And soon if I have the first version of

>>> FFI-Callback to show you :)

>>>

>>> Because those aliases become fully transparent after compiling the

>>> method for the FFI call into an instance of ExternalLibraryFunction and

>>> after compiling the field specs of external structures into compiledSpec

>>> for struct types.

>>>

>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a

>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And

>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into

>>> Squeak FFI vocabulary. Nothing more. Nothing less.

>>>

>>> Having that, please do not use class-side methods in ExternalType to

>>> alias a (complex) struct type. Use them only for renaming atomic types.

>>> Please. We may want to add checks to enforce that.

>>>

>>> I suppose that this discussion focuses on type aliases that are

>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>>> thus get their own struct type. So, let's ignore this other mechanism for

>>> now.

>>>

>>> > This has one advantage: type safety. You cannot pass a Foo to a

>>> function expecting a Bar, even if they both alias int...

>>>

>>> Yes! So, we are now talking about type aliases to atomic types. The

>>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>>> this:

>>>

>>> (...forgive my "creative" representation of WordArray here...)

>>>

>>> compiledSpec: 0A 04 00 01

>>> referentClass: Foo

>>>

>>> How does the atomic type for 'char' look like?

>>>

>>> compiledSpec: 0A 04 00 01

>>> referentClass: nil

>>>

>>> Consequently, argument coercing will fail if you pass a 'char' when a

>>> 'Foo' is expected. Because the referentClass does not match --- even if the

>>> compiledSpec does match.

>>>

>>> > But this has one major drawback: with current FFI, you cannot pass a

>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>>> an integer type...

>>>

>>> That's correct. If the origin of such an argument is from within the

>>> image, you have to wrap it. But when returned from another call ... well

>>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>>

>>> > FFI plugin is clever enough to recognize an atomic type alias and

>>> simply return a Smalltalk Integer...

>>> > So it's not Bar all the way down, I have to manually wrap Bar...

>>> > This is not consistent.

>>>

>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>>> because it does so for argument coercing.

>>>

>>> Now I understand why the code generation of struct-field accessors was

>>> so apparently broken for me. I changed that so that having an alias type as

>>> part of a larger struct will automatically wrap that for you into the alias

>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>>> ... Ha ha. ;-) #ffiLongVsInt.

>>>

>>> It should be "Bar all the way down". Definitely.

>>>

>>> > If we return an int, we must accept an int, otherwise, we cannot chain

>>> FFI calls...

>>>

>>> Well, not for type aliases. I would like keep the path of type safety

>>> here, you mentioned above.

>>>

>>> Just fix the FFI plugin to respect referentClass when packaging the

>>> return value.

>>>

>>> > I could use the lightweight alias instead, [...]

>>>

>>> Please don't. See above. Those "lightweight alias" are for renaming

>>> atomic types only. Let's keep it simple and try to address you concern here

>>> with struct types and ExternalTypeAlias. :-)

>>>

>>> > there was another usage where it was convenient to use

>>> ExternalTypeAlias: enum.

>>> > Indeed, I simply defined all enum symbols as class side method.

>>>

>>> I like that! It reads like a next step for ExternalPool. Once you have

>>> extracted the constant values from header files, and once you have those

>>> values for your platform in classVars in your external pool, use that pool

>>> to define enums through ExternalStructure.

>>>

>>> Like this:

>>>

>>> ExternalTypeAlias subclass: #MyEnumBar

>>> instanceVariableNames: ''

>>> classVariableNames: ''

>>> poolDictionaries: 'HDF5Pool'

>>> category: 'HDF5'

>>>

>>> MyEnumBar class >> #poolFields

>>> "

>>> self defineFields.

>>> "

>>> ^ #(

>>> (beg HDF5_BEG)

>>> (mid HDF5_MID)

>>> (end HDF5_END)

>>> )

>>>

>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>>> external-pool definitions. See class comment in ExternalPool to get started.

>>>

>>> Note that #poolFields would translate to class-side methods as you

>>> suggested.

>>>

>>> > If I use a simpler alias, where are those ids going to?

>>>

>>> Please don't. See above. :-)

>>>

>>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>>> inputs/outputs behave consistently.

>>>

>>> Absolutely!

>>>

>>> > - should a function expecting an ExternalTypeAlias of atomic type

>>> accept an immediate value of compatible type?

>>>

>>> Considering type safety, I think not.

>>>

>>> Well ... since this is a convenience issue and thus not about saving

>>> run-time cost ... What about adding a callback into the image to react on

>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>>> stored in ExternalLibraryFunction since this is accessible to the plugin

>>> anyway?

>>>

>>>

>>>

>>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>>

>>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>>> FFI-Callback. But ignore me here :)

>>>

>>> > - should a function returning an ExternalTypeAlias of atomic type

>>> instantiate the Alias?

>>>

>>> Yes, please! See above.

>>>

>>> If you want to trade type safety in for a performance gain, just use a

>>> "lightweight alias" as explained above. And package that return value

>>> manually.

>>>

>>> Best,

>>> Marcel

>>>

>>>

>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:

>>>

>>> Hi all,

>>> following the question of Marcel about usage of ExternalTypeAlias:

>>>

>>> Type alias are subclasses of ExternalStructure.

>>> I used them in HDF5 because there was no other means to define an alias

>>> at the time.

>>> Now we have another mean by adding a class side method in ExternalType

>>> (I would have rather imagined the usage of some annotation to get a

>>> declarative style)

>>>

>>> This has one advantage: type safety. You cannot pass a Foo to a function

>>> expecting a Bar, even if they both alias int...

>>>

>>> But this has one major drawback: with current FFI, you cannot pass a

>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>>> an integer type...

>>>

>>> Thus, you have to wrap every Bar argument into a Bar instance...

>>> Gasp, this is going too far...

>>>

>>> What about functions returning such alias?

>>> Function returning struct by value allocate a struct, but it's not the

>>> case here, FFI plugin is clever enough to recognize an atomic type alias

>>> and simply return a Smalltalk Integer...

>>> So it's not Bar all the way down, I have to manually wrap Bar...

>>> This is not consistent.

>>> If we return an int, we must accept an int, otherwise, we cannot chain

>>> FFI calls...

>>>

>>> I could use the lightweight alias instead, but there was another usage

>>> where it was convenient to use ExternalTypeAlias: enum.

>>> Indeed, I simply defined all enum symbols as class side method. For

>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>>> This way, I normally can pass a Bar mid to an external function.

>>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>>

>>> If I use a simpler alias, where are those ids going to?

>>>

>>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>>> inputs/outputs behave consistently.

>>> The choice is this one:

>>> - should a function expecting an ExternalTypeAlias of atomic type accept

>>> an immediate value of compatible type? (the permissive implementation)

>>> - should a function returning an ExternalTypeAlias of atomic type

>>> instantiate the Alias? (the typesafe implementation, with higher runtime

>>> cost due to pressure on GC)

>>>

>>>

>>>

>>>

>>>

>>>

>>>

>>

>



Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :
 












Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...
The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).
This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,
Marcel






Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak


specific!











Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :





> Hi Marcel,


> thanks for your vote. Anyone else?


>


> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a


> écrit :


>


>> Hi Nicolas, hi all! :-)


>>


>> > Type alias are subclasses of ExternalStructure.


>>


>> And with it, they get their own instance of ExternalType, which is


>> managed in the classVar StructTypes. Let's call them struct type.


>>


>> > Now we have another mean by adding a class side method in ExternalType.


>>


>> That kind of aliasing is for compile-time type referencing only -- such


>> as in FFI calls (i.e. pragmas) or struct-field definitions


>> (i.e. #fields). (And soon if I have the first version of


>> FFI-Callback to show you :)


>>


>> Because those aliases become fully transparent after compiling the method


>> for the FFI call into an instance of ExternalLibraryFunction and after


>> compiling the field specs of external structures into compiledSpec for


>> struct types.


>>


>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',


>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'


>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI


>> vocabulary. Nothing more. Nothing less.


>>


>> Having that, please do not use class-side methods in ExternalType to


>> alias a (complex) struct type. Use them only for renaming atomic types.


>> Please. We may want to add checks to enforce that.


>>


>> I suppose that this discussion focuses on type aliases that are


>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and


>> thus get their own struct type. So, let's ignore this other mechanism for


>> now.


>>


>> > This has one advantage: type safety. You cannot pass a Foo to a


>> function expecting a Bar, even if they both alias int...


>>


>> Yes! So, we are now talking about type aliases to atomic types. The


>> struct type you get when aliasing a 'char' as Foo, for example, looks like


>> this:


>>


>> (...forgive my "creative" representation of WordArray here...)


>>


>> compiledSpec: 0A 04 00 01


>> referentClass: Foo


>>


>> How does the atomic type for 'char' look like?


>>


>> compiledSpec: 0A 04 00 01


>> referentClass: nil


>>


>> Consequently, argument coercing will fail if you pass a 'char' when a


>> 'Foo' is expected. Because the referentClass does not match --- even if the


>> compiledSpec does match.


>>


>> > But this has one major drawback: with current FFI, you cannot pass a


>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for


>> an integer type...


>>


>> That's correct. If the origin of such an argument is from within the


>> image, you have to wrap it. But when returned from another call ... well


>> ... that wrapping should have happened in the plugin ... But read on! :-)


>>


>> > FFI plugin is clever enough to recognize an atomic type alias and


>> simply return a Smalltalk Integer...


>> > So it's not Bar all the way down, I have to manually wrap Bar...


>> > This is not consistent.


>>


>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass


>> because it does so for argument coercing.


>>


>> Now I understand why the code generation of struct-field accessors was so


>> apparently broken for me. I changed that so that having an alias type as


>> part of a larger struct will automatically wrap that for you into the alias


>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'


>> ... Ha ha. ;-) #ffiLongVsInt.


>>


>> It should be "Bar all the way down". Definitely.


>>


>> > If we return an int, we must accept an int, otherwise, we cannot chain


>> FFI calls...


>>


>> Well, not for type aliases. I would like keep the path of type safety


>> here, you mentioned above.


>>


>> Just fix the FFI plugin to respect referentClass when packaging the


>> return value.


>>


>> > I could use the lightweight alias instead, [...]


>>


>> Please don't. See above. Those "lightweight alias" are for renaming


>> atomic types only. Let's keep it simple and try to address you concern here


>> with struct types and ExternalTypeAlias. :-)


>>


>> > there was another usage where it was convenient to use


>> ExternalTypeAlias: enum.


>> > Indeed, I simply defined all enum symbols as class side method.


>>


>> I like that! It reads like a next step for ExternalPool. Once you have


>> extracted the constant values from header files, and once you have those


>> values for your platform in classVars in your external pool, use that pool


>> to define enums through ExternalStructure.


>>


>> Like this:


>>


>> ExternalTypeAlias subclass: #MyEnumBar


>> instanceVariableNames: ''


>> classVariableNames: ''


>> poolDictionaries: 'HDF5Pool'


>> category: 'HDF5'


>>


>> MyEnumBar class >> #poolFields


>> "


>> self defineFields.


>> "


>> ^ #(


>> (beg HDF5_BEG)


>> (mid HDF5_MID)


>> (end HDF5_END)


>> )


>>


>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via


>> external-pool definitions. See class comment in ExternalPool to get started.


>>


>> Note that #poolFields would translate to class-side methods as you


>> suggested.


>>


>> > If I use a simpler alias, where are those ids going to?


>>


>> Please don't. See above. :-)


>>


>> > IMO we cannot keep status quo in the plugin, we gotta to make the


>> inputs/outputs behave consistently.


>>


>> Absolutely!


>>


>> > - should a function expecting an ExternalTypeAlias of atomic type


>> accept an immediate value of compatible type?


>>


>> Considering type safety, I think not.


>>


>> Well ... since this is a convenience issue and thus not about saving


>> run-time cost ... What about adding a callback into the image to react on


>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be


>> stored in ExternalLibraryFunction since this is accessible to the plugin


>> anyway?


>>


>>


>>


>> ... would it be possible? Like that #doesNotUnderstand: callback?


>>


>> I would love to do ad-hoc packaging of arguments. (Also thinking about


>> FFI-Callback. But ignore me here :)


>>


>> > - should a function returning an ExternalTypeAlias of atomic type


>> instantiate the Alias?


>>


>> Yes, please! See above.


>>


>> If you want to trade type safety in for a performance gain, just use a


>> "lightweight alias" as explained above. And package that return value


>> manually.


>>


>> Best,


>> Marcel


>>


>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:


>> Hi all,


>> following the question of Marcel about usage of ExternalTypeAlias:


>>


>> Type alias are subclasses of ExternalStructure.


>> I used them in HDF5 because there was no other means to define an alias


>> at the time.


>> Now we have another mean by adding a class side method in ExternalType (I


>> would have rather imagined the usage of some annotation to get a


>> declarative style)


>>


>> This has one advantage: type safety. You cannot pass a Foo to a function


>> expecting a Bar, even if they both alias int...


>>


>> But this has one major drawback: with current FFI, you cannot pass a


>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for


>> an integer type...


>>


>> Thus, you have to wrap every Bar argument into a Bar instance...


>> Gasp, this is going too far...


>>


>> What about functions returning such alias?


>> Function returning struct by value allocate a struct, but it's not the


>> case here, FFI plugin is clever enough to recognize an atomic type alias


>> and simply return a Smalltalk Integer...


>> So it's not Bar all the way down, I have to manually wrap Bar...


>> This is not consistent.


>> If we return an int, we must accept an int, otherwise, we cannot chain


>> FFI calls...


>>


>> I could use the lightweight alias instead, but there was another usage


>> where it was convenient to use ExternalTypeAlias: enum.


>> Indeed, I simply defined all enum symbols as class side method. For


>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar


>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.


>> This way, I normally can pass a Bar mid to an external function.


>> (currently, we must defne mid ^Bar new value: 1; yourself)


>>


>> If I use a simpler alias, where are those ids going to?


>>


>> IMO we cannot keep statu quo in the plugin, we gotta to make the


>> inputs/outputs behave consistently.


>> The choice is this one:


>> - should a function expecting an ExternalTypeAlias of atomic type accept


>> an immediate value of compatible type? (the permissive implementation)


>> - should a function returning an ExternalTypeAlias of atomic type


>> instantiate the Alias? (the typesafe implementation, with higher runtime


>> cost due to pressure on GC)


>>


>>


>>


>>


Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :



Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel



Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:




Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)

























Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

marcel.taeumel
In reply to this post by Nicolas Cellier
 
I'd like to have a 3rd possibility: pointing to a sub-region of a ByteArray
> (ByteArray + offset)

For this, you could add "offset" to ExternalData (in addition to "type" and "size"). This would also postpone the expensive pointer arithmetic implemented in ExternalAddress >> #+ to the last possible moment. :-) 

Best,
Marcel

Am 19.06.2020 18:52:02 schrieb Nicolas Cellier <[hidden email]>:

Well, my mental model is this one:

ExternalData needs to refer to an area of memory, so as to be able to act
as a pointer (single element or array does not really matter).
If this memory area lies in external heap, then we need an ExternalAddress
as handle.
If this memory area lies in object memory, it must be a ByteArray.
We cannot handle pointers to object memory, because object memory is
relocatable...
Well, unless we pin the object - that why we pass the ByteArray itself, not
its address..

This is independent of how we interpret the contents.
IMO, it's more useful to avoid the pointer indirection in type, because we
cannot handle double pointer currently...
Of course, the only pointers that the memory area could contain are
pointers to external heap (or eventually pinned objects).

I'd like to have a 3rd possibility: pointing to a sub-region of a ByteArray
(ByteArray + offset).
That may be very useful for emulating.
An example in Smallapack: pass a pointer to the imaginary part of a complex
array.
Since real and imaginary parts are interleaved, just offsetting the pointer
and using a stride of 2 elements does the trick when one wants to access
real or imaginary part
(most lapack/blas routines are equipped to handle simple strides).
This is currently possible with ExternalAddress, but NOT when matrix data
lies in ObjectMemory.
Another example would be if you wanted to implement an interpreter for a
low level language... (emulate a processor, whatever).



Le ven. 19 juin 2020 à 18:04, Marcel Taeumel a
écrit :

>
> So, what about this:
>
> ExternalData<@00f3a3ee, int="">
>
> I think it could mean that you want to interpret the very pointer address
> itself as an integer. That is, you do not want to follow to where it points
> to.
>
> Then what about this:
>
> ExternalData<@00f3a3ee, double="">
>
> Ha! An error, I suppose. Unless ... given 8-byte pointers, you can try to
> read a double out of it. :-D Just kidding. It should be an error.
>
> .
> .
> .
>
> > > (On a side note: I think that ExternalData should always have a
> pointer type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
> > Let me disagree here...
>
> The examples I just wrote down in the previous messages let me realize why
> you did disagree. :-D
>
> I hope this helps you with improving argument coercing and return value
> packaging in the FFI plugin. ^__^
>
> Best,
> Marcel
>
> Am 19.06.2020 17:53:21 schrieb Marcel Taeumel :
> > then you would use int* and not int**
>
> My bad: " then you would use int** and not int* " Sorry for the noise
> -.-"
>
> Am 19.06.2020 17:52:21 schrieb Marcel Taeumel :
> > If you want to make a pointer array out of it, change the type to int**
>
> Correction: If you happen to now, that this external address points to an
> array of pointers to int, then you would use int* and not int**. You cannot
> just "decide" what the data behind that external address should be. .. you
> know what I mean. ;-)
>
> best,
> Marcel
>
> Am 19.06.2020 17:47:36 schrieb Marcel Taeumel :
> Hi Nicolas.
>
> Maybe one more thought on
>
> ExternalData
>
> There is no need to have
>
> ExternalData<1312301580, int="">
>
> because it can just be 1312301580.
>
> On the other hand:
>
> ExternalData<#[ 12="" 34="" 56="" 78="" ],="" int="">
>
> Tells you that (1) this data wants to be interpreted as signed int and (2)
> this data is already in Squeak's object memory.
>
> Finally:
>
> ExternalData<@00f3a3ee, int*="">
>
> Tells you that (1) this data wants to be interpreted as signed int and (2)
> this data is in external memory.
>
> If you want to make an array out of it, make use of the "size" field in
> ExternalData.
>
> If you want to make a pointer array out of it, change the type to int**
>
> ExternalData<@00f3a3ee, int**="">
>
> So, what does this mean then:
>
> ExternalData<#[ 12="" 34="" 56="" 78="" ],="" int*="">
>
> Well, it tells you that there is a pointer to an int stored in a byte
> array. This comes very handy if you think about type aliases to pointer
> types such as "typedef int* INT_P". Because then it would be
>
> ExternalData<#[ 12="" 34="" 56="" 78="" ],="" int_p="">
>
> To where this pointer points to? Probably garbage. I just made it up for
> this example. :-D
>
> Best,
> Marcel
>
> Am 19.06.2020 17:15:23 schrieb Marcel Taeumel :
> > The type should better describe the contents of the handle (be it an
> > ExternalAddress or a zone of object memory (ByteArray)).
> > Typically, if you have an external (global) variable of type foo (extern
> > foo my_var;) then the natural type is foo.
> > Having foo*, would mean that we handle an array of foo*, or a pointer to
> > foo* (foo**).
> > This way, we make no difference between type surrogates
> > (ExternalStruct/ExternalTypeAlias) and other atomic cases.
>
> We have the "size" field in ExternalData encode whether "int*" points to a
> single or multiple things. No need to treat a global variable sitting in
> external memory any different here. ... IMO :-)
>
> I know, that this is my personal mental model for Squeak FFI to treat
> pointer types as things in external memory and non-pointer types as things
> in Squeak's object memory. The latter includes ByteArray and Integer and
> Float and also IntegerArray etc. Just not ExternalAddress.
>
> This basically comes from the fact that, until now, the FFI plugin gave me
> a ByteArray in return when I told it to be "Foo", not "Foo*". So,
> non-pointer type means ByteArray. Pointer-type means ExternalAddress.
>
> Type aliases are not really different here. Each alias as a corresponding
> struct type, which has again a non-pointer variant and a pointer variant.
>
> Type aliases to pointer types are just special because they store the
> pointer address in a byte array. The corresponding non-pointer type will do
> fine for such aliases. Their pointer type is not relevant in general, I
> suppose.
>
> > Having foo*, would mean that we handle an array of foo*, or a pointer to foo*
> (foo**).
>
> Well, I have a simple idea on how to represent foo** (and other
> dimensions) without the FFI plugin needing to know. ... unless you are
> concerned about type safety for that, too. Then we would really need to
> encode that in the compiledSpec.
>
> If not, ExternalData with foo** type will just answer ExternalData with
> foo* when asked via "at: 1" for example. :-)
>
> > or an Alien as we may want to support that too.
>
> I don't think that's necessary. Alien has its own call-out mechanism
> through the IA32ABI plugin and also a different layout/usage of ByteArray.
>
> Best,
> Marcel
>
> Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <>
> [hidden email]>:
> Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
> écrit :
>
> >
> > > Err, I move this thread to Opensmalltalk VM dev, because it is not
> > Squeak specific!
> >
> > Absolutely. :-)
> >
> > (On a side note: I think that ExternalData should always have a pointer
> > type in its "type" field unless "handle" is a ByteArray instead of
> > ExternalAddress)
> >
> > Let me disagree here...
> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.
>
> (On another side note: I also think that "handle" in ExternalData should
> > never ever be an atomic Smalltalk object (Integer or Float) but always
> > either ByteArray or ExternalAddress. ... which makes it different from
> what
> > "handle" can be in ExternalStructs that represent type aliases. :-)
> >
> > Yes!
> It could be an ExternalAddress, or a zone of object memory (ByteArray) or
> an Alien as we may want to support that too.
> I'd be inclined to support doubleByte, word, and doubleWord arrays if the
> size of atomic type matches
> (Typically passing a DoubleByteArray, WordArray, a Float32Array, a
> Float64Array to a short */int */float*/double*).
>
> Best,
> > Marcel
> >
> > Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> > [hidden email]>:
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak
> > specific!
> >
> >
> >
> > Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> > [hidden email]> a écrit :
> >
> > > Hi Marcel,
> > > thanks for your vote. Anyone else?
> > >
> > > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > > écrit :
> > >
> > >> Hi Nicolas, hi all! :-)
> > >>
> > >> > Type alias are subclasses of ExternalStructure.
> > >>
> > >> And with it, they get their own instance of ExternalType, which is
> > >> managed in the classVar StructTypes. Let's call them struct type.
> > >>
> > >> > Now we have another mean by adding a class side method in
> > ExternalType.
> > >>
> > >> That kind of aliasing is for compile-time type referencing only --
> such
> > >> as in FFI calls (i.e. pragmas) or struct-field definitions
> > >> (i.e. #fields). (And soon if I have the first version of
> > >> FFI-Callback to show you :)
> > >>
> > >> Because those aliases become fully transparent after compiling the
> > method
> > >> for the FFI call into an instance of ExternalLibraryFunction and
> after
> > >> compiling the field specs of external structures into compiledSpec
> for
> > >> struct types.
> > >>
> > >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> > 'long',
> > >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> > 'int32_t'
> > >> also becomes a 'long'. So, it just translates C vocabulary into
> Squeak
> > FFI
> > >> vocabulary. Nothing more. Nothing less.
> > >>
> > >> Having that, please do not use class-side methods in ExternalType to
> > >> alias a (complex) struct type. Use them only for renaming atomic
> types.
> > >> Please. We may want to add checks to enforce that.
> > >>
> > >> I suppose that this discussion focuses on type aliases that are
> > >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> > and
> > >> thus get their own struct type. So, let's ignore this other mechanism
> > for
> > >> now.
> > >>
> > >> > This has one advantage: type safety. You cannot pass a Foo to a
> > >> function expecting a Bar, even if they both alias int...
> > >>
> > >> Yes! So, we are now talking about type aliases to atomic types. The
> > >> struct type you get when aliasing a 'char' as Foo, for example, looks
> > like
> > >> this:
> > >>
> > >> (...forgive my "creative" representation of WordArray here...)
> > >>
> > >> compiledSpec: 0A 04 00 01
> > >> referentClass: Foo
> > >>
> > >> How does the atomic type for 'char' look like?
> > >>
> > >> compiledSpec: 0A 04 00 01
> > >> referentClass: nil
> > >>
> > >> Consequently, argument coercing will fail if you pass a 'char' when a
> > >> 'Foo' is expected. Because the referentClass does not match --- even
> if
> > the
> > >> compiledSpec does match.
> > >>
> > >> > But this has one major drawback: with current FFI, you cannot pass
> a
> > >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias
> > for
> > >> an integer type...
> > >>
> > >> That's correct. If the origin of such an argument is from within the
> > >> image, you have to wrap it. But when returned from another call ...
> > well
> > >> ... that wrapping should have happened in the plugin ... But read on!
> > :-)
> > >>
> > >> > FFI plugin is clever enough to recognize an atomic type alias and
> > >> simply return a Smalltalk Integer...
> > >> > So it's not Bar all the way down, I have to manually wrap Bar...
> > >> > This is not consistent.
> > >>
> > >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> > >> because it does so for argument coercing.
> > >>
> > >> Now I understand why the code generation of struct-field accessors
> was
> > so
> > >> apparently broken for me. I changed that so that having an alias type
> > as
> > >> part of a larger struct will automatically wrap that for you into the
> > alias
> > >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> > 'long'
> > >> ... Ha ha. ;-) #ffiLongVsInt.
> > >>
> > >> It should be "Bar all the way down". Definitely.
> > >>
> > >> > If we return an int, we must accept an int, otherwise, we cannot
> > chain
> > >> FFI calls...
> > >>
> > >> Well, not for type aliases. I would like keep the path of type safety
> > >> here, you mentioned above.
> > >>
> > >> Just fix the FFI plugin to respect referentClass when packaging the
> > >> return value.
> > >>
> > >> > I could use the lightweight alias instead, [...]
> > >>
> > >> Please don't. See above. Those "lightweight alias" are for renaming
> > >> atomic types only. Let's keep it simple and try to address you
> concern
> > here
> > >> with struct types and ExternalTypeAlias. :-)
> > >>
> > >> > there was another usage where it was convenient to use
> > >> ExternalTypeAlias: enum.
> > >> > Indeed, I simply defined all enum symbols as class side method.
> > >>
> > >> I like that! It reads like a next step for ExternalPool. Once you
> have
> > >> extracted the constant values from header files, and once you have
> > those
> > >> values for your platform in classVars in your external pool, use that
> > pool
> > >> to define enums through ExternalStructure.
> > >>
> > >> Like this:
> > >>
> > >> ExternalTypeAlias subclass: #MyEnumBar
> > >> instanceVariableNames: ''
> > >> classVariableNames: ''
> > >> poolDictionaries: 'HDF5Pool'
> > >> category: 'HDF5'
> > >>
> > >> MyEnumBar class >> #poolFields
> > >> "
> > >> self defineFields.
> > >> "
> > >> ^ #(
> > >> (beg HDF5_BEG)
> > >> (mid HDF5_MID)
> > >> (end HDF5_END)
> > >> )
> > >>
> > >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> > >> external-pool definitions. See class comment in ExternalPool to get
> > started.
> > >>
> > >> Note that #poolFields would translate to class-side methods as you
> > >> suggested.
> > >>
> > >> > If I use a simpler alias, where are those ids going to?
> > >>
> > >> Please don't. See above. :-)
> > >>
> > >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> > >> inputs/outputs behave consistently.
> > >>
> > >> Absolutely!
> > >>
> > >> > - should a function expecting an ExternalTypeAlias of atomic type
> > >> accept an immediate value of compatible type?
> > >>
> > >> Considering type safety, I think not.
> > >>
> > >> Well ... since this is a convenience issue and thus not about saving
> > >> run-time cost ... What about adding a callback into the image to
> react
> > on
> > >> FFIErrorCoercionFailed? Maybe the selector for such a callback could
> be
> > >> stored in ExternalLibraryFunction since this is accessible to the
> > plugin
> > >> anyway?
> > >>
> > >>
> > >>
> > >> ... would it be possible? Like that #doesNotUnderstand: callback?
> > >>
> > >> I would love to do ad-hoc packaging of arguments. (Also thinking
> about
> > >> FFI-Callback. But ignore me here :)
> > >>
> > >> > - should a function returning an ExternalTypeAlias of atomic type
> > >> instantiate the Alias?
> > >>
> > >> Yes, please! See above.
> > >>
> > >> If you want to trade type safety in for a performance gain, just use
> a
> > >> "lightweight alias" as explained above. And package that return value
> > >> manually.
> > >>
> > >> Best,
> > >> Marcel
> > >>
> > >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> > >> [hidden email]>:
> > >> Hi all,
> > >> following the question of Marcel about usage of ExternalTypeAlias:
> > >>
> > >> Type alias are subclasses of ExternalStructure.
> > >> I used them in HDF5 because there was no other means to define an
> alias
> > >> at the time.
> > >> Now we have another mean by adding a class side method in
> ExternalType
> > (I
> > >> would have rather imagined the usage of some annotation to get a
> > >> declarative style)
> > >>
> > >> This has one advantage: type safety. You cannot pass a Foo to a
> > function
> > >> expecting a Bar, even if they both alias int...
> > >>
> > >> But this has one major drawback: with current FFI, you cannot pass a
> > >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias
> > for
> > >> an integer type...
> > >>
> > >> Thus, you have to wrap every Bar argument into a Bar instance...
> > >> Gasp, this is going too far...
> > >>
> > >> What about functions returning such alias?
> > >> Function returning struct by value allocate a struct, but it's not
> the
> > >> case here, FFI plugin is clever enough to recognize an atomic type
> > alias
> > >> and simply return a Smalltalk Integer...
> > >> So it's not Bar all the way down, I have to manually wrap Bar...
> > >> This is not consistent.
> > >> If we return an int, we must accept an int, otherwise, we cannot
> chain
> > >> FFI calls...
> > >>
> > >> I could use the lightweight alias instead, but there was another
> usage
> > >> where it was convenient to use ExternalTypeAlias: enum.
> > >> Indeed, I simply defined all enum symbols as class side method. For
> > >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> > Bar
> > >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> > >> This way, I normally can pass a Bar mid to an external function.
> > >> (currently, we must defne mid ^Bar new value: 1; yourself)
> > >>
> > >> If I use a simpler alias, where are those ids going to?
> > >>
> > >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> > >> inputs/outputs behave consistently.
> > >> The choice is this one:
> > >> - should a function expecting an ExternalTypeAlias of atomic type
> > accept
> > >> an immediate value of compatible type? (the permissive
> implementation)
> > >> - should a function returning an ExternalTypeAlias of atomic type
> > >> instantiate the Alias? (the typesafe implementation, with higher
> > runtime
> > >> cost due to pressure on GC)
> > >>
> > >>
> > >>
> > >>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak
> > specific!
> >
> >
> >
> > Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> > [hidden email]> a écrit :
> >
> >> Hi Marcel,
> >> thanks for your vote. Anyone else?
> >>
> >> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> >> écrit :
> >>
> >>>
> >>> Hi Nicolas, hi all! :-)
> >>>
> >>> > Type alias are subclasses of ExternalStructure.
> >>>
> >>> And with it, they get their own instance of ExternalType, which is
> >>> managed in the classVar StructTypes. Let's call them struct type.
> >>>
> >>> > Now we have another mean by adding a class side method in
> ExternalType.
> >>>
> >>> That kind of aliasing is for compile-time type referencing only --
> such
> >>> as in FFI calls (i.e. pragmas) or struct-field definitions
> >>> (i.e. #fields). (And soon if I have the first version of
> >>> FFI-Callback to show you :)
> >>>
> >>> Because those aliases become fully transparent after compiling the
> >>> method for the FFI call into an instance of ExternalLibraryFunction
> and
> >>> after compiling the field specs of external structures into
> compiledSpec
> >>> for struct types.
> >>>
> >>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> >>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> >>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary
> into
> >>> Squeak FFI vocabulary. Nothing more. Nothing less.
> >>>
> >>> Having that, please do not use class-side methods in ExternalType to
> >>> alias a (complex) struct type. Use them only for renaming atomic
> types.
> >>> Please. We may want to add checks to enforce that.
> >>>
> >>> I suppose that this discussion focuses on type aliases that are
> >>> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >>> thus get their own struct type. So, let's ignore this other mechanism
> for
> >>> now.
> >>>
> >>> > This has one advantage: type safety. You cannot pass a Foo to a
> >>> function expecting a Bar, even if they both alias int...
> >>>
> >>> Yes! So, we are now talking about type aliases to atomic types. The
> >>> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >>> this:
> >>>
> >>> (...forgive my "creative" representation of WordArray here...)
> >>>
> >>> compiledSpec: 0A 04 00 01
> >>> referentClass: Foo
> >>>
> >>> How does the atomic type for 'char' look like?
> >>>
> >>> compiledSpec: 0A 04 00 01
> >>> referentClass: nil
> >>>
> >>> Consequently, argument coercing will fail if you pass a 'char' when a
> >>> 'Foo' is expected. Because the referentClass does not match --- even
> if the
> >>> compiledSpec does match.
> >>>
> >>> > But this has one major drawback: with current FFI, you cannot pass a
> >>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias for
> >>> an integer type...
> >>>
> >>> That's correct. If the origin of such an argument is from within the
> >>> image, you have to wrap it. But when returned from another call ...
> well
> >>> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>>
> >>> > FFI plugin is clever enough to recognize an atomic type alias and
> >>> simply return a Smalltalk Integer...
> >>> > So it's not Bar all the way down, I have to manually wrap Bar...
> >>> > This is not consistent.
> >>>
> >>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >>> because it does so for argument coercing.
> >>>
> >>> Now I understand why the code generation of struct-field accessors was
> >>> so apparently broken for me. I changed that so that having an alias
> type as
> >>> part of a larger struct will automatically wrap that for you into the
> alias
> >>> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >>> ... Ha ha. ;-) #ffiLongVsInt.
> >>>
> >>> It should be "Bar all the way down". Definitely.
> >>>
> >>> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >>> FFI calls...
> >>>
> >>> Well, not for type aliases. I would like keep the path of type safety
> >>> here, you mentioned above.
> >>>
> >>> Just fix the FFI plugin to respect referentClass when packaging the
> >>> return value.
> >>>
> >>> > I could use the lightweight alias instead, [...]
> >>>
> >>> Please don't. See above. Those "lightweight alias" are for renaming
> >>> atomic types only. Let's keep it simple and try to address you concern
> here
> >>> with struct types and ExternalTypeAlias. :-)
> >>>
> >>> > there was another usage where it was convenient to use
> >>> ExternalTypeAlias: enum.
> >>> > Indeed, I simply defined all enum symbols as class side method.
> >>>
> >>> I like that! It reads like a next step for ExternalPool. Once you have
> >>> extracted the constant values from header files, and once you have
> those
> >>> values for your platform in classVars in your external pool, use that
> pool
> >>> to define enums through ExternalStructure.
> >>>
> >>> Like this:
> >>>
> >>> ExternalTypeAlias subclass: #MyEnumBar
> >>> instanceVariableNames: ''
> >>> classVariableNames: ''
> >>> poolDictionaries: 'HDF5Pool'
> >>> category: 'HDF5'
> >>>
> >>> MyEnumBar class >> #poolFields
> >>> "
> >>> self defineFields.
> >>> "
> >>> ^ #(
> >>> (beg HDF5_BEG)
> >>> (mid HDF5_MID)
> >>> (end HDF5_END)
> >>> )
> >>>
> >>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >>> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>>
> >>> Note that #poolFields would translate to class-side methods as you
> >>> suggested.
> >>>
> >>> > If I use a simpler alias, where are those ids going to?
> >>>
> >>> Please don't. See above. :-)
> >>>
> >>> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >>> inputs/outputs behave consistently.
> >>>
> >>> Absolutely!
> >>>
> >>> > - should a function expecting an ExternalTypeAlias of atomic type
> >>> accept an immediate value of compatible type?
> >>>
> >>> Considering type safety, I think not.
> >>>
> >>> Well ... since this is a convenience issue and thus not about saving
> >>> run-time cost ... What about adding a callback into the image to react
> on
> >>> FFIErrorCoercionFailed? Maybe the selector for such a callback could
> be
> >>> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >>> anyway?
> >>>
> >>>
> >>>
> >>> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>>
> >>> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >>> FFI-Callback. But ignore me here :)
> >>>
> >>> > - should a function returning an ExternalTypeAlias of atomic type
> >>> instantiate the Alias?
> >>>
> >>> Yes, please! See above.
> >>>
> >>> If you want to trade type safety in for a performance gain, just use a
> >>> "lightweight alias" as explained above. And package that return value
> >>> manually.
> >>>
> >>> Best,
> >>> Marcel
> >>>
> >>>
> >>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >>> [hidden email]>:
> >>>
> >>> Hi all,
> >>> following the question of Marcel about usage of ExternalTypeAlias:
> >>>
> >>> Type alias are subclasses of ExternalStructure.
> >>> I used them in HDF5 because there was no other means to define an
> alias
> >>> at the time.
> >>> Now we have another mean by adding a class side method in ExternalType
> >>> (I would have rather imagined the usage of some annotation to get a
> >>> declarative style)
> >>>
> >>> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >>> expecting a Bar, even if they both alias int...
> >>>
> >>> But this has one major drawback: with current FFI, you cannot pass a
> >>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias for
> >>> an integer type...
> >>>
> >>> Thus, you have to wrap every Bar argument into a Bar instance...
> >>> Gasp, this is going too far...
> >>>
> >>> What about functions returning such alias?
> >>> Function returning struct by value allocate a struct, but it's not the
> >>> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >>> and simply return a Smalltalk Integer...
> >>> So it's not Bar all the way down, I have to manually wrap Bar...
> >>> This is not consistent.
> >>> If we return an int, we must accept an int, otherwise, we cannot chain
> >>> FFI calls...
> >>>
> >>> I could use the lightweight alias instead, but there was another usage
> >>> where it was convenient to use ExternalTypeAlias: enum.
> >>> Indeed, I simply defined all enum symbols as class side method. For
> >>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >>> This way, I normally can pass a Bar mid to an external function.
> >>> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>>
> >>> If I use a simpler alias, where are those ids going to?
> >>>
> >>> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >>> inputs/outputs behave consistently.
> >>> The choice is this one:
> >>> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >>> an immediate value of compatible type? (the permissive implementation)
> >>> - should a function returning an ExternalTypeAlias of atomic type
> >>> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >>> cost due to pressure on GC)
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >
>
>
> Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
> écrit :
>
>>
>>
>>
>>
>>
>>
>>
>> > Err, I move this thread to Opensmalltalk VM dev, because it is not
>> Squeak specific!
>>
>> Absolutely. :-)
>>
>> (On a side note: I think that ExternalData should always have a pointer
>> type in its "type" field unless "handle" is a ByteArray instead of
>> ExternalAddress)
>>
>> Let me disagree here...
> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.
>
> (On another side note: I also think that "handle" in ExternalData should
>> never ever be an atomic Smalltalk object (Integer or Float) but always
>> either ByteArray or ExternalAddress. ... which makes it different from what
>> "handle" can be in ExternalStructs that represent type aliases. :-)
>>
>> Yes!
> It could be an ExternalAddress, or a zone of object memory (ByteArray) or
> an Alien as we may want to support that too.
> I'd be inclined to support doubleByte, word, and doubleWord arrays if the
> size of atomic type matches
> (Typically passing a DoubleByteArray, WordArray, a Float32Array, a
> Float64Array to a short */int */float*/double*).
>
> Best,
>> Marcel
>>
>>
>>
>> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
>> [hidden email]>:
>> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
>>
>> specific!
>>
>>
>>
>>
>>
>>
>>
>> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
>> [hidden email]> a écrit :
>>
>>
>>
>> > Hi Marcel,
>>
>> > thanks for your vote. Anyone else?
>>
>> >
>>
>> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>>
>> > écrit :
>>
>> >
>>
>> >> Hi Nicolas, hi all! :-)
>>
>> >>
>>
>> >> > Type alias are subclasses of ExternalStructure.
>>
>> >>
>>
>> >> And with it, they get their own instance of ExternalType, which is
>>
>> >> managed in the classVar StructTypes. Let's call them struct type.
>>
>> >>
>>
>> >> > Now we have another mean by adding a class side method in
>> ExternalType.
>>
>> >>
>>
>> >> That kind of aliasing is for compile-time type referencing only --
>> such
>>
>> >> as in FFI calls (i.e. pragmas) or struct-field definitions
>>
>> >> (i.e. #fields). (And soon if I have the first version of
>>
>> >> FFI-Callback to show you :)
>>
>> >>
>>
>> >> Because those aliases become fully transparent after compiling the
>> method
>>
>> >> for the FFI call into an instance of ExternalLibraryFunction and after
>>
>> >> compiling the field specs of external structures into compiledSpec for
>>
>> >> struct types.
>>
>> >>
>>
>> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>> 'long',
>>
>> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>> 'int32_t'
>>
>> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
>> FFI
>>
>> >> vocabulary. Nothing more. Nothing less.
>>
>> >>
>>
>> >> Having that, please do not use class-side methods in ExternalType to
>>
>> >> alias a (complex) struct type. Use them only for renaming atomic
>> types.
>>
>> >> Please. We may want to add checks to enforce that.
>>
>> >>
>>
>> >> I suppose that this discussion focuses on type aliases that are
>>
>> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
>> and
>>
>> >> thus get their own struct type. So, let's ignore this other mechanism
>> for
>>
>> >> now.
>>
>> >>
>>
>> >> > This has one advantage: type safety. You cannot pass a Foo to a
>>
>> >> function expecting a Bar, even if they both alias int...
>>
>> >>
>>
>> >> Yes! So, we are now talking about type aliases to atomic types. The
>>
>> >> struct type you get when aliasing a 'char' as Foo, for example, looks
>> like
>>
>> >> this:
>>
>> >>
>>
>> >> (...forgive my "creative" representation of WordArray here...)
>>
>> >>
>>
>> >> compiledSpec: 0A 04 00 01
>>
>> >> referentClass: Foo
>>
>> >>
>>
>> >> How does the atomic type for 'char' look like?
>>
>> >>
>>
>> >> compiledSpec: 0A 04 00 01
>>
>> >> referentClass: nil
>>
>> >>
>>
>> >> Consequently, argument coercing will fail if you pass a 'char' when a
>>
>> >> 'Foo' is expected. Because the referentClass does not match --- even
>> if the
>>
>> >> compiledSpec does match.
>>
>> >>
>>
>> >> > But this has one major drawback: with current FFI, you cannot pass a
>>
>> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
>> alias for
>>
>> >> an integer type...
>>
>> >>
>>
>> >> That's correct. If the origin of such an argument is from within the
>>
>> >> image, you have to wrap it. But when returned from another call ...
>> well
>>
>> >> ... that wrapping should have happened in the plugin ... But read on!
>> :-)
>>
>> >>
>>
>> >> > FFI plugin is clever enough to recognize an atomic type alias and
>>
>> >> simply return a Smalltalk Integer...
>>
>> >> > So it's not Bar all the way down, I have to manually wrap Bar...
>>
>> >> > This is not consistent.
>>
>> >>
>>
>> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>
>> >> because it does so for argument coercing.
>>
>> >>
>>
>> >> Now I understand why the code generation of struct-field accessors was
>> so
>>
>> >> apparently broken for me. I changed that so that having an alias type
>> as
>>
>> >> part of a larger struct will automatically wrap that for you into the
>> alias
>>
>> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
>> 'long'
>>
>> >> ... Ha ha. ;-) #ffiLongVsInt.
>>
>> >>
>>
>> >> It should be "Bar all the way down". Definitely.
>>
>> >>
>>
>> >> > If we return an int, we must accept an int, otherwise, we cannot
>> chain
>>
>> >> FFI calls...
>>
>> >>
>>
>> >> Well, not for type aliases. I would like keep the path of type safety
>>
>> >> here, you mentioned above.
>>
>> >>
>>
>> >> Just fix the FFI plugin to respect referentClass when packaging the
>>
>> >> return value.
>>
>> >>
>>
>> >> > I could use the lightweight alias instead, [...]
>>
>> >>
>>
>> >> Please don't. See above. Those "lightweight alias" are for renaming
>>
>> >> atomic types only. Let's keep it simple and try to address you concern
>> here
>>
>> >> with struct types and ExternalTypeAlias. :-)
>>
>> >>
>>
>> >> > there was another usage where it was convenient to use
>>
>> >> ExternalTypeAlias: enum.
>>
>> >> > Indeed, I simply defined all enum symbols as class side method.
>>
>> >>
>>
>> >> I like that! It reads like a next step for ExternalPool. Once you have
>>
>> >> extracted the constant values from header files, and once you have
>> those
>>
>> >> values for your platform in classVars in your external pool, use that
>> pool
>>
>> >> to define enums through ExternalStructure.
>>
>> >>
>>
>> >> Like this:
>>
>> >>
>>
>> >> ExternalTypeAlias subclass: #MyEnumBar
>>
>> >> instanceVariableNames: ''
>>
>> >> classVariableNames: ''
>>
>> >> poolDictionaries: 'HDF5Pool'
>>
>> >> category: 'HDF5'
>>
>> >>
>>
>> >> MyEnumBar class >> #poolFields
>>
>> >> "
>>
>> >> self defineFields.
>>
>> >> "
>>
>> >> ^ #(
>>
>> >> (beg HDF5_BEG)
>>
>> >> (mid HDF5_MID)
>>
>> >> (end HDF5_END)
>>
>> >> )
>>
>> >>
>>
>> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>
>> >> external-pool definitions. See class comment in ExternalPool to get
>> started.
>>
>> >>
>>
>> >> Note that #poolFields would translate to class-side methods as you
>>
>> >> suggested.
>>
>> >>
>>
>> >> > If I use a simpler alias, where are those ids going to?
>>
>> >>
>>
>> >> Please don't. See above. :-)
>>
>> >>
>>
>> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>
>> >> inputs/outputs behave consistently.
>>
>> >>
>>
>> >> Absolutely!
>>
>> >>
>>
>> >> > - should a function expecting an ExternalTypeAlias of atomic type
>>
>> >> accept an immediate value of compatible type?
>>
>> >>
>>
>> >> Considering type safety, I think not.
>>
>> >>
>>
>> >> Well ... since this is a convenience issue and thus not about saving
>>
>> >> run-time cost ... What about adding a callback into the image to react
>> on
>>
>> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could
>> be
>>
>> >> stored in ExternalLibraryFunction since this is accessible to the
>> plugin
>>
>> >> anyway?
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >> ... would it be possible? Like that #doesNotUnderstand: callback?
>>
>> >>
>>
>> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>
>> >> FFI-Callback. But ignore me here :)
>>
>> >>
>>
>> >> > - should a function returning an ExternalTypeAlias of atomic type
>>
>> >> instantiate the Alias?
>>
>> >>
>>
>> >> Yes, please! See above.
>>
>> >>
>>
>> >> If you want to trade type safety in for a performance gain, just use a
>>
>> >> "lightweight alias" as explained above. And package that return value
>>
>> >> manually.
>>
>> >>
>>
>> >> Best,
>>
>> >> Marcel
>>
>> >>
>>
>> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> >> [hidden email]>:
>>
>> >> Hi all,
>>
>> >> following the question of Marcel about usage of ExternalTypeAlias:
>>
>> >>
>>
>> >> Type alias are subclasses of ExternalStructure.
>>
>> >> I used them in HDF5 because there was no other means to define an
>> alias
>>
>> >> at the time.
>>
>> >> Now we have another mean by adding a class side method in ExternalType
>> (I
>>
>> >> would have rather imagined the usage of some annotation to get a
>>
>> >> declarative style)
>>
>> >>
>>
>> >> This has one advantage: type safety. You cannot pass a Foo to a
>> function
>>
>> >> expecting a Bar, even if they both alias int...
>>
>> >>
>>
>> >> But this has one major drawback: with current FFI, you cannot pass a
>>
>> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
>> alias for
>>
>> >> an integer type...
>>
>> >>
>>
>> >> Thus, you have to wrap every Bar argument into a Bar instance...
>>
>> >> Gasp, this is going too far...
>>
>> >>
>>
>> >> What about functions returning such alias?
>>
>> >> Function returning struct by value allocate a struct, but it's not the
>>
>> >> case here, FFI plugin is clever enough to recognize an atomic type
>> alias
>>
>> >> and simply return a Smalltalk Integer...
>>
>> >> So it's not Bar all the way down, I have to manually wrap Bar...
>>
>> >> This is not consistent.
>>
>> >> If we return an int, we must accept an int, otherwise, we cannot chain
>>
>> >> FFI calls...
>>
>> >>
>>
>> >> I could use the lightweight alias instead, but there was another usage
>>
>> >> where it was convenient to use ExternalTypeAlias: enum.
>>
>> >> Indeed, I simply defined all enum symbols as class side method. For
>>
>> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
>> Bar
>>
>> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>
>> >> This way, I normally can pass a Bar mid to an external function.
>>
>> >> (currently, we must defne mid ^Bar new value: 1; yourself)
>>
>> >>
>>
>> >> If I use a simpler alias, where are those ids going to?
>>
>> >>
>>
>> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>
>> >> inputs/outputs behave consistently.
>>
>> >> The choice is this one:
>>
>> >> - should a function expecting an ExternalTypeAlias of atomic type
>> accept
>>
>> >> an immediate value of compatible type? (the permissive implementation)
>>
>> >> - should a function returning an ExternalTypeAlias of atomic type
>>
>> >> instantiate the Alias? (the typesafe implementation, with higher
>> runtime
>>
>> >> cost due to pressure on GC)
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
>> specific!
>>
>>
>>
>> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
>> [hidden email]> a écrit :
>>
>>> Hi Marcel,
>>> thanks for your vote. Anyone else?
>>>
>>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>>> écrit :
>>>
>>>>
>>>>
>>>> Hi Nicolas, hi all! :-)
>>>>
>>>> > Type alias are subclasses of ExternalStructure.
>>>>
>>>> And with it, they get their own instance of ExternalType, which is
>>>> managed in the classVar StructTypes. Let's call them struct type.
>>>>
>>>> > Now we have another mean by adding a class side method in
>>>> ExternalType.
>>>>
>>>> That kind of aliasing is for compile-time type referencing only -- such
>>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>>> (i.e. #fields). (And soon if I have the first version of
>>>> FFI-Callback to show you :)
>>>>
>>>> Because those aliases become fully transparent after compiling the
>>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>>> after compiling the field specs of external structures into compiledSpec
>>>> for struct types.
>>>>
>>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>>
>>>> Having that, please do not use class-side methods in ExternalType to
>>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>>> Please. We may want to add checks to enforce that.
>>>>
>>>> I suppose that this discussion focuses on type aliases that are
>>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>>> thus get their own struct type. So, let's ignore this other mechanism for
>>>> now.
>>>>
>>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>>> function expecting a Bar, even if they both alias int...
>>>>
>>>> Yes! So, we are now talking about type aliases to atomic types. The
>>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>>> this:
>>>>
>>>> (...forgive my "creative" representation of WordArray here...)
>>>>
>>>> compiledSpec: 0A 04 00 01
>>>> referentClass: Foo
>>>>
>>>> How does the atomic type for 'char' look like?
>>>>
>>>> compiledSpec: 0A 04 00 01
>>>> referentClass: nil
>>>>
>>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>>> compiledSpec does match.
>>>>
>>>> > But this has one major drawback: with current FFI, you cannot pass a
>>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>>> an integer type...
>>>>
>>>> That's correct. If the origin of such an argument is from within the
>>>> image, you have to wrap it. But when returned from another call ... well
>>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>>
>>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>>> simply return a Smalltalk Integer...
>>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>>> > This is not consistent.
>>>>
>>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>>> because it does so for argument coercing.
>>>>
>>>> Now I understand why the code generation of struct-field accessors was
>>>> so apparently broken for me. I changed that so that having an alias type as
>>>> part of a larger struct will automatically wrap that for you into the alias
>>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>>
>>>> It should be "Bar all the way down". Definitely.
>>>>
>>>> > If we return an int, we must accept an int, otherwise, we cannot
>>>> chain FFI calls...
>>>>
>>>> Well, not for type aliases. I would like keep the path of type safety
>>>> here, you mentioned above.
>>>>
>>>> Just fix the FFI plugin to respect referentClass when packaging the
>>>> return value.
>>>>
>>>> > I could use the lightweight alias instead, [...]
>>>>
>>>> Please don't. See above. Those "lightweight alias" are for renaming
>>>> atomic types only. Let's keep it simple and try to address you concern here
>>>> with struct types and ExternalTypeAlias. :-)
>>>>
>>>> > there was another usage where it was convenient to use
>>>> ExternalTypeAlias: enum.
>>>> > Indeed, I simply defined all enum symbols as class side method.
>>>>
>>>> I like that! It reads like a next step for ExternalPool. Once you have
>>>> extracted the constant values from header files, and once you have those
>>>> values for your platform in classVars in your external pool, use that pool
>>>> to define enums through ExternalStructure.
>>>>
>>>> Like this:
>>>>
>>>> ExternalTypeAlias subclass: #MyEnumBar
>>>> instanceVariableNames: ''
>>>> classVariableNames: ''
>>>> poolDictionaries: 'HDF5Pool'
>>>> category: 'HDF5'
>>>>
>>>> MyEnumBar class >> #poolFields
>>>> "
>>>> self defineFields.
>>>> "
>>>> ^ #(
>>>> (beg HDF5_BEG)
>>>> (mid HDF5_MID)
>>>> (end HDF5_END)
>>>> )
>>>>
>>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>>
>>>> Note that #poolFields would translate to class-side methods as you
>>>> suggested.
>>>>
>>>> > If I use a simpler alias, where are those ids going to?
>>>>
>>>> Please don't. See above. :-)
>>>>
>>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>>> inputs/outputs behave consistently.
>>>>
>>>> Absolutely!
>>>>
>>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>>> accept an immediate value of compatible type?
>>>>
>>>> Considering type safety, I think not.
>>>>
>>>> Well ... since this is a convenience issue and thus not about saving
>>>> run-time cost ... What about adding a callback into the image to react on
>>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>>> anyway?
>>>>
>>>>
>>>>
>>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>>
>>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>>> FFI-Callback. But ignore me here :)
>>>>
>>>> > - should a function returning an ExternalTypeAlias of atomic type
>>>> instantiate the Alias?
>>>>
>>>> Yes, please! See above.
>>>>
>>>> If you want to trade type safety in for a performance gain, just use a
>>>> "lightweight alias" as explained above. And package that return value
>>>> manually.
>>>>
>>>> Best,
>>>> Marcel
>>>>
>>>>
>>>>
>>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>>> [hidden email]>:
>>>>
>>>>
>>>> Hi all,
>>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>>
>>>> Type alias are subclasses of ExternalStructure.
>>>> I used them in HDF5 because there was no other means to define an alias
>>>> at the time.
>>>> Now we have another mean by adding a class side method in ExternalType
>>>> (I would have rather imagined the usage of some annotation to get a
>>>> declarative style)
>>>>
>>>> This has one advantage: type safety. You cannot pass a Foo to a
>>>> function expecting a Bar, even if they both alias int...
>>>>
>>>> But this has one major drawback: with current FFI, you cannot pass a
>>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>>> an integer type...
>>>>
>>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>>> Gasp, this is going too far...
>>>>
>>>> What about functions returning such alias?
>>>> Function returning struct by value allocate a struct, but it's not the
>>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>>> and simply return a Smalltalk Integer...
>>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>>> This is not consistent.
>>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>>> FFI calls...
>>>>
>>>> I could use the lightweight alias instead, but there was another usage
>>>> where it was convenient to use ExternalTypeAlias: enum.
>>>> Indeed, I simply defined all enum symbols as class side method. For
>>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>>> This way, I normally can pass a Bar mid to an external function.
>>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>>
>>>> If I use a simpler alias, where are those ids going to?
>>>>
>>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>>> inputs/outputs behave consistently.
>>>> The choice is this one:
>>>> - should a function expecting an ExternalTypeAlias of atomic type
>>>> accept an immediate value of compatible type? (the permissive
>>>> implementation)
>>>> - should a function returning an ExternalTypeAlias of atomic type
>>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>>> cost due to pressure on GC)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
Well, my mental model is this one:

ExternalData needs to refer to an area of memory, so as to be able to act as a pointer (single element or array does not really matter).
If this memory area lies in external heap, then we need an ExternalAddress as handle.
If this memory area lies in object memory, it must be a ByteArray.
We cannot handle pointers to object memory, because object memory is relocatable...
Well, unless we pin the object - that why we pass the ByteArray itself, not its address..

This is independent of how we interpret the contents.
IMO, it's more useful to avoid the pointer indirection in type, because we cannot handle double pointer currently...
Of course, the only pointers that the memory area could contain are pointers to external heap (or eventually pinned objects).

I'd like to have a 3rd possibility: pointing to a sub-region of a ByteArray (ByteArray + offset).
That may be very useful for emulating.
An example in Smallapack: pass a pointer to the imaginary part of a complex array.
Since real and imaginary parts are interleaved, just offsetting the pointer and using a stride of 2 elements does the trick when one wants to access real or imaginary part
(most lapack/blas routines are equipped to handle simple strides).
This is currently possible with ExternalAddress, but NOT when matrix data lies in ObjectMemory.
Another example would be if you wanted to implement an interpreter for a low level language... (emulate a processor, whatever).



Le ven. 19 juin 2020 à 18:04, Marcel Taeumel <[hidden email]> a écrit :
 

So, what about this:

ExternalData<@00F3A3EE, int>

I think it could mean that you want to interpret the very pointer address itself as an integer. That is, you do not want to follow to where it points to.

Then what about this:

ExternalData<@00F3A3EE, double>

Ha! An error, I suppose. Unless ... given 8-byte pointers, you can try to read a double out of it. :-D Just kidding. It should be an error.

.
.
.

> > (On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)
> Let me disagree here...

The examples I just wrote down in the previous messages let me realize why you did disagree. :-D

I hope this helps you with improving argument coercing and return value packaging in the FFI plugin. ^__^

Best,
Marcel

Am 19.06.2020 17:53:21 schrieb Marcel Taeumel <[hidden email]>:


then you would use int* and not int**

My bad: "  then you would use int** and not int*  " Sorry for the noise -.-"

Am 19.06.2020 17:52:21 schrieb Marcel Taeumel <[hidden email]>:


If you want to make a pointer array out of it, change the type to int**

Correction: If you happen to now, that this external address points to an array of pointers to int, then you would use int* and not int**. You cannot just "decide" what the data behind that external address should be. .. you know what I mean. ;-)

best,
Marcel

Am 19.06.2020 17:47:36 schrieb Marcel Taeumel <[hidden email]>:


Hi Nicolas.

Maybe one more thought on

ExternalData<handle, type>

There is no need to have

ExternalData<1312301580, int>

because it can just be 1312301580.

On the other hand:

ExternalData<#[ 12 34 56 78 ], int>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is already in Squeak's object memory.

Finally:

ExternalData<@00F3A3EE, int*>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is in external memory.

If you want to make an array out of it, make use of the "size" field in ExternalData.

If you want to make a pointer array out of it, change the type to int**

ExternalData<@00F3A3EE, int**>

So, what does this mean then:

ExternalData<#[ 12 34 56 78 ], int*>

Well, it tells you that there is a pointer to an int stored in a byte array. This comes very handy if you think about type aliases to pointer types such as "typedef int* INT_P". Because then it would be

ExternalData<#[ 12 34 56 78 ], INT_P>

To where this pointer points to? Probably garbage. I just made it up for this example. :-D

Best,
Marcel

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:







> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.



We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,
Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a

écrit :



>

> > Err, I move this thread to Opensmalltalk VM dev, because it is not

> Squeak specific!

>

> Absolutely. :-)

>

> (On a side note: I think that ExternalData should always have a pointer

> type in its "type" field unless "handle" is a ByteArray instead of

> ExternalAddress)

>

> Let me disagree here...

The type should better describe the contents of the handle (be it an

ExternalAddress or a zone of object memory (ByteArray)).

Typically, if you have an external (global) variable of type foo (extern

foo my_var;) then the natural type is foo.

Having foo*, would mean that we handle an array of foo*, or a pointer to

foo* (foo**).

This way, we make no difference between type surrogates

(ExternalStruct/ExternalTypeAlias) and other atomic cases.



(On another side note: I also think that "handle" in ExternalData should

> never ever be an atomic Smalltalk object (Integer or Float) but always

> either ByteArray or ExternalAddress. ... which makes it different from what

> "handle" can be in ExternalStructs that represent type aliases. :-)

>

> Yes!

It could be an ExternalAddress, or a zone of object memory (ByteArray) or

an Alien as we may want to support that too.

I'd be inclined to support doubleByte, word, and doubleWord arrays if the

size of atomic type matches

(Typically passing a DoubleByteArray, WordArray, a Float32Array, a

Float64Array to a short */int */float*/double*).



Best,

> Marcel

>

> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:

> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

> specific!

>

>

>

> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>

> [hidden email]> a écrit :

>

> > Hi Marcel,

> > thanks for your vote. Anyone else?

> >

> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> > écrit :

> >

> >> Hi Nicolas, hi all! :-)

> >>

> >> > Type alias are subclasses of ExternalStructure.

> >>

> >> And with it, they get their own instance of ExternalType, which is

> >> managed in the classVar StructTypes. Let's call them struct type.

> >>

> >> > Now we have another mean by adding a class side method in

> ExternalType.

> >>

> >> That kind of aliasing is for compile-time type referencing only -- such

> >> as in FFI calls (i.e. pragmas) or struct-field definitions

> >> (i.e. #fields). (And soon if I have the first version of

> >> FFI-Callback to show you :)

> >>

> >> Because those aliases become fully transparent after compiling the

> method

> >> for the FFI call into an instance of ExternalLibraryFunction and after

> >> compiling the field specs of external structures into compiledSpec for

> >> struct types.

> >>

> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a

> 'long',

> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And

> 'int32_t'

> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak

> FFI

> >> vocabulary. Nothing more. Nothing less.

> >>

> >> Having that, please do not use class-side methods in ExternalType to

> >> alias a (complex) struct type. Use them only for renaming atomic types.

> >> Please. We may want to add checks to enforce that.

> >>

> >> I suppose that this discussion focuses on type aliases that are

> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)

> and

> >> thus get their own struct type. So, let's ignore this other mechanism

> for

> >> now.

> >>

> >> > This has one advantage: type safety. You cannot pass a Foo to a

> >> function expecting a Bar, even if they both alias int...

> >>

> >> Yes! So, we are now talking about type aliases to atomic types. The

> >> struct type you get when aliasing a 'char' as Foo, for example, looks

> like

> >> this:

> >>

> >> (...forgive my "creative" representation of WordArray here...)

> >>

> >> compiledSpec: 0A 04 00 01

> >> referentClass: Foo

> >>

> >> How does the atomic type for 'char' look like?

> >>

> >> compiledSpec: 0A 04 00 01

> >> referentClass: nil

> >>

> >> Consequently, argument coercing will fail if you pass a 'char' when a

> >> 'Foo' is expected. Because the referentClass does not match --- even if

> the

> >> compiledSpec does match.

> >>

> >> > But this has one major drawback: with current FFI, you cannot pass a

> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias

> for

> >> an integer type...

> >>

> >> That's correct. If the origin of such an argument is from within the

> >> image, you have to wrap it. But when returned from another call ...

> well

> >> ... that wrapping should have happened in the plugin ... But read on!

> :-)

> >>

> >> > FFI plugin is clever enough to recognize an atomic type alias and

> >> simply return a Smalltalk Integer...

> >> > So it's not Bar all the way down, I have to manually wrap Bar...

> >> > This is not consistent.

> >>

> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

> >> because it does so for argument coercing.

> >>

> >> Now I understand why the code generation of struct-field accessors was

> so

> >> apparently broken for me. I changed that so that having an alias type

> as

> >> part of a larger struct will automatically wrap that for you into the

> alias

> >> structure. :-) Even if you are just aliasing a plain 'int' ... or

> 'long'

> >> ... Ha ha. ;-) #ffiLongVsInt.

> >>

> >> It should be "Bar all the way down". Definitely.

> >>

> >> > If we return an int, we must accept an int, otherwise, we cannot

> chain

> >> FFI calls...

> >>

> >> Well, not for type aliases. I would like keep the path of type safety

> >> here, you mentioned above.

> >>

> >> Just fix the FFI plugin to respect referentClass when packaging the

> >> return value.

> >>

> >> > I could use the lightweight alias instead, [...]

> >>

> >> Please don't. See above. Those "lightweight alias" are for renaming

> >> atomic types only. Let's keep it simple and try to address you concern

> here

> >> with struct types and ExternalTypeAlias. :-)

> >>

> >> > there was another usage where it was convenient to use

> >> ExternalTypeAlias: enum.

> >> > Indeed, I simply defined all enum symbols as class side method.

> >>

> >> I like that! It reads like a next step for ExternalPool. Once you have

> >> extracted the constant values from header files, and once you have

> those

> >> values for your platform in classVars in your external pool, use that

> pool

> >> to define enums through ExternalStructure.

> >>

> >> Like this:

> >>

> >> ExternalTypeAlias subclass: #MyEnumBar

> >> instanceVariableNames: ''

> >> classVariableNames: ''

> >> poolDictionaries: 'HDF5Pool'

> >> category: 'HDF5'

> >>

> >> MyEnumBar class >> #poolFields

> >> "

> >> self defineFields.

> >> "

> >> ^ #(

> >> (beg HDF5_BEG)

> >> (mid HDF5_MID)

> >> (end HDF5_END)

> >> )

> >>

> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

> >> external-pool definitions. See class comment in ExternalPool to get

> started.

> >>

> >> Note that #poolFields would translate to class-side methods as you

> >> suggested.

> >>

> >> > If I use a simpler alias, where are those ids going to?

> >>

> >> Please don't. See above. :-)

> >>

> >> > IMO we cannot keep status quo in the plugin, we gotta to make the

> >> inputs/outputs behave consistently.

> >>

> >> Absolutely!

> >>

> >> > - should a function expecting an ExternalTypeAlias of atomic type

> >> accept an immediate value of compatible type?

> >>

> >> Considering type safety, I think not.

> >>

> >> Well ... since this is a convenience issue and thus not about saving

> >> run-time cost ... What about adding a callback into the image to react

> on

> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

> >> stored in ExternalLibraryFunction since this is accessible to the

> plugin

> >> anyway?

> >>

> >>

> >>

> >> ... would it be possible? Like that #doesNotUnderstand: callback?

> >>

> >> I would love to do ad-hoc packaging of arguments. (Also thinking about

> >> FFI-Callback. But ignore me here :)

> >>

> >> > - should a function returning an ExternalTypeAlias of atomic type

> >> instantiate the Alias?

> >>

> >> Yes, please! See above.

> >>

> >> If you want to trade type safety in for a performance gain, just use a

> >> "lightweight alias" as explained above. And package that return value

> >> manually.

> >>

> >> Best,

> >> Marcel

> >>

> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>

> >> [hidden email]>:

> >> Hi all,

> >> following the question of Marcel about usage of ExternalTypeAlias:

> >>

> >> Type alias are subclasses of ExternalStructure.

> >> I used them in HDF5 because there was no other means to define an alias

> >> at the time.

> >> Now we have another mean by adding a class side method in ExternalType

> (I

> >> would have rather imagined the usage of some annotation to get a

> >> declarative style)

> >>

> >> This has one advantage: type safety. You cannot pass a Foo to a

> function

> >> expecting a Bar, even if they both alias int...

> >>

> >> But this has one major drawback: with current FFI, you cannot pass a

> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias

> for

> >> an integer type...

> >>

> >> Thus, you have to wrap every Bar argument into a Bar instance...

> >> Gasp, this is going too far...

> >>

> >> What about functions returning such alias?

> >> Function returning struct by value allocate a struct, but it's not the

> >> case here, FFI plugin is clever enough to recognize an atomic type

> alias

> >> and simply return a Smalltalk Integer...

> >> So it's not Bar all the way down, I have to manually wrap Bar...

> >> This is not consistent.

> >> If we return an int, we must accept an int, otherwise, we cannot chain

> >> FFI calls...

> >>

> >> I could use the lightweight alias instead, but there was another usage

> >> where it was convenient to use ExternalTypeAlias: enum.

> >> Indeed, I simply defined all enum symbols as class side method. For

> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type

> Bar

> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

> >> This way, I normally can pass a Bar mid to an external function.

> >> (currently, we must defne mid ^Bar new value: 1; yourself)

> >>

> >> If I use a simpler alias, where are those ids going to?

> >>

> >> IMO we cannot keep statu quo in the plugin, we gotta to make the

> >> inputs/outputs behave consistently.

> >> The choice is this one:

> >> - should a function expecting an ExternalTypeAlias of atomic type

> accept

> >> an immediate value of compatible type? (the permissive implementation)

> >> - should a function returning an ExternalTypeAlias of atomic type

> >> instantiate the Alias? (the typesafe implementation, with higher

> runtime

> >> cost due to pressure on GC)

> >>

> >>

> >>

> >>

> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

> specific!

>

>

>

> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :

>

>> Hi Marcel,

>> thanks for your vote. Anyone else?

>>

>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

>> écrit :

>>

>>>

>>> Hi Nicolas, hi all! :-)

>>>

>>> > Type alias are subclasses of ExternalStructure.

>>>

>>> And with it, they get their own instance of ExternalType, which is

>>> managed in the classVar StructTypes. Let's call them struct type.

>>>

>>> > Now we have another mean by adding a class side method in ExternalType.

>>>

>>> That kind of aliasing is for compile-time type referencing only -- such

>>> as in FFI calls (i.e. pragmas) or struct-field definitions

>>> (i.e. #fields). (And soon if I have the first version of

>>> FFI-Callback to show you :)

>>>

>>> Because those aliases become fully transparent after compiling the

>>> method for the FFI call into an instance of ExternalLibraryFunction and

>>> after compiling the field specs of external structures into compiledSpec

>>> for struct types.

>>>

>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a

>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And

>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into

>>> Squeak FFI vocabulary. Nothing more. Nothing less.

>>>

>>> Having that, please do not use class-side methods in ExternalType to

>>> alias a (complex) struct type. Use them only for renaming atomic types.

>>> Please. We may want to add checks to enforce that.

>>>

>>> I suppose that this discussion focuses on type aliases that are

>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>>> thus get their own struct type. So, let's ignore this other mechanism for

>>> now.

>>>

>>> > This has one advantage: type safety. You cannot pass a Foo to a

>>> function expecting a Bar, even if they both alias int...

>>>

>>> Yes! So, we are now talking about type aliases to atomic types. The

>>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>>> this:

>>>

>>> (...forgive my "creative" representation of WordArray here...)

>>>

>>> compiledSpec: 0A 04 00 01

>>> referentClass: Foo

>>>

>>> How does the atomic type for 'char' look like?

>>>

>>> compiledSpec: 0A 04 00 01

>>> referentClass: nil

>>>

>>> Consequently, argument coercing will fail if you pass a 'char' when a

>>> 'Foo' is expected. Because the referentClass does not match --- even if the

>>> compiledSpec does match.

>>>

>>> > But this has one major drawback: with current FFI, you cannot pass a

>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>>> an integer type...

>>>

>>> That's correct. If the origin of such an argument is from within the

>>> image, you have to wrap it. But when returned from another call ... well

>>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>>

>>> > FFI plugin is clever enough to recognize an atomic type alias and

>>> simply return a Smalltalk Integer...

>>> > So it's not Bar all the way down, I have to manually wrap Bar...

>>> > This is not consistent.

>>>

>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>>> because it does so for argument coercing.

>>>

>>> Now I understand why the code generation of struct-field accessors was

>>> so apparently broken for me. I changed that so that having an alias type as

>>> part of a larger struct will automatically wrap that for you into the alias

>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>>> ... Ha ha. ;-) #ffiLongVsInt.

>>>

>>> It should be "Bar all the way down". Definitely.

>>>

>>> > If we return an int, we must accept an int, otherwise, we cannot chain

>>> FFI calls...

>>>

>>> Well, not for type aliases. I would like keep the path of type safety

>>> here, you mentioned above.

>>>

>>> Just fix the FFI plugin to respect referentClass when packaging the

>>> return value.

>>>

>>> > I could use the lightweight alias instead, [...]

>>>

>>> Please don't. See above. Those "lightweight alias" are for renaming

>>> atomic types only. Let's keep it simple and try to address you concern here

>>> with struct types and ExternalTypeAlias. :-)

>>>

>>> > there was another usage where it was convenient to use

>>> ExternalTypeAlias: enum.

>>> > Indeed, I simply defined all enum symbols as class side method.

>>>

>>> I like that! It reads like a next step for ExternalPool. Once you have

>>> extracted the constant values from header files, and once you have those

>>> values for your platform in classVars in your external pool, use that pool

>>> to define enums through ExternalStructure.

>>>

>>> Like this:

>>>

>>> ExternalTypeAlias subclass: #MyEnumBar

>>> instanceVariableNames: ''

>>> classVariableNames: ''

>>> poolDictionaries: 'HDF5Pool'

>>> category: 'HDF5'

>>>

>>> MyEnumBar class >> #poolFields

>>> "

>>> self defineFields.

>>> "

>>> ^ #(

>>> (beg HDF5_BEG)

>>> (mid HDF5_MID)

>>> (end HDF5_END)

>>> )

>>>

>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>>> external-pool definitions. See class comment in ExternalPool to get started.

>>>

>>> Note that #poolFields would translate to class-side methods as you

>>> suggested.

>>>

>>> > If I use a simpler alias, where are those ids going to?

>>>

>>> Please don't. See above. :-)

>>>

>>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>>> inputs/outputs behave consistently.

>>>

>>> Absolutely!

>>>

>>> > - should a function expecting an ExternalTypeAlias of atomic type

>>> accept an immediate value of compatible type?

>>>

>>> Considering type safety, I think not.

>>>

>>> Well ... since this is a convenience issue and thus not about saving

>>> run-time cost ... What about adding a callback into the image to react on

>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>>> stored in ExternalLibraryFunction since this is accessible to the plugin

>>> anyway?

>>>

>>>

>>>

>>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>>

>>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>>> FFI-Callback. But ignore me here :)

>>>

>>> > - should a function returning an ExternalTypeAlias of atomic type

>>> instantiate the Alias?

>>>

>>> Yes, please! See above.

>>>

>>> If you want to trade type safety in for a performance gain, just use a

>>> "lightweight alias" as explained above. And package that return value

>>> manually.

>>>

>>> Best,

>>> Marcel

>>>

>>>

>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:

>>>

>>> Hi all,

>>> following the question of Marcel about usage of ExternalTypeAlias:

>>>

>>> Type alias are subclasses of ExternalStructure.

>>> I used them in HDF5 because there was no other means to define an alias

>>> at the time.

>>> Now we have another mean by adding a class side method in ExternalType

>>> (I would have rather imagined the usage of some annotation to get a

>>> declarative style)

>>>

>>> This has one advantage: type safety. You cannot pass a Foo to a function

>>> expecting a Bar, even if they both alias int...

>>>

>>> But this has one major drawback: with current FFI, you cannot pass a

>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>>> an integer type...

>>>

>>> Thus, you have to wrap every Bar argument into a Bar instance...

>>> Gasp, this is going too far...

>>>

>>> What about functions returning such alias?

>>> Function returning struct by value allocate a struct, but it's not the

>>> case here, FFI plugin is clever enough to recognize an atomic type alias

>>> and simply return a Smalltalk Integer...

>>> So it's not Bar all the way down, I have to manually wrap Bar...

>>> This is not consistent.

>>> If we return an int, we must accept an int, otherwise, we cannot chain

>>> FFI calls...

>>>

>>> I could use the lightweight alias instead, but there was another usage

>>> where it was convenient to use ExternalTypeAlias: enum.

>>> Indeed, I simply defined all enum symbols as class side method. For

>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>>> This way, I normally can pass a Bar mid to an external function.

>>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>>

>>> If I use a simpler alias, where are those ids going to?

>>>

>>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>>> inputs/outputs behave consistently.

>>> The choice is this one:

>>> - should a function expecting an ExternalTypeAlias of atomic type accept

>>> an immediate value of compatible type? (the permissive implementation)

>>> - should a function returning an ExternalTypeAlias of atomic type

>>> instantiate the Alias? (the typesafe implementation, with higher runtime

>>> cost due to pressure on GC)

>>>

>>>

>>>

>>>

>>>

>>>

>>>

>>

>



Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :
 












Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...
The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).
This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,
Marcel






Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak


specific!











Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :





> Hi Marcel,


> thanks for your vote. Anyone else?


>


> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a


> écrit :


>


>> Hi Nicolas, hi all! :-)


>>


>> > Type alias are subclasses of ExternalStructure.


>>


>> And with it, they get their own instance of ExternalType, which is


>> managed in the classVar StructTypes. Let's call them struct type.


>>


>> > Now we have another mean by adding a class side method in ExternalType.


>>


>> That kind of aliasing is for compile-time type referencing only -- such


>> as in FFI calls (i.e. pragmas) or struct-field definitions


>> (i.e. #fields). (And soon if I have the first version of


>> FFI-Callback to show you :)


>>


>> Because those aliases become fully transparent after compiling the method


>> for the FFI call into an instance of ExternalLibraryFunction and after


>> compiling the field specs of external structures into compiledSpec for


>> struct types.


>>


>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',


>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'


>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI


>> vocabulary. Nothing more. Nothing less.


>>


>> Having that, please do not use class-side methods in ExternalType to


>> alias a (complex) struct type. Use them only for renaming atomic types.


>> Please. We may want to add checks to enforce that.


>>


>> I suppose that this discussion focuses on type aliases that are


>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and


>> thus get their own struct type. So, let's ignore this other mechanism for


>> now.


>>


>> > This has one advantage: type safety. You cannot pass a Foo to a


>> function expecting a Bar, even if they both alias int...


>>


>> Yes! So, we are now talking about type aliases to atomic types. The


>> struct type you get when aliasing a 'char' as Foo, for example, looks like


>> this:


>>


>> (...forgive my "creative" representation of WordArray here...)


>>


>> compiledSpec: 0A 04 00 01


>> referentClass: Foo


>>


>> How does the atomic type for 'char' look like?


>>


>> compiledSpec: 0A 04 00 01


>> referentClass: nil


>>


>> Consequently, argument coercing will fail if you pass a 'char' when a


>> 'Foo' is expected. Because the referentClass does not match --- even if the


>> compiledSpec does match.


>>


>> > But this has one major drawback: with current FFI, you cannot pass a


>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for


>> an integer type...


>>


>> That's correct. If the origin of such an argument is from within the


>> image, you have to wrap it. But when returned from another call ... well


>> ... that wrapping should have happened in the plugin ... But read on! :-)


>>


>> > FFI plugin is clever enough to recognize an atomic type alias and


>> simply return a Smalltalk Integer...


>> > So it's not Bar all the way down, I have to manually wrap Bar...


>> > This is not consistent.


>>


>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass


>> because it does so for argument coercing.


>>


>> Now I understand why the code generation of struct-field accessors was so


>> apparently broken for me. I changed that so that having an alias type as


>> part of a larger struct will automatically wrap that for you into the alias


>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'


>> ... Ha ha. ;-) #ffiLongVsInt.


>>


>> It should be "Bar all the way down". Definitely.


>>


>> > If we return an int, we must accept an int, otherwise, we cannot chain


>> FFI calls...


>>


>> Well, not for type aliases. I would like keep the path of type safety


>> here, you mentioned above.


>>


>> Just fix the FFI plugin to respect referentClass when packaging the


>> return value.


>>


>> > I could use the lightweight alias instead, [...]


>>


>> Please don't. See above. Those "lightweight alias" are for renaming


>> atomic types only. Let's keep it simple and try to address you concern here


>> with struct types and ExternalTypeAlias. :-)


>>


>> > there was another usage where it was convenient to use


>> ExternalTypeAlias: enum.


>> > Indeed, I simply defined all enum symbols as class side method.


>>


>> I like that! It reads like a next step for ExternalPool. Once you have


>> extracted the constant values from header files, and once you have those


>> values for your platform in classVars in your external pool, use that pool


>> to define enums through ExternalStructure.


>>


>> Like this:


>>


>> ExternalTypeAlias subclass: #MyEnumBar


>> instanceVariableNames: ''


>> classVariableNames: ''


>> poolDictionaries: 'HDF5Pool'


>> category: 'HDF5'


>>


>> MyEnumBar class >> #poolFields


>> "


>> self defineFields.


>> "


>> ^ #(


>> (beg HDF5_BEG)


>> (mid HDF5_MID)


>> (end HDF5_END)


>> )


>>


>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via


>> external-pool definitions. See class comment in ExternalPool to get started.


>>


>> Note that #poolFields would translate to class-side methods as you


>> suggested.


>>


>> > If I use a simpler alias, where are those ids going to?


>>


>> Please don't. See above. :-)


>>


>> > IMO we cannot keep status quo in the plugin, we gotta to make the


>> inputs/outputs behave consistently.


>>


>> Absolutely!


>>


>> > - should a function expecting an ExternalTypeAlias of atomic type


>> accept an immediate value of compatible type?


>>


>> Considering type safety, I think not.


>>


>> Well ... since this is a convenience issue and thus not about saving


>> run-time cost ... What about adding a callback into the image to react on


>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be


>> stored in ExternalLibraryFunction since this is accessible to the plugin


>> anyway?


>>


>>


>>


>> ... would it be possible? Like that #doesNotUnderstand: callback?


>>


>> I would love to do ad-hoc packaging of arguments. (Also thinking about


>> FFI-Callback. But ignore me here :)


>>


>> > - should a function returning an ExternalTypeAlias of atomic type


>> instantiate the Alias?


>>


>> Yes, please! See above.


>>


>> If you want to trade type safety in for a performance gain, just use a


>> "lightweight alias" as explained above. And package that return value


>> manually.


>>


>> Best,


>> Marcel


>>


>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:


>> Hi all,


>> following the question of Marcel about usage of ExternalTypeAlias:


>>


>> Type alias are subclasses of ExternalStructure.


>> I used them in HDF5 because there was no other means to define an alias


>> at the time.


>> Now we have another mean by adding a class side method in ExternalType (I


>> would have rather imagined the usage of some annotation to get a


>> declarative style)


>>


>> This has one advantage: type safety. You cannot pass a Foo to a function


>> expecting a Bar, even if they both alias int...


>>


>> But this has one major drawback: with current FFI, you cannot pass a


>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for


>> an integer type...


>>


>> Thus, you have to wrap every Bar argument into a Bar instance...


>> Gasp, this is going too far...


>>


>> What about functions returning such alias?


>> Function returning struct by value allocate a struct, but it's not the


>> case here, FFI plugin is clever enough to recognize an atomic type alias


>> and simply return a Smalltalk Integer...


>> So it's not Bar all the way down, I have to manually wrap Bar...


>> This is not consistent.


>> If we return an int, we must accept an int, otherwise, we cannot chain


>> FFI calls...


>>


>> I could use the lightweight alias instead, but there was another usage


>> where it was convenient to use ExternalTypeAlias: enum.


>> Indeed, I simply defined all enum symbols as class side method. For


>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar


>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.


>> This way, I normally can pass a Bar mid to an external function.


>> (currently, we must defne mid ^Bar new value: 1; yourself)


>>


>> If I use a simpler alias, where are those ids going to?


>>


>> IMO we cannot keep statu quo in the plugin, we gotta to make the


>> inputs/outputs behave consistently.


>> The choice is this one:


>> - should a function expecting an ExternalTypeAlias of atomic type accept


>> an immediate value of compatible type? (the permissive implementation)


>> - should a function returning an ExternalTypeAlias of atomic type


>> instantiate the Alias? (the typesafe implementation, with higher runtime


>> cost due to pressure on GC)


>>


>>


>>


>>


Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :



Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel



Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:




Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)

























Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

marcel.taeumel
 
Btw: Type aliases to pointer types can already be handled through their referent sub-instances of ExternalTypeAlias. I did already put #doesNotUnderstand: there to forward the calls to #value.

INT_P class >> #originalTypeName
   ^ 'int*'

...
data := ExternalData
   fromHandle: #[ 12 13 14 15 ]
   type: INT_P externalType.
...
myIntP := data asExternalStructure. "myIntP class == INT_P."
anInt := myIntP at: 1. "anInt isKindOf: Integer"

Maybe also take a look at ExternalStructure >> #asExternalData.

You would only lose track of the type INT_P if you do this:

myIntP value asExternalData.

After that you have int* and cannot go back to that ExternalTypeAlias:

myIntP value "OK" asExternalData "OK" asExternalStructure "ERROR"

For aliases to struct pointers you will also lose track of the alias

myStructP value "OK" asExternalData "OK" asExternalStructure "OK but different class"

Best,
Marcel

Am 19.06.2020 19:01:50 schrieb Marcel Taeumel <[hidden email]>:

I'd like to have a 3rd possibility: pointing to a sub-region of a ByteArray
> (ByteArray + offset)

For this, you could add "offset" to ExternalData (in addition to "type" and "size"). This would also postpone the expensive pointer arithmetic implemented in ExternalAddress >> #+ to the last possible moment. :-) 

Best,
Marcel

Am 19.06.2020 18:52:02 schrieb Nicolas Cellier <[hidden email]>:

Well, my mental model is this one:

ExternalData needs to refer to an area of memory, so as to be able to act
as a pointer (single element or array does not really matter).
If this memory area lies in external heap, then we need an ExternalAddress
as handle.
If this memory area lies in object memory, it must be a ByteArray.
We cannot handle pointers to object memory, because object memory is
relocatable...
Well, unless we pin the object - that why we pass the ByteArray itself, not
its address..

This is independent of how we interpret the contents.
IMO, it's more useful to avoid the pointer indirection in type, because we
cannot handle double pointer currently...
Of course, the only pointers that the memory area could contain are
pointers to external heap (or eventually pinned objects).

I'd like to have a 3rd possibility: pointing to a sub-region of a ByteArray
(ByteArray + offset).
That may be very useful for emulating.
An example in Smallapack: pass a pointer to the imaginary part of a complex
array.
Since real and imaginary parts are interleaved, just offsetting the pointer
and using a stride of 2 elements does the trick when one wants to access
real or imaginary part
(most lapack/blas routines are equipped to handle simple strides).
This is currently possible with ExternalAddress, but NOT when matrix data
lies in ObjectMemory.
Another example would be if you wanted to implement an interpreter for a
low level language... (emulate a processor, whatever).



Le ven. 19 juin 2020 à 18:04, Marcel Taeumel a
écrit :

>
> So, what about this:
>
> ExternalData<@00f3a3ee, int="">
>
> I think it could mean that you want to interpret the very pointer address
> itself as an integer. That is, you do not want to follow to where it points
> to.
>
> Then what about this:
>
> ExternalData<@00f3a3ee, double="">
>
> Ha! An error, I suppose. Unless ... given 8-byte pointers, you can try to
> read a double out of it. :-D Just kidding. It should be an error.
>
> .
> .
> .
>
> > > (On a side note: I think that ExternalData should always have a
> pointer type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
> > Let me disagree here...
>
> The examples I just wrote down in the previous messages let me realize why
> you did disagree. :-D
>
> I hope this helps you with improving argument coercing and return value
> packaging in the FFI plugin. ^__^
>
> Best,
> Marcel
>
> Am 19.06.2020 17:53:21 schrieb Marcel Taeumel :
> > then you would use int* and not int**
>
> My bad: " then you would use int** and not int* " Sorry for the noise
> -.-"
>
> Am 19.06.2020 17:52:21 schrieb Marcel Taeumel :
> > If you want to make a pointer array out of it, change the type to int**
>
> Correction: If you happen to now, that this external address points to an
> array of pointers to int, then you would use int* and not int**. You cannot
> just "decide" what the data behind that external address should be. .. you
> know what I mean. ;-)
>
> best,
> Marcel
>
> Am 19.06.2020 17:47:36 schrieb Marcel Taeumel :
> Hi Nicolas.
>
> Maybe one more thought on
>
> ExternalData
>
> There is no need to have
>
> ExternalData<1312301580, int="">
>
> because it can just be 1312301580.
>
> On the other hand:
>
> ExternalData<#[ 12="" 34="" 56="" 78="" ],="" int="">
>
> Tells you that (1) this data wants to be interpreted as signed int and (2)
> this data is already in Squeak's object memory.
>
> Finally:
>
> ExternalData<@00f3a3ee, int*="">
>
> Tells you that (1) this data wants to be interpreted as signed int and (2)
> this data is in external memory.
>
> If you want to make an array out of it, make use of the "size" field in
> ExternalData.
>
> If you want to make a pointer array out of it, change the type to int**
>
> ExternalData<@00f3a3ee, int**="">
>
> So, what does this mean then:
>
> ExternalData<#[ 12="" 34="" 56="" 78="" ],="" int*="">
>
> Well, it tells you that there is a pointer to an int stored in a byte
> array. This comes very handy if you think about type aliases to pointer
> types such as "typedef int* INT_P". Because then it would be
>
> ExternalData<#[ 12="" 34="" 56="" 78="" ],="" int_p="">
>
> To where this pointer points to? Probably garbage. I just made it up for
> this example. :-D
>
> Best,
> Marcel
>
> Am 19.06.2020 17:15:23 schrieb Marcel Taeumel :
> > The type should better describe the contents of the handle (be it an
> > ExternalAddress or a zone of object memory (ByteArray)).
> > Typically, if you have an external (global) variable of type foo (extern
> > foo my_var;) then the natural type is foo.
> > Having foo*, would mean that we handle an array of foo*, or a pointer to
> > foo* (foo**).
> > This way, we make no difference between type surrogates
> > (ExternalStruct/ExternalTypeAlias) and other atomic cases.
>
> We have the "size" field in ExternalData encode whether "int*" points to a
> single or multiple things. No need to treat a global variable sitting in
> external memory any different here. ... IMO :-)
>
> I know, that this is my personal mental model for Squeak FFI to treat
> pointer types as things in external memory and non-pointer types as things
> in Squeak's object memory. The latter includes ByteArray and Integer and
> Float and also IntegerArray etc. Just not ExternalAddress.
>
> This basically comes from the fact that, until now, the FFI plugin gave me
> a ByteArray in return when I told it to be "Foo", not "Foo*". So,
> non-pointer type means ByteArray. Pointer-type means ExternalAddress.
>
> Type aliases are not really different here. Each alias as a corresponding
> struct type, which has again a non-pointer variant and a pointer variant.
>
> Type aliases to pointer types are just special because they store the
> pointer address in a byte array. The corresponding non-pointer type will do
> fine for such aliases. Their pointer type is not relevant in general, I
> suppose.
>
> > Having foo*, would mean that we handle an array of foo*, or a pointer to foo*
> (foo**).
>
> Well, I have a simple idea on how to represent foo** (and other
> dimensions) without the FFI plugin needing to know. ... unless you are
> concerned about type safety for that, too. Then we would really need to
> encode that in the compiledSpec.
>
> If not, ExternalData with foo** type will just answer ExternalData with
> foo* when asked via "at: 1" for example. :-)
>
> > or an Alien as we may want to support that too.
>
> I don't think that's necessary. Alien has its own call-out mechanism
> through the IA32ABI plugin and also a different layout/usage of ByteArray.
>
> Best,
> Marcel
>
> Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <>
> [hidden email]>:
> Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
> écrit :
>
> >
> > > Err, I move this thread to Opensmalltalk VM dev, because it is not
> > Squeak specific!
> >
> > Absolutely. :-)
> >
> > (On a side note: I think that ExternalData should always have a pointer
> > type in its "type" field unless "handle" is a ByteArray instead of
> > ExternalAddress)
> >
> > Let me disagree here...
> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.
>
> (On another side note: I also think that "handle" in ExternalData should
> > never ever be an atomic Smalltalk object (Integer or Float) but always
> > either ByteArray or ExternalAddress. ... which makes it different from
> what
> > "handle" can be in ExternalStructs that represent type aliases. :-)
> >
> > Yes!
> It could be an ExternalAddress, or a zone of object memory (ByteArray) or
> an Alien as we may want to support that too.
> I'd be inclined to support doubleByte, word, and doubleWord arrays if the
> size of atomic type matches
> (Typically passing a DoubleByteArray, WordArray, a Float32Array, a
> Float64Array to a short */int */float*/double*).
>
> Best,
> > Marcel
> >
> > Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> > [hidden email]>:
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak
> > specific!
> >
> >
> >
> > Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> > [hidden email]> a écrit :
> >
> > > Hi Marcel,
> > > thanks for your vote. Anyone else?
> > >
> > > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > > écrit :
> > >
> > >> Hi Nicolas, hi all! :-)
> > >>
> > >> > Type alias are subclasses of ExternalStructure.
> > >>
> > >> And with it, they get their own instance of ExternalType, which is
> > >> managed in the classVar StructTypes. Let's call them struct type.
> > >>
> > >> > Now we have another mean by adding a class side method in
> > ExternalType.
> > >>
> > >> That kind of aliasing is for compile-time type referencing only --
> such
> > >> as in FFI calls (i.e. pragmas) or struct-field definitions
> > >> (i.e. #fields). (And soon if I have the first version of
> > >> FFI-Callback to show you :)
> > >>
> > >> Because those aliases become fully transparent after compiling the
> > method
> > >> for the FFI call into an instance of ExternalLibraryFunction and
> after
> > >> compiling the field specs of external structures into compiledSpec
> for
> > >> struct types.
> > >>
> > >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> > 'long',
> > >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> > 'int32_t'
> > >> also becomes a 'long'. So, it just translates C vocabulary into
> Squeak
> > FFI
> > >> vocabulary. Nothing more. Nothing less.
> > >>
> > >> Having that, please do not use class-side methods in ExternalType to
> > >> alias a (complex) struct type. Use them only for renaming atomic
> types.
> > >> Please. We may want to add checks to enforce that.
> > >>
> > >> I suppose that this discussion focuses on type aliases that are
> > >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> > and
> > >> thus get their own struct type. So, let's ignore this other mechanism
> > for
> > >> now.
> > >>
> > >> > This has one advantage: type safety. You cannot pass a Foo to a
> > >> function expecting a Bar, even if they both alias int...
> > >>
> > >> Yes! So, we are now talking about type aliases to atomic types. The
> > >> struct type you get when aliasing a 'char' as Foo, for example, looks
> > like
> > >> this:
> > >>
> > >> (...forgive my "creative" representation of WordArray here...)
> > >>
> > >> compiledSpec: 0A 04 00 01
> > >> referentClass: Foo
> > >>
> > >> How does the atomic type for 'char' look like?
> > >>
> > >> compiledSpec: 0A 04 00 01
> > >> referentClass: nil
> > >>
> > >> Consequently, argument coercing will fail if you pass a 'char' when a
> > >> 'Foo' is expected. Because the referentClass does not match --- even
> if
> > the
> > >> compiledSpec does match.
> > >>
> > >> > But this has one major drawback: with current FFI, you cannot pass
> a
> > >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias
> > for
> > >> an integer type...
> > >>
> > >> That's correct. If the origin of such an argument is from within the
> > >> image, you have to wrap it. But when returned from another call ...
> > well
> > >> ... that wrapping should have happened in the plugin ... But read on!
> > :-)
> > >>
> > >> > FFI plugin is clever enough to recognize an atomic type alias and
> > >> simply return a Smalltalk Integer...
> > >> > So it's not Bar all the way down, I have to manually wrap Bar...
> > >> > This is not consistent.
> > >>
> > >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> > >> because it does so for argument coercing.
> > >>
> > >> Now I understand why the code generation of struct-field accessors
> was
> > so
> > >> apparently broken for me. I changed that so that having an alias type
> > as
> > >> part of a larger struct will automatically wrap that for you into the
> > alias
> > >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> > 'long'
> > >> ... Ha ha. ;-) #ffiLongVsInt.
> > >>
> > >> It should be "Bar all the way down". Definitely.
> > >>
> > >> > If we return an int, we must accept an int, otherwise, we cannot
> > chain
> > >> FFI calls...
> > >>
> > >> Well, not for type aliases. I would like keep the path of type safety
> > >> here, you mentioned above.
> > >>
> > >> Just fix the FFI plugin to respect referentClass when packaging the
> > >> return value.
> > >>
> > >> > I could use the lightweight alias instead, [...]
> > >>
> > >> Please don't. See above. Those "lightweight alias" are for renaming
> > >> atomic types only. Let's keep it simple and try to address you
> concern
> > here
> > >> with struct types and ExternalTypeAlias. :-)
> > >>
> > >> > there was another usage where it was convenient to use
> > >> ExternalTypeAlias: enum.
> > >> > Indeed, I simply defined all enum symbols as class side method.
> > >>
> > >> I like that! It reads like a next step for ExternalPool. Once you
> have
> > >> extracted the constant values from header files, and once you have
> > those
> > >> values for your platform in classVars in your external pool, use that
> > pool
> > >> to define enums through ExternalStructure.
> > >>
> > >> Like this:
> > >>
> > >> ExternalTypeAlias subclass: #MyEnumBar
> > >> instanceVariableNames: ''
> > >> classVariableNames: ''
> > >> poolDictionaries: 'HDF5Pool'
> > >> category: 'HDF5'
> > >>
> > >> MyEnumBar class >> #poolFields
> > >> "
> > >> self defineFields.
> > >> "
> > >> ^ #(
> > >> (beg HDF5_BEG)
> > >> (mid HDF5_MID)
> > >> (end HDF5_END)
> > >> )
> > >>
> > >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> > >> external-pool definitions. See class comment in ExternalPool to get
> > started.
> > >>
> > >> Note that #poolFields would translate to class-side methods as you
> > >> suggested.
> > >>
> > >> > If I use a simpler alias, where are those ids going to?
> > >>
> > >> Please don't. See above. :-)
> > >>
> > >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> > >> inputs/outputs behave consistently.
> > >>
> > >> Absolutely!
> > >>
> > >> > - should a function expecting an ExternalTypeAlias of atomic type
> > >> accept an immediate value of compatible type?
> > >>
> > >> Considering type safety, I think not.
> > >>
> > >> Well ... since this is a convenience issue and thus not about saving
> > >> run-time cost ... What about adding a callback into the image to
> react
> > on
> > >> FFIErrorCoercionFailed? Maybe the selector for such a callback could
> be
> > >> stored in ExternalLibraryFunction since this is accessible to the
> > plugin
> > >> anyway?
> > >>
> > >>
> > >>
> > >> ... would it be possible? Like that #doesNotUnderstand: callback?
> > >>
> > >> I would love to do ad-hoc packaging of arguments. (Also thinking
> about
> > >> FFI-Callback. But ignore me here :)
> > >>
> > >> > - should a function returning an ExternalTypeAlias of atomic type
> > >> instantiate the Alias?
> > >>
> > >> Yes, please! See above.
> > >>
> > >> If you want to trade type safety in for a performance gain, just use
> a
> > >> "lightweight alias" as explained above. And package that return value
> > >> manually.
> > >>
> > >> Best,
> > >> Marcel
> > >>
> > >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> > >> [hidden email]>:
> > >> Hi all,
> > >> following the question of Marcel about usage of ExternalTypeAlias:
> > >>
> > >> Type alias are subclasses of ExternalStructure.
> > >> I used them in HDF5 because there was no other means to define an
> alias
> > >> at the time.
> > >> Now we have another mean by adding a class side method in
> ExternalType
> > (I
> > >> would have rather imagined the usage of some annotation to get a
> > >> declarative style)
> > >>
> > >> This has one advantage: type safety. You cannot pass a Foo to a
> > function
> > >> expecting a Bar, even if they both alias int...
> > >>
> > >> But this has one major drawback: with current FFI, you cannot pass a
> > >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias
> > for
> > >> an integer type...
> > >>
> > >> Thus, you have to wrap every Bar argument into a Bar instance...
> > >> Gasp, this is going too far...
> > >>
> > >> What about functions returning such alias?
> > >> Function returning struct by value allocate a struct, but it's not
> the
> > >> case here, FFI plugin is clever enough to recognize an atomic type
> > alias
> > >> and simply return a Smalltalk Integer...
> > >> So it's not Bar all the way down, I have to manually wrap Bar...
> > >> This is not consistent.
> > >> If we return an int, we must accept an int, otherwise, we cannot
> chain
> > >> FFI calls...
> > >>
> > >> I could use the lightweight alias instead, but there was another
> usage
> > >> where it was convenient to use ExternalTypeAlias: enum.
> > >> Indeed, I simply defined all enum symbols as class side method. For
> > >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> > Bar
> > >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> > >> This way, I normally can pass a Bar mid to an external function.
> > >> (currently, we must defne mid ^Bar new value: 1; yourself)
> > >>
> > >> If I use a simpler alias, where are those ids going to?
> > >>
> > >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> > >> inputs/outputs behave consistently.
> > >> The choice is this one:
> > >> - should a function expecting an ExternalTypeAlias of atomic type
> > accept
> > >> an immediate value of compatible type? (the permissive
> implementation)
> > >> - should a function returning an ExternalTypeAlias of atomic type
> > >> instantiate the Alias? (the typesafe implementation, with higher
> > runtime
> > >> cost due to pressure on GC)
> > >>
> > >>
> > >>
> > >>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak
> > specific!
> >
> >
> >
> > Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> > [hidden email]> a écrit :
> >
> >> Hi Marcel,
> >> thanks for your vote. Anyone else?
> >>
> >> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> >> écrit :
> >>
> >>>
> >>> Hi Nicolas, hi all! :-)
> >>>
> >>> > Type alias are subclasses of ExternalStructure.
> >>>
> >>> And with it, they get their own instance of ExternalType, which is
> >>> managed in the classVar StructTypes. Let's call them struct type.
> >>>
> >>> > Now we have another mean by adding a class side method in
> ExternalType.
> >>>
> >>> That kind of aliasing is for compile-time type referencing only --
> such
> >>> as in FFI calls (i.e. pragmas) or struct-field definitions
> >>> (i.e. #fields). (And soon if I have the first version of
> >>> FFI-Callback to show you :)
> >>>
> >>> Because those aliases become fully transparent after compiling the
> >>> method for the FFI call into an instance of ExternalLibraryFunction
> and
> >>> after compiling the field specs of external structures into
> compiledSpec
> >>> for struct types.
> >>>
> >>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> >>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> >>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary
> into
> >>> Squeak FFI vocabulary. Nothing more. Nothing less.
> >>>
> >>> Having that, please do not use class-side methods in ExternalType to
> >>> alias a (complex) struct type. Use them only for renaming atomic
> types.
> >>> Please. We may want to add checks to enforce that.
> >>>
> >>> I suppose that this discussion focuses on type aliases that are
> >>> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >>> thus get their own struct type. So, let's ignore this other mechanism
> for
> >>> now.
> >>>
> >>> > This has one advantage: type safety. You cannot pass a Foo to a
> >>> function expecting a Bar, even if they both alias int...
> >>>
> >>> Yes! So, we are now talking about type aliases to atomic types. The
> >>> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >>> this:
> >>>
> >>> (...forgive my "creative" representation of WordArray here...)
> >>>
> >>> compiledSpec: 0A 04 00 01
> >>> referentClass: Foo
> >>>
> >>> How does the atomic type for 'char' look like?
> >>>
> >>> compiledSpec: 0A 04 00 01
> >>> referentClass: nil
> >>>
> >>> Consequently, argument coercing will fail if you pass a 'char' when a
> >>> 'Foo' is expected. Because the referentClass does not match --- even
> if the
> >>> compiledSpec does match.
> >>>
> >>> > But this has one major drawback: with current FFI, you cannot pass a
> >>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias for
> >>> an integer type...
> >>>
> >>> That's correct. If the origin of such an argument is from within the
> >>> image, you have to wrap it. But when returned from another call ...
> well
> >>> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>>
> >>> > FFI plugin is clever enough to recognize an atomic type alias and
> >>> simply return a Smalltalk Integer...
> >>> > So it's not Bar all the way down, I have to manually wrap Bar...
> >>> > This is not consistent.
> >>>
> >>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >>> because it does so for argument coercing.
> >>>
> >>> Now I understand why the code generation of struct-field accessors was
> >>> so apparently broken for me. I changed that so that having an alias
> type as
> >>> part of a larger struct will automatically wrap that for you into the
> alias
> >>> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >>> ... Ha ha. ;-) #ffiLongVsInt.
> >>>
> >>> It should be "Bar all the way down". Definitely.
> >>>
> >>> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >>> FFI calls...
> >>>
> >>> Well, not for type aliases. I would like keep the path of type safety
> >>> here, you mentioned above.
> >>>
> >>> Just fix the FFI plugin to respect referentClass when packaging the
> >>> return value.
> >>>
> >>> > I could use the lightweight alias instead, [...]
> >>>
> >>> Please don't. See above. Those "lightweight alias" are for renaming
> >>> atomic types only. Let's keep it simple and try to address you concern
> here
> >>> with struct types and ExternalTypeAlias. :-)
> >>>
> >>> > there was another usage where it was convenient to use
> >>> ExternalTypeAlias: enum.
> >>> > Indeed, I simply defined all enum symbols as class side method.
> >>>
> >>> I like that! It reads like a next step for ExternalPool. Once you have
> >>> extracted the constant values from header files, and once you have
> those
> >>> values for your platform in classVars in your external pool, use that
> pool
> >>> to define enums through ExternalStructure.
> >>>
> >>> Like this:
> >>>
> >>> ExternalTypeAlias subclass: #MyEnumBar
> >>> instanceVariableNames: ''
> >>> classVariableNames: ''
> >>> poolDictionaries: 'HDF5Pool'
> >>> category: 'HDF5'
> >>>
> >>> MyEnumBar class >> #poolFields
> >>> "
> >>> self defineFields.
> >>> "
> >>> ^ #(
> >>> (beg HDF5_BEG)
> >>> (mid HDF5_MID)
> >>> (end HDF5_END)
> >>> )
> >>>
> >>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >>> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>>
> >>> Note that #poolFields would translate to class-side methods as you
> >>> suggested.
> >>>
> >>> > If I use a simpler alias, where are those ids going to?
> >>>
> >>> Please don't. See above. :-)
> >>>
> >>> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >>> inputs/outputs behave consistently.
> >>>
> >>> Absolutely!
> >>>
> >>> > - should a function expecting an ExternalTypeAlias of atomic type
> >>> accept an immediate value of compatible type?
> >>>
> >>> Considering type safety, I think not.
> >>>
> >>> Well ... since this is a convenience issue and thus not about saving
> >>> run-time cost ... What about adding a callback into the image to react
> on
> >>> FFIErrorCoercionFailed? Maybe the selector for such a callback could
> be
> >>> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >>> anyway?
> >>>
> >>>
> >>>
> >>> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>>
> >>> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >>> FFI-Callback. But ignore me here :)
> >>>
> >>> > - should a function returning an ExternalTypeAlias of atomic type
> >>> instantiate the Alias?
> >>>
> >>> Yes, please! See above.
> >>>
> >>> If you want to trade type safety in for a performance gain, just use a
> >>> "lightweight alias" as explained above. And package that return value
> >>> manually.
> >>>
> >>> Best,
> >>> Marcel
> >>>
> >>>
> >>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >>> [hidden email]>:
> >>>
> >>> Hi all,
> >>> following the question of Marcel about usage of ExternalTypeAlias:
> >>>
> >>> Type alias are subclasses of ExternalStructure.
> >>> I used them in HDF5 because there was no other means to define an
> alias
> >>> at the time.
> >>> Now we have another mean by adding a class side method in ExternalType
> >>> (I would have rather imagined the usage of some annotation to get a
> >>> declarative style)
> >>>
> >>> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >>> expecting a Bar, even if they both alias int...
> >>>
> >>> But this has one major drawback: with current FFI, you cannot pass a
> >>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
> alias for
> >>> an integer type...
> >>>
> >>> Thus, you have to wrap every Bar argument into a Bar instance...
> >>> Gasp, this is going too far...
> >>>
> >>> What about functions returning such alias?
> >>> Function returning struct by value allocate a struct, but it's not the
> >>> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >>> and simply return a Smalltalk Integer...
> >>> So it's not Bar all the way down, I have to manually wrap Bar...
> >>> This is not consistent.
> >>> If we return an int, we must accept an int, otherwise, we cannot chain
> >>> FFI calls...
> >>>
> >>> I could use the lightweight alias instead, but there was another usage
> >>> where it was convenient to use ExternalTypeAlias: enum.
> >>> Indeed, I simply defined all enum symbols as class side method. For
> >>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >>> This way, I normally can pass a Bar mid to an external function.
> >>> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>>
> >>> If I use a simpler alias, where are those ids going to?
> >>>
> >>> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >>> inputs/outputs behave consistently.
> >>> The choice is this one:
> >>> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >>> an immediate value of compatible type? (the permissive implementation)
> >>> - should a function returning an ExternalTypeAlias of atomic type
> >>> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >>> cost due to pressure on GC)
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >
>
>
> Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
> écrit :
>
>>
>>
>>
>>
>>
>>
>>
>> > Err, I move this thread to Opensmalltalk VM dev, because it is not
>> Squeak specific!
>>
>> Absolutely. :-)
>>
>> (On a side note: I think that ExternalData should always have a pointer
>> type in its "type" field unless "handle" is a ByteArray instead of
>> ExternalAddress)
>>
>> Let me disagree here...
> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.
>
> (On another side note: I also think that "handle" in ExternalData should
>> never ever be an atomic Smalltalk object (Integer or Float) but always
>> either ByteArray or ExternalAddress. ... which makes it different from what
>> "handle" can be in ExternalStructs that represent type aliases. :-)
>>
>> Yes!
> It could be an ExternalAddress, or a zone of object memory (ByteArray) or
> an Alien as we may want to support that too.
> I'd be inclined to support doubleByte, word, and doubleWord arrays if the
> size of atomic type matches
> (Typically passing a DoubleByteArray, WordArray, a Float32Array, a
> Float64Array to a short */int */float*/double*).
>
> Best,
>> Marcel
>>
>>
>>
>> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
>> [hidden email]>:
>> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
>>
>> specific!
>>
>>
>>
>>
>>
>>
>>
>> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
>> [hidden email]> a écrit :
>>
>>
>>
>> > Hi Marcel,
>>
>> > thanks for your vote. Anyone else?
>>
>> >
>>
>> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>>
>> > écrit :
>>
>> >
>>
>> >> Hi Nicolas, hi all! :-)
>>
>> >>
>>
>> >> > Type alias are subclasses of ExternalStructure.
>>
>> >>
>>
>> >> And with it, they get their own instance of ExternalType, which is
>>
>> >> managed in the classVar StructTypes. Let's call them struct type.
>>
>> >>
>>
>> >> > Now we have another mean by adding a class side method in
>> ExternalType.
>>
>> >>
>>
>> >> That kind of aliasing is for compile-time type referencing only --
>> such
>>
>> >> as in FFI calls (i.e. pragmas) or struct-field definitions
>>
>> >> (i.e. #fields). (And soon if I have the first version of
>>
>> >> FFI-Callback to show you :)
>>
>> >>
>>
>> >> Because those aliases become fully transparent after compiling the
>> method
>>
>> >> for the FFI call into an instance of ExternalLibraryFunction and after
>>
>> >> compiling the field specs of external structures into compiledSpec for
>>
>> >> struct types.
>>
>> >>
>>
>> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>> 'long',
>>
>> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>> 'int32_t'
>>
>> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
>> FFI
>>
>> >> vocabulary. Nothing more. Nothing less.
>>
>> >>
>>
>> >> Having that, please do not use class-side methods in ExternalType to
>>
>> >> alias a (complex) struct type. Use them only for renaming atomic
>> types.
>>
>> >> Please. We may want to add checks to enforce that.
>>
>> >>
>>
>> >> I suppose that this discussion focuses on type aliases that are
>>
>> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
>> and
>>
>> >> thus get their own struct type. So, let's ignore this other mechanism
>> for
>>
>> >> now.
>>
>> >>
>>
>> >> > This has one advantage: type safety. You cannot pass a Foo to a
>>
>> >> function expecting a Bar, even if they both alias int...
>>
>> >>
>>
>> >> Yes! So, we are now talking about type aliases to atomic types. The
>>
>> >> struct type you get when aliasing a 'char' as Foo, for example, looks
>> like
>>
>> >> this:
>>
>> >>
>>
>> >> (...forgive my "creative" representation of WordArray here...)
>>
>> >>
>>
>> >> compiledSpec: 0A 04 00 01
>>
>> >> referentClass: Foo
>>
>> >>
>>
>> >> How does the atomic type for 'char' look like?
>>
>> >>
>>
>> >> compiledSpec: 0A 04 00 01
>>
>> >> referentClass: nil
>>
>> >>
>>
>> >> Consequently, argument coercing will fail if you pass a 'char' when a
>>
>> >> 'Foo' is expected. Because the referentClass does not match --- even
>> if the
>>
>> >> compiledSpec does match.
>>
>> >>
>>
>> >> > But this has one major drawback: with current FFI, you cannot pass a
>>
>> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
>> alias for
>>
>> >> an integer type...
>>
>> >>
>>
>> >> That's correct. If the origin of such an argument is from within the
>>
>> >> image, you have to wrap it. But when returned from another call ...
>> well
>>
>> >> ... that wrapping should have happened in the plugin ... But read on!
>> :-)
>>
>> >>
>>
>> >> > FFI plugin is clever enough to recognize an atomic type alias and
>>
>> >> simply return a Smalltalk Integer...
>>
>> >> > So it's not Bar all the way down, I have to manually wrap Bar...
>>
>> >> > This is not consistent.
>>
>> >>
>>
>> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>
>> >> because it does so for argument coercing.
>>
>> >>
>>
>> >> Now I understand why the code generation of struct-field accessors was
>> so
>>
>> >> apparently broken for me. I changed that so that having an alias type
>> as
>>
>> >> part of a larger struct will automatically wrap that for you into the
>> alias
>>
>> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
>> 'long'
>>
>> >> ... Ha ha. ;-) #ffiLongVsInt.
>>
>> >>
>>
>> >> It should be "Bar all the way down". Definitely.
>>
>> >>
>>
>> >> > If we return an int, we must accept an int, otherwise, we cannot
>> chain
>>
>> >> FFI calls...
>>
>> >>
>>
>> >> Well, not for type aliases. I would like keep the path of type safety
>>
>> >> here, you mentioned above.
>>
>> >>
>>
>> >> Just fix the FFI plugin to respect referentClass when packaging the
>>
>> >> return value.
>>
>> >>
>>
>> >> > I could use the lightweight alias instead, [...]
>>
>> >>
>>
>> >> Please don't. See above. Those "lightweight alias" are for renaming
>>
>> >> atomic types only. Let's keep it simple and try to address you concern
>> here
>>
>> >> with struct types and ExternalTypeAlias. :-)
>>
>> >>
>>
>> >> > there was another usage where it was convenient to use
>>
>> >> ExternalTypeAlias: enum.
>>
>> >> > Indeed, I simply defined all enum symbols as class side method.
>>
>> >>
>>
>> >> I like that! It reads like a next step for ExternalPool. Once you have
>>
>> >> extracted the constant values from header files, and once you have
>> those
>>
>> >> values for your platform in classVars in your external pool, use that
>> pool
>>
>> >> to define enums through ExternalStructure.
>>
>> >>
>>
>> >> Like this:
>>
>> >>
>>
>> >> ExternalTypeAlias subclass: #MyEnumBar
>>
>> >> instanceVariableNames: ''
>>
>> >> classVariableNames: ''
>>
>> >> poolDictionaries: 'HDF5Pool'
>>
>> >> category: 'HDF5'
>>
>> >>
>>
>> >> MyEnumBar class >> #poolFields
>>
>> >> "
>>
>> >> self defineFields.
>>
>> >> "
>>
>> >> ^ #(
>>
>> >> (beg HDF5_BEG)
>>
>> >> (mid HDF5_MID)
>>
>> >> (end HDF5_END)
>>
>> >> )
>>
>> >>
>>
>> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>
>> >> external-pool definitions. See class comment in ExternalPool to get
>> started.
>>
>> >>
>>
>> >> Note that #poolFields would translate to class-side methods as you
>>
>> >> suggested.
>>
>> >>
>>
>> >> > If I use a simpler alias, where are those ids going to?
>>
>> >>
>>
>> >> Please don't. See above. :-)
>>
>> >>
>>
>> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>
>> >> inputs/outputs behave consistently.
>>
>> >>
>>
>> >> Absolutely!
>>
>> >>
>>
>> >> > - should a function expecting an ExternalTypeAlias of atomic type
>>
>> >> accept an immediate value of compatible type?
>>
>> >>
>>
>> >> Considering type safety, I think not.
>>
>> >>
>>
>> >> Well ... since this is a convenience issue and thus not about saving
>>
>> >> run-time cost ... What about adding a callback into the image to react
>> on
>>
>> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could
>> be
>>
>> >> stored in ExternalLibraryFunction since this is accessible to the
>> plugin
>>
>> >> anyway?
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >> ... would it be possible? Like that #doesNotUnderstand: callback?
>>
>> >>
>>
>> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>
>> >> FFI-Callback. But ignore me here :)
>>
>> >>
>>
>> >> > - should a function returning an ExternalTypeAlias of atomic type
>>
>> >> instantiate the Alias?
>>
>> >>
>>
>> >> Yes, please! See above.
>>
>> >>
>>
>> >> If you want to trade type safety in for a performance gain, just use a
>>
>> >> "lightweight alias" as explained above. And package that return value
>>
>> >> manually.
>>
>> >>
>>
>> >> Best,
>>
>> >> Marcel
>>
>> >>
>>
>> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> >> [hidden email]>:
>>
>> >> Hi all,
>>
>> >> following the question of Marcel about usage of ExternalTypeAlias:
>>
>> >>
>>
>> >> Type alias are subclasses of ExternalStructure.
>>
>> >> I used them in HDF5 because there was no other means to define an
>> alias
>>
>> >> at the time.
>>
>> >> Now we have another mean by adding a class side method in ExternalType
>> (I
>>
>> >> would have rather imagined the usage of some annotation to get a
>>
>> >> declarative style)
>>
>> >>
>>
>> >> This has one advantage: type safety. You cannot pass a Foo to a
>> function
>>
>> >> expecting a Bar, even if they both alias int...
>>
>> >>
>>
>> >> But this has one major drawback: with current FFI, you cannot pass a
>>
>> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an
>> alias for
>>
>> >> an integer type...
>>
>> >>
>>
>> >> Thus, you have to wrap every Bar argument into a Bar instance...
>>
>> >> Gasp, this is going too far...
>>
>> >>
>>
>> >> What about functions returning such alias?
>>
>> >> Function returning struct by value allocate a struct, but it's not the
>>
>> >> case here, FFI plugin is clever enough to recognize an atomic type
>> alias
>>
>> >> and simply return a Smalltalk Integer...
>>
>> >> So it's not Bar all the way down, I have to manually wrap Bar...
>>
>> >> This is not consistent.
>>
>> >> If we return an int, we must accept an int, otherwise, we cannot chain
>>
>> >> FFI calls...
>>
>> >>
>>
>> >> I could use the lightweight alias instead, but there was another usage
>>
>> >> where it was convenient to use ExternalTypeAlias: enum.
>>
>> >> Indeed, I simply defined all enum symbols as class side method. For
>>
>> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
>> Bar
>>
>> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>
>> >> This way, I normally can pass a Bar mid to an external function.
>>
>> >> (currently, we must defne mid ^Bar new value: 1; yourself)
>>
>> >>
>>
>> >> If I use a simpler alias, where are those ids going to?
>>
>> >>
>>
>> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>
>> >> inputs/outputs behave consistently.
>>
>> >> The choice is this one:
>>
>> >> - should a function expecting an ExternalTypeAlias of atomic type
>> accept
>>
>> >> an immediate value of compatible type? (the permissive implementation)
>>
>> >> - should a function returning an ExternalTypeAlias of atomic type
>>
>> >> instantiate the Alias? (the typesafe implementation, with higher
>> runtime
>>
>> >> cost due to pressure on GC)
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> >>
>>
>> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
>> specific!
>>
>>
>>
>> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
>> [hidden email]> a écrit :
>>
>>> Hi Marcel,
>>> thanks for your vote. Anyone else?
>>>
>>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>>> écrit :
>>>
>>>>
>>>>
>>>> Hi Nicolas, hi all! :-)
>>>>
>>>> > Type alias are subclasses of ExternalStructure.
>>>>
>>>> And with it, they get their own instance of ExternalType, which is
>>>> managed in the classVar StructTypes. Let's call them struct type.
>>>>
>>>> > Now we have another mean by adding a class side method in
>>>> ExternalType.
>>>>
>>>> That kind of aliasing is for compile-time type referencing only -- such
>>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>>> (i.e. #fields). (And soon if I have the first version of
>>>> FFI-Callback to show you :)
>>>>
>>>> Because those aliases become fully transparent after compiling the
>>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>>> after compiling the field specs of external structures into compiledSpec
>>>> for struct types.
>>>>
>>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>>
>>>> Having that, please do not use class-side methods in ExternalType to
>>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>>> Please. We may want to add checks to enforce that.
>>>>
>>>> I suppose that this discussion focuses on type aliases that are
>>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>>> thus get their own struct type. So, let's ignore this other mechanism for
>>>> now.
>>>>
>>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>>> function expecting a Bar, even if they both alias int...
>>>>
>>>> Yes! So, we are now talking about type aliases to atomic types. The
>>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>>> this:
>>>>
>>>> (...forgive my "creative" representation of WordArray here...)
>>>>
>>>> compiledSpec: 0A 04 00 01
>>>> referentClass: Foo
>>>>
>>>> How does the atomic type for 'char' look like?
>>>>
>>>> compiledSpec: 0A 04 00 01
>>>> referentClass: nil
>>>>
>>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>>> compiledSpec does match.
>>>>
>>>> > But this has one major drawback: with current FFI, you cannot pass a
>>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>>> an integer type...
>>>>
>>>> That's correct. If the origin of such an argument is from within the
>>>> image, you have to wrap it. But when returned from another call ... well
>>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>>
>>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>>> simply return a Smalltalk Integer...
>>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>>> > This is not consistent.
>>>>
>>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>>> because it does so for argument coercing.
>>>>
>>>> Now I understand why the code generation of struct-field accessors was
>>>> so apparently broken for me. I changed that so that having an alias type as
>>>> part of a larger struct will automatically wrap that for you into the alias
>>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>>
>>>> It should be "Bar all the way down". Definitely.
>>>>
>>>> > If we return an int, we must accept an int, otherwise, we cannot
>>>> chain FFI calls...
>>>>
>>>> Well, not for type aliases. I would like keep the path of type safety
>>>> here, you mentioned above.
>>>>
>>>> Just fix the FFI plugin to respect referentClass when packaging the
>>>> return value.
>>>>
>>>> > I could use the lightweight alias instead, [...]
>>>>
>>>> Please don't. See above. Those "lightweight alias" are for renaming
>>>> atomic types only. Let's keep it simple and try to address you concern here
>>>> with struct types and ExternalTypeAlias. :-)
>>>>
>>>> > there was another usage where it was convenient to use
>>>> ExternalTypeAlias: enum.
>>>> > Indeed, I simply defined all enum symbols as class side method.
>>>>
>>>> I like that! It reads like a next step for ExternalPool. Once you have
>>>> extracted the constant values from header files, and once you have those
>>>> values for your platform in classVars in your external pool, use that pool
>>>> to define enums through ExternalStructure.
>>>>
>>>> Like this:
>>>>
>>>> ExternalTypeAlias subclass: #MyEnumBar
>>>> instanceVariableNames: ''
>>>> classVariableNames: ''
>>>> poolDictionaries: 'HDF5Pool'
>>>> category: 'HDF5'
>>>>
>>>> MyEnumBar class >> #poolFields
>>>> "
>>>> self defineFields.
>>>> "
>>>> ^ #(
>>>> (beg HDF5_BEG)
>>>> (mid HDF5_MID)
>>>> (end HDF5_END)
>>>> )
>>>>
>>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>>
>>>> Note that #poolFields would translate to class-side methods as you
>>>> suggested.
>>>>
>>>> > If I use a simpler alias, where are those ids going to?
>>>>
>>>> Please don't. See above. :-)
>>>>
>>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>>> inputs/outputs behave consistently.
>>>>
>>>> Absolutely!
>>>>
>>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>>> accept an immediate value of compatible type?
>>>>
>>>> Considering type safety, I think not.
>>>>
>>>> Well ... since this is a convenience issue and thus not about saving
>>>> run-time cost ... What about adding a callback into the image to react on
>>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>>> anyway?
>>>>
>>>>
>>>>
>>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>>
>>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>>> FFI-Callback. But ignore me here :)
>>>>
>>>> > - should a function returning an ExternalTypeAlias of atomic type
>>>> instantiate the Alias?
>>>>
>>>> Yes, please! See above.
>>>>
>>>> If you want to trade type safety in for a performance gain, just use a
>>>> "lightweight alias" as explained above. And package that return value
>>>> manually.
>>>>
>>>> Best,
>>>> Marcel
>>>>
>>>>
>>>>
>>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>>> [hidden email]>:
>>>>
>>>>
>>>> Hi all,
>>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>>
>>>> Type alias are subclasses of ExternalStructure.
>>>> I used them in HDF5 because there was no other means to define an alias
>>>> at the time.
>>>> Now we have another mean by adding a class side method in ExternalType
>>>> (I would have rather imagined the usage of some annotation to get a
>>>> declarative style)
>>>>
>>>> This has one advantage: type safety. You cannot pass a Foo to a
>>>> function expecting a Bar, even if they both alias int...
>>>>
>>>> But this has one major drawback: with current FFI, you cannot pass a
>>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>>> an integer type...
>>>>
>>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>>> Gasp, this is going too far...
>>>>
>>>> What about functions returning such alias?
>>>> Function returning struct by value allocate a struct, but it's not the
>>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>>> and simply return a Smalltalk Integer...
>>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>>> This is not consistent.
>>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>>> FFI calls...
>>>>
>>>> I could use the lightweight alias instead, but there was another usage
>>>> where it was convenient to use ExternalTypeAlias: enum.
>>>> Indeed, I simply defined all enum symbols as class side method. For
>>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>>> This way, I normally can pass a Bar mid to an external function.
>>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>>
>>>> If I use a simpler alias, where are those ids going to?
>>>>
>>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>>> inputs/outputs behave consistently.
>>>> The choice is this one:
>>>> - should a function expecting an ExternalTypeAlias of atomic type
>>>> accept an immediate value of compatible type? (the permissive
>>>> implementation)
>>>> - should a function returning an ExternalTypeAlias of atomic type
>>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>>> cost due to pressure on GC)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
Well, my mental model is this one:

ExternalData needs to refer to an area of memory, so as to be able to act as a pointer (single element or array does not really matter).
If this memory area lies in external heap, then we need an ExternalAddress as handle.
If this memory area lies in object memory, it must be a ByteArray.
We cannot handle pointers to object memory, because object memory is relocatable...
Well, unless we pin the object - that why we pass the ByteArray itself, not its address..

This is independent of how we interpret the contents.
IMO, it's more useful to avoid the pointer indirection in type, because we cannot handle double pointer currently...
Of course, the only pointers that the memory area could contain are pointers to external heap (or eventually pinned objects).

I'd like to have a 3rd possibility: pointing to a sub-region of a ByteArray (ByteArray + offset).
That may be very useful for emulating.
An example in Smallapack: pass a pointer to the imaginary part of a complex array.
Since real and imaginary parts are interleaved, just offsetting the pointer and using a stride of 2 elements does the trick when one wants to access real or imaginary part
(most lapack/blas routines are equipped to handle simple strides).
This is currently possible with ExternalAddress, but NOT when matrix data lies in ObjectMemory.
Another example would be if you wanted to implement an interpreter for a low level language... (emulate a processor, whatever).



Le ven. 19 juin 2020 à 18:04, Marcel Taeumel <[hidden email]> a écrit :
 

So, what about this:

ExternalData<@00F3A3EE, int>

I think it could mean that you want to interpret the very pointer address itself as an integer. That is, you do not want to follow to where it points to.

Then what about this:

ExternalData<@00F3A3EE, double>

Ha! An error, I suppose. Unless ... given 8-byte pointers, you can try to read a double out of it. :-D Just kidding. It should be an error.

.
.
.

> > (On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)
> Let me disagree here...

The examples I just wrote down in the previous messages let me realize why you did disagree. :-D

I hope this helps you with improving argument coercing and return value packaging in the FFI plugin. ^__^

Best,
Marcel

Am 19.06.2020 17:53:21 schrieb Marcel Taeumel <[hidden email]>:


then you would use int* and not int**

My bad: "  then you would use int** and not int*  " Sorry for the noise -.-"

Am 19.06.2020 17:52:21 schrieb Marcel Taeumel <[hidden email]>:


If you want to make a pointer array out of it, change the type to int**

Correction: If you happen to now, that this external address points to an array of pointers to int, then you would use int* and not int**. You cannot just "decide" what the data behind that external address should be. .. you know what I mean. ;-)

best,
Marcel

Am 19.06.2020 17:47:36 schrieb Marcel Taeumel <[hidden email]>:


Hi Nicolas.

Maybe one more thought on

ExternalData<handle, type>

There is no need to have

ExternalData<1312301580, int>

because it can just be 1312301580.

On the other hand:

ExternalData<#[ 12 34 56 78 ], int>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is already in Squeak's object memory.

Finally:

ExternalData<@00F3A3EE, int*>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is in external memory.

If you want to make an array out of it, make use of the "size" field in ExternalData.

If you want to make a pointer array out of it, change the type to int**

ExternalData<@00F3A3EE, int**>

So, what does this mean then:

ExternalData<#[ 12 34 56 78 ], int*>

Well, it tells you that there is a pointer to an int stored in a byte array. This comes very handy if you think about type aliases to pointer types such as "typedef int* INT_P". Because then it would be

ExternalData<#[ 12 34 56 78 ], INT_P>

To where this pointer points to? Probably garbage. I just made it up for this example. :-D

Best,
Marcel

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:







> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.



We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,
Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a

écrit :



>

> > Err, I move this thread to Opensmalltalk VM dev, because it is not

> Squeak specific!

>

> Absolutely. :-)

>

> (On a side note: I think that ExternalData should always have a pointer

> type in its "type" field unless "handle" is a ByteArray instead of

> ExternalAddress)

>

> Let me disagree here...

The type should better describe the contents of the handle (be it an

ExternalAddress or a zone of object memory (ByteArray)).

Typically, if you have an external (global) variable of type foo (extern

foo my_var;) then the natural type is foo.

Having foo*, would mean that we handle an array of foo*, or a pointer to

foo* (foo**).

This way, we make no difference between type surrogates

(ExternalStruct/ExternalTypeAlias) and other atomic cases.



(On another side note: I also think that "handle" in ExternalData should

> never ever be an atomic Smalltalk object (Integer or Float) but always

> either ByteArray or ExternalAddress. ... which makes it different from what

> "handle" can be in ExternalStructs that represent type aliases. :-)

>

> Yes!

It could be an ExternalAddress, or a zone of object memory (ByteArray) or

an Alien as we may want to support that too.

I'd be inclined to support doubleByte, word, and doubleWord arrays if the

size of atomic type matches

(Typically passing a DoubleByteArray, WordArray, a Float32Array, a

Float64Array to a short */int */float*/double*).



Best,

> Marcel

>

> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:

> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

> specific!

>

>

>

> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>

> [hidden email]> a écrit :

>

> > Hi Marcel,

> > thanks for your vote. Anyone else?

> >

> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> > écrit :

> >

> >> Hi Nicolas, hi all! :-)

> >>

> >> > Type alias are subclasses of ExternalStructure.

> >>

> >> And with it, they get their own instance of ExternalType, which is

> >> managed in the classVar StructTypes. Let's call them struct type.

> >>

> >> > Now we have another mean by adding a class side method in

> ExternalType.

> >>

> >> That kind of aliasing is for compile-time type referencing only -- such

> >> as in FFI calls (i.e. pragmas) or struct-field definitions

> >> (i.e. #fields). (And soon if I have the first version of

> >> FFI-Callback to show you :)

> >>

> >> Because those aliases become fully transparent after compiling the

> method

> >> for the FFI call into an instance of ExternalLibraryFunction and after

> >> compiling the field specs of external structures into compiledSpec for

> >> struct types.

> >>

> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a

> 'long',

> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And

> 'int32_t'

> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak

> FFI

> >> vocabulary. Nothing more. Nothing less.

> >>

> >> Having that, please do not use class-side methods in ExternalType to

> >> alias a (complex) struct type. Use them only for renaming atomic types.

> >> Please. We may want to add checks to enforce that.

> >>

> >> I suppose that this discussion focuses on type aliases that are

> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)

> and

> >> thus get their own struct type. So, let's ignore this other mechanism

> for

> >> now.

> >>

> >> > This has one advantage: type safety. You cannot pass a Foo to a

> >> function expecting a Bar, even if they both alias int...

> >>

> >> Yes! So, we are now talking about type aliases to atomic types. The

> >> struct type you get when aliasing a 'char' as Foo, for example, looks

> like

> >> this:

> >>

> >> (...forgive my "creative" representation of WordArray here...)

> >>

> >> compiledSpec: 0A 04 00 01

> >> referentClass: Foo

> >>

> >> How does the atomic type for 'char' look like?

> >>

> >> compiledSpec: 0A 04 00 01

> >> referentClass: nil

> >>

> >> Consequently, argument coercing will fail if you pass a 'char' when a

> >> 'Foo' is expected. Because the referentClass does not match --- even if

> the

> >> compiledSpec does match.

> >>

> >> > But this has one major drawback: with current FFI, you cannot pass a

> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias

> for

> >> an integer type...

> >>

> >> That's correct. If the origin of such an argument is from within the

> >> image, you have to wrap it. But when returned from another call ...

> well

> >> ... that wrapping should have happened in the plugin ... But read on!

> :-)

> >>

> >> > FFI plugin is clever enough to recognize an atomic type alias and

> >> simply return a Smalltalk Integer...

> >> > So it's not Bar all the way down, I have to manually wrap Bar...

> >> > This is not consistent.

> >>

> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

> >> because it does so for argument coercing.

> >>

> >> Now I understand why the code generation of struct-field accessors was

> so

> >> apparently broken for me. I changed that so that having an alias type

> as

> >> part of a larger struct will automatically wrap that for you into the

> alias

> >> structure. :-) Even if you are just aliasing a plain 'int' ... or

> 'long'

> >> ... Ha ha. ;-) #ffiLongVsInt.

> >>

> >> It should be "Bar all the way down". Definitely.

> >>

> >> > If we return an int, we must accept an int, otherwise, we cannot

> chain

> >> FFI calls...

> >>

> >> Well, not for type aliases. I would like keep the path of type safety

> >> here, you mentioned above.

> >>

> >> Just fix the FFI plugin to respect referentClass when packaging the

> >> return value.

> >>

> >> > I could use the lightweight alias instead, [...]

> >>

> >> Please don't. See above. Those "lightweight alias" are for renaming

> >> atomic types only. Let's keep it simple and try to address you concern

> here

> >> with struct types and ExternalTypeAlias. :-)

> >>

> >> > there was another usage where it was convenient to use

> >> ExternalTypeAlias: enum.

> >> > Indeed, I simply defined all enum symbols as class side method.

> >>

> >> I like that! It reads like a next step for ExternalPool. Once you have

> >> extracted the constant values from header files, and once you have

> those

> >> values for your platform in classVars in your external pool, use that

> pool

> >> to define enums through ExternalStructure.

> >>

> >> Like this:

> >>

> >> ExternalTypeAlias subclass: #MyEnumBar

> >> instanceVariableNames: ''

> >> classVariableNames: ''

> >> poolDictionaries: 'HDF5Pool'

> >> category: 'HDF5'

> >>

> >> MyEnumBar class >> #poolFields

> >> "

> >> self defineFields.

> >> "

> >> ^ #(

> >> (beg HDF5_BEG)

> >> (mid HDF5_MID)

> >> (end HDF5_END)

> >> )

> >>

> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

> >> external-pool definitions. See class comment in ExternalPool to get

> started.

> >>

> >> Note that #poolFields would translate to class-side methods as you

> >> suggested.

> >>

> >> > If I use a simpler alias, where are those ids going to?

> >>

> >> Please don't. See above. :-)

> >>

> >> > IMO we cannot keep status quo in the plugin, we gotta to make the

> >> inputs/outputs behave consistently.

> >>

> >> Absolutely!

> >>

> >> > - should a function expecting an ExternalTypeAlias of atomic type

> >> accept an immediate value of compatible type?

> >>

> >> Considering type safety, I think not.

> >>

> >> Well ... since this is a convenience issue and thus not about saving

> >> run-time cost ... What about adding a callback into the image to react

> on

> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

> >> stored in ExternalLibraryFunction since this is accessible to the

> plugin

> >> anyway?

> >>

> >>

> >>

> >> ... would it be possible? Like that #doesNotUnderstand: callback?

> >>

> >> I would love to do ad-hoc packaging of arguments. (Also thinking about

> >> FFI-Callback. But ignore me here :)

> >>

> >> > - should a function returning an ExternalTypeAlias of atomic type

> >> instantiate the Alias?

> >>

> >> Yes, please! See above.

> >>

> >> If you want to trade type safety in for a performance gain, just use a

> >> "lightweight alias" as explained above. And package that return value

> >> manually.

> >>

> >> Best,

> >> Marcel

> >>

> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>

> >> [hidden email]>:

> >> Hi all,

> >> following the question of Marcel about usage of ExternalTypeAlias:

> >>

> >> Type alias are subclasses of ExternalStructure.

> >> I used them in HDF5 because there was no other means to define an alias

> >> at the time.

> >> Now we have another mean by adding a class side method in ExternalType

> (I

> >> would have rather imagined the usage of some annotation to get a

> >> declarative style)

> >>

> >> This has one advantage: type safety. You cannot pass a Foo to a

> function

> >> expecting a Bar, even if they both alias int...

> >>

> >> But this has one major drawback: with current FFI, you cannot pass a

> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias

> for

> >> an integer type...

> >>

> >> Thus, you have to wrap every Bar argument into a Bar instance...

> >> Gasp, this is going too far...

> >>

> >> What about functions returning such alias?

> >> Function returning struct by value allocate a struct, but it's not the

> >> case here, FFI plugin is clever enough to recognize an atomic type

> alias

> >> and simply return a Smalltalk Integer...

> >> So it's not Bar all the way down, I have to manually wrap Bar...

> >> This is not consistent.

> >> If we return an int, we must accept an int, otherwise, we cannot chain

> >> FFI calls...

> >>

> >> I could use the lightweight alias instead, but there was another usage

> >> where it was convenient to use ExternalTypeAlias: enum.

> >> Indeed, I simply defined all enum symbols as class side method. For

> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type

> Bar

> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

> >> This way, I normally can pass a Bar mid to an external function.

> >> (currently, we must defne mid ^Bar new value: 1; yourself)

> >>

> >> If I use a simpler alias, where are those ids going to?

> >>

> >> IMO we cannot keep statu quo in the plugin, we gotta to make the

> >> inputs/outputs behave consistently.

> >> The choice is this one:

> >> - should a function expecting an ExternalTypeAlias of atomic type

> accept

> >> an immediate value of compatible type? (the permissive implementation)

> >> - should a function returning an ExternalTypeAlias of atomic type

> >> instantiate the Alias? (the typesafe implementation, with higher

> runtime

> >> cost due to pressure on GC)

> >>

> >>

> >>

> >>

> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

> specific!

>

>

>

> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :

>

>> Hi Marcel,

>> thanks for your vote. Anyone else?

>>

>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

>> écrit :

>>

>>>

>>> Hi Nicolas, hi all! :-)

>>>

>>> > Type alias are subclasses of ExternalStructure.

>>>

>>> And with it, they get their own instance of ExternalType, which is

>>> managed in the classVar StructTypes. Let's call them struct type.

>>>

>>> > Now we have another mean by adding a class side method in ExternalType.

>>>

>>> That kind of aliasing is for compile-time type referencing only -- such

>>> as in FFI calls (i.e. pragmas) or struct-field definitions

>>> (i.e. #fields). (And soon if I have the first version of

>>> FFI-Callback to show you :)

>>>

>>> Because those aliases become fully transparent after compiling the

>>> method for the FFI call into an instance of ExternalLibraryFunction and

>>> after compiling the field specs of external structures into compiledSpec

>>> for struct types.

>>>

>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a

>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And

>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into

>>> Squeak FFI vocabulary. Nothing more. Nothing less.

>>>

>>> Having that, please do not use class-side methods in ExternalType to

>>> alias a (complex) struct type. Use them only for renaming atomic types.

>>> Please. We may want to add checks to enforce that.

>>>

>>> I suppose that this discussion focuses on type aliases that are

>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>>> thus get their own struct type. So, let's ignore this other mechanism for

>>> now.

>>>

>>> > This has one advantage: type safety. You cannot pass a Foo to a

>>> function expecting a Bar, even if they both alias int...

>>>

>>> Yes! So, we are now talking about type aliases to atomic types. The

>>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>>> this:

>>>

>>> (...forgive my "creative" representation of WordArray here...)

>>>

>>> compiledSpec: 0A 04 00 01

>>> referentClass: Foo

>>>

>>> How does the atomic type for 'char' look like?

>>>

>>> compiledSpec: 0A 04 00 01

>>> referentClass: nil

>>>

>>> Consequently, argument coercing will fail if you pass a 'char' when a

>>> 'Foo' is expected. Because the referentClass does not match --- even if the

>>> compiledSpec does match.

>>>

>>> > But this has one major drawback: with current FFI, you cannot pass a

>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>>> an integer type...

>>>

>>> That's correct. If the origin of such an argument is from within the

>>> image, you have to wrap it. But when returned from another call ... well

>>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>>

>>> > FFI plugin is clever enough to recognize an atomic type alias and

>>> simply return a Smalltalk Integer...

>>> > So it's not Bar all the way down, I have to manually wrap Bar...

>>> > This is not consistent.

>>>

>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>>> because it does so for argument coercing.

>>>

>>> Now I understand why the code generation of struct-field accessors was

>>> so apparently broken for me. I changed that so that having an alias type as

>>> part of a larger struct will automatically wrap that for you into the alias

>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>>> ... Ha ha. ;-) #ffiLongVsInt.

>>>

>>> It should be "Bar all the way down". Definitely.

>>>

>>> > If we return an int, we must accept an int, otherwise, we cannot chain

>>> FFI calls...

>>>

>>> Well, not for type aliases. I would like keep the path of type safety

>>> here, you mentioned above.

>>>

>>> Just fix the FFI plugin to respect referentClass when packaging the

>>> return value.

>>>

>>> > I could use the lightweight alias instead, [...]

>>>

>>> Please don't. See above. Those "lightweight alias" are for renaming

>>> atomic types only. Let's keep it simple and try to address you concern here

>>> with struct types and ExternalTypeAlias. :-)

>>>

>>> > there was another usage where it was convenient to use

>>> ExternalTypeAlias: enum.

>>> > Indeed, I simply defined all enum symbols as class side method.

>>>

>>> I like that! It reads like a next step for ExternalPool. Once you have

>>> extracted the constant values from header files, and once you have those

>>> values for your platform in classVars in your external pool, use that pool

>>> to define enums through ExternalStructure.

>>>

>>> Like this:

>>>

>>> ExternalTypeAlias subclass: #MyEnumBar

>>> instanceVariableNames: ''

>>> classVariableNames: ''

>>> poolDictionaries: 'HDF5Pool'

>>> category: 'HDF5'

>>>

>>> MyEnumBar class >> #poolFields

>>> "

>>> self defineFields.

>>> "

>>> ^ #(

>>> (beg HDF5_BEG)

>>> (mid HDF5_MID)

>>> (end HDF5_END)

>>> )

>>>

>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>>> external-pool definitions. See class comment in ExternalPool to get started.

>>>

>>> Note that #poolFields would translate to class-side methods as you

>>> suggested.

>>>

>>> > If I use a simpler alias, where are those ids going to?

>>>

>>> Please don't. See above. :-)

>>>

>>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>>> inputs/outputs behave consistently.

>>>

>>> Absolutely!

>>>

>>> > - should a function expecting an ExternalTypeAlias of atomic type

>>> accept an immediate value of compatible type?

>>>

>>> Considering type safety, I think not.

>>>

>>> Well ... since this is a convenience issue and thus not about saving

>>> run-time cost ... What about adding a callback into the image to react on

>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>>> stored in ExternalLibraryFunction since this is accessible to the plugin

>>> anyway?

>>>

>>>

>>>

>>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>>

>>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>>> FFI-Callback. But ignore me here :)

>>>

>>> > - should a function returning an ExternalTypeAlias of atomic type

>>> instantiate the Alias?

>>>

>>> Yes, please! See above.

>>>

>>> If you want to trade type safety in for a performance gain, just use a

>>> "lightweight alias" as explained above. And package that return value

>>> manually.

>>>

>>> Best,

>>> Marcel

>>>

>>>

>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:

>>>

>>> Hi all,

>>> following the question of Marcel about usage of ExternalTypeAlias:

>>>

>>> Type alias are subclasses of ExternalStructure.

>>> I used them in HDF5 because there was no other means to define an alias

>>> at the time.

>>> Now we have another mean by adding a class side method in ExternalType

>>> (I would have rather imagined the usage of some annotation to get a

>>> declarative style)

>>>

>>> This has one advantage: type safety. You cannot pass a Foo to a function

>>> expecting a Bar, even if they both alias int...

>>>

>>> But this has one major drawback: with current FFI, you cannot pass a

>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>>> an integer type...

>>>

>>> Thus, you have to wrap every Bar argument into a Bar instance...

>>> Gasp, this is going too far...

>>>

>>> What about functions returning such alias?

>>> Function returning struct by value allocate a struct, but it's not the

>>> case here, FFI plugin is clever enough to recognize an atomic type alias

>>> and simply return a Smalltalk Integer...

>>> So it's not Bar all the way down, I have to manually wrap Bar...

>>> This is not consistent.

>>> If we return an int, we must accept an int, otherwise, we cannot chain

>>> FFI calls...

>>>

>>> I could use the lightweight alias instead, but there was another usage

>>> where it was convenient to use ExternalTypeAlias: enum.

>>> Indeed, I simply defined all enum symbols as class side method. For

>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>>> This way, I normally can pass a Bar mid to an external function.

>>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>>

>>> If I use a simpler alias, where are those ids going to?

>>>

>>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>>> inputs/outputs behave consistently.

>>> The choice is this one:

>>> - should a function expecting an ExternalTypeAlias of atomic type accept

>>> an immediate value of compatible type? (the permissive implementation)

>>> - should a function returning an ExternalTypeAlias of atomic type

>>> instantiate the Alias? (the typesafe implementation, with higher runtime

>>> cost due to pressure on GC)

>>>

>>>

>>>

>>>

>>>

>>>

>>>

>>

>



Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :
 












Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...
The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).
This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,
Marcel






Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak


specific!











Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :





> Hi Marcel,


> thanks for your vote. Anyone else?


>


> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a


> écrit :


>


>> Hi Nicolas, hi all! :-)


>>


>> > Type alias are subclasses of ExternalStructure.


>>


>> And with it, they get their own instance of ExternalType, which is


>> managed in the classVar StructTypes. Let's call them struct type.


>>


>> > Now we have another mean by adding a class side method in ExternalType.


>>


>> That kind of aliasing is for compile-time type referencing only -- such


>> as in FFI calls (i.e. pragmas) or struct-field definitions


>> (i.e. #fields). (And soon if I have the first version of


>> FFI-Callback to show you :)


>>


>> Because those aliases become fully transparent after compiling the method


>> for the FFI call into an instance of ExternalLibraryFunction and after


>> compiling the field specs of external structures into compiledSpec for


>> struct types.


>>


>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',


>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'


>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI


>> vocabulary. Nothing more. Nothing less.


>>


>> Having that, please do not use class-side methods in ExternalType to


>> alias a (complex) struct type. Use them only for renaming atomic types.


>> Please. We may want to add checks to enforce that.


>>


>> I suppose that this discussion focuses on type aliases that are


>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and


>> thus get their own struct type. So, let's ignore this other mechanism for


>> now.


>>


>> > This has one advantage: type safety. You cannot pass a Foo to a


>> function expecting a Bar, even if they both alias int...


>>


>> Yes! So, we are now talking about type aliases to atomic types. The


>> struct type you get when aliasing a 'char' as Foo, for example, looks like


>> this:


>>


>> (...forgive my "creative" representation of WordArray here...)


>>


>> compiledSpec: 0A 04 00 01


>> referentClass: Foo


>>


>> How does the atomic type for 'char' look like?


>>


>> compiledSpec: 0A 04 00 01


>> referentClass: nil


>>


>> Consequently, argument coercing will fail if you pass a 'char' when a


>> 'Foo' is expected. Because the referentClass does not match --- even if the


>> compiledSpec does match.


>>


>> > But this has one major drawback: with current FFI, you cannot pass a


>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for


>> an integer type...


>>


>> That's correct. If the origin of such an argument is from within the


>> image, you have to wrap it. But when returned from another call ... well


>> ... that wrapping should have happened in the plugin ... But read on! :-)


>>


>> > FFI plugin is clever enough to recognize an atomic type alias and


>> simply return a Smalltalk Integer...


>> > So it's not Bar all the way down, I have to manually wrap Bar...


>> > This is not consistent.


>>


>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass


>> because it does so for argument coercing.


>>


>> Now I understand why the code generation of struct-field accessors was so


>> apparently broken for me. I changed that so that having an alias type as


>> part of a larger struct will automatically wrap that for you into the alias


>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'


>> ... Ha ha. ;-) #ffiLongVsInt.


>>


>> It should be "Bar all the way down". Definitely.


>>


>> > If we return an int, we must accept an int, otherwise, we cannot chain


>> FFI calls...


>>


>> Well, not for type aliases. I would like keep the path of type safety


>> here, you mentioned above.


>>


>> Just fix the FFI plugin to respect referentClass when packaging the


>> return value.


>>


>> > I could use the lightweight alias instead, [...]


>>


>> Please don't. See above. Those "lightweight alias" are for renaming


>> atomic types only. Let's keep it simple and try to address you concern here


>> with struct types and ExternalTypeAlias. :-)


>>


>> > there was another usage where it was convenient to use


>> ExternalTypeAlias: enum.


>> > Indeed, I simply defined all enum symbols as class side method.


>>


>> I like that! It reads like a next step for ExternalPool. Once you have


>> extracted the constant values from header files, and once you have those


>> values for your platform in classVars in your external pool, use that pool


>> to define enums through ExternalStructure.


>>


>> Like this:


>>


>> ExternalTypeAlias subclass: #MyEnumBar


>> instanceVariableNames: ''


>> classVariableNames: ''


>> poolDictionaries: 'HDF5Pool'


>> category: 'HDF5'


>>


>> MyEnumBar class >> #poolFields


>> "


>> self defineFields.


>> "


>> ^ #(


>> (beg HDF5_BEG)


>> (mid HDF5_MID)


>> (end HDF5_END)


>> )


>>


>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via


>> external-pool definitions. See class comment in ExternalPool to get started.


>>


>> Note that #poolFields would translate to class-side methods as you


>> suggested.


>>


>> > If I use a simpler alias, where are those ids going to?


>>


>> Please don't. See above. :-)


>>


>> > IMO we cannot keep status quo in the plugin, we gotta to make the


>> inputs/outputs behave consistently.


>>


>> Absolutely!


>>


>> > - should a function expecting an ExternalTypeAlias of atomic type


>> accept an immediate value of compatible type?


>>


>> Considering type safety, I think not.


>>


>> Well ... since this is a convenience issue and thus not about saving


>> run-time cost ... What about adding a callback into the image to react on


>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be


>> stored in ExternalLibraryFunction since this is accessible to the plugin


>> anyway?


>>


>>


>>


>> ... would it be possible? Like that #doesNotUnderstand: callback?


>>


>> I would love to do ad-hoc packaging of arguments. (Also thinking about


>> FFI-Callback. But ignore me here :)


>>


>> > - should a function returning an ExternalTypeAlias of atomic type


>> instantiate the Alias?


>>


>> Yes, please! See above.


>>


>> If you want to trade type safety in for a performance gain, just use a


>> "lightweight alias" as explained above. And package that return value


>> manually.


>>


>> Best,


>> Marcel


>>


>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:


>> Hi all,


>> following the question of Marcel about usage of ExternalTypeAlias:


>>


>> Type alias are subclasses of ExternalStructure.


>> I used them in HDF5 because there was no other means to define an alias


>> at the time.


>> Now we have another mean by adding a class side method in ExternalType (I


>> would have rather imagined the usage of some annotation to get a


>> declarative style)


>>


>> This has one advantage: type safety. You cannot pass a Foo to a function


>> expecting a Bar, even if they both alias int...


>>


>> But this has one major drawback: with current FFI, you cannot pass a


>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for


>> an integer type...


>>


>> Thus, you have to wrap every Bar argument into a Bar instance...


>> Gasp, this is going too far...


>>


>> What about functions returning such alias?


>> Function returning struct by value allocate a struct, but it's not the


>> case here, FFI plugin is clever enough to recognize an atomic type alias


>> and simply return a Smalltalk Integer...


>> So it's not Bar all the way down, I have to manually wrap Bar...


>> This is not consistent.


>> If we return an int, we must accept an int, otherwise, we cannot chain


>> FFI calls...


>>


>> I could use the lightweight alias instead, but there was another usage


>> where it was convenient to use ExternalTypeAlias: enum.


>> Indeed, I simply defined all enum symbols as class side method. For


>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar


>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.


>> This way, I normally can pass a Bar mid to an external function.


>> (currently, we must defne mid ^Bar new value: 1; yourself)


>>


>> If I use a simpler alias, where are those ids going to?


>>


>> IMO we cannot keep statu quo in the plugin, we gotta to make the


>> inputs/outputs behave consistently.


>> The choice is this one:


>> - should a function expecting an ExternalTypeAlias of atomic type accept


>> an immediate value of compatible type? (the permissive implementation)


>> - should a function returning an ExternalTypeAlias of atomic type


>> instantiate the Alias? (the typesafe implementation, with higher runtime


>> cost due to pressure on GC)


>>


>>


>>


>>


Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :



Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel



Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:




Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)

























Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] FFI ExternalTypeAlias

Nicolas Cellier
In reply to this post by marcel.taeumel
 


Le ven. 19 juin 2020 à 17:15, Marcel Taeumel <[hidden email]> a écrit :
 
> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.

We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

Beware, we have two different things here:
- ByteArray and ExternalAddress both define a zone of memory.
  one is in ObjectMemory and accessed directly, the other lies in Heap and is accessed indirectly thru the ExternalAddress
  But that's details, they behave the same and can be viewed with the same type from outside.
  It's like direct vs indirect Alien, a single class with both capabilities.
- Float Integer Character true false nil...
  those are immediate values and cannot be passed by reference
  We could eventually copy them in somewhere on the stack and pass a reference to that if we are just that those are read-only (const).
  But currently, we have no way to know that, so we'd better not.
  Maybe they are passed by reference so that we get the value back.

So the handle of an ExternalData can only be of the first kind.

A more specialized Float32Array or Float64Array could be used in place of the ByteArray.
But there is some kind of redundancy in type representation: RawBitsArray  subclasses implicitly encodes the type in the class.
They are idempotent to ExternalData and we do not need an ExternalData on: anExternalData...
I had the very same idea, but I just wonder if it is useful.
That makes me think that those classes/objects should be responsible for checking parameter spec conformance by their own...
Enlarging the zoo of classes known by the plugin is a no go.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Ah, but I fixed something related to that on pointer return.
If the type is atomic non pointer, or struct non pointer, we'll get a Foo now.
That's only one (duplicated) change.

If the type is pointer to whatever, we'll get an ExternalData.
Ah, I forgot the case of char*, we'll get a String (hoping it's null terminated!).
If we don't expect a NULL terminated string, then we must use byte * instead of char*.

A side effect is that a VoidPtr alias to void* will now answer an ExternalData rather than an ExternalTypeAlias when returning VoidPtr...
A reference to such aliases would be problematic only if we try to interpret the contents by ourselves...
For opaque type, it's OK because we do not look inside.
What is lacking though is the ability to pass a pointer to a VoidPtr *, so as to retrieve a different VoidPtr.
All the responsibility is at the client side...

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

That's something we need to inquire about.
Some FFI examples define such opaque struct as void *, some others as intptr_t...
We must support both in a reasonable manner.
By now, the FFI plugin will behave differently and I'm not sure it does it consistently, nor whether it meets user expectations...
We have to add test cases for documenting the (expected) behavior.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

Either we have the type checks and coercing in the image side, or we have to encode this in the spec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

Yes, at image side we definitely have to support arity if we need to access contents...
Not for opaque, but there are other use cases.
IMO there are more complex uses cases than HDF5, i think of GUIs, but you know that well :)

or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Alien support is required for Callback now.
Otherwise, it's completely redundant to a ByteArray/ExternalAddress (except that it can encode both cases...).

Best,
Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...
The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>


Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :
 






Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...
The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).
This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,
Marcel



Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :
Hi Marcel,
thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :


Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01
referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01
referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
> So it's not Bar all the way down, I have to manually wrap Bar...
> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.
> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: 'HDF5Pool'
category: 'HDF5'

MyEnumBar class >> #poolFields
   "
   self defineFields.
   "
   ^ #(
      (beg HDF5_BEG)
      (mid HDF5_MID)
      (end HDF5_END)
   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,
Marcel


Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:



Hi all,
following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.
I used them in HDF5 because there was no other means to define an alias at the time.
Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...
Gasp, this is going too far...

What about functions returning such alias?
Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...
So it's not Bar all the way down, I have to manually wrap Bar...
This is not consistent.
If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.
Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.
This way, I normally can pass a Bar mid to an external function.
(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.
 The choice is this one:
- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)
- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)