Re: [squeak-dev] FFI ExternalTypeAlias

Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
specific!



Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :

> Hi Marcel,
> thanks for your vote. Anyone else?
>
> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> écrit :
>
>> Hi Nicolas, hi all! :-)
>>
>> > Type alias are subclasses of ExternalStructure.
>>
>> And with it, they get their own instance of ExternalType, which is
>> managed in the classVar StructTypes. Let's call them struct type.
>>
>> > Now we have another mean by adding a class side method in ExternalType.
>>
>> That kind of aliasing is for compile-time type referencing only -- such
>> as in FFI calls (i.e. pragmas) or struct-field definitions
>> (i.e. #fields). (And soon if I have the first version of
>> FFI-Callback to show you :)
>>
>> Because those aliases become fully transparent after compiling the method
>> for the FFI call into an instance of ExternalLibraryFunction and after
>> compiling the field specs of external structures into compiledSpec for
>> struct types.
>>
>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',
>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'
>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI
>> vocabulary. Nothing more. Nothing less.
>>
>> Having that, please do not use class-side methods in ExternalType to
>> alias a (complex) struct type. Use them only for renaming atomic types.
>> Please. We may want to add checks to enforce that.
>>
>> I suppose that this discussion focuses on type aliases that are
>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>> thus get their own struct type. So, let's ignore this other mechanism for
>> now.
>>
>> > This has one advantage: type safety. You cannot pass a Foo to a
>> function expecting a Bar, even if they both alias int...
>>
>> Yes! So, we are now talking about type aliases to atomic types. The
>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>> this:
>>
>> (...forgive my "creative" representation of WordArray here...)
>>
>> compiledSpec: 0A 04 00 01
>> referentClass: Foo
>>
>> How does the atomic type for 'char' look like?
>>
>> compiledSpec: 0A 04 00 01
>> referentClass: nil
>>
>> Consequently, argument coercing will fail if you pass a 'char' when a
>> 'Foo' is expected. Because the referentClass does not match --- even if the
>> compiledSpec does match.
>>
>> > But this has one major drawback: with current FFI, you cannot pass a
>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>> an integer type...
>>
>> That's correct. If the origin of such an argument is from within the
>> image, you have to wrap it. But when returned from another call ... well
>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>
>> > FFI plugin is clever enough to recognize an atomic type alias and
>> simply return a Smalltalk Integer...
>> > So it's not Bar all the way down, I have to manually wrap Bar...
>> > This is not consistent.
>>
>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>> because it does so for argument coercing.
>>
>> Now I understand why the code generation of struct-field accessors was so
>> apparently broken for me. I changed that so that having an alias type as
>> part of a larger struct will automatically wrap that for you into the alias
>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>> ... Ha ha. ;-) #ffiLongVsInt.
>>
>> It should be "Bar all the way down". Definitely.
>>
>> > If we return an int, we must accept an int, otherwise, we cannot chain
>> FFI calls...
>>
>> Well, not for type aliases. I would like keep the path of type safety
>> here, you mentioned above.
>>
>> Just fix the FFI plugin to respect referentClass when packaging the
>> return value.
>>
>> > I could use the lightweight alias instead, [...]
>>
>> Please don't. See above. Those "lightweight alias" are for renaming
>> atomic types only. Let's keep it simple and try to address you concern here
>> with struct types and ExternalTypeAlias. :-)
>>
>> > there was another usage where it was convenient to use
>> ExternalTypeAlias: enum.
>> > Indeed, I simply defined all enum symbols as class side method.
>>
>> I like that! It reads like a next step for ExternalPool. Once you have
>> extracted the constant values from header files, and once you have those
>> values for your platform in classVars in your external pool, use that pool
>> to define enums through ExternalStructure.
>>
>> Like this:
>>
>> ExternalTypeAlias subclass: #MyEnumBar
>> instanceVariableNames: ''
>> classVariableNames: ''
>> poolDictionaries: 'HDF5Pool'
>> category: 'HDF5'
>>
>> MyEnumBar class >> #poolFields
>> "
>> self defineFields.
>> "
>> ^ #(
>> (beg HDF5_BEG)
>> (mid HDF5_MID)
>> (end HDF5_END)
>> )
>>
>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>> external-pool definitions. See class comment in ExternalPool to get started.
>>
>> Note that #poolFields would translate to class-side methods as you
>> suggested.
>>
>> > If I use a simpler alias, where are those ids going to?
>>
>> Please don't. See above. :-)
>>
>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>> inputs/outputs behave consistently.
>>
>> Absolutely!
>>
>> > - should a function expecting an ExternalTypeAlias of atomic type
>> accept an immediate value of compatible type?
>>
>> Considering type safety, I think not.
>>
>> Well ... since this is a convenience issue and thus not about saving
>> run-time cost ... What about adding a callback into the image to react on
>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>> stored in ExternalLibraryFunction since this is accessible to the plugin
>> anyway?
>>
>>
>>
>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>
>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>> FFI-Callback. But ignore me here :)
>>
>> > - should a function returning an ExternalTypeAlias of atomic type
>> instantiate the Alias?
>>
>> Yes, please! See above.
>>
>> If you want to trade type safety in for a performance gain, just use a
>> "lightweight alias" as explained above. And package that return value
>> manually.
>>
>> Best,
>> Marcel
>>
>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:
>> Hi all,
>> following the question of Marcel about usage of ExternalTypeAlias:
>>
>> Type alias are subclasses of ExternalStructure.
>> I used them in HDF5 because there was no other means to define an alias
>> at the time.
>> Now we have another mean by adding a class side method in ExternalType (I
>> would have rather imagined the usage of some annotation to get a
>> declarative style)
>>
>> This has one advantage: type safety. You cannot pass a Foo to a function
>> expecting a Bar, even if they both alias int...
>>
>> But this has one major drawback: with current FFI, you cannot pass a
>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>> an integer type...
>>
>> Thus, you have to wrap every Bar argument into a Bar instance...
>> Gasp, this is going too far...
>>
>> What about functions returning such alias?
>> Function returning struct by value allocate a struct, but it's not the
>> case here, FFI plugin is clever enough to recognize an atomic type alias
>> and simply return a Smalltalk Integer...
>> So it's not Bar all the way down, I have to manually wrap Bar...
>> This is not consistent.
>> If we return an int, we must accept an int, otherwise, we cannot chain
>> FFI calls...
>>
>> I could use the lightweight alias instead, but there was another usage
>> where it was convenient to use ExternalTypeAlias: enum.
>> Indeed, I simply defined all enum symbols as class side method. For
>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>> This way, I normally can pass a Bar mid to an external function.
>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>
>> If I use a simpler alias, where are those ids going to?
>>
>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>> inputs/outputs behave consistently.
>> The choice is this one:
>> - should a function expecting an ExternalTypeAlias of atomic type accept
>> an immediate value of compatible type? (the permissive implementation)
>> - should a function returning an ExternalTypeAlias of atomic type
>> instantiate the Alias? (the typesafe implementation, with higher runtime
>> cost due to pressure on GC)
>>
>>
>>
>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!

Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :

Hi Marcel,

thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01

referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01

referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

> So it's not Bar all the way down, I have to manually wrap Bar...

> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.

> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar

instanceVariableNames: ''

classVariableNames: ''

poolDictionaries: 'HDF5Pool'

category: 'HDF5'

MyEnumBar class >> #poolFields

   "

   self defineFields.

   "

   ^ #(

      (beg HDF5_BEG)

      (mid HDF5_MID)

      (end HDF5_END)

   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,

Marcel

Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:

Hi all,

following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.

I used them in HDF5 because there was no other means to define an alias at the time.

Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...

Gasp, this is going too far...

What about functions returning such alias?

Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

So it's not Bar all the way down, I have to manually wrap Bar...

This is not consistent.

If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.

Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.

This way, I normally can pass a Bar mid to an external function.

(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.

 The choice is this one:

- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)

- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...

The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :

 

> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...

The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).

Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.

Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!

It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.

I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches

(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,

Marcel

Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!

Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :

Hi Marcel,

thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01

referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01

referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

> So it's not Bar all the way down, I have to manually wrap Bar...

> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.

> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar

instanceVariableNames: ''

classVariableNames: ''

poolDictionaries: 'HDF5Pool'

category: 'HDF5'

MyEnumBar class >> #poolFields

   "

   self defineFields.

   "

   ^ #(

      (beg HDF5_BEG)

      (mid HDF5_MID)

      (end HDF5_END)

   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,

Marcel

Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:

Hi all,

following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.

I used them in HDF5 because there was no other means to define an alias at the time.

Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...

Gasp, this is going too far...

What about functions returning such alias?

Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

So it's not Bar all the way down, I have to manually wrap Bar...

This is not consistent.

If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.

Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.

This way, I normally can pass a Bar mid to an external function.

(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.

 The choice is this one:

- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)

- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:

> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.

We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

> or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,

Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...

The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :

 

> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...

The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).

Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.

Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!

It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.

I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches

(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,

Marcel

Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!

Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :

Hi Marcel,

thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01

referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01

referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

> So it's not Bar all the way down, I have to manually wrap Bar...

> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.

> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar

instanceVariableNames: ''

classVariableNames: ''

poolDictionaries: 'HDF5Pool'

category: 'HDF5'

MyEnumBar class >> #poolFields

   "

   self defineFields.

   "

   ^ #(

      (beg HDF5_BEG)

      (mid HDF5_MID)

      (end HDF5_END)

   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,

Marcel

Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:

Hi all,

following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.

I used them in HDF5 because there was no other means to define an alias at the time.

Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...

Gasp, this is going too far...

What about functions returning such alias?

Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

So it's not Bar all the way down, I have to manually wrap Bar...

This is not consistent.

If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.

Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.

This way, I normally can pass a Bar mid to an external function.

(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.

 The choice is this one:

- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)

- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)

Am 19.06.2020 17:47:36 schrieb Marcel Taeumel <[hidden email]>:

Hi Nicolas.

Maybe one more thought on

ExternalData<handle, type>

There is no need to have

ExternalData<1312301580, int>

because it can just be 1312301580.

On the other hand:

ExternalData<#[ 12 34 56 78 ], int>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is already in Squeak's object memory.

Finally:

ExternalData<@00F3A3EE, int*>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is in external memory.

If you want to make an array out of it, make use of the "size" field in ExternalData.

If you want to make a pointer array out of it, change the type to int**

ExternalData<@00F3A3EE, int**>

So, what does this mean then:

ExternalData<#[ 12 34 56 78 ], int*>

Well, it tells you that there is a pointer to an int stored in a byte array. This comes very handy if you think about type aliases to pointer types such as "typedef int* INT_P". Because then it would be

ExternalData<#[ 12 34 56 78 ], INT_P>

To where this pointer points to? Probably garbage. I just made it up for this example. :-D

Best,

Marcel

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:

> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.

We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

> or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,

Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...

The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :

 

> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...

The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).

Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.

Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!

It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.

I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches

(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,

Marcel

Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!

Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :

Hi Marcel,

thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01

referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01

referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

> So it's not Bar all the way down, I have to manually wrap Bar...

> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.

> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar

instanceVariableNames: ''

classVariableNames: ''

poolDictionaries: 'HDF5Pool'

category: 'HDF5'

MyEnumBar class >> #poolFields

   "

   self defineFields.

   "

   ^ #(

      (beg HDF5_BEG)

      (mid HDF5_MID)

      (end HDF5_END)

   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,

Marcel

Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:

Hi all,

following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.

I used them in HDF5 because there was no other means to define an alias at the time.

Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...

Gasp, this is going too far...

What about functions returning such alias?

Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

So it's not Bar all the way down, I have to manually wrap Bar...

This is not consistent.

If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.

Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.

This way, I normally can pass a Bar mid to an external function.

(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.

 The choice is this one:

- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)

- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)

Am 19.06.2020 17:52:21 schrieb Marcel Taeumel <[hidden email]>:

> If you want to make a pointer array out of it, change the type to int**

Correction: If you happen to now, that this external address points to an array of pointers to int, then you would use int* and not int**. You cannot just "decide" what the data behind that external address should be. .. you know what I mean. ;-)

best,

Marcel

Am 19.06.2020 17:47:36 schrieb Marcel Taeumel <[hidden email]>:

Hi Nicolas.

Maybe one more thought on

ExternalData<handle, type>

There is no need to have

ExternalData<1312301580, int>

because it can just be 1312301580.

On the other hand:

ExternalData<#[ 12 34 56 78 ], int>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is already in Squeak's object memory.

Finally:

ExternalData<@00F3A3EE, int*>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is in external memory.

If you want to make an array out of it, make use of the "size" field in ExternalData.

If you want to make a pointer array out of it, change the type to int**

ExternalData<@00F3A3EE, int**>

So, what does this mean then:

ExternalData<#[ 12 34 56 78 ], int*>

Well, it tells you that there is a pointer to an int stored in a byte array. This comes very handy if you think about type aliases to pointer types such as "typedef int* INT_P". Because then it would be

ExternalData<#[ 12 34 56 78 ], INT_P>

To where this pointer points to? Probably garbage. I just made it up for this example. :-D

Best,

Marcel

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:

> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.

We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

> or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,

Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...

The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :

 

> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...

The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).

Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.

Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!

It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.

I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches

(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,

Marcel

Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!

Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :

Hi Marcel,

thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01

referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01

referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

> So it's not Bar all the way down, I have to manually wrap Bar...

> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.

> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar

instanceVariableNames: ''

classVariableNames: ''

poolDictionaries: 'HDF5Pool'

category: 'HDF5'

MyEnumBar class >> #poolFields

   "

   self defineFields.

   "

   ^ #(

      (beg HDF5_BEG)

      (mid HDF5_MID)

      (end HDF5_END)

   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,

Marcel

Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:

Hi all,

following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.

I used them in HDF5 because there was no other means to define an alias at the time.

Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...

Gasp, this is going too far...

What about functions returning such alias?

Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

So it's not Bar all the way down, I have to manually wrap Bar...

This is not consistent.

If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.

Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.

This way, I normally can pass a Bar mid to an external function.

(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.

 The choice is this one:

- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)

- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)

Am 19.06.2020 17:53:21 schrieb Marcel Taeumel <[hidden email]>:

> then you would use int* and not int**

My bad: "  then you would use int** and not int*  " Sorry for the noise -.-"

Am 19.06.2020 17:52:21 schrieb Marcel Taeumel <[hidden email]>:

> If you want to make a pointer array out of it, change the type to int**

Correction: If you happen to now, that this external address points to an array of pointers to int, then you would use int* and not int**. You cannot just "decide" what the data behind that external address should be. .. you know what I mean. ;-)

best,

Marcel

Am 19.06.2020 17:47:36 schrieb Marcel Taeumel <[hidden email]>:

Hi Nicolas.

Maybe one more thought on

ExternalData<handle, type>

There is no need to have

ExternalData<1312301580, int>

because it can just be 1312301580.

On the other hand:

ExternalData<#[ 12 34 56 78 ], int>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is already in Squeak's object memory.

Finally:

ExternalData<@00F3A3EE, int*>

Tells you that (1) this data wants to be interpreted as signed int and (2) this data is in external memory.

If you want to make an array out of it, make use of the "size" field in ExternalData.

If you want to make a pointer array out of it, change the type to int**

ExternalData<@00F3A3EE, int**>

So, what does this mean then:

ExternalData<#[ 12 34 56 78 ], int*>

Well, it tells you that there is a pointer to an int stored in a byte array. This comes very handy if you think about type aliases to pointer types such as "typedef int* INT_P". Because then it would be

ExternalData<#[ 12 34 56 78 ], INT_P>

To where this pointer points to? Probably garbage. I just made it up for this example. :-D

Best,

Marcel

Am 19.06.2020 17:15:23 schrieb Marcel Taeumel <[hidden email]>:

> The type should better describe the contents of the handle (be it an
> ExternalAddress or a zone of object memory (ByteArray)).
> Typically, if you have an external (global) variable of type foo (extern
> foo my_var;) then the natural type is foo.
> Having foo*, would mean that we handle an array of foo*, or a pointer to
> foo* (foo**).
> This way, we make no difference between type surrogates
> (ExternalStruct/ExternalTypeAlias) and other atomic cases.

We have the "size" field in ExternalData encode whether "int*" points to a single or multiple things. No need to treat a global variable sitting in external memory any different here.  ... IMO :-)

I know, that this is my personal mental model for Squeak FFI to treat pointer types as things in external memory and non-pointer types as things in Squeak's object memory. The latter includes ByteArray and Integer and Float and also IntegerArray etc. Just not ExternalAddress.

This basically comes from the fact that, until now, the FFI plugin gave me a ByteArray in return when I told it to be "Foo", not "Foo*". So, non-pointer type means ByteArray. Pointer-type means ExternalAddress.

Type aliases are not really different here. Each alias as a corresponding struct type, which has again a non-pointer variant and a pointer variant.

Type aliases to pointer types are just special because they store the pointer address in a byte array. The corresponding non-pointer type will do fine for such aliases. Their pointer type is not relevant in general, I suppose.

> Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

Well, I have a simple idea on how to represent foo** (and other dimensions) without the FFI plugin needing to know. ... unless you are concerned about type safety for that, too. Then we would really need to encode that in the compiledSpec.

If not, ExternalData with foo** type will just answer ExternalData with foo* when asked via "at: 1" for example. :-)

> or an Alien as we may want to support that too.

I don't think that's necessary. Alien has its own call-out mechanism through the IA32ABI plugin and also a different layout/usage of ByteArray.

Best,

Marcel

Am 19.06.2020 16:40:50 schrieb Nicolas Cellier <[hidden email]>:

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel a
écrit :

>
> > Err, I move this thread to Opensmalltalk VM dev, because it is not
> Squeak specific!
>
> Absolutely. :-)
>
> (On a side note: I think that ExternalData should always have a pointer
> type in its "type" field unless "handle" is a ByteArray instead of
> ExternalAddress)
>
> Let me disagree here...

The type should better describe the contents of the handle (be it an
ExternalAddress or a zone of object memory (ByteArray)).
Typically, if you have an external (global) variable of type foo (extern
foo my_var;) then the natural type is foo.
Having foo*, would mean that we handle an array of foo*, or a pointer to
foo* (foo**).
This way, we make no difference between type surrogates
(ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should
> never ever be an atomic Smalltalk object (Integer or Float) but always
> either ByteArray or ExternalAddress. ... which makes it different from what
> "handle" can be in ExternalStructs that represent type aliases. :-)
>
> Yes!
It could be an ExternalAddress, or a zone of object memory (ByteArray) or
an Alien as we may want to support that too.
I'd be inclined to support doubleByte, word, and doubleWord arrays if the
size of atomic type matches
(Typically passing a DoubleByteArray, WordArray, a Float32Array, a
Float64Array to a short */int */float*/double*).

Best,

> Marcel
>
> Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <>
> [hidden email]>:
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
> > Hi Marcel,
> > thanks for your vote. Anyone else?
> >
> > Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
> > écrit :
> >
> >> Hi Nicolas, hi all! :-)
> >>
> >> > Type alias are subclasses of ExternalStructure.
> >>
> >> And with it, they get their own instance of ExternalType, which is
> >> managed in the classVar StructTypes. Let's call them struct type.
> >>
> >> > Now we have another mean by adding a class side method in
> ExternalType.
> >>
> >> That kind of aliasing is for compile-time type referencing only -- such
> >> as in FFI calls (i.e. pragmas) or struct-field definitions
> >> (i.e. #fields). (And soon if I have the first version of
> >> FFI-Callback to show you :)
> >>
> >> Because those aliases become fully transparent after compiling the
> method
> >> for the FFI call into an instance of ExternalLibraryFunction and after
> >> compiling the field specs of external structures into compiledSpec for
> >> struct types.
> >>
> >> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
> 'long',
> >> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
> 'int32_t'
> >> also becomes a 'long'. So, it just translates C vocabulary into Squeak
> FFI
> >> vocabulary. Nothing more. Nothing less.
> >>
> >> Having that, please do not use class-side methods in ExternalType to
> >> alias a (complex) struct type. Use them only for renaming atomic types.
> >> Please. We may want to add checks to enforce that.
> >>
> >> I suppose that this discussion focuses on type aliases that are
> >> represented as subclasses of ExternalStructure (or ExternalTypeAlias)
> and
> >> thus get their own struct type. So, let's ignore this other mechanism
> for
> >> now.
> >>
> >> > This has one advantage: type safety. You cannot pass a Foo to a
> >> function expecting a Bar, even if they both alias int...
> >>
> >> Yes! So, we are now talking about type aliases to atomic types. The
> >> struct type you get when aliasing a 'char' as Foo, for example, looks
> like
> >> this:
> >>
> >> (...forgive my "creative" representation of WordArray here...)
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: Foo
> >>
> >> How does the atomic type for 'char' look like?
> >>
> >> compiledSpec: 0A 04 00 01
> >> referentClass: nil
> >>
> >> Consequently, argument coercing will fail if you pass a 'char' when a
> >> 'Foo' is expected. Because the referentClass does not match --- even if
> the
> >> compiledSpec does match.
> >>
> >> > But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> That's correct. If the origin of such an argument is from within the
> >> image, you have to wrap it. But when returned from another call ...
> well
> >> ... that wrapping should have happened in the plugin ... But read on!
> :-)
> >>
> >> > FFI plugin is clever enough to recognize an atomic type alias and
> >> simply return a Smalltalk Integer...
> >> > So it's not Bar all the way down, I have to manually wrap Bar...
> >> > This is not consistent.
> >>
> >> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
> >> because it does so for argument coercing.
> >>
> >> Now I understand why the code generation of struct-field accessors was
> so
> >> apparently broken for me. I changed that so that having an alias type
> as
> >> part of a larger struct will automatically wrap that for you into the
> alias
> >> structure. :-) Even if you are just aliasing a plain 'int' ... or
> 'long'
> >> ... Ha ha. ;-) #ffiLongVsInt.
> >>
> >> It should be "Bar all the way down". Definitely.
> >>
> >> > If we return an int, we must accept an int, otherwise, we cannot
> chain
> >> FFI calls...
> >>
> >> Well, not for type aliases. I would like keep the path of type safety
> >> here, you mentioned above.
> >>
> >> Just fix the FFI plugin to respect referentClass when packaging the
> >> return value.
> >>
> >> > I could use the lightweight alias instead, [...]
> >>
> >> Please don't. See above. Those "lightweight alias" are for renaming
> >> atomic types only. Let's keep it simple and try to address you concern
> here
> >> with struct types and ExternalTypeAlias. :-)
> >>
> >> > there was another usage where it was convenient to use
> >> ExternalTypeAlias: enum.
> >> > Indeed, I simply defined all enum symbols as class side method.
> >>
> >> I like that! It reads like a next step for ExternalPool. Once you have
> >> extracted the constant values from header files, and once you have
> those
> >> values for your platform in classVars in your external pool, use that
> pool
> >> to define enums through ExternalStructure.
> >>
> >> Like this:
> >>
> >> ExternalTypeAlias subclass: #MyEnumBar
> >> instanceVariableNames: ''
> >> classVariableNames: ''
> >> poolDictionaries: 'HDF5Pool'
> >> category: 'HDF5'
> >>
> >> MyEnumBar class >> #poolFields
> >> "
> >> self defineFields.
> >> "
> >> ^ #(
> >> (beg HDF5_BEG)
> >> (mid HDF5_MID)
> >> (end HDF5_END)
> >> )
> >>
> >> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
> >> external-pool definitions. See class comment in ExternalPool to get
> started.
> >>
> >> Note that #poolFields would translate to class-side methods as you
> >> suggested.
> >>
> >> > If I use a simpler alias, where are those ids going to?
> >>
> >> Please don't. See above. :-)
> >>
> >> > IMO we cannot keep status quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >>
> >> Absolutely!
> >>
> >> > - should a function expecting an ExternalTypeAlias of atomic type
> >> accept an immediate value of compatible type?
> >>
> >> Considering type safety, I think not.
> >>
> >> Well ... since this is a convenience issue and thus not about saving
> >> run-time cost ... What about adding a callback into the image to react
> on
> >> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
> >> stored in ExternalLibraryFunction since this is accessible to the
> plugin
> >> anyway?
> >>
> >>
> >>
> >> ... would it be possible? Like that #doesNotUnderstand: callback?
> >>
> >> I would love to do ad-hoc packaging of arguments. (Also thinking about
> >> FFI-Callback. But ignore me here :)
> >>
> >> > - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias?
> >>
> >> Yes, please! See above.
> >>
> >> If you want to trade type safety in for a performance gain, just use a
> >> "lightweight alias" as explained above. And package that return value
> >> manually.
> >>
> >> Best,
> >> Marcel
> >>
> >> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
> >> [hidden email]>:
> >> Hi all,
> >> following the question of Marcel about usage of ExternalTypeAlias:
> >>
> >> Type alias are subclasses of ExternalStructure.
> >> I used them in HDF5 because there was no other means to define an alias
> >> at the time.
> >> Now we have another mean by adding a class side method in ExternalType
> (I
> >> would have rather imagined the usage of some annotation to get a
> >> declarative style)
> >>
> >> This has one advantage: type safety. You cannot pass a Foo to a
> function
> >> expecting a Bar, even if they both alias int...
> >>
> >> But this has one major drawback: with current FFI, you cannot pass a
> >> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias
> for
> >> an integer type...
> >>
> >> Thus, you have to wrap every Bar argument into a Bar instance...
> >> Gasp, this is going too far...
> >>
> >> What about functions returning such alias?
> >> Function returning struct by value allocate a struct, but it's not the
> >> case here, FFI plugin is clever enough to recognize an atomic type
> alias
> >> and simply return a Smalltalk Integer...
> >> So it's not Bar all the way down, I have to manually wrap Bar...
> >> This is not consistent.
> >> If we return an int, we must accept an int, otherwise, we cannot chain
> >> FFI calls...
> >>
> >> I could use the lightweight alias instead, but there was another usage
> >> where it was convenient to use ExternalTypeAlias: enum.
> >> Indeed, I simply defined all enum symbols as class side method. For
> >> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type
> Bar
> >> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
> >> This way, I normally can pass a Bar mid to an external function.
> >> (currently, we must defne mid ^Bar new value: 1; yourself)
> >>
> >> If I use a simpler alias, where are those ids going to?
> >>
> >> IMO we cannot keep statu quo in the plugin, we gotta to make the
> >> inputs/outputs behave consistently.
> >> The choice is this one:
> >> - should a function expecting an ExternalTypeAlias of atomic type
> accept
> >> an immediate value of compatible type? (the permissive implementation)
> >> - should a function returning an ExternalTypeAlias of atomic type
> >> instantiate the Alias? (the typesafe implementation, with higher
> runtime
> >> cost due to pressure on GC)
> >>
> >>
> >>
> >>
> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak
> specific!
>
>
>
> Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
> [hidden email]> a écrit :
>
>> Hi Marcel,
>> thanks for your vote. Anyone else?
>>
>> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a
>> écrit :
>>
>>>
>>> Hi Nicolas, hi all! :-)
>>>
>>> > Type alias are subclasses of ExternalStructure.
>>>
>>> And with it, they get their own instance of ExternalType, which is
>>> managed in the classVar StructTypes. Let's call them struct type.
>>>
>>> > Now we have another mean by adding a class side method in ExternalType.
>>>
>>> That kind of aliasing is for compile-time type referencing only -- such
>>> as in FFI calls (i.e. pragmas) or struct-field definitions
>>> (i.e. #fields). (And soon if I have the first version of
>>> FFI-Callback to show you :)
>>>
>>> Because those aliases become fully transparent after compiling the
>>> method for the FFI call into an instance of ExternalLibraryFunction and
>>> after compiling the field specs of external structures into compiledSpec
>>> for struct types.
>>>
>>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a
>>> 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And
>>> 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into
>>> Squeak FFI vocabulary. Nothing more. Nothing less.
>>>
>>> Having that, please do not use class-side methods in ExternalType to
>>> alias a (complex) struct type. Use them only for renaming atomic types.
>>> Please. We may want to add checks to enforce that.
>>>
>>> I suppose that this discussion focuses on type aliases that are
>>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and
>>> thus get their own struct type. So, let's ignore this other mechanism for
>>> now.
>>>
>>> > This has one advantage: type safety. You cannot pass a Foo to a
>>> function expecting a Bar, even if they both alias int...
>>>
>>> Yes! So, we are now talking about type aliases to atomic types. The
>>> struct type you get when aliasing a 'char' as Foo, for example, looks like
>>> this:
>>>
>>> (...forgive my "creative" representation of WordArray here...)
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: Foo
>>>
>>> How does the atomic type for 'char' look like?
>>>
>>> compiledSpec: 0A 04 00 01
>>> referentClass: nil
>>>
>>> Consequently, argument coercing will fail if you pass a 'char' when a
>>> 'Foo' is expected. Because the referentClass does not match --- even if the
>>> compiledSpec does match.
>>>
>>> > But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> That's correct. If the origin of such an argument is from within the
>>> image, you have to wrap it. But when returned from another call ... well
>>> ... that wrapping should have happened in the plugin ... But read on! :-)
>>>
>>> > FFI plugin is clever enough to recognize an atomic type alias and
>>> simply return a Smalltalk Integer...
>>> > So it's not Bar all the way down, I have to manually wrap Bar...
>>> > This is not consistent.
>>>
>>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass
>>> because it does so for argument coercing.
>>>
>>> Now I understand why the code generation of struct-field accessors was
>>> so apparently broken for me. I changed that so that having an alias type as
>>> part of a larger struct will automatically wrap that for you into the alias
>>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'
>>> ... Ha ha. ;-) #ffiLongVsInt.
>>>
>>> It should be "Bar all the way down". Definitely.
>>>
>>> > If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> Well, not for type aliases. I would like keep the path of type safety
>>> here, you mentioned above.
>>>
>>> Just fix the FFI plugin to respect referentClass when packaging the
>>> return value.
>>>
>>> > I could use the lightweight alias instead, [...]
>>>
>>> Please don't. See above. Those "lightweight alias" are for renaming
>>> atomic types only. Let's keep it simple and try to address you concern here
>>> with struct types and ExternalTypeAlias. :-)
>>>
>>> > there was another usage where it was convenient to use
>>> ExternalTypeAlias: enum.
>>> > Indeed, I simply defined all enum symbols as class side method.
>>>
>>> I like that! It reads like a next step for ExternalPool. Once you have
>>> extracted the constant values from header files, and once you have those
>>> values for your platform in classVars in your external pool, use that pool
>>> to define enums through ExternalStructure.
>>>
>>> Like this:
>>>
>>> ExternalTypeAlias subclass: #MyEnumBar
>>> instanceVariableNames: ''
>>> classVariableNames: ''
>>> poolDictionaries: 'HDF5Pool'
>>> category: 'HDF5'
>>>
>>> MyEnumBar class >> #poolFields
>>> "
>>> self defineFields.
>>> "
>>> ^ #(
>>> (beg HDF5_BEG)
>>> (mid HDF5_MID)
>>> (end HDF5_END)
>>> )
>>>
>>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via
>>> external-pool definitions. See class comment in ExternalPool to get started.
>>>
>>> Note that #poolFields would translate to class-side methods as you
>>> suggested.
>>>
>>> > If I use a simpler alias, where are those ids going to?
>>>
>>> Please don't. See above. :-)
>>>
>>> > IMO we cannot keep status quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>>
>>> Absolutely!
>>>
>>> > - should a function expecting an ExternalTypeAlias of atomic type
>>> accept an immediate value of compatible type?
>>>
>>> Considering type safety, I think not.
>>>
>>> Well ... since this is a convenience issue and thus not about saving
>>> run-time cost ... What about adding a callback into the image to react on
>>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be
>>> stored in ExternalLibraryFunction since this is accessible to the plugin
>>> anyway?
>>>
>>>
>>>
>>> ... would it be possible? Like that #doesNotUnderstand: callback?
>>>
>>> I would love to do ad-hoc packaging of arguments. (Also thinking about
>>> FFI-Callback. But ignore me here :)
>>>
>>> > - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias?
>>>
>>> Yes, please! See above.
>>>
>>> If you want to trade type safety in for a performance gain, just use a
>>> "lightweight alias" as explained above. And package that return value
>>> manually.
>>>
>>> Best,
>>> Marcel
>>>
>>>
>>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>>> [hidden email]>:
>>>
>>> Hi all,
>>> following the question of Marcel about usage of ExternalTypeAlias:
>>>
>>> Type alias are subclasses of ExternalStructure.
>>> I used them in HDF5 because there was no other means to define an alias
>>> at the time.
>>> Now we have another mean by adding a class side method in ExternalType
>>> (I would have rather imagined the usage of some annotation to get a
>>> declarative style)
>>>
>>> This has one advantage: type safety. You cannot pass a Foo to a function
>>> expecting a Bar, even if they both alias int...
>>>
>>> But this has one major drawback: with current FFI, you cannot pass a
>>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for
>>> an integer type...
>>>
>>> Thus, you have to wrap every Bar argument into a Bar instance...
>>> Gasp, this is going too far...
>>>
>>> What about functions returning such alias?
>>> Function returning struct by value allocate a struct, but it's not the
>>> case here, FFI plugin is clever enough to recognize an atomic type alias
>>> and simply return a Smalltalk Integer...
>>> So it's not Bar all the way down, I have to manually wrap Bar...
>>> This is not consistent.
>>> If we return an int, we must accept an int, otherwise, we cannot chain
>>> FFI calls...
>>>
>>> I could use the lightweight alias instead, but there was another usage
>>> where it was convenient to use ExternalTypeAlias: enum.
>>> Indeed, I simply defined all enum symbols as class side method. For
>>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar
>>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.
>>> This way, I normally can pass a Bar mid to an external function.
>>> (currently, we must defne mid ^Bar new value: 1; yourself)
>>>
>>> If I use a simpler alias, where are those ids going to?
>>>
>>> IMO we cannot keep statu quo in the plugin, we gotta to make the
>>> inputs/outputs behave consistently.
>>> The choice is this one:
>>> - should a function expecting an ExternalTypeAlias of atomic type accept
>>> an immediate value of compatible type? (the permissive implementation)
>>> - should a function returning an ExternalTypeAlias of atomic type
>>> instantiate the Alias? (the typesafe implementation, with higher runtime
>>> cost due to pressure on GC)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Le ven. 19 juin 2020 à 16:16, Marcel Taeumel <[hidden email]> a écrit :

 

> Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific! 

Absolutely. :-)

(On a side note: I think that ExternalData should always have a pointer type in its "type" field unless "handle" is a ByteArray instead of ExternalAddress)

Let me disagree here...

The type should better describe the contents of the handle (be it an ExternalAddress or a zone of object memory (ByteArray)).

Typically, if you have an external (global) variable of type foo (extern foo my_var;) then the natural type is foo.

Having foo*, would mean that we handle an array of foo*, or a pointer to foo* (foo**).

This way, we make no difference between type surrogates (ExternalStruct/ExternalTypeAlias) and other atomic cases.

(On another side note: I also think that "handle" in ExternalData should never ever be an atomic Smalltalk object (Integer or Float) but always either ByteArray or ExternalAddress. ... which makes it different from what "handle" can be in ExternalStructs that represent type aliases. :-)

Yes!

It could be an ExternalAddress, or a zone of object memory (ByteArray) or an Alien as we may want to support that too.

I'd be inclined to support doubleByte, word, and doubleWord arrays if the size of atomic type matches

(Typically passing a DoubleByteArray, WordArray, a Float32Array, a Float64Array to a short */int */float*/double*).

Best,

Marcel

Am 19.06.2020 16:03:10 schrieb Nicolas Cellier <[hidden email]>:

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak

specific!







Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <>
[hidden email]> a écrit :



> Hi Marcel,

> thanks for your vote. Anyone else?

>

> Le ven. 19 juin 2020 à 09:30, Marcel Taeumel a

> écrit :

>

>> Hi Nicolas, hi all! :-)

>>

>> > Type alias are subclasses of ExternalStructure.

>>

>> And with it, they get their own instance of ExternalType, which is

>> managed in the classVar StructTypes. Let's call them struct type.

>>

>> > Now we have another mean by adding a class side method in ExternalType.

>>

>> That kind of aliasing is for compile-time type referencing only -- such

>> as in FFI calls (i.e. pragmas) or struct-field definitions

>> (i.e. #fields). (And soon if I have the first version of

>> FFI-Callback to show you :)

>>

>> Because those aliases become fully transparent after compiling the method

>> for the FFI call into an instance of ExternalLibraryFunction and after

>> compiling the field specs of external structures into compiledSpec for

>> struct types.

>>

>> For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long',

>> which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t'

>> also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI

>> vocabulary. Nothing more. Nothing less.

>>

>> Having that, please do not use class-side methods in ExternalType to

>> alias a (complex) struct type. Use them only for renaming atomic types.

>> Please. We may want to add checks to enforce that.

>>

>> I suppose that this discussion focuses on type aliases that are

>> represented as subclasses of ExternalStructure (or ExternalTypeAlias) and

>> thus get their own struct type. So, let's ignore this other mechanism for

>> now.

>>

>> > This has one advantage: type safety. You cannot pass a Foo to a

>> function expecting a Bar, even if they both alias int...

>>

>> Yes! So, we are now talking about type aliases to atomic types. The

>> struct type you get when aliasing a 'char' as Foo, for example, looks like

>> this:

>>

>> (...forgive my "creative" representation of WordArray here...)

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: Foo

>>

>> How does the atomic type for 'char' look like?

>>

>> compiledSpec: 0A 04 00 01

>> referentClass: nil

>>

>> Consequently, argument coercing will fail if you pass a 'char' when a

>> 'Foo' is expected. Because the referentClass does not match --- even if the

>> compiledSpec does match.

>>

>> > But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> That's correct. If the origin of such an argument is from within the

>> image, you have to wrap it. But when returned from another call ... well

>> ... that wrapping should have happened in the plugin ... But read on! :-)

>>

>> > FFI plugin is clever enough to recognize an atomic type alias and

>> simply return a Smalltalk Integer...

>> > So it's not Bar all the way down, I have to manually wrap Bar...

>> > This is not consistent.

>>

>> Uhh! That's a bug. The FFI Plugin must take a look at referentClass

>> because it does so for argument coercing.

>>

>> Now I understand why the code generation of struct-field accessors was so

>> apparently broken for me. I changed that so that having an alias type as

>> part of a larger struct will automatically wrap that for you into the alias

>> structure. :-) Even if you are just aliasing a plain 'int' ... or 'long'

>> ... Ha ha. ;-) #ffiLongVsInt.

>>

>> It should be "Bar all the way down". Definitely.

>>

>> > If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> Well, not for type aliases. I would like keep the path of type safety

>> here, you mentioned above.

>>

>> Just fix the FFI plugin to respect referentClass when packaging the

>> return value.

>>

>> > I could use the lightweight alias instead, [...]

>>

>> Please don't. See above. Those "lightweight alias" are for renaming

>> atomic types only. Let's keep it simple and try to address you concern here

>> with struct types and ExternalTypeAlias. :-)

>>

>> > there was another usage where it was convenient to use

>> ExternalTypeAlias: enum.

>> > Indeed, I simply defined all enum symbols as class side method.

>>

>> I like that! It reads like a next step for ExternalPool. Once you have

>> extracted the constant values from header files, and once you have those

>> values for your platform in classVars in your external pool, use that pool

>> to define enums through ExternalStructure.

>>

>> Like this:

>>

>> ExternalTypeAlias subclass: #MyEnumBar

>> instanceVariableNames: ''

>> classVariableNames: ''

>> poolDictionaries: 'HDF5Pool'

>> category: 'HDF5'

>>

>> MyEnumBar class >> #poolFields

>> "

>> self defineFields.

>> "

>> ^ #(

>> (beg HDF5_BEG)

>> (mid HDF5_MID)

>> (end HDF5_END)

>> )

>>

>> Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via

>> external-pool definitions. See class comment in ExternalPool to get started.

>>

>> Note that #poolFields would translate to class-side methods as you

>> suggested.

>>

>> > If I use a simpler alias, where are those ids going to?

>>

>> Please don't. See above. :-)

>>

>> > IMO we cannot keep status quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>>

>> Absolutely!

>>

>> > - should a function expecting an ExternalTypeAlias of atomic type

>> accept an immediate value of compatible type?

>>

>> Considering type safety, I think not.

>>

>> Well ... since this is a convenience issue and thus not about saving

>> run-time cost ... What about adding a callback into the image to react on

>> FFIErrorCoercionFailed? Maybe the selector for such a callback could be

>> stored in ExternalLibraryFunction since this is accessible to the plugin

>> anyway?

>>

>>

>>

>> ... would it be possible? Like that #doesNotUnderstand: callback?

>>

>> I would love to do ad-hoc packaging of arguments. (Also thinking about

>> FFI-Callback. But ignore me here :)

>>

>> > - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias?

>>

>> Yes, please! See above.

>>

>> If you want to trade type safety in for a performance gain, just use a

>> "lightweight alias" as explained above. And package that return value

>> manually.

>>

>> Best,

>> Marcel

>>

>> Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <>
>> [hidden email]>:

>> Hi all,

>> following the question of Marcel about usage of ExternalTypeAlias:

>>

>> Type alias are subclasses of ExternalStructure.

>> I used them in HDF5 because there was no other means to define an alias

>> at the time.

>> Now we have another mean by adding a class side method in ExternalType (I

>> would have rather imagined the usage of some annotation to get a

>> declarative style)

>>

>> This has one advantage: type safety. You cannot pass a Foo to a function

>> expecting a Bar, even if they both alias int...

>>

>> But this has one major drawback: with current FFI, you cannot pass a

>> Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for

>> an integer type...

>>

>> Thus, you have to wrap every Bar argument into a Bar instance...

>> Gasp, this is going too far...

>>

>> What about functions returning such alias?

>> Function returning struct by value allocate a struct, but it's not the

>> case here, FFI plugin is clever enough to recognize an atomic type alias

>> and simply return a Smalltalk Integer...

>> So it's not Bar all the way down, I have to manually wrap Bar...

>> This is not consistent.

>> If we return an int, we must accept an int, otherwise, we cannot chain

>> FFI calls...

>>

>> I could use the lightweight alias instead, but there was another usage

>> where it was convenient to use ExternalTypeAlias: enum.

>> Indeed, I simply defined all enum symbols as class side method. For

>> example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar

>> aliasing int and having class side methods beg ^0, mid ^1, end ^2.

>> This way, I normally can pass a Bar mid to an external function.

>> (currently, we must defne mid ^Bar new value: 1; yourself)

>>

>> If I use a simpler alias, where are those ids going to?

>>

>> IMO we cannot keep statu quo in the plugin, we gotta to make the

>> inputs/outputs behave consistently.

>> The choice is this one:

>> - should a function expecting an ExternalTypeAlias of atomic type accept

>> an immediate value of compatible type? (the permissive implementation)

>> - should a function returning an ExternalTypeAlias of atomic type

>> instantiate the Alias? (the typesafe implementation, with higher runtime

>> cost due to pressure on GC)

>>

>>

>>

>>

Err, I move this thread to Opensmalltalk VM dev, because it is not Squeak specific!

Le ven. 19 juin 2020 à 15:10, Nicolas Cellier <[hidden email]> a écrit :

Hi Marcel,

thanks for your vote. Anyone else?

Le ven. 19 juin 2020 à 09:30, Marcel Taeumel <[hidden email]> a écrit :

Hi Nicolas, hi all! :-)

> Type alias are subclasses of ExternalStructure.

And with it, they get their own instance of ExternalType, which is managed in the classVar StructTypes. Let's call them struct type.

> Now we have another mean by adding a class side method in ExternalType.

That kind of aliasing is for compile-time type referencing only -- such as in FFI calls (i.e. <apicall:...> pragmas) or struct-field definitions (i.e. #fields). (And soon <callback: ...> if I have the first version of FFI-Callback to show you :)

Because those aliases become fully transparent after compiling the method for the FFI call into an instance of ExternalLibraryFunction and after compiling the field specs of external structures into compiledSpec for struct types. 

For example, on my machine in 32-bit Squeak, a 'size_t' becomes a 'long', which is Squeak FFI term for 'int'. :-) See #ffiLongVsInt. And 'int32_t' also becomes a 'long'. So, it just translates C vocabulary into Squeak FFI vocabulary. Nothing more. Nothing less.

Having that, please do not use class-side methods in ExternalType to alias a (complex) struct type. Use them only for renaming atomic types. Please. We may want to add checks to enforce that.

I suppose that this discussion focuses on type aliases that are represented as subclasses of ExternalStructure (or ExternalTypeAlias) and thus get their own struct type. So, let's ignore this other mechanism for now.

> This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

Yes! So, we are now talking about type aliases to atomic types. The struct type you get when aliasing a 'char' as Foo, for example, looks like this:

(...forgive my "creative" representation of WordArray here...)

compiledSpec: 0A 04 00 01

referentClass: Foo

How does the atomic type for 'char' look like?

compiledSpec: 0A 04 00 01

referentClass: nil

Consequently, argument coercing will fail if you pass a 'char' when a 'Foo' is expected. Because the referentClass does not match --- even if the compiledSpec does match.

> But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

That's correct. If the origin of such an argument is from within the image, you have to wrap it. But when returned from another call ... well ... that wrapping should have happened in the plugin ... But read on! :-)

> FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

> So it's not Bar all the way down, I have to manually wrap Bar...

> This is not consistent.

Uhh! That's a bug. The FFI Plugin must take a look at referentClass because it does so for argument coercing.

Now I understand why the code generation of struct-field accessors was so apparently broken for me. I changed that so that having an alias type as part of a larger struct will automatically wrap that for you into the alias structure. :-) Even if you are just aliasing a plain 'int' ... or 'long' ... Ha ha. ;-) #ffiLongVsInt.

It should be "Bar all the way down". Definitely.

> If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

Well, not for type aliases. I would like keep the path of type safety here, you mentioned above.

Just fix the FFI plugin to respect referentClass when packaging the return value.

> I could use the lightweight alias instead, [...]

Please don't. See above. Those "lightweight alias" are for renaming atomic types only. Let's keep it simple and try to address you concern here with struct types and ExternalTypeAlias. :-)

> there was another usage where it was convenient to use ExternalTypeAlias: enum.

> Indeed, I simply defined all enum symbols as class side method. 

I like that! It reads like a next step for ExternalPool. Once you have extracted the constant values from header files, and once you have those values for your platform in classVars in your external pool, use that pool to define enums through ExternalStructure.

Like this:

ExternalTypeAlias subclass: #MyEnumBar

instanceVariableNames: ''

classVariableNames: ''

poolDictionaries: 'HDF5Pool'

category: 'HDF5'

MyEnumBar class >> #poolFields

   "

   self defineFields.

   "

   ^ #(

      (beg HDF5_BEG)

      (mid HDF5_MID)

      (end HDF5_END)

   )

Note that HDF5_BEG etc. are from the HDF5Pool, which is managed via external-pool definitions. See class comment in ExternalPool to get started.

Note that #poolFields would translate to class-side methods as you suggested.

> If I use a simpler alias, where are those ids going to?

Please don't. See above. :-)

> IMO we cannot keep status quo in the plugin, we gotta to make the inputs/outputs behave consistently.

Absolutely!

> - should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type?

Considering type safety, I think not.

Well ... since this is a convenience issue and thus not about saving run-time cost ... What about adding a callback into the image to react on FFIErrorCoercionFailed? Maybe the selector for such a callback could be stored in ExternalLibraryFunction since this is accessible to the plugin anyway?

<apicall: void 'foo' (MyInt YourInt in) ifFailCoerceVia: #wrapInt: >

... would it be possible? Like that #doesNotUnderstand: callback?

I would love to do ad-hoc packaging of arguments. (Also thinking about FFI-Callback. But ignore me here :)

> - should a function returning an ExternalTypeAlias of atomic type instantiate the Alias?

Yes, please! See above.

If you want to trade type safety in for a performance gain, just use a "lightweight alias" as explained above. And package that return value manually.

Best,

Marcel

Am 18.06.2020 21:03:53 schrieb Nicolas Cellier <[hidden email]>:

Hi all,

following the question of Marcel about usage of ExternalTypeAlias:

Type alias are subclasses of ExternalStructure.

I used them in HDF5 because there was no other means to define an alias at the time.

Now we have another mean by adding a class side method in ExternalType (I would have rather imagined the usage of some annotation to get a declarative style)

This has one advantage: type safety. You cannot pass a Foo to a function expecting a Bar, even if they both alias int...

But this has one major drawback: with current FFI, you cannot pass a Smalltalk Integer, to a method expecting a Bar, even if Bar is an alias for an integer type...

Thus, you have to wrap every Bar argument into a Bar instance...

Gasp, this is going too far...

What about functions returning such alias?

Function returning struct by value allocate a struct, but it's not the case here, FFI plugin is clever enough to recognize an atomic type alias and simply return a Smalltalk Integer...

So it's not Bar all the way down, I have to manually wrap Bar...

This is not consistent.

If we return an int, we must accept an int, otherwise, we cannot chain FFI calls...

I could use the lightweight alias instead, but there was another usage where it was convenient to use ExternalTypeAlias: enum.

Indeed, I simply defined all enum symbols as class side method. For example for enum bar { beg=0, mid=1, end=2 }; I simply create a type Bar aliasing int and having class side methods beg ^0, mid ^1, end ^2.

This way, I normally can pass a Bar mid to an external function.

(currently, we must defne mid ^Bar new value: 1; yourself)

If I use a simpler alias, where are those ids going to?

IMO we cannot keep statu quo in the plugin, we gotta to make the inputs/outputs behave consistently.

 The choice is this one:

- should a function expecting an ExternalTypeAlias of atomic type accept an immediate value of compatible type? (the permissive implementation)

- should a function returning an ExternalTypeAlias of atomic type instantiate the Alias? (the typesafe implementation, with higher runtime cost due to pressure on GC)