FFI callbacks

> This community needs a repository for plugins. A directory at
> http://ftp.squeak.org ought to do it.
>
> If I want to try Alien I need the plugin. Where am I going to get that?
> I could get VMMaker, read NewSqueakIA32ABIPlugin three times to see it
> is not saying Newspeak and realize it compiles a plugin called IA32ABI,
> which - coincidentally - is the name of the plugin the error Alien
> exampleCqsort will gives me. (NewSqueakIA32ABIPlugin class>> moduleName
> returns IA32ABI).
>
> Does anybody know what the Genie plugin does? Does Paul DeBrucker need
> to store a BerkeleyDB plugin dll he got by emailing somebody on GitHub?
>
> If I understand this correctly, if such a directory existed, then all a
> person would need to do to try the above Alien code would be to download
> a pre-compiled plugin from - say -
> http://ftp.squeak/plugins/Alien/fooVersion, put it in their VM directory
> and start the image. Then check with SmalltalkImage current
> listLoadedModules to see if you're ready to go.
>
> I nominate Fabio Niephaus to be the person the community can send
> pre-compiled plugins for inclusion in such a directory. Eliot's been
> banging away about Alien for some time [1]. If a person could go, get
> the plugin and give the code shot, he could have more productive
> conversations about Alien.
>
> So, if, Eliot, you have a pre-compiled plugin for Alien, perhaps in an
> idle moment, you could email it to Fabio? And when, Fabio, you've
> created the directory in http://ftp.squeak.org you could announce it
> here on Squeak-dev?
>
> FWIW,
>
> Chris

>
> [1] http://wiki.squeak.org/squeak/uploads/6100/
>
> On 2015-09-03 7:07 PM, Eliot Miranda wrote:
> >Hi Craig,
> >
> >    you need Alien-eem.24
> >
> >Here's Alien class>>exampleCqsort
> >"Call the libc qsort function (which requires a callback)."
> >"Alien exampleCqsort"
> >"(Time millisecondsToRun: [100 timesRepeat: [Alien exampleCqsort]]) /
> >100.0"
> >| cb rand nElements sizeofDouble values orig sort |
> >rand := Random new.
> >values := Alien newC: (nElements := 100) * (sizeofDouble := 8).
> >1 to: values dataSize by: sizeofDouble do:
> >[:i| values doubleAt: i put: rand next].
> >orig := (1 to: values dataSize by: sizeofDouble) collect: [:i| values
> >doubleAt: i].
> >cb := Callback
> >signature:  #(int (*)(const void *, const void *))
> >block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign].
> >(Alien lookup: 'qsort' inLibrary: Alien libcName)
> >primFFICallResult: nil
> >with: values pointer
> >with: nElements
> >with: sizeofDouble
> >with: cb thunk.
> >sort := (1 to: values dataSize by: sizeofDouble) collect: [:i| values
> >doubleAt: i].
> >values free.
> >^orig -> sort
> >
> >The above example uses Alien to make the callout.  To use it with the
> >FFI simply pass cb pointer as the argument as is done above.
> >
> >
> >Implementation:
> >
> >The way that it works is in two parts
> >- the VM passes up a pointer to a structure from which all arguments,
> >stacked and in registers (because the VM has copied the register args
> >into the struct) can be accessed, and through which the result can be
> >returned.
> >- the image level provides marshalling methods that match the
> >signature in the callback
> >
> >So e.g. with a callback of
> >Callback
> >signature:  #(int (*)(const void *, const void *))
> >block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign]
> >the marshalling methods are in Callback's signature protocol:
> >
> >Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
> ><signature: #(int (*)(const void *, const void *)) abi: 'IA32'>
> >^callbackContext wordResult:
> >(block
> >value: (Alien forPointer: (spAlien unsignedLongAt: 1))
> >value: (Alien forPointer: (spAlien unsignedLongAt: 5)))
> >
> >where spAlien is an Alien pointing to a VMCallbackContext32.
> >
> >For ARM support we would add
> >
> >Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
> >intRegArgs: regsAlien
> ><signature: #(int (*)(const void *, const void *)) abi: 'ARMV5'>
> >^callbackContext wordResult:
> >(block
> >value: (Alien forPointer: (regsAlien unsignedLongAt: 1))
> >value: (Alien forPointer: (regsAlien unsignedLongAt: 5)))
> >
> >Basically the idea is that the selector of the method doesn't matter
> >except for the number of arguments.  What's important is the pragma
> >which defines the signature and the ABI for which this is a valid
> >marshalling method.
> >
> >When the callback is instantiated, Callback introspects to find the
> >marshalling method that matches the signature.  If one doesn't already
> >exist you can write one.  Hopefully we'll write an ABI compiler that
> >will automatically generate these marshalling methods according to the
> >platform's ABI, but for now its a manual process.  But at least it's
> >open and flexible.  When the callback is invoked the evaluator is
> >performed with the current callbackContext and pointer(s) to the
> >arguments. There is a 32-bit and a 64-bit callback context, and it can
> >have a stack pointer, integer register args and floating point
> >register args.  So it's general enough for any callback.
> >
> >To pass back the result, a value is assigned into the struct via the
> >accessor in the marshalling method and control returns to teh point
> >where teh callback comes in, and this uses a primitive to return.  
> >Inside the callbackCOntext is a jmpbuf from a setjmp.  The primitive
> >longjmp's back to the entry point in the VM which extracts the result
> >and the code for the kind of result and returns:
> >
> >Callback class>>invokeCallbackContext: vmCallbackContextAddress
> >"<Integer>" "^<FFICallbackReturnValue>"
> >"The low-level entry-point for callbacks sent from the VM/IA32ABI plugin.
> >Return via primReturnFromContext:through:.  thisContext's sender is the
> >call-out context."
> >| callbackAlien type |
> >callbackAlien := (Smalltalk wordSize = 4
> >ifTrue: [VMCallbackContext32]
> >ifFalse: [VMCallbackContext64])
> >atAddress: vmCallbackContextAddress.
> >[type := Callback evaluateCallbackForContext: callbackAlien]
> >ifCurtailed: [self error: 'attempt to non-local return across a
> >callback'].
> >type ifNil:
> >[type := 1. callbackAlien wordResult: -1].
> >callbackAlien primReturnAs: type fromContext: thisContext
> >
> >On Thu, Sep 3, 2015 at 5:22 AM, Craig Latta <[hidden email]
> ><mailto:[hidden email]>> wrote:
> >
> >
> >    Hi all--
> >
> >         I'd like to use a C shared library via FFI from Squeak or Pharo.
> >    Specifically, I want to call a C function which takes as one of its
> >    parameters a pointer to an array of callback function addresses. Does
> >    one of the FFI approaches floating around handle that, and provide a
> >    relatively pleasant mapping between Smalltalk block closures and C
> >    callback function addresses?
> >
> >         I was doing this so far in GemStone, but apparently its FFI only
> >    supports passing callback function addresses one at a time as direct
> >    parameters of C callouts. I suppose I could write a wrapper C library
> >    around the one I really want to use, that provides the callback setup
> >    interface that GemStone expects, but whenever I have to write actual C
> >    code I start to think "Hm, why don't I just use a Smalltalk VM for
> >    which
> >    I have the source?" :)  All the fancy distributed object-database
> >    stuff
> >    that my project also wants can wait for a bit.
> >
> >
> >         thanks!
> >
> >    -C
> >
> >    --
> >    Craig Latta
> >    netjam.org <http://netjam.org>
> >    +31 6 2757 7177 <tel:%2B31%20%20%206%202757%207177> (SMS ok)
> >    + 1 415 287 3547 <tel:%2B%201%20415%20%20287%203547> (no SMS)
> >
> >
> >
> >
> >
> >--
> >_,,,^..^,,,_
> >best, Eliot
> >
> >
>

> On Thu, Sep 03, 2015 at 09:05:07PM -0400, Chris Cunnington wrote:
>> This community needs a repository for plugins. A directory at
>> http://ftp.squeak.org ought to do it.
>>
>> If I want to try Alien I need the plugin. Where am I going to get that?
>> I could get VMMaker, read NewSqueakIA32ABIPlugin three times to see it
>> is not saying Newspeak and realize it compiles a plugin called IA32ABI,
>> which - coincidentally - is the name of the plugin the error Alien
>> exampleCqsort will gives me. (NewSqueakIA32ABIPlugin class>> moduleName
>> returns IA32ABI).
>>
>> Does anybody know what the Genie plugin does? Does Paul DeBrucker need
>> to store a BerkeleyDB plugin dll he got by emailing somebody on GitHub?
>>
>> If I understand this correctly, if such a directory existed, then all a
>> person would need to do to try the above Alien code would be to download
>> a pre-compiled plugin from - say -
>> http://ftp.squeak/plugins/Alien/fooVersion, put it in their VM directory
>> and start the image. Then check with SmalltalkImage current
>> listLoadedModules to see if you're ready to go.
>>
>> I nominate Fabio Niephaus to be the person the community can send
>> pre-compiled plugins for inclusion in such a directory. Eliot's been
>> banging away about Alien for some time [1]. If a person could go, get
>> the plugin and give the code shot, he could have more productive
>> conversations about Alien.
>>
>> So, if, Eliot, you have a pre-compiled plugin for Alien, perhaps in an
>> idle moment, you could email it to Fabio? And when, Fabio, you've
>> created the directory in http://ftp.squeak.org you could announce it
>> here on Squeak-dev?
>>
>> FWIW,
>>
>> Chris
> A word of caution:
>
> I think the idea is good in principle, but it needs to be accompanied
> by a commitment by someone to do the hard work of keeping track of
> dependencies on the various flavors of OS, image word size, VM word size,
> spur/V3, and so forth. If someone would like to take that on, great.
> That said, we don't want to start a project that gets 80% completed, and
> then expect "the community" to do the rest of the work.
>
> I cannot speak for Fabio, but I don't think it is fair for us to expect
> him to take on this kind of workload unless he has indicated an interest
> in doing it.
>
> Maybe a restricted repository focused just on compiled plugins known
> to work with the Spur VM on a limited range of OS flavors would be helpful
> at this stage. It probably does not belong on ftp.squeak.org, but it could
> be a very useful resource if someone (Chris Cunnington?) wants to set it up.
>
> Dave
>
>
>> [1] http://wiki.squeak.org/squeak/uploads/6100/
>>
>> On 2015-09-03 7:07 PM, Eliot Miranda wrote:
>>> Hi Craig,
>>>
>>>     you need Alien-eem.24
>>>
>>> Here's Alien class>>exampleCqsort
>>> "Call the libc qsort function (which requires a callback)."
>>> "Alien exampleCqsort"
>>> "(Time millisecondsToRun: [100 timesRepeat: [Alien exampleCqsort]]) /
>>> 100.0"
>>> | cb rand nElements sizeofDouble values orig sort |
>>> rand := Random new.
>>> values := Alien newC: (nElements := 100) * (sizeofDouble := 8).
>>> 1 to: values dataSize by: sizeofDouble do:
>>> [:i| values doubleAt: i put: rand next].
>>> orig := (1 to: values dataSize by: sizeofDouble) collect: [:i| values
>>> doubleAt: i].
>>> cb := Callback
>>> signature:  #(int (*)(const void *, const void *))
>>> block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign].
>>> (Alien lookup: 'qsort' inLibrary: Alien libcName)
>>> primFFICallResult: nil
>>> with: values pointer
>>> with: nElements
>>> with: sizeofDouble
>>> with: cb thunk.
>>> sort := (1 to: values dataSize by: sizeofDouble) collect: [:i| values
>>> doubleAt: i].
>>> values free.
>>> ^orig -> sort
>>>
>>> The above example uses Alien to make the callout.  To use it with the
>>> FFI simply pass cb pointer as the argument as is done above.
>>>
>>>
>>> Implementation:
>>>
>>> The way that it works is in two parts
>>> - the VM passes up a pointer to a structure from which all arguments,
>>> stacked and in registers (because the VM has copied the register args
>>> into the struct) can be accessed, and through which the result can be
>>> returned.
>>> - the image level provides marshalling methods that match the
>>> signature in the callback
>>>
>>> So e.g. with a callback of
>>> Callback
>>> signature:  #(int (*)(const void *, const void *))
>>> block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign]
>>> the marshalling methods are in Callback's signature protocol:
>>>
>>> Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
>>> <signature: #(int (*)(const void *, const void *)) abi: 'IA32'>
>>> ^callbackContext wordResult:
>>> (block
>>> value: (Alien forPointer: (spAlien unsignedLongAt: 1))
>>> value: (Alien forPointer: (spAlien unsignedLongAt: 5)))
>>>
>>> where spAlien is an Alien pointing to a VMCallbackContext32.
>>>
>>> For ARM support we would add
>>>
>>> Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
>>> intRegArgs: regsAlien
>>> <signature: #(int (*)(const void *, const void *)) abi: 'ARMV5'>
>>> ^callbackContext wordResult:
>>> (block
>>> value: (Alien forPointer: (regsAlien unsignedLongAt: 1))
>>> value: (Alien forPointer: (regsAlien unsignedLongAt: 5)))
>>>
>>> Basically the idea is that the selector of the method doesn't matter
>>> except for the number of arguments.  What's important is the pragma
>>> which defines the signature and the ABI for which this is a valid
>>> marshalling method.
>>>
>>> When the callback is instantiated, Callback introspects to find the
>>> marshalling method that matches the signature.  If one doesn't already
>>> exist you can write one.  Hopefully we'll write an ABI compiler that
>>> will automatically generate these marshalling methods according to the
>>> platform's ABI, but for now its a manual process.  But at least it's
>>> open and flexible.  When the callback is invoked the evaluator is
>>> performed with the current callbackContext and pointer(s) to the
>>> arguments. There is a 32-bit and a 64-bit callback context, and it can
>>> have a stack pointer, integer register args and floating point
>>> register args.  So it's general enough for any callback.
>>>
>>> To pass back the result, a value is assigned into the struct via the
>>> accessor in the marshalling method and control returns to teh point
>>> where teh callback comes in, and this uses a primitive to return.
>>> Inside the callbackCOntext is a jmpbuf from a setjmp.  The primitive
>>> longjmp's back to the entry point in the VM which extracts the result
>>> and the code for the kind of result and returns:
>>>
>>> Callback class>>invokeCallbackContext: vmCallbackContextAddress
>>> "<Integer>" "^<FFICallbackReturnValue>"
>>> "The low-level entry-point for callbacks sent from the VM/IA32ABI plugin.
>>> Return via primReturnFromContext:through:.  thisContext's sender is the
>>> call-out context."
>>> | callbackAlien type |
>>> callbackAlien := (Smalltalk wordSize = 4
>>> ifTrue: [VMCallbackContext32]
>>> ifFalse: [VMCallbackContext64])
>>> atAddress: vmCallbackContextAddress.
>>> [type := Callback evaluateCallbackForContext: callbackAlien]
>>> ifCurtailed: [self error: 'attempt to non-local return across a
>>> callback'].
>>> type ifNil:
>>> [type := 1. callbackAlien wordResult: -1].
>>> callbackAlien primReturnAs: type fromContext: thisContext
>>>
>>> On Thu, Sep 3, 2015 at 5:22 AM, Craig Latta <[hidden email]
>>> <mailto:[hidden email]>> wrote:
>>>
>>>
>>>     Hi all--
>>>
>>>          I'd like to use a C shared library via FFI from Squeak or Pharo.
>>>     Specifically, I want to call a C function which takes as one of its
>>>     parameters a pointer to an array of callback function addresses. Does
>>>     one of the FFI approaches floating around handle that, and provide a
>>>     relatively pleasant mapping between Smalltalk block closures and C
>>>     callback function addresses?
>>>
>>>          I was doing this so far in GemStone, but apparently its FFI only
>>>     supports passing callback function addresses one at a time as direct
>>>     parameters of C callouts. I suppose I could write a wrapper C library
>>>     around the one I really want to use, that provides the callback setup
>>>     interface that GemStone expects, but whenever I have to write actual C
>>>     code I start to think "Hm, why don't I just use a Smalltalk VM for
>>>     which
>>>     I have the source?" :)  All the fancy distributed object-database
>>>     stuff
>>>     that my project also wants can wait for a bit.
>>>
>>>
>>>          thanks!
>>>
>>>     -C
>>>
>>>     --
>>>     Craig Latta
>>>     netjam.org <http://netjam.org>
>>>     +31 6 2757 7177 <tel:%2B31%20%20%206%202757%207177> (SMS ok)
>>>     + 1 415 287 3547 <tel:%2B%201%20415%20%20287%203547> (no SMS)
>>>
>>>
>>>
>>>
>>>
>>> --
>>> _,,,^..^,,,_
>>> best, Eliot
>>>
>>>
>

> Perhaps the best thing is to leave it as an open question for  a few weeks
> before settling on any kind of solution. A common location for plugins is
> about as far as my thinking takes me at the moment. I'm sure the collective
> imagination of the community can come up with a better idea.
>
> Chris
>
>
> On 2015-09-03 10:07 PM, David T. Lewis wrote:
>>
>> On Thu, Sep 03, 2015 at 09:05:07PM -0400, Chris Cunnington wrote:
>>>
>>> This community needs a repository for plugins. A directory at
>>> http://ftp.squeak.org ought to do it.
>>>
>>> If I want to try Alien I need the plugin. Where am I going to get that?
>>> I could get VMMaker, read NewSqueakIA32ABIPlugin three times to see it
>>> is not saying Newspeak and realize it compiles a plugin called IA32ABI,
>>> which - coincidentally - is the name of the plugin the error Alien
>>> exampleCqsort will gives me. (NewSqueakIA32ABIPlugin class>> moduleName
>>> returns IA32ABI).
>>>
>>> Does anybody know what the Genie plugin does? Does Paul DeBrucker need
>>> to store a BerkeleyDB plugin dll he got by emailing somebody on GitHub?
>>>
>>> If I understand this correctly, if such a directory existed, then all a
>>> person would need to do to try the above Alien code would be to download
>>> a pre-compiled plugin from - say -
>>> http://ftp.squeak/plugins/Alien/fooVersion, put it in their VM directory
>>> and start the image. Then check with SmalltalkImage current
>>> listLoadedModules to see if you're ready to go.
>>>
>>> I nominate Fabio Niephaus to be the person the community can send
>>> pre-compiled plugins for inclusion in such a directory. Eliot's been
>>> banging away about Alien for some time [1]. If a person could go, get
>>> the plugin and give the code shot, he could have more productive
>>> conversations about Alien.
>>>
>>> So, if, Eliot, you have a pre-compiled plugin for Alien, perhaps in an
>>> idle moment, you could email it to Fabio? And when, Fabio, you've
>>> created the directory in http://ftp.squeak.org you could announce it
>>> here on Squeak-dev?
>>>
>>> FWIW,
>>>
>>> Chris
>>
>> A word of caution:
>>
>> I think the idea is good in principle, but it needs to be accompanied
>> by a commitment by someone to do the hard work of keeping track of
>> dependencies on the various flavors of OS, image word size, VM word size,
>> spur/V3, and so forth. If someone would like to take that on, great.
>> That said, we don't want to start a project that gets 80% completed, and
>> then expect "the community" to do the rest of the work.
>>
>> I cannot speak for Fabio, but I don't think it is fair for us to expect
>> him to take on this kind of workload unless he has indicated an interest
>> in doing it.
>>
>> Maybe a restricted repository focused just on compiled plugins known
>> to work with the Spur VM on a limited range of OS flavors would be helpful
>> at this stage. It probably does not belong on ftp.squeak.org, but it could
>> be a very useful resource if someone (Chris Cunnington?) wants to set it
>> up.
>>
>> Dave
>>
>>
>>> [1] http://wiki.squeak.org/squeak/uploads/6100/
>>>
>>> On 2015-09-03 7:07 PM, Eliot Miranda wrote:
>>>>
>>>> Hi Craig,
>>>>
>>>>     you need Alien-eem.24
>>>>
>>>> Here's Alien class>>exampleCqsort
>>>> "Call the libc qsort function (which requires a callback)."
>>>> "Alien exampleCqsort"
>>>> "(Time millisecondsToRun: [100 timesRepeat: [Alien exampleCqsort]]) /
>>>> 100.0"
>>>> | cb rand nElements sizeofDouble values orig sort |
>>>> rand := Random new.
>>>> values := Alien newC: (nElements := 100) * (sizeofDouble := 8).
>>>> 1 to: values dataSize by: sizeofDouble do:
>>>> [:i| values doubleAt: i put: rand next].
>>>> orig := (1 to: values dataSize by: sizeofDouble) collect: [:i| values
>>>> doubleAt: i].
>>>> cb := Callback
>>>> signature:  #(int (*)(const void *, const void *))
>>>> block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign].
>>>> (Alien lookup: 'qsort' inLibrary: Alien libcName)
>>>> primFFICallResult: nil
>>>> with: values pointer
>>>> with: nElements
>>>> with: sizeofDouble
>>>> with: cb thunk.
>>>> sort := (1 to: values dataSize by: sizeofDouble) collect: [:i| values
>>>> doubleAt: i].
>>>> values free.
>>>> ^orig -> sort
>>>>
>>>> The above example uses Alien to make the callout.  To use it with the
>>>> FFI simply pass cb pointer as the argument as is done above.
>>>>
>>>>
>>>> Implementation:
>>>>
>>>> The way that it works is in two parts
>>>> - the VM passes up a pointer to a structure from which all arguments,
>>>> stacked and in registers (because the VM has copied the register args
>>>> into the struct) can be accessed, and through which the result can be
>>>> returned.
>>>> - the image level provides marshalling methods that match the
>>>> signature in the callback
>>>>
>>>> So e.g. with a callback of
>>>> Callback
>>>> signature:  #(int (*)(const void *, const void *))
>>>> block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign]
>>>> the marshalling methods are in Callback's signature protocol:
>>>>
>>>> Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
>>>> <signature: #(int (*)(const void *, const void *)) abi: 'IA32'>
>>>> ^callbackContext wordResult:
>>>> (block
>>>> value: (Alien forPointer: (spAlien unsignedLongAt: 1))
>>>> value: (Alien forPointer: (spAlien unsignedLongAt: 5)))
>>>>
>>>> where spAlien is an Alien pointing to a VMCallbackContext32.
>>>>
>>>> For ARM support we would add
>>>>
>>>> Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
>>>> intRegArgs: regsAlien
>>>> <signature: #(int (*)(const void *, const void *)) abi: 'ARMV5'>
>>>> ^callbackContext wordResult:
>>>> (block
>>>> value: (Alien forPointer: (regsAlien unsignedLongAt: 1))
>>>> value: (Alien forPointer: (regsAlien unsignedLongAt: 5)))
>>>>
>>>> Basically the idea is that the selector of the method doesn't matter
>>>> except for the number of arguments.  What's important is the pragma
>>>> which defines the signature and the ABI for which this is a valid
>>>> marshalling method.
>>>>
>>>> When the callback is instantiated, Callback introspects to find the
>>>> marshalling method that matches the signature.  If one doesn't already
>>>> exist you can write one.  Hopefully we'll write an ABI compiler that
>>>> will automatically generate these marshalling methods according to the
>>>> platform's ABI, but for now its a manual process.  But at least it's
>>>> open and flexible.  When the callback is invoked the evaluator is
>>>> performed with the current callbackContext and pointer(s) to the
>>>> arguments. There is a 32-bit and a 64-bit callback context, and it can
>>>> have a stack pointer, integer register args and floating point
>>>> register args.  So it's general enough for any callback.
>>>>
>>>> To pass back the result, a value is assigned into the struct via the
>>>> accessor in the marshalling method and control returns to teh point
>>>> where teh callback comes in, and this uses a primitive to return.
>>>> Inside the callbackCOntext is a jmpbuf from a setjmp.  The primitive
>>>> longjmp's back to the entry point in the VM which extracts the result
>>>> and the code for the kind of result and returns:
>>>>
>>>> Callback class>>invokeCallbackContext: vmCallbackContextAddress
>>>> "<Integer>" "^<FFICallbackReturnValue>"
>>>> "The low-level entry-point for callbacks sent from the VM/IA32ABI
>>>> plugin.
>>>> Return via primReturnFromContext:through:.  thisContext's sender is the
>>>> call-out context."
>>>> | callbackAlien type |
>>>> callbackAlien := (Smalltalk wordSize = 4
>>>> ifTrue: [VMCallbackContext32]
>>>> ifFalse: [VMCallbackContext64])
>>>> atAddress: vmCallbackContextAddress.
>>>> [type := Callback evaluateCallbackForContext: callbackAlien]
>>>> ifCurtailed: [self error: 'attempt to non-local return across a
>>>> callback'].
>>>> type ifNil:
>>>> [type := 1. callbackAlien wordResult: -1].
>>>> callbackAlien primReturnAs: type fromContext: thisContext
>>>>
>>>> On Thu, Sep 3, 2015 at 5:22 AM, Craig Latta <[hidden email]
>>>> <mailto:[hidden email]>> wrote:
>>>>
>>>>
>>>>     Hi all--
>>>>
>>>>          I'd like to use a C shared library via FFI from Squeak or
>>>> Pharo.
>>>>     Specifically, I want to call a C function which takes as one of its
>>>>     parameters a pointer to an array of callback function addresses.
>>>> Does
>>>>     one of the FFI approaches floating around handle that, and provide a
>>>>     relatively pleasant mapping between Smalltalk block closures and C
>>>>     callback function addresses?
>>>>
>>>>          I was doing this so far in GemStone, but apparently its FFI
>>>> only
>>>>     supports passing callback function addresses one at a time as direct
>>>>     parameters of C callouts. I suppose I could write a wrapper C
>>>> library
>>>>     around the one I really want to use, that provides the callback
>>>> setup
>>>>     interface that GemStone expects, but whenever I have to write actual
>>>> C
>>>>     code I start to think "Hm, why don't I just use a Smalltalk VM for
>>>>     which
>>>>     I have the source?" :)  All the fancy distributed object-database
>>>>     stuff
>>>>     that my project also wants can wait for a bit.
>>>>
>>>>
>>>>          thanks!
>>>>
>>>>     -C
>>>>
>>>>     --
>>>>     Craig Latta
>>>>     netjam.org <http://netjam.org>
>>>>     +31 6 2757 7177 <tel:%2B31%20%20%206%202757%207177> (SMS ok)
>>>>     + 1 415 287 3547 <tel:%2B%201%20415%20%20287%203547> (no SMS)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> _,,,^..^,,,_
>>>> best, Eliot
>>>>
>>>>
>>
>
>

>
>> I
>> hadn't considered that plugins would need meta data or labels.
>
> They could be put into a .zip file and stored and cataloged somewhere
> which would be a well-known place for Squeak.  And, each plugin would
> be tagged with one or more hierarchical Category's to indicate the
> platform and whatever else we wanted...
>
> And, if the plugin is compiled and hosted somewhere else, like GitHub,
> then the Squeak-well-known place should direct the user to THAT
> external place instead of hosting copies on squeak.org servers...
> Except as a backup, of course..
>
> Hmm, if only we have something that could do all that...    Oh wait,
> we do!!!!    ;-)
>
>> Perhaps the best thing is to leave it as an open question for  a few weeks
>> before settling on any kind of solution. A common location for plugins is
>> about as far as my thinking takes me at the moment. I'm sure the collective
>> imagination of the community can come up with a better idea.
>>
>> Chris
>>
>>
>> On 2015-09-03 10:07 PM, David T. Lewis wrote:
>>>
>>> On Thu, Sep 03, 2015 at 09:05:07PM -0400, Chris Cunnington wrote:
>>>>
>>>> This community needs a repository for plugins. A directory at
>>>> http://ftp.squeak.org ought to do it.
>>>>
>>>> If I want to try Alien I need the plugin. Where am I going to get that?
>>>> I could get VMMaker, read NewSqueakIA32ABIPlugin three times to see it
>>>> is not saying Newspeak and realize it compiles a plugin called IA32ABI,
>>>> which - coincidentally - is the name of the plugin the error Alien
>>>> exampleCqsort will gives me. (NewSqueakIA32ABIPlugin class>> moduleName
>>>> returns IA32ABI).
>>>>
>>>> Does anybody know what the Genie plugin does? Does Paul DeBrucker need
>>>> to store a BerkeleyDB plugin dll he got by emailing somebody on GitHub?
>>>>
>>>> If I understand this correctly, if such a directory existed, then all a
>>>> person would need to do to try the above Alien code would be to download
>>>> a pre-compiled plugin from - say -
>>>> http://ftp.squeak/plugins/Alien/fooVersion, put it in their VM directory
>>>> and start the image. Then check with SmalltalkImage current
>>>> listLoadedModules to see if you're ready to go.
>>>>
>>>> I nominate Fabio Niephaus to be the person the community can send
>>>> pre-compiled plugins for inclusion in such a directory. Eliot's been
>>>> banging away about Alien for some time [1]. If a person could go, get
>>>> the plugin and give the code shot, he could have more productive
>>>> conversations about Alien.
>>>>
>>>> So, if, Eliot, you have a pre-compiled plugin for Alien, perhaps in an
>>>> idle moment, you could email it to Fabio? And when, Fabio, you've
>>>> created the directory in http://ftp.squeak.org you could announce it
>>>> here on Squeak-dev?
>>>>
>>>> FWIW,
>>>>
>>>> Chris
>>>
>>> A word of caution:
>>>
>>> I think the idea is good in principle, but it needs to be accompanied
>>> by a commitment by someone to do the hard work of keeping track of
>>> dependencies on the various flavors of OS, image word size, VM word size,
>>> spur/V3, and so forth. If someone would like to take that on, great.
>>> That said, we don't want to start a project that gets 80% completed, and
>>> then expect "the community" to do the rest of the work.
>>>
>>> I cannot speak for Fabio, but I don't think it is fair for us to expect
>>> him to take on this kind of workload unless he has indicated an interest
>>> in doing it.
>>>
>>> Maybe a restricted repository focused just on compiled plugins known
>>> to work with the Spur VM on a limited range of OS flavors would be helpful
>>> at this stage. It probably does not belong on ftp.squeak.org, but it could
>>> be a very useful resource if someone (Chris Cunnington?) wants to set it
>>> up.
>>>
>>> Dave
>>>
>>>
>>>> [1] http://wiki.squeak.org/squeak/uploads/6100/
>>>>
>>>> On 2015-09-03 7:07 PM, Eliot Miranda wrote:
>>>>>
>>>>> Hi Craig,
>>>>>
>>>>>    you need Alien-eem.24
>>>>>
>>>>> Here's Alien class>>exampleCqsort
>>>>> "Call the libc qsort function (which requires a callback)."
>>>>> "Alien exampleCqsort"
>>>>> "(Time millisecondsToRun: [100 timesRepeat: [Alien exampleCqsort]]) /
>>>>> 100.0"
>>>>> | cb rand nElements sizeofDouble values orig sort |
>>>>> rand := Random new.
>>>>> values := Alien newC: (nElements := 100) * (sizeofDouble := 8).
>>>>> 1 to: values dataSize by: sizeofDouble do:
>>>>> [:i| values doubleAt: i put: rand next].
>>>>> orig := (1 to: values dataSize by: sizeofDouble) collect: [:i| values
>>>>> doubleAt: i].
>>>>> cb := Callback
>>>>> signature:  #(int (*)(const void *, const void *))
>>>>> block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign].
>>>>> (Alien lookup: 'qsort' inLibrary: Alien libcName)
>>>>> primFFICallResult: nil
>>>>> with: values pointer
>>>>> with: nElements
>>>>> with: sizeofDouble
>>>>> with: cb thunk.
>>>>> sort := (1 to: values dataSize by: sizeofDouble) collect: [:i| values
>>>>> doubleAt: i].
>>>>> values free.
>>>>> ^orig -> sort
>>>>>
>>>>> The above example uses Alien to make the callout.  To use it with the
>>>>> FFI simply pass cb pointer as the argument as is done above.
>>>>>
>>>>>
>>>>> Implementation:
>>>>>
>>>>> The way that it works is in two parts
>>>>> - the VM passes up a pointer to a structure from which all arguments,
>>>>> stacked and in registers (because the VM has copied the register args
>>>>> into the struct) can be accessed, and through which the result can be
>>>>> returned.
>>>>> - the image level provides marshalling methods that match the
>>>>> signature in the callback
>>>>>
>>>>> So e.g. with a callback of
>>>>> Callback
>>>>> signature:  #(int (*)(const void *, const void *))
>>>>> block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign]
>>>>> the marshalling methods are in Callback's signature protocol:
>>>>>
>>>>> Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
>>>>> <signature: #(int (*)(const void *, const void *)) abi: 'IA32'>
>>>>> ^callbackContext wordResult:
>>>>> (block
>>>>> value: (Alien forPointer: (spAlien unsignedLongAt: 1))
>>>>> value: (Alien forPointer: (spAlien unsignedLongAt: 5)))
>>>>>
>>>>> where spAlien is an Alien pointing to a VMCallbackContext32.
>>>>>
>>>>> For ARM support we would add
>>>>>
>>>>> Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
>>>>> intRegArgs: regsAlien
>>>>> <signature: #(int (*)(const void *, const void *)) abi: 'ARMV5'>
>>>>> ^callbackContext wordResult:
>>>>> (block
>>>>> value: (Alien forPointer: (regsAlien unsignedLongAt: 1))
>>>>> value: (Alien forPointer: (regsAlien unsignedLongAt: 5)))
>>>>>
>>>>> Basically the idea is that the selector of the method doesn't matter
>>>>> except for the number of arguments.  What's important is the pragma
>>>>> which defines the signature and the ABI for which this is a valid
>>>>> marshalling method.
>>>>>
>>>>> When the callback is instantiated, Callback introspects to find the
>>>>> marshalling method that matches the signature.  If one doesn't already
>>>>> exist you can write one.  Hopefully we'll write an ABI compiler that
>>>>> will automatically generate these marshalling methods according to the
>>>>> platform's ABI, but for now its a manual process.  But at least it's
>>>>> open and flexible.  When the callback is invoked the evaluator is
>>>>> performed with the current callbackContext and pointer(s) to the
>>>>> arguments. There is a 32-bit and a 64-bit callback context, and it can
>>>>> have a stack pointer, integer register args and floating point
>>>>> register args.  So it's general enough for any callback.
>>>>>
>>>>> To pass back the result, a value is assigned into the struct via the
>>>>> accessor in the marshalling method and control returns to teh point
>>>>> where teh callback comes in, and this uses a primitive to return.
>>>>> Inside the callbackCOntext is a jmpbuf from a setjmp.  The primitive
>>>>> longjmp's back to the entry point in the VM which extracts the result
>>>>> and the code for the kind of result and returns:
>>>>>
>>>>> Callback class>>invokeCallbackContext: vmCallbackContextAddress
>>>>> "<Integer>" "^<FFICallbackReturnValue>"
>>>>> "The low-level entry-point for callbacks sent from the VM/IA32ABI
>>>>> plugin.
>>>>> Return via primReturnFromContext:through:.  thisContext's sender is the
>>>>> call-out context."
>>>>> | callbackAlien type |
>>>>> callbackAlien := (Smalltalk wordSize = 4
>>>>> ifTrue: [VMCallbackContext32]
>>>>> ifFalse: [VMCallbackContext64])
>>>>> atAddress: vmCallbackContextAddress.
>>>>> [type := Callback evaluateCallbackForContext: callbackAlien]
>>>>> ifCurtailed: [self error: 'attempt to non-local return across a
>>>>> callback'].
>>>>> type ifNil:
>>>>> [type := 1. callbackAlien wordResult: -1].
>>>>> callbackAlien primReturnAs: type fromContext: thisContext
>>>>>
>>>>> On Thu, Sep 3, 2015 at 5:22 AM, Craig Latta <[hidden email]
>>>>> <mailto:[hidden email]>> wrote:
>>>>>
>>>>>
>>>>>    Hi all--
>>>>>
>>>>>         I'd like to use a C shared library via FFI from Squeak or
>>>>> Pharo.
>>>>>    Specifically, I want to call a C function which takes as one of its
>>>>>    parameters a pointer to an array of callback function addresses.
>>>>> Does
>>>>>    one of the FFI approaches floating around handle that, and provide a
>>>>>    relatively pleasant mapping between Smalltalk block closures and C
>>>>>    callback function addresses?
>>>>>
>>>>>         I was doing this so far in GemStone, but apparently its FFI
>>>>> only
>>>>>    supports passing callback function addresses one at a time as direct
>>>>>    parameters of C callouts. I suppose I could write a wrapper C
>>>>> library
>>>>>    around the one I really want to use, that provides the callback
>>>>> setup
>>>>>    interface that GemStone expects, but whenever I have to write actual
>>>>> C
>>>>>    code I start to think "Hm, why don't I just use a Smalltalk VM for
>>>>>    which
>>>>>    I have the source?" :)  All the fancy distributed object-database
>>>>>    stuff
>>>>>    that my project also wants can wait for a bit.
>>>>>
>>>>>
>>>>>         thanks!
>>>>>
>>>>>    -C
>>>>>
>>>>>    --
>>>>>    Craig Latta
>>>>>    netjam.org <http://netjam.org>
>>>>>    +31 6 2757 7177 <tel:%2B31%20%20%206%202757%207177> (SMS ok)
>>>>>    + 1 415 287 3547 <tel:%2B%201%20415%20%20287%203547> (no SMS)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> _,,,^..^,,,_
>>>>> best, Eliot

>
> On 04.09.2015, at 05:46, Chris Muller <[hidden email]> wrote:
>
>>> I do think that anything  is better than the current situation. A bucket of
>>> any kind in an agreed upon location seems like a good place to start.
>>
>> I totally agree!
>
> what about
>
> http://github.com/squeak-vm/plugins
>
> or
>
> http://github.com/squeak-smalltalk/vm-plugins

> Best
>         -tobias
>
> PS: we should look at https://github.com/search?q=squeak+vm
>
>>
>>> I
>>> hadn't considered that plugins would need meta data or labels.
>>
>> They could be put into a .zip file and stored and cataloged somewhere
>> which would be a well-known place for Squeak.  And, each plugin would
>> be tagged with one or more hierarchical Category's to indicate the
>> platform and whatever else we wanted...
>>
>> And, if the plugin is compiled and hosted somewhere else, like GitHub,
>> then the Squeak-well-known place should direct the user to THAT
>> external place instead of hosting copies on squeak.org servers...
>> Except as a backup, of course..
>>
>> Hmm, if only we have something that could do all that...    Oh wait,
>> we do!!!!    ;-)
>>
>>> Perhaps the best thing is to leave it as an open question for  a few weeks
>>> before settling on any kind of solution. A common location for plugins is
>>> about as far as my thinking takes me at the moment. I'm sure the collective
>>> imagination of the community can come up with a better idea.
>>>
>>> Chris
>>>
>>>
>>> On 2015-09-03 10:07 PM, David T. Lewis wrote:
>>>>
>>>> On Thu, Sep 03, 2015 at 09:05:07PM -0400, Chris Cunnington wrote:
>>>>>
>>>>> This community needs a repository for plugins. A directory at
>>>>> http://ftp.squeak.org ought to do it.
>>>>>
>>>>> If I want to try Alien I need the plugin. Where am I going to get that?
>>>>> I could get VMMaker, read NewSqueakIA32ABIPlugin three times to see it
>>>>> is not saying Newspeak and realize it compiles a plugin called IA32ABI,
>>>>> which - coincidentally - is the name of the plugin the error Alien
>>>>> exampleCqsort will gives me. (NewSqueakIA32ABIPlugin class>> moduleName
>>>>> returns IA32ABI).
>>>>>
>>>>> Does anybody know what the Genie plugin does? Does Paul DeBrucker need
>>>>> to store a BerkeleyDB plugin dll he got by emailing somebody on GitHub?
>>>>>
>>>>> If I understand this correctly, if such a directory existed, then all a
>>>>> person would need to do to try the above Alien code would be to download
>>>>> a pre-compiled plugin from - say -
>>>>> http://ftp.squeak/plugins/Alien/fooVersion, put it in their VM directory
>>>>> and start the image. Then check with SmalltalkImage current
>>>>> listLoadedModules to see if you're ready to go.
>>>>>
>>>>> I nominate Fabio Niephaus to be the person the community can send
>>>>> pre-compiled plugins for inclusion in such a directory. Eliot's been
>>>>> banging away about Alien for some time [1]. If a person could go, get
>>>>> the plugin and give the code shot, he could have more productive
>>>>> conversations about Alien.
>>>>>
>>>>> So, if, Eliot, you have a pre-compiled plugin for Alien, perhaps in an
>>>>> idle moment, you could email it to Fabio? And when, Fabio, you've
>>>>> created the directory in http://ftp.squeak.org you could announce it
>>>>> here on Squeak-dev?
>>>>>
>>>>> FWIW,
>>>>>
>>>>> Chris
>>>>
>>>> A word of caution:
>>>>
>>>> I think the idea is good in principle, but it needs to be accompanied
>>>> by a commitment by someone to do the hard work of keeping track of
>>>> dependencies on the various flavors of OS, image word size, VM word size,
>>>> spur/V3, and so forth. If someone would like to take that on, great.
>>>> That said, we don't want to start a project that gets 80% completed, and
>>>> then expect "the community" to do the rest of the work.
>>>>
>>>> I cannot speak for Fabio, but I don't think it is fair for us to expect
>>>> him to take on this kind of workload unless he has indicated an interest
>>>> in doing it.
>>>>
>>>> Maybe a restricted repository focused just on compiled plugins known
>>>> to work with the Spur VM on a limited range of OS flavors would be helpful
>>>> at this stage. It probably does not belong on ftp.squeak.org, but it could
>>>> be a very useful resource if someone (Chris Cunnington?) wants to set it
>>>> up.
>>>>
>>>> Dave
>>>>
>>>>
>>>>> [1] http://wiki.squeak.org/squeak/uploads/6100/
>>>>>
>>>>> On 2015-09-03 7:07 PM, Eliot Miranda wrote:
>>>>>>
>>>>>> Hi Craig,
>>>>>>
>>>>>>    you need Alien-eem.24
>>>>>>
>>>>>> Here's Alien class>>exampleCqsort
>>>>>> "Call the libc qsort function (which requires a callback)."
>>>>>> "Alien exampleCqsort"
>>>>>> "(Time millisecondsToRun: [100 timesRepeat: [Alien exampleCqsort]]) /
>>>>>> 100.0"
>>>>>> | cb rand nElements sizeofDouble values orig sort |
>>>>>> rand := Random new.
>>>>>> values := Alien newC: (nElements := 100) * (sizeofDouble := 8).
>>>>>> 1 to: values dataSize by: sizeofDouble do:
>>>>>> [:i| values doubleAt: i put: rand next].
>>>>>> orig := (1 to: values dataSize by: sizeofDouble) collect: [:i| values
>>>>>> doubleAt: i].
>>>>>> cb := Callback
>>>>>> signature:  #(int (*)(const void *, const void *))
>>>>>> block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign].
>>>>>> (Alien lookup: 'qsort' inLibrary: Alien libcName)
>>>>>> primFFICallResult: nil
>>>>>> with: values pointer
>>>>>> with: nElements
>>>>>> with: sizeofDouble
>>>>>> with: cb thunk.
>>>>>> sort := (1 to: values dataSize by: sizeofDouble) collect: [:i| values
>>>>>> doubleAt: i].
>>>>>> values free.
>>>>>> ^orig -> sort
>>>>>>
>>>>>> The above example uses Alien to make the callout.  To use it with the
>>>>>> FFI simply pass cb pointer as the argument as is done above.
>>>>>>
>>>>>>
>>>>>> Implementation:
>>>>>>
>>>>>> The way that it works is in two parts
>>>>>> - the VM passes up a pointer to a structure from which all arguments,
>>>>>> stacked and in registers (because the VM has copied the register args
>>>>>> into the struct) can be accessed, and through which the result can be
>>>>>> returned.
>>>>>> - the image level provides marshalling methods that match the
>>>>>> signature in the callback
>>>>>>
>>>>>> So e.g. with a callback of
>>>>>> Callback
>>>>>> signature:  #(int (*)(const void *, const void *))
>>>>>> block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1)) sign]
>>>>>> the marshalling methods are in Callback's signature protocol:
>>>>>>
>>>>>> Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
>>>>>> <signature: #(int (*)(const void *, const void *)) abi: 'IA32'>
>>>>>> ^callbackContext wordResult:
>>>>>> (block
>>>>>> value: (Alien forPointer: (spAlien unsignedLongAt: 1))
>>>>>> value: (Alien forPointer: (spAlien unsignedLongAt: 5)))
>>>>>>
>>>>>> where spAlien is an Alien pointing to a VMCallbackContext32.
>>>>>>
>>>>>> For ARM support we would add
>>>>>>
>>>>>> Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
>>>>>> intRegArgs: regsAlien
>>>>>> <signature: #(int (*)(const void *, const void *)) abi: 'ARMV5'>
>>>>>> ^callbackContext wordResult:
>>>>>> (block
>>>>>> value: (Alien forPointer: (regsAlien unsignedLongAt: 1))
>>>>>> value: (Alien forPointer: (regsAlien unsignedLongAt: 5)))
>>>>>>
>>>>>> Basically the idea is that the selector of the method doesn't matter
>>>>>> except for the number of arguments.  What's important is the pragma
>>>>>> which defines the signature and the ABI for which this is a valid
>>>>>> marshalling method.
>>>>>>
>>>>>> When the callback is instantiated, Callback introspects to find the
>>>>>> marshalling method that matches the signature.  If one doesn't already
>>>>>> exist you can write one.  Hopefully we'll write an ABI compiler that
>>>>>> will automatically generate these marshalling methods according to the
>>>>>> platform's ABI, but for now its a manual process.  But at least it's
>>>>>> open and flexible.  When the callback is invoked the evaluator is
>>>>>> performed with the current callbackContext and pointer(s) to the
>>>>>> arguments. There is a 32-bit and a 64-bit callback context, and it can
>>>>>> have a stack pointer, integer register args and floating point
>>>>>> register args.  So it's general enough for any callback.
>>>>>>
>>>>>> To pass back the result, a value is assigned into the struct via the
>>>>>> accessor in the marshalling method and control returns to teh point
>>>>>> where teh callback comes in, and this uses a primitive to return.
>>>>>> Inside the callbackCOntext is a jmpbuf from a setjmp.  The primitive
>>>>>> longjmp's back to the entry point in the VM which extracts the result
>>>>>> and the code for the kind of result and returns:
>>>>>>
>>>>>> Callback class>>invokeCallbackContext: vmCallbackContextAddress
>>>>>> "<Integer>" "^<FFICallbackReturnValue>"
>>>>>> "The low-level entry-point for callbacks sent from the VM/IA32ABI
>>>>>> plugin.
>>>>>> Return via primReturnFromContext:through:.  thisContext's sender is the
>>>>>> call-out context."
>>>>>> | callbackAlien type |
>>>>>> callbackAlien := (Smalltalk wordSize = 4
>>>>>> ifTrue: [VMCallbackContext32]
>>>>>> ifFalse: [VMCallbackContext64])
>>>>>> atAddress: vmCallbackContextAddress.
>>>>>> [type := Callback evaluateCallbackForContext: callbackAlien]
>>>>>> ifCurtailed: [self error: 'attempt to non-local return across a
>>>>>> callback'].
>>>>>> type ifNil:
>>>>>> [type := 1. callbackAlien wordResult: -1].
>>>>>> callbackAlien primReturnAs: type fromContext: thisContext
>>>>>>
>>>>>> On Thu, Sep 3, 2015 at 5:22 AM, Craig Latta <[hidden email]
>>>>>> <mailto:[hidden email]>> wrote:
>>>>>>
>>>>>>
>>>>>>    Hi all--
>>>>>>
>>>>>>         I'd like to use a C shared library via FFI from Squeak or
>>>>>> Pharo.
>>>>>>    Specifically, I want to call a C function which takes as one of its
>>>>>>    parameters a pointer to an array of callback function addresses.
>>>>>> Does
>>>>>>    one of the FFI approaches floating around handle that, and provide a
>>>>>>    relatively pleasant mapping between Smalltalk block closures and C
>>>>>>    callback function addresses?
>>>>>>
>>>>>>         I was doing this so far in GemStone, but apparently its FFI
>>>>>> only
>>>>>>    supports passing callback function addresses one at a time as direct
>>>>>>    parameters of C callouts. I suppose I could write a wrapper C
>>>>>> library
>>>>>>    around the one I really want to use, that provides the callback
>>>>>> setup
>>>>>>    interface that GemStone expects, but whenever I have to write actual
>>>>>> C
>>>>>>    code I start to think "Hm, why don't I just use a Smalltalk VM for
>>>>>>    which
>>>>>>    I have the source?" :)  All the fancy distributed object-database
>>>>>>    stuff
>>>>>>    that my project also wants can wait for a bit.
>>>>>>
>>>>>>
>>>>>>         thanks!
>>>>>>
>>>>>>    -C
>>>>>>
>>>>>>    --
>>>>>>    Craig Latta
>>>>>>    netjam.org <http://netjam.org>
>>>>>>    +31 6 2757 7177 <tel:%2B31%20%20%206%202757%207177> (SMS ok)
>>>>>>    + 1 415 287 3547 <tel:%2B%201%20415%20%20287%203547> (no SMS)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> _,,,^..^,,,_
>>>>>> best, Eliot
>
>
>

> On Fri, Sep 4, 2015 at 9:26 AM Frank Shearar <[hidden email]>
> wrote:
>
>> On 4 September 2015 at 07:01, Tobias Pape <[hidden email]> wrote:
>> >
>> > On 04.09.2015, at 05:46, Chris Muller <[hidden email]> wrote:
>> >
>> >>> I do think that anything  is better than the current situation. A
>> bucket of
>> >>> any kind in an agreed upon location seems like a good place to start.
>> >>
>> >> I totally agree!
>> >
>> > what about
>> >
>> > http://github.com/squeak-vm/plugins
>> >
>> > or
>> >
>> > http://github.com/squeak-smalltalk/vm-plugins
>>
>> My preferred choice: the organisation already exists.
>>
>> frank
>>
>> > Best
>> >         -tobias
>> >
>> > PS: we should look at https://github.com/search?q=squeak+vm
>> >
>> >>
>> >>> I
>> >>> hadn't considered that plugins would need meta data or labels.
>> >>
>> >> They could be put into a .zip file and stored and cataloged somewhere
>> >> which would be a well-known place for Squeak.  And, each plugin would
>> >> be tagged with one or more hierarchical Category's to indicate the
>> >> platform and whatever else we wanted...
>> >>
>> >> And, if the plugin is compiled and hosted somewhere else, like GitHub,
>> >> then the Squeak-well-known place should direct the user to THAT
>> >> external place instead of hosting copies on squeak.org servers...
>> >> Except as a backup, of course..
>> >>
>> >> Hmm, if only we have something that could do all that...    Oh wait,
>> >> we do!!!!    ;-)
>> >>
>> >>> Perhaps the best thing is to leave it as an open question for  a few
>> weeks
>> >>> before settling on any kind of solution. A common location for
>> >>> plugins
>> is
>> >>> about as far as my thinking takes me at the moment. I'm sure the
>> collective
>> >>> imagination of the community can come up with a better idea.
>> >>>
>> >>> Chris
>> >>>
>> >>>
>> >>> On 2015-09-03 10:07 PM, David T. Lewis wrote:
>> >>>>
>> >>>> On Thu, Sep 03, 2015 at 09:05:07PM -0400, Chris Cunnington wrote:
>> >>>>>
>> >>>>> This community needs a repository for plugins. A directory at
>> >>>>> http://ftp.squeak.org ought to do it.
>> >>>>>
>> >>>>> If I want to try Alien I need the plugin. Where am I going to get
>> that?
>> >>>>> I could get VMMaker, read NewSqueakIA32ABIPlugin three times to see
>> it
>> >>>>> is not saying Newspeak and realize it compiles a plugin called
>> IA32ABI,
>> >>>>> which - coincidentally - is the name of the plugin the error Alien
>> >>>>> exampleCqsort will gives me. (NewSqueakIA32ABIPlugin class>>
>> moduleName
>> >>>>> returns IA32ABI).
>> >>>>>
>> >>>>> Does anybody know what the Genie plugin does? Does Paul DeBrucker
>> need
>> >>>>> to store a BerkeleyDB plugin dll he got by emailing somebody on
>> GitHub?
>> >>>>>
>> >>>>> If I understand this correctly, if such a directory existed, then
>> all a
>> >>>>> person would need to do to try the above Alien code would be to
>> download
>> >>>>> a pre-compiled plugin from - say -
>> >>>>> http://ftp.squeak/plugins/Alien/fooVersion, put it in their VM
>> directory
>> >>>>> and start the image. Then check with SmalltalkImage current
>> >>>>> listLoadedModules to see if you're ready to go.
>> >>>>>
>> >>>>> I nominate Fabio Niephaus to be the person the community can send
>> >>>>> pre-compiled plugins for inclusion in such a directory. Eliot's
>> >>>>> been
>> >>>>> banging away about Alien for some time [1]. If a person could go,
>> >>>>> get
>> >>>>> the plugin and give the code shot, he could have more productive
>> >>>>> conversations about Alien.
>> >>>>>
>> >>>>> So, if, Eliot, you have a pre-compiled plugin for Alien, perhaps in
>> an
>> >>>>> idle moment, you could email it to Fabio? And when, Fabio, you've
>> >>>>> created the directory in http://ftp.squeak.org you could announce
>> >>>>> it
>> >>>>> here on Squeak-dev?
>> >>>>>
>> >>>>> FWIW,
>> >>>>>
>> >>>>> Chris
>> >>>>
>> >>>> A word of caution:
>> >>>>
>> >>>> I think the idea is good in principle, but it needs to be
>> >>>> accompanied
>> >>>> by a commitment by someone to do the hard work of keeping track of
>> >>>> dependencies on the various flavors of OS, image word size, VM word
>> size,
>> >>>> spur/V3, and so forth. If someone would like to take that on, great.
>> >>>> That said, we don't want to start a project that gets 80% completed,
>> and
>> >>>> then expect "the community" to do the rest of the work.
>> >>>>
>> >>>> I cannot speak for Fabio, but I don't think it is fair for us to
>> expect
>> >>>> him to take on this kind of workload unless he has indicated an
>> interest
>> >>>> in doing it.
>> >>>>
>> >>>> Maybe a restricted repository focused just on compiled plugins known
>> >>>> to work with the Spur VM on a limited range of OS flavors would be
>> helpful
>> >>>> at this stage. It probably does not belong on ftp.squeak.org, but it
>> could
>> >>>> be a very useful resource if someone (Chris Cunnington?) wants to
>> >>>> set
>> it
>> >>>> up.
>> >>>>
>> >>>> Dave
>> >>>>
>> >>>>
>> >>>>> [1] http://wiki.squeak.org/squeak/uploads/6100/
>> >>>>>
>> >>>>> On 2015-09-03 7:07 PM, Eliot Miranda wrote:
>> >>>>>>
>> >>>>>> Hi Craig,
>> >>>>>>
>> >>>>>>    you need Alien-eem.24
>> >>>>>>
>> >>>>>> Here's Alien class>>exampleCqsort
>> >>>>>> "Call the libc qsort function (which requires a callback)."
>> >>>>>> "Alien exampleCqsort"
>> >>>>>> "(Time millisecondsToRun: [100 timesRepeat: [Alien
>> >>>>>> exampleCqsort]])
>> /
>> >>>>>> 100.0"
>> >>>>>> | cb rand nElements sizeofDouble values orig sort |
>> >>>>>> rand := Random new.
>> >>>>>> values := Alien newC: (nElements := 100) * (sizeofDouble := 8).
>> >>>>>> 1 to: values dataSize by: sizeofDouble do:
>> >>>>>> [:i| values doubleAt: i put: rand next].
>> >>>>>> orig := (1 to: values dataSize by: sizeofDouble) collect: [:i|
>> values
>> >>>>>> doubleAt: i].
>> >>>>>> cb := Callback
>> >>>>>> signature:  #(int (*)(const void *, const void *))
>> >>>>>> block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1))
>> sign].
>> >>>>>> (Alien lookup: 'qsort' inLibrary: Alien libcName)
>> >>>>>> primFFICallResult: nil
>> >>>>>> with: values pointer
>> >>>>>> with: nElements
>> >>>>>> with: sizeofDouble
>> >>>>>> with: cb thunk.
>> >>>>>> sort := (1 to: values dataSize by: sizeofDouble) collect: [:i|
>> values
>> >>>>>> doubleAt: i].
>> >>>>>> values free.
>> >>>>>> ^orig -> sort
>> >>>>>>
>> >>>>>> The above example uses Alien to make the callout.  To use it with
>> the
>> >>>>>> FFI simply pass cb pointer as the argument as is done above.
>> >>>>>>
>> >>>>>>
>> >>>>>> Implementation:
>> >>>>>>
>> >>>>>> The way that it works is in two parts
>> >>>>>> - the VM passes up a pointer to a structure from which all
>> arguments,
>> >>>>>> stacked and in registers (because the VM has copied the register
>> args
>> >>>>>> into the struct) can be accessed, and through which the result can
>> be
>> >>>>>> returned.
>> >>>>>> - the image level provides marshalling methods that match the
>> >>>>>> signature in the callback
>> >>>>>>
>> >>>>>> So e.g. with a callback of
>> >>>>>> Callback
>> >>>>>> signature:  #(int (*)(const void *, const void *))
>> >>>>>> block: [ :arg1 :arg2 | ((arg1 doubleAt: 1) - (arg2 doubleAt: 1))
>> sign]
>> >>>>>> the marshalling methods are in Callback's signature protocol:
>> >>>>>>
>> >>>>>> Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
>> >>>>>> <signature: #(int (*)(const void *, const void *)) abi: 'IA32'>
>> >>>>>> ^callbackContext wordResult:
>> >>>>>> (block
>> >>>>>> value: (Alien forPointer: (spAlien unsignedLongAt: 1))
>> >>>>>> value: (Alien forPointer: (spAlien unsignedLongAt: 5)))
>> >>>>>>
>> >>>>>> where spAlien is an Alien pointing to a VMCallbackContext32.
>> >>>>>>
>> >>>>>> For ARM support we would add
>> >>>>>>
>> >>>>>> Callback>>voidstarvoidstarRetint: callbackContext sp: spAlien
>> >>>>>> intRegArgs: regsAlien
>> >>>>>> <signature: #(int (*)(const void *, const void *)) abi: 'ARMV5'>
>> >>>>>> ^callbackContext wordResult:
>> >>>>>> (block
>> >>>>>> value: (Alien forPointer: (regsAlien unsignedLongAt: 1))
>> >>>>>> value: (Alien forPointer: (regsAlien unsignedLongAt: 5)))
>> >>>>>>
>> >>>>>> Basically the idea is that the selector of the method doesn't
>> >>>>>> matter
>> >>>>>> except for the number of arguments.  What's important is the
>> >>>>>> pragma
>> >>>>>> which defines the signature and the ABI for which this is a valid
>> >>>>>> marshalling method.
>> >>>>>>
>> >>>>>> When the callback is instantiated, Callback introspects to find
>> >>>>>> the
>> >>>>>> marshalling method that matches the signature.  If one doesn't
>> already
>> >>>>>> exist you can write one.  Hopefully we'll write an ABI compiler
>> >>>>>> that
>> >>>>>> will automatically generate these marshalling methods according to
>> the
>> >>>>>> platform's ABI, but for now its a manual process.  But at least
>> >>>>>> it's
>> >>>>>> open and flexible.  When the callback is invoked the evaluator is
>> >>>>>> performed with the current callbackContext and pointer(s) to the
>> >>>>>> arguments. There is a 32-bit and a 64-bit callback context, and it
>> can
>> >>>>>> have a stack pointer, integer register args and floating point
>> >>>>>> register args.  So it's general enough for any callback.
>> >>>>>>
>> >>>>>> To pass back the result, a value is assigned into the struct via
>> >>>>>> the
>> >>>>>> accessor in the marshalling method and control returns to teh
>> >>>>>> point
>> >>>>>> where teh callback comes in, and this uses a primitive to return.
>> >>>>>> Inside the callbackCOntext is a jmpbuf from a setjmp.  The
>> >>>>>> primitive
>> >>>>>> longjmp's back to the entry point in the VM which extracts the
>> result
>> >>>>>> and the code for the kind of result and returns:
>> >>>>>>
>> >>>>>> Callback class>>invokeCallbackContext: vmCallbackContextAddress
>> >>>>>> "<Integer>" "^<FFICallbackReturnValue>"
>> >>>>>> "The low-level entry-point for callbacks sent from the VM/IA32ABI
>> >>>>>> plugin.
>> >>>>>> Return via primReturnFromContext:through:.  thisContext's sender
>> >>>>>> is
>> the
>> >>>>>> call-out context."
>> >>>>>> | callbackAlien type |
>> >>>>>> callbackAlien := (Smalltalk wordSize = 4
>> >>>>>> ifTrue: [VMCallbackContext32]
>> >>>>>> ifFalse: [VMCallbackContext64])
>> >>>>>> atAddress: vmCallbackContextAddress.
>> >>>>>> [type := Callback evaluateCallbackForContext: callbackAlien]
>> >>>>>> ifCurtailed: [self error: 'attempt to non-local return across a
>> >>>>>> callback'].
>> >>>>>> type ifNil:
>> >>>>>> [type := 1. callbackAlien wordResult: -1].
>> >>>>>> callbackAlien primReturnAs: type fromContext: thisContext
>> >>>>>>
>> >>>>>> On Thu, Sep 3, 2015 at 5:22 AM, Craig Latta <[hidden email]
>> >>>>>> <mailto:[hidden email]>> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>>    Hi all--
>> >>>>>>
>> >>>>>>         I'd like to use a C shared library via FFI from Squeak or
>> >>>>>> Pharo.
>> >>>>>>    Specifically, I want to call a C function which takes as one of
>> its
>> >>>>>>    parameters a pointer to an array of callback function
>> >>>>>> addresses.
>> >>>>>> Does
>> >>>>>>    one of the FFI approaches floating around handle that, and
>> provide a
>> >>>>>>    relatively pleasant mapping between Smalltalk block closures
>> >>>>>> and
>> C
>> >>>>>>    callback function addresses?
>> >>>>>>
>> >>>>>>         I was doing this so far in GemStone, but apparently its
>> >>>>>> FFI
>> >>>>>> only
>> >>>>>>    supports passing callback function addresses one at a time as
>> direct
>> >>>>>>    parameters of C callouts. I suppose I could write a wrapper C
>> >>>>>> library
>> >>>>>>    around the one I really want to use, that provides the callback
>> >>>>>> setup
>> >>>>>>    interface that GemStone expects, but whenever I have to write
>> actual
>> >>>>>> C
>> >>>>>>    code I start to think "Hm, why don't I just use a Smalltalk VM
>> for
>> >>>>>>    which
>> >>>>>>    I have the source?" :)  All the fancy distributed
>> >>>>>> object-database
>> >>>>>>    stuff
>> >>>>>>    that my project also wants can wait for a bit.
>> >>>>>>
>> >>>>>>
>> >>>>>>         thanks!
>> >>>>>>
>> >>>>>>    -C
>> >>>>>>
>> >>>>>>    --
>> >>>>>>    Craig Latta
>> >>>>>>    netjam.org <http://netjam.org>
>> >>>>>>    +31 6 2757 7177 <tel:%2B31%20%20%206%202757%207177> (SMS ok)
>> >>>>>>    + 1 415 287 3547 <tel:%2B%201%20415%20%20287%203547> (no SMS)
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> _,,,^..^,,,_
>> >>>>>> best, Eliot
>> >
>> >
>> >
>>
>>
> I'd agree that the best thing for now is to leave it as an open question
> and then we should go for a solution that everyone feels comfortable
> with.
>
> I'm not sure though if a single GitHub repository will do the trick
> considering the meta information mentioned by David.
> However, we could e.g. use different branches for different Squeak
> versions, but we really need to think this through.
>
> A related question: how hard would it be to build plugins
> automatically (e.g. on Travis CI or on AppVeyor) from source?
>

>
> On 05.09.2015, at 07:31, H. Hirzel <[hidden email]> wrote:
>
>> What about using new SqueakMap entries to find the plugins?
>>
>> The plugins may be in many places.
>
> This won't help for the binaries, tho.
>
> Best regards
> -Tobias
>
>
>

> I mean to use the function of SqueakMap being a catalogue which
> references artefacts available elsewhere.
>
> For dealing with binaries additional provision has to be made to deal with
> it.
>
> --Hannes
>
> On 9/5/15, Tobias Pape <[hidden email]> wrote:
>>
>> On 05.09.2015, at 07:31, H. Hirzel <[hidden email]> wrote:
>>
>>> What about using new SqueakMap entries to find the plugins?
>>>
>>> The plugins may be in many places.
>>
>> This won't help for the binaries, tho.
>>
>> Best regards
>> -Tobias
>>
>>
>>
>

> On Sat, Sep 5, 2015 at 2:19 AM, H. Hirzel <[hidden email]> wrote:
>> I mean to use the function of SqueakMap being a catalogue which
>> references artefacts available elsewhere.
>
> Yep, binaries are no problem for SqueakMap, it was designed to solve
> exactly this problem -- the cataloging of any software, in any format,
> hosted at any location.
>
>> For dealing with binaries additional provision has to be made to deal with it.
>
> No additional provisions need to be made, except possibly adding a
> "Plugin" category.

>> On Sat, Sep 5, 2015 at 2:19 AM, H. Hirzel <[hidden email]> wrote:
>>> I mean to use the function of SqueakMap being a catalogue which
>>> references artefacts available elsewhere.
>>
>> Yep, binaries are no problem for SqueakMap, it was designed to solve
>> exactly this problem -- the cataloging of any software, in any format,
>> hosted at any location.
>>
>>> For dealing with binaries additional provision has to be made to deal with it.
>>
>> No additional provisions need to be made, except possibly adding a
>> "Plugin" category.
>
> I'm unsure. Its external things, nothing that is in image.

> > I loaded Alien-eem.25 into Squeak 5.0.15110 on Mac OS 10.11.15A279b
> > from the Monticello HTTP repository at [1]. I ran AlienSunit in the
> > Test Runner, and it crashed Squeak. Before I get into posting error
> > logs, can anyone confirm that AlienSunit runs successfully on their
> > machine, on any host platform, or that I shouldn't expect it to run
> > successfully? Are there other Monticello packages I should be
> > loading, and if so, in what order?
>
> I get 15 passes and one failure, that being testLongLong.  But that's
> in my VMMaker image, which has some extras in it.  AFAIA tough Alien
> should just work.  I'll look at the testLongLong failure soon cuz I'm
> starting on the x64 JIT and that requires the above to work to access
> the processor alien.