FFI Callbacks

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

FFI Callbacks

Ron Teitelbaum

All,

 

I saw that there was work on a call back structure for plug-ins, by any chance has any of this work spilled over into FFI?

 

I would like to attach to a third party callback from FFI, is that possible?

 

Ron Teitelbaum



Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Andreas.Raab
Ron Teitelbaum wrote:
> I saw that there was work on a call back structure for plug-ins, by any
> chance has any of this work spilled over into FFI?

No. The main issue is that for a C callback you need to be able to "pop
the arguments from the C stack" and there is no support for this yet.

> I would like to attach to a third party callback from FFI, is that possible?

Yes. You can write a plugin that takes the callback and signals a
semaphore (or does other interesting things).

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

RE: FFI Callbacks

Ron Teitelbaum
I should have said without having to write a plug-in.  

Your suggestion sounds simple for simple notifying callbacks, but for
callback that sends in lots of data and has to interact with the image it
sounds like I'm better off writing the whole thing as a plug in and skip the
FFI.  I guess that means learning some SLANG (how about *#&$*#)

A quick question, if callbacks are supported by the VM, and a plug-in can be
written to handle callbacks couldn't someone (notice I didn't say "I" or
"WE" hoping maybe one of the VM gurus would consider it) combine those
features to write a generalized callback framework that would be available
to the image?  For example it would define the function name return value
and parameters, just like FFI but maybe be called <apicallback: ...>.  Then
that callback could be invoked within a semaphore on a separate thread.

Ron


> From: Andreas Raab
> Sent: Thursday, August 31, 2006 8:34 PM
>
> Ron Teitelbaum wrote:
> > I saw that there was work on a call back structure for plug-ins, by any
> > chance has any of this work spilled over into FFI?
>
> No. The main issue is that for a C callback you need to be able to "pop
> the arguments from the C stack" and there is no support for this yet.
>
> > I would like to attach to a third party callback from FFI, is that
> possible?
>
> Yes. You can write a plugin that takes the callback and signals a
> semaphore (or does other interesting things).
>
> Cheers,
>    - Andreas
>



Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Andreas.Raab
Ron Teitelbaum wrote:
> A quick question, if callbacks are supported by the VM, and a plug-in can be
> written to handle callbacks couldn't someone (notice I didn't say "I" or
> "WE" hoping maybe one of the VM gurus would consider it) combine those
> features to write a generalized callback framework that would be available
> to the image?  For example it would define the function name return value
> and parameters, just like FFI but maybe be called <apicallback: ...>.  Then
> that callback could be invoked within a semaphore on a separate thread.

Yes. That is basically what I meant when I said "you need to pop the
arguments from the C stack". What you have to do goes somewhat along the
lines of the following:
1. Set up a pool of callback functions that are varargs based. This
avoids the need to generate callbacks stubs (which may be preferrable on
some platforms but it doesn't matter). These functions need to "put
away" the stack pointer for the image to use, signal an FFI semaphore
and callback into the interpreter.
2. When the callback is picked up in the image, the image needs to call
a set of support functions to take the arguments correctly from the
stack. This would be based on some ffi spec.
3. Once all the arguments are picked up, you run your callback code.
4. To return, you need to call another set of support functions for
storing the return value and return from the callback.

See, it's only four steps. How hard could it possibly be? ;-)

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Alejandro F. Reimondo

> See, it's only four steps. How hard could it possibly be? ;-)
Fine!
Now we are ready to add the point number 5.

5.- it must be realized in 100% smalltalk.
e.g. included in FFI services to be understood
 by squeak users (with at least one example of use).

One point to be considered is that "functions" on smalltalk
 side are full messages (including a receiver) and
 not compiled methods, because each callback instance
 must be managed by a particular receiver.

best,
Ale.


----- Original Message -----
From: "Andreas Raab" <[hidden email]>
To: "The general-purpose Squeak developers list"
<[hidden email]>
Sent: Thursday, August 31, 2006 10:57 PM
Subject: Re: FFI Callbacks


> Ron Teitelbaum wrote:
> > A quick question, if callbacks are supported by the VM, and a plug-in
can be
> > written to handle callbacks couldn't someone (notice I didn't say "I" or
> > "WE" hoping maybe one of the VM gurus would consider it) combine those
> > features to write a generalized callback framework that would be
available
> > to the image?  For example it would define the function name return
value
> > and parameters, just like FFI but maybe be called <apicallback: ...>.
Then

> > that callback could be invoked within a semaphore on a separate thread.
>
> Yes. That is basically what I meant when I said "you need to pop the
> arguments from the C stack". What you have to do goes somewhat along the
> lines of the following:
> 1. Set up a pool of callback functions that are varargs based. This
> avoids the need to generate callbacks stubs (which may be preferrable on
> some platforms but it doesn't matter). These functions need to "put
> away" the stack pointer for the image to use, signal an FFI semaphore
> and callback into the interpreter.
> 2. When the callback is picked up in the image, the image needs to call
> a set of support functions to take the arguments correctly from the
> stack. This would be based on some ffi spec.
> 3. Once all the arguments are picked up, you run your callback code.
> 4. To return, you need to call another set of support functions for
> storing the return value and return from the callback.
>
> See, it's only four steps. How hard could it possibly be? ;-)
>
> Cheers,
>    - Andreas
>


Reply | Threaded
Open this post in threaded view
|

RE: FFI Callbacks

Ron Teitelbaum
I've been thinking a bit about that.  I agree it is important that we define
a standard way to handle the api. It seems to me that a call to a call back
would automatically suspend a separate process with a semaphore.  This would
be automatic to the framework.  How the system handles the callback could
include one of the following:

1) The system would then have to wait on some structure like a SharedQueue
for a response; this would allow polling in the main thread for developers
to "attach" too.  It simplifies the framework since the framework ends with
depositing information into a thread safe area.

2) We could allow the event loop to poll for callbacks and process them in
turn.  This would allow a developer to write code that handles the
invocation by writing the handling code directly after the invocation, and
would remove some of the threading implications of the framework.  This
approach seems contrary to the way we normally think of callbacks, which
sends a non-blocking call and requires the developer to write a separate
method that gets invoked by the external program.  (although it is kinda
cool being able to write a branch of code that gets invoked and goes off on
its own I think it would be confusing)

3) We could let the developer write a return method after the
<apicallback:..> which would be called with some set priority and have
access to the values sent in and returned.  This to me would allow for
higher priority callbacks but since the code runs in its own thread the
developer would need to understand all the threading implications.  But it
does allow the receiver to be explicitly defined.

We could also combine features of the above to get something different.

Do you have other suggestions?

Ron Teitelbaum
(Note discussion should not be considered to be an acceptance of the task to
write the framework, volunteers are welcome)



> From: Alejandro F. Reimondo
> Sent: Friday, September 01, 2006 9:16 AM
>
>
> > See, it's only four steps. How hard could it possibly be? ;-)
> Fine!
> Now we are ready to add the point number 5.
>
> 5.- it must be realized in 100% smalltalk.
> e.g. included in FFI services to be understood
>  by squeak users (with at least one example of use).
>
> One point to be considered is that "functions" on smalltalk
>  side are full messages (including a receiver) and
>  not compiled methods, because each callback instance
>  must be managed by a particular receiver.
>
> best,
> Ale.
>
>
> ----- Original Message -----
> From: "Andreas Raab" <[hidden email]>
> To: "The general-purpose Squeak developers list"
> <[hidden email]>
> Sent: Thursday, August 31, 2006 10:57 PM
> Subject: Re: FFI Callbacks
>
>
> > Ron Teitelbaum wrote:
> > > A quick question, if callbacks are supported by the VM, and a plug-in
> can be
> > > written to handle callbacks couldn't someone (notice I didn't say "I"
> or
> > > "WE" hoping maybe one of the VM gurus would consider it) combine those
> > > features to write a generalized callback framework that would be
> available
> > > to the image?  For example it would define the function name return
> value
> > > and parameters, just like FFI but maybe be called <apicallback: ...>.
> Then
> > > that callback could be invoked within a semaphore on a separate
> thread.
> >
> > Yes. That is basically what I meant when I said "you need to pop the
> > arguments from the C stack". What you have to do goes somewhat along the
> > lines of the following:
> > 1. Set up a pool of callback functions that are varargs based. This
> > avoids the need to generate callbacks stubs (which may be preferrable on
> > some platforms but it doesn't matter). These functions need to "put
> > away" the stack pointer for the image to use, signal an FFI semaphore
> > and callback into the interpreter.
> > 2. When the callback is picked up in the image, the image needs to call
> > a set of support functions to take the arguments correctly from the
> > stack. This would be based on some ffi spec.
> > 3. Once all the arguments are picked up, you run your callback code.
> > 4. To return, you need to call another set of support functions for
> > storing the return value and return from the callback.
> >
> > See, it's only four steps. How hard could it possibly be? ;-)
> >
> > Cheers,
> >    - Andreas
> >
>



Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Alejandro F. Reimondo
We can model the callback as a simple full
 message (aMessageSend) to be performed
 when activated.
Any objects that responds to:
    #arguments(:)
    #perform (or #evaluate )
 can be used as a callback hook.
I think there is no reason to hook aCompiledMethod...
Ale.

----- Original Message -----
From: "Ron Teitelbaum" <[hidden email]>
To: "'The general-purpose Squeak developers list'"
<[hidden email]>
Sent: Friday, September 01, 2006 11:04 AM
Subject: RE: FFI Callbacks


> I've been thinking a bit about that.  I agree it is important that we
define
> a standard way to handle the api. It seems to me that a call to a call
back
> would automatically suspend a separate process with a semaphore.  This
would
> be automatic to the framework.  How the system handles the callback could
> include one of the following:
>
> 1) The system would then have to wait on some structure like a SharedQueue
> for a response; this would allow polling in the main thread for developers
> to "attach" too.  It simplifies the framework since the framework ends
with

> depositing information into a thread safe area.
>
> 2) We could allow the event loop to poll for callbacks and process them in
> turn.  This would allow a developer to write code that handles the
> invocation by writing the handling code directly after the invocation, and
> would remove some of the threading implications of the framework.  This
> approach seems contrary to the way we normally think of callbacks, which
> sends a non-blocking call and requires the developer to write a separate
> method that gets invoked by the external program.  (although it is kinda
> cool being able to write a branch of code that gets invoked and goes off
on

> its own I think it would be confusing)
>
> 3) We could let the developer write a return method after the
> <apicallback:..> which would be called with some set priority and have
> access to the values sent in and returned.  This to me would allow for
> higher priority callbacks but since the code runs in its own thread the
> developer would need to understand all the threading implications.  But it
> does allow the receiver to be explicitly defined.
>
> We could also combine features of the above to get something different.
>
> Do you have other suggestions?
>
> Ron Teitelbaum
> (Note discussion should not be considered to be an acceptance of the task
to

> write the framework, volunteers are welcome)
>
>
>
> > From: Alejandro F. Reimondo
> > Sent: Friday, September 01, 2006 9:16 AM
> >
> >
> > > See, it's only four steps. How hard could it possibly be? ;-)
> > Fine!
> > Now we are ready to add the point number 5.
> >
> > 5.- it must be realized in 100% smalltalk.
> > e.g. included in FFI services to be understood
> >  by squeak users (with at least one example of use).
> >
> > One point to be considered is that "functions" on smalltalk
> >  side are full messages (including a receiver) and
> >  not compiled methods, because each callback instance
> >  must be managed by a particular receiver.
> >
> > best,
> > Ale.
> >
> >
> > ----- Original Message -----
> > From: "Andreas Raab" <[hidden email]>
> > To: "The general-purpose Squeak developers list"
> > <[hidden email]>
> > Sent: Thursday, August 31, 2006 10:57 PM
> > Subject: Re: FFI Callbacks
> >
> >
> > > Ron Teitelbaum wrote:
> > > > A quick question, if callbacks are supported by the VM, and a
plug-in
> > can be
> > > > written to handle callbacks couldn't someone (notice I didn't say
"I"
> > or
> > > > "WE" hoping maybe one of the VM gurus would consider it) combine
those
> > > > features to write a generalized callback framework that would be
> > available
> > > > to the image?  For example it would define the function name return
> > value
> > > > and parameters, just like FFI but maybe be called <apicallback:
...>.
> > Then
> > > > that callback could be invoked within a semaphore on a separate
> > thread.
> > >
> > > Yes. That is basically what I meant when I said "you need to pop the
> > > arguments from the C stack". What you have to do goes somewhat along
the
> > > lines of the following:
> > > 1. Set up a pool of callback functions that are varargs based. This
> > > avoids the need to generate callbacks stubs (which may be preferrable
on
> > > some platforms but it doesn't matter). These functions need to "put
> > > away" the stack pointer for the image to use, signal an FFI semaphore
> > > and callback into the interpreter.
> > > 2. When the callback is picked up in the image, the image needs to
call

> > > a set of support functions to take the arguments correctly from the
> > > stack. This would be based on some ffi spec.
> > > 3. Once all the arguments are picked up, you run your callback code.
> > > 4. To return, you need to call another set of support functions for
> > > storing the return value and return from the callback.
> > >
> > > See, it's only four steps. How hard could it possibly be? ;-)
> > >
> > > Cheers,
> > >    - Andreas
> > >
> >
>
>
>


Reply | Threaded
Open this post in threaded view
|

RE: FFI Callbacks

Ron Teitelbaum
That is how it would be called if the callback was defined in the primitive.
We would parse the return method and do a message send with the values from
the callback.

Which threading model do you prefer?

1) Thread ends at callback, access results through sharedQueue
2) Thread continues at callback, in main thread through eventloop
3) Thread continues at callback in its own thread at specified priority.

Which api do you prefer?

A) Access callback values in a queue
B) Suspend and continue processing at callback from invocation.
C) Specify callback method in primitive

Based on your response preferring a message send I would assume you would
prefer either 2C or 3C.

I could be completely wrong but it seems to me that 1A is the easiest to
build and the safest to use.

2B is confusing to use but safe.

2C is easy to understand and matches what people would expect and is safe in
terms of threading but might have performance problems.

3C is easy to understand and matches what people would expect but it
requires additional care because of threading issues (anything the return
method does will have to be properly protected).

Ron Teitelbaum

> From: Alejandro F. Reimondo
> Sent: Friday, September 01, 2006 10:30 AM
>
> We can model the callback as a simple full
>  message (aMessageSend) to be performed
>  when activated.
> Any objects that responds to:
>     #arguments(:)
>     #perform (or #evaluate )
>  can be used as a callback hook.
> I think there is no reason to hook aCompiledMethod...
> Ale.
>
> ----- Original Message -----
> From: "Ron Teitelbaum" <[hidden email]>
> To: "'The general-purpose Squeak developers list'"
> <[hidden email]>
> Sent: Friday, September 01, 2006 11:04 AM
> Subject: RE: FFI Callbacks
>
>
> > I've been thinking a bit about that.  I agree it is important that we
> define
> > a standard way to handle the api. It seems to me that a call to a call
> back
> > would automatically suspend a separate process with a semaphore.  This
> would
> > be automatic to the framework.  How the system handles the callback
> could
> > include one of the following:
> >
> > 1) The system would then have to wait on some structure like a
> SharedQueue
> > for a response; this would allow polling in the main thread for
> developers
> > to "attach" too.  It simplifies the framework since the framework ends
> with
> > depositing information into a thread safe area.
> >
> > 2) We could allow the event loop to poll for callbacks and process them
> in
> > turn.  This would allow a developer to write code that handles the
> > invocation by writing the handling code directly after the invocation,
> and
> > would remove some of the threading implications of the framework.  This
> > approach seems contrary to the way we normally think of callbacks, which
> > sends a non-blocking call and requires the developer to write a separate
> > method that gets invoked by the external program.  (although it is kinda
> > cool being able to write a branch of code that gets invoked and goes off
> on
> > its own I think it would be confusing)
> >
> > 3) We could let the developer write a return method after the
> > <apicallback:..> which would be called with some set priority and have
> > access to the values sent in and returned.  This to me would allow for
> > higher priority callbacks but since the code runs in its own thread the
> > developer would need to understand all the threading implications.  But
> it
> > does allow the receiver to be explicitly defined.
> >
> > We could also combine features of the above to get something different.
> >
> > Do you have other suggestions?
> >
> > Ron Teitelbaum
> > (Note discussion should not be considered to be an acceptance of the
> task
> to
> > write the framework, volunteers are welcome)
> >
> >
> >
> > > From: Alejandro F. Reimondo
> > > Sent: Friday, September 01, 2006 9:16 AM
> > >
> > >
> > > > See, it's only four steps. How hard could it possibly be? ;-)
> > > Fine!
> > > Now we are ready to add the point number 5.
> > >
> > > 5.- it must be realized in 100% smalltalk.
> > > e.g. included in FFI services to be understood
> > >  by squeak users (with at least one example of use).
> > >
> > > One point to be considered is that "functions" on smalltalk
> > >  side are full messages (including a receiver) and
> > >  not compiled methods, because each callback instance
> > >  must be managed by a particular receiver.
> > >
> > > best,
> > > Ale.
> > >
> > >
> > > ----- Original Message -----
> > > From: "Andreas Raab" <[hidden email]>
> > > To: "The general-purpose Squeak developers list"
> > > <[hidden email]>
> > > Sent: Thursday, August 31, 2006 10:57 PM
> > > Subject: Re: FFI Callbacks
> > >
> > >
> > > > Ron Teitelbaum wrote:
> > > > > A quick question, if callbacks are supported by the VM, and a
> plug-in
> > > can be
> > > > > written to handle callbacks couldn't someone (notice I didn't say
> "I"
> > > or
> > > > > "WE" hoping maybe one of the VM gurus would consider it) combine
> those
> > > > > features to write a generalized callback framework that would be
> > > available
> > > > > to the image?  For example it would define the function name
> return
> > > value
> > > > > and parameters, just like FFI but maybe be called <apicallback:
> ...>.
> > > Then
> > > > > that callback could be invoked within a semaphore on a separate
> > > thread.
> > > >
> > > > Yes. That is basically what I meant when I said "you need to pop the
> > > > arguments from the C stack". What you have to do goes somewhat along
> the
> > > > lines of the following:
> > > > 1. Set up a pool of callback functions that are varargs based. This
> > > > avoids the need to generate callbacks stubs (which may be
> preferrable
> on
> > > > some platforms but it doesn't matter). These functions need to "put
> > > > away" the stack pointer for the image to use, signal an FFI
> semaphore
> > > > and callback into the interpreter.
> > > > 2. When the callback is picked up in the image, the image needs to
> call
> > > > a set of support functions to take the arguments correctly from the
> > > > stack. This would be based on some ffi spec.
> > > > 3. Once all the arguments are picked up, you run your callback code.
> > > > 4. To return, you need to call another set of support functions for
> > > > storing the return value and return from the callback.
> > > >
> > > > See, it's only four steps. How hard could it possibly be? ;-)
> > > >
> > > > Cheers,
> > > >    - Andreas
> > > >
> > >
> >
> >
> >
>
>



Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Diego Fernández
In reply to this post by Andreas.Raab
I don't know anything about the internals but I like how callbacks are
declared in VAST:

windowProc := EsEntryPoint
       receiver: [:hwnd :lParam | "call back object to be executed" ]
       selector: #value:value:
       callingConvention: 'c'
       arrayBased: false
       parameterTypes: #(uint32 pointer)
       returnType: #boolean.

platformFunction := PlatformFunction
       callingConvention: 'c'
       function: 'EnumWindows'
       library: 'user32'
       parameterTypes: #(pointer uint32)
       returnType: #boolean.

platformFunction callWith: address with: 0

An off topic question:
I never used FFI, It's possible to make function calls like in the
example shown above, or I must to use the special syntax: <...> ?

Regards,
Diego

Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Andreas.Raab
Diego Fernandez wrote:
> An off topic question:
> I never used FFI, It's possible to make function calls like in the
> example shown above, or I must to use the special syntax: <...> ?

Why anyone would want to do this is beyond me, but here you go:

aFunction := ExternalLibraryFunction
        name:'EnumWindows'
        module: 'user32'
        callType: ExternalFunction callTypeAPI
        returnType: ExternalType bool
        argumentTypes: {
                ExternalType char asPointerType.
                ExternalType uint32
        }.
aFunction invokeWith: address with: 0.

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Alejandro F. Reimondo
In reply to this post by Diego Fernández
Hi Diego,
The <...> syntax is a short syntax notation to make a method
 as a primitive method, calling a function (object) like
 you instantiate for platformFunction (the object will be
 instantiated by the compiler and bounded to the method
 as a primitive) and returning the result (following normal
 primitive failure policy).
Dynamic function calls (instantiate functions on demand)
 can be used, but are more long expressions.
With dynamic functions you can call more than
 one function in a method, but invalidate
 senders/implementors as a powerful tool for browsing
 (#call:with:... will return all API calls in the universe).
Using <...> syntax let you choose better (smalltalk) selectors
 for calls and let you bind the calls without reducing
 browsing effectiveness.
Ale.


----- Original Message -----
From: "Diego Fernandez" <[hidden email]>
To: "The general-purpose Squeak developers list"
<[hidden email]>
Sent: Friday, September 01, 2006 12:45 PM
Subject: Re: FFI Callbacks


> I don't know anything about the internals but I like how callbacks are
> declared in VAST:
>
> windowProc := EsEntryPoint
>        receiver: [:hwnd :lParam | "call back object to be executed" ]
>        selector: #value:value:
>        callingConvention: 'c'
>        arrayBased: false
>        parameterTypes: #(uint32 pointer)
>        returnType: #boolean.
>
> platformFunction := PlatformFunction
>        callingConvention: 'c'
>        function: 'EnumWindows'
>        library: 'user32'
>        parameterTypes: #(pointer uint32)
>        returnType: #boolean.
>
> platformFunction callWith: address with: 0
>
> An off topic question:
> I never used FFI, It's possible to make function calls like in the
> example shown above, or I must to use the special syntax: <...> ?
>
> Regards,
> Diego
>


Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Alejandro F. Reimondo
In reply to this post by Ron Teitelbaum
Ron,
I prefer the most natural (naïve) model as default.
If there is more that one practical use of callbacks,
 subclassification (of Callback class) can be used to
 define the policy.
I will not put any preference for security, because
 I think that objects must not protect themselves
 (an object that protect itself will break other objects)
best,
Ale.

----- Original Message -----
From: "Ron Teitelbaum" <[hidden email]>
To: "'The general-purpose Squeak developers list'"
<[hidden email]>
Sent: Friday, September 01, 2006 12:23 PM
Subject: RE: FFI Callbacks


> That is how it would be called if the callback was defined in the
primitive.
> We would parse the return method and do a message send with the values
from

> the callback.
>
> Which threading model do you prefer?
>
> 1) Thread ends at callback, access results through sharedQueue
> 2) Thread continues at callback, in main thread through eventloop
> 3) Thread continues at callback in its own thread at specified priority.
>
> Which api do you prefer?
>
> A) Access callback values in a queue
> B) Suspend and continue processing at callback from invocation.
> C) Specify callback method in primitive
>
> Based on your response preferring a message send I would assume you would
> prefer either 2C or 3C.
>
> I could be completely wrong but it seems to me that 1A is the easiest to
> build and the safest to use.
>
> 2B is confusing to use but safe.
>
> 2C is easy to understand and matches what people would expect and is safe
in

> terms of threading but might have performance problems.
>
> 3C is easy to understand and matches what people would expect but it
> requires additional care because of threading issues (anything the return
> method does will have to be properly protected).
>
> Ron Teitelbaum
>
> > From: Alejandro F. Reimondo
> > Sent: Friday, September 01, 2006 10:30 AM
> >
> > We can model the callback as a simple full
> >  message (aMessageSend) to be performed
> >  when activated.
> > Any objects that responds to:
> >     #arguments(:)
> >     #perform (or #evaluate )
> >  can be used as a callback hook.
> > I think there is no reason to hook aCompiledMethod...
> > Ale.
> >
> > ----- Original Message -----
> > From: "Ron Teitelbaum" <[hidden email]>
> > To: "'The general-purpose Squeak developers list'"
> > <[hidden email]>
> > Sent: Friday, September 01, 2006 11:04 AM
> > Subject: RE: FFI Callbacks
> >
> >
> > > I've been thinking a bit about that.  I agree it is important that we
> > define
> > > a standard way to handle the api. It seems to me that a call to a call
> > back
> > > would automatically suspend a separate process with a semaphore.  This
> > would
> > > be automatic to the framework.  How the system handles the callback
> > could
> > > include one of the following:
> > >
> > > 1) The system would then have to wait on some structure like a
> > SharedQueue
> > > for a response; this would allow polling in the main thread for
> > developers
> > > to "attach" too.  It simplifies the framework since the framework ends
> > with
> > > depositing information into a thread safe area.
> > >
> > > 2) We could allow the event loop to poll for callbacks and process
them
> > in
> > > turn.  This would allow a developer to write code that handles the
> > > invocation by writing the handling code directly after the invocation,
> > and
> > > would remove some of the threading implications of the framework.
This
> > > approach seems contrary to the way we normally think of callbacks,
which
> > > sends a non-blocking call and requires the developer to write a
separate
> > > method that gets invoked by the external program.  (although it is
kinda
> > > cool being able to write a branch of code that gets invoked and goes
off
> > on
> > > its own I think it would be confusing)
> > >
> > > 3) We could let the developer write a return method after the
> > > <apicallback:..> which would be called with some set priority and have
> > > access to the values sent in and returned.  This to me would allow for
> > > higher priority callbacks but since the code runs in its own thread
the
> > > developer would need to understand all the threading implications.
But
> > it
> > > does allow the receiver to be explicitly defined.
> > >
> > > We could also combine features of the above to get something
different.

> > >
> > > Do you have other suggestions?
> > >
> > > Ron Teitelbaum
> > > (Note discussion should not be considered to be an acceptance of the
> > task
> > to
> > > write the framework, volunteers are welcome)
> > >
> > >
> > >
> > > > From: Alejandro F. Reimondo
> > > > Sent: Friday, September 01, 2006 9:16 AM
> > > >
> > > >
> > > > > See, it's only four steps. How hard could it possibly be? ;-)
> > > > Fine!
> > > > Now we are ready to add the point number 5.
> > > >
> > > > 5.- it must be realized in 100% smalltalk.
> > > > e.g. included in FFI services to be understood
> > > >  by squeak users (with at least one example of use).
> > > >
> > > > One point to be considered is that "functions" on smalltalk
> > > >  side are full messages (including a receiver) and
> > > >  not compiled methods, because each callback instance
> > > >  must be managed by a particular receiver.
> > > >
> > > > best,
> > > > Ale.
> > > >
> > > >
> > > > ----- Original Message -----
> > > > From: "Andreas Raab" <[hidden email]>
> > > > To: "The general-purpose Squeak developers list"
> > > > <[hidden email]>
> > > > Sent: Thursday, August 31, 2006 10:57 PM
> > > > Subject: Re: FFI Callbacks
> > > >
> > > >
> > > > > Ron Teitelbaum wrote:
> > > > > > A quick question, if callbacks are supported by the VM, and a
> > plug-in
> > > > can be
> > > > > > written to handle callbacks couldn't someone (notice I didn't
say

> > "I"
> > > > or
> > > > > > "WE" hoping maybe one of the VM gurus would consider it) combine
> > those
> > > > > > features to write a generalized callback framework that would be
> > > > available
> > > > > > to the image?  For example it would define the function name
> > return
> > > > value
> > > > > > and parameters, just like FFI but maybe be called <apicallback:
> > ...>.
> > > > Then
> > > > > > that callback could be invoked within a semaphore on a separate
> > > > thread.
> > > > >
> > > > > Yes. That is basically what I meant when I said "you need to pop
the
> > > > > arguments from the C stack". What you have to do goes somewhat
along
> > the
> > > > > lines of the following:
> > > > > 1. Set up a pool of callback functions that are varargs based.
This
> > > > > avoids the need to generate callbacks stubs (which may be
> > preferrable
> > on
> > > > > some platforms but it doesn't matter). These functions need to
"put
> > > > > away" the stack pointer for the image to use, signal an FFI
> > semaphore
> > > > > and callback into the interpreter.
> > > > > 2. When the callback is picked up in the image, the image needs to
> > call
> > > > > a set of support functions to take the arguments correctly from
the
> > > > > stack. This would be based on some ffi spec.
> > > > > 3. Once all the arguments are picked up, you run your callback
code.
> > > > > 4. To return, you need to call another set of support functions
for

> > > > > storing the return value and return from the callback.
> > > > >
> > > > > See, it's only four steps. How hard could it possibly be? ;-)
> > > > >
> > > > > Cheers,
> > > > >    - Andreas
> > > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Diego Fernández
In reply to this post by Andreas.Raab
Cool, thanks for the reply :) to both (Andreas and Alejandro).

On 9/1/06, Andreas Raab <[hidden email]> wrote:
> Why anyone would want to do this is beyond me, but here you go:

Beacuse I don't have to learn a new syntax, is just the simple object
message syntax.
I don't care if I have to type a little more... but it's just my
personal taste :)

Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Diego Fernández
In reply to this post by Alejandro F. Reimondo
On 9/1/06, Alejandro F. Reimondo <[hidden email]> wrote:
> The <...> syntax is a short syntax notation to make a method
>  as a primitive method, calling a function (object) like
>  you instantiate for platformFunction (the object will be
>  instantiated by the compiler and bounded to the method
>  as a primitive) and returning the result (following normal
>  primitive failure policy).

I know that <> is used in primitives, but in calls to external
libraries there is a performance reason for this kind of notation?
Have this kind of methods different bycodes (like in primitives) or
it's only a shorthand for the longer expression?

> Dynamic function calls (instantiate functions on demand)
>  can be used, but are more long expressions.
> With dynamic functions you can call more than
>  one function in a method, but invalidate
>  senders/implementors as a powerful tool for browsing
>  (#call:with:... will return all API calls in the universe).
> Using <...> syntax let you choose better (smalltalk) selectors
>  for calls and let you bind the calls without reducing
>  browsing effectiveness.

What are the differences in browsing?
I don't see much appart from less typing, since you can find
references to ExternalLibraryFunction, and follow some "best practice"
to declare the function in one place.

Regards,
Diego.-

Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Alejandro F. Reimondo
Hi Diego,

> I know that <> is used in primitives, but in calls to external
> libraries there is a performance reason for this kind of notation?

The method will be activated ONLY on primitive failure.
Activation of method contexts requires processing.
In this situation, the api call will run faster than normal methods.
You must have in mind that the #invoque:... -#call:-
 methods are also primitive methods.
IMHO the reason to use short notation is that it preserve
 the value of senders/implementors ... (user performance reason)

> Have this kind of methods different bycodes (like in primitives) or
> it's only a shorthand for the longer expression?

The api methods are identical to primitive methods, their
 bytecode will be used only on call/primitive failure.
It is success, no bytecode is executed (no method activation).

The no-activation of contexts is an important point to
 consider with FFI mechanisms because it relates with
 the general perfomance.

> > Dynamic function calls (instantiate functions on demand)
> >  can be used, but are more long expressions.
> > With dynamic functions you can call more than
> >  one function in a method, but invalidate
> >  senders/implementors as a powerful tool for browsing
> >  (#call:with:... will return all API calls in the universe).
> > Using <...> syntax let you choose better (smalltalk) selectors
> >  for calls and let you bind the calls without reducing
> >  browsing effectiveness.
>
> What are the differences in browsing?

If you call the senders of #call: a huge amount of calls
 will be reported (all API calls use the same message).
When you use <...> notation, the senders & implementors
 works as usual, retrieving the senders of the API you
 are searching.

> I don't see much appart from less typing, since you can find
> references to ExternalLibraryFunction, and follow some "best practice"
> to declare the function in one place.

The function will be "declared" once if you implement it in one method. :-)
Implementing a function and using it as an object requires
 a global reference.
The reference will be hidden in a method if <...> notation is used.

[si no se entiende, decime y te escribo en castellano a tu casilla]
cheers,
Ale.


----- Original Message -----
From: "Diego Fernandez" <[hidden email]>
To: "The general-purpose Squeak developers list"
<[hidden email]>
Sent: Friday, September 01, 2006 3:40 PM
Subject: Re: FFI Callbacks


> On 9/1/06, Alejandro F. Reimondo <[hidden email]> wrote:
> > The <...> syntax is a short syntax notation to make a method
> >  as a primitive method, calling a function (object) like
> >  you instantiate for platformFunction (the object will be
> >  instantiated by the compiler and bounded to the method
> >  as a primitive) and returning the result (following normal
> >  primitive failure policy).
>
> I know that <> is used in primitives, but in calls to external
> libraries there is a performance reason for this kind of notation?
> Have this kind of methods different bycodes (like in primitives) or
> it's only a shorthand for the longer expression?
>
> > Dynamic function calls (instantiate functions on demand)
> >  can be used, but are more long expressions.
> > With dynamic functions you can call more than
> >  one function in a method, but invalidate
> >  senders/implementors as a powerful tool for browsing
> >  (#call:with:... will return all API calls in the universe).
> > Using <...> syntax let you choose better (smalltalk) selectors
> >  for calls and let you bind the calls without reducing
> >  browsing effectiveness.
>
> What are the differences in browsing?
> I don't see much appart from less typing, since you can find
> references to ExternalLibraryFunction, and follow some "best practice"
> to declare the function in one place.
>
> Regards,
> Diego.-
>


Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Andreas.Raab
In reply to this post by Diego Fernández
Diego Fernandez wrote:
> I know that <> is used in primitives, but in calls to external
> libraries there is a performance reason for this kind of notation?
> Have this kind of methods different bycodes (like in primitives) or
> it's only a shorthand for the longer expression?

It's both a shorthand and a "statically compiled version" of the longer
expression. Here is something that is essentially equivalent:

1. Using <> syntax:

Foo>>callSomething
      <cdecl: void 'callSomething' () module: 'Foo'>
      Transcript cr; show: 'Failed to call function'.

2. Using the (mostly) equivalent explicit version:

Object subclass: #Foo
     instanceVariableNames: ''
     classVariableNames: 'CallSomething'
     poolDictionaries: ''
     category: 'Croquet-Harness'

Foo class>>initialize
     "Construct the FFI call for callSomething"
     CallSomething := ExternalLibraryFunction
         name:'callSomething'
         module: 'Foo'
         callType: ExternalFunction callTypeCDecl
         returnType: ExternalType void
         argumentTypes: #()

Foo>>callSomething
     ^[CallSomething invoke] on: PrimitiveFailed do:[
         Transcript cr; show: 'Failed to call function'.
      ].

The reason this is only "mostly" equivalent is that there isn't a
PrimitiveFailed exception (but there should be ;-)

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Alejandro F. Reimondo
Hi Andreas,
Using the option (2) will activate the method context for Foo>>callSomething
 (a context with a block context....)
The option (1) will execute the primitive WITHOUT method activation.
I think that perfomance of option (1) will be similar to primitives/plugIn
calls
Also the indirection to the global association (CallSomething function)
 must be added; in option (1) the method serve as reference to the function.
Without an optimized VM the penalty of method activation
 and indirection can make a diference.
cheers,
Ale.


----- Original Message -----
From: "Andreas Raab" <[hidden email]>
To: "The general-purpose Squeak developers list"
<[hidden email]>
Sent: Friday, September 01, 2006 5:26 PM
Subject: Re: FFI Callbacks


> Diego Fernandez wrote:
> > I know that <> is used in primitives, but in calls to external
> > libraries there is a performance reason for this kind of notation?
> > Have this kind of methods different bycodes (like in primitives) or
> > it's only a shorthand for the longer expression?
>
> It's both a shorthand and a "statically compiled version" of the longer
> expression. Here is something that is essentially equivalent:
>
> 1. Using <> syntax:
>
> Foo>>callSomething
>       <cdecl: void 'callSomething' () module: 'Foo'>
>       Transcript cr; show: 'Failed to call function'.
>
> 2. Using the (mostly) equivalent explicit version:
>
> Object subclass: #Foo
>      instanceVariableNames: ''
>      classVariableNames: 'CallSomething'
>      poolDictionaries: ''
>      category: 'Croquet-Harness'
>
> Foo class>>initialize
>      "Construct the FFI call for callSomething"
>      CallSomething := ExternalLibraryFunction
>          name:'callSomething'
>          module: 'Foo'
>          callType: ExternalFunction callTypeCDecl
>          returnType: ExternalType void
>          argumentTypes: #()
>
> Foo>>callSomething
>      ^[CallSomething invoke] on: PrimitiveFailed do:[
>          Transcript cr; show: 'Failed to call function'.
>       ].
>
> The reason this is only "mostly" equivalent is that there isn't a
> PrimitiveFailed exception (but there should be ;-)
>
> Cheers,
>    - Andreas
>


Reply | Threaded
Open this post in threaded view
|

Re: FFI Callbacks

Diego Fernández
Again thanks to both for the reply... :)

[se entendio perfecto, gracias :)]

On 9/1/06, Alejandro F. Reimondo <[hidden email]> wrote:

> Hi Andreas,
> Using the option (2) will activate the method context for Foo>>callSomething
>  (a context with a block context....)
> The option (1) will execute the primitive WITHOUT method activation.
> I think that perfomance of option (1) will be similar to primitives/plugIn
> calls
> Also the indirection to the global association (CallSomething function)
>  must be added; in option (1) the method serve as reference to the function.
> Without an optimized VM the penalty of method activation
>  and indirection can make a diference.
> cheers,
> Ale.
>
>
> ----- Original Message -----
> From: "Andreas Raab" <[hidden email]>
> To: "The general-purpose Squeak developers list"
> <[hidden email]>
> Sent: Friday, September 01, 2006 5:26 PM
> Subject: Re: FFI Callbacks
>
>
> > Diego Fernandez wrote:
> > > I know that <> is used in primitives, but in calls to external
> > > libraries there is a performance reason for this kind of notation?
> > > Have this kind of methods different bycodes (like in primitives) or
> > > it's only a shorthand for the longer expression?
> >
> > It's both a shorthand and a "statically compiled version" of the longer
> > expression. Here is something that is essentially equivalent:
> >
> > 1. Using <> syntax:
> >
> > Foo>>callSomething
> >       <cdecl: void 'callSomething' () module: 'Foo'>
> >       Transcript cr; show: 'Failed to call function'.
> >
> > 2. Using the (mostly) equivalent explicit version:
> >
> > Object subclass: #Foo
> >      instanceVariableNames: ''
> >      classVariableNames: 'CallSomething'
> >      poolDictionaries: ''
> >      category: 'Croquet-Harness'
> >
> > Foo class>>initialize
> >      "Construct the FFI call for callSomething"
> >      CallSomething := ExternalLibraryFunction
> >          name:'callSomething'
> >          module: 'Foo'
> >          callType: ExternalFunction callTypeCDecl
> >          returnType: ExternalType void
> >          argumentTypes: #()
> >
> > Foo>>callSomething
> >      ^[CallSomething invoke] on: PrimitiveFailed do:[
> >          Transcript cr; show: 'Failed to call function'.
> >       ].
> >
> > The reason this is only "mostly" equivalent is that there isn't a
> > PrimitiveFailed exception (but there should be ;-)
> >
> > Cheers,
> >    - Andreas
> >
>
>
>