Stability of the external plugin interface

Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Stability of the external plugin interface

Philip Bernhart
Hello,

I want to write an external plugin - I think,
for having a fast interface to a C library.

How is that currently done? How stable is that
interface, will it change in the near future?

What is the current best documentation for doing
that? Is there an example plugin?


Thanks for your time,
Philip
_______________________________________________
VM-beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/vm-beginners
Reply | Threaded
Open this post in threaded view
|

Re: Stability of the external plugin interface

Eliot Miranda-2
Hi Philip,

On Sat, May 23, 2020 at 9:27 AM Philip Bernhart <[hidden email]> wrote:
Hello,

I want to write an external plugin - I think,
for having a fast interface to a C library.

How is that currently done?

Either by writing a plugin or directly through the FFI.  We can discuss how to make the choice between the two but I lean towards the FFI.
 
How stable is that
interface, will it change in the near future?

The interface is not stable, but it is stable w.r.t. backward compatibility.  So while the interface is evolving, it is safe to write against.
 
What is the current best documentation for doing
that? Is there an example plugin?

This Squeak Wiki page is an overview and a series of pointers to articles, but I don't know how up-to-date it or the articles are. 


 Tim, any suggestions on the most up-to-date external plugin doc is?

Thanks for your time,
Philip
_______________________________________________
VM-beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/vm-beginners


--
_,,,^..^,,,_
best, Eliot

_______________________________________________
VM-beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/vm-beginners
Reply | Threaded
Open this post in threaded view
|

Re: Stability of the external plugin interface

Philip Bernhart
Hi Eliot,

Eliot Miranda <[hidden email]> writes:

> Hi Philip,
>
> On Sat, May 23, 2020 at 9:27 AM Philip Bernhart <[hidden email]>
> wrote:
>
>> Hello,
>>
>> I want to write an external plugin - I think,
>> for having a fast interface to a C library.
>>
>> How is that currently done?
>
>
> Either by writing a plugin or directly through the FFI.  We can discuss how
> to make the choice between the two but I lean towards the FFI.

Why is that? I see the case for calling external libraries in a more
general fashion FFI. I see why people don't want to have additional
plugins in the VM too, more stuff to maintain -> more bugs, also some
VM internal resource exhaustion issues associated with internal
plugins(?), etc.
I'm not asking for that, I'm talking about external plugins.

I read a little bit the Swiki articles about FFI / Plugins and people
who wrote them generally seemed to lean toward plugins, when it comes to
a layer provided to the users of the functionality which is supposed
to be put into the plugin / library. And they seemed to be a little
bit faster than the FFI interface.

So what are your reasons for prefering FFI over External Plugins? :-)


Thanks for your time,
Philip
_______________________________________________
VM-beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/vm-beginners
Reply | Threaded
Open this post in threaded view
|

Re: Stability of the external plugin interface

Philip Bernhart
Hi Eliot,


Eliot Miranda <[hidden email]> writes:

> I have just spent several months porting the 32-bit
> Qwaq/Teleplace/3DICC plug-ins to 64-bit in Terf, the
> Croquet-based business communication tool, first on macOS,
> second on Windows.  It has been an enormously expensive and slow
> process.  Part of the problem is that some plugins involve three
> languages, C in the Slang-generated wrapper, C++ in the generic Qwaq
> library and the boost support libs (queues, locking, synchronization,
> etc), and Objective-C.  Part of the problem is using low-level
> debuggers for the static languages involved.  It is incredibly slow
> and costly work.
>
> Ronie has had to implement a new rendering framework because (I think,
> Ronie can explain better than me), Apple is not interested in
> supporting OpenGL in 64-bits as the world transitions to graphics
> co-processing for rendering, and also because he can do a better job
> himself than being concerned strained by an aging API such as OpenGL.
>
> Apart from the fact that Ronie is a better programmer than I, Ronie’s
> productivity has been much higher. I was porting existing code, and he
> was writing new code.  I was working with low-level code expressed as
> plugins. He had built a meta-programming facility in Smalltalk and
> generates an FFI binding and a backend composed of gpu-specific back
> end dlls in C++.
>
> Apart from security and packaging, plugins are a disaster: - having to
> generate the plugin wrapper means two focii for development, in
> Smalltalk and on the platform (Ronie’s approach has the same issue but
> not so bad because apart from building and debugging the gpu-specific
> dlls the focus is in Smalltalk, where it should be).
>
>- architecturally plug-ins force inefficiencies due to marshaling across
> the Smalltalk/platform boundary and force data copying (we have
> pinning now, so copying can and should be reduced, in using plugins or
> using the FFI, but the Qwaq plugins were written before pinning was
> available).  - browsing the plugins cannot be done in a single IDE;
> there aren’t good IDEs on linux, MacOS and Windows don’t share a
> common IDE, so changing a plug-in API takes a lot of browsing effort

> - architecturally the plugin forces the definition of *three* APIs,
> the plugin API between Smalltalk and the plugin’s internals, and the
> common internal API between the marshaled Smalltalk objects in the
> Slang-generated C plugin wrapper and the platform-specific code, and
> finally between this common internal API and each platform’s
> facilities. In an FFI-based implementation there is at most one API
> per platform, and it is easier to use facilities such as pinning to
> ease passing Smallta lk objects to external code.

Ok, I think I haven't yet understood the way of how external plugins
interact with the VM. I thought I saw only one. The interpreterProxy
defined in sqVirtualMachine.h and the initialiseModule, setInterpreter,
getModuleName and shutdownModule callbacks.

> Going forward, we will have a much more effective optimizer in
> Sista/Scorch and a much more flexible threaded FFI, along with an
> incremental global garbage collector.  This should provide facilities
> that make it feasible to move more computation into Smalltalk. Within
> Smalltalk the facilities for creating platform-neutral APIs that
> internally map to platform-specific APIs accesses through the FFI are
> much better than the static language facilities provided by
> C/C++/Objective-C et al, and the programming experience is much easier
> because there is only one foreign API to deal with, not three.  The
> rate of development is much much faster.

I'm looking forward to it, but in the meantime I need to write packages
for Cuis Smalltalk which I need which in turn go over the slow legacy
Squeak FFI with no callbacks. Is there a roadmap, some public plan of
these implementations? I think that ideas are already around a long time
already and it seems that the VM development is heavily underfunded or
else it would already be here.

> For these reasons I think it would be a huge mistake to develop any new
> rendering framework or platform interface using plugins.  The
> development should be in Smalltalk across the FFI.  Once a facility
> has been developed then one could auto generate a sandboxed
> implementation which replaces an open and potentially insecure FFI
> interface with a black box plugin, but this would be a packaging
> option, and not the main development line.

So.. if people would generate the whole "plugin" from something like
Slang and could just decide by a flag or something if the interface to
it should be over the FFI or an external plugin, would that be ok?
I have seen tries like this like the lowtalk implementation
and my current archeological project https://github.com/Phantasus/Cuis-Smalltalk-Slang/


May you feel well and safe,
Philip
_______________________________________________
VM-beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/vm-beginners