Re: [Vm-beginners] Stability of the external plugin interface

 

Hi Eliot,

Eliot Miranda <[hidden email]> writes:

Hi Philip,

On Sat, May 23, 2020 at 9:27 AM Philip Bernhart <[hidden email]>

wrote:

Hello,

I want to write an external plugin - I think,

for having a fast interface to a C library.

How is that currently done?

Either by writing a plugin or directly through the FFI.  We can discuss how

to make the choice between the two but I lean towards the FFI.

Why is that? I see the case for calling external libraries in a more
general fashion FFI. I see why people don't want to have additional
plugins in the VM too, more stuff to maintain -> more bugs, also some
VM internal resource exhaustion issues associated with internal
plugins(?), etc.
I'm not asking for that, I'm talking about external plugins.

I read a little bit the Swiki articles about FFI / Plugins and people
who wrote them generally seemed to lean toward plugins, when it comes to
a layer provided to the users of the functionality which is supposed
to be put into the plugin / library. And they seemed to be a little
bit faster than the FFI interface.

So what are your reasons for prefering FFI over External Plugins? :-)

here is an extract of a recent conversation with Marcel Taumael and others that started off-line.  It states my reasons for preferring FFI over plugins.

 

> On 2020-05-24, at 3:57 PM, Eliot Miranda <[hidden email]> wrote:
>
> Plugins emerged from the lack of an FFI, later a deficient FFI (slow, non-reentrant, no callbacks), and the need for security, at a time when graphics co-processing was very rare and not at all commodity technology.  The world looks different now.  We should evolve to match the new environment.  I agree that security is very important.  But I do not believe that security implies no FFI. I think we can have both security and the flexibility, development speed and architectural efficiency of an FFI-based implementation.


The VM plugins were an attempt to get away from a few problems that plagued VM development at the end of the last millennium (and I don't mean March this year) with things like a limited number of prim table entires, the lack of flexibility a static table engendered, the time it took to compile an entire VM for a tiny change/test cycle etc (IIRC of the order of half an hour back then). The original external plugins stuff meant you could get to outside code via a name system that didn't screw the prim table (and I'd still love to find away to get rid of it entirely, but dream on)  and that let you build/load/try/unload/rebuild quickly. We were also starting the beginnings of the ffi stuff around then. Andy Greenberg came up with the SmartSyntaxPlugin stuff, for example, at the end of '99. Apparently I sent Dan/Andreas a final pluginized linux vm late April 2000.

It's worth remembering just how awful Windows & Mac OS were back then as part of the wider scenario - non-preemptive processscheduling, weird callback based event reading, general nonsense. Checking ancient emails to find dates etc reminded me of a huge list of other stuff we were working on back then that *still* needs finishing! And the early pinned objects stuff... and private methods! And...!) Oh-ho - and as an amusing aside, an email from Eliot of that sort of date was pointing out how amazing it was to be able to get a 30Gb disk for a mere $190; you can get 10TB now.

I definitely agree that for an awful lot of cases using a good ffi is better nowadays. It does have some costs though; but in some case I think those are costs we should be pleased to pay. For example, a number of current plugins would be better replaced with ffi classes that actually work with the OS facilities rather than trying to work around them. The FilePlugin comes readily to mind. How much nicer it would be to handle files with direct ffi calls to OS routines and have the complicated stuff in Smalltalk bridge classes. Hmm, now where did we have this in the past; oh yes, in the BrouHaHa images 30-some years ago.

Moving to ffi-only would allow removing a fair bit of cruft from the VM, though there's our old friend 'backward compatibility' to argue with. It would mean some rewriting (mostly simplification I think?) of existing plugins code. And what to do about stuff that we pretty much always compile as internal plugins? Rewrite as external libraries and pay the (hopefully small) performance cost, or rewrite as appropriate to drop any pretence of being able to be external? SoundPlugin should absolutely be replaced by a bridge class and ffi calls to the OS. LargeInterger? Probably much better to pull properly into the vm core. MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.

Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: DCVP: Destroy another Computer Via Phone-link

 

+1

> Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.

+1000

> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> Strange OpCodes: DCVP: Destroy another Computer Via Phone-link
>
>

 

> On 2020-05-24, at 3:57 PM, Eliot Miranda <[hidden email]> wrote:
>
> Plugins emerged from the lack of an FFI, later a deficient FFI (slow, non-reentrant, no callbacks), and the need for security, at a time when graphics co-processing was very rare and not at all commodity technology. The world looks different now. We should evolve to match the new environment. I agree that security is very important. But I do not believe that security implies no FFI. I think we can have both security and the flexibility, development speed and architectural efficiency of an FFI-based implementation.


The VM plugins were an attempt to get away from a few problems that plagued VM development at the end of the last millennium (and I don't mean March this year) with things like a limited number of prim table entires, the lack of flexibility a static table engendered, the time it took to compile an entire VM for a tiny change/test cycle etc (IIRC of the order of half an hour back then). The original external plugins stuff meant you could get to outside code via a name system that didn't screw the prim table (and I'd still love to find away to get rid of it entirely, but dream on) and that let you build/load/try/unload/rebuild quickly. We were also starting the beginnings of the ffi stuff around then. Andy Greenberg came up with the SmartSyntaxPlugin stuff, for example, at the end of '99. Apparently I sent Dan/Andreas a final pluginized linux vm late April 2000.

It's worth remembering just how awful Windows & Mac OS were back then as part of the wider scenario - non-preemptive processscheduling, weird callback based event reading, general nonsense. Checking ancient emails to find dates etc reminded me of a huge list of other stuff we were working on back then that *still* needs finishing! And the early pinned objects stuff... and private methods! And...!) Oh-ho - and as an amusing aside, an email from Eliot of that sort of date was pointing out how amazing it was to be able to get a 30Gb disk for a mere $190; you can get 10TB now.

I definitely agree that for an awful lot of cases using a good ffi is better nowadays. It does have some costs though; but in some case I think those are costs we should be pleased to pay. For example, a number of current plugins would be better replaced with ffi classes that actually work with the OS facilities rather than trying to work around them. The FilePlugin comes readily to mind. How much nicer it would be to handle files with direct ffi calls to OS routines and have the complicated stuff in Smalltalk bridge classes. Hmm, now where did we have this in the past; oh yes, in the BrouHaHa images 30-some years ago.

Moving to ffi-only would allow removing a fair bit of cruft from the VM, though there's our old friend 'backward compatibility' to argue with. It would mean some rewriting (mostly simplification I think?) of existing plugins code. And what to do about stuff that we pretty much always compile as internal plugins? Rewrite as external libraries and pay the (hopefully small) performance cost, or rewrite as appropriate to drop any pretence of being able to be external? SoundPlugin should absolutely be replaced by a bridge class and ffi calls to the OS. LargeInterger? Probably much better to pull properly into the vm core. MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.

Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.

How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms. 

 
Hi
> On 25.05.2020, at 05:35, Eliot Miranda <[hidden email]> wrote:
>>

>> Moving to ffi-only would allow removing a fair bit of cruft from the VM, though there's our old friend 'backward compatibility' to argue with. It would mean some rewriting (mostly simplification I think?) of existing plugins code. And what to do about stuff that we pretty much always compile as internal plugins? Rewrite as external libraries and pay the (hopefully small) performance cost, or rewrite as appropriate to drop any pretence of being able to be external? SoundPlugin should absolutely be replaced by a bridge class and ffi calls to the OS. LargeInterger? Probably much better to pull properly into the vm core. MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.
>
> +1
>
>> Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.
>
> +1000


One thing tho.

This all boils down to "who moves faster".

Here's a story.

When I tried to get the Self VM running again some years ago, one of the agonizing parts was its
extensive use of FFI. Why so? Well, the FFI was coded against a mid-1990 X11. Which, in that form
did not exist on either Mac nor Linux. Some parts worked, but a lot of things needed attention
in _both_ the VM _and_ the Image. In some respect, the "libraries interfaced via FFI" had moved
faster than the VM.

Had Self had a Display Plugin just like Squeak, changes would have been much more simple, since the
assumptions in the Self world had nod changed at all.

So, if things move slowly in VM/Image land and more quickly in library land, the answer is Plugins.
If things move quickly in VM/Image land and are stable in library land, the answer is FFI.

I see this in SqueakSSL very prominently. The stable interface from the Image land helps a lot.
It would be a nightmare to have this via FFI. The effect would be that we have to
settle on one SSL library on all platforms, and keep that updated and ship it with the VM.
That's by the way what the Pharo VM does and I think it is the wrong way for the problem.

Best regards
        -Tobias

 

 

Rather the latter. I wouldn't call it hot-fix but rather evolve stuff as they change.
That's how we do with Display and SqueakSSL.
You don't need to touch the image, but you could.
I don't understand how "proper modules" in whatever sense would fix that.

:)

Best regards
        -Tobias

 
OK, I'll try to answer some of the thoughts on this -

> On 2020-05-24, at 11:07 PM, Robert <[hidden email]> wrote:
>
> How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
>
> Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?

Right now (assuming I'm remembering it all correctly and with the proviso that I haven't written a new plugin in several years) an external plugin is compiled from the assorted code required (some plugins are solely slang generated code, some are solely hand-written C, some are slang + C + calls to other libraries and so on) and a platform specific form of shared library is made. Are they in a form that is callable via 'normal' ffi calls? Dunno, but clearly they *could* be made that way with not very much change.



> On 2020-05-24, at 11:26 PM, Tobias Pape <[hidden email]> wrote:
>  The effect would be that we have to
> settle on one SSL library on all platforms, and keep that updated and ship it with the VM.
> That's by the way what the Pharo VM does and I think it is the wrong way for the problem.

Actually no, that isn't how you *have* to do it. You can have interfaces from the image to many different libraries; hence the mentions of 'bridge classes' in my message. It's not much different to how we currently choose the platform appropriate class of file directory on startup.

When the interface to a subsystem changes you have to make corresponding changes *somewhere*. With current plugins we have to rewrite the relevant parts, often the hand-written C-shim stuff. In one sense that is a virtue since we can do that with 'normal' tools like edlin, and enjoy all the fun of that world of high quality IDEs (sarcasm? Moi?). If we used bridge classes to ffi calls direct to the external system then we would have to do some work in the image; now of course in general we'd all prefer that but I can imagine cases where the changed interface prevents us running the image to write the code to handle the changed interface. This is where having a way to make the system load code at startup helps.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Eagles may soar, but weasels aren't sucked into jet engines.

 
On Sun, 24 May 2020, tim Rowledge wrote:

> MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.

Below is a benchmark measuring the primitive to calculate string and byte
array hashes. The in-image variant is fully JIT-friendly: AFAIK all its
message sends have jitted implementations (#bitAnd:, #size, #to:do:, #+,
#basicAt:, #hashMultiply).

| b |
b := ByteArray new: 10000.
{[ b hashWithInitialHash: 0 ] benchFor: 1 seconds.
[ String stringHash: b initialHash: 0 ] benchFor: 1 seconds }.
#('88,500 per second. 11.3 microseconds per run. 0 % GC time.' '11,500 per
second. 86.9 microseconds per run. 0 % GC time.')"

In this case, the primitive version is 7.7x faster on my machine.

The actual performance difference depends on the size of the collection.
For sizes smaller than 4, the primitive version is slower. That could
probably be improved by making it a numberered primitive and providing a
jitted version of the primitive; something that was done with
#compare:with:collated: (see primitive 158).

At this point we clearly can't drop those primitives. Some of them is
probably worth to add to the VM directly, others may not be necessary at
all in the future.


Levente

>
>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> Strange OpCodes: DCVP: Destroy another Computer Via Phone-link

 

Per se, true. But try discriminating openssl 0.9.6, 1.0.1, and 1.1.1. It's a nightmare. The overlay somewhat works, but even that is not so easy.

It's just that I've seen the bridge-class stuff made things hard for me in Self.
But that might just be the long time gap where X11 evolved and the Self part did not. Problem was I had no tools because I broke my tools with my tools (so to speak, because no UI, no way of fixing the UI properly).


>
> When the interface to a subsystem changes you have to make corresponding changes *somewhere*. With current plugins we have to rewrite the relevant parts, often the hand-written C-shim stuff. In one sense that is a virtue since we can do that with 'normal' tools like edlin, and enjoy all the fun of that world of high quality IDEs (sarcasm? Moi?).

Not that I like digging in C more than using Squeak proper...

> If we used bridge classes to ffi calls direct to the external system then we would have to do some work in the image; now of course in general we'd all prefer that but I can imagine cases where the changed interface prevents us running the image to write the code to handle the changed interface. This is where having a way to make the system load code at startup helps.


Again, FFI is fine. My point is rather that the pace of change can indicate whether a plugin is better or FFI.

Also, the other VM-implementations can provide Plugins much more easily than FFI.
I would argue this is true for SqueakJS, RSqueak, JSqueak/Potato, and to an extend, TruffleSqueak.
These systems do not necessarily provide access to C libraries and reduce the need for in-image changes for each VM.


TL;DR: FFI good, Plugins also good.

just my 2ct.

Best regards
        -Tobias

 
On 25/05/20 5:04 pm, Jakob Reschke wrote:
>
> If a library breaks compatibility, you would have to update the plugin
> and maybe think about how to stay compatible with older Squeaks. If it
> were FFI with a proper facade that finds out how to link to the external
> library, you would need to update the facade and worry there about
> staying compatible, but you could do it in Smalltalk. It requires
> discipline and guidance to only use the facade, of course.
Libraries like OpenGL are frustrating because of poorly abstracted, low
level functions. Trying to stable facade around its warts will not solve
the core problem of having to deal with static code and compile cycles.

The frustration is having to regress from live programming into
code-compile-link-test cycle in 2020! Given a choice between live
programming and backward compatibility, I would choose live programming
because I can close any gaps through live coding.

For instance, a few years back I developed live previews for LaTeX
without leaving Squeak (Etoys) by using OSProcess to build a LaTeX
pipeline that converted code into image as the code was being typed. It
is an inefficient design but it took just half a day and has served well
over the years.

But this was at process level. If only I could do the same with DLLs. If
only Squeak had a tool (ELF reader?) to reify a DLL, extract its
instruction set, interface definitions, state variables and calling
conventions. Then I can livecode a class and dynamically compile it (now
that we have JIT) to call functions DLL without leaving the image. The
code may not be 'portable' in traditional sense. But it will be a lot
more manageable than static designs.

Regards .. Subbu

 
On Mon, May 25, 2020 at 12:46, tim Rowledge <[hidden email]> wrote:

OK, I'll try to answer some of the thoughts on this -

> On 2020-05-24, at 11:07 PM, Robert <[hidden email]> wrote:
>
> How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
>
> Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?

Right now (assuming I'm remembering it all correctly and with the proviso that I haven't written a new plugin in several years) an external plugin is compiled from the assorted code required (some plugins are solely slang generated code, some are solely hand-written C, some are slang + C + calls to other libraries and so on) and a platform specific form of shared library is made. Are they in a form that is callable via 'normal' ffi calls? Dunno, but clearly they *could* be made that way with not very much change.



> On 2020-05-24, at 11:26 PM, Tobias Pape <[hidden email]> wrote:
> The effect would be that we have to
> settle on one SSL library on all platforms, and keep that updated and ship it with the VM.
> That's by the way what the Pharo VM does and I think it is the wrong way for the problem.

Actually no, that isn't how you *have* to do it. You can have interfaces from the image to many different libraries; hence the mentions of 'bridge classes' in my message. It's not much different to how we currently choose the platform appropriate class of file directory on startup.

When the interface to a subsystem changes you have to make corresponding changes *somewhere*. With current plugins we have to rewrite the relevant parts, often the hand-written C-shim stuff. In one sense that is a virtue since we can do that with 'normal' tools like edlin, and enjoy all the fun of that world of high quality IDEs (sarcasm? Moi?). If we used bridge classes to ffi calls direct to the external system then we would have to do some work in the image; now of course in general we'd all prefer that but I can imagine cases where the changed interface prevents us running the image to write the code to handle the changed interface. This is where having a way to make the system load code at startup helps.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Eagles may soar, but weasels aren't sucked into jet engines.

 

OK, I'll try to answer some of the thoughts on this -

> On 2020-05-24, at 11:07 PM, Robert <[hidden email]> wrote:
>
> How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
>
> Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?

Right now (assuming I'm remembering it all correctly and with the proviso that I haven't written a new plugin in several years) an external plugin is compiled from the assorted code required (some plugins are solely slang generated code, some are solely hand-written C, some are slang + C + calls to other libraries and so on) and a platform specific form of shared library is made. Are they in a form that is callable via 'normal' ffi calls? Dunno, but clearly they *could* be made that way with not very much change.

Tim, does this mean that the plugin generation and linking as appropriate continues a pace, while the in-image named prims table switches all entries to FFI calls? CryptographyPlugins have the advantage of slang generated only code.

 


> On 2020-05-25, at 10:25 AM, K K Subbu <[hidden email]> wrote:
>
> On 25/05/20 5:04 pm, Jakob Reschke wrote:
>> If a library breaks compatibility, you would have to update the plugin and maybe think about how to stay compatible with older Squeaks. If it were FFI with a proper facade that finds out how to link to the external library, you would need to update the facade and worry there about staying compatible, but you could do it in Smalltalk. It requires discipline and guidance to only use the facade, of course.
> Libraries like OpenGL are frustrating because of poorly abstracted, low level functions. Trying to stable facade around its warts will not solve the core problem of having to deal with static code and compile cycles.
>
> The frustration is having to regress from live programming into code-compile-link-test cycle in 2020! Given a choice between live programming and backward compatibility, I would choose live programming because I can close any gaps through live coding.


Nicely put.
>

> For instance, a few years back I developed live previews for LaTeX without leaving Squeak (Etoys) by using OSProcess to build a LaTeX pipeline that converted code into image as the code was being typed. It is an inefficient design but it took just half a day and has served well over the years.
>
> But this was at process level. If only I could do the same with DLLs. If only Squeak had a tool (ELF reader?) to reify a DLL, extract its instruction set, interface definitions, state variables and calling conventions. Then I can livecode a class and dynamically compile it (now that we have JIT) to call functions DLL without leaving the image. The code may not be 'portable' in traditional sense. But it will be a lot more manageable than static designs.


VW has a parser of C header files that is I suppose the right way to do this sort of thing - you parse the headers (warning - I seem to recall parsing 'windows.h' taking several days in the original VW ddl product) and build proxies of all the types/structs/etc.  Is that a practical way to do it? Hell of a lot of work to do, certainly.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: QVC: Question Valid Command

 
On 26/05/20 2:20 am, tim Rowledge wrote:
We don't know what source code was used to compile the DLL. Having an
inspector for DLLs would be a good start. If we pull out the dynamic
link maps into the image, then we can simulate in the image, what VM
currently does through dlopen() calls. Like what we do in doPrimitive.

Regards .. Subbu

 


Certainly true in a lot of cases; the idea used to be that you could get a header file that described the interface and so compile appropriate code. I'm sure there are plenty of cases where that is not what actually happens.

> Having an inspector for DLLs would be a good start. If we pull out the dynamic link maps into the image, then we can simulate in the image, what VM currently does through dlopen() calls. Like what we do in doPrimitive.

That would be nice. And of course Apple bundles, unix .so's and RISC OS shared libraries. A small matter of programming :-)


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful Latin Phrases:- Magister Mundi sum! = I am the Master of the Universe!

 
I was hoping getting a solid libCLANG interface would make this easy.

 
Hi Tim,

Emphatically not.  DLLCC’s C parser is a nightmare to maintain, and with a little cunning, essentially unnecessary. See all the conversation around generating a C program which when run outputs information that the image can easily parse, and as a result (re)populate a shared pool with relevant platform specifics.  We have this in FFISharedPool (by Shaping IIRC, after an idea/suggestion of mine, implemented as a prototype before I left Cincom, and as a part of UFFI by Mariano Martinez Peck, following discussion a few years back).

Generating a program has several advantages

- no need to implement any part of the C toolchain (doesn’t reinvent the wheel)

- uses the platform’s information more-or-less directly, rather than trying to mimic it

- the binaries can be shared between developers (a library of them?) and deployed with cross-platform apps

- only the FFI api author needs a C development toolset


So what the FFI author does is define a shared pool which has platform-specific metadata, specifically the necessary include files and version selection defines for each relevant platform.  The FFI shared pool machinery takes care of generating the program when necessary (only when authoring an FFI), and running it when necessary (only once for a single-platform app, only when launching on a different platform to the one snapshotted on)


>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> Strange OpCodes: QVC: Question Valid Command
>
>

 

I was hoping getting a solid libCLANG interface would make this easy.

But I haven't been able to get past a problem with some functions returning structs by value and the FFI failing to handle this.

Bugs were fixed on x86_64 recently and ARMv8 less recently.

 


> On 2020-05-26, at 8:10 PM, Eliot Miranda <[hidden email]> wrote:
> DLLCC’s C parser is a nightmare to maintain,

I certainly remember *that*. Poor Marcel Schelvis practically went mad working on it.

>  So what the FFI author does is define a shared pool which has platform-specific metadata, specifically the necessary include files and version selection defines for each relevant platform.  The FFI shared pool machinery takes care of generating the program when necessary (only when authoring an FFI), and running it when necessary (only once for a single-platform app, only when launching on a different platform to the one snapshotted on)

Sounds good to me. Beats parsing secondhand and possibly inaccurate information intended for an inferior language


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Fractured Idiom:- MERCI RIEN - Thanks for nothin'.