Re: [Vm-beginners] Stability of the external plugin interface

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Eliot Miranda-2
 
Hi Philip, Hi All,

On May 24, 2020, at 2:37 PM, Philip Bernhart <[hidden email]> wrote:

Hi Eliot,

Eliot Miranda <[hidden email]> writes:

Hi Philip,

On Sat, May 23, 2020 at 9:27 AM Philip Bernhart <[hidden email]>
wrote:

Hello,

I want to write an external plugin - I think,
for having a fast interface to a C library.

How is that currently done?


Either by writing a plugin or directly through the FFI.  We can discuss how
to make the choice between the two but I lean towards the FFI.

Why is that? I see the case for calling external libraries in a more
general fashion FFI. I see why people don't want to have additional
plugins in the VM too, more stuff to maintain -> more bugs, also some
VM internal resource exhaustion issues associated with internal
plugins(?), etc.
I'm not asking for that, I'm talking about external plugins.

I read a little bit the Swiki articles about FFI / Plugins and people
who wrote them generally seemed to lean toward plugins, when it comes to
a layer provided to the users of the functionality which is supposed
to be put into the plugin / library. And they seemed to be a little
bit faster than the FFI interface.

So what are your reasons for prefering FFI over External Plugins? :-)

here is an extract of a recent conversation with Marcel Taumael and others that started off-line.  It states my reasons for preferring FFI over plugins.

———8<———
On May 13, 2020, at 5:58 AM, Marcel Taeumel <[hidden email]> wrote:


Hi Nicolas, hi Eliot, hi Levente, hi Vanessa.

What is your opinion about "SDL through FFI" as it is proposed as the big thing in the "minheadless" branch and heavily used in recent Pharo versions?

I think that
- Yes, it is a good idea to reduce the need for platform-specific code.
- No, it is a terrible idea to let SDL (or any C) internals bleed in to the regular Squeak/Smalltalk image.
- Instead, we should design a simple "WindowPlugin" that can easily map to SDL in a cross-platform fashion. Such a plugin can be extended to provide debugging support in a domain-specific way.
- However, it should always remain possible to provide alternative backends to SDL (such as Qt or native OS libs) for experimentation or workarounds or bugfixes. Just like the "Project" abstraction in Squeak to keep MVC and Morphic and whatever around at the same time.

Isn't that what the unix part is doing anyway? vm-display-X11, vm-display-Quartz ...

I have just spent several months porting the 32-bit Qwaq/Teleplace/3DICC plug-ins  to 64-bit in Terf, the Croquet-based business communication tool, first on macOS, second on Windows.  It has been an enormously expensive and slow process.  Part of the problem is that some plugins involve three languages, C in the Slang-generated wrapper, C++ in the generic Qwaq library and the boost support libs (queues, locking, synchronization, etc), and Objective-C.  Part of the problem is using low-level debuggers for the static languages involved.  It is incredibly slow and costly work.

Ronie has had to implement a new rendering framework because (I think, Ronie can explain better than me), Apple is not interested in supporting OpenGL in 64-bits as the world transitions to graphics co-processing for rendering, and also because he can do a better job himself than being concerned strained by an aging API such as OpenGL.

Apart from the fact that Ronie is a better programmer than I, Ronie’s productivity has been much higher. I was porting existing code, and he was writing new code.  I was working with low-level code expressed as plugins. He had built a meta-programming facility in Smalltalk and generates an FFI binding and a backend composed of gpu-specific back end dlls in C++.

Apart from security and packaging, plugins are a disaster:
- having to generate the plugin wrapper means two focii for development, in Smalltalk and on the platform (Ronie’s approach has the same issue but not so bad because apart from building and debugging the gpu-specific dlls the focus is in Smalltalk, where it should be).
- architecturally plug-ins force inefficiencies due to marshaling across the Smalltalk/platform boundary and force data copying (we have pinning now, so copying can and should be reduced, in using plugins or using the FFI, but the Qwaq plugins were written before pinning was available).
- browsing the plugins cannot be done in a single IDE; there aren’t good IDEs on linux, MacOS and Windows don’t share a common IDE, so changing a plug-in API takes a lot of browsing effort
- architecturally the plugin forces the definition of *three* APIs, the plugin API between Smalltalk and the plugin’s internals, and the common internal API between the marshaled Smalltalk objects in the Slang-generated C plugin wrapper and the platform-specific code, and finally between this common internal API and each platform’s facilities. In an FFI-based implementation there is at most one API per platform, and it is easier to use facilities such as pinning to ease passing Smalltalk objects to external code.

Going forward, we will have a much more effective optimizer in Sista/Scorch and a much more flexible threaded FFI, along with an incremental global garbage collector.  This should provide facilities that make it feasible to move more computation into Smalltalk. Within Smalltalk the facilities for creating platform-neutral APIs that internally map to platform-specific APIs accesses through the FFI are much better than the static language facilities provided by C/C++/Objective-C et al, and the programming experience is much easier because there is only one foreign API to deal with, not three.  The rate of development is much much faster.

If we stay with plugins we fall into the Python trap.  The plugin legacy is hugely expensive and restrictive, and prevents us from evolving, both because there is substantial investment and performance in the plugins and because they are inherently more viscous (for the 3 API reasons above).

For these reasons I think it would be a huge mistake to develop any new rendering framework or platform interface using plugins.  The development should be in Smalltalk across the FFI.  Once a facility has been developed then one could auto generate a sandboxed implementation which replaces an open and potentially insecure FFI interface with a black box plugin, but this would be a packaging option, and not the main development line.

Further, we could perhaps develop a different style of security, building on the ideas of the SecurityPlugin.  Perhaps we could parameterize the FFI so it will only allow certain functions to be called.  Imagine for example an FFISecurityPlugin which is built statically that combines a list of names (the names of the foreign functions that the image is allowed to call), with the dlls the FFI is allowed to use (either the dlls themselves or their names).  Attempts to use foreign functions or dlls outside the names in the predefined set, would be outlawed.

So my experience, not just the recent experience with 3D ICC, but also many years with VisualWorks, and some before with BrouHaHa, leads me to push strongly for embracing the FFI and addressing security issues directly, without tying our hands and putting an albatross around our necks to weigh us down.

Plugins emerged from the lack of an FFI, later a deficient FFI (slow, non-reentrant, no callbacks), and the need for security, at a time when graphics co-processing was very rare and not at all commodity technology.  The world looks different now.  We should evolve to match the new environment.  I agree that security is very important.  But I do not believe that security implies no FFI. I think we can have both security and the flexibility, development speed and architectural efficiency of an FFI-based implementation.
———8<———



Thanks for your time,
Philip
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

timrowledge
 

> On 2020-05-24, at 3:57 PM, Eliot Miranda <[hidden email]> wrote:
>
> Plugins emerged from the lack of an FFI, later a deficient FFI (slow, non-reentrant, no callbacks), and the need for security, at a time when graphics co-processing was very rare and not at all commodity technology.  The world looks different now.  We should evolve to match the new environment.  I agree that security is very important.  But I do not believe that security implies no FFI. I think we can have both security and the flexibility, development speed and architectural efficiency of an FFI-based implementation.


The VM plugins were an attempt to get away from a few problems that plagued VM development at the end of the last millennium (and I don't mean March this year) with things like a limited number of prim table entires, the lack of flexibility a static table engendered, the time it took to compile an entire VM for a tiny change/test cycle etc (IIRC of the order of half an hour back then). The original external plugins stuff meant you could get to outside code via a name system that didn't screw the prim table (and I'd still love to find away to get rid of it entirely, but dream on)  and that let you build/load/try/unload/rebuild quickly. We were also starting the beginnings of the ffi stuff around then. Andy Greenberg came up with the SmartSyntaxPlugin stuff, for example, at the end of '99. Apparently I sent Dan/Andreas a final pluginized linux vm late April 2000.

It's worth remembering just how awful Windows & Mac OS were back then as part of the wider scenario - non-preemptive processscheduling, weird callback based event reading, general nonsense. Checking ancient emails to find dates etc reminded me of a huge list of other stuff we were working on back then that *still* needs finishing! And the early pinned objects stuff... and private methods! And...!) Oh-ho - and as an amusing aside, an email from Eliot of that sort of date was pointing out how amazing it was to be able to get a 30Gb disk for a mere $190; you can get 10TB now.

I definitely agree that for an awful lot of cases using a good ffi is better nowadays. It does have some costs though; but in some case I think those are costs we should be pleased to pay. For example, a number of current plugins would be better replaced with ffi classes that actually work with the OS facilities rather than trying to work around them. The FilePlugin comes readily to mind. How much nicer it would be to handle files with direct ffi calls to OS routines and have the complicated stuff in Smalltalk bridge classes. Hmm, now where did we have this in the past; oh yes, in the BrouHaHa images 30-some years ago.

Moving to ffi-only would allow removing a fair bit of cruft from the VM, though there's our old friend 'backward compatibility' to argue with. It would mean some rewriting (mostly simplification I think?) of existing plugins code. And what to do about stuff that we pretty much always compile as internal plugins? Rewrite as external libraries and pay the (hopefully small) performance cost, or rewrite as appropriate to drop any pretence of being able to be external? SoundPlugin should absolutely be replaced by a bridge class and ffi calls to the OS. LargeInterger? Probably much better to pull properly into the vm core. MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.

Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: DCVP: Destroy another Computer Via Phone-link


Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Eliot Miranda-2
 

> On May 24, 2020, at 8:14 PM, tim Rowledge <[hidden email]> wrote:
>
> 
>
>> On 2020-05-24, at 3:57 PM, Eliot Miranda <[hidden email]> wrote:
>>
>> Plugins emerged from the lack of an FFI, later a deficient FFI (slow, non-reentrant, no callbacks), and the need for security, at a time when graphics co-processing was very rare and not at all commodity technology.  The world looks different now.  We should evolve to match the new environment.  I agree that security is very important.  But I do not believe that security implies no FFI. I think we can have both security and the flexibility, development speed and architectural efficiency of an FFI-based implementation.
>
>
> The VM plugins were an attempt to get away from a few problems that plagued VM development at the end of the last millennium (and I don't mean March this year) with things like a limited number of prim table entires, the lack of flexibility a static table engendered, the time it took to compile an entire VM for a tiny change/test cycle etc (IIRC of the order of half an hour back then). The original external plugins stuff meant you could get to outside code via a name system that didn't screw the prim table (and I'd still love to find away to get rid of it entirely, but dream on)  and that let you build/load/try/unload/rebuild quickly. We were also starting the beginnings of the ffi stuff around then. Andy Greenberg came up with the SmartSyntaxPlugin stuff, for example, at the end of '99. Apparently I sent Dan/Andreas a final pluginized linux vm late April 2000.
>
> It's worth remembering just how awful Windows & Mac OS were back then as part of the wider scenario - non-preemptive processscheduling, weird callback based event reading, general nonsense. Checking ancient emails to find dates etc reminded me of a huge list of other stuff we were working on back then that *still* needs finishing! And the early pinned objects stuff... and private methods! And...!) Oh-ho - and as an amusing aside, an email from Eliot of that sort of date was pointing out how amazing it was to be able to get a 30Gb disk for a mere $190; you can get 10TB now.
>
> I definitely agree that for an awful lot of cases using a good ffi is better nowadays. It does have some costs though; but in some case I think those are costs we should be pleased to pay. For example, a number of current plugins would be better replaced with ffi classes that actually work with the OS facilities rather than trying to work around them. The FilePlugin comes readily to mind. How much nicer it would be to handle files with direct ffi calls to OS routines and have the complicated stuff in Smalltalk bridge classes. Hmm, now where did we have this in the past; oh yes, in the BrouHaHa images 30-some years ago.
>
> Moving to ffi-only would allow removing a fair bit of cruft from the VM, though there's our old friend 'backward compatibility' to argue with. It would mean some rewriting (mostly simplification I think?) of existing plugins code. And what to do about stuff that we pretty much always compile as internal plugins? Rewrite as external libraries and pay the (hopefully small) performance cost, or rewrite as appropriate to drop any pretence of being able to be external? SoundPlugin should absolutely be replaced by a bridge class and ffi calls to the OS. LargeInterger? Probably much better to pull properly into the vm core. MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.

+1

> Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.

+1000

> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> Strange OpCodes: DCVP: Destroy another Computer Via Phone-link
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Robert Withers-2
In reply to this post by timrowledge
 
On Sun, May 24, 2020 at 23:14, tim Rowledge <[hidden email]> wrote:


> On 2020-05-24, at 3:57 PM, Eliot Miranda <[hidden email]> wrote:
>
> Plugins emerged from the lack of an FFI, later a deficient FFI (slow, non-reentrant, no callbacks), and the need for security, at a time when graphics co-processing was very rare and not at all commodity technology. The world looks different now. We should evolve to match the new environment. I agree that security is very important. But I do not believe that security implies no FFI. I think we can have both security and the flexibility, development speed and architectural efficiency of an FFI-based implementation.


The VM plugins were an attempt to get away from a few problems that plagued VM development at the end of the last millennium (and I don't mean March this year) with things like a limited number of prim table entires, the lack of flexibility a static table engendered, the time it took to compile an entire VM for a tiny change/test cycle etc (IIRC of the order of half an hour back then). The original external plugins stuff meant you could get to outside code via a name system that didn't screw the prim table (and I'd still love to find away to get rid of it entirely, but dream on) and that let you build/load/try/unload/rebuild quickly. We were also starting the beginnings of the ffi stuff around then. Andy Greenberg came up with the SmartSyntaxPlugin stuff, for example, at the end of '99. Apparently I sent Dan/Andreas a final pluginized linux vm late April 2000.

It's worth remembering just how awful Windows & Mac OS were back then as part of the wider scenario - non-preemptive processscheduling, weird callback based event reading, general nonsense. Checking ancient emails to find dates etc reminded me of a huge list of other stuff we were working on back then that *still* needs finishing! And the early pinned objects stuff... and private methods! And...!) Oh-ho - and as an amusing aside, an email from Eliot of that sort of date was pointing out how amazing it was to be able to get a 30Gb disk for a mere $190; you can get 10TB now.

I definitely agree that for an awful lot of cases using a good ffi is better nowadays. It does have some costs though; but in some case I think those are costs we should be pleased to pay. For example, a number of current plugins would be better replaced with ffi classes that actually work with the OS facilities rather than trying to work around them. The FilePlugin comes readily to mind. How much nicer it would be to handle files with direct ffi calls to OS routines and have the complicated stuff in Smalltalk bridge classes. Hmm, now where did we have this in the past; oh yes, in the BrouHaHa images 30-some years ago.

Moving to ffi-only would allow removing a fair bit of cruft from the VM, though there's our old friend 'backward compatibility' to argue with. It would mean some rewriting (mostly simplification I think?) of existing plugins code. And what to do about stuff that we pretty much always compile as internal plugins? Rewrite as external libraries and pay the (hopefully small) performance cost, or rewrite as appropriate to drop any pretence of being able to be external? SoundPlugin should absolutely be replaced by a bridge class and ffi calls to the OS. LargeInterger? Probably much better to pull properly into the vm core. MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.

Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.
How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms. 

Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them? 

I feel like a special SharedObjectLibPlugin super class could generate the lib and the FFI calls to use it. Then the plugin slang could still be managed in-image, yet use FFI. Is it so? 

Another use case I can think of for use of FFI, is non-blocking socket code, NIO Sockets.

Kindly,
Robert


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: DCVP: Destroy another Computer Via Phone-link




Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Tobias Pape
In reply to this post by Eliot Miranda-2
 
Hi
> On 25.05.2020, at 05:35, Eliot Miranda <[hidden email]> wrote:
>>

>> Moving to ffi-only would allow removing a fair bit of cruft from the VM, though there's our old friend 'backward compatibility' to argue with. It would mean some rewriting (mostly simplification I think?) of existing plugins code. And what to do about stuff that we pretty much always compile as internal plugins? Rewrite as external libraries and pay the (hopefully small) performance cost, or rewrite as appropriate to drop any pretence of being able to be external? SoundPlugin should absolutely be replaced by a bridge class and ffi calls to the OS. LargeInterger? Probably much better to pull properly into the vm core. MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.
>
> +1
>
>> Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.
>
> +1000


One thing tho.

This all boils down to "who moves faster".

Here's a story.

When I tried to get the Self VM running again some years ago, one of the agonizing parts was its
extensive use of FFI. Why so? Well, the FFI was coded against a mid-1990 X11. Which, in that form
did not exist on either Mac nor Linux. Some parts worked, but a lot of things needed attention
in _both_ the VM _and_ the Image. In some respect, the "libraries interfaced via FFI" had moved
faster than the VM.

Had Self had a Display Plugin just like Squeak, changes would have been much more simple, since the
assumptions in the Self world had nod changed at all.

So, if things move slowly in VM/Image land and more quickly in library land, the answer is Plugins.
If things move quickly in VM/Image land and are stable in library land, the answer is FFI.

I see this in SqueakSSL very prominently. The stable interface from the Image land helps a lot.
It would be a nightmare to have this via FFI. The effect would be that we have to
settle on one SSL library on all platforms, and keep that updated and ship it with the VM.
That's by the way what the Pharo VM does and I think it is the wrong way for the problem.

Best regards
        -Tobias
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Jakob Reschke
 


Tobias Pape <[hidden email]> schrieb am Mo., 25. Mai 2020, 08:26:
 

This all boils down to "who moves faster".

Here's a story.

When I tried to get the Self VM running again some years ago, one of the agonizing parts was its
extensive use of FFI. Why so? Well, the FFI was coded against a mid-1990 X11. Which, in that form
did not exist on either Mac nor Linux. Some parts worked, but a lot of things needed attention
in _both_ the VM _and_ the Image. In some respect, the "libraries interfaced via FFI" had moved
faster than the VM.

Had Self had a Display Plugin just like Squeak, changes would have been much more simple, since the
assumptions in the Self world had nod changed at all.

So, if things move slowly in VM/Image land and more quickly in library land, the answer is Plugins.
If things move quickly in VM/Image land and are stable in library land, the answer is FFI.

I see this in SqueakSSL very prominently. The stable interface from the Image land helps a lot.
It would be a nightmare to have this via FFI. The effect would be that we have to
settle on one SSL library on all platforms, and keep that updated and ship it with the VM.
That's by the way what the Pharo VM does and I think it is the wrong way for the problem.

How does using plugins solve this problem?

If a library breaks compatibility, you would have to update the plugin and maybe think about how to stay compatible with older Squeaks. If it were FFI with a proper facade that finds out how to link to the external library, you would need to update the facade and worry there about staying compatible, but you could do it in Smalltalk. It requires discipline and guidance to only use the facade, of course.

Do you mean you can hot-fix a plugin underneath the image, so you can run ancient unmodified images on a newer platform/library base? Then maybe we do not really want plugins, but proper modules (Smalltalk shared libraries/image fragments/image components/parcels/...).
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Tobias Pape
 

> On 25.05.2020, at 13:34, Jakob Reschke <[hidden email]> wrote:
>
>
>
> Tobias Pape <[hidden email]> schrieb am Mo., 25. Mai 2020, 08:26:
>  
>
> This all boils down to "who moves faster".
>
> Here's a story.
>
> When I tried to get the Self VM running again some years ago, one of the agonizing parts was its
> extensive use of FFI. Why so? Well, the FFI was coded against a mid-1990 X11. Which, in that form
> did not exist on either Mac nor Linux. Some parts worked, but a lot of things needed attention
> in _both_ the VM _and_ the Image. In some respect, the "libraries interfaced via FFI" had moved
> faster than the VM.
>
> Had Self had a Display Plugin just like Squeak, changes would have been much more simple, since the
> assumptions in the Self world had nod changed at all.
>
> So, if things move slowly in VM/Image land and more quickly in library land, the answer is Plugins.
> If things move quickly in VM/Image land and are stable in library land, the answer is FFI.
>
> I see this in SqueakSSL very prominently. The stable interface from the Image land helps a lot.
> It would be a nightmare to have this via FFI. The effect would be that we have to
> settle on one SSL library on all platforms, and keep that updated and ship it with the VM.
> That's by the way what the Pharo VM does and I think it is the wrong way for the problem.
>
> How does using plugins solve this problem?
>
> If a library breaks compatibility, you would have to update the plugin and maybe think about how to stay compatible with older Squeaks. If it were FFI with a proper facade that finds out how to link to the external library, you would need to update the facade and worry there about staying compatible, but you could do it in Smalltalk. It requires discipline and guidance to only use the facade, of course.
>
> Do you mean you can hot-fix a plugin underneath the image, so you can run ancient unmodified images on a newer platform/library base? Then maybe we do not really want plugins, but proper modules (Smalltalk shared libraries/image fragments/image components/parcels/...).

Rather the latter. I wouldn't call it hot-fix but rather evolve stuff as they change.
That's how we do with Display and SqueakSSL.
You don't need to touch the image, but you could.
I don't understand how "proper modules" in whatever sense would fix that.

:)

Best regards
        -Tobias



Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

timrowledge
In reply to this post by Robert Withers-2
 
OK, I'll try to answer some of the thoughts on this -

> On 2020-05-24, at 11:07 PM, Robert <[hidden email]> wrote:
>
> How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
>
> Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?

Right now (assuming I'm remembering it all correctly and with the proviso that I haven't written a new plugin in several years) an external plugin is compiled from the assorted code required (some plugins are solely slang generated code, some are solely hand-written C, some are slang + C + calls to other libraries and so on) and a platform specific form of shared library is made. Are they in a form that is callable via 'normal' ffi calls? Dunno, but clearly they *could* be made that way with not very much change.



> On 2020-05-24, at 11:26 PM, Tobias Pape <[hidden email]> wrote:
>  The effect would be that we have to
> settle on one SSL library on all platforms, and keep that updated and ship it with the VM.
> That's by the way what the Pharo VM does and I think it is the wrong way for the problem.

Actually no, that isn't how you *have* to do it. You can have interfaces from the image to many different libraries; hence the mentions of 'bridge classes' in my message. It's not much different to how we currently choose the platform appropriate class of file directory on startup.

When the interface to a subsystem changes you have to make corresponding changes *somewhere*. With current plugins we have to rewrite the relevant parts, often the hand-written C-shim stuff. In one sense that is a virtue since we can do that with 'normal' tools like edlin, and enjoy all the fun of that world of high quality IDEs (sarcasm? Moi?). If we used bridge classes to ffi calls direct to the external system then we would have to do some work in the image; now of course in general we'd all prefer that but I can imagine cases where the changed interface prevents us running the image to write the code to handle the changed interface. This is where having a way to make the system load code at startup helps.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Eagles may soar, but weasels aren't sucked into jet engines.


Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Levente Uzonyi
In reply to this post by timrowledge
 
On Sun, 24 May 2020, tim Rowledge wrote:

> MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.

Below is a benchmark measuring the primitive to calculate string and byte
array hashes. The in-image variant is fully JIT-friendly: AFAIK all its
message sends have jitted implementations (#bitAnd:, #size, #to:do:, #+,
#basicAt:, #hashMultiply).

| b |
b := ByteArray new: 10000.
{[ b hashWithInitialHash: 0 ] benchFor: 1 seconds.
[ String stringHash: b initialHash: 0 ] benchFor: 1 seconds }.
#('88,500 per second. 11.3 microseconds per run. 0 % GC time.' '11,500 per
second. 86.9 microseconds per run. 0 % GC time.')"

In this case, the primitive version is 7.7x faster on my machine.

The actual performance difference depends on the size of the collection.
For sizes smaller than 4, the primitive version is slower. That could
probably be improved by making it a numberered primitive and providing a
jitted version of the primitive; something that was done with
#compare:with:collated: (see primitive 158).

At this point we clearly can't drop those primitives. Some of them is
probably worth to add to the VM directly, others may not be necessary at
all in the future.


Levente

>
>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> Strange OpCodes: DCVP: Destroy another Computer Via Phone-link
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Tobias Pape
In reply to this post by timrowledge
 

> On 25.05.2020, at 18:46, tim Rowledge <[hidden email]> wrote:
>
>
> OK, I'll try to answer some of the thoughts on this -
>
>> On 2020-05-24, at 11:07 PM, Robert <[hidden email]> wrote:
>>
>> How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
>>
>> Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?
>
> Right now (assuming I'm remembering it all correctly and with the proviso that I haven't written a new plugin in several years) an external plugin is compiled from the assorted code required (some plugins are solely slang generated code, some are solely hand-written C, some are slang + C + calls to other libraries and so on) and a platform specific form of shared library is made. Are they in a form that is callable via 'normal' ffi calls? Dunno, but clearly they *could* be made that way with not very much change.
>
>
>
>> On 2020-05-24, at 11:26 PM, Tobias Pape <[hidden email]> wrote:
>> The effect would be that we have to
>> settle on one SSL library on all platforms, and keep that updated and ship it with the VM.
>> That's by the way what the Pharo VM does and I think it is the wrong way for the problem.
>
> Actually no, that isn't how you *have* to do it. You can have interfaces from the image to many different libraries; hence the mentions of 'bridge classes' in my message. It's not much different to how we currently choose the platform appropriate class of file directory on startup.

Per se, true. But try discriminating openssl 0.9.6, 1.0.1, and 1.1.1. It's a nightmare. The overlay somewhat works, but even that is not so easy.

It's just that I've seen the bridge-class stuff made things hard for me in Self.
But that might just be the long time gap where X11 evolved and the Self part did not. Problem was I had no tools because I broke my tools with my tools (so to speak, because no UI, no way of fixing the UI properly).


>
> When the interface to a subsystem changes you have to make corresponding changes *somewhere*. With current plugins we have to rewrite the relevant parts, often the hand-written C-shim stuff. In one sense that is a virtue since we can do that with 'normal' tools like edlin, and enjoy all the fun of that world of high quality IDEs (sarcasm? Moi?).

Not that I like digging in C more than using Squeak proper...

> If we used bridge classes to ffi calls direct to the external system then we would have to do some work in the image; now of course in general we'd all prefer that but I can imagine cases where the changed interface prevents us running the image to write the code to handle the changed interface. This is where having a way to make the system load code at startup helps.


Again, FFI is fine. My point is rather that the pace of change can indicate whether a plugin is better or FFI.

Also, the other VM-implementations can provide Plugins much more easily than FFI.
I would argue this is true for SqueakJS, RSqueak, JSqueak/Potato, and to an extend, TruffleSqueak.
These systems do not necessarily provide access to C libraries and reduce the need for in-image changes for each VM.


TL;DR: FFI good, Plugins also good.

just my 2ct.

Best regards
        -Tobias




Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

K K Subbu
In reply to this post by Jakob Reschke
 
On 25/05/20 5:04 pm, Jakob Reschke wrote:
>
> If a library breaks compatibility, you would have to update the plugin
> and maybe think about how to stay compatible with older Squeaks. If it
> were FFI with a proper facade that finds out how to link to the external
> library, you would need to update the facade and worry there about
> staying compatible, but you could do it in Smalltalk. It requires
> discipline and guidance to only use the facade, of course.
Libraries like OpenGL are frustrating because of poorly abstracted, low
level functions. Trying to stable facade around its warts will not solve
the core problem of having to deal with static code and compile cycles.

The frustration is having to regress from live programming into
code-compile-link-test cycle in 2020! Given a choice between live
programming and backward compatibility, I would choose live programming
because I can close any gaps through live coding.

For instance, a few years back I developed live previews for LaTeX
without leaving Squeak (Etoys) by using OSProcess to build a LaTeX
pipeline that converted code into image as the code was being typed. It
is an inefficient design but it took just half a day and has served well
over the years.

But this was at process level. If only I could do the same with DLLs. If
only Squeak had a tool (ELF reader?) to reify a DLL, extract its
instruction set, interface definitions, state variables and calling
conventions. Then I can livecode a class and dynamically compile it (now
that we have JIT) to call functions DLL without leaving the image. The
code may not be 'portable' in traditional sense. But it will be a lot
more manageable than static designs.

Regards .. Subbu
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Robert Withers-2
In reply to this post by timrowledge
 
OMG, I have had a huge brain fart. I totally forgot that plugins are built *as* shared object Libs. 

So our choice to invoke said SOLib through named primitives are just as workable as somehow invoking FFI on them. What would that look like 👀? Is there an example for calling FFI, for one of the Crypto plugins? Let’s evaluate.

Kindly,
Robert


On Mon, May 25, 2020 at 12:46, tim Rowledge <[hidden email]> wrote:

OK, I'll try to answer some of the thoughts on this -

> On 2020-05-24, at 11:07 PM, Robert <[hidden email]> wrote:
>
> How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
>
> Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?

Right now (assuming I'm remembering it all correctly and with the proviso that I haven't written a new plugin in several years) an external plugin is compiled from the assorted code required (some plugins are solely slang generated code, some are solely hand-written C, some are slang + C + calls to other libraries and so on) and a platform specific form of shared library is made. Are they in a form that is callable via 'normal' ffi calls? Dunno, but clearly they *could* be made that way with not very much change.



> On 2020-05-24, at 11:26 PM, Tobias Pape <[hidden email]> wrote:
> The effect would be that we have to
> settle on one SSL library on all platforms, and keep that updated and ship it with the VM.
> That's by the way what the Pharo VM does and I think it is the wrong way for the problem.

Actually no, that isn't how you *have* to do it. You can have interfaces from the image to many different libraries; hence the mentions of 'bridge classes' in my message. It's not much different to how we currently choose the platform appropriate class of file directory on startup.

When the interface to a subsystem changes you have to make corresponding changes *somewhere*. With current plugins we have to rewrite the relevant parts, often the hand-written C-shim stuff. In one sense that is a virtue since we can do that with 'normal' tools like edlin, and enjoy all the fun of that world of high quality IDEs (sarcasm? Moi?). If we used bridge classes to ffi calls direct to the external system then we would have to do some work in the image; now of course in general we'd all prefer that but I can imagine cases where the changed interface prevents us running the image to write the code to handle the changed interface. This is where having a way to make the system load code at startup helps.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Eagles may soar, but weasels aren't sucked into jet engines.




Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Robert Withers-2
In reply to this post by timrowledge
 
On Mon, May 25, 2020 at 12:46, tim Rowledge <[hidden email]> wrote:

OK, I'll try to answer some of the thoughts on this -

> On 2020-05-24, at 11:07 PM, Robert <[hidden email]> wrote:
>
> How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
>
> Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?

Right now (assuming I'm remembering it all correctly and with the proviso that I haven't written a new plugin in several years) an external plugin is compiled from the assorted code required (some plugins are solely slang generated code, some are solely hand-written C, some are slang + C + calls to other libraries and so on) and a platform specific form of shared library is made. Are they in a form that is callable via 'normal' ffi calls? Dunno, but clearly they *could* be made that way with not very much change.
Tim, does this mean that the plugin generation and linking as appropriate continues a pace, while the in-image named prims table switches all entries to FFI calls? CryptographyPlugins have the advantage of slang generated only code.

Would it be possible to take advantage of a GPU??

K, r

Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

timrowledge
In reply to this post by K K Subbu
 


> On 2020-05-25, at 10:25 AM, K K Subbu <[hidden email]> wrote:
>
> On 25/05/20 5:04 pm, Jakob Reschke wrote:
>> If a library breaks compatibility, you would have to update the plugin and maybe think about how to stay compatible with older Squeaks. If it were FFI with a proper facade that finds out how to link to the external library, you would need to update the facade and worry there about staying compatible, but you could do it in Smalltalk. It requires discipline and guidance to only use the facade, of course.
> Libraries like OpenGL are frustrating because of poorly abstracted, low level functions. Trying to stable facade around its warts will not solve the core problem of having to deal with static code and compile cycles.
>
> The frustration is having to regress from live programming into code-compile-link-test cycle in 2020! Given a choice between live programming and backward compatibility, I would choose live programming because I can close any gaps through live coding.


Nicely put.
>

> For instance, a few years back I developed live previews for LaTeX without leaving Squeak (Etoys) by using OSProcess to build a LaTeX pipeline that converted code into image as the code was being typed. It is an inefficient design but it took just half a day and has served well over the years.
>
> But this was at process level. If only I could do the same with DLLs. If only Squeak had a tool (ELF reader?) to reify a DLL, extract its instruction set, interface definitions, state variables and calling conventions. Then I can livecode a class and dynamically compile it (now that we have JIT) to call functions DLL without leaving the image. The code may not be 'portable' in traditional sense. But it will be a lot more manageable than static designs.


VW has a parser of C header files that is I suppose the right way to do this sort of thing - you parse the headers (warning - I seem to recall parsing 'windows.h' taking several days in the original VW ddl product) and build proxies of all the types/structs/etc.  Is that a practical way to do it? Hell of a lot of work to do, certainly.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: QVC: Question Valid Command


Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

K K Subbu
 
On 26/05/20 2:20 am, tim Rowledge wrote:

>> But this was at process level. If only I could do the same with
>> DLLs. If only Squeak had a tool (ELF reader?) to reify a DLL,
>> extract its instruction set, interface definitions, state variables
>> and calling conventions. Then I can livecode a class and
>> dynamically compile it (now that we have JIT) to call functions DLL
>> without leaving the image. The code may not be 'portable' in
>> traditional sense. But it will be a lot more manageable than static
>> designs.
>
> VW has a parser of C header files that is I suppose the right way to
> do this sort of thing - you parse the headers (warning - I seem to
> recall parsing 'windows.h' taking several days in the original VW ddl
> product) and build proxies of all the types/structs/etc.  Is that a
> practical way to do it? Hell of a lot of work to do, certainly.

We don't know what source code was used to compile the DLL. Having an
inspector for DLLs would be a good start. If we pull out the dynamic
link maps into the image, then we can simulate in the image, what VM
currently does through dlopen() calls. Like what we do in doPrimitive.

Regards .. Subbu
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

timrowledge
 


> On 2020-05-26, at 4:20 AM, K K Subbu <[hidden email]> wrote:
>
> On 26/05/20 2:20 am, tim Rowledge wrote:
>>> But this was at process level. If only I could do the same with
>>> DLLs. If only Squeak had a tool (ELF reader?) to reify a DLL,
>>> extract its instruction set, interface definitions, state variables
>>> and calling conventions. Then I can livecode a class and
>>> dynamically compile it (now that we have JIT) to call functions DLL
>>> without leaving the image. The code may not be 'portable' in
>>> traditional sense. But it will be a lot more manageable than static
>>> designs.
>> VW has a parser of C header files that is I suppose the right way to
>> do this sort of thing - you parse the headers (warning - I seem to
>> recall parsing 'windows.h' taking several days in the original VW ddl
>> product) and build proxies of all the types/structs/etc.  Is that a
>> practical way to do it? Hell of a lot of work to do, certainly.
>
> We don't know what source code was used to compile the DLL.

Certainly true in a lot of cases; the idea used to be that you could get a header file that described the interface and so compile appropriate code. I'm sure there are plenty of cases where that is not what actually happens.

> Having an inspector for DLLs would be a good start. If we pull out the dynamic link maps into the image, then we can simulate in the image, what VM currently does through dlopen() calls. Like what we do in doPrimitive.

That would be nice. And of course Apple bundles, unix .so's and RISC OS shared libraries. A small matter of programming :-)


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful Latin Phrases:- Magister Mundi sum! = I am the Master of the Universe!


Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

tblanchard
In reply to this post by timrowledge
 
I was hoping getting a solid libCLANG interface would make this easy.

But I haven't been able to get past a problem with some functions returning structs by value and the FFI failing to handle this.

Might be fixed now.  Worth a look.

On May 25, 2020, at 1:50 PM, tim Rowledge <[hidden email]> wrote:

VW has a parser of C header files that is I suppose the right way to do this sort of thing - you parse the headers (warning - I seem to recall parsing 'windows.h' taking several days in the original VW ddl product) and build proxies of all the types/structs/etc.  Is that a practical way to do it? Hell of a lot of work to do, certainly.

Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Eliot Miranda-2
In reply to this post by timrowledge
 
Hi Tim,

> On May 25, 2020, at 1:50 PM, tim Rowledge <[hidden email]> wrote:
>
>
>> On 2020-05-25, at 10:25 AM, K K Subbu <[hidden email]> wrote:
>>
>>> On 25/05/20 5:04 pm, Jakob Reschke wrote:
>>> If a library breaks compatibility, you would have to update the plugin and maybe think about how to stay compatible with older Squeaks. If it were FFI with a proper facade that finds out how to link to the external library, you would need to update the facade and worry there about staying compatible, but you could do it in Smalltalk. It requires discipline and guidance to only use the facade, of course.
>> Libraries like OpenGL are frustrating because of poorly abstracted, low level functions. Trying to stable facade around its warts will not solve the core problem of having to deal with static code and compile cycles.
>>
>> The frustration is having to regress from live programming into code-compile-link-test cycle in 2020! Given a choice between live programming and backward compatibility, I would choose live programming because I can close any gaps through live coding.
>
>
> Nicely put.
>>
>
>> For instance, a few years back I developed live previews for LaTeX without leaving Squeak (Etoys) by using OSProcess to build a LaTeX pipeline that converted code into image as the code was being typed. It is an inefficient design but it took just half a day and has served well over the years.
>>
>> But this was at process level. If only I could do the same with DLLs. If only Squeak had a tool (ELF reader?) to reify a DLL, extract its instruction set, interface definitions, state variables and calling conventions. Then I can livecode a class and dynamically compile it (now that we have JIT) to call functions DLL without leaving the image. The code may not be 'portable' in traditional sense. But it will be a lot more manageable than static designs.
>
>
> VW has a parser of C header files that is I suppose the right way to do this sort of thing - you parse the headers (warning - I seem to recall parsing 'windows.h' taking several days in the original VW ddl product) and build proxies of all the types/structs/etc.  Is that a practical way to do it? Hell of a lot of work to do, certainly.

Emphatically not.  DLLCC’s C parser is a nightmare to maintain, and with a little cunning, essentially unnecessary. See all the conversation around generating a C program which when run outputs information that the image can easily parse, and as a result (re)populate a shared pool with relevant platform specifics.  We have this in FFISharedPool (by Shaping IIRC, after an idea/suggestion of mine, implemented as a prototype before I left Cincom, and as a part of UFFI by Mariano Martinez Peck, following discussion a few years back).

Generating a program has several advantages

- no need to implement any part of the C toolchain (doesn’t reinvent the wheel)

- uses the platform’s information more-or-less directly, rather than trying to mimic it

- the binaries can be shared between developers (a library of them?) and deployed with cross-platform apps

- only the FFI api author needs a C development toolset


So what the FFI author does is define a shared pool which has platform-specific metadata, specifically the necessary include files and version selection defines for each relevant platform.  The FFI shared pool machinery takes care of generating the program when necessary (only when authoring an FFI), and running it when necessary (only once for a single-platform app, only when launching on a different platform to the one snapshotted on)


>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> Strange OpCodes: QVC: Question Valid Command
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

Eliot Miranda-2
In reply to this post by tblanchard
 


On May 26, 2020, at 6:51 PM, Todd Blanchard <[hidden email]> wrote:

I was hoping getting a solid libCLANG interface would make this easy.

But I haven't been able to get past a problem with some functions returning structs by value and the FFI failing to handle this.

Bugs were fixed on x86_64 recently and ARMv8 less recently.

There really isn’t any advantage in using a different FFI implementation.  We have the capability of producing a correct FFI.  What we need is good bug reports and help.  What we *dont* need is a different FFI, incoherent or split community effort, etc.


Might be fixed now.  Worth a look.

On May 25, 2020, at 1:50 PM, tim Rowledge <[hidden email]> wrote:

VW has a parser of C header files that is I suppose the right way to do this sort of thing - you parse the headers (warning - I seem to recall parsing 'windows.h' taking several days in the original VW ddl product) and build proxies of all the types/structs/etc.  Is that a practical way to do it? Hell of a lot of work to do, certainly.

Reply | Threaded
Open this post in threaded view
|

Re: [Vm-beginners] Stability of the external plugin interface

timrowledge
In reply to this post by Eliot Miranda-2
 


> On 2020-05-26, at 8:10 PM, Eliot Miranda <[hidden email]> wrote:
> DLLCC’s C parser is a nightmare to maintain,

I certainly remember *that*. Poor Marcel Schelvis practically went mad working on it.

>  So what the FFI author does is define a shared pool which has platform-specific metadata, specifically the necessary include files and version selection defines for each relevant platform.  The FFI shared pool machinery takes care of generating the program when necessary (only when authoring an FFI), and running it when necessary (only once for a single-platform app, only when launching on a different platform to the one snapshotted on)

Sounds good to me. Beats parsing secondhand and possibly inaccurate information intended for an inferior language


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Fractured Idiom:- MERCI RIEN - Thanks for nothin'.


12