building plugins outside the tree

Hi,

Has anyone tried to build a plugin outside the vm tree?

In a VM build, how do I generate the necessary header files and libraries to be able to build plugins later outside the source tree?

TIA .. Subbu

Hi,

Has anyone tried to build a plugin outside the vm tree?

In a vm build, how do I generate the necessary header files and libraries to be able to build plugins later outside the source tree?

TIA .. Subbu

On Thursday 15 March 2018 09:42 AM, Eliot Miranda wrote:

    In a vm build, how do I generate the necessary header files and
    libraries to be able to build plugins later outside the source tree?


Not sure I understand the question.  All the include files are in the platforms (platforms/Cross/vm & platforms/Cross/plugins/*/*.h) and relevant source tree.  So if you look at the include paths for a build that'll give you the relevant includes.

The src tree combines headers needed to build vm from those needed for plugins. Most of the build time is spent on plugins. If the build process is split into four+ stages:

first-stage(buildconfig, $VMDK) -> $VMDK/include/, $VMDK/include/$PLAT, $VMDK/lib/, $VMDK/lib/$PLAT, $VMDK/lib64/, ....

second-stage($VMDK) - update the above with vm-specific files (say sqst.h, sqcog.h sqspur.h, libstvm.a, libcog.a libspur.a etc). $VMDK contents are packaged for developers (say libostvm-dev).

third-stage($VMDK) - for each plugin do build plugin using headers and libraries in $VMDK/. install header and libraries under $VMDK/. This would consume most of the build time, so it could be done in parallel or even outside the source directory at a later date. The build would depend only on files in $VMDK. Each plugin could choose its code structure, platforms and makefiles. Internal plugins would be generated in both obj and DLL while external plugins would be DLLs only.

fourth-stage($VMDK with internal plugins) -> squeakvm binary.
  Install libostvm-dev and the dependent external plugin packages. Generate multiple headless or GUI launchers and platform-specific packages and main package (say ostvm).

f you're building a plugin you *have* to build different versions for V3 (the original pre-Spur VM), 32-bit Spur and 64-bit Spur.  So for example you'll see that src/vm/interp.h spursrc/vm/interp.h and spur64src/vm/interp.h all differ slightly.  So, other than writing a plugin-specific source file, e.g. platforms/Cross/plugins/FilePlugin/FilePluginh) and platform-and/or-library-specific include files, everything else should be taken from the opensmalltalk-vm source tree.

I see plugins falling into two types - algorithms (e.g. BitBlt) and device reifiers (e.g. SoundPlugin). Source level coupling crimps device plugin developers. Device plugins tend to be small and the overhead of doing a full build is very high. Also, in embedded or IoT deployments, the life cycle for plugins can be very different from that of the main VM. So it is imperative that they be decoupled from VM build process through an explicit SDK.

Regards .. Subbu

I'd even argue that we would be better off *not* having copies of the generated code publicly available and looking like you can edit them. Generating the files from Slang is not really all that time consuming (even on a Pi) and you don't have to do all of it every time, and we could probably improve detection of when you do.

> On 19-03-2018, at 4:46 AM, K K Subbu <[hidden email]> wrote:
> [snip]
> Generating C code is the least of the problem. The complexity is in compiling it to a shared library. Hardware plugins face a much bigger issue than algorithmic plugins because they have to integrate slang generated code with hand-coded and local headers/libs.

You'll have the complexity of the build system whatever you do within a C/make world. You need the relevant headers and hand-written code and make related files. Nothing is likely to change that much.
You edit the hand-written code, generate any Slang code from your VMMaker image, go to the right directory and type make.

What could we do to make that easier?

I guess *if* everything worked properly on all platforms (which it never has and likely never will) we could imagine each plugin directory containing links to every relevant file so that the insane complexity of paths that look like ../../../../../platforms/unix/sysV4/plugins/WibbleWriter could be avoided. That was actually a part of my original plan when I started the VMMaker stuff back in ... 2000. Eek. But linking simply didn't work across platforms back then and I don't honestly imagine it would work neatly now. Andreas insisted on a strange arrangement of files for Window because he didn't want to sort out nmake (which was the windows make equivalent back then and since I had had some very unpleasant experience making it work well enough to build VisualWorks a few years before that I couldn't really argue). RISC OS could do links but only really of directories. Mac couldn't do links at all, just sometihng I can't even remember the name of now, that had to be manually interpreted. Perhaps some git magic might make it work so that all the file appear in the plugin build directcory?

We could conceivably build a tool in Squeak to drive it all a bit more pleasantly, using stuff from OSProcess and CommandTool etc. Platform spread, as so often, is a major cause of work here.

We could certainly improve documentation so it is easier to find out what is needed.

I'd even argue that we would be better off *not* having copies of the generated code publicly available and looking like you can edit them. Generating the files from Slang is not really all that time consuming (even on a Pi) and you don't have to do all of it every time, and we could probably improve detection of when you do.

On Tuesday 20 March 2018 11:21 PM, Eliot Miranda wrote:

Over my dead body.  Look, the generated sources all start with comments that say

/* Automatically generated by

they could say

/* Automatically generated by
.....
   DO NOT EDIT
  */

Is it necessary to generate human-readable C code?

If the code can be filled out on a line completely, it should dissuade people from editing it while keeping the compiler happy. E.g.
------interp.c-------------------------------------------
typedef struct _StackPage { char *stackLimit; char
*headSP; char *headFP; char *baseFP; char *baseAddress;
char *realStackLimit; char *lastAddress; int trace;
struct _StackPage *nextPage; struct _StackPage *prevPage;
} StackPage;
---------------------------------------------------------

The code can be restored if necessary with indent(1).

Regards .. Subbu

> On 20-03-2018, at 10:51 AM, Eliot Miranda <[hidden email]> wrote:
>
> Over my dead body.  Look, the generated sources all start with comments that say
>
> /* Automatically generated by
>
> they could say
>
> /* Automatically generated by
> .....
>   DO NOT EDIT
>  */


OR we could run the generated code through an obfuscator... wait, what am I saying, this is C code already. An important problem here is that I can pretty much guarantee nobody ever 'see's that comment, nor would take any notice of it if they did. Because code mokeys have all taken off their hoodies, handed over their beer/red bull/mountain dew/whatever and told their buddies "ok, watch *this* dude!"

>
> But going back to generating the VM sources every time one wants to build a VM, having, as Ben points out the possibility of changes in Slang, variability due to sets changing generated sources, date stamps changing, etc, etc.  NO!!!!!!!  I can't believe you're serious.

Who needs to be serious? Fair point about the variability of details important to other tools though. I can't imagine we couldn't work around all that but as you say, who has the time?