Modular Plugins

Hi,

Currently, the *Plugin.c file is generated from Slang code which is pre-loaded
into the vmm image and not distributed with the plugins/ directory. The first
line in such generated files is a cryptic:
 /* Automatically generated from Squeak on #(7 March 2007 2:25:53 pm) */

This line contains no clue about the version of the slang code or the vmm
which translated it to C.

I have two proposals towards modular plugins:

1. Keep all sources, including the Slang code, in the *Plugins/ directories
(Cross/, ${PLATFORM})? When a plugin is selected as internal/external, its
changeset can be loaded into VMM ready for translation/testing. The changeset
blob can be subject to version control just like other hand-coded files.

2. Generate support header files like sqVirtualMachine.h along with *Plugin.c
file, so that the code can be built independent of the VM directory layout.
This will be useful while porting sq*.c files to a different platform - say like
hacking device names or timings.

BTW, I am sending this to squeak-dev and not vm-dev to elicit broader
feedback.

Subbu

On Mon, Dec 21, 2009 at 09:04:21AM +0530, K. K. Subramaniam wrote:
> Hi,
>
> Currently, the *Plugin.c file is generated from Slang code which is pre-loaded
> into the vmm image and not distributed with the plugins/ directory. The first
> line in such generated files is a cryptic:
>  /* Automatically generated from Squeak on #(7 March 2007 2:25:53 pm) */
>
> This line contains no clue about the version of the slang code or the vmm
> which translated it to C.

Are you using an up to date VMMaker from SqueakSource? If so, you should
be seeing comments that look like this:

/* Automatically generated from Squeak on 20 December 2009 11:41:57 pm
   by VMMaker 3.11.10
 */

The "3.11.10" version stamp comes from VMMaker class>>versionString.
The #versionString method is updated by hand, and therefore is subject
to human error, but it works fairly well in practice.

On the unix VMs (not sure of the others yet), the VM identifies itself
as e.g. "3.11.10-2146", which refers to a VM built with generated code
from VMMaker version 3.11.10 and platform sources from Subversion
revision level 2146. This provides at least a reasonable level of
traceability to the source code, some of which is managed under
Monticello, and some of which is managed under Subversion.

Having part of the source managed under Monticello and part of
it managed under Subversion does cause confusion, especially for
people who expect source code to be written in C and compiled with
"configure; make; make install". I don't know of a good solution
to this, although I have a personal bias towards keeping as much
of the system as possible in Smalltalk, hence Monticello. But
that's my just personal opinion, and I suspect that many others
are more comfortable keeping things in external files. I'll be
interested to hear other folks' POV on this.

One thing I do have a strong opinion on: The generated C source
code that is translated from Slang should *not* be treated as
source code. It is intermediate code, like the *.o from a C
compiler. I understand that Linux/GNU distribution maintainers
do need a copy of source code from which a VM can be built, and
the C version is much more convenient for this (hence the
platforms/unix/src files that are kept for this purpose).
But the real source is written in Smalltalk, and Monticello
is a much more effective version control system for this.

Dave

2009/12/21 David T. Lewis <[hidden email]>:
Sorry, you won't hear from me anything which would be against your opinion :)
Keep as much as possible in smalltalk. I helped me a lot when making a HydraVM,
and it definitely will help others who would want to port it on
different platform or even language.
+1 here.
I already said before, that keeping a generated sources in version
tracker is overkill.
There was arguments for keeping them so you can build it w/o running VM.
Or building VM from first principles (on completely virgin platform)
where you don't have anything but C compiler..
(this is a bit stretched, as to me, since you can always do
preparations on different platform, where VM already can be run to
generate those damned C sources :)

Other aspect is to compare generated sources by your with those which
in repository.. Never did that.. maybe its useful?

> Dave
>
>
>



--
Best regards,
Igor Stasenko AKA sig.

On Mon, Dec 21, 2009 at 09:04:21AM +0530, K. K. Subramaniam wrote:
>
> 2. Generate support header files like sqVirtualMachine.h along with *Plugin.c
> file, so that the code can be built independent of the VM directory layout.
> This will be useful while porting sq*.c files to a different platform - say like
> hacking device names or timings.

What to folks think about this idea? I think that the intent of this
suggestion is to make it potentially easer to generate and build an
individual plugin without the need to build the entire VM. The
down side would be that it could cause problems for the people
who write and support the platform code.

I have no preference one way or the other, but I'm curious what
others think.

Dave

David T. Lewis wrote:
I'm not sure what is meant with the above. Most plugin header files are
not generated but hand written. Those that are (interp.h etc) already
are generated side-by-side with the plugin code. So I'm not sure what is
being requested here.

(I will make one side note though that in the past on Win32 VMMaker
copied all of hand written and generated files into a single directory
precisely to simplify the build structure. This, however had a major
disadvantage for maintenance since there wasn't an easy way to ensure
that the files in the build match the files in SVN. As a consequence, I
opted for a more complex build structure to leave the files in their
original location)

Cheers,
   - Andreas

On Monday 21 December 2009 11:06:51 am David T. Lewis wrote: I picked the header from
./unix/plugins/VideoForLinuxPlugin/VideoForLinuxPlugin.c randomly from the
unix*vmm.tar.gz sources distributed from squeakvm.org/unix. Even the new style
header only includes the VM version and not the plugin source (cs) version.
Shouldn't this appear in the intermediate .c or .so file for traceability?

>One thing I do have a strong opinion on: The generated C source
>code that is translated from Slang should not be treated as
>source code. It is intermediate code, like the *.o from a C compiler.
I fully agree. My suggestion was to put cs under version control and not the
generated C code. The exact version control tool is not relevant as long as we
can trace the object (.so or .c) to the source (.cs). The cs file, like .y or
.lex needs a converter. AFAIK, the complication is that there is no command
line converter to extract a version of .cs and convert it to .c so one cannot
run it from a batch process like make(1) or cmake(1) say:
    squeak -headless vmm.image translate-plugin.st src p.cs obj p.c dir /tmp

Knowing Squeak, this should be a simple matter of programming (SMOP) but my
Squeak skills are insufficient to figure this out :-(. I suppose the first
interactive editing and regeneration of plugin .c file is tolerated because it
changes rarely compared to porting changes to sq*.c files.

Subbu

On Monday 21 December 2009 11:43:34 pm David T. Lewis wrote:
> What to folks think about this idea? I think that the intent of this
> suggestion is to make it potentially easer to generate and build an
> individual plugin without the need to build the entire VM. The
> down side would be that it could cause problems for the people
> who write and support the platform code.
Yes. that is my intent and I expect that it would simplify life for people who
know platform (say iPhone or Maemo) but not Squeak VM.

Say Joe Gizmo wants to port plug P to gizmo platform.

1. Download p-1.1.zip from squeakvm. Extract to:
     /tmp/p-1.1/
                README, Changelog ....
                p.cs
                unix/configure, sqUnixp.c  .....
                win32/p.prj, sqWin32p.c ...
2. Joe fires up VMM on his host platform and files in p.cs. He then generates:
    /tmp/p-1.1/
                p.c
                sqVirtualMachine.h
    (can this be run in batch mode?)

3. Joe port unix/* files to gizmo/, making necessary changes.
    /tmp/p-1.1/
                gizmo/configure, sqGizmop.c sqGizmop.h,...
    sqGizmop.c will refer to include directory ../ for sqVirtualMachine.h
    and link against p.o generated from ../p.c

   Changes in gizmo/ are checked into SVN and /tmp/p-1.1 is archived as p.zip

4. Joe unpacks archive on target platform in
      /tmp/p-src/
 and builds the binaries in
      /tmp/p-gizmo/
           ../p-src/gizmo/configure; make
5. The new plugin is ready to be loaded and tested in his target squeak image.
Changes in gizmo/* sources are checked back into SVN.

6. Once tested, p.zip can be uploaded to squeakvm as p-1.2.zip for general
consumption.

Subbu

On Tue, Dec 22, 2009 at 12:12:07PM +0530, K. K. Subramaniam wrote:
Ha! Interesting "random" choice. I don't know why this C source is being
kept in the platforms tree, but it does illustrate the sort of confusion
that arises from allowing generated C files to be mixed in with the platforms
source tree. I think that Video4Linux comes from Etoys, see
  http://wiki.squeak.org/squeak/3765

I'm not sure where the real master copy of the source code is, possibly
it's this change set:
  http://wiki.squeak.org/squeak/uploads/3765/VideoForSqueak-dgd.cs.gz
or (hopefully) it's being maintained in a Monticello repository somewhere.

In any case, I suspect that at some point (as needed for the OLPC project),
a C file was generated from the Smalltalk source, and this was added to
unix/plugins/VideoForLinuxPlugin. That would be the file you are looking
at. It is *not* the real source code for this plugin.

IMO this is an error and should be fixed.

> Even the new style
> header only includes the VM version and not the plugin source (cs) version.
> Shouldn't this appear in the intermediate .c or .so file for traceability?

This is a good idea. Unfortunately I don't know any way to do it in an
automated fashion. That leaves it up to the author of the plugin to make
it happen. For example, in OSProcessPlugin you will see something like
this about 30 lines into the generated source:

/* D T Lewis - UnixOSProcessPlugin.c translated from class
   UnixOSProcessPlugin of OSProcessPlugin version 4.3.3 */

There is also a primitive that answers the plugin version string at
runtime. But this is a manual process that relies on the author of
the plugin writing the necessary code and keeping the version string
up to date.

If anyone can think of a better way to do this, I'd be all for it!

Dave

On Tuesday 22 December 2009 01:21:36 am Andreas Raab wrote: Files like sq.h, sqMemoryAccess.h, sqVirtualMachine.h are all generated by
VMM. The current procedure assumes that all plugins are being generated at
once along with the vm. It does not consider the case where someone is working
on a plugin outside the vm source tree. For a plugin where only a few hundred
lines of code are involved, full vm sources are an overkill.

Subbu

K. K. Subramaniam wrote:
I'm sorry but that is incorrect. None of these files are generated; they
are hand-edited and checked into SVN, see

http://squeakvm.org/cgi-bin/viewcvs.cgi/trunk/platforms/Cross/vm/

> VMM. The current procedure assumes that all plugins are being generated at
> once along with the vm. It does not consider the case where someone is working
> on a plugin outside the vm source tree. For a plugin where only a few hundred
> lines of code are involved, full vm sources are an overkill.

Yes, and there's no need for that. I've successfully written plugins
that either had a very small or even no Slang plugin behind it.

Cheers,
   - Andreas

On Wednesday 23 December 2009 09:25:18 pm Andreas Raab wrote: You are right. I stand corrected. Thanks. A closer look at SVN shows only
unix/src/ and Mac OS/.. contain auto-generated files. Though why these
intermediate files (but not their source .cs/.st files) are in SVN is a mystery.

> > VMM. The current procedure assumes that all plugins are being generated
> > at once along with the vm. It does not consider the case where someone is
> > working on a plugin outside the vm source tree. For a plugin where only a
> > few hundred lines of code are involved, full vm sources are an overkill.
>
> Yes, and there's no need for that. I've successfully written plugins
> that either had a very small or even no Slang plugin behind it.
Are these in public domain? url?

I have seen Scratch plugin sources packaged for ex-vm tree builds. It extracts
only the needed header files from SVN. But it complicates maintenance like you
stated in an earlier mail. Either option (full-tree vs. copied tree) look
extreme to me. All that is needed is a zip file with just the vm headers and
source files. External plugins can be packaged and built individually. They
could even become squeaksource projects like VMMaker.

Thanks .. Subbu

K. K. Subramaniam wrote:
>>> Files like sq.h, sqMemoryAccess.h, sqVirtualMachine.h are all generated
>>> by
>> I'm sorry but that is incorrect. None of these files are generated; they
>> are hand-edited and checked into SVN, see
>>
>> http://squeakvm.org/cgi-bin/viewcvs.cgi/trunk/platforms/Cross/vm/
> You are right. I stand corrected. Thanks. A closer look at SVN shows only
> unix/src/ and Mac OS/.. contain auto-generated files. Though why these
> intermediate files (but not their source .cs/.st files) are in SVN is a mystery.

For the same reason that these files are included in the build files on
Windows as well. They allow one to recompile precisely the same version
that was used in the binary you're using which is tremendously helpful
for debugging. We do the same at Teleplace and it's really only the
difference in build structure (i.e, winbuild/ next to platforms/) that
prevents me from doing the same for the Windows VM.

>>> VMM. The current procedure assumes that all plugins are being generated
>>> at once along with the vm. It does not consider the case where someone is
>>> working on a plugin outside the vm source tree. For a plugin where only a
>>> few hundred lines of code are involved, full vm sources are an overkill.
>> Yes, and there's no need for that. I've successfully written plugins
>> that either had a very small or even no Slang plugin behind it.
> Are these in public domain? url?

It's very simple: Make an InterpreterPlugin subclass and generate the
empty SqueakPlugin.c file. That's your starting template. Getting it
compiled means your build setup is sane. Then add your stuff. Rinse and
repeat.

Cheers,
   - Andreas

On Thursday 24 December 2009 12:04:45 am Andreas Raab wrote: I find this confusing given David's earlier mail about .c files being an
intermediate file. How is this use case different from using -E option in C
compilers to get pre-processed output for debugging purposes? We don't put the
.i file into SVN. We simply generate it on demand from the .c file.

>It's very simple: Make an InterpreterPlugin subclass and generate the
>empty SqueakPlugin.c file. That's your starting template. Getting it
>compiled means your build setup is sane. Then add your stuff. Rinse and
>repeat.
Thanks for the tip. You have been patient in answering my queries.

I used the following script SamplePlugin.st:
-----
Preferences disable: #warnIfNoSourcesFile; disable: #warnIfNoChangesFile.
InterpreterPlugin subclass: #SamplePlugin instanceVariableNames: ''
  classVariableNames: '' poolDictionaries: '' category: 'VMMaker-Plugins'.
SamplePlugin translateInDirectory: (FileDirectory on: '/tmp/vmm')
   doInlining: true.
SmalltalkImage current snapshot: false andQuit: true.
---
and could generate the SamplePlugin.c file with:
  $ squeak -headless unix-3.11.3.image $PWD/SamplePlugin.st

The code refers to sqVirtualMachine.h, sqConfig.h, sqPlatformSpecific.h and
sqMemoryAccess.h. I suppose I can change the script to generate these files for
a given target platform, if necessary.

What holds us back from using vmm as a code translator tool in build scripts
to translate .st/.cs files (checked out from Monticello or SVN) and its
supporting header files on the fly before compilation?

What am I missing here? .. Subbu

K. K. Subramaniam wrote:
The main point of checking in these files is to have a reference that
can be used to create *precisely* the VM that you're using, as close as
possible. The same thing could be achieved by checking in all the
VMMaker code into SVN and regenerate it from there. The choice of
checking in the .c files is rather arbitrary in this context - mostly
driven by the need to fire up an image, load VMMaker, configure it
correctly, generate the .c files. Since we don't have good scripts to do
that, it's somewhat risky to assume everyone will be able to load
VMMaker in precisely the version, using precisely the configuration that
was used for the original build, to recreate these files. Consequently
they are being checked in as well.

I would just check out the required support code and set things up
accordingly. Something along the lines of:

/SamplePlugin
        /cross (check out of platforms/Cross/vm)
        /win32 (check out of platforms/Win32/vm)
        /unix  (check out of platforms/Unix/vm)
        /macos (check out of platforms/Mac OS/vm)

        /src   (your source code)

This way you can stay in sync with the relevant bits at squeakvm.org.

> What holds us back from using vmm as a code translator tool in build scripts
> to translate .st/.cs files (checked out from Monticello or SVN) and its
> supporting header files on the fly before compilation?

Someone has to do it. I suspect that the reason why this has never been
an issue is simply that during development you always have an image open
already so there's little practical need to make that scripted.

Cheers,
   - Andreas