plugin location

 
Does anyone use the plugin mechanism to load libraries that are not  
plugins on Unix?  Or to override just one or two of the installed  
ones from a user directory?

I'm thinking of simplifying the search strategy (which is indeed  
broken w.r.t. overriding installed plugins as Subbu points out) along  
with all the junk related to probing for a zillion prefixes and  
suffixes.  Unlike libtool, CMake manages to build loadable modules  
with predictable 'lib*.so' names regardless of the platform.  That,  
combined with a launch script that can add a -plugins option to the  
VM args, suggests it ought to be possible to find the plugin  
precisely on the first attempt, without having to search at all.

The advantage is vastly simpler logic that is completely predictable.

The disadvantage is that it will not be possible to subvert the  
plugin mechanism to load system libraries, and it will not be  
possible to override the installed plugins with a single plugin built  
in a different directory.  Would either of these be a noticeable loss?

Cheers,
Ian

 
Would this affect the FFI and its attempts to load platform libraries?

Cheers,
   - Andreas

Ian Piumarta wrote:

 
On Wed, Aug 26, 2009 at 07:16:42PM -0700, Ian Piumarta wrote:
Predictable and easy to understand is more important than convenience,
so it sounds like a net win to me.

I like being able to work on a plugin that I can build and run from
a local build directory (cd ./build; make; ./squeak myimage) and have
it find the plugin in the ./build directory. I also like to be able
to run my ./build/squeak under gdb for debugging the plugin. It's nice
if that mechanism continues to work but not essential.

Dave

 
I've a flag in the mac carbon vm that turns off oh 90% of the  
searching, that is ON by default,
people haven't complained. Your milage might be different for Linux  
users.


SqueakPluginsBuiltInOrLocalOnly

when set to true makes the lookup logic only consider unix libraries  
or os-x bundles in
the ./Plugins folder in the same directory as the app, or the  
application Resources folder,
or as a foo.framework in the /System/Library/Frameworks/.

To turn this feature off and then consider the other 80+ choices of  
library locations and
library names set SqueakPluginsBuiltInOrLocalOnly to false.Turning the  
feature on
avoids a couple of hundred stat calls at startup time as the VM hunts  
for external versions
of internal plugins. It was added to reduce overhead and help the  
macIntel migration were
we provided new universal binaries that live in the application  
Resources folder since we cann't
load powerpc plugins on macintel machines if we found them first in  
the ./Pugins folder


We skip looking for

   static char *prefixes[]= { "", "lib", 0 };
   static char *suffixes[]= { "", ".so", ".dylib",0 };

and for local or built in does the vm directory path (which is the os-
x bundle resource folder), then ./Plugins

          if ( (handle= tryLoading( vmDirPath, pluginName))
                || (handle= tryLoading( pluginDirPath, pluginName))

and skips
                        || (handle= tryLoading(    "./", pluginName))
                          || (handle= tryLoadingPath("SQUEAK_PLUGIN_PATH", pluginName))
                          || (handle= tryLoading(    VM_LIBDIR"/", pluginName))
                          || (handle= tryLoadingPath("LD_LIBRARY_PATH",
pluginName))
                          || (handle= tryLoading(    "", pluginName))
                #    if defined(VM_X11DIR)
                          || (handle= tryLoading(VM_X11DIR"/", pluginName))
                #    endif
                          )

A special case to this is a bit more hunting if the plugin is  
*.framework since we then consider the vmDir, the puginDirpath, and  
the systemFolder

Otherwise it would also hunt in
        "/CoreServices.framework/Frameworks",
        "/ApplicationServices.framework/Frameworks",
        "/Carbon.framework/Frameworks",



On 26-Aug-09, at 7:16 PM, Ian Piumarta wrote:

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>   Twitter:  
squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================

 
On Aug 26, 2009, at 8:02 PM, Andreas Raab wrote:

> Would this affect the FFI and its attempts to load platform libraries?

It would continue to find anything that dlopen() would find; i.e.,  
anything with the platform's expected prefix and suffix in the  
standard shared library locations including whatever influence  
environment variables might have on the search.

In other words, I am suggesting:

1. The SQUEAK_PLUGINS or -plugins location (which could be multiple  
paths separated by ':'s) with 'lib' prefix and '.so' suffix.  This  
finds the VM plugins on all platforms regardless of local library  
naming conventions and permits non-installed libraries to override  
installed ones.

2. The default places searched by dlopen() with the platform's  
default library prefix and suffix added, unless there is a / in the  
name in which case I'd pass it verbatim.  This finds the FFI  
libraries (which AFAICT don't tend to have prefix/suffix in their  
module name in FFI declarations).

Anyone wanting to find platform libraries in non-standard places  
(e.g., X11 on many non-Linux platforms) would have to provide a full  
path to the library or set LD_LIBRARY_PATH (or whatever) as required.

John's report makes me think the above is almost reasonable. :)

Cheers,
Ian

 
On Thursday 27 Aug 2009 9:15:39 am Ian Piumarta wrote:
> Anyone wanting to find platform libraries in non-standard places  
> (e.g., X11 on many non-Linux platforms) would have to provide a full  
> path to the library or set LD_LIBRARY_PATH (or whatever) as required.
>
> John's report makes me think the above is almost reasonable. :)
"there must be a simpler way" .. Dan Ingalls (talk on lively kernel)

Subbu

On Thu, Aug 27, 2009 at 4:16 AM, Ian Piumarta<[hidden email]> wrote:
Agreed.

> The disadvantage is that it will not be possible to subvert the plugin
> mechanism to load system libraries, and it will not be possible to override
> the installed plugins with a single plugin built in a different directory.
>  Would either of these be a noticeable loss?

Nope.  Go for it!

- Andrew

 
Ian Piumarta wrote:
>> Would this affect the FFI and its attempts to load platform libraries?
>
> It would continue to find anything that dlopen() would find; i.e.,
> anything with the platform's expected prefix and suffix in the standard
> shared library locations including whatever influence environment
> variables might have on the search.

But it wouldn't find it in any of the "additional" locations that you
were looking at, right? It's probably okay (most Unix users know that
they have to set LD_LIBRARY_PATH) but I'm still a little concerned that
this may break apps if people switch from one VM to the next. Perhaps
leave it as an option for the FFI and drop it for good on the 4.0
switch? (changes in major versions are just easier to explain than the
change from 3.10-6 to 3.10-7 ;-)

It's your choice of course.

Cheers,
   - Andreas

 
On Thursday 27 Aug 2009 2:15:19 pm Andreas Raab wrote:
> > It would continue to find anything that dlopen() would find; i.e.,
> > anything with the platform's expected prefix and suffix in the standard
> > shared library locations including whatever influence environment
> > variables might have on the search.
>
> But it wouldn't find it in any of the "additional" locations that you
> were looking at, right?
if it did, wouldn't it be a bug? .. Subbu

 
On 27.08.2009, at 10:36, Andrew Gaylard wrote:


+1

As for debugging, what I usually do is symlink the plugin from my  
build dir to the default plugin dir.

Out of curiosity, which other extensions than .so are in use currently?

- Bert -

 
On Thu, Aug 27, 2009 at 11:10:45AM +0200, Bert Freudenberg wrote:
>
> As for debugging, what I usually do is symlink the plugin from my  
> build dir to the default plugin dir.

Good idea, thanks.

 
Andreas Raab wrote: On platforms using ELF executables you can embed paths for the run time
linker into the exe itself so each vm (and plugin) can be built to look
in its own specific places.

 
On Aug 27, 2009, at 5:21 AM, Douglas Brebner wrote:

> On platforms using ELF executables you can embed paths for the run  
> time
> linker into the exe itself so each vm (and plugin) can be built to  
> look
> in its own specific places.

The ideal would be to set an RPATH on libSqueakFFIPrims.so that  
contains the complete list of likely places to search for FFI  
libraries, but not all platforms are ELF and it appears that RPATH is  
not inherited by the main program (where the dlopen() call is made)  
from dynamically-loaded libraries even when control has passed  
through such a library to reach the dlopen() call.  (This is arguably  
a huge bug in the semantics of C.)

Part of the complication arises because loading a primitive plugin  
and loading a FFI library are conflated in the support code but  
should behave differently.

There is a distinction within the Interpreter between loading a  
"plugin module" for primitives and loading a "library module" for FFI  
callout.  Unfortunately the distinction is destroyed by the time we  
reach ioLoadModule and ioFindExternalFunction in the support code.

However, there are two functions in the vm proxy structure used  
exclusively by FFI to find libraries and symbols that appear to come  
before the plugin/library distinction is destroyed.  It would be  
possible to wrap those trivially such that the platform support can  
tell if the ioLoadModule came from a primitive or from the FFI.  The  
former would look for a piece of VM (by adding plugindir/prefix  
and .suffix to the module name) and the latter would look for a  
library (by doing whatever dlopen() does by default:  FFI library not  
found?  Go RTFM page of dlopen, rtld, dyld, whatever...).

Cheers,
Ian

 

 
On 27.08.2009, at 17:59, Eliot Miranda wrote:

> On Thu, Aug 27, 2009 at 2:10 AM, Bert Freudenberg <[hidden email]
> > wrote:
>
>> Out of curiosity, which other extensions than .so are in use  
>> currently?
>
> HPUX uses .sl & Mac OS X uses .dylib


Hmm. So maybe cmake could figure this out?

- Bert -

 
On Aug 27, 2009, at 9:32 AM, Bert Freudenberg wrote:

>> HPUX uses .sl & Mac OS X uses .dylib
>
>
> Hmm. So maybe cmake could figure this out?

It does.  add_library(SHARED ...) uses the platform's native format.  
add_library(MODULE ...) uses 'lib' in front and '.so' in back.  There  
are variables to tell you what the native prefix and sufix are for  
libraries.

The biggest issue was libtool using some totally random combination  
of name, lib (or not), .so or .dylib (or not), version number (in one  
two or three parts, correct or all zeros, placed before or after  
the .so and maybe separated by a -, or maybe not) each according to  
the ls bit in /dev/random multiplied by the phase of the moon.  
Buried somewhere in a .lobscure file someplace might be a clue as to  
what it decided to do.

Cheers,
Ian (a cmaker who is never going back to autohell)

 
On Thu, Aug 27, 2009 at 5:59 PM, Eliot Miranda<[hidden email]> wrote:
>> Out of curiosity, which other extensions than .so are in use currently?
>
> HPUX uses .sl & Mac OS X uses .dylib

Yeah, but let's not worry about HP-UX.  I've never gotten squeak to run on
it, despite considerable trying.  And HP-UX isn't ELF (not on 32-bit PA-RISC
anyway) so there's no RPATH either.   It has its own equivalent way of
storing library paths in the SOM binaries.  But honestly, I've never heard of
any squeakers on HP-UX, so...


- Andrew