Linux vm

Hello all,

Is there a way to get the pre-compiled linux vm to describe what the full path is to a library it is trying to load?    I can hack the source to do so, but then I start wondering what I might have disturbed in the process.  Out of the box, all I get is an external library with a nil handle and no clue what did or did not happen.

At various points in my struggle today, I saw what looked like a tendency to load plugins from the vmpath//so.pluginName (two slashes rather than one).  It raises the question of whether the code is adding an extra slash when it is not given an explicit plugins path??  The separate concern is that I need information about what is happening when one of my libraries fails to load.

One possibility is that a particular .so can't load.  How would I find out whether that this is case?  Is there a way to test load it or to get the vm to tell me the cause of a failure to load it?

Bill

It has been a long day, and it might not have needed to be so.  Within 15 minutes of seeing "undefined symbol: libusb_close" from dlerror(), I had the problem not only diagnosed but fixed; seeing the error in something related to USB support was enough to suggest I needed to link that library into the .so I was trying to load.  I was still guessing, but it was an educated guess based on a concrete lead.

Is there a way to get such diagnostic information without hacking the vm source?  If not, please consider the following as a proposed patch.  I have tried to strike a balance between seeing enough to make sense of errant plugin paths and giving detailed information about failures when given an absolute path.  Behavior changes (I intend/think) only if something won't load (it says what it was trying to do) and if the library is specified by absolute path, in which case it gives diagnostic output on any failure to load.

Bill


( Squeak-...-src/unix/vm/sqUnixExternalPrims.c )
static void *tryLoadModule(char *in, char *name)
{
  char path[PATH_MAX], *out= path;
  void *handle= 0;
  int c;
  while ((c= *in++) && ':' != c) { /* copy next plugin path to path[] */
    switch (c) {
    case '%':
      if ('n' == *in || 'N' == *in) { /* replace %n with name of plugin */
        ++in;
        strcpy(out, name);
        out += strlen(name);
        continue;
      }
      if ('%' == *in) {
        ++in;
        *out++= '%';
        continue;
      }
      /* fall through... */
    default:
      *out++= c;
      continue;
    }
  }
  sprintf(out, "/" MODULE_PREFIX "%s" MODULE_SUFFIX, name);
  handle= dlopen(path, RTLD_NOW | RTLD_GLOBAL);
  fdebugf((stderr, "tryLoading(%s) = %p\n", path, handle));

  // 8-10 wks - no silent failures
        if( !handle ){
                // 8-10 - if a load fails, tell the user what we tried to load:
                fprintf(stdout, "Load failed for: %s\n", path);

                // 8-10 - is the given path absolute?  If so, try again with it
                // as given and report diagnostic information if it fails.
                if( (int)(name[0])==47 ) {
                        handle=dlopen(name, RTLD_NOW | RTLD_GLOBAL);
                        fprintf(stdout, "Load-by-name(%s) = %p\n", name, handle);
                        if(!handle){
                                fprintf(stdout,"Error loading library: %s\n",dlerror());
                        }
                }
        }

  if (!handle) {
    struct stat buf;
    if ((0 == stat(path, &buf)) && ! S_ISDIR(buf.st_mode))
      fprintf(stderr, "%s\n", dlerror());
  }

  return handle;
}

________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Schwab,Wilhelm K [[hidden email]]
Sent: Saturday, August 21, 2010 8:19 PM
To: [hidden email]; [hidden email]
Subject: [Pharo-project] Linux vm

Hello all,

Is there a way to get the pre-compiled linux vm to describe what the full path is to a library it is trying to load?    I can hack the source to do so, but then I start wondering what I might have disturbed in the process.  Out of the box, all I get is an external library with a nil handle and no clue what did or did not happen.

At various points in my struggle today, I saw what looked like a tendency to load plugins from the vmpath//so.pluginName (two slashes rather than one).  It raises the question of whether the code is adding an extra slash when it is not given an explicit plugins path??  The separate concern is that I need information about what is happening when one of my libraries fails to load.

One possibility is that a particular .so can't load.  How would I find out whether that this is case?  Is there a way to test load it or to get the vm to tell me the cause of a failure to load it?

Bill


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

 
Hi Bill,

Eliot,

Thanks for you prompt and helpful reply.  I will admit I have not tried this yet, but I see where you are going and think it is better to cross-post this into Pharo-dev so it will be searchable via the Gmane archive than to actually see the messages that result; I will sooner or later get to the latter.

There is still the possible problem of an errant / in the default plugins path, and that the vm does not provide feedback on failures, at least not that I can find.  Problems like I had yesterday go from a frustrating mystery to readily solved with a tiny hint that (in this case) is available from dlerror(); we should pass it along to the programmer.

Bill


====================================
Eliot Miranda eliot.miranda at gmail.com
Sun Aug 22 02:53:30 UTC 2010

What I do is manually repeat the link command using options that will cause
the linker to complain about unresolved symbols. You'll get some false
positives (dynamically-linked runtime routines) but you'll also get any
actually undefined symbols.  Use  -Wl,--warn-unresolved-symbols
-Wl,--no-allow-shlib-undefined when manually repeating the link command.

So while this is after the fact you should try remaking your plugin with the
libusb support missing, copy the link command and repeat it
adding -Wl,--warn-unresolved-symbols -Wl,--no-allow-shlib-undefined and this
tie you should see the link fail while listing libusb_close et al.

HTH
Eliot

________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Schwab,Wilhelm K [[hidden email]]
Sent: Saturday, August 21, 2010 10:39 PM
To: [hidden email]; [hidden email]
Subject: Re: [Pharo-project] Linux vm

It has been a long day, and it might not have needed to be so.  Within 15 minutes of seeing "undefined symbol: libusb_close" from dlerror(), I had the problem not only diagnosed but fixed; seeing the error in something related to USB support was enough to suggest I needed to link that library into the .so I was trying to load.  I was still guessing, but it was an educated guess based on a concrete lead.

Is there a way to get such diagnostic information without hacking the vm source?  If not, please consider the following as a proposed patch.  I have tried to strike a balance between seeing enough to make sense of errant plugin paths and giving detailed information about failures when given an absolute path.  Behavior changes (I intend/think) only if something won't load (it says what it was trying to do) and if the library is specified by absolute path, in which case it gives diagnostic output on any failure to load.

Bill


( Squeak-...-src/unix/vm/sqUnixExternalPrims.c )
static void *tryLoadModule(char *in, char *name)
{
  char path[PATH_MAX], *out= path;
  void *handle= 0;
  int c;
  while ((c= *in++) && ':' != c) {      /* copy next plugin path to path[] */
    switch (c) {
    case '%':
      if ('n' == *in || 'N' == *in) {   /* replace %n with name of plugin */
        ++in;
        strcpy(out, name);
        out += strlen(name);
        continue;
      }
      if ('%' == *in) {
        ++in;
        *out++= '%';
        continue;
      }
      /* fall through... */
    default:
      *out++= c;
      continue;
    }
  }
  sprintf(out, "/" MODULE_PREFIX "%s" MODULE_SUFFIX, name);
  handle= dlopen(path, RTLD_NOW | RTLD_GLOBAL);
  fdebugf((stderr, "tryLoading(%s) = %p\n", path, handle));

        // 8-10 wks - no silent failures
        if( !handle ){
                // 8-10 - if a load fails, tell the user what we tried to load:
                fprintf(stdout, "Load failed for: %s\n", path);

                // 8-10 - is the given path absolute?  If so, try again with it
                // as given and report diagnostic information if it fails.
                if( (int)(name[0])==47 ) {
                        handle=dlopen(name, RTLD_NOW | RTLD_GLOBAL);
                        fprintf(stdout, "Load-by-name(%s) = %p\n", name, handle);
                        if(!handle){
                                fprintf(stdout,"Error loading library: %s\n",dlerror());
                        }
                }
        }

  if (!handle) {
    struct stat buf;
    if ((0 == stat(path, &buf)) && ! S_ISDIR(buf.st_mode))
      fprintf(stderr, "%s\n", dlerror());
  }

  return handle;
}

________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Schwab,Wilhelm K [[hidden email]]
Sent: Saturday, August 21, 2010 8:19 PM
To: [hidden email]; [hidden email]
Subject: [Pharo-project] Linux vm

Hello all,

Is there a way to get the pre-compiled linux vm to describe what the full path is to a library it is trying to load?    I can hack the source to do so, but then I start wondering what I might have disturbed in the process.  Out of the box, all I get is an external library with a nil handle and no clue what did or did not happen.

At various points in my struggle today, I saw what looked like a tendency to load plugins from the vmpath//so.pluginName (two slashes rather than one).  It raises the question of whether the code is adding an extra slash when it is not given an explicit plugins path??  The separate concern is that I need information about what is happening when one of my libraries fails to load.

One possibility is that a particular .so can't load.  How would I find out whether that this is case?  Is there a way to test load it or to get the vm to tell me the cause of a failure to load it?

Bill


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

 
On Sunday 22 Aug 2010 8:09:22 am Schwab,Wilhelm K wrote:
> Is there a way to get such diagnostic information without hacking the vm
> source?  If not, please consider the following as a proposed patch.  I
> have tried to strike a balance between seeing enough to make sense of
> errant plugin paths and giving detailed information about failures when
> given an absolute path
In your place, I would have tried ldd(1) or strace(1) to detect path load
errors.

In general, failing to load a plugin module is not a fatal error. Sometimes, a
primtive can fallback to using internal routines if the plugin is not
available.

Subbu

Subbu,

It can still be a serious problem as performance can suffer.  In fact, in this case, I am talking about Shared Libraries, not plugins.  There is no fallback code; one succeeds in loading the library or the game is over.

strace might do the job, but what is wrong with having the vm log what it is trying to do?

Bill


________________________________________
From: K. K. Subramaniam [[hidden email]]
Sent: Monday, August 23, 2010 2:33 AM
To: [hidden email]
Cc: Schwab,Wilhelm K; [hidden email]
Subject: Re: [Vm-dev] RE: Linux vm

On Sunday 22 Aug 2010 8:09:22 am Schwab,Wilhelm K wrote:
> Is there a way to get such diagnostic information without hacking the vm
> source?  If not, please consider the following as a proposed patch.  I
> have tried to strike a balance between seeing enough to make sense of
> errant plugin paths and giving detailed information about failures when
> given an absolute path
In your place, I would have tried ldd(1) or strace(1) to detect path load
errors.

In general, failing to load a plugin module is not a fatal error. Sometimes, a
primtive can fallback to using internal routines if the plugin is not
available.

Subbu

 
On Monday 23 Aug 2010 6:49:43 pm Schwab,Wilhelm K wrote:
> strace might do the job, but what is wrong with having the vm log what it
> is trying to do?
You could also use ltrace(1).

The vm already logs entries in debug mode. Non-debug run-times can be traced
with strace/ltrace. Performance can be optimized by changing LD_LIBRARY_PATH
and -plugins options in the launcher script.

In any case, you can send in your suggestions (complete file, not a diff) for
Ian's consideration.

Subbu

Subbu,

What is the debug mode?  Is it a build option, or something more readily available to an end user?

As for the code, I provided the entire function.  Depending on what falls out of the question above, I might find that all is well as-is; otherwise I can send the entire file (where?) if needed.

Bill



________________________________________
From: K. K. Subramaniam [[hidden email]]
Sent: Monday, August 23, 2010 11:06 AM
To: Schwab,Wilhelm K
Cc: [hidden email]
Subject: Re: [Vm-dev] RE: Linux vm

On Monday 23 Aug 2010 6:49:43 pm Schwab,Wilhelm K wrote:
> strace might do the job, but what is wrong with having the vm log what it
> is trying to do?
You could also use ltrace(1).

The vm already logs entries in debug mode. Non-debug run-times can be traced
with strace/ltrace. Performance can be optimized by changing LD_LIBRARY_PATH
and -plugins options in the launcher script.

In any case, you can send in your suggestions (complete file, not a diff) for
Ian's consideration.

Subbu

 
On Monday 23 Aug 2010 8:47:31 pm Schwab,Wilhelm K wrote:
> Subbu,
>
> What is the debug mode?  Is it a build option, or something more readily
> available to an end user?
It is build option (see debug.h) -DDEBUG=1. If turned on, you get detailed
dump on stderr.

> As for the code, I provided the entire function.  Depending on what falls
> out of the question above, I might find that all is well as-is; otherwise
> I can send the entire file (where?) if needed.
See Ian's announcement in http://lists.squeakfoundation.org/pipermail/vm-
dev/2009-September/003206.html

Subbu