Stupidifying FFI/library loading mechanism
Locked
12
12
Stupidifying FFI/library loading mechanism
 
|
Hello, i am fed up seeing the messages like "FFI can't find a library" or "VM can't find my plugin". I think we should trash too clever intrinsic library lookup mechanism and provide a simple primitive, which attempts to load a library by given file name & path and if it fails, give an extended response, why: - library found, but can't be loaded (because it 64bit while you running 32 bit), or it can't be found and so on. It should be up to the user to find and load the library first, and only after you obtained a valid library handle, attempt to do FFI calls. IMO, same should be done to plugin loading mechanism. VM should give you the paths: - what is the current directory - what is the directory where VM located then user code could attempt to load external plugin by specifying one of these paths or any other to the library which representing plugin. And only after he knows that library successfully loaded, he could attempt to use its primitives (or do FFI calls). The approach is simple: - first you making sure that you loaded external module/plugin, then you start using it. And it is your responsibility to make sure you loaded it, not VM's. This could be done at image startup or at code entry point, or even inside a generic code which handling a primitive failure. VM should behave like a stupid idiot: - if module with name, which specified in primitive pragma <primitive: 'foo' module: 'bar' > not registered already in internal list of VM's modules, it should fail unconditionally without even trying to load external library. Even for internal plugins: A user code can handle this error and instruct VM explicitly to load & initialize internal plugin, but not implicitly like currently VM does for internal and external plugins. If we put a module loading mechanism at image side, it will give us much more flexibility, and all future/potential problems with it could be fixed there without touching VM code anymore. I know that it means quite a lot of work (some named primitives even assume that they never fail). But i think it is worth it. Because if it will stay like that, we will always have issues with it, over and over again. -- Best regards, Igor Stasenko AKA sig. |
Re: Stupidifying FFI/library loading mechanism
|
|
I am not sure if it applies to your proposal here, but just in case I want to mention one note of caution. The #primLoadModule that was discussed previously in the "Sandbox" thread will not work on a 64-bit platform, for the same reason that FFI does not work on a 64-bit platform. Dave On Thu, May 26, 2011 at 01:48:42PM +0200, Igor Stasenko wrote: > > Hello, > i am fed up seeing the messages like "FFI can't find a library" or "VM > can't find my plugin". > > I think we should trash too clever intrinsic library lookup mechanism > and provide a simple primitive, which > attempts to load a library by given file name & path and if it fails, > give an extended response, why: > - library found, but can't be loaded (because it 64bit while you > running 32 bit), or it can't be found and so on. > > It should be up to the user to find and load the library first, and > only after you obtained a valid library > handle, attempt to do FFI calls. > > IMO, same should be done to plugin loading mechanism. VM should give > you the paths: > - what is the current directory > - what is the directory where VM located > > then user code could attempt to load external plugin by specifying one > of these paths or any other to the library which representing plugin. > And only after he knows that library successfully loaded, he could > attempt to use its primitives (or do FFI calls). > > The approach is simple: > - first you making sure that you loaded external module/plugin, then > you start using it. And it is your responsibility to make sure you > loaded it, > not VM's. > This could be done at image startup or at code entry point, or even > inside a generic code which handling a primitive failure. > > VM should behave like a stupid idiot: > - if module with name, which specified in primitive pragma > <primitive: 'foo' module: 'bar' > > not registered already in internal list of VM's modules, it should > fail unconditionally without even trying to load external library. > Even for internal plugins: A user code can handle this error and > instruct VM explicitly to load & initialize > internal plugin, but not implicitly like currently VM does for > internal and external plugins. > > If we put a module loading mechanism at image side, it will give us > much more flexibility, and all future/potential problems > with it could be fixed there without touching VM code anymore. > > I know that it means quite a lot of work (some named primitives even > assume that they never fail). But i think it is worth it. > Because if it will stay like that, we will always have issues with it, > over and over again. > > -- > Best regards, > Igor Stasenko AKA sig. |
Re: Stupidifying FFI/library loading mechanism
|
|
On 26 May 2011 14:16, David T. Lewis <[hidden email]> wrote: > > I am not sure if it applies to your proposal here, but just in > case I want to mention one note of caution. The #primLoadModule > that was discussed previously in the "Sandbox" thread will not > work on a 64-bit platform, for the same reason that FFI does not > work on a 64-bit platform. > Sorry, i din't followed that issue (cant remember). The reason because of mixing 32/64bit libraries, or something else? The primitive you mentioning reusing the same routine to load external module as VM internally for primitives do. So, it is step in right direction (explicitly instruct VM to load modules), but more work needs to be done. Now imagine that you have a primitives: loadInternalModule: moduleName loadExternalModule: moduleName path: pathToModuleLibrary and by analogy, for FFI: loadExternalLibrary: pathToExternalLibrary and in case of failure, these prims could then give you an extended response: - module already loaded - internal module not exists - external library file not found - external library file found but cannot be loaded (not a library, mix 64/32bit mode etc) - module loaded but failed to initialize .. etc Then you can easily diagnose the problems per each attempt to load some library. But not like today.. if you telling to load module named 'GL' , VM tries all different combinations like libGL.so GL.so libGL.dynlib multiplied on various crazy locations in file system.. Is it only me who thinks that, this looks like shooting blindly into the sky hoping there are enough ducks flying so you can hit one? Also, if you explicitly control the module loading then you can also control which module takes priority - internal or external one. if you know, today's VMs first attempting to load external library with given name, and only then internal plugins. Apparently, if you wanna play games in sandbox, you'd be willing to change a priority or even do not attempt to use any external modules at all. -- Best regards, Igor Stasenko AKA sig. |
Re: Stupidifying FFI/library loading mechanism
|
|
2011/5/26 Igor Stasenko <[hidden email]>
+1 paths used to look for external libraries (FFI or plugins) should be simpler and documented. I am always looking into log to know why it failed. IMO, it should also be easy to share external libs between VMs. I already share some libraries but it requires to discover some strange paths. #Luc
|
Re: Stupidifying FFI/library loading mechanism
|
|
In reply to this post by Igor Stasenko
On Thu, May 26, 2011 at 02:39:52PM +0200, Igor Stasenko wrote: > > On 26 May 2011 14:16, David T. Lewis <[hidden email]> wrote: > > > > I am not sure if it applies to your proposal here, but just in > > case I want to mention one note of caution. The #primLoadModule > > that was discussed previously in the "Sandbox" thread will not > > work on a 64-bit platform, for the same reason that FFI does not > > work on a 64-bit platform. > > > Sorry, i din't followed that issue (cant remember). The reason because > of mixing 32/64bit libraries, or something else? Something else. It's basic type declaration issues, storing 64-bit pointers in 32-bit ints etc. <http://bugs.squeak.org/view.php?id=7237> <http://lists.squeakfoundation.org/pipermail/vm-dev/2008-May/001945.html> Dave |
Re: Stupidifying FFI/library loading mechanism
|
|
In reply to this post by Igor Stasenko
On 26.05.2011, at 13:48, Igor Stasenko wrote: > Hello, > i am fed up seeing the messages like "FFI can't find a library" or "VM > can't find my plugin". I agree for FFI libraries. I do not agree for VM modules. IMHO it is a terrible idea to lump the two together. They happen to share an implementation, but that is rather incidental. Your new primitives should be added to the FFI. Heck, since it's FFI you should be able to just call whatever OS function you want to find out the right path. For FFI this okay because the code in the image needs to deal with highly platform-dependent stuff anyway. VM modules (a.k.a. plugins) are a totally different kind of beast. They are just an implementation detail of the VM. Nothing in the image should need to care about where those plugins are. There are virtually no problems with built-in modules. There are very few and minor problems with external modules. And they are almost exclusive to Linux, where they have gotten much easier to fix since Ian made "squeakvm" be a shell script that sets the right paths and can easily be edited. I see no good reason to abandon this infrastructure. - Bert - |
Re: Stupidifying FFI/library loading mechanism
|
|
In reply to this post by Igor Stasenko
Hello 2011/5/26 Igor Stasenko <[hidden email]>
Can you explain what the difference between this kind of module loading. Why not enough just loadLibrary: pathToLibrary? |
Re: Stupidifying FFI/library loading mechanism
|
|
In reply to this post by Bert Freudenberg
On 26 May 2011 20:12, Bert Freudenberg <[hidden email]> wrote: > > On 26.05.2011, at 13:48, Igor Stasenko wrote: > >> Hello, >> i am fed up seeing the messages like "FFI can't find a library" or "VM >> can't find my plugin". > > I agree for FFI libraries. I do not agree for VM modules. > > IMHO it is a terrible idea to lump the two together. They happen to share an implementation, but that is rather incidental. > library twice. A way to do it is keep an eye of everything what loaded. But anyways, i prefer a user code to reason what to load, not VM. > Your new primitives should be added to the FFI. Heck, since it's FFI you should be able to just call whatever OS function you want to find out the right path. For FFI this okay because the code in the image needs to deal with highly platform-dependent stuff anyway. > > VM modules (a.k.a. plugins) are a totally different kind of beast. They are just an implementation detail of the VM. Nothing in the image should need to care about where those plugins are. There are virtually no problems with built-in modules. There are very few and minor problems with external modules. And they are almost exclusive to Linux, where they have gotten much easier to fix since Ian made "squeakvm" be a shell script that sets the right paths and can easily be edited. I see no good reason to abandon this infrastructure. > Maybe. Depends, what you proposing to answer to future "i can't load plugin" and "i don't know why it not loading" problems. > - Bert - -- Best regards, Igor Stasenko AKA sig. |
Re: Stupidifying FFI/library loading mechanism
|
|
In reply to this post by Denis Kudriashov
On 26.05.2011, at 20:41, Denis Kudriashov wrote: Hello VM modules are not just arbitrary libraries. They provide primitive Smalltalk methods. They have special entry points like setInterpreter to be able to interface with the VM directly. Their location is known to the VM. Internal modules are already loaded. But they are still only "activated" when one of their primitive is called. They do the name-to-function-address mapping on their own rather than using the OS's name resolution mechanism. Libraries loaded using FFI are just C libraries, nothing special about them. The provide C functions. Name lookup is done using OS functions. The VM can only guess about their location. Igor was proposing to take the guessing out of the VM and put it into the image. - Bert - |
Re: Stupidifying FFI/library loading mechanism
|
|
In reply to this post by Denis Kudriashov
On 26 May 2011 20:41, Denis Kudriashov <[hidden email]> wrote: > > Hello > > 2011/5/26 Igor Stasenko <[hidden email]> >> >> loadInternalModule: moduleName >> loadExternalModule: moduleName path: pathToModuleLibrary >> >> and by analogy, for FFI: >> >> loadExternalLibrary: pathToExternalLibrary >> > > Can you explain what the difference between this kind of module loading. Why not enough just loadLibrary: pathToLibrary? > If you loading VM plugin, is expects that dynamic library has setInterpreter() function to initialize it. Otherwise it reports failure. Then VM calls this function and if function returns "ok" , then module is registered in module list and you can start using primitives provided by it. With FFI its different: an external library can hold (or not) anything. There is no any conventions. -- Best regards, Igor Stasenko AKA sig. |
Re: Stupidifying FFI/library loading mechanism
|
|
In reply to this post by Bert Freudenberg
On 26 May 2011 23:35, Bert Freudenberg <[hidden email]> wrote: > > > On 26.05.2011, at 20:41, Denis Kudriashov wrote: > > Hello > > 2011/5/26 Igor Stasenko <[hidden email]> >> >> loadInternalModule: moduleName >> loadExternalModule: moduleName path: pathToModuleLibrary >> >> and by analogy, for FFI: >> >> loadExternalLibrary: pathToExternalLibrary >> > > Can you explain what the difference between this kind of module loading. Why not enough just loadLibrary: pathToLibrary? > > VM modules are not just arbitrary libraries. They provide primitive Smalltalk methods. They have special entry points like setInterpreter to be able to interface with the VM directly. Their location is known to the VM. Internal modules are already loaded. But they are still only "activated" when one of their primitive is called. They do the name-to-function-address mapping on their own rather than using the OS's name resolution mechanism. > Libraries loaded using FFI are just C libraries, nothing special about them. The provide C functions. Name lookup is done using OS functions. The VM can only guess about their location. Igor was proposing to take the guessing out of the VM and put it into the image. What i finding fun that once you load B3D plugin, it already loads GL library, because it is linked with it. But it doesn't means that FFI can locate/use it :) > - Bert - > > > -- Best regards, Igor Stasenko AKA sig. |
Re: Stupidifying FFI/library loading mechanism
|
|
On 26.05.2011, at 23:45, Igor Stasenko wrote: > On 26 May 2011 23:35, Bert Freudenberg <[hidden email]> wrote: >> >> On 26.05.2011, at 20:41, Denis Kudriashov wrote: >> >> Hello >> >> 2011/5/26 Igor Stasenko <[hidden email]> >>> >>> loadInternalModule: moduleName >>> loadExternalModule: moduleName path: pathToModuleLibrary >>> >>> and by analogy, for FFI: >>> >>> loadExternalLibrary: pathToExternalLibrary >>> >> >> Can you explain what the difference between this kind of module loading. Why not enough just loadLibrary: pathToLibrary? >> >> VM modules are not just arbitrary libraries. They provide primitive Smalltalk methods. They have special entry points like setInterpreter to be able to interface with the VM directly. Their location is known to the VM. Internal modules are already loaded. But they are still only "activated" when one of their primitive is called. They do the name-to-function-address mapping on their own rather than using the OS's name resolution mechanism. >> Libraries loaded using FFI are just C libraries, nothing special about them. The provide C functions. Name lookup is done using OS functions. The VM can only guess about their location. Igor was proposing to take the guessing out of the VM and put it into the image. > > What i finding fun that once you load B3D plugin, it already loads GL > library, because it is linked with it. > But it doesn't means that FFI can locate/use it :) That is one advantage of using proper plugins, yes. Because linking the plugin uses the system linker. It will e.g. link it correctly to libGL.so.1 if that's what your system is using. Even though the name given in the Makefile is just "GL". One problem people are experiencing with FFI under Linux in particular is that there is no "libGL.so" on most machines. That symlink only gets installed by the developer packages. "Regular" software links against specific library versions (e.g. libGL.so.1). I wonder how other systems solve that problem. E.g. GNOME 3 introduces "GObject Introspection" which basically provides type annotations to generate FFI calls. But it still would need to figure out the proper library version. Maybe that's listed in the type library file? I don't know. - Bert - |
Re: Stupidifying FFI/library loading mechanism
|
|
In reply to this post by Bert Freudenberg
Thanks for answer. 2011/5/27 Bert Freudenberg <[hidden email]>
|
Re: Stupidifying FFI/library loading mechanism
|
|
In reply to this post by Bert Freudenberg
On 5/26/2011 20:12, Bert Freudenberg wrote: > On 26.05.2011, at 13:48, Igor Stasenko wrote: >> Hello, >> i am fed up seeing the messages like "FFI can't find a library" or "VM >> can't find my plugin". > I agree for FFI libraries. I do not agree for VM modules. > > IMHO it is a terrible idea to lump the two together. They happen to share an implementation, but that is rather incidental. +1. It's a mistake to lump those two together for sure. +1 also for the idea of having more (and better) failure information on why loading of any of them failed (this holds both for the FFI as well as the plugins because their load failure is often even more mysterious). However, a big -1 on the idea of a "dumb" FFI loading function if by "dumb" we mean it to ignore the standard system paths on the platform (LD_LIBRARY_PATH and friends). The whole idea of these is to provide executable with the proper information to find those modules; why on earth would we ignore the information that is specifically provided for that purpose? Cheers, - Andreas |
Re: Stupidifying FFI/library loading mechanism
|
|
On 27 May 2011 10:53, Andreas Raab <[hidden email]> wrote: > > On 5/26/2011 20:12, Bert Freudenberg wrote: >> >> On 26.05.2011, at 13:48, Igor Stasenko wrote: >>> >>> Hello, >>> i am fed up seeing the messages like "FFI can't find a library" or "VM >>> can't find my plugin". >> >> I agree for FFI libraries. I do not agree for VM modules. >> >> IMHO it is a terrible idea to lump the two together. They happen to share >> an implementation, but that is rather incidental. > > +1. It's a mistake to lump those two together for sure. +1 also for the idea > of having more (and better) failure information on why loading of any of > them failed (this holds both for the FFI as well as the plugins because > their load failure is often even more mysterious). > > However, a big -1 on the idea of a "dumb" FFI loading function if by "dumb" > we mean it to ignore the standard system paths on the platform > (LD_LIBRARY_PATH and friends). The whole idea of these is to provide > executable with the proper information to find those modules; why on earth > would we ignore the information that is specifically provided for that > purpose? who said that you will ignore it? I said that VM should expose all useful paths to language side and it should be up to language side, where to look for library. But not magically like it doing now. > > Cheers, > - Andreas > -- Best regards, Igor Stasenko AKA sig. |
Re: Stupidifying FFI/library loading mechanism
|
|
On 5/27/2011 12:40, Igor Stasenko wrote: > On 27 May 2011 10:53, Andreas Raab<[hidden email]> wrote: >> However, a big -1 on the idea of a "dumb" FFI loading function if by "dumb" >> we mean it to ignore the standard system paths on the platform >> (LD_LIBRARY_PATH and friends). The whole idea of these is to provide >> executable with the proper information to find those modules; why on earth >> would we ignore the information that is specifically provided for that >> purpose? > who said that you will ignore it? I said that VM should expose all > useful paths to language side > and it should be up to language side, where to look for library. > But not magically like it doing now. Then perhaps you can clarify what you mean by "I think we should trash too clever intrinsic library lookup mechanism and provide a simple primitive, which attempts to load a library by given file name & path ". To me, this sounded as if you want to deliberately abandon implicit search paths on the platforms. If not, then I don't understand what is different compared with today (other than having better failure information with which I agree). Cheers, - Andreas |
Re: Stupidifying FFI/library loading mechanism
|
|
On 27 May 2011 12:52, Andreas Raab <[hidden email]> wrote: > > On 5/27/2011 12:40, Igor Stasenko wrote: >> >> On 27 May 2011 10:53, Andreas Raab<[hidden email]> wrote: >>> >>> However, a big -1 on the idea of a "dumb" FFI loading function if by >>> "dumb" >>> we mean it to ignore the standard system paths on the platform >>> (LD_LIBRARY_PATH and friends). The whole idea of these is to provide >>> executable with the proper information to find those modules; why on >>> earth >>> would we ignore the information that is specifically provided for that >>> purpose? >> >> who said that you will ignore it? I said that VM should expose all >> useful paths to language side >> and it should be up to language side, where to look for library. >> But not magically like it doing now. > > Then perhaps you can clarify what you mean by "I think we should trash too > clever intrinsic library lookup mechanism and provide a simple primitive, > which attempts to load a library by given file name & path ". To me, this > sounded as if you want to deliberately abandon implicit search paths on the > platforms. If not, then I don't understand what is different compared with > today (other than having better failure information with which I agree). > Clarification: if search is implicitly managed by platform, then there's no problem. (and actually you cannot trash this logic since it sits in platform functions, and VM have to use these functions anyway). But VM should not add its own implicit logic in addition to what provided by platform. > Cheers, > - Andreas -- Best regards, Igor Stasenko AKA sig. |
Re: Stupidifying FFI/library loading mechanism
|
|
On 5/27/2011 13:01, Igor Stasenko wrote: > On 27 May 2011 12:52, Andreas Raab<[hidden email]> wrote: >> Then perhaps you can clarify what you mean by "I think we should trash too >> clever intrinsic library lookup mechanism and provide a simple primitive, >> which attempts to load a library by given file name& path ". To me, this >> sounded as if you want to deliberately abandon implicit search paths on the >> platforms. If not, then I don't understand what is different compared with >> today (other than having better failure information with which I agree). > Clarification: > if search is implicitly managed by platform, then there's no problem. > (and actually you cannot trash this logic since it sits in platform > functions, and VM have to use these functions anyway). > But VM should not add its own implicit logic in addition to what > provided by platform. It seems to me that a good way to address the implicit logic is simply by by exposing (a list of) search paths to the image. In which case you can set the search paths to be as wide, or as narrow, as you'd like and the VM can list what additional search paths it would use by default, thus making this both forwards and backwards compatible. Cheers, - Andreas |
Re: Stupidifying FFI/library loading mechanism
|
|
On 27 May 2011 13:25, Andreas Raab <[hidden email]> wrote: > > On 5/27/2011 13:01, Igor Stasenko wrote: >> >> On 27 May 2011 12:52, Andreas Raab<[hidden email]> wrote: >>> >>> Then perhaps you can clarify what you mean by "I think we should trash >>> too >>> clever intrinsic library lookup mechanism and provide a simple primitive, >>> which attempts to load a library by given file name& path ". To me, this >>> sounded as if you want to deliberately abandon implicit search paths on >>> the >>> platforms. If not, then I don't understand what is different compared >>> with >>> today (other than having better failure information with which I agree). >> >> Clarification: >> if search is implicitly managed by platform, then there's no problem. >> (and actually you cannot trash this logic since it sits in platform >> functions, and VM have to use these functions anyway). >> But VM should not add its own implicit logic in addition to what >> provided by platform. > > It seems to me that a good way to address the implicit logic is simply by by > exposing (a list of) search paths to the image. In which case you can set > the search paths to be as wide, or as narrow, as you'd like and the VM can > list what additional search paths it would use by default, thus making this > both forwards and backwards compatible. > Indeed. In this way it is language side, are the only who responsible for finding a library, while VM could provide a hints , like default search paths, which are most appropriate for current platform. So, if we agreed on that, lets sketch the plan what we need to provide: - extended error handling - explicit module loading mechanism - providing hints for library search by VM -- Best regards, Igor Stasenko AKA sig. |
Re: Stupidifying FFI/library loading mechanism
|
|
On 5/27/2011 13:51, Igor Stasenko wrote: > > On 27 May 2011 13:25, Andreas Raab<[hidden email]> wrote: >> On 5/27/2011 13:01, Igor Stasenko wrote: >>> On 27 May 2011 12:52, Andreas Raab<[hidden email]> wrote: >>>> Then perhaps you can clarify what you mean by "I think we should trash >>>> too >>>> clever intrinsic library lookup mechanism and provide a simple primitive, >>>> which attempts to load a library by given file name& path ". To me, this >>>> sounded as if you want to deliberately abandon implicit search paths on >>>> the >>>> platforms. If not, then I don't understand what is different compared >>>> with >>>> today (other than having better failure information with which I agree). >>> Clarification: >>> if search is implicitly managed by platform, then there's no problem. >>> (and actually you cannot trash this logic since it sits in platform >>> functions, and VM have to use these functions anyway). >>> But VM should not add its own implicit logic in addition to what >>> provided by platform. >> It seems to me that a good way to address the implicit logic is simply by by >> exposing (a list of) search paths to the image. In which case you can set >> the search paths to be as wide, or as narrow, as you'd like and the VM can >> list what additional search paths it would use by default, thus making this >> both forwards and backwards compatible. >> > Indeed. > In this way it is language side, are the only who responsible for > finding a library, > while VM could provide a hints , like default search paths, which are > most appropriate for current platform. > > So, if we agreed on that, lets sketch the plan what we need to provide: > > - extended error handling > - explicit module loading mechanism > - providing hints for library search by VM Almost :-) I really can't see a reason for requiring explicit module loading. Amongst other things it bloats applications (which will attempt to preload all libraries they might *possibly* need instead of those they *actually* need) and slow down system startup (for the same reason) or otherwise require people to write lots of either repeated code (as an example, just look at how many senders of Socket initializeNetwork there are all of which should be removed) or equally pointless extra primitive/ffi failure code. I can really see no good reason for requiring explicit loading other than ensure some library is available and that can be done with better error reporting. Cheers, - Andreas |
«
Return to Squeak VM
|
1 view|%1 views
| Free forum by Nabble | Edit this page |