Smalltalk › Pharo › Pharo Smalltalk Users

compiling libraries for Pharo on macOS

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

11 messages Options

Michel Onoff

compiling libraries for Pharo on macOS

Hello,

I've made some attempts to use the Pharo FFI. I've gone through the
documentation [1] and tried some examples.

While the documentation is quite clear about using external libraries
from Pharo, it lacks details on how to build own libraries to be used
from Pharo.

Specifically, I'm trying to build on all supported platforms, namely
macOS, Windows and Linux. Currently I've got Linux and Windows under
control, but I cannot find a systematic way to build on macOS.

Here's an example for a library that I can use from a small test program
but not from Pharo. Unfortunately, the diagnostic on Pharo is not
detailed: it simply reports that it cannot load the module.

clang -I <include-path> -O3 -c <source-path>
...
clang -dynamiclib <object-files> -o lib<name>.dylib -undefined
dynamic_lookup

cp lib<name>.dylib <pharo-folder>

Is there good documentation about how to correctly build libraries so
that they can be used from Pharo?

Alternatively, are there details on how Pharo FFI searches for libraries
and links them in the Pharo process?

Regards
MO

----

[1] https://files.pharo.org/books-pdfs/booklet-uFFI/UFFIDRAFT.pdf

kilon.alios

Re: compiling libraries for Pharo on macOS

I advice first of all the use of CMake , you can thank me later.

CMake if you are not aware is the standard build system for C/C++ , it makes it super easy to build and compile projects so you dont have to do what you are doing, call the compiler and pass the inifite amount of compiler flags that it needs.

To understand how to build a library you have to understand first what a library is.

Libraries used by Pharo are known as either Shared Libraries on Macos (dylib) and Linux (so) but Dynamically Linked Libraries on Windows (DLLs). They are executables designed to be called from another executable (this includes another library as well).

Those libraries operate in 3 modes

1) Static linking

2) semi dynamic linking

3) dynamic linking

(1) may come as suprise but yes those libraries can be linked to the executable and build inside the exrecutable. But in our case we done want that because it would mean to rebuild the VM.

(2) is not what we want either because eventhough it builds the library independently generating the usual exetension files I mentioned above it includes its symbole table in the executable so you wont have to do what UFFI does, wrap or map your code to those library functions. The equivelant for Pharo would mean that we would not need UFFI at all but we do need to rebuld the VM to include ths symbol table. The symbole table basically includes the name of the functions, their signatures (type of arguments) and their corresponing adresses in memory needed to access them.

(3) is what UFFI does, in that cause the executable is not affected at all so you dont need to to rebuild the vm. But you do need to know where to find the library , to access its symbol table and make sure the variables your executable uses match the type and signature in the symbol table or else the executable wont able to locate and correctly call these functions. Of course libraries can contain structs and global variables as well but the concept is the same. Now this messy part is handled by the UFFI although barely because it still needs from you to provide the correct signature to the function.

I am giving you the long version because you have to make sure you build you library with (3) way and not (2). (1) can be easily avoid but it is easy to mix (2) and (3). From there on its easy , locating the library is provided as a string returned by a method , see how UFFI utilises LibC for example and from there on you just convert the signatures to symbol arrays as described by the UFFI tutorials.

For cmake you only need to use -> add_library(libraryname MODULE ${SRC} ). You can find a CmakeList.txt file that I use here https://gist.github.com/kilon/ff4b973bdeb3b0c0253ac373a4b2506a

If you are wondering why its so big, its because the project I link against is huge and because I move the DLL to its build folder instead of building it in the same folder as my source code. It CMake's why of keeping things seperate and organised. Observe also that I use dll exension for Windows and MacOS, The extensions plays no role so I decided to go for the same extension to save time although usually its preferable go for the standard extensions.

Beware that if your library uses any external code even parts of the C standard library those will have to be included , usually you need only the header files, which is what SET INC is doing in my example, while SET SRC includes only the library files. The inclusion of headers is super important because they are used to build the symbol table which is the most important part of the library. Without a correctly build symbol table the library is useless even when used from C/C++.

On Wed, Oct 17, 2018 at 11:09 AM Michel Onoff <[hidden email]> wrote:

Hello,

I've made some attempts to use the Pharo FFI. I've gone through the
documentation [1] and tried some examples.

While the documentation is quite clear about using external libraries
from Pharo, it lacks details on how to build own libraries to be used
from Pharo.

Specifically, I'm trying to build on all supported platforms, namely
macOS, Windows and Linux. Currently I've got Linux and Windows under
control, but I cannot find a systematic way to build on macOS.

Here's an example for a library that I can use from a small test program
but not from Pharo. Unfortunately, the diagnostic on Pharo is not
detailed: it simply reports that it cannot load the module.

clang -I <include-path> -O3 -c <source-path>
...
clang -dynamiclib <object-files> -o lib<name>.dylib -undefined
dynamic_lookup

cp lib<name>.dylib <pharo-folder>

Is there good documentation about how to correctly build libraries so
that they can be used from Pharo?

Alternatively, are there details on how Pharo FFI searches for libraries
and links them in the Pharo process?

Regards
MO

----

[1] https://files.pharo.org/books-pdfs/booklet-uFFI/UFFIDRAFT.pdf

Michel Onoff

Re: compiling libraries for Pharo on macOS

Hi Dimitris,

thanks for the long explanation.

As you can see from my example, I build the library with the
-undefined dynamic_lookup
flag to clang, so this should be compatible with your (3) option if I
understand you correctly. Compiling/linking this way should allow the
use of dlopen(3) from UFFI during the execution of Pharo code, exactly
as I try to simulate with my small test programs.

Pharo complains about not being able to load the module (the library),
even if it appears in the Pharo main folder. It does not report whether
it cannot find it or if it cannot bind it using dlopen(3).

Moreover, it is unclear to me how Pharo interacts with DYLD_LIBRARY_PATH
(on macOS) or LD_LIBRARY_PATH (on Linux) environment variables if they
are set.

In other words, Where does it search for libraries?

Anyway, I'll try your script on cmake.

Regards
MO

for use with dlopen()

On 2018-10-17 11:32, Dimitris Chloupis wrote:

> I advice first of all the use of CMake , you can thank me later.
>
> CMake if you are not aware is the standard build system for C/C++ , it
> makes it super easy to build and compile projects so you dont have to do
> what you are doing, call the compiler and pass the inifite amount of
> compiler flags that it needs.
>
> To understand how to build a library you have to understand first what a
> library is.
>
> Libraries used by Pharo are known as either Shared Libraries on Macos
> (dylib) and Linux (so) but Dynamically Linked Libraries on Windows
> (DLLs). They are executables designed to be called from another
> executable (this includes another library as well).
>
> Those libraries operate in 3 modes
> 1) Static linking
> 2) semi dynamic linking
> 3) dynamic linking
>
> (1) may come as suprise but yes those libraries can be linked to the
> executable and build inside the exrecutable. But in our case we done
> want that because it would mean to rebuild the VM.
> (2) is not what we want either because eventhough it builds the library
> independently generating the usual exetension files I mentioned above it
> includes its symbole table in the executable so you wont have to do what
> UFFI does, wrap or map your code to those library functions. The
> equivelant for Pharo would mean that we would not need UFFI at all but
> we do need to rebuld the VM to include ths symbol table. The symbole
> table basically includes the name of the functions, their signatures
> (type of arguments) and their corresponing adresses in memory needed to
> access them.
>
> (3) is what UFFI does, in that cause the executable is not affected at
> all so you dont need to to rebuild the vm. But you do need to know where
> to find the library , to access its symbol table and make sure the
> variables your executable uses match the type and signature in the
> symbol table or else the executable wont able to locate and correctly
> call these functions. Of course libraries can contain structs and global
> variables as well but the concept is the same. Now this messy part is
> handled by the UFFI although barely because it still needs from you to
> provide the correct signature to the function.
>
> I am giving you the long version because you have to make sure you build
> you library with (3) way and not (2). (1) can be easily avoid but it is
> easy to mix (2) and (3). From there on its easy , locating the library
> is provided as a string returned by a method , see how UFFI utilises
> LibC for example and from there on you just convert the signatures to
> symbol arrays as described by the UFFI tutorials.
>
> For cmake you only need to use -> add_library(libraryname MODULE ${SRC}
> ). You can find a CmakeList.txt file that I use here
> https://gist.github.com/kilon/ff4b973bdeb3b0c0253ac373a4b2506a
>
> If you are wondering why its so big, its because the project I link
> against is huge and because I move the DLL to its build folder instead
> of building it in the same folder as my source code. It CMake's why of
> keeping things seperate and organised. Observe also that I use dll
> exension for Windows and MacOS, The extensions plays no role so I
> decided to go for the same extension to save time although usually its
> preferable go for the standard extensions.
>
> Beware that if your library uses any external code even parts of the C
> standard library those will have to be included , usually you need only
> the header files, which is what SET INC is doing in my example, while
> SET SRC includes only the library files. The inclusion of headers is
> super important because they are used to build the symbol table which is
> the most important part of the library. Without a correctly build symbol
> table the library is useless even when used from C/C++.
>
> On Wed, Oct 17, 2018 at 11:09 AM Michel Onoff <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> Hello,
>
> I've made some attempts to use the Pharo FFI. I've gone through the
> documentation [1] and tried some examples.
>
> While the documentation is quite clear about using external libraries
> from Pharo, it lacks details on how to build own libraries to be used
> from Pharo.
>
> Specifically, I'm trying to build on all supported platforms, namely
> macOS, Windows and Linux. Currently I've got Linux and Windows under
> control, but I cannot find a systematic way to build on macOS.
>
> Here's an example for a library that I can use from a small test
> program
> but not from Pharo. Unfortunately, the diagnostic on Pharo is not
> detailed: it simply reports that it cannot load the module.
>
>
> clang -I <include-path> -O3 -c <source-path>
> ...
> clang -dynamiclib <object-files> -o lib<name>.dylib -undefined
> dynamic_lookup
>
> cp lib<name>.dylib <pharo-folder>
>
>
> Is there good documentation about how to correctly build libraries so
> that they can be used from Pharo?
>
> Alternatively, are there details on how Pharo FFI searches for
> libraries
> and links them in the Pharo process?
>
>
> Regards
> MO
>
> ----
>
> [1] https://files.pharo.org/books-pdfs/booklet-uFFI/UFFIDRAFT.pdf
>
>

kilon.alios

Re: compiling libraries for Pharo on macOS

yeah I apologise for the wrong example , LibC utilises the enviroment/system variables.

Generally speaking all OSes have a PATH variable that defines the places where common libraries and tools are to be found. If you are on windows you can type in the command prompt , "PATH", in Linux and MacOS is "$PATH" . You can add your path in Windows using the system variables dialog window, in MacOS and Linux it usually happens via .bash-config file that setups the path but I am not so sure.

The easiest way is to do something like macModuleName does in the LGitLibrary class the code is

macModuleName
    | pluginDir |
    pluginDir := Smalltalk vm binary parent / 'Plugins'.
    #('libgit2.dylib' 'libgit2.0.dylib')
        detect: [ :each | (pluginDir / each) exists ]
        ifFound: [ :libName | ^ libName ].

    self error: 'Module not found.'

Hope that helps

On Wed, Oct 17, 2018 at 12:54 PM Michel Onoff <[hidden email]> wrote:

Hi Dimitris,

thanks for the long explanation.

As you can see from my example, I build the library with the
-undefined dynamic_lookup
flag to clang, so this should be compatible with your (3) option if I
understand you correctly. Compiling/linking this way should allow the
use of dlopen(3) from UFFI during the execution of Pharo code, exactly
as I try to simulate with my small test programs.

Pharo complains about not being able to load the module (the library),
even if it appears in the Pharo main folder. It does not report whether
it cannot find it or if it cannot bind it using dlopen(3).

Moreover, it is unclear to me how Pharo interacts with DYLD_LIBRARY_PATH
(on macOS) or LD_LIBRARY_PATH (on Linux) environment variables if they
are set.

In other words, Where does it search for libraries?

Anyway, I'll try your script on cmake.

Regards
MO

for use with dlopen()

On 2018-10-17 11:32, Dimitris Chloupis wrote:
> I advice first of all the use of CMake , you can thank me later.
>
> CMake if you are not aware is the standard build system for C/C++ , it
> makes it super easy to build and compile projects so you dont have to do
> what you are doing, call the compiler and pass the inifite amount of
> compiler flags that it needs.
>
> To understand how to build a library you have to understand first what a
> library is.
>
> Libraries used by Pharo are known as either Shared Libraries on Macos
> (dylib) and Linux (so) but Dynamically Linked Libraries on Windows
> (DLLs). They are executables designed to be called from another
> executable (this includes another library as well).
>
> Those libraries operate in 3 modes
> 1) Static linking
> 2) semi dynamic linking
> 3) dynamic linking
>
> (1) may come as suprise but yes those libraries can be linked to the
> executable and build inside the exrecutable. But in our case we done
> want that because it would mean to rebuild the VM.
> (2) is not what we want either because eventhough it builds the library
> independently generating the usual exetension files I mentioned above it
> includes its symbole table in the executable so you wont have to do what
> UFFI does, wrap or map your code to those library functions. The
> equivelant for Pharo would mean that we would not need UFFI at all but
> we do need to rebuld the VM to include ths symbol table. The symbole
> table basically includes the name of the functions, their signatures
> (type of arguments) and their corresponing adresses in memory needed to
> access them.
>
> (3) is what UFFI does, in that cause the executable is not affected at
> all so you dont need to to rebuild the vm. But you do need to know where
> to find the library , to access its symbol table and make sure the
> variables your executable uses match the type and signature in the
> symbol table or else the executable wont able to locate and correctly
> call these functions. Of course libraries can contain structs and global
> variables as well but the concept is the same. Now this messy part is
> handled by the UFFI although barely because it still needs from you to
> provide the correct signature to the function.
>
> I am giving you the long version because you have to make sure you build
> you library with (3) way and not (2). (1) can be easily avoid but it is
> easy to mix (2) and (3). From there on its easy , locating the library
> is provided as a string returned by a method , see how UFFI utilises
> LibC for example and from there on you just convert the signatures to
> symbol arrays as described by the UFFI tutorials.
>
> For cmake you only need to use -> add_library(libraryname MODULE ${SRC}
> ). You can find a CmakeList.txt file that I use here
> https://gist.github.com/kilon/ff4b973bdeb3b0c0253ac373a4b2506a
>
> If you are wondering why its so big, its because the project I link
> against is huge and because I move the DLL to its build folder instead
> of building it in the same folder as my source code. It CMake's why of
> keeping things seperate and organised. Observe also that I use dll
> exension for Windows and MacOS, The extensions plays no role so I
> decided to go for the same extension to save time although usually its
> preferable go for the standard extensions.
>
> Beware that if your library uses any external code even parts of the C
> standard library those will have to be included , usually you need only
> the header files, which is what SET INC is doing in my example, while
> SET SRC includes only the library files. The inclusion of headers is
> super important because they are used to build the symbol table which is
> the most important part of the library. Without a correctly build symbol
> table the library is useless even when used from C/C++.
>
> On Wed, Oct 17, 2018 at 11:09 AM Michel Onoff <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> Hello,
>
> I've made some attempts to use the Pharo FFI. I've gone through the
> documentation [1] and tried some examples.
>
> While the documentation is quite clear about using external libraries
> from Pharo, it lacks details on how to build own libraries to be used
> from Pharo.
>
> Specifically, I'm trying to build on all supported platforms, namely
> macOS, Windows and Linux. Currently I've got Linux and Windows under
> control, but I cannot find a systematic way to build on macOS.
>
> Here's an example for a library that I can use from a small test
> program
> but not from Pharo. Unfortunately, the diagnostic on Pharo is not
> detailed: it simply reports that it cannot load the module.
>
>
> clang -I <include-path> -O3 -c <source-path>
> ...
> clang -dynamiclib <object-files> -o lib<name>.dylib -undefined
> dynamic_lookup
>
> cp lib<name>.dylib <pharo-folder>
>
>
> Is there good documentation about how to correctly build libraries so
> that they can be used from Pharo?
>
> Alternatively, are there details on how Pharo FFI searches for
> libraries
> and links them in the Pharo process?
>
>
> Regards
> MO
>
> ----
>
> [1] https://files.pharo.org/books-pdfs/booklet-uFFI/UFFIDRAFT.pdf
>
>

Michel Onoff

Re: compiling libraries for Pharo on macOS

Afaik, dlopen(3), which I presume UFFI uses internally to load (shared)
libraries on demand (I cannot imagine anything else), looks in
LD_LIBRARY_PATH (DYLD_LIBRARY_PATH on macOS), not in PATH.

So let's focus on one of my question again, reworded here:

What is the strategy that UFFI internally uses in its own implementation
to search for a library? What if that library depends on other libraries
in turn? Where would these ones be searched for?

On 2018-10-17 12:05, Dimitris Chloupis wrote:

> yeah I apologise for the wrong example , LibC utilises the
> enviroment/system variables.
>
> Generally speaking all OSes have a PATH variable that defines the places
> where common libraries and tools are to be found. If you are on windows
> you can type in the command prompt , "PATH", in Linux and MacOS is
> "$PATH" . You can add your path in Windows using the system variables
> dialog window, in MacOS and Linux it usually happens via .bash-config
> file that setups the path but I am not so sure.
>
> The easiest way is to do something like macModuleName does in the
> LGitLibrary class the code is
>
> macModuleName
>      | pluginDir |
>      pluginDir := Smalltalk vm binary parent / 'Plugins'.
>      #('libgit2.dylib' 'libgit2.0.dylib')
>          detect: [ :each | (pluginDir / each) exists ]
>          ifFound: [ :libName | ^ libName ].
>
>      self error: 'Module not found.'
>
> Hope that helps
>
> On Wed, Oct 17, 2018 at 12:54 PM Michel Onoff <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> Hi Dimitris,
>
> thanks for the long explanation.
>
> As you can see from my example, I build the library with the
>    -undefined dynamic_lookup
> flag to clang, so this should be compatible with your (3) option if I
> understand you correctly. Compiling/linking this way should allow the
> use of dlopen(3) from UFFI during the execution of Pharo code, exactly
> as I try to simulate with my small test programs.
>
> Pharo complains about not being able to load the module (the library),
> even if it appears in the Pharo main folder. It does not report whether
> it cannot find it or if it cannot bind it using dlopen(3).
>
> Moreover, it is unclear to me how Pharo interacts with
> DYLD_LIBRARY_PATH
> (on macOS) or LD_LIBRARY_PATH (on Linux) environment variables if they
> are set.
>
> In other words, Where does it search for libraries?
>
>
>
> Anyway, I'll try your script on cmake.
>
>
> Regards
> MO
>
>
>
>
> for use with dlopen()
>
> On 2018-10-17 11:32, Dimitris Chloupis wrote:
> > I advice first of all the use of CMake , you can thank me later.
> >
> > CMake if you are not aware is the standard build system for C/C++
> , it
> > makes it super easy to build and compile projects so you dont
> have to do
> > what you are doing, call the compiler and pass the inifite amount of
> > compiler flags that it needs.
> >
> > To understand how to build a library you have to understand first
> what a
> > library is.
> >
> > Libraries used by Pharo are known as either Shared Libraries on
> Macos
> > (dylib) and Linux (so) but Dynamically Linked Libraries on Windows
> > (DLLs). They are executables designed to be called from another
> > executable (this includes another library as well).
> >
> > Those libraries operate in 3 modes
> > 1) Static linking
> > 2) semi dynamic linking
> > 3) dynamic linking
> >
> > (1) may come as suprise but yes those libraries can be linked to the
> > executable and build inside the exrecutable. But in our case we done
> > want that because it would mean to rebuild the VM.
> > (2) is not what we want either because eventhough it builds the
> library
> > independently generating the usual exetension files I mentioned
> above it
> > includes its symbole table in the executable so you wont have to
> do what
> > UFFI does, wrap or map your code to those library functions. The
> > equivelant for Pharo would mean that we would not need UFFI at
> all but
> > we do need to rebuld the VM to include ths symbol table. The symbole
> > table basically includes the name of the functions, their signatures
> > (type of arguments) and their corresponing adresses in memory
> needed to
> > access them.
> >
> > (3) is what UFFI does, in that cause the executable is not
> affected at
> > all so you dont need to to rebuild the vm. But you do need to
> know where
> > to find the library , to access its symbol table and make sure the
> > variables your executable uses match the type and signature in the
> > symbol table or else the executable wont able to locate and
> correctly
> > call these functions. Of course libraries can contain structs and
> global
> > variables as well but the concept is the same. Now this messy
> part is
> > handled by the UFFI although barely because it still needs from
> you to
> > provide the correct signature to the function.
> >
> > I am giving you the long version because you have to make sure
> you build
> > you library with (3) way and not (2). (1) can be easily avoid but
> it is
> > easy to mix (2) and (3). From there on its easy , locating the
> library
> > is provided as a string returned by a method , see how UFFI utilises
> > LibC for example and from there on you just convert the
> signatures to
> > symbol arrays as described by the UFFI tutorials.
> >
> > For cmake you only need to use -> add_library(libraryname MODULE
> ${SRC}
> > ). You can find a CmakeList.txt file that I use here
> > https://gist.github.com/kilon/ff4b973bdeb3b0c0253ac373a4b2506a
> >
> > If you are wondering why its so big, its because the project I link
> > against is huge and because I move the DLL to its build folder
> instead
> > of building it in the same folder as my source code. It CMake's
> why of
> > keeping things seperate and organised. Observe also that I use dll
> > exension for Windows and MacOS, The extensions plays no role so I
> > decided to go for the same extension to save time although
> usually its
> > preferable go for the standard extensions.
> >
> > Beware that if your library uses any external code even parts of
> the C
> > standard library those will have to be included , usually you
> need only
> > the header files, which is what SET INC is doing in my example,
> while
> > SET SRC includes only the library files. The inclusion of headers is
> > super important because they are used to build the symbol table
> which is
> > the most important part of the library. Without a correctly build
> symbol
> > table the library is useless even when used from C/C++.
> >
> > On Wed, Oct 17, 2018 at 11:09 AM Michel Onoff
> <[hidden email] <mailto:[hidden email]>
> > <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
> >
> > Hello,
> >
> > I've made some attempts to use the Pharo FFI. I've gone
> through the
> > documentation [1] and tried some examples.
> >
> > While the documentation is quite clear about using external
> libraries
> > from Pharo, it lacks details on how to build own libraries to
> be used
> > from Pharo.
> >
> > Specifically, I'm trying to build on all supported platforms,
> namely
> > macOS, Windows and Linux. Currently I've got Linux and
> Windows under
> > control, but I cannot find a systematic way to build on macOS.
> >
> > Here's an example for a library that I can use from a small test
> > program
> > but not from Pharo. Unfortunately, the diagnostic on Pharo is not
> > detailed: it simply reports that it cannot load the module.
> >
> >
> > clang -I <include-path> -O3 -c <source-path>
> > ...
> > clang -dynamiclib <object-files> -o lib<name>.dylib -undefined
> > dynamic_lookup
> >
> > cp lib<name>.dylib <pharo-folder>
> >
> >
> > Is there good documentation about how to correctly build
> libraries so
> > that they can be used from Pharo?
> >
> > Alternatively, are there details on how Pharo FFI searches for
> > libraries
> > and links them in the Pharo process?
> >
> >
> > Regards
> > MO
> >
> > ----
> >
> > [1] https://files.pharo.org/books-pdfs/booklet-uFFI/UFFIDRAFT.pdf
> >
> >
>
>

kilon.alios

Re: compiling libraries for Pharo on macOS

yes exactly There is for Unix (which includes macos and linux)

UnixDynamicLoader >>loadLibrary: filename flag: flag
^ self ffiCall: #(void *dlopen(const char *filename, int flag))

while for windows is

WindowsDynamicLoader>>loadLibrary: lpFileName
^ self ffiCall: #(void *LoadLibrary(String lpFileName))

So UFFI apparently does not seem to deal with path checking at , instead it lets the DLL loading C functions to do their job. How they do this is completely dependant on the platform, or you can use my approach I linked earlier by including your library inside the Pharo folder relative to the folder of the VM.

But if you want to do this the standard then yes you will have to do this the standard way ;)

So look at how C does this and you will have your answer. This is a C question rather a Pharo questions, unless I am missing something here.

On Wed, Oct 17, 2018 at 1:33 PM Michel Onoff <[hidden email]> wrote:

Afaik, dlopen(3), which I presume UFFI uses internally to load (shared)
libraries on demand (I cannot imagine anything else), looks in
LD_LIBRARY_PATH (DYLD_LIBRARY_PATH on macOS), not in PATH.

So let's focus on one of my question again, reworded here:

What is the strategy that UFFI internally uses in its own implementation
to search for a library? What if that library depends on other libraries
in turn? Where would these ones be searched for?

On 2018-10-17 12:05, Dimitris Chloupis wrote:
> yeah I apologise for the wrong example , LibC utilises the
> enviroment/system variables.
>
> Generally speaking all OSes have a PATH variable that defines the places
> where common libraries and tools are to be found. If you are on windows
> you can type in the command prompt , "PATH", in Linux and MacOS is
> "$PATH" . You can add your path in Windows using the system variables
> dialog window, in MacOS and Linux it usually happens via .bash-config
> file that setups the path but I am not so sure.
>
> The easiest way is to do something like macModuleName does in the
> LGitLibrary class the code is
>
> macModuleName
>     | pluginDir |
>     pluginDir := Smalltalk vm binary parent / 'Plugins'.
>     #('libgit2.dylib' 'libgit2.0.dylib')
>         detect: [ :each | (pluginDir / each) exists ]
>         ifFound: [ :libName | ^ libName ].
>
>     self error: 'Module not found.'
>
> Hope that helps
>
> On Wed, Oct 17, 2018 at 12:54 PM Michel Onoff <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> Hi Dimitris,
>
> thanks for the long explanation.
>
> As you can see from my example, I build the library with the
> -undefined dynamic_lookup
> flag to clang, so this should be compatible with your (3) option if I
> understand you correctly. Compiling/linking this way should allow the
> use of dlopen(3) from UFFI during the execution of Pharo code, exactly
> as I try to simulate with my small test programs.
>
> Pharo complains about not being able to load the module (the library),
> even if it appears in the Pharo main folder. It does not report whether
> it cannot find it or if it cannot bind it using dlopen(3).
>
> Moreover, it is unclear to me how Pharo interacts with
> DYLD_LIBRARY_PATH
> (on macOS) or LD_LIBRARY_PATH (on Linux) environment variables if they
> are set.
>
> In other words, Where does it search for libraries?
>
>
>
> Anyway, I'll try your script on cmake.
>
>
> Regards
> MO
>
>
>
>
> for use with dlopen()
>
> On 2018-10-17 11:32, Dimitris Chloupis wrote:
> > I advice first of all the use of CMake , you can thank me later.
> >
> > CMake if you are not aware is the standard build system for C/C++
> , it
> > makes it super easy to build and compile projects so you dont
> have to do
> > what you are doing, call the compiler and pass the inifite amount of
> > compiler flags that it needs.
> >
> > To understand how to build a library you have to understand first
> what a
> > library is.
> >
> > Libraries used by Pharo are known as either Shared Libraries on
> Macos
> > (dylib) and Linux (so) but Dynamically Linked Libraries on Windows
> > (DLLs). They are executables designed to be called from another
> > executable (this includes another library as well).
> >
> > Those libraries operate in 3 modes
> > 1) Static linking
> > 2) semi dynamic linking
> > 3) dynamic linking
> >
> > (1) may come as suprise but yes those libraries can be linked to the
> > executable and build inside the exrecutable. But in our case we done
> > want that because it would mean to rebuild the VM.
> > (2) is not what we want either because eventhough it builds the
> library
> > independently generating the usual exetension files I mentioned
> above it
> > includes its symbole table in the executable so you wont have to
> do what
> > UFFI does, wrap or map your code to those library functions. The
> > equivelant for Pharo would mean that we would not need UFFI at
> all but
> > we do need to rebuld the VM to include ths symbol table. The symbole
> > table basically includes the name of the functions, their signatures
> > (type of arguments) and their corresponing adresses in memory
> needed to
> > access them.
> >
> > (3) is what UFFI does, in that cause the executable is not
> affected at
> > all so you dont need to to rebuild the vm. But you do need to
> know where
> > to find the library , to access its symbol table and make sure the
> > variables your executable uses match the type and signature in the
> > symbol table or else the executable wont able to locate and
> correctly
> > call these functions. Of course libraries can contain structs and
> global
> > variables as well but the concept is the same. Now this messy
> part is
> > handled by the UFFI although barely because it still needs from
> you to
> > provide the correct signature to the function.
> >
> > I am giving you the long version because you have to make sure
> you build
> > you library with (3) way and not (2). (1) can be easily avoid but
> it is
> > easy to mix (2) and (3). From there on its easy , locating the
> library
> > is provided as a string returned by a method , see how UFFI utilises
> > LibC for example and from there on you just convert the
> signatures to
> > symbol arrays as described by the UFFI tutorials.
> >
> > For cmake you only need to use -> add_library(libraryname MODULE
> ${SRC}
> > ). You can find a CmakeList.txt file that I use here
> > https://gist.github.com/kilon/ff4b973bdeb3b0c0253ac373a4b2506a
> >
> > If you are wondering why its so big, its because the project I link
> > against is huge and because I move the DLL to its build folder
> instead
> > of building it in the same folder as my source code. It CMake's
> why of
> > keeping things seperate and organised. Observe also that I use dll
> > exension for Windows and MacOS, The extensions plays no role so I
> > decided to go for the same extension to save time although
> usually its
> > preferable go for the standard extensions.
> >
> > Beware that if your library uses any external code even parts of
> the C
> > standard library those will have to be included , usually you
> need only
> > the header files, which is what SET INC is doing in my example,
> while
> > SET SRC includes only the library files. The inclusion of headers is
> > super important because they are used to build the symbol table
> which is
> > the most important part of the library. Without a correctly build
> symbol
> > table the library is useless even when used from C/C++.
> >
> > On Wed, Oct 17, 2018 at 11:09 AM Michel Onoff
> <[hidden email] <mailto:[hidden email]>
> > <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
> >
> > Hello,
> >
> > I've made some attempts to use the Pharo FFI. I've gone
> through the
> > documentation [1] and tried some examples.
> >
> > While the documentation is quite clear about using external
> libraries
> > from Pharo, it lacks details on how to build own libraries to
> be used
> > from Pharo.
> >
> > Specifically, I'm trying to build on all supported platforms,
> namely
> > macOS, Windows and Linux. Currently I've got Linux and
> Windows under
> > control, but I cannot find a systematic way to build on macOS.
> >
> > Here's an example for a library that I can use from a small test
> > program
> > but not from Pharo. Unfortunately, the diagnostic on Pharo is not
> > detailed: it simply reports that it cannot load the module.
> >
> >
> > clang -I <include-path> -O3 -c <source-path>
> > ...
> > clang -dynamiclib <object-files> -o lib<name>.dylib -undefined
> > dynamic_lookup
> >
> > cp lib<name>.dylib <pharo-folder>
> >
> >
> > Is there good documentation about how to correctly build
> libraries so
> > that they can be used from Pharo?
> >
> > Alternatively, are there details on how Pharo FFI searches for
> > libraries
> > and links them in the Pharo process?
> >
> >
> > Regards
> > MO
> >
> > ----
> >
> > [1] https://files.pharo.org/books-pdfs/booklet-uFFI/UFFIDRAFT.pdf
> >
> >
>
>

Ben Coman

Re: compiling libraries for Pharo on macOS

In reply to this post by Michel Onoff

On Wed, 17 Oct 2018 at 18:33, Michel Onoff <[hidden email]> wrote:

Afaik, dlopen(3), which I presume UFFI uses internally to load (shared)
libraries on demand (I cannot imagine anything else), looks in
LD_LIBRARY_PATH (DYLD_LIBRARY_PATH on macOS), not in PATH.

What is the strategy that UFFI internally uses in its own implementation
to search for a library? What if that library depends on other libraries
in turn? Where would these ones be searched for?

I'm not really familiar with the mechanism, but a quick trawl of the VM sources found these...

[1] https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/vm/sqUnixExternalPrims.c

[2] https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/Mac%20OS/vm/sqMacUnixExternalPrims.c

[3] https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/iOS/vm/OSX/sqMacUnixExternalPrims.m

[4] https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/iOS/vm/iPhone/sqMacUnixExternalPrims.m

For [1] and [2], ioLoadModule() seems to use LD_LIBRARY_PATH,

but [3] and [4] don't.

IIUC, [2] is a deprecated version and [3] is the one you are concerned about.

btw, If you want to discuss those files, consider cross posting to vm-dev.

http://lists.squeakfoundation.org/mailman/listinfo/vm-dev

cheers -ben

Michel Onoff

Re: compiling libraries for Pharo on macOS

Ben, thanks for the references.

On 2018-10-17 13:57, Ben Coman wrote:

> On Wed, 17 Oct 2018 at 18:33, Michel Onoff <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> Afaik, dlopen(3), which I presume UFFI uses internally to load (shared)
> libraries on demand (I cannot imagine anything else), looks in
> LD_LIBRARY_PATH (DYLD_LIBRARY_PATH on macOS), not in PATH.
>
> What is the strategy that UFFI internally uses in its own
> implementation
> to search for a library? What if that library depends on other
> libraries
> in turn? Where would these ones be searched for?
>
>
> I'm not really familiar with the mechanism, but a quick trawl of the VM
> sources found these...
> [1]
> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/vm/sqUnixExternalPrims.c
> [2]
> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/Mac%20OS/vm/sqMacUnixExternalPrims.c
> [3]
> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/iOS/vm/OSX/sqMacUnixExternalPrims.m
> [4]
> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/iOS/vm/iPhone/sqMacUnixExternalPrims.m
>
> For [1] and [2], ioLoadModule() seems to use LD_LIBRARY_PATH,
> but [3] and [4] don't.
>
> IIUC, [2] is a deprecated version and [3] is the one you are concerned
> about.
>
> btw, If you want to discuss those files, consider cross posting to vm-dev.
> http://lists.squeakfoundation.org/mailman/listinfo/vm-dev
>
> cheers -ben

Pierce Ng-3

Re: compiling libraries for Pharo on macOS

In reply to this post by Michel Onoff

On Wed, Oct 17, 2018 at 10:08:58AM +0200, Michel Onoff wrote:
> Specifically, I'm trying to build on all supported platforms, namely macOS,
> Windows and Linux. Currently I've got Linux and Windows under control, but I
> cannot find a systematic way to build on macOS.

When I last tried either on Yosemite or El Capitan, I found that the
Pharo at that time could not load 32/64-bit universal libraries.
Building a 32-bit library for 32-bit Pharo fixed it for me. You might
want to try it.

Pierce

Michel Onoff

Re: compiling libraries for Pharo on macOS

On 2018-10-18 05:31, Pierce Ng wrote:

> On Wed, Oct 17, 2018 at 10:08:58AM +0200, Michel Onoff wrote:
>> Specifically, I'm trying to build on all supported platforms, namely macOS,
>> Windows and Linux. Currently I've got Linux and Windows under control, but I
>> cannot find a systematic way to build on macOS.
>
> When I last tried either on Yosemite or El Capitan, I found that the
> Pharo at that time could not load 32/64-bit universal libraries.
> Building a 32-bit library for 32-bit Pharo fixed it for me. You might
> want to try it.
>
> Pierce
>
>

In the end the problem was not building, which is rather simple even
using clang directly, but the locations that Pharo searches for
libraries. Putting the library there or using a full path solved the
problem for my 64 ibt Pharo 6.1/macOS Mojave installation.

Thanks
MO

Sean P. DeNigris

Re: compiling libraries for Pharo on macOS

Administrator

Michel Onoff wrote
> In the end the problem was… but the locations that Pharo searches for
> libraries. Putting the library there or using a full path solved the
> problem for my 64 ibt Pharo 6.1/macOS Mojave installation.

Great! Thanks for posting the solution :)

-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Cheers,
Sean