Pharo VM crash in bootstrap with Inconsistency detected by ld.so

 
Hi Tobias, Esteban and Fabio,

Attempting to bootstrap the image with Pharo 7.0 VMs newer than 30 Jan
are failing with:



+ vm/pharo Pharo7.0-32bit-eabf524.image eval --save "Metacello new
baseline: 'Tonel';repository:
'github://pharo-vcs/tonel:v1.0.5';onWarning: [ :e | Error signal: e
messageText in: e signalerContext ]; load: 'core'"

...

...finished baselinelinear load :
linear load : baseline [BaselineOfTonel]
load : MonticelloTonel-Core-cypress.1
load : MonticelloTonel-FileSystem-cypress.1
Inconsistency detected by ld.so: dl-close.c: 811: _dl_close: Assertion
`map->l_init_called' failed!



VMs prior to this succeed (e.g. the current stable vm70).

The error is happening after the image is saved, so it is actually
possible to continue and successfully bootstrap the image, but
obviously that is not solving the core issue.

Building a debug VM gives more information:



$ gdb ../vmtarget/lib/pharo/5.0-201803091840-21462-Open-a-FileStream-based-on-fd-or-FILE/pharo
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from
../vmtarget/lib/pharo/5.0-201803091840-21462-Open-a-FileStream-based-on-fd-or-FILE/pharo...done.
(gdb) run --headless Pharo7.0-32bit-eabf524.image eval --save
"Metacello new baseline: 'Tonel';repository:
'github://pharo-vcs/tonel:v1.0.5';onWarning: [ :e | Error signal: e
messageText in: e signalerContext ]; load: 'core'"
Starting program:
/home/alistair/pharo7/bootstrap/pharo/vmtarget/lib/pharo/5.0-201803091840-21462-Open-a-FileStream-based-on-fd-or-FILE/pharo
--headless Pharo7.0-32bit-eabf524.image eval --save "Metacello new
baseline: 'Tonel';repository:
'github://pharo-vcs/tonel:v1.0.5';onWarning: [ :e | Error signal: e
messageText in: e signalerContext ]; load: 'core'"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0xf7a7cb40 (LWP 27000)]
Loading (null)
Loading libssl.so
Found libssl.so at libssl.so.1.0.0
Found symbol ASN1_STRING_data in dlhandle_ssl
Found symbol ASN1_STRING_length in RTLD_DEFAULT
Found symbol BIO_free_all in RTLD_DEFAULT
Found symbol BIO_new in RTLD_DEFAULT
Found symbol BIO_s_mem in RTLD_DEFAULT
Found symbol BIO_ctrl_pending in RTLD_DEFAULT
Found symbol BIO_ctrl in RTLD_DEFAULT
Found symbol BIO_write in RTLD_DEFAULT
Found symbol BIO_read in RTLD_DEFAULT
Found symbol ERR_print_errors_fp in RTLD_DEFAULT
Found symbol SSL_CTX_free in RTLD_DEFAULT
Found symbol SSL_CTX_set_cipher_list in RTLD_DEFAULT
Found symbol SSL_CTX_set_default_verify_paths in RTLD_DEFAULT
Found symbol SSL_CTX_ctrl in RTLD_DEFAULT
Found symbol SSL_CTX_use_PrivateKey_file in RTLD_DEFAULT
Found symbol SSL_CTX_use_certificate_file in RTLD_DEFAULT
Found symbol SSL_accept in RTLD_DEFAULT
Found symbol SSL_connect in RTLD_DEFAULT
Found symbol SSL_free in RTLD_DEFAULT
Found symbol SSL_ctrl in RTLD_DEFAULT
Found symbol SSL_get_error in RTLD_DEFAULT
Found symbol SSL_get_peer_certificate in RTLD_DEFAULT
Found symbol SSL_get_verify_result in RTLD_DEFAULT
Found symbol SSL_new in RTLD_DEFAULT
Found symbol SSL_read in RTLD_DEFAULT
Found symbol SSL_set_accept_state in RTLD_DEFAULT
Found symbol SSL_set_bio in RTLD_DEFAULT
Found symbol SSL_set_connect_state in RTLD_DEFAULT
Found symbol SSL_write in RTLD_DEFAULT
Found symbol X509_NAME_get_text_by_NID in RTLD_DEFAULT
Found symbol X509_get_subject_name in RTLD_DEFAULT
Found symbol X509_get_ext_d2i in RTLD_DEFAULT
Found symbol X509_free in RTLD_DEFAULT
Found symbol X509_check_ip_asc in RTLD_DEFAULT
Found symbol X509_check_host in RTLD_DEFAULT
Loading libcrypto.so
Found libcrypto.so at libcrypto.so.1.0.0
Cannot find SSL_CTX_set_options
Found symbol BIO_test_flags in RTLD_DEFAULT
Cannot find ASN1_STRING_get0_data
Found symbol sk_new_null in RTLD_DEFAULT
Found symbol sk_push in RTLD_DEFAULT
Found symbol sk_free in RTLD_DEFAULT
Found symbol sk_value in RTLD_DEFAULT
Found symbol sk_num in RTLD_DEFAULT
Found symbol sk_pop_free in RTLD_DEFAULT
Found symbol SSLv23_method in RTLD_DEFAULT
Found symbol SSL_CTX_new in RTLD_DEFAULT
Cannot find OPENSSL_sk_new_null
Cannot find OPENSSL_sk_push
Cannot find OPENSSL_sk_free
Cannot find OPENSSL_sk_value
Cannot find OPENSSL_sk_num
Cannot find OPENSSL_sk_pop_free
Cannot find TLS_method
Found symbol SSL_library_init in RTLD_DEFAULT
Found symbol SSL_load_error_strings in RTLD_DEFAULT
Cannot find OPENSSL_init_ssl

Fetched -> BaselineOfTonel-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- github://pharo-vcs/tonel:v1.0.5
Loaded -> BaselineOfTonel-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- github://pharo-vcs/tonel:v1.0.5
Loading baseline of BaselineOfTonel...
Fetched -> MonticelloTonel-Core-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- github://pharo-vcs/tonel:v1.0.5
Fetched -> MonticelloTonel-FileSystem-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- github://pharo-vcs/tonel:v1.0.5
Loaded -> MonticelloTonel-Core-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- cache
Loaded -> MonticelloTonel-FileSystem-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- cache
...finished baselinelinear load :
linear load : baseline [BaselineOfTonel]
load : MonticelloTonel-Core-cypress.1
load : MonticelloTonel-FileSystem-cypress.1
Inconsistency detected by ld.so: dl-close.c: 811: _dl_close: Assertion
`map->l_init_called' failed!
[Thread 0xf7a7cb40 (LWP 27000) exited]
[Inferior 1 (process 26996) exited with code 0177]
(gdb)




The problem occurrs some time between:


VM 30 Jan 2018 14:55 commit 0d66d2a  (bootstrap succeeds)

and:

VM 31 Jan 2018 14:13 commit 0456605  (bootstrap fails)


These are two consecutive VMs on files.pharo.org, so I can't check
commits in between by downloading, and my attempts to build the VMs
failed.


The commit messages are:


commit 045660598ed0a743d9430b600f3b2ac9cf31d4c4
Merge: b251d57 74e443c
Author: Fabio Niephaus <[hidden email]>
Date:   Wed Jan 31 15:13:15 2018 +0100

    Merge pull request #205 from OpenSmalltalk/dl-openssl

    [SqueakSSL] Overlay OpenSSL for linux/unix

commit 74e443c1e2c1492ee7677471730182393f5ae996
Author: Fabio Niephaus <[hidden email]>
Date:   Wed Jan 31 14:35:45 2018 +0100

    Nit picks [ci skip]

commit 438311cb201cfc01dcb03ddcf034cfb5176b57f4
Author: Tobias Pape <[hidden email]>
Date:   Wed Jan 31 14:27:31 2018 +0100

    Refactor the overlay, comments where due

commit b251d57fec7da2f19945c0555b3847f1e397cd71
Merge: 0d66d2a 66d5790
Author: Fabio Niephaus <[hidden email]>
Date:   Wed Jan 31 10:50:39 2018 +0100

    Merge pull request #206 from estebanlm/add-pharo-certificates

    new try with certificates

commit 66d579096b9d954ed208b621770a937639f58031
Author: Esteban Lorenzano <[hidden email]>
Date:   Wed Jan 31 09:33:11 2018 +0100

    new try

commit 8e48e6f2a94ecac2010550b68c2ceab6cf7cde91
Author: Tobias Pape <[hidden email]>
Date:   Tue Jan 30 21:15:51 2018 +0100

    Allow for possibly undefined X509 constant

commit 49353ed2783531b3c8fcb7a522dc6f578ea5414c
Author: Tobias Pape <[hidden email]>
Date:   Tue Jan 30 20:09:38 2018 +0100

    [SqueakSSL] Overlay OpenSSL for linux/unix

    When we dynamically link against OpenSSL, the bundles are not portable,
    as CentOS and friends use other SO_NAMEs than Debian and friends. Also,
    soft-fallback for later features such as host name verification is hard.

    When we statically link, we might lack behind the OS, the binaries are
    bigger, and the legal situation is less clear.

    So we now support not linking at all but rather lookup all necessary
    functions/symbols at runtime.

    This can be disabled with SQSSL_OPENSSL_LINKED which effectively results
    in the dynamically-linked behavior. (This is preferable for platform
    builds, eg, debs and rpms)



I'm struggling to figure this one out at the moment and am hoping that
one of you might recognise something to help track it down...

P.S. I also tried compiling with -DSQSSL_OPENSSL_LINKED, but that
produced compiler errors.


Help :-)
Alistair

 

cool :(


which?

Best regards
        -Tobias

 
Hi Alistair

I've committed a probable fix in 802de0a1b. Please tell me whether this works.

Best regards
        -Tobias

 
Hi Tobias,

On 12 March 2018 at 11:22, Tobias Pape <[hidden email]> wrote:

c99 -m32 -g3 -O0 -fwrapv -DDEBUGVM=1 -DPharoVM -DIMMUTABILITY=1
-DSQSSL_OPENSSL_LINKED -msse2 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64
-DCOGMTVM=0 -DLSB_FIRST=1 -DHAVE_CONFIG_H
-I/home/alistair/vmmaker/opensmalltalk-vm/build.linux32x86/pharo.cog.spur/build.debug
-I/home/alistair/vmmaker/opensmalltalk-vm/build.linux32x86/pharo.cog.spur/build.debug
-I/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/vm
-I/home/alistair/vmmaker/opensmalltalk-vm/platforms/Cross/vm
-I/home/alistair/vmmaker/opensmalltalk-vm/spursrc/vm
-I/home/alistair/vmmaker/opensmalltalk-vm/platforms/Cross/plugins/SqueakSSL
-I/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL
-c /home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c
 -fPIC -DPIC -DPIC -o sqUnixOpenSSL.o
In file included from
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:4:0:
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/openssl_overlay.h:
In function 'loadLibrary':
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/openssl_overlay.h:140:30:
error: too many arguments to function '0u'
 #define sqo_OPENSSL_init_ssl ((void*(*)(void))NULL)
                              ^
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/openssl_overlay.h:886:7:
note: in expansion of macro 'sqo_OPENSSL_init_ssl'
       sqo_OPENSSL_init_ssl(sqo_OPENSSL_INIT_LOAD_SSL_STRINGS |
       ^
In file included from /usr/include/openssl/crypto.h:129:0,
                 from /usr/include/openssl/comp.h:5,
                 from /usr/include/openssl/ssl.h:149,
                 from
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/openssl_overlay.h:33,
                 from
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:4:
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:
In function 'sqVerifyNameInner':
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:116:39:
error: 'sk_GENERAL_NAME_freefunc' undeclared (first use in this
function)
   sqo_sk_GENERAL_NAME_pop_free(sANs,
(sk_GENERAL_NAME_freefunc)sqo_OPENSSL_sk_free);
                                       ^
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:116:3:
note: in expansion of macro 'sqo_sk_GENERAL_NAME_pop_free'
   sqo_sk_GENERAL_NAME_pop_free(sANs,
(sk_GENERAL_NAME_freefunc)sqo_OPENSSL_sk_free);
   ^
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:116:39:
note: each undeclared identifier is reported only once for each
function it appears in
   sqo_sk_GENERAL_NAME_pop_free(sANs,
(sk_GENERAL_NAME_freefunc)sqo_OPENSSL_sk_free);
                                       ^
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:116:3:
note: in expansion of macro 'sqo_sk_GENERAL_NAME_pop_free'
   sqo_sk_GENERAL_NAME_pop_free(sANs,
(sk_GENERAL_NAME_freefunc)sqo_OPENSSL_sk_free);
   ^
In file included from
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:4:0:
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:
In function 'sqVerifySAN':
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/openssl_overlay.h:149:35:
error: too many arguments to function '0u'
 #define sqo_ASN1_STRING_get0_data ((void*(*)(void))NULL)
                                   ^
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:162:6:
note: in expansion of macro 'sqo_ASN1_STRING_get0_data'
    ? sqo_ASN1_STRING_get0_data(sAN->d.ia5)
      ^
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:
In function 'sqSetupSSL':
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/openssl_overlay.h:138:33:
error: too many arguments to function '0u'
 #define sqo_SSL_CTX_set_options ((void*(*)(void))NULL)
                                 ^
/home/alistair/vmmaker/opensmalltalk-vm/platforms/unix/plugins/SqueakSSL/sqUnixOpenSSL.c:241:2:
note: in expansion of macro 'sqo_SSL_CTX_set_options'
  sqo_SSL_CTX_set_options(ssl->ctx, SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3);
  ^
Makefile:192: recipe for target 'sqUnixOpenSSL.lo' failed
make[1]: *** [sqUnixOpenSSL.lo] Error 1
Makefile:552: recipe for target 'SqueakSSL.la' failed
make: *** [SqueakSSL.la] Error 2
~/vmmaker/opensmalltalk-vm/build.linux32x86/pharo.cog.spur/build.debug
~/vmmaker/opensmalltalk-vm/build.linux32x86/pharo.cog.spur/build.debug
cp -f /home/alistair/vmmaker/opensmalltalk-vm/.thirdparty-cache/linux/i386/lib/libSDL2-2.0.so.0.4.1
/home/alistair/vmmaker/opensmalltalk-vm/products/debug/phcogspurlinuxht/lib/squeak/5.0-201803091840
Installing links
for each in /home/alistair/vmmaker/opensmalltalk-vm/.thirdparty-cache/linux/i386/lib/libSDL2*so*;
do \
if [ -L $each ]; then \
cp -d $each /home/alistair/vmmaker/opensmalltalk-vm/products/debug/phcogspurlinuxht/lib/squeak/5.0-201803091840;
\
fi \
done
DONE


Let me know if you want the complete log file (417K).


> I've committed a probable fix in 802de0a1b. Please tell me whether this works.

Thanks!  I'll let you know.


Cheers,
Alistair

 
Hi Tobias,

On 12 March 2018 at 11:51, Tobias Pape <[hidden email]> wrote:
Yep, that looks good - I was able to run the Pharo 7 bootstrap process
(32 bit) and load up my normal development environment (64 bit).

The Travis CI job succeeded, but nothing appeared on bintray, and the
zip on files.pharo.org is incomplete.  I'll send a separate email
about those.

Thanks!
Alistair

 
Hi Alistair
Thanks.

Can you try compiling 2783b4429 linked? I think that should fix that :)

Best regards
        -Tobias

>
>
>> I've committed a probable fix in 802de0a1b. Please tell me whether this works.
>
> Thanks!  I'll let you know.
>
>
> Cheers,
> Alistair

 
Hi Tobias,

On 12 March 2018 at 21:06, Tobias Pape <[hidden email]> wrote:
Yep, that works fine.

Thanks!
Alistair

 
Hi,

Ok, that's great!

Best regards
        -Tobias