Esteban's ChangeLog week of 6 March 2017

Hello!

This is my weekly ChangeLog, from 6 March 2017 to 12 March 2017.
You can see it in a better format by going here: http://log.smallworks.eu/web/search?from=6/3/2017&to=12/3/2017

ChangeLog
=========

10 March 2017:
--------------

*    === libgit2(ssh) and linux
    An update on the libgit2 (and consequently iceberg) problem on linux: SSH will not work on ITIMER VM (which
    is the zeroconf default, at least for now).

    Here the reason:

    ITIMER sends an signal to the VM thread using +pthread_kill(tickerThread, TICKER_SIGNAL)+ and that aborts
    system calls, which are used by +libssh2+, producing a 'wait for packet' to fail answering a timeout (
    even if that timeout does not happens for real).

    Here is an interesting comment from +sqUnixITimerTickerHeartbeat.c+ :

    ----
    /* While we use SA_RESTART to ensure system calls are restarted, this is
     * not universally effective.  In particular, connect calls can abort if
     * a system call is made in the signal handler, i.e. the pthread_kill in
     * prodHighPriorityThread.  So we avoid this if possible by not prodding
     * the high-priority thread unless there are high-priority tickees as
     * indicated by numAsyncTickees > 0.
     */
    ----

    ... and sadly, as +OSSubprocess+, +libssh2+ calls falls under the category of "not universally effective"
    (they are connect calls... once disconnected, you cannot connect again 'as is').

    So, the only workaround possible (at least for now), is to use the threaded version of VM... I improved the
    error message it throws when this fails, now looks like this:

    ----
    This VM uses a thread heartbeat who requires a special configuration to work.
    You need to allow it to run higher priority threads (real time), to allow clock to work properly
    You need to add a conf file to /etc/security/limits.d, executing this:

    sudo cat >/etc/security/limits.d/%s.conf <<END\n", VMNAME);
    *       hard    rtprio  2
    *       soft    rtprio  2
    END

    You need to log out and log back in for the limits to take effect.
    For more information read https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux
    ----

    which at least shows what's happens and how you can easily fix it.

    Now, I think this is a *sad solution*. Needing to touch system configuration in order to run a VM
    does not feels "professional". I would like to have some time to find a fix.

9 March 2017:
-------------

*    I spent last two days again fighting with linux dependencies, because what I thought was fixed (the
    loading of +libgit2.so+), actually was not working in all conditions.

    Now, I think I finally arrived to a working solution (that involved add an +rpath+ to Pharo, not
    to dependent libraries).

    There is a new +vmLatest60+, let's see how it works...


6 March 2017:
-------------

*    I spent time fixing libgit2 (and dependents) compilation, both for linux and windows.

    As a result, now windows is working again (there was problems after version 0.26)... but linux, who seemed
    working last friday, is now again not working properly :(


cheers!
Esteban

Hi Igor, 

  you are right with the idea of having a "setup" script with the downloadable VM. 

However, I think that the proposed solution, the change in the configuration is not good. It works, but the solution that we are proposing alters the permissions for all the users and processes. It creates a security hole (I don't know how much dangerous it is, but usually the linux configuration does not allow to change the threads to real time priority). 

We should provide a solution using a specific running user for the VM or doing something else if we want the VM to run with special privileges.

I think the professional solution is not to ask to run as root or ask to lower the security of the whole system. 

Thierry, 

    I think the VM that you are looking for is in: http://files.pharo.org/vm/pharo-spur32/linux/latest-threaded.zip 

    However I won't enter in panic, we have only seen problems with LibSSH and OSSubProcesses, basically because depends of the system calls used and if they can be restarted.

Cheers.

On Mon, Mar 13, 2017 at 10:31 AM, Igor Stasenko <[hidden email]> wrote:

On 13 March 2017 at 09:00, <[hidden email]> wrote:

Hello!

This is my weekly ChangeLog, from 6 March 2017 to 12 March 2017.
You can see it in a better format by going here: http://log.smallworks.eu/web/search?from=6/3/2017&to=12/3/2017

ChangeLog
=========

10 March 2017:
--------------

*    === libgit2(ssh) and linux
    An update on the libgit2 (and consequently iceberg) problem on linux: SSH will not work on ITIMER VM (which
    is the zeroconf default, at least for now).

    Here the reason:

    ITIMER sends an signal to the VM thread using +pthread_kill(tickerThread, TICKER_SIGNAL)+ and that aborts
    system calls, which are used by +libssh2+, producing a 'wait for packet' to fail answering a timeout (
    even if that timeout does not happens for real).

    Here is an interesting comment from +sqUnixITimerTickerHeartbeat.c+ :

    ----
    /* While we use SA_RESTART to ensure system calls are restarted, this is
     * not universally effective.  In particular, connect calls can abort if
     * a system call is made in the signal handler, i.e. the pthread_kill in
     * prodHighPriorityThread.  So we avoid this if possible by not prodding
     * the high-priority thread unless there are high-priority tickees as
     * indicated by numAsyncTickees > 0.
     */
    ----

    ... and sadly, as +OSSubprocess+, +libssh2+ calls falls under the category of "not universally effective"
    (they are connect calls... once disconnected, you cannot connect again 'as is').

    So, the only workaround possible (at least for now), is to use the threaded version of VM... I improved the
    error message it throws when this fails, now looks like this:

    ----
    This VM uses a thread heartbeat who requires a special configuration to work.
    You need to allow it to run higher priority threads (real time), to allow clock to work properly
    You need to add a conf file to /etc/security/limits.d, executing this:

    sudo cat >/etc/security/limits.d/%s.conf <<END\n", VMNAME);
    *       hard    rtprio  2
    *       soft    rtprio  2
    END

    You need to log out and log back in for the limits to take effect.
    For more information read https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux
    ----

    which at least shows what's happens and how you can easily fix it.

    Now, I think this is a *sad solution*. Needing to touch system configuration in order to run a VM
    does not feels "professional". I would like to have some time to find a fix.

No, it does feels right, according to common *nix lifestyle :)

No, seriously, it is not an issue for linux to set up configs for installed software , that would require root privileges.
VM is quite a beast, and this is not a surprise, that it requires fine tuning & deep & broader interfacing with OS, comparing to regular software.

Of course, the sad moment is that you need to write down an installation package and put this .conf file there which will install it,
in comparison to usual "download and run" style, this surely looks like a drawback.
But we can do smarter: - add small script to bundle with big letters "RUN ME" , that will do what is should to copy/concat .conf file
for VM binary located at given folder or wherever it does.. and everyone's happy :)
 

9 March 2017:
-------------

*    I spent last two days again fighting with linux dependencies, because what I thought was fixed (the
    loading of +libgit2.so+), actually was not working in all conditions.

    Now, I think I finally arrived to a working solution (that involved add an +rpath+ to Pharo, not
    to dependent libraries).

    There is a new +vmLatest60+, let's see how it works...


6 March 2017:
-------------

*    I spent time fixing libgit2 (and dependents) compilation, both for linux and windows.

    As a result, now windows is working again (there was problems after version 0.26)... but linux, who seemed
    working last friday, is now again not working properly :(


cheers!
Esteban

--

Best regards,
Igor Stasenko.

--

Pablo Tesone.
[hidden email]