Fwd: [Vm-dev] why GT forked the VM and how to kill our fork

From: Eliot Miranda <[hidden email]>
Date: April 26, 2020 at 10:39:35 AM PDT
To: Open Smalltalk Virtual Machine Development Discussion <[hidden email]>
Subject: Re:  [Vm-dev] why GT forked the VM and how to kill our fork

Hi Doru,

On Apr 26, 2020, at 12:57 AM, Tudor Girba <[hidden email]> wrote:



Hi,

Currently, at feenk we have feenkcom/opensmalltalk-vm:

https://github.com/feenkcom/opensmalltalk-vm

This is a small fork of the headless branch from pharo-project/opensmalltalk-vm that appeared out of practical necessities, but that we would like to avoid having. This post briefly describes the changes in the feenkcom/opensmalltalk-vm repo and the functionality those changes provide for Glamorous Toolkit.

For Glamorous Toolkit we aimed for the following functionality:

  • Open the GUI natively and have native display quality (GUI opened through FFI calls)

  • Have a Glamorous Toolkit app for Mac OS that works as any other apps for Mac OS

  • Create end-user applications that are fully customisable (executable name, menus, etc)

  • Use Github actions for doing all compilations of external libraries and the vm instead of Travis CI.

  • Have Iceberg running in native windows (which requires nested FFI callbacks)

There has been work on these issues in both OpenSmalltalk/opensmalltalk-vm and pharo-project/opensmalltalk-vm but they were not entirely addressed. We needed to have something reliable a few months ago, and forking and doing some quick changes made that possible.

Ideally we want to be able to run Glamorous Toolkit on both OpenSmalltalk/opensmalltalk-vm and pharo-project/opensmalltalk-vm.

To have native GUIs we relied on Ronie Salgado’s work on the headless vm and started with pharo-project/opensmalltalk-vm - headless branch:

https://github.com/pharo-project/opensmalltalk-vm/tree/headless

That provided a solution for opening the GUI from the image through FFI calls. Currently we use Glutin (a library for OpenGL context creation, written in pure Rust) and this made it possible to run the entire Glamorous Toolkit inside a callback.

On macOS when running an app, even a notarized one, the OS warns the user that the app is downloaded from the internet, and the user needs to confirm that they agree. Once the user agrees the app should automatically start. This is not currently possible with Pharo apps (for example PharoLaunched.app) and users have to again run the app manually after giving permission. Also Gatekeeper in macOS runs applications downloaded from zips in a randomized read-only DMG. We do not want this behaviour as users not copying Glamorous Toolkit to the Applications folder on macOS would then experience incorrect application behaviour.

To create end-user applications we also need to fully customize the executable name (what the user sees in the Task Runner/Activity monitor), icons, native menus. Part of this work is already integrated in the pharo-project/opensmalltalk-vm - headless branch (Customizing the OS X icons, Brand the VM executable and package).

Since last year Github offers Github Actions similar to Travis. We found it much easier to use than Travis for external libraries and the vm. Also we get to manage the code and the builds in the same place. This work is already integrated in the pharo-project/opensmalltalk-vm - headless branch (Build the VM under GitHub actions: https://github.com/pharo-project/opensmalltalk-vm/pull/56).

The issues related to running Iceberg is a bit more technical. By moving to the headless vm we are running the entire image computation inside a callback from Glutin (https://github.com/rust-windowing/glutin/). When using Iceberg we get nested callbacks which we could not get to work using Alien. Instead we are using the ThreadedFFI Plugin and running all callback from Iceberg and Glutin using the Threaded FFI plugin (https://github.com/pharo-project/threadedFFI-Plugin). Currently we have a small fork of this plugin (feenkcom/threadedFFI-Plugin) and we also ship a custom plugin with the VM to fix a race condition due to having two copies of the callback stack (a pull request is here: https://github.com/pharo-project/threadedFFI-Plugin/pull/17).

One detail here is that Alien provides the callback machinery used alongside ThreadedFFIPlugin.  The Alien call-out machinery is an old hack used only by Newspeak, that could be eliminated in non-Newspeak builds.  But AFAIA Alien’s callback machinery is *the* callback machinery everywhere for the OpenSmalltalk vm.

While not specific to our environment, openssl1.0 is no longer supported, and we are seeing users who are unable to run Pharo due to version conflicts, as reported in https://github.com/pharo-project/opensmalltalk-vm/issues/62.

To sum up, a fork was the easiest way to get all this running. Now some changes are already in the pharo-project/opensmalltalk-vm - headless branch. What we are still missing are the changes that get the VM to work nicely with Mac OS and a bug fix in ThreadedFFI.

We would also love it to have all these changes integrated in OpenSmalltalk/opensmalltalk-vm in the headless vm. This requires additional coordination as the required changes are somewhat deeper.

Can it be done with pull requests?  Should we try and firm a team of volunteers, eg one from feenk and one from OpenSmalltalk to try and merge in the feenk code?

I would volunteer but I am way too busy getting the Terf system working.  TFir e able, the ARMv8 JIT has been working for months but I’ve had no time to release it because of Terf and other payed work (got to pay the bills).

Please let us know you would prefer to coordinate.

Any volunteers?

Cheers,

Tudor, on behalf of the feenk team

--

feenk.com

"The coherence of a trip is given by the clearness of the goal."