Metacello Tonel Support Proposal

Hi everyone.

We are currently working on improving the support for different code-serialization formats in Squeak. This includes Tonel [2], which is used by developers who are working on their projects in the Pharo environment and storing code artifacts on GitHub such as PolyMath [6]. While those projects can be virtually compatible with Squeak, support for the Tonel format is required to actually load the source code -- be it via Monticello, Metacello, or another front-end.

We wrote briefly about our intentions in the rather long discussion about performance on the vm-dev mailing list [4]. You can find the pull request for the necessary changes to Metacello on GitHub [5]. A version of Tonel with Squeak-specific compatibility changes is already available [2].

Please try it out. You can support our efforts by helping us find bugs.

We encourage all of you to load your favorite projects not only in a fresh Trunk image [7] but also in the image you are using anyway -- after making a backup copy :). Please report debuggers, performance drops, or other issues. Of course, do also mention any ideas you have on the topic.

Here is how to test your projects: (1) install our pull request for Metacello and (2) load your project through Metacello.

**Step 1 - Install Metacello from our pull request**

*Note that, in a fresh Squeak image, `Metacello new` will first install the official Metacello version. After that, the following code snippet will update that version with our pull request. That’s why you need an extra `get` and `load.`.*

```Smalltalk
Metacello new
   baseline: 'Metacello';
   repository: 'github://tom95/metacello:squeak-download-repo-split/repository';
   get; load: #default.
```

**Step 2 - Load your project via Metacello**

We can load PolyMath [6] in the following example. Note, however, that not all PolyMath classes can be loaded in Squeak because of several missing classes, which are specific to Pharo’s standard library. Yet, loading starts fine, which indicates a working support of the Tonel format in Squeak. Again, don’t forget to backup your production images.

*Note that loading an already installed project will not trigger the new code paths, we added to Metacello. So, remove those projects from the Metacello registry first or use a fresh Squeak image instead.*

```Smalltalk
Metacello new
   repository: 'github://PolyMathOrg/PolyMath/src';
   baseline: 'PolyMath';
   load.
```

**Further advice**

When you try to load a repository that uses Tonel for the first time, the installer will ask you whether to install Tonel support for Squeak. If you deny this request, you will have to do it three times because of Metacello's internal retry logic.

**Emergency brake:** Should anything out of the ordinary happen, you might easily get back to Metacello’s current loading logic by simply removing the checks for `type = ’github’` and `type=’bitbucket’` at the end of `MetacelloSqueakPlatform>>#createRepository:`.

We -- Marcel (mt), Fabio (fn), Jakob (jr), and Tom (tobe) -- thank you for your support.

Happy testing! :)

-----------------------

**More Bits on Tonel**

Pharo switched to Tonel as their default format a while ago, making it more challenging to maintain cross-dialect support for repositories. There are some projects that strive for compatibility -- such as [1] --, but that entails an extra maintenance effort. Only brave people are willing to go that extra mile from what we've seen.

Tonel is responsible for writing Smalltalk source code into the file system. Entire classes are stored in files, rather than methods as in filetree [8], which also improves performance when storing or loading code artifacts. Considering Git support, metadata (e.g. method timestamps) is not part of the serialization format, which fosters clean merge requests and a reduction of noise throughout the (external/git) commit history. Consequently, Tonel puts less pressure on the operating system’s file system, which can especially be beneficial on Windows-based platforms.

However, since Tonel-based project repositories are omitting Squeak’s method timestamps, existing code-browsing tools would have to be extended to fetch that metadata from the particular Git repository. Otherwise, both authorship and time of last edit wouldn’t be available in Squeak, even though they would be updated through a Git commit/push process -- that is, if Tonel is used in combination with Git, which is not mandatory, considering Monticello’s (not Metacello’s) connection to SqueakSource and the local file system. So, Tonel poses a tricky compatibility challenge on Squeak’s tools. We plan to add computation of timestamps based on the git history to Squot, our Git infrastructure, in the foreseeable future (which not only benefits Tonel, but should also allow us to recover the full version history in metadataless filetree repositories).

A second drawback of Tonel might be limited support for custom parsers, which has been mentioned as a valuable use case by  the Squeak community, if I understand correctly [9].

Our proposed implementation [2] employs Metacello’s platform-specific package concept. While the core of Metacello is shared across Smalltalk dialects, there are packages for Squeak or Pharo. One of the key aspects for each dialect is code loading. We decouple **transport** (e.g., downloading from a git repository) from **deserialization** (e.g., parsing files from Tonel or FileTree and creating Monticello’s MCDefinitions). For this, we propose the `MCFilesystemFetchOnlyRepository`, which couples common fetching logic, as a wrapper around `MCFiletreeRepository` and `TonelRepository`, which specialize parsing respectively.

In the long term, we want to enable and support Squeak compatibility upstream. In the spirit of common open source practice, we therefore forked the original Tonel repository as squeak-tonel [2] (as well as its dependency on Filesystem [10]). Having the pull request for Metacello as described above, this enables us to quickly address compatibility issues in Squeak. However, our goal is to push our contributions upstream. Consequently, any compatibility fixes will be opened as pull requests for review.

Lastly, note that our efforts to support Tonel in Squeak do also entail the support of the Smalltalk Object Notation (STON) for Squeak [3]. STON is vital to identify whether a repository is using Tonel or not when loading projects through Metacello. Any heuristic approach would likely be too error-prone considering Metacello’s expected reliability. Therefore, we added STON as a *mandatory dependency* to Metacello in our pull request. Like squeak-tonel, we want to enable and support Squeak compatibility upstream for squeak-ston [3] in the long term.

[1] https://github.com/ronsaldo/abstract-gpu
[2] https://github.com/squeak-smalltalk/squeak-tonel
[3] https://github.com/squeak-smalltalk/squeak-ston
[4] http://forum.world.st/Squeak-and-Pharo-speed-differences-td5116635.html
[5] https://github.com/Metacello/metacello/pull/515
[6] https://github.com/PolyMathOrg/PolyMath
[7] http://files.squeak.org/6.0alpha/
[8] https://github.com/dalehenrich/filetree
[9] http://forum.world.st/Squeak-and-Tonel-td5095284.html#a5095317
[10] https://github.com/squeak-smalltalk/squeak-filesystem

FWIW, I am working on a project called Rowan that is a replacement for Metacello ... Right now I am working on integrating Rowan into the GemStone build process and the plan is that Rowan will be included in the next major release of GemStone ... Like Metacello, Rowan is aimed at multiple dialects of Smalltalk.

All of the project/package metadata is stored external to the packages directory. Here is a very simple example project[1] that has 2 packages[2] RowanSample9-Core and RowanSample9-Tests. The RowanSample-Tests package is conditionally loaded. if you poke around in the rowan directory[1], you will see that the project meta data is being stored on disk in STON format.

Rowan supports both Filetree and Tonel package formats.

There are nearly 60 different sample projects in the RowanSample9 repository[3] each on it's own branch and all of the projects were created and updated using the Rowan API. The Rowan API encompasses not only creating and managing project meta data, but the creation and management of projects/packages/classes/methods.

The "Rowan API" is segregated into two major pieces: disk format and in-image api.

The most important piece of the API is the disk format as that is the key to sharing projects between smalltalk dialects. At a minimum all dialects must be able to share the disk format.

The second component is the in-image api. The in-image api is in roughly three pieces: Rowan definitions (similar to Monticello definitions ... extended to include projects and the in-image representation of the meta data; a loader api which manages the work of installing definitions into the image; and the loaded representation of projects, packages, classes and methods.

I'm imagining that different dialects will choose to support Rowan at different levels. Sharing disk format only; sharing the definition implementation (which would include reading and writing definitions to/from disk); sharing the loader implementation; and sharing the loaded implementation.

Esteban, Mariano and I talked in December/January about incorporating  sub applications, etc. into Rowan, but we were all under time constraints at the time and it seems that incorporating the Envy meta data was going to take a little bit of work. Rowan has the notion of `components` which are very similar to subapplications, but subapplications are more restrictive than components[4] and there was no opportunity for directly mapping between the two --- without spending a bit of time that we didn't have ... with that said I am optimistic that Envy metadata can be incorporated.

The fact that Rowan serializes objects to disk in STON format, means that we can have quite a bit of latitude in what how we shape the metadata that is stored on disk --- which means that if push comes to shove, Envy style projects may use a different set of classes than the "standard Rowan project (akin to Metacello/Monticello projects)". Of course, the Envy style projects should be loadable and writable on all supported platforms ...

I am still a couple months away from being done with V2.0 of Rowan (Rowan V1.0 went into production at one of our customer's sites 1.5 years ago) and I will have other tasks related to the next release of GemStone, so I can't forecast when I will start porting Rowan to other platforms, but that is my intention.

In the mean time, if there are folks interested in helping with the port to other platforms, please drop me a line and we can arrange a chat :)

Dale

On Fri, May 29, 2020 at 4:34 PM Jakob Reschke <[hidden email]> wrote:

Hi Mariano,

That is interesting to hear, thank you!

What is VA Smalltalk's stance about Tonel interoperability between the
Smalltalks _in the same project_? For example, if the main developers
use VA Smalltalk, there will be the additional ENVY properties for
SubApplications and such in the meta files. But when someone loads the
project in Squeak or Pharo and writes another version, this will not
include these additional properties. So if this person created a pull
request, merging it without manual polishing will "destroy" the
additional setup for VA Smalltalk.

I am asking because we are facing similar problems between Squeak and
Pharo, although they are still much closer to one another than VA
Smalltalk is to any of them.

Yes, that would be the current situation. Although, if you can live with certain constrains (like not having subapps, loosing whether a method is public or private, etc etc), then its much closer to be interoperable. You can read more here: https://github.com/instantiations/tonel-vast#compatibility-recommendations

Also, we have an open  issue to address the possibility of storing the VA specific metadata in a separate place: https://github.com/instantiations/tonel-vast/issues/49

But we have a bunch of other higher priority things first. 

Hope this helps. 

--

Mariano Martinez Peck

Email: [hidden email]

Twitter: @MartinezPeck

LinkedIn: www.linkedin.com/in/mariano-martinez-peck

Blog: https://marianopeck.wordpress.com/