Overview of the OpenSmalltalk VM and its build process

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Overview of the OpenSmalltalk VM and its build process

raffaello.giulietti
 
Hi,

I'm interested in understanding the overall OpenSmalltalk VM structure
and design and its build process.

I read [1] and [2] to get a high level picture of the vm and its
interesting strategy to development (Slang, VMMaker, etc.)

Apart from these, are there other useful documents/sites/tutorials on
the vm development process in general and about the underlying ideas in
the build process? I'm aware that vm construction is a rather special
topic but any useful reference or hint is welcome.

I'm already familiar with the interesting [3] and [4] websites and their
authors papers, talks and slides, but these do not seem to help much in
clarifying the development process itself.

In particular, I'm targeting Pharo 64 bit/Win 64 bit (x64-86). I hope to
be able someday to actively participate in bringing Cog/Spur/Sista to a
more mature status on this platform.

The questions I would like to find an answer to are of the following kind:
* Where should I start?
* Should I use Squaek or can I use Pharo for vm development purposes?
* What are the restrictions of Slang?
* Where does VMMaker live?
* Which C toolchain shall I use?
* Which release of gcc, ld, make are known to work?
* What about clang?
* Which version of cygwin64 or mingw64 are OK?
* etc.



Greetings
Raffaello

---

[1] http://www.vpri.org/pdf/tr1997001_backto.pdf
[2] http://design.cs.iastate.edu/vmil/2011/papers/p03-miranda.pdf
[3] http://www.mirandabanda.org/cogblog/
[4] https://clementbera.wordpress.com/
Reply | Threaded
Open this post in threaded view
|

Re: Overview of the OpenSmalltalk VM and its build process

Clément Bera-4
 
Hi Raffaello,

Answers inlined...

On Tue, Oct 17, 2017 at 3:07 PM, Raffaello Giulietti <[hidden email]> wrote:

Hi,

I'm interested in understanding the overall OpenSmalltalk VM structure and design and its build process.

I read [1] and [2] to get a high level picture of the vm and its interesting strategy to development (Slang, VMMaker, etc.)

Apart from these, are there other useful documents/sites/tutorials on the vm development process in general and about the underlying ideas in the build process? I'm aware that vm construction is a rather special topic but any useful reference or hint is welcome.

I'm already familiar with the interesting [3] and [4] websites and their authors papers, talks and slides, but these do not seem to help much in clarifying the development process itself.

In particular, I'm targeting Pharo 64 bit/Win 64 bit (x64-86). I hope to be able someday to actively participate in bringing Cog/Spur/Sista to a more mature status on this platform.

The questions I would like to find an answer to are of the following kind:
* Where should I start?

A good start would be to load the Squeak VMMaker dev image (See here: http://www.mirandabanda.org/cogblog/build-image/) and read the different opened workspaces. In addition, ... 

Compilation toolchain

To check you have everything installed to compile, you can go to the one of the build folder (One of the windows build folder if you're on windows, for example build.Win64x64, in most case we don't recommend cross compilation), read the HowToBuild file, then go to one of the inner folder and use the mvm script to compile a VM from pre-generated C sources.

If it does not work, check on the openSmalltalk-VM repo how the VM is built using Travis CI and set-up your machine similarly to the Travis slaves.

Changes in Slang (GC/Interpreter/JIT/...)

In general when you want to change something in the GC / Interpreter the process is as follow:

1) Make your changes work with the StackVM (interpreter-only) simulator
2) Make your changes work with the compiled StackVM (interpreter-only) 
3) Make your changes work with the CogVM simulator
4) Make your changes work with the compiled CogVM

In general when you want to change something in the JIT the process is as follow:

1) Make your changes generate correct in-image machine code
2) Make your changes work with the CogVM simulator
3) Make your changes work with the compiled CogVM

So a good place to start is to have the simulators working. You could use this to help:

Once the simulator works with your changes, you can generate C sources with something like "VMMaker generateConfiguration"

Changes in Platform code

Go to the platform folder at the root and change the C files with a text editor
 
* Should I use Squaek or can I use Pharo for vm development purposes?

Currently I still use Squeak but I plan to migrate to Pharo. I'd say it's still a bit easier from Squeak due to package compatibility making it hard to commit from Pharo without uncommitting something else, some scripts are also present by default in the Squeak dev image, but everything should work on both environments. No big deal in using one of the other.
 
* What are the restrictions of Slang?

...
As long as the VM compiles and run, this is correct Slang code, else it's not.
In general you should just think in C and write Smalltalk instead. 
I don't know what to say. This is not dynamic at all. No dictionaries, no look-up, no nothing. Even arrays look like Smalltalk arrays but like C arrays you don't have the actual size.
Blocks work as long as they're inlined and removed when performing inlining in the Slang to C compiler.

* Where does VMMaker live?

Half there:
Accessible from git

Half there:
Accessible from Monticello

* Which C toolchain shall I use? 
* Which release of gcc, ld, make are known to work?
* What about clang?

Depends on the platform. The mvm script normally chooses for you. 

All VMs are built from Travis CI so you can check how to set up your machine from the Travis set-up.

On linux it's gcc by default. All recent gcc versions should work since 4.4
On Mac, gcc is not supported anymore, so clang is used. I've used many different versions of clang and they all worked.

Specific compiler versions that don't work are mentioned in the HowToBuild file in the platform folders (if any).
 
* Which version of cygwin64 or mingw64 are OK?

Arf. Both worked at some point, I don't know what the default script use right now I would start with the version used by the mvm script / the Travis build.

 
* etc.



Greetings
Raffaello

---

[1] http://www.vpri.org/pdf/tr1997001_backto.pdf
[2] http://design.cs.iastate.edu/vmil/2011/papers/p03-miranda.pdf
[3] http://www.mirandabanda.org/cogblog/
[4] https://clementbera.wordpress.com/



--
Clément Béra
Pharo consortium engineer
Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq
Reply | Threaded
Open this post in threaded view
|

Re: Overview of the OpenSmalltalk VM and its build process

Nicolas Cellier
 


2017-10-17 16:00 GMT+02:00 Clément Bera <[hidden email]>:
 
Hi Raffaello,

Answers inlined...

On Tue, Oct 17, 2017 at 3:07 PM, Raffaello Giulietti <[hidden email]> wrote:

Hi,

I'm interested in understanding the overall OpenSmalltalk VM structure and design and its build process.

I read [1] and [2] to get a high level picture of the vm and its interesting strategy to development (Slang, VMMaker, etc.)

Apart from these, are there other useful documents/sites/tutorials on the vm development process in general and about the underlying ideas in the build process? I'm aware that vm construction is a rather special topic but any useful reference or hint is welcome.

I'm already familiar with the interesting [3] and [4] websites and their authors papers, talks and slides, but these do not seem to help much in clarifying the development process itself.

In particular, I'm targeting Pharo 64 bit/Win 64 bit (x64-86). I hope to be able someday to actively participate in bringing Cog/Spur/Sista to a more mature status on this platform.

The questions I would like to find an answer to are of the following kind:
* Where should I start?

A good start would be to load the Squeak VMMaker dev image (See here: http://www.mirandabanda.org/cogblog/build-image/) and read the different opened workspaces. In addition, ... 

Compilation toolchain

To check you have everything installed to compile, you can go to the one of the build folder (One of the windows build folder if you're on windows, for example build.Win64x64, in most case we don't recommend cross compilation), read the HowToBuild file, then go to one of the inner folder and use the mvm script to compile a VM from pre-generated C sources.

See below though, on windows, we target mingw so it's kind of cross-compilation anyway.


If it does not work, check on the openSmalltalk-VM repo how the VM is built using Travis CI and set-up your machine similarly to the Travis slaves.

Changes in Slang (GC/Interpreter/JIT/...)

In general when you want to change something in the GC / Interpreter the process is as follow:

1) Make your changes work with the StackVM (interpreter-only) simulator
2) Make your changes work with the compiled StackVM (interpreter-only) 
3) Make your changes work with the CogVM simulator
4) Make your changes work with the compiled CogVM

In general when you want to change something in the JIT the process is as follow:

1) Make your changes generate correct in-image machine code
2) Make your changes work with the CogVM simulator
3) Make your changes work with the compiled CogVM

So a good place to start is to have the simulators working. You could use this to help:

Once the simulator works with your changes, you can generate C sources with something like "VMMaker generateConfiguration"

Changes in Platform code

Go to the platform folder at the root and change the C files with a text editor
 
* Should I use Squaek or can I use Pharo for vm development purposes?

Currently I still use Squeak but I plan to migrate to Pharo. I'd say it's still a bit easier from Squeak due to package compatibility making it hard to commit from Pharo without uncommitting something else, some scripts are also present by default in the Squeak dev image, but everything should work on both environments. No big deal in using one of the other.
 
* What are the restrictions of Slang?

...
As long as the VM compiles and run, this is correct Slang code, else it's not.
In general you should just think in C and write Smalltalk instead. 
I don't know what to say. This is not dynamic at all. No dictionaries, no look-up, no nothing. Even arrays look like Smalltalk arrays but like C arrays you don't have the actual size.
Blocks work as long as they're inlined and removed when performing inlining in the Slang to C compiler.

* Where does VMMaker live?

Half there:
Accessible from git

Half there:
Accessible from Monticello

* Which C toolchain shall I use? 
* Which release of gcc, ld, make are known to work?
* What about clang?

Depends on the platform. The mvm script normally chooses for you. 

All VMs are built from Travis CI so you can check how to set up your machine from the Travis set-up.

On linux it's gcc by default. All recent gcc versions should work since 4.4
On Mac, gcc is not supported anymore, so clang is used. I've used many different versions of clang and they all worked.

Specific compiler versions that don't work are mentioned in the HowToBuild file in the platform folders (if any).
 
* Which version of cygwin64 or mingw64 are OK?

Arf. Both worked at some point, I don't know what the default script use right now I would start with the version used by the mvm script / the Travis build.


gcc is broken for win64 (or at least not satisfying our expectations about stack management)
So we have to use clang.
For this reason, we prefer to compile thru cygwin which has prebuilt (downlodable) clang modules.
It's much more difficult to get a working clang in a mingw environment, so this brand is not maintained anymore.

But the target is a mingw target, not a cygwin target (we don't want VM to depend on cygwin layer, we want to stay minimal).
So we use a cross compiler. See the line

TOOLPREFIX:=x86_64-w64-mingw32-


Using cross-compiler means that we can eventually build a x86_64 .exe from a cygwin32.
But cygwin32 does not provide a debugger (gdb) for x86_64, that's why we prefer developing from a cygwin64 environment.

Eventually, I have a hand-assembled visual studio project for compiling the vm with MSVC (2015 if I remember...)
Of course that kind of project should better be generated (as was possible via cmake) rather than hand-assembled...
It's absolutely cumbersome to maintain such .vc(x)proj across several versions of MSVC.
Anyway, it does not produce working VM yet.
I retried with 2017 but encountered a code generation bug.

We could try to adapt the gnu makefiles to use MSVC thru command line, but IMO the main interest of MSVC is to perform complementary analysis thru IDE.




--
Clément Béra
Pharo consortium engineer
Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq


Reply | Threaded
Open this post in threaded view
|

Re: Overview of the OpenSmalltalk VM and its build process

Nicolas Cellier
 


2017-10-17 19:03 GMT+02:00 Nicolas Cellier <[hidden email]>:


2017-10-17 16:00 GMT+02:00 Clément Bera <[hidden email]>:
 
Hi Raffaello,

Answers inlined...

On Tue, Oct 17, 2017 at 3:07 PM, Raffaello Giulietti <[hidden email]> wrote:

Hi,

I'm interested in understanding the overall OpenSmalltalk VM structure and design and its build process.

I read [1] and [2] to get a high level picture of the vm and its interesting strategy to development (Slang, VMMaker, etc.)

Apart from these, are there other useful documents/sites/tutorials on the vm development process in general and about the underlying ideas in the build process? I'm aware that vm construction is a rather special topic but any useful reference or hint is welcome.

I'm already familiar with the interesting [3] and [4] websites and their authors papers, talks and slides, but these do not seem to help much in clarifying the development process itself.

In particular, I'm targeting Pharo 64 bit/Win 64 bit (x64-86). I hope to be able someday to actively participate in bringing Cog/Spur/Sista to a more mature status on this platform.

The questions I would like to find an answer to are of the following kind:
* Where should I start?

A good start would be to load the Squeak VMMaker dev image (See here: http://www.mirandabanda.org/cogblog/build-image/) and read the different opened workspaces. In addition, ... 

Compilation toolchain

To check you have everything installed to compile, you can go to the one of the build folder (One of the windows build folder if you're on windows, for example build.Win64x64, in most case we don't recommend cross compilation), read the HowToBuild file, then go to one of the inner folder and use the mvm script to compile a VM from pre-generated C sources.

See below though, on windows, we target mingw so it's kind of cross-compilation anyway.


If it does not work, check on the openSmalltalk-VM repo how the VM is built using Travis CI and set-up your machine similarly to the Travis slaves.

Changes in Slang (GC/Interpreter/JIT/...)

In general when you want to change something in the GC / Interpreter the process is as follow:

1) Make your changes work with the StackVM (interpreter-only) simulator
2) Make your changes work with the compiled StackVM (interpreter-only) 
3) Make your changes work with the CogVM simulator
4) Make your changes work with the compiled CogVM

In general when you want to change something in the JIT the process is as follow:

1) Make your changes generate correct in-image machine code
2) Make your changes work with the CogVM simulator
3) Make your changes work with the compiled CogVM

So a good place to start is to have the simulators working. You could use this to help:

Once the simulator works with your changes, you can generate C sources with something like "VMMaker generateConfiguration"

Changes in Platform code

Go to the platform folder at the root and change the C files with a text editor
 
* Should I use Squaek or can I use Pharo for vm development purposes?

Currently I still use Squeak but I plan to migrate to Pharo. I'd say it's still a bit easier from Squeak due to package compatibility making it hard to commit from Pharo without uncommitting something else, some scripts are also present by default in the Squeak dev image, but everything should work on both environments. No big deal in using one of the other.
 
* What are the restrictions of Slang?

...
As long as the VM compiles and run, this is correct Slang code, else it's not.
In general you should just think in C and write Smalltalk instead. 
I don't know what to say. This is not dynamic at all. No dictionaries, no look-up, no nothing. Even arrays look like Smalltalk arrays but like C arrays you don't have the actual size.
Blocks work as long as they're inlined and removed when performing inlining in the Slang to C compiler.

* Where does VMMaker live?

Half there:
Accessible from git

Half there:
Accessible from Monticello

* Which C toolchain shall I use? 
* Which release of gcc, ld, make are known to work?
* What about clang?

Depends on the platform. The mvm script normally chooses for you. 

All VMs are built from Travis CI so you can check how to set up your machine from the Travis set-up.

On linux it's gcc by default. All recent gcc versions should work since 4.4
On Mac, gcc is not supported anymore, so clang is used. I've used many different versions of clang and they all worked.

Specific compiler versions that don't work are mentioned in the HowToBuild file in the platform folders (if any).
 
* Which version of cygwin64 or mingw64 are OK?

Arf. Both worked at some point, I don't know what the default script use right now I would start with the version used by the mvm script / the Travis build.


gcc is broken for win64 (or at least not satisfying our expectations about stack management)
So we have to use clang.
For this reason, we prefer to compile thru cygwin which has prebuilt (downlodable) clang modules.
It's much more difficult to get a working clang in a mingw environment, so this brand is not maintained anymore.


Ah, and there was another grief with mingw...
Currently the VM still uses DirectX-9.
cygwin provides support for this (maybe it's not very future proof now...)
mingw status is unclear, and it seems that recent distributions lack this support...


But the target is a mingw target, not a cygwin target (we don't want VM to depend on cygwin layer, we want to stay minimal).
So we use a cross compiler. See the line

TOOLPREFIX:=x86_64-w64-mingw32-


Using cross-compiler means that we can eventually build a x86_64 .exe from a cygwin32.
But cygwin32 does not provide a debugger (gdb) for x86_64, that's why we prefer developing from a cygwin64 environment.

Eventually, I have a hand-assembled visual studio project for compiling the vm with MSVC (2015 if I remember...)
Of course that kind of project should better be generated (as was possible via cmake) rather than hand-assembled...
It's absolutely cumbersome to maintain such .vc(x)proj across several versions of MSVC.
Anyway, it does not produce working VM yet.
I retried with 2017 but encountered a code generation bug.

We could try to adapt the gnu makefiles to use MSVC thru command line, but IMO the main interest of MSVC is to perform complementary analysis thru IDE.




--
Clément Béra
Pharo consortium engineer
Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq



Reply | Threaded
Open this post in threaded view
|

Re: Overview of the OpenSmalltalk VM and its build process

raffaello.giulietti
 
Hi Clément and Nicolas,

thanks for the useful hints and background information.




On 17/10/17 19:22, Nicolas Cellier wrote:

>  
>
>
>
>
>
> 2017-10-17 19:03 GMT+02:00 Nicolas Cellier
> <[hidden email]
> <mailto:[hidden email]>>:
>
>
>
>     2017-10-17 16:00 GMT+02:00 Clément Bera <[hidden email]
>     <mailto:[hidden email]>>:
>
>          
>         Hi Raffaello,
>
>         Answers inlined...
>
>         On Tue, Oct 17, 2017 at 3:07 PM, Raffaello Giulietti
>         <[hidden email]
>         <mailto:[hidden email]>> wrote:
>
>
>             Hi,
>
>             I'm interested in understanding the overall OpenSmalltalk VM
>             structure and design and its build process.
>
>             I read [1] and [2] to get a high level picture of the vm and
>             its interesting strategy to development (Slang, VMMaker, etc.)
>
>             Apart from these, are there other useful
>             documents/sites/tutorials on the vm development process in
>             general and about the underlying ideas in the build process?
>             I'm aware that vm construction is a rather special topic but
>             any useful reference or hint is welcome.
>
>             I'm already familiar with the interesting [3] and [4]
>             websites and their authors papers, talks and slides, but
>             these do not seem to help much in clarifying the development
>             process itself.
>
>             In particular, I'm targeting Pharo 64 bit/Win 64 bit
>             (x64-86). I hope to be able someday to actively participate
>             in bringing Cog/Spur/Sista to a more mature status on this
>             platform.
>
>             The questions I would like to find an answer to are of the
>             following kind:
>             * Where should I start?
>
>
>         A good start would be to load the Squeak VMMaker dev image (See
>         here: http://www.mirandabanda.org/cogblog/build-image/
>         <http://www.mirandabanda.org/cogblog/build-image/>) and read the
>         different opened workspaces. In addition, ... 
>
>         *Compilation toolchain*
>
>         To check you have everything installed to compile, you can go to
>         the one of the build folder (One of the windows build folder if
>         you're on windows, for example build.Win64x64, in most case we
>         don't recommend cross compilation), read the HowToBuild file,
>         then go to one of the inner folder and use the mvm script to
>         compile a VM from pre-generated C sources.
>
>
>     See below though, on windows, we target mingw so it's kind of
>     cross-compilation anyway.
>
>
>         If it does not work, check on the openSmalltalk-VM repo how the
>         VM is built using Travis CI and set-up your machine similarly to
>         the Travis slaves.
>
>         *Changes in Slang (GC/Interpreter/JIT/...)*
>
>         In general when you want to change something in the GC /
>         Interpreter the process is as follow:
>
>         1) Make your changes work with the StackVM (interpreter-only)
>         simulator
>         2) Make your changes work with the compiled StackVM
>         (interpreter-only) 
>         3) Make your changes work with the CogVM simulator
>         4) Make your changes work with the compiled CogVM
>
>         In general when you want to change something in the JIT the
>         process is as follow:
>
>         1) Make your changes generate correct in-image machine code
>         2) Make your changes work with the CogVM simulator
>         3) Make your changes work with the compiled CogVM
>
>         So a good place to start is to have the simulators working. You
>         could use this to help:
>         https://clementbera.wordpress.com/2016/05/30/simulating-the-cog-vm/
>         <https://clementbera.wordpress.com/2016/05/30/simulating-the-cog-vm/>
>
>         Once the simulator works with your changes, you can generate C
>         sources with something like "VMMaker generateConfiguration"
>
>         *Changes in Platform code*
>         *
>         *
>         Go to the platform folder at the root and change the C files
>         with a text editor
>          
>
>             * Should I use Squaek or can I use Pharo for vm development
>             purposes?
>
>
>         Currently I still use Squeak but I plan to migrate to Pharo. I'd
>         say it's still a bit easier from Squeak due to package
>         compatibility making it hard to commit from Pharo without
>         uncommitting something else, some scripts are also present by
>         default in the Squeak dev image, but everything should work on
>         both environments. No big deal in using one of the other.
>          
>
>             * What are the restrictions of Slang?
>
>
>         ...
>         As long as the VM compiles and run, this is correct Slang code,
>         else it's not.
>         In general you should just think in C and write Smalltalk instead. 
>         I don't know what to say. This is not dynamic at all. No
>         dictionaries, no look-up, no nothing. Even arrays look like
>         Smalltalk arrays but like C arrays you don't have the actual size.
>         Blocks work as long as they're inlined and removed when
>         performing inlining in the Slang to C compiler.
>
>             * Where does VMMaker live?
>
>
>         Half there:
>         https://github.com/OpenSmalltalk/opensmalltalk-vm
>         <https://github.com/OpenSmalltalk/opensmalltalk-vm>
>         Accessible from git
>
>         Half there:
>         http://source.squeak.org/VMMaker/
>         <http://source.squeak.org/VMMaker/>
>         Accessible from Monticello
>
>             * Which C toolchain shall I use? 
>
>             * Which release of gcc, ld, make are known to work?
>             * What about clang?
>
>
>         Depends on the platform. The mvm script normally chooses for you. 
>
>         All VMs are built from Travis CI so you can check how to set up
>         your machine from the Travis set-up.
>
>         On linux it's gcc by default. All recent gcc versions should
>         work since 4.4
>         On Mac, gcc is not supported anymore, so clang is used. I've
>         used many different versions of clang and they all worked.
>
>         Specific compiler versions that don't work are mentioned in the
>         HowToBuild file in the platform folders (if any).
>          
>
>             * Which version of cygwin64 or mingw64 are OK?
>
>
>         Arf. Both worked at some point, I don't know what the default
>         script use right now I would start with the version used by the
>         mvm script / the Travis build.
>
>
>     gcc is broken for win64 (or at least not satisfying our expectations
>     about stack management)
>     So we have to use clang.
>     For this reason, we prefer to compile thru cygwin which has prebuilt
>     (downlodable) clang modules.
>     It's much more difficult to get a working clang in a mingw
>     environment, so this brand is not maintained anymore.
>
>
> Ah, and there was another grief with mingw...
> Currently the VM still uses DirectX-9.
> cygwin provides support for this (maybe it's not very future proof now...)
> mingw status is unclear, and it seems that recent distributions lack
> this support...
>
>
>     But the target is a mingw target, not a cygwin target (we don't want
>     VM to depend on cygwin layer, we want to stay minimal).
>     So we use a cross compiler. See the line
>
>     TOOLPREFIX:=x86_64-w64-mingw32-
>
>     in
>     https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/build.win64x64/common/Makefile.tools
>     <https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/build.win64x64/common/Makefile.tools>
>
>     Using cross-compiler means that we can eventually build a x86_64
>     .exe from a cygwin32.
>     But cygwin32 does not provide a debugger (gdb) for x86_64, that's
>     why we prefer developing from a cygwin64 environment.
>
>     Eventually, I have a hand-assembled visual studio project for
>     compiling the vm with MSVC (2015 if I remember...)
>     Of course that kind of project should better be generated (as was
>     possible via cmake) rather than hand-assembled...
>     It's absolutely cumbersome to maintain such .vc(x)proj across
>     several versions of MSVC.
>     Anyway, it does not produce working VM yet.
>     I retried with 2017 but encountered a code generation bug.
>
>     We could try to adapt the gnu makefiles to use MSVC thru command
>     line, but IMO the main interest of MSVC is to perform complementary
>     analysis thru IDE.
>
>          
>
>             * etc.
>
>
>
>             Greetings
>             Raffaello
>
>             ---
>
>             [1] http://www.vpri.org/pdf/tr1997001_backto.pdf
>             <http://www.vpri.org/pdf/tr1997001_backto.pdf>
>             [2]
>             http://design.cs.iastate.edu/vmil/2011/papers/p03-miranda.pdf <http://design.cs.iastate.edu/vmil/2011/papers/p03-miranda.pdf>
>             [3] http://www.mirandabanda.org/cogblog/
>             <http://www.mirandabanda.org/cogblog/>
>             [4] https://clementbera.wordpress.com/
>             <https://clementbera.wordpress.com/>
>
>
>
>
>         --
>         Clément Béra
>         Pharo consortium engineer
>         https://clementbera.wordpress.com/
>         <https://clementbera.wordpress.com/>
>         Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Overview of the OpenSmalltalk VM and its build process

Mariano Martinez Peck
 

On Wed, Oct 18, 2017 at 3:11 AM, <[hidden email]> wrote:

Hi Clément and Nicolas,

thanks for the useful hints and background information.




On 17/10/17 19:22, Nicolas Cellier wrote:
>
>
>
>
>
>
> 2017-10-17 19:03 GMT+02:00 Nicolas Cellier
> <[hidden email]
> <mailto:[hidden email]>>:
>
>
>
>     2017-10-17 16:00 GMT+02:00 Clément Bera <[hidden email]
>     <mailto:[hidden email]>>:
>
>          
>         Hi Raffaello,
>
>         Answers inlined...
>
>         On Tue, Oct 17, 2017 at 3:07 PM, Raffaello Giulietti
>         <[hidden email]
>         <mailto:[hidden email]>> wrote:
>
>
>             Hi,
>
>             I'm interested in understanding the overall OpenSmalltalk VM
>             structure and design and its build process.
>
>             I read [1] and [2] to get a high level picture of the vm and
>             its interesting strategy to development (Slang, VMMaker, etc.)
>
>             Apart from these, are there other useful
>             documents/sites/tutorials on the vm development process in
>             general and about the underlying ideas in the build process?
>             I'm aware that vm construction is a rather special topic but
>             any useful reference or hint is welcome.
>
>             I'm already familiar with the interesting [3] and [4]
>             websites and their authors papers, talks and slides, but
>             these do not seem to help much in clarifying the development
>             process itself.
>
>             In particular, I'm targeting Pharo 64 bit/Win 64 bit
>             (x64-86). I hope to be able someday to actively participate
>             in bringing Cog/Spur/Sista to a more mature status on this
>             platform.
>
>             The questions I would like to find an answer to are of the
>             following kind:
>             * Where should I start?
>
>
>         A good start would be to load the Squeak VMMaker dev image (See
>         here: http://www.mirandabanda.org/cogblog/build-image/
>         <http://www.mirandabanda.org/cogblog/build-image/>) and read the
>         different opened workspaces. In addition, ... 
>
>         *Compilation toolchain*
>
>         To check you have everything installed to compile, you can go to
>         the one of the build folder (One of the windows build folder if
>         you're on windows, for example build.Win64x64, in most case we
>         don't recommend cross compilation), read the HowToBuild file,
>         then go to one of the inner folder and use the mvm script to
>         compile a VM from pre-generated C sources.
>
>
>     See below though, on windows, we target mingw so it's kind of
>     cross-compilation anyway.
>
>
>         If it does not work, check on the openSmalltalk-VM repo how the
>         VM is built using Travis CI and set-up your machine similarly to
>         the Travis slaves.
>
>         *Changes in Slang (GC/Interpreter/JIT/...)*
>
>         In general when you want to change something in the GC /
>         Interpreter the process is as follow:
>
>         1) Make your changes work with the StackVM (interpreter-only)
>         simulator
>         2) Make your changes work with the compiled StackVM
>         (interpreter-only) 
>         3) Make your changes work with the CogVM simulator
>         4) Make your changes work with the compiled CogVM
>
>         In general when you want to change something in the JIT the
>         process is as follow:
>
>         1) Make your changes generate correct in-image machine code
>         2) Make your changes work with the CogVM simulator
>         3) Make your changes work with the compiled CogVM
>
>         So a good place to start is to have the simulators working. You
>         could use this to help:
>         https://clementbera.wordpress.com/2016/05/30/simulating-the-cog-vm/
>         <https://clementbera.wordpress.com/2016/05/30/simulating-the-cog-vm/>
>
>         Once the simulator works with your changes, you can generate C
>         sources with something like "VMMaker generateConfiguration"
>
>         *Changes in Platform code*
>         *
>         *
>         Go to the platform folder at the root and change the C files
>         with a text editor
>          
>
>             * Should I use Squaek or can I use Pharo for vm development
>             purposes?
>
>
>         Currently I still use Squeak but I plan to migrate to Pharo. I'd
>         say it's still a bit easier from Squeak due to package
>         compatibility making it hard to commit from Pharo without
>         uncommitting something else, some scripts are also present by
>         default in the Squeak dev image, but everything should work on
>         both environments. No big deal in using one of the other.
>          
>
>             * What are the restrictions of Slang?
>
>
>         ...
>         As long as the VM compiles and run, this is correct Slang code,
>         else it's not.
>         In general you should just think in C and write Smalltalk instead. 
>         I don't know what to say. This is not dynamic at all. No
>         dictionaries, no look-up, no nothing. Even arrays look like
>         Smalltalk arrays but like C arrays you don't have the actual size.
>         Blocks work as long as they're inlined and removed when
>         performing inlining in the Slang to C compiler.
>
>             * Where does VMMaker live?
>
>
>         Half there:
>         https://github.com/OpenSmalltalk/opensmalltalk-vm
>         <https://github.com/OpenSmalltalk/opensmalltalk-vm>
>         Accessible from git
>
>         Half there:
>         http://source.squeak.org/VMMaker/
>         <http://source.squeak.org/VMMaker/>
>         Accessible from Monticello
>
>             * Which C toolchain shall I use? 
>
>             * Which release of gcc, ld, make are known to work?
>             * What about clang?
>
>
>         Depends on the platform. The mvm script normally chooses for you. 
>
>         All VMs are built from Travis CI so you can check how to set up
>         your machine from the Travis set-up.
>
>         On linux it's gcc by default. All recent gcc versions should
>         work since 4.4
>         On Mac, gcc is not supported anymore, so clang is used. I've
>         used many different versions of clang and they all worked.
>
>         Specific compiler versions that don't work are mentioned in the
>         HowToBuild file in the platform folders (if any).
>          
>
>             * Which version of cygwin64 or mingw64 are OK?
>
>
>         Arf. Both worked at some point, I don't know what the default
>         script use right now I would start with the version used by the
>         mvm script / the Travis build.
>
>
>     gcc is broken for win64 (or at least not satisfying our expectations
>     about stack management)
>     So we have to use clang.
>     For this reason, we prefer to compile thru cygwin which has prebuilt
>     (downlodable) clang modules.
>     It's much more difficult to get a working clang in a mingw
>     environment, so this brand is not maintained anymore.
>
>
> Ah, and there was another grief with mingw...
> Currently the VM still uses DirectX-9.
> cygwin provides support for this (maybe it's not very future proof now...)
> mingw status is unclear, and it seems that recent distributions lack
> this support...
>
>
>     But the target is a mingw target, not a cygwin target (we don't want
>     VM to depend on cygwin layer, we want to stay minimal).
>     So we use a cross compiler. See the line
>
>     TOOLPREFIX:=x86_64-w64-mingw32-
>
>     in
>     https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/build.win64x64/common/Makefile.tools
>     <https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/build.win64x64/common/Makefile.tools>
>
>     Using cross-compiler means that we can eventually build a x86_64
>     .exe from a cygwin32.
>     But cygwin32 does not provide a debugger (gdb) for x86_64, that's
>     why we prefer developing from a cygwin64 environment.
>
>     Eventually, I have a hand-assembled visual studio project for
>     compiling the vm with MSVC (2015 if I remember...)
>     Of course that kind of project should better be generated (as was
>     possible via cmake) rather than hand-assembled...
>     It's absolutely cumbersome to maintain such .vc(x)proj across
>     several versions of MSVC.
>     Anyway, it does not produce working VM yet.
>     I retried with 2017 but encountered a code generation bug.
>
>     We could try to adapt the gnu makefiles to use MSVC thru command
>     line, but IMO the main interest of MSVC is to perform complementary
>     analysis thru IDE.
>
>          
>
>             * etc.
>
>
>
>             Greetings
>             Raffaello
>
>             ---
>
>             [1] http://www.vpri.org/pdf/tr1997001_backto.pdf
>             <http://www.vpri.org/pdf/tr1997001_backto.pdf>
>             [2]
>             http://design.cs.iastate.edu/vmil/2011/papers/p03-miranda.pdf <http://design.cs.iastate.edu/vmil/2011/papers/p03-miranda.pdf>
>             [3] http://www.mirandabanda.org/cogblog/
>             <http://www.mirandabanda.org/cogblog/>
>             [4] https://clementbera.wordpress.com/
>             <https://clementbera.wordpress.com/>
>
>
>
>
>         --
>         Clément Béra
>         Pharo consortium engineer
>         https://clementbera.wordpress.com/
>         <https://clementbera.wordpress.com/>
>         Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq
>
>
>




--
Reply | Threaded
Open this post in threaded view
|

Re: Overview of the OpenSmalltalk VM and its build process

raffaello.giulietti
 
Thanks Mariano, I'll soon take a look at your promising blogs.




On 2017-10-18 13:37, Mariano Martinez Peck wrote:

>  
>
>
>
> It's a bit outdated and aimed for newbies but you can see my posts about
> the VM:
>
> https://marianopeck.wordpress.com/2011/04/28/the-first-part-of-the-journey-is-over/
> https://marianopeck.wordpress.com/2011/07/18/the-second-part-of-the-journey-is-over/
>
> Cheers,
>
>
>
> On Wed, Oct 18, 2017 at 3:11 AM, <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>     Hi Clément and Nicolas,
>
>     thanks for the useful hints and background information.
>
>
>
>
>     On 17/10/17 19:22, Nicolas Cellier wrote:
>     >
>     >
>     >
>     >
>     >
>     >
>     > 2017-10-17 19:03 GMT+02:00 Nicolas Cellier
>     > <[hidden email]
>     <mailto:[hidden email]>
>     > <mailto:[hidden email]
>     <mailto:[hidden email]>>>:
>     >
>     >
>     >
>     >     2017-10-17 16:00 GMT+02:00 Clément Bera
>     <[hidden email] <mailto:[hidden email]>
>     >     <mailto:[hidden email] <mailto:[hidden email]>>>:
>     >
>     >          
>     >         Hi Raffaello,
>     >
>     >         Answers inlined...
>     >
>     >         On Tue, Oct 17, 2017 at 3:07 PM, Raffaello Giulietti
>     >         <[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>> wrote:
>     >
>     >
>     >             Hi,
>     >
>     >             I'm interested in understanding the overall OpenSmalltalk VM
>     >             structure and design and its build process.
>     >
>     >             I read [1] and [2] to get a high level picture of the vm and
>     >             its interesting strategy to development (Slang, VMMaker, etc.)
>     >
>     >             Apart from these, are there other useful
>     >             documents/sites/tutorials on the vm development process in
>     >             general and about the underlying ideas in the build process?
>     >             I'm aware that vm construction is a rather special topic but
>     >             any useful reference or hint is welcome.
>     >
>     >             I'm already familiar with the interesting [3] and [4]
>     >             websites and their authors papers, talks and slides, but
>     >             these do not seem to help much in clarifying the development
>     >             process itself.
>     >
>     >             In particular, I'm targeting Pharo 64 bit/Win 64 bit
>     >             (x64-86). I hope to be able someday to actively participate
>     >             in bringing Cog/Spur/Sista to a more mature status on this
>     >             platform.
>     >
>     >             The questions I would like to find an answer to are of the
>     >             following kind:
>     >             * Where should I start?
>     >
>     >
>     >         A good start would be to load the Squeak VMMaker dev image
>     (See
>     >         here: http://www.mirandabanda.org/cogblog/build-image/
>     <http://www.mirandabanda.org/cogblog/build-image/>
>     >         <http://www.mirandabanda.org/cogblog/build-image/
>     <http://www.mirandabanda.org/cogblog/build-image/>>) and read the
>     >         different opened workspaces. In addition, ... 
>     >
>     >         *Compilation toolchain*
>     >
>     >         To check you have everything installed to compile, you can
>     go to
>     >         the one of the build folder (One of the windows build
>     folder if
>     >         you're on windows, for example build.Win64x64, in most case we
>     >         don't recommend cross compilation), read the HowToBuild file,
>     >         then go to one of the inner folder and use the mvm script to
>     >         compile a VM from pre-generated C sources.
>     >
>     >
>     >     See below though, on windows, we target mingw so it's kind of
>     >     cross-compilation anyway.
>     >
>     >
>     >         If it does not work, check on the openSmalltalk-VM repo
>     how the
>     >         VM is built using Travis CI and set-up your machine
>     similarly to
>     >         the Travis slaves.
>     >
>     >         *Changes in Slang (GC/Interpreter/JIT/...)*
>     >
>     >         In general when you want to change something in the GC /
>     >         Interpreter the process is as follow:
>     >
>     >         1) Make your changes work with the StackVM (interpreter-only)
>     >         simulator
>     >         2) Make your changes work with the compiled StackVM
>     >         (interpreter-only) 
>     >         3) Make your changes work with the CogVM simulator
>     >         4) Make your changes work with the compiled CogVM
>     >
>     >         In general when you want to change something in the JIT the
>     >         process is as follow:
>     >
>     >         1) Make your changes generate correct in-image machine code
>     >         2) Make your changes work with the CogVM simulator
>     >         3) Make your changes work with the compiled CogVM
>     >
>     >         So a good place to start is to have the simulators
>     working. You
>     >         could use this to help:
>     >       
>      https://clementbera.wordpress.com/2016/05/30/simulating-the-cog-vm/
>     <https://clementbera.wordpress.com/2016/05/30/simulating-the-cog-vm/>
>     >       
>      <https://clementbera.wordpress.com/2016/05/30/simulating-the-cog-vm/ <https://clementbera.wordpress.com/2016/05/30/simulating-the-cog-vm/>>
>     >
>     >         Once the simulator works with your changes, you can generate C
>     >         sources with something like "VMMaker generateConfiguration"
>     >
>     >         *Changes in Platform code*
>     >         *
>     >         *
>     >         Go to the platform folder at the root and change the C files
>     >         with a text editor
>     >          
>     >
>     >             * Should I use Squaek or can I use Pharo for vm development
>     >             purposes?
>     >
>     >
>     >         Currently I still use Squeak but I plan to migrate to
>     Pharo. I'd
>     >         say it's still a bit easier from Squeak due to package
>     >         compatibility making it hard to commit from Pharo without
>     >         uncommitting something else, some scripts are also present by
>     >         default in the Squeak dev image, but everything should work on
>     >         both environments. No big deal in using one of the other.
>     >          
>     >
>     >             * What are the restrictions of Slang?
>     >
>     >
>     >         ...
>     >         As long as the VM compiles and run, this is correct Slang
>     code,
>     >         else it's not.
>     >         In general you should just think in C and write Smalltalk
>     instead. 
>     >         I don't know what to say. This is not dynamic at all. No
>     >         dictionaries, no look-up, no nothing. Even arrays look like
>     >         Smalltalk arrays but like C arrays you don't have the
>     actual size.
>     >         Blocks work as long as they're inlined and removed when
>     >         performing inlining in the Slang to C compiler.
>     >
>     >             * Where does VMMaker live?
>     >
>     >
>     >         Half there:
>     >         https://github.com/OpenSmalltalk/opensmalltalk-vm
>     <https://github.com/OpenSmalltalk/opensmalltalk-vm>
>     >         <https://github.com/OpenSmalltalk/opensmalltalk-vm
>     <https://github.com/OpenSmalltalk/opensmalltalk-vm>>
>     >         Accessible from git
>     >
>     >         Half there:
>     >         http://source.squeak.org/VMMaker/
>     <http://source.squeak.org/VMMaker/>
>     >         <http://source.squeak.org/VMMaker/
>     <http://source.squeak.org/VMMaker/>>
>     >         Accessible from Monticello
>     >
>     >             * Which C toolchain shall I use? 
>     >
>     >             * Which release of gcc, ld, make are known to work?
>     >             * What about clang?
>     >
>     >
>     >         Depends on the platform. The mvm script normally chooses
>     for you. 
>     >
>     >         All VMs are built from Travis CI so you can check how to
>     set up
>     >         your machine from the Travis set-up.
>     >
>     >         On linux it's gcc by default. All recent gcc versions should
>     >         work since 4.4
>     >         On Mac, gcc is not supported anymore, so clang is used. I've
>     >         used many different versions of clang and they all worked.
>     >
>     >         Specific compiler versions that don't work are mentioned
>     in the
>     >         HowToBuild file in the platform folders (if any).
>     >          
>     >
>     >             * Which version of cygwin64 or mingw64 are OK?
>     >
>     >
>     >         Arf. Both worked at some point, I don't know what the default
>     >         script use right now I would start with the version used
>     by the
>     >         mvm script / the Travis build.
>     >
>     >
>     >     gcc is broken for win64 (or at least not satisfying our
>     expectations
>     >     about stack management)
>     >     So we have to use clang.
>     >     For this reason, we prefer to compile thru cygwin which has
>     prebuilt
>     >     (downlodable) clang modules.
>     >     It's much more difficult to get a working clang in a mingw
>     >     environment, so this brand is not maintained anymore.
>     >
>     >
>     > Ah, and there was another grief with mingw...
>     > Currently the VM still uses DirectX-9.
>     > cygwin provides support for this (maybe it's not very future proof
>     now...)
>     > mingw status is unclear, and it seems that recent distributions lack
>     > this support...
>     >
>     >
>     >     But the target is a mingw target, not a cygwin target (we
>     don't want
>     >     VM to depend on cygwin layer, we want to stay minimal).
>     >     So we use a cross compiler. See the line
>     >
>     >     TOOLPREFIX:=x86_64-w64-mingw32-
>     >
>     >     in
>     >   
>      https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/build.win64x64/common/Makefile.tools
>     <https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/build.win64x64/common/Makefile.tools>
>     >   
>      <https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/build.win64x64/common/Makefile.tools
>     <https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/build.win64x64/common/Makefile.tools>>
>     >
>     >     Using cross-compiler means that we can eventually build a x86_64
>     >     .exe from a cygwin32.
>     >     But cygwin32 does not provide a debugger (gdb) for x86_64, that's
>     >     why we prefer developing from a cygwin64 environment.
>     >
>     >     Eventually, I have a hand-assembled visual studio project for
>     >     compiling the vm with MSVC (2015 if I remember...)
>     >     Of course that kind of project should better be generated (as was
>     >     possible via cmake) rather than hand-assembled...
>     >     It's absolutely cumbersome to maintain such .vc(x)proj across
>     >     several versions of MSVC.
>     >     Anyway, it does not produce working VM yet.
>     >     I retried with 2017 but encountered a code generation bug.
>     >
>     >     We could try to adapt the gnu makefiles to use MSVC thru command
>     >     line, but IMO the main interest of MSVC is to perform
>     complementary
>     >     analysis thru IDE.
>     >
>     >          
>     >
>     >             * etc.
>     >
>     >
>     >
>     >             Greetings
>     >             Raffaello
>     >
>     >             ---
>     >
>     >             [1] http://www.vpri.org/pdf/tr1997001_backto.pdf
>     <http://www.vpri.org/pdf/tr1997001_backto.pdf>
>     >             <http://www.vpri.org/pdf/tr1997001_backto.pdf
>     <http://www.vpri.org/pdf/tr1997001_backto.pdf>>
>     >             [2]
>     >           
>      http://design.cs.iastate.edu/vmil/2011/papers/p03-miranda.pdf
>     <http://design.cs.iastate.edu/vmil/2011/papers/p03-miranda.pdf>
>     <http://design.cs.iastate.edu/vmil/2011/papers/p03-miranda.pdf
>     <http://design.cs.iastate.edu/vmil/2011/papers/p03-miranda.pdf>>
>     >             [3] http://www.mirandabanda.org/cogblog/
>     <http://www.mirandabanda.org/cogblog/>
>     >             <http://www.mirandabanda.org/cogblog/
>     <http://www.mirandabanda.org/cogblog/>>
>     >             [4] https://clementbera.wordpress.com/ <https://clementbera.wordpress.com/>
>     >             <https://clementbera.wordpress.com/
>     <https://clementbera.wordpress.com/>>
>     >
>     >
>     >
>     >
>     >         --
>     >         Clément Béra
>     >         Pharo consortium engineer
>     >         https://clementbera.wordpress.com/
>     <https://clementbera.wordpress.com/>
>     >         <https://clementbera.wordpress.com/
>     <https://clementbera.wordpress.com/>>
>     >         Bâtiment B 40, avenue Halley 59650 Villeneuve d'Ascq
>     >
>     >
>     >
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com