Status of Pharo VM for FreeBSD

Hi Eliot,

Am 11.03.2015 um 23:15 schrieb Eliot Miranda <[hidden email]>:

HI Andreas,

On Wed, Mar 11, 2015 at 9:55 AM, Andreas Wacknitz <[hidden email]> wrote:

 

Hi Clement,

Am 11.03.2015 um 09:23 schrieb Clément Bera <[hidden email]>:

Hello,

About the FreeBSD VM, Holger Freyther worked on it so he's the best person to answer. I think some people used it and it was at least partially working.

That’s my impression. The VMMaker contains some FreeBSD classes but I have the impression that they are not complete (and probably outdated).

About your NativeBoost bug on openSolaris,  need more information:

- Can you confirm that you use an intel processor on your openSolaris machine ? I assume that yes but I ask because solaris were typically running on other processors. NativeBoost, as of today, works only with intel processor.

Yes, my Sun Ultra 24 is an Intel based Workstation (Q9300).

- Do you build the Cog VM or Stack VM ? I mean PharoVMBuild or PharoSVMBuild ? I think the PharoSVMBuild does not include NativeBoost by default, that may be your problem. There's a fix somewhere...

PharoVM from "branch 'master' of https://github.com/pharo-project/pharo-vm" (thus Cog VM).

I would like to fold back any changes into the svn master repository for Cog.   What are the diffs?  (If you have time to send me the diffs that would save me a lot of time).

I don’t know whether there is much to harvest from what I did. As far as I remember most of my work was hacking the generator image created by the pharo vm scripts (for my Mac) in order to make

the resulting C code to compile under openindiana. The basis for Solaris was already there (and as far as I can see it is also in the Squeak VM sources). I only tweaked some definitions and includes.

I will look at my notes tomorrow and will post if I will find something relevant.

I am curios about the future of the PharoVM. The main development of the VM seem to happen in the SqueakVM (by you). Getting the Spur changes into the PharoVM seem to be a lot of work.

Wouldn’t it be better to move back the changes of the PharoVM into the SqueakVM and have a united development?

And what is the status and what are the plans regarding Timothy’s (gettimothy) work?

Regards

Andreas

On 02 Apr 2015, at 19:20, Eliot Miranda <[hidden email]> wrote:

Hi Andreas,

    sorry to be late in replying.  This has been a busy month (I moved house).

On Sat, Mar 14, 2015 at 10:33 AM, Andreas Wacknitz <[hidden email]> wrote:

 

Hi Eliot,

Am 11.03.2015 um 23:15 schrieb Eliot Miranda <[hidden email]>:

HI Andreas,

On Wed, Mar 11, 2015 at 9:55 AM, Andreas Wacknitz <[hidden email]> wrote:

 

Hi Clement,

Am 11.03.2015 um 09:23 schrieb Clément Bera <[hidden email]>:

Hello,

About the FreeBSD VM, Holger Freyther worked on it so he's the best person to answer. I think some people used it and it was at least partially working.

That’s my impression. The VMMaker contains some FreeBSD classes but I have the impression that they are not complete (and probably outdated).

About your NativeBoost bug on openSolaris,  need more information:

- Can you confirm that you use an intel processor on your openSolaris machine ? I assume that yes but I ask because solaris were typically running on other processors. NativeBoost, as of today, works only with intel processor.

Yes, my Sun Ultra 24 is an Intel based Workstation (Q9300).

- Do you build the Cog VM or Stack VM ? I mean PharoVMBuild or PharoSVMBuild ? I think the PharoSVMBuild does not include NativeBoost by default, that may be your problem. There's a fix somewhere...

PharoVM from "branch 'master' of https://github.com/pharo-project/pharo-vm" (thus Cog VM).

I would like to fold back any changes into the svn master repository for Cog.   What are the diffs?  (If you have time to send me the diffs that would save me a lot of time).

I don’t know whether there is much to harvest from what I did. As far as I remember most of my work was hacking the generator image created by the pharo vm scripts (for my Mac) in order to make

the resulting C code to compile under openindiana. The basis for Solaris was already there (and as far as I can see it is also in the Squeak VM sources). I only tweaked some definitions and includes.

I will look at my notes tomorrow and will post if I will find something relevant.

I am curios about the future of the PharoVM. The main development of the VM seem to happen in the SqueakVM (by you). Getting the Spur changes into the PharoVM seem to be a lot of work.

Note that this will happen (or is already happening).  Esteban is working on building the Spur version of Pharo, so he is doing this work.  But actually it *isn't* that much work.  There is basically a trio of new memory management files for each platform, e.g. platforms/unix/vm/sqUnixSpurMemory.c, and a new source tree for the spur vm, spursrc/vm.  The system is already set up to build multiple VMs (at least the svn tree is).

Yes, this is already done. We are building spur VMs and images since awhile now. You can find all the related jobs here: 

https://ci.inria.fr/pharo/view/4.0-VM-Spur/

And as Eliot says… is not *much* work… except when it is :)

In fact, we were planning to release Pharo 4 (next week) with a Spur VM, but we didn’t finish all the small things around. So we will release next July (or around) a Pharo 4S (S, for Spur) with “official” spur support. We do not want to stay to much time in older versions. Also, our development process is different

than squeak, AFAIK… we drop backward compatibility in a regular basis. Which basically means we will move to spur and we will drop support for older versions.

 

Wouldn’t it be better to move back the changes of the PharoVM into the SqueakVM and have a united development?

Well, I don't think the Pharo community will be willing to move to svn.  SOme time I may be able to move to git.  But yes, I *would* like to see important fixes merged back into the SqueakVM.  I think this is very important.  I'm too overloaded to look at the pharovm so I'm dependent on those working on the pharovm in giut to send me changes for integration.

Right, we are happy with our process and I do not see it fitting with svn. We changed a lot of “organisational” stuff to ensure traceability and “buildability” (if such word exists…). And we have made a lot of progress in that area using git and github infrastructure at a point most of the time to incorporate a change we just accept a pull request. 

To be able to do that:

- we have to be sure what version of each component (vm, plugin, platform source) is part of the commit info. That’s why we keep together both platform sources and image sources (using filetree monticello format). That way each commit has everything we need to build the new vm. In fact… I have a script “./newVM <commit>” that does a clone, prepares an image, generates sources and builds the vm… then I can test if a pull request is valid. But most of the time that is not needed, because:

-  for each pull request, we fire a travis job that creates a vm from scratch and then runs all tests we have in Pharo (and we have improved a lot in that area latest years). They are not “vm specific tests”, but since they tests all the system, if vm does not crashes and tests are run, we can be sure is working (this wouldn’t be possible without right traceability). 

- we also build the vm using CMake, but not directly, we use CMakeMaker which allow us to define the build in smalltalk.

- finally, we would like to use the other capabilities (for documentation, etc.) we gain for free by using github. Not that we are already using it… but we would like, in the future. 

So… obviously all of this can be achieved without using git and github… but there the infrastructure is already done. 

Said that. Even if we actually have a different process, we (Myself, particularly) are trying to reduce the gap between both VMs. And right now this is the status:

- in the VM itself there is almost no change. AFAIR, just two small things: 

a) I include setjmp.h somewhere, because compiler was asking for it (We use different versions than Eliot)

b) the macro to read the image is changed, because we needed to change it for allow build an iOS image. This is just one line in the image and the addition of one macro in 4 platform sources (Linux, Win and Mac redirects to old macro, but iOS implements something different)

- In the platforms we have the most important difference, because we deprecated the “Mac OS” branch in favor of “iOS”, which in fact should be called OSX, because is the Cocoa version. I understand Eliot want to go in that direction soon so we will align in that area too (btw, that branch has growth organically so we'll need to do some reorganisation to clarify it, eventually)

- in the plugins, we try to adopt a different approach than the previous one: instead using particularities of the platform, we want to align sources as much as possible, so we use the posix libraries. Again, that’s just when is possible (and when we have time). The most important change we produced here is with FilePlugin: we changed it to provide posix-permissions (and soon we will add primitives to retrieve also ownership). To allow that, we changed a lot in the windows version of the plugin, because instead windows functions we use MinGW. We would like to see this changes merged.

After that, I think there will be some other minor changes… not many, and most probably we can remove those differences. 

hope this clarify all :)

Am 03.04.15 11:13, schrieb Esteban Lorenzano:

On 02 Apr 2015, at 19:20, Eliot Miranda <[hidden email]> wrote:

Hi Andreas,

    sorry to be late in replying.  This has been a busy month (I moved house).

On Sat, Mar 14, 2015 at 10:33 AM, Andreas Wacknitz <[hidden email]> wrote:

 

Hi Eliot,

Am 11.03.2015 um 23:15 schrieb Eliot Miranda <[hidden email]>:

HI Andreas,

On Wed, Mar 11, 2015 at 9:55 AM, Andreas Wacknitz <[hidden email]> wrote:

 

Hi Clement,

Am 11.03.2015 um 09:23 schrieb Clément Bera <[hidden email]>:

Hello,

About the FreeBSD VM, Holger Freyther worked on it so he's the best person to answer. I think some people used it and it was at least partially working.

That’s my impression. The VMMaker contains some FreeBSD classes but I have the impression that they are not complete (and probably outdated).

About your NativeBoost bug on openSolaris,  need more information:

- Can you confirm that you use an intel processor on your openSolaris machine ? I assume that yes but I ask because solaris were typically running on other processors. NativeBoost, as of today, works only with intel processor.

Yes, my Sun Ultra 24 is an Intel based Workstation (Q9300).

- Do you build the Cog VM or Stack VM ? I mean PharoVMBuild or PharoSVMBuild ? I think the PharoSVMBuild does not include NativeBoost by default, that may be your problem. There's a fix somewhere...

PharoVM from "branch 'master' of https://github.com/pharo-project/pharo-vm" (thus Cog VM).

I would like to fold back any changes into the svn master repository for Cog.   What are the diffs?  (If you have time to send me the diffs that would save me a lot of time).

I don’t know whether there is much to harvest from what I did. As far as I remember most of my work was hacking the generator image created by the pharo vm scripts (for my Mac) in order to make

the resulting C code to compile under openindiana. The basis for Solaris was already there (and as far as I can see it is also in the Squeak VM sources). I only tweaked some definitions and includes.

I will look at my notes tomorrow and will post if I will find something relevant.

I am curios about the future of the PharoVM. The main development of the VM seem to happen in the SqueakVM (by you). Getting the Spur changes into the PharoVM seem to be a lot of work.

Note that this will happen (or is already happening).  Esteban is working on building the Spur version of Pharo, so he is doing this work.  But actually it *isn't* that much work.  There is basically a trio of new memory management files for each platform, e.g. platforms/unix/vm/sqUnixSpurMemory.c, and a new source tree for the spur vm, spursrc/vm.  The system is already set up to build multiple VMs (at least the svn tree is).

Yes, this is already done. We are building spur VMs and images since awhile now. You can find all the related jobs here: 

https://ci.inria.fr/pharo/view/4.0-VM-Spur/

If I follow this link and what is being used there brings me to the ordinary PharoVm project on github:
https://github.com/pharo-project/pharo-vm
There are three branches: master, develop and spur64. Which one is being used to build PharoVM-spur32?

And as Eliot says… is not *much* work… except when it is :)

In fact, we were planning to release Pharo 4 (next week) with a Spur VM, but we didn’t finish all the small things around. So we will release next July (or around) a Pharo 4S (S, for Spur) with “official” spur support. We do not want to stay to much time in older versions. Also, our development process is different

This explanation irritates me: Pharo 4 will be released soon with a Spur VM? And then around summer Pharo 4S? Isn't it a contradiction?

than squeak, AFAIK… we drop backward compatibility in a regular basis. Which basically means we will move to spur and we will drop support for older versions.

That's OK, but I am still, hmm say confused, because Eliot is changing A LOT (just look at what has been released during the last days), but PharoVM hasn't been
changed for some days (I am following the master branch closely). So there is a rapid development in the Cog branch of the SqueakVM. The changes in the PharoVM are much slower (at least as I recognise it).

On 03 Apr 2015, at 13:04, Andreas Wacknitz <[hidden email]> wrote:

Am 03.04.15 11:13, schrieb Esteban Lorenzano:

On 02 Apr 2015, at 19:20, Eliot Miranda <[hidden email]> wrote:

Hi Andreas,

    sorry to be late in replying.  This has been a busy month (I moved house).

On Sat, Mar 14, 2015 at 10:33 AM, Andreas Wacknitz <[hidden email]> wrote:

 

Hi Eliot,

Am 11.03.2015 um 23:15 schrieb Eliot Miranda <[hidden email]>:

HI Andreas,

On Wed, Mar 11, 2015 at 9:55 AM, Andreas Wacknitz <[hidden email]> wrote:

 

Hi Clement,

Am 11.03.2015 um 09:23 schrieb Clément Bera <[hidden email]>:

Hello,

About the FreeBSD VM, Holger Freyther worked on it so he's the best person to answer. I think some people used it and it was at least partially working.

That’s my impression. The VMMaker contains some FreeBSD classes but I have the impression that they are not complete (and probably outdated).

About your NativeBoost bug on openSolaris,  need more information:

- Can you confirm that you use an intel processor on your openSolaris machine ? I assume that yes but I ask because solaris were typically running on other processors. NativeBoost, as of today, works only with intel processor.

Yes, my Sun Ultra 24 is an Intel based Workstation (Q9300).

- Do you build the Cog VM or Stack VM ? I mean PharoVMBuild or PharoSVMBuild ? I think the PharoSVMBuild does not include NativeBoost by default, that may be your problem. There's a fix somewhere...

PharoVM from "branch 'master' of https://github.com/pharo-project/pharo-vm" (thus Cog VM).

I would like to fold back any changes into the svn master repository for Cog.   What are the diffs?  (If you have time to send me the diffs that would save me a lot of time).

I don’t know whether there is much to harvest from what I did. As far as I remember most of my work was hacking the generator image created by the pharo vm scripts (for my Mac) in order to make

the resulting C code to compile under openindiana. The basis for Solaris was already there (and as far as I can see it is also in the Squeak VM sources). I only tweaked some definitions and includes.

I will look at my notes tomorrow and will post if I will find something relevant.

I am curios about the future of the PharoVM. The main development of the VM seem to happen in the SqueakVM (by you). Getting the Spur changes into the PharoVM seem to be a lot of work.

Note that this will happen (or is already happening).  Esteban is working on building the Spur version of Pharo, so he is doing this work.  But actually it *isn't* that much work.  There is basically a trio of new memory management files for each platform, e.g. platforms/unix/vm/sqUnixSpurMemory.c, and a new source tree for the spur vm, spursrc/vm.  The system is already set up to build multiple VMs (at least the svn tree is).

Yes, this is already done. We are building spur VMs and images since awhile now. You can find all the related jobs here: 

https://ci.inria.fr/pharo/view/4.0-VM-Spur/

If I follow this link and what is being used there brings me to the ordinary PharoVm project on github:
https://github.com/pharo-project/pharo-vm
There are three branches: master, develop and spur64. Which one is being used to build PharoVM-spur32?

yes, I still didn’t merged with master. 

master still builds a regular cog vm. 

I’m working on spur64 branch now, here:

https://github.com/estebanlm/pharo-vm/tree/spur64

the spur jobs are built against my repository for the moment.

And as Eliot says… is not *much* work… except when it is :)

In fact, we were planning to release Pharo 4 (next week) with a Spur VM, but we didn’t finish all the small things around. So we will release next July (or around) a Pharo 4S (S, for Spur) with “official” spur support. We do not want to stay to much time in older versions. Also, our development process is different

This explanation irritates me: Pharo 4 will be released soon with a Spur VM? And then around summer Pharo 4S? Isn't it a contradiction?

no, Pharo 4 will be released *without* spur, then we will release a 4S *with* spur.

than squeak, AFAIK… we drop backward compatibility in a regular basis. Which basically means we will move to spur and we will drop support for older versions.

That's OK, but I am still, hmm say confused, because Eliot is changing A LOT (just look at what has been released during the last days), but PharoVM hasn't been
changed for some days (I am following the master branch closely). So there is a rapid development in the Cog branch of the SqueakVM. The changes in the PharoVM are much slower (at least as I recognise it).

just because you checked the master repository, not mine… mine changed yesterday :)

I have more questions but I am reluctant to disturb you further as you must be quite busy atm.

ask, I will answer when I can :)

Best regards
Andreas

 

Wouldn’t it be better to move back the changes of the PharoVM into the SqueakVM and have a united development?

Well, I don't think the Pharo community will be willing to move to svn.  SOme time I may be able to move to git.  But yes, I *would* like to see important fixes merged back into the SqueakVM.  I think this is very important.  I'm too overloaded to look at the pharovm so I'm dependent on those working on the pharovm in giut to send me changes for integration.

Right, we are happy with our process and I do not see it fitting with svn. We changed a lot of “organisational” stuff to ensure traceability and “buildability” (if such word exists…). And we have made a lot of progress in that area using git and github infrastructure at a point most of the time to incorporate a change we just accept a pull request. 

To be able to do that:

- we have to be sure what version of each component (vm, plugin, platform source) is part of the commit info. That’s why we keep together both platform sources and image sources (using filetree monticello format). That way each commit has everything we need to build the new vm. In fact… I have a script “./newVM <commit>” that does a clone, prepares an image, generates sources and builds the vm… then I can test if a pull request is valid. But most of the time that is not needed, because:

-  for each pull request, we fire a travis job that creates a vm from scratch and then runs all tests we have in Pharo (and we have improved a lot in that area latest years). They are not “vm specific tests”, but since they tests all the system, if vm does not crashes and tests are run, we can be sure is working (this wouldn’t be possible without right traceability). 

- we also build the vm using CMake, but not directly, we use CMakeMaker which allow us to define the build in smalltalk.

- finally, we would like to use the other capabilities (for documentation, etc.) we gain for free by using github. Not that we are already using it… but we would like, in the future. 

Is there an up-to-date documentation of the processes? Especially if I want to add more platform support (like openindiana or FreeBSD)?

the process, no… for now we accept pull requests… and we handle issues using the pharo issue tracker. This is not good, I would like to have a centralised issue tracker… but well, we will talk with Eliot about it soon (I suppose having alll sources in github also could lead us to us the issue tracker of github)

the build process is explained in the README.md.

My time is very limited (my day job is quite different from what I do in my spare time + my family and my house also need a lot attention + next to my Smalltalk interets I am also interested in operating systems) so my reaction time is sometimes quite slow :)
 

So… obviously all of this can be achieved without using git and github… but there the infrastructure is already done. 

Said that. Even if we actually have a different process, we (Myself, particularly) are trying to reduce the gap between both VMs. And right now this is the status:

- in the VM itself there is almost no change. AFAIR, just two small things: 

a) I include setjmp.h somewhere, because compiler was asking for it (We use different versions than Eliot)

b) the macro to read the image is changed, because we needed to change it for allow build an iOS image. This is just one line in the image and the addition of one macro in 4 platform sources (Linux, Win and Mac redirects to old macro, but iOS implements something different)

- In the platforms we have the most important difference, because we deprecated the “Mac OS” branch in favor of “iOS”, which in fact should be called OSX, because is the Cocoa version. I understand Eliot want to go in that direction soon so we will align in that area too (btw, that branch has growth organically so we'll need to do some reorganisation to clarify it, eventually)

- in the plugins, we try to adopt a different approach than the previous one: instead using particularities of the platform, we want to align sources as much as possible, so we use the posix libraries. Again, that’s just when is possible (and when we have time). The most important change we produced here is with FilePlugin: we changed it to provide posix-permissions (and soon we will add primitives to retrieve also ownership). To allow that, we changed a lot in the windows version of the plugin, because instead windows functions we use MinGW. We would like to see this changes merged.

After that, I think there will be some other minor changes… not many, and most probably we can remove those differences. 

hope this clarify all :)

That's a nice explanation at least.
But I am still confused. Especially because you didn't mention NativeBoost :)

Nativeboost is just a plugin (and a cpp flag in generated sources to allocate executable memory).

The ffi area also seem to be different and i a flux.

but we are going to align that also (already half way from there) :)

we want to be as close as possible… there is no point on keeping not needed divergences. 

And still: How to integrate more platforms?

depends on the platform. 

regular process would be: 

1) create a /platform directory, for example /platform/freebsd

but since freebsd is a unix, probably you do not need to do that and you can handle it in same /platform/unix directory

2) create a PharoSpur32FreeBSDConfig, taking CogFreeBSDConfig and PharoSpur32UnixConfig (already there) as models.

no idea what changes you need to have. 

we could probably get a freebsd slave to add the build to our building process, I don’t know :)