New stable VM for Pharo8?

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.

The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8

?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,

Guille and Pablo

On 2 Apr 2019, at 17:53, ducasse <[hidden email]> wrote:

Ok thanks ronie. 

This is why the headless VM is on P8 roadmap. 

I think that Esteban will start to work on it in May. 

But he can confirm his agenda. 

Stef

On 2 Apr 2019, at 17:46, Ronie Salgado <[hidden email]> wrote:

Hi,

 

I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(

The plugin itself is optional, but it depends on code being exposed from the VM windowing system in order to blit a final rendered rectangle on top the VM window. Since Apple is deprecating OpenGL, then supporting Metal (or another Apple supported alternative) is a necessity. I am more of a fan of the no VM window solution.

My biggest problem with Metal is on testing with older versions of OS X, and with older versions of the hardware itself. It works perfectly on my Macbook air Early 2015 (The VM, and Woden). But it does not work for example on Alexandre Bergel older desktop machine (black windows with Woden). Tomorrow I will be debugging Woden on Alexandre machine, and I will take some time on trying to debug this VM problem, if I can reproduce it.

The way of debugging a Metal application is by running it with the XCode debugger, enabling GPU debugging, and capturing some frames in the debugger that which allows to inspect the draw calls and the resources that are used for generating the frame.

Best regards,

Ronie

El mar., 2 abr. 2019 a las 12:28, ducasse (<[hidden email]>) escribió:

Thanks ronie for the explanation.

I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(

When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :(

May be Pharo should have a branch so that we can promote a Pharo stable VM. 

I feel uncomfortable to have to remember which commits is the stable one for us because

how can we progress there? I do not see how we can jump over commits. 

And my concern is how can we improve the time you and Pablo and Guille are spending 

handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle

regressions and unsure that we can blessed a VM a stable for real.

We should discuss it during PharoDays consortium meeting and the next consortium online meeting. 

Stef

On 2 Apr 2019, at 16:56, Ronie Salgado <[hidden email]> wrote:

Hi,

For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):

-metal Use metal if the Metal detection code works.

-opengl Use OpenGL. The previous default.

-core-graphics Use the older CoreGraphics based backend.

-headless Null without rendering

This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.

On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.

Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.

Best regards,

Ronie

El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<[hidden email]>) escribió:

On Tue, Apr 2, 2019 at 2:11 PM ducasse <[hidden email]> wrote:

Thanks guille and pablo for you dedication for Pharo, I love you :)

Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?

I don't know exactly on what versions it is not working because I have Windows10 and Mojave.

But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).

AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX.

In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.

I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation.

In the meanwhile, using the build I say in the email above may suffice us all I think ^^.

 

Stef (back writing research proposal…..) 

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.

The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,

Guille and Pablo

--

Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr Web: http://guillep.github.io Phone: +33 06 52 70 66 13

On 2 Apr 2019, at 20:30, Esteban Lorenzano <[hidden email]> wrote:

On 2 Apr 2019, at 17:53, ducasse <[hidden email]> wrote:

Ok thanks ronie. 

This is why the headless VM is on P8 roadmap. 

I think that Esteban will start to work on it in May. 

But he can confirm his agenda. 

Well, indirectly we are already working on it :)

Pablo will concentrate on it in may (while I continue working on Spec2 until we have a beta).

I wanted to work on it myself, but this time distributions fits better :)

Esteban

Stef

On 2 Apr 2019, at 17:46, Ronie Salgado <[hidden email]> wrote:

Hi,

 

I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(

The plugin itself is optional, but it depends on code being exposed from the VM windowing system in order to blit a final rendered rectangle on top the VM window. Since Apple is deprecating OpenGL, then supporting Metal (or another Apple supported alternative) is a necessity. I am more of a fan of the no VM window solution.

My biggest problem with Metal is on testing with older versions of OS X, and with older versions of the hardware itself. It works perfectly on my Macbook air Early 2015 (The VM, and Woden). But it does not work for example on Alexandre Bergel older desktop machine (black windows with Woden). Tomorrow I will be debugging Woden on Alexandre machine, and I will take some time on trying to debug this VM problem, if I can reproduce it.

The way of debugging a Metal application is by running it with the XCode debugger, enabling GPU debugging, and capturing some frames in the debugger that which allows to inspect the draw calls and the resources that are used for generating the frame.

Best regards,

Ronie

El mar., 2 abr. 2019 a las 12:28, ducasse (<[hidden email]>) escribió:

Thanks ronie for the explanation.

I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(

When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :(

May be Pharo should have a branch so that we can promote a Pharo stable VM. 

I feel uncomfortable to have to remember which commits is the stable one for us because

how can we progress there? I do not see how we can jump over commits. 

And my concern is how can we improve the time you and Pablo and Guille are spending 

handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle

regressions and unsure that we can blessed a VM a stable for real.

We should discuss it during PharoDays consortium meeting and the next consortium online meeting. 

Stef

On 2 Apr 2019, at 16:56, Ronie Salgado <[hidden email]> wrote:

Hi,

For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):

-metal Use metal if the Metal detection code works.

-opengl Use OpenGL. The previous default.

-core-graphics Use the older CoreGraphics based backend.

-headless Null without rendering

This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.

On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.

Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.

Best regards,

Ronie

El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<[hidden email]>) escribió:

On Tue, Apr 2, 2019 at 2:11 PM ducasse <[hidden email]> wrote:

Thanks guille and pablo for you dedication for Pharo, I love you :)

Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?

I don't know exactly on what versions it is not working because I have Windows10 and Mojave.

But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).

AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX.

In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.

I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation.

In the meanwhile, using the build I say in the email above may suffice us all I think ^^.

 

Stef (back writing research proposal…..) 

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.

The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,

Guille and Pablo

--

Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr Web: http://guillep.github.io Phone: +33 06 52 70 66 13