New stable VM for Pharo8?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

New stable VM for Pharo8?

Guillermo Polito
Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.
The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,
Guille and Pablo
Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

ducasse
Thanks guille and pablo for you dedication for Pharo, I love you :)

Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?

Stef (back writing research proposal…..) 

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.
The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,
Guille and Pablo

Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

Guillermo Polito


On Tue, Apr 2, 2019 at 2:11 PM ducasse <[hidden email]> wrote:
Thanks guille and pablo for you dedication for Pharo, I love you :)

Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?

I don't know exactly on what versions it is not working because I have Windows10 and Mojave.
But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).

AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX.
In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.

I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation.
In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
 

Stef (back writing research proposal…..) 

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.
The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,
Guille and Pablo



--

   

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr


Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

alistairgrant
In reply to this post by Guillermo Polito
Hi Guille,

On Tue, 2 Apr 2019 at 13:40, Guillermo Polito <[hidden email]> wrote:

>
> Hi all, Esteban,
>
> In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.
> The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.
>
> While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.
>
> Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.
>
> Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).
>
> We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?
>
> Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Cyril has already reported this issue in Pharo 8, so I agree that an
updated VM for Pharo 8 would be nice.

As you say, this doesn't affect Pharo 7 (or earlier).


There is another, similar, issue that does affect Pharo 7 and earlier:

https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/368

which is resolved by:


commit e9fee6facb61c1163e820673bd8734b59a831535
Author: AlistairGrant <[hidden email]>
Date:   Wed Feb 6 08:23:32 2019 +0100

    FilePlugin: reopen the directory when jumping around

    In dir_Lookup, when jumping around, i.e. index ~= lastIndex+1, reopen
    the directory rather than rewinding.

    This is required as the open directory caches entries (at least with
    CIFS mounted file system), and if files are deleted between calls the
    wrong answer will be returned.

    Issue: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/368


This issue doesn't affect Pharo 8 since dir_Lookup() isn't used anymore.

I'm only aware of this being reported once, and the issue has probably
been there since 2010, or maybe even earlier.

It is worthwhile being aware of it, but probably not enough to force a
new VM for Pharo 7 immediately.

Cheers,
Alistair

Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

Ronie Salgado
In reply to this post by Guillermo Polito
Hi,

For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):

-metal Use metal if the Metal detection code works.
-opengl Use OpenGL. The previous default.
-core-graphics Use the older CoreGraphics based backend.
-headless Null without rendering

This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.

On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.

Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.

Best regards,
Ronie


El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<[hidden email]>) escribió:


On Tue, Apr 2, 2019 at 2:11 PM ducasse <[hidden email]> wrote:
Thanks guille and pablo for you dedication for Pharo, I love you :)

Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?

I don't know exactly on what versions it is not working because I have Windows10 and Mojave.
But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).

AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX.
In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.

I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation.
In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
 

Stef (back writing research proposal…..) 

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.
The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,
Guille and Pablo



--

   

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr


Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

ducasse
Thanks ronie for the explanation.
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(
When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :(

May be Pharo should have a branch so that we can promote a Pharo stable VM. 
I feel uncomfortable to have to remember which commits is the stable one for us because
how can we progress there? I do not see how we can jump over commits. 

And my concern is how can we improve the time you and Pablo and Guille are spending 
handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle
regressions and unsure that we can blessed a VM a stable for real.

We should discuss it during PharoDays consortium meeting and the next consortium online meeting. 

Stef

On 2 Apr 2019, at 16:56, Ronie Salgado <[hidden email]> wrote:

Hi,

For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):

-metal Use metal if the Metal detection code works.
-opengl Use OpenGL. The previous default.
-core-graphics Use the older CoreGraphics based backend.
-headless Null without rendering

This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.

On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.

Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.

Best regards,
Ronie


El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<[hidden email]>) escribió:


On Tue, Apr 2, 2019 at 2:11 PM ducasse <[hidden email]> wrote:
Thanks guille and pablo for you dedication for Pharo, I love you :)

Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?

I don't know exactly on what versions it is not working because I have Windows10 and Mojave.
But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).

AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX.
In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.

I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation.
In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
 

Stef (back writing research proposal…..) 

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.
The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,
Guille and Pablo



--
   
Guille Polito
Research Engineer


Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - http://www.cnrs.fr

Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

Ronie Salgado
Hi,
 
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(
The plugin itself is optional, but it depends on code being exposed from the VM windowing system in order to blit a final rendered rectangle on top the VM window. Since Apple is deprecating OpenGL, then supporting Metal (or another Apple supported alternative) is a necessity. I am more of a fan of the no VM window solution.

My biggest problem with Metal is on testing with older versions of OS X, and with older versions of the hardware itself. It works perfectly on my Macbook air Early 2015 (The VM, and Woden). But it does not work for example on Alexandre Bergel older desktop machine (black windows with Woden). Tomorrow I will be debugging Woden on Alexandre machine, and I will take some time on trying to debug this VM problem, if I can reproduce it.

The way of debugging a Metal application is by running it with the XCode debugger, enabling GPU debugging, and capturing some frames in the debugger that which allows to inspect the draw calls and the resources that are used for generating the frame.

Best regards,
Ronie

El mar., 2 abr. 2019 a las 12:28, ducasse (<[hidden email]>) escribió:
Thanks ronie for the explanation.
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(
When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :(

May be Pharo should have a branch so that we can promote a Pharo stable VM. 
I feel uncomfortable to have to remember which commits is the stable one for us because
how can we progress there? I do not see how we can jump over commits. 

And my concern is how can we improve the time you and Pablo and Guille are spending 
handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle
regressions and unsure that we can blessed a VM a stable for real.

We should discuss it during PharoDays consortium meeting and the next consortium online meeting. 

Stef

On 2 Apr 2019, at 16:56, Ronie Salgado <[hidden email]> wrote:

Hi,

For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):

-metal Use metal if the Metal detection code works.
-opengl Use OpenGL. The previous default.
-core-graphics Use the older CoreGraphics based backend.
-headless Null without rendering

This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.

On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.

Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.

Best regards,
Ronie


El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<[hidden email]>) escribió:


On Tue, Apr 2, 2019 at 2:11 PM ducasse <[hidden email]> wrote:
Thanks guille and pablo for you dedication for Pharo, I love you :)

Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?

I don't know exactly on what versions it is not working because I have Windows10 and Mojave.
But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).

AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX.
In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.

I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation.
In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
 

Stef (back writing research proposal…..) 

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.
The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,
Guille and Pablo



--
   
Guille Polito
Research Engineer


Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - http://www.cnrs.fr

Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

ducasse
Ok thanks ronie. 
This is why the headless VM is on P8 roadmap. 
I think that Esteban will start to work on it in May. 
But he can confirm his agenda. 

Stef

On 2 Apr 2019, at 17:46, Ronie Salgado <[hidden email]> wrote:

Hi,
 
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(
The plugin itself is optional, but it depends on code being exposed from the VM windowing system in order to blit a final rendered rectangle on top the VM window. Since Apple is deprecating OpenGL, then supporting Metal (or another Apple supported alternative) is a necessity. I am more of a fan of the no VM window solution.

My biggest problem with Metal is on testing with older versions of OS X, and with older versions of the hardware itself. It works perfectly on my Macbook air Early 2015 (The VM, and Woden). But it does not work for example on Alexandre Bergel older desktop machine (black windows with Woden). Tomorrow I will be debugging Woden on Alexandre machine, and I will take some time on trying to debug this VM problem, if I can reproduce it.

The way of debugging a Metal application is by running it with the XCode debugger, enabling GPU debugging, and capturing some frames in the debugger that which allows to inspect the draw calls and the resources that are used for generating the frame.

Best regards,
Ronie

El mar., 2 abr. 2019 a las 12:28, ducasse (<[hidden email]>) escribió:
Thanks ronie for the explanation.
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(
When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :(

May be Pharo should have a branch so that we can promote a Pharo stable VM. 
I feel uncomfortable to have to remember which commits is the stable one for us because
how can we progress there? I do not see how we can jump over commits. 

And my concern is how can we improve the time you and Pablo and Guille are spending 
handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle
regressions and unsure that we can blessed a VM a stable for real.

We should discuss it during PharoDays consortium meeting and the next consortium online meeting. 

Stef

On 2 Apr 2019, at 16:56, Ronie Salgado <[hidden email]> wrote:

Hi,

For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):

-metal Use metal if the Metal detection code works.
-opengl Use OpenGL. The previous default.
-core-graphics Use the older CoreGraphics based backend.
-headless Null without rendering

This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.

On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.

Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.

Best regards,
Ronie


El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<[hidden email]>) escribió:


On Tue, Apr 2, 2019 at 2:11 PM ducasse <[hidden email]> wrote:
Thanks guille and pablo for you dedication for Pharo, I love you :)

Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?

I don't know exactly on what versions it is not working because I have Windows10 and Mojave.
But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).

AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX.
In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.

I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation.
In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
 

Stef (back writing research proposal…..) 

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.
The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,
Guille and Pablo



--
   
Guille Polito
Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - http://www.cnrs.fr

Phone: +33 06 52 70 66 13


Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

EstebanLM
In reply to this post by Guillermo Polito
Hi,

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.
The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8

I will do it tomorrow morning :)

Esteban

?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,
Guille and Pablo

Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

EstebanLM
In reply to this post by ducasse


On 2 Apr 2019, at 17:53, ducasse <[hidden email]> wrote:

Ok thanks ronie. 
This is why the headless VM is on P8 roadmap. 
I think that Esteban will start to work on it in May. 
But he can confirm his agenda. 

Well, indirectly we are already working on it :)
Pablo will concentrate on it in may (while I continue working on Spec2 until we have a beta).

I wanted to work on it myself, but this time distributions fits better :)

Esteban


Stef

On 2 Apr 2019, at 17:46, Ronie Salgado <[hidden email]> wrote:

Hi,
 
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(
The plugin itself is optional, but it depends on code being exposed from the VM windowing system in order to blit a final rendered rectangle on top the VM window. Since Apple is deprecating OpenGL, then supporting Metal (or another Apple supported alternative) is a necessity. I am more of a fan of the no VM window solution.

My biggest problem with Metal is on testing with older versions of OS X, and with older versions of the hardware itself. It works perfectly on my Macbook air Early 2015 (The VM, and Woden). But it does not work for example on Alexandre Bergel older desktop machine (black windows with Woden). Tomorrow I will be debugging Woden on Alexandre machine, and I will take some time on trying to debug this VM problem, if I can reproduce it.

The way of debugging a Metal application is by running it with the XCode debugger, enabling GPU debugging, and capturing some frames in the debugger that which allows to inspect the draw calls and the resources that are used for generating the frame.

Best regards,
Ronie

El mar., 2 abr. 2019 a las 12:28, ducasse (<[hidden email]>) escribió:
Thanks ronie for the explanation.
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(
When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :(

May be Pharo should have a branch so that we can promote a Pharo stable VM. 
I feel uncomfortable to have to remember which commits is the stable one for us because
how can we progress there? I do not see how we can jump over commits. 

And my concern is how can we improve the time you and Pablo and Guille are spending 
handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle
regressions and unsure that we can blessed a VM a stable for real.

We should discuss it during PharoDays consortium meeting and the next consortium online meeting. 

Stef

On 2 Apr 2019, at 16:56, Ronie Salgado <[hidden email]> wrote:

Hi,

For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):

-metal Use metal if the Metal detection code works.
-opengl Use OpenGL. The previous default.
-core-graphics Use the older CoreGraphics based backend.
-headless Null without rendering

This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.

On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.

Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.

Best regards,
Ronie


El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<[hidden email]>) escribió:


On Tue, Apr 2, 2019 at 2:11 PM ducasse <[hidden email]> wrote:
Thanks guille and pablo for you dedication for Pharo, I love you :)

Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?

I don't know exactly on what versions it is not working because I have Windows10 and Mojave.
But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).

AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX.
In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.

I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation.
In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
 

Stef (back writing research proposal…..) 

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.
The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,
Guille and Pablo



--
   
Guille Polito
Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - http://www.cnrs.fr

Phone: +33 06 52 70 66 13



Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

EstebanLM


On 2 Apr 2019, at 20:30, Esteban Lorenzano <[hidden email]> wrote:



On 2 Apr 2019, at 17:53, ducasse <[hidden email]> wrote:

Ok thanks ronie. 
This is why the headless VM is on P8 roadmap. 
I think that Esteban will start to work on it in May. 
But he can confirm his agenda. 

Well, indirectly we are already working on it :)
Pablo will concentrate on it in may (while I continue working on Spec2 until we have a beta).

And many thanks to Schmidt on Lifeware, because of them I can say “Pablo will do it” instead saying “I will see how I multiply myself to work on it” :D

Esteban


I wanted to work on it myself, but this time distributions fits better :)

Esteban


Stef

On 2 Apr 2019, at 17:46, Ronie Salgado <[hidden email]> wrote:

Hi,
 
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(
The plugin itself is optional, but it depends on code being exposed from the VM windowing system in order to blit a final rendered rectangle on top the VM window. Since Apple is deprecating OpenGL, then supporting Metal (or another Apple supported alternative) is a necessity. I am more of a fan of the no VM window solution.

My biggest problem with Metal is on testing with older versions of OS X, and with older versions of the hardware itself. It works perfectly on my Macbook air Early 2015 (The VM, and Woden). But it does not work for example on Alexandre Bergel older desktop machine (black windows with Woden). Tomorrow I will be debugging Woden on Alexandre machine, and I will take some time on trying to debug this VM problem, if I can reproduce it.

The way of debugging a Metal application is by running it with the XCode debugger, enabling GPU debugging, and capturing some frames in the debugger that which allows to inspect the draw calls and the resources that are used for generating the frame.

Best regards,
Ronie

El mar., 2 abr. 2019 a las 12:28, ducasse (<[hidden email]>) escribió:
Thanks ronie for the explanation.
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(
When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :(

May be Pharo should have a branch so that we can promote a Pharo stable VM. 
I feel uncomfortable to have to remember which commits is the stable one for us because
how can we progress there? I do not see how we can jump over commits. 

And my concern is how can we improve the time you and Pablo and Guille are spending 
handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle
regressions and unsure that we can blessed a VM a stable for real.

We should discuss it during PharoDays consortium meeting and the next consortium online meeting. 

Stef

On 2 Apr 2019, at 16:56, Ronie Salgado <[hidden email]> wrote:

Hi,

For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):

-metal Use metal if the Metal detection code works.
-opengl Use OpenGL. The previous default.
-core-graphics Use the older CoreGraphics based backend.
-headless Null without rendering

This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.

On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.

Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.

Best regards,
Ronie


El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<[hidden email]>) escribió:


On Tue, Apr 2, 2019 at 2:11 PM ducasse <[hidden email]> wrote:
Thanks guille and pablo for you dedication for Pharo, I love you :)

Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?

I don't know exactly on what versions it is not working because I have Windows10 and Mojave.
But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).

AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX.
In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.

I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation.
In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
 

Stef (back writing research proposal…..) 

On 2 Apr 2019, at 13:40, Guillermo Polito <[hidden email]> wrote:

Hi all, Esteban,

In the last sprint with Pablo and Pierre we have detected several misbehaviours with files.
The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.

While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.

Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.

Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28298534dafbd0c26ea34fb).

We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?

Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)

Ideas?

Thanks,
Guille and Pablo



--
   
Guille Polito
Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - http://www.cnrs.fr

Phone: +33 06 52 70 66 13




Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

ducasse
Which reminds me https://www.amazon.com/Scientific-Progress-Goes-Boink-Collection/dp/0836218787

is about calvin creating a machine to create clones so that they clean his room. 
And after the clones are getting as lazy as him and they fight and the machine explodes :)

Reply | Threaded
Open this post in threaded view
|

Re: New stable VM for Pharo8?

Guillermo Polito
Just bumping, I've marked with Pablo the VM a838346b1a67712cc28298534dafbd0c26ea34fb as stable for Pharo8.
This should fix the directory issues and the primitive ones.

For those using the Pharo launcher:
 - if you use 64 bits, use the VM manager to update your VMs (remove the old 80 VM)
image.png 
image.png

 - If you re using 32 bits, you'll have to wait a bit more. I've found an issue in the middle, fix in progress here (with a test, yay!)

Guille and Pablo

On Tue, Apr 2, 2019 at 8:53 PM ducasse <[hidden email]> wrote:
Which reminds me https://www.amazon.com/Scientific-Progress-Goes-Boink-Collection/dp/0836218787

is about calvin creating a machine to create clones so that they clean his room. 
And after the clones are getting as lazy as him and they fight and the machine explodes :)



--

   

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr


Web: http://guillep.github.io

Phone: +33 06 52 70 66 13