two issues: GC and wrong pointers

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

two issues: GC and wrong pointers

Tudor Girba-2
Hi,

During our work we encountered two issues we think should be considered critical.

The first is that the garbage collector is unreliable on larger images:
https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391
We stumble on this when we manipulate larger pieces of data quite consistently in plain Pharo (7 or 8). We lack the knowledge of how to approach it, but we would happily pair and try to reproduce it with someone.

The second one is that the sources contain wrong pointers, and this makes it hard to work with TFFI:
https://github.com/pharo-project/pharo/issues/4967
This is already marked as important. We spent quite some time investigating this, especially in the context of TFFI, and we would be happy to pair to work through it.

Would anyone be available for a collaboration in the following days?

Cheers,
Doru


--
feenk.com

"Not knowing how to do something is not an argument for how it cannot be done."


Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

Marcus Denker-4


> On 11 Nov 2019, at 12:57, Tudor Girba <[hidden email]> wrote:
>
> Hi,
>
> During our work we encountered two issues we think should be considered critical.
>
> The first is that the garbage collector is unreliable on larger images:
> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391
> We stumble on this when we manipulate larger pieces of data quite consistently in plain Pharo (7 or 8). We lack the knowledge of how to approach it, but we would happily pair and try to reproduce it with someone.
>
> The second one is that the sources contain wrong pointers, and this makes it hard to work with TFFI:
> https://github.com/pharo-project/pharo/issues/4967
> This is already marked as important. We spent quite some time investigating this, especially in the context of TFFI, and we would be happy to pair to work through it.
>
This one I want to have a look at… I think the solution is to remove the magic and maybe even think about integrating the FFI plugin into the compiler.

I am available Tuesday afternoon.

        Marcus


Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

ducasse
In reply to this post by Tudor Girba-2
Hi doru


Can you tell us more? How many threads are running? how many times did you save the image?
Pablo is investigating a bug related to the GC that happens when we have many threads and saved the image
multiple times. He was on it since a couple of days.

Now Pablo is attending Smalltalks and will be on vacation until end of the month.
Guille is on father leave and coming back to work one day a week.

So we are concerned about this problem but we cannot really allocate people on it before beginning of december.

S.

> On 11 Nov 2019, at 12:57, Tudor Girba <[hidden email]> wrote:
>
> Hi,
>
> During our work we encountered two issues we think should be considered critical.
>
> The first is that the garbage collector is unreliable on larger images:
> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391
> We stumble on this when we manipulate larger pieces of data quite consistently in plain Pharo (7 or 8). We lack the knowledge of how to approach it, but we would happily pair and try to reproduce it with someone.
>
> The second one is that the sources contain wrong pointers, and this makes it hard to work with TFFI:
> https://github.com/pharo-project/pharo/issues/4967
> This is already marked as important. We spent quite some time investigating this, especially in the context of TFFI, and we would be happy to pair to work through it.
>
> Would anyone be available for a collaboration in the following days?
>
> Cheers,
> Doru
>
>
> --
> feenk.com
>
> "Not knowing how to do something is not an argument for how it cannot be done."
>
>



Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

Tudor Girba-2
Hi,

For the GC problem from https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391, there is no threading involved. Just invoking full GC in a large image.

Cheers,
Doru



> On Nov 11, 2019, at 1:47 PM, ducasse <[hidden email]> wrote:
>
> Hi doru
>
>
> Can you tell us more? How many threads are running? how many times did you save the image?
> Pablo is investigating a bug related to the GC that happens when we have many threads and saved the image
> multiple times. He was on it since a couple of days.
>
> Now Pablo is attending Smalltalks and will be on vacation until end of the month.
> Guille is on father leave and coming back to work one day a week.
>
> So we are concerned about this problem but we cannot really allocate people on it before beginning of december.
>
> S.
>
>> On 11 Nov 2019, at 12:57, Tudor Girba <[hidden email]> wrote:
>>
>> Hi,
>>
>> During our work we encountered two issues we think should be considered critical.
>>
>> The first is that the garbage collector is unreliable on larger images:
>> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391
>> We stumble on this when we manipulate larger pieces of data quite consistently in plain Pharo (7 or 8). We lack the knowledge of how to approach it, but we would happily pair and try to reproduce it with someone.
>>
>> The second one is that the sources contain wrong pointers, and this makes it hard to work with TFFI:
>> https://github.com/pharo-project/pharo/issues/4967
>> This is already marked as important. We spent quite some time investigating this, especially in the context of TFFI, and we would be happy to pair to work through it.
>>
>> Would anyone be available for a collaboration in the following days?
>>
>> Cheers,
>> Doru
>>
>>
>> --
>> feenk.com
>>
>> "Not knowing how to do something is not an argument for how it cannot be done."
>>
>>
>
>
>

--
feenk.com

"Value is always contextual."







Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

Marcus Denker-4


Hi,

During our work we encountered two issues we think should be considered critical.

The first is that the garbage collector is unreliable on larger images:
https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391
We stumble on this when we manipulate larger pieces of data quite consistently in plain Pharo (7 or 8). We lack the knowledge of how to approach it, but we would happily pair and try to reproduce it with someone.

The second one is that the sources contain wrong pointers, and this makes it hard to work with TFFI:
https://github.com/pharo-project/pharo/issues/4967
This is already marked as important. We spent quite some time investigating this, especially in the context of TFFI, and we would be happy to pair to work through it.


I think my PR fixes it… the PR is green and the downloaded artefact seems to not show any problematic methods anymore:




Marcus
Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

ducasse
In reply to this post by Tudor Girba-2
May be we have two problems then :(
Because we have problem without a large image.

Stef

> On 11 Nov 2019, at 14:44, Tudor Girba <[hidden email]> wrote:
>
> Hi,
>
> For the GC problem from https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391, there is no threading involved. Just invoking full GC in a large image.
>
> Cheers,
> Doru
>
>
>
>> On Nov 11, 2019, at 1:47 PM, ducasse <[hidden email]> wrote:
>>
>> Hi doru
>>
>>
>> Can you tell us more? How many threads are running? how many times did you save the image?
>> Pablo is investigating a bug related to the GC that happens when we have many threads and saved the image
>> multiple times. He was on it since a couple of days.
>>
>> Now Pablo is attending Smalltalks and will be on vacation until end of the month.
>> Guille is on father leave and coming back to work one day a week.
>>
>> So we are concerned about this problem but we cannot really allocate people on it before beginning of december.
>>
>> S.
>>
>>> On 11 Nov 2019, at 12:57, Tudor Girba <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> During our work we encountered two issues we think should be considered critical.
>>>
>>> The first is that the garbage collector is unreliable on larger images:
>>> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391
>>> We stumble on this when we manipulate larger pieces of data quite consistently in plain Pharo (7 or 8). We lack the knowledge of how to approach it, but we would happily pair and try to reproduce it with someone.
>>>
>>> The second one is that the sources contain wrong pointers, and this makes it hard to work with TFFI:
>>> https://github.com/pharo-project/pharo/issues/4967
>>> This is already marked as important. We spent quite some time investigating this, especially in the context of TFFI, and we would be happy to pair to work through it.
>>>
>>> Would anyone be available for a collaboration in the following days?
>>>
>>> Cheers,
>>> Doru
>>>
>>>
>>> --
>>> feenk.com
>>>
>>> "Not knowing how to do something is not an argument for how it cannot be done."
>>>
>>>
>>
>>
>>
>
> --
> feenk.com
>
> "Value is always contextual."
>
>
>
>
>
>
>



Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

ducasse
In reply to this post by Marcus Denker-4
marcus I read your fix and I do not understand why the previous way was a problem.
What was your hypothesis?

Stef

On 11 Nov 2019, at 14:46, Marcus Denker <[hidden email]> wrote:



Hi,

During our work we encountered two issues we think should be considered critical.

The first is that the garbage collector is unreliable on larger images:
https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391
We stumble on this when we manipulate larger pieces of data quite consistently in plain Pharo (7 or 8). We lack the knowledge of how to approach it, but we would happily pair and try to reproduce it with someone.

The second one is that the sources contain wrong pointers, and this makes it hard to work with TFFI:
https://github.com/pharo-project/pharo/issues/4967
This is already marked as important. We spent quite some time investigating this, especially in the context of TFFI, and we would be happy to pair to work through it.


I think my PR fixes it… the PR is green and the downloaded artefact seems to not show any problematic methods anymore:




Marcus

Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

Marcus Denker-4


On 11 Nov 2019, at 15:04, ducasse <[hidden email]> wrote:

marcus I read your fix and I do not understand why the previous way was a problem.
What was your hypothesis?


I think the problem is that it forces recompiling methods while already loading code, which leads to some strange effect.. that I think
the MethodChanged that monticello does at the end references the old method, but that got recompiled in the meantime.

Something like that.

Stef

On 11 Nov 2019, at 14:46, Marcus Denker <[hidden email]> wrote:



Hi,

During our work we encountered two issues we think should be considered critical.

The first is that the garbage collector is unreliable on larger images:
https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391
We stumble on this when we manipulate larger pieces of data quite consistently in plain Pharo (7 or 8). We lack the knowledge of how to approach it, but we would happily pair and try to reproduce it with someone.

The second one is that the sources contain wrong pointers, and this makes it hard to work with TFFI:
https://github.com/pharo-project/pharo/issues/4967
This is already marked as important. We spent quite some time investigating this, especially in the context of TFFI, and we would be happy to pair to work through it.


I think my PR fixes it… the PR is green and the downloaded artefact seems to not show any problematic methods anymore:




Marcus


Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

Tudor Girba-2
Thanks Marcus!

Alistair reviewed the issue (https://github.com/pharo-project/pharo/pull/5120) and the pull request looks great.

Cool work :).

Cheers,
Doru



> On Nov 11, 2019, at 3:51 PM, Marcus Denker <[hidden email]> wrote:
>
>
>
>> On 11 Nov 2019, at 15:04, ducasse <[hidden email]> wrote:
>>
>> marcus I read your fix and I do not understand why the previous way was a problem.
>> What was your hypothesis?
>>
>
> I think the problem is that it forces recompiling methods while already loading code, which leads to some strange effect.. that I think
> the MethodChanged that monticello does at the end references the old method, but that got recompiled in the meantime.
>
> Something like that.
>
>> Stef
>>
>>> On 11 Nov 2019, at 14:46, Marcus Denker <[hidden email]> wrote:
>>>
>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> During our work we encountered two issues we think should be considered critical.
>>>>>>
>>>>>> The first is that the garbage collector is unreliable on larger images:
>>>>>> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391
>>>>>> We stumble on this when we manipulate larger pieces of data quite consistently in plain Pharo (7 or 8). We lack the knowledge of how to approach it, but we would happily pair and try to reproduce it with someone.
>>>>>>
>>>>>> The second one is that the sources contain wrong pointers, and this makes it hard to work with TFFI:
>>>>>> https://github.com/pharo-project/pharo/issues/4967
>>>>>> This is already marked as important. We spent quite some time investigating this, especially in the context of TFFI, and we would be happy to pair to work through it.
>>>>>>
>>>
>>> I think my PR fixes it… the PR is green and the downloaded artefact seems to not show any problematic methods anymore:
>>>
>>> https://github.com/pharo-project/pharo/pull/5120
>>>
>>>
>>>
>>> Marcus
>>
>

--
feenk.com

"To utilize feedback, you first have to acquire it."


Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

ducasse
In reply to this post by Tudor Girba-2


> On 11 Nov 2019, at 12:57, Tudor Girba <[hidden email]> wrote:
>
> Hi,
>
> During our work we encountered two issues we think should be considered critical.
>
> The first is that the garbage collector is unreliable on larger images:
> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391
> We stumble on this when we manipulate larger pieces of data quite consistently in plain Pharo (7 or 8). We lack the knowledge of how to approach it, but we would happily pair and try to reproduce it with someone.


Doru

To me this is not really a good bug report.
        How can we reproduce it?
        What is a big image?
               
The first thing to do is to get a simple reproducible case.
This is what pavel did with the crash on normal images.
Can you see if you can do something like that?
It will help us trying to fix it.

S.



Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

Tudor Girba-2
Hi,

Yes, I know it’s not a good public bug report. That is why I asked for help.

The problem is that this is not easily reproducible outside of specific cases. We encountered it in the context of GT which has double the code as compared to Pharo. So, to test it outside of GT, for the last couple of weeks we manipulated some data in plain Pharo, and we still got to the problem a couple of times (though it appears less often). We do not know yet how to reproduce it with anonymous data. That is why I said that some pairing might help.

In parallel we are trying to reproduce it, like we do with all other bug reports.

Cheers,
Doru


> On Nov 12, 2019, at 8:00 AM, ducasse <[hidden email]> wrote:
>
>
>
>> On 11 Nov 2019, at 12:57, Tudor Girba <[hidden email]> wrote:
>>
>> Hi,
>>
>> During our work we encountered two issues we think should be considered critical.
>>
>> The first is that the garbage collector is unreliable on larger images:
>> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391
>> We stumble on this when we manipulate larger pieces of data quite consistently in plain Pharo (7 or 8). We lack the knowledge of how to approach it, but we would happily pair and try to reproduce it with someone.
>
>
> Doru
>
> To me this is not really a good bug report.
> How can we reproduce it?
> What is a big image?
>
> The first thing to do is to get a simple reproducible case.
> This is what pavel did with the crash on normal images.
> Can you see if you can do something like that?
> It will help us trying to fix it.
>
> S.
>
>
>

--
feenk.com

"What is more important: To be happy, or to make happy?"


Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

ducasse
ok
It may be related to the bug pablo is working on. 
So it would be good to see how many times the image is saved. 

Stef

On 12 Nov 2019, at 08:58, Tudor Girba <[hidden email]> wrote:

ith all other bug reports.

Reply | Threaded
Open this post in threaded view
|

Re: two issues: GC and wrong pointers

Pharo Smalltalk Developers mailing list
Hi,

we found a script that reproduces the GC segmentation fault. The report is here
- which is likely the same issue as reported here: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/391 

Best,
Juraj


On 12 Nov 2019, at 12:40, ducasse <[hidden email]> wrote:

ok
It may be related to the bug pablo is working on. 
So it would be good to see how many times the image is saved. 

Stef

On 12 Nov 2019, at 08:58, Tudor Girba <[hidden email]> wrote:

ith all other bug reports.