More Windows VM Crashing -_-

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

More Windows VM Crashing -_-

Peter Uhnak
Hi,

Originally I thought my VM is crashing because of FT, so I applied some fix from FogBugz (that is now part of Pharo 6), but that didn't help.

So I disabled FT in settings and changed fonts to non-FT ones; didn't help.
So I completely removed FT2Plugin.dll (now settings just states that FT fonts are not available), but that didn't help either.

And obviously there is no crash.dmp whatsoever (this is latest Windows VM).

This crash is extremely frustrating because the crash happens _during saving_, which means I am losing code all the time and have to constantly recover them, which really makes my blood boil.

Without crash dump I don't even know where and how to start investigating. Windows kindly opens Visual Studio when Pharo crashes, but all it shows to me is assembly, which is not very useful to me. I can somewhat orient myself in the C code of the VM (as I've alredy fixed some other issues in the VM), but here I am lost.

<SNIP>
00407D7F  ret  
00407D80  mov         eax,dword ptr ds:[00596E44h]  
00407D85  test        al,3  
00407D87  jne         00407DB2  
00407D89  mov         ecx,dword ptr ds:[596E08h]  
00407D8F  lea         edx,[ecx-1]  
00407D92  cmp         eax,edx  
00407D94  jb          00407DB2  
00407D96  cmp         dword ptr ds:[596E10h],eax  
00407D9C  jb          00407DB2  
00407D9E  mov         ecx,dword ptr ds:[5B89D0h]  
00407DA4  cmp         dword ptr [eax-4],ecx  
00407DA7  jb          00407DB2  
00407DA9  mov         eax,dword ptr ds:[00596E28h]  
---> 00407DAE  movzx       eax,byte ptr [eax]  
00407DB1  ret  

eax = 00000000
ecx = 04700000
</SNIP>

So my question I guess is:
How can we fix this? How can I help fix this? Because I've lost countless days of time and energy on this f***ing problem...

Peter

Reply | Threaded
Open this post in threaded view
|

Re: More Windows VM Crashing -_-

Nicolai Hess-3-2


2017-03-22 10:31 GMT+01:00 Peter Uhnak <[hidden email]>:
Hi,

Originally I thought my VM is crashing because of FT, so I applied some fix from FogBugz (that is now part of Pharo 6), but that didn't help.

So I disabled FT in settings and changed fonts to non-FT ones; didn't help.
So I completely removed FT2Plugin.dll (now settings just states that FT fonts are not available), but that didn't help either.

And obviously there is no crash.dmp whatsoever (this is latest Windows VM).

This crash is extremely frustrating because the crash happens _during saving_, which means I am losing code all the time and have to constantly recover them, which really makes my blood boil.

Without crash dump I don't even know where and how to start investigating. Windows kindly opens Visual Studio when Pharo crashes, but all it shows to me is assembly, which is not very useful to me. I can somewhat orient myself in the C code of the VM (as I've alredy fixed some other issues in the VM), but here I am lost.

<SNIP>
00407D7F  ret
00407D80  mov         eax,dword ptr ds:[00596E44h]
00407D85  test        al,3
00407D87  jne         00407DB2
00407D89  mov         ecx,dword ptr ds:[596E08h]
00407D8F  lea         edx,[ecx-1]
00407D92  cmp         eax,edx
00407D94  jb          00407DB2
00407D96  cmp         dword ptr ds:[596E10h],eax
00407D9C  jb          00407DB2
00407D9E  mov         ecx,dword ptr ds:[5B89D0h]
00407DA4  cmp         dword ptr [eax-4],ecx
00407DA7  jb          00407DB2
00407DA9  mov         eax,dword ptr ds:[00596E28h]
---> 00407DAE  movzx       eax,byte ptr [eax]
00407DB1  ret

eax = 00000000
ecx = 04700000
</SNIP>

So my question I guess is:
How can we fix this? How can I help fix this? Because I've lost countless days of time and energy on this f***ing problem...

Peter


Maybe it would help if other people try to reproduce this error.
Can you share the image/code you are using ?

Do you have a mingw build environment ? Sometimes it helps to start the vm with gdb, and if an exception occurs that does not write a crash.dmp, if you are lucky, gdb will show some additional information.


(for me, the latest windows vm *sometimes* does create a crash.dmp).



Reply | Threaded
Open this post in threaded view
|

Re: More Windows VM Crashing -_-

Eliot Miranda-2
In reply to this post by Peter Uhnak
Hi Peter,

> On Mar 22, 2017, at 2:31 AM, Peter Uhnak <[hidden email]> wrote:
>
> Hi,
>
> Originally I thought my VM is crashing because of FT, so I applied some fix from FogBugz (that is now part of Pharo 6), but that didn't help.
>
> So I disabled FT in settings and changed fonts to non-FT ones; didn't help.
> So I completely removed FT2Plugin.dll (now settings just states that FT fonts are not available), but that didn't help either.
>
> And obviously there is no crash.dmp whatsoever (this is latest Windows VM).
>
> This crash is extremely frustrating because the crash happens _during saving_, which means I am losing code all the time and have to constantly recover them, which really makes my blood boil.
>
> Without crash dump I don't even know where and how to start investigating. Windows kindly opens Visual Studio when Pharo crashes, but all it shows to me is assembly, which is not very useful to me. I can somewhat orient myself in the C code of the VM (as I've alredy fixed some other issues in the VM), but here I am lost.
>
> <SNIP>
> 00407D7F  ret  
> 00407D80  mov         eax,dword ptr ds:[00596E44h]  
> 00407D85  test        al,3  
> 00407D87  jne         00407DB2  
> 00407D89  mov         ecx,dword ptr ds:[596E08h]  
> 00407D8F  lea         edx,[ecx-1]  
> 00407D92  cmp         eax,edx  
> 00407D94  jb          00407DB2  
> 00407D96  cmp         dword ptr ds:[596E10h],eax  
> 00407D9C  jb          00407DB2  
> 00407D9E  mov         ecx,dword ptr ds:[5B89D0h]  
> 00407DA4  cmp         dword ptr [eax-4],ecx  
> 00407DA7  jb          00407DB2  
> 00407DA9  mov         eax,dword ptr ds:[00596E28h]  
> ---> 00407DAE  movzx       eax,byte ptr [eax]  
> 00407DB1  ret  
>
> eax = 00000000
> ecx = 04700000
> </SNIP>

Can you post a link to the exact VM you're using?  I may be able to make sense of this.

First, I'm very sorry you're experiencing these crashes.  They're likely a bug in the new compactor.  It makes multiple compaction passes when there are lots of objects, only on snapshot.  In a normal GC it only ever makes one pass.

Therefore, one thing that may allow you to work around the bug is to modify the snapshot:andQuit: method to explicitly do a Smalltalk garbageCollect before invoking the snapshot primitive.  If the bug is only on subsequent passes then the explicit GC should not crash and will make it much less likely that the GC done as part of the snapshot will require a second pass, and hence less likely to crash.

> So my question I guess is:
> How can we fix this? How can I help fix this? Because I've lost countless days of time and energy on this f***ing problem...

Helping me match the code above with the VM will be really helpful.  I want to know the values in registers, and the value of nilObj too.

> Peter

_,,,^..^,,,_ (phone)
Reply | Threaded
Open this post in threaded view
|

Re: More Windows VM Crashing -_-

Peter Uhnak
On Wed, Mar 22, 2017 at 07:06:30AM -0700, Eliot Miranda wrote:

> Hi Peter,
>
> > On Mar 22, 2017, at 2:31 AM, Peter Uhnak <[hidden email]> wrote:
> >
> > Hi,
> >
> > Originally I thought my VM is crashing because of FT, so I applied some fix from FogBugz (that is now part of Pharo 6), but that didn't help.
> >
> > So I disabled FT in settings and changed fonts to non-FT ones; didn't help.
> > So I completely removed FT2Plugin.dll (now settings just states that FT fonts are not available), but that didn't help either.
> >
> > And obviously there is no crash.dmp whatsoever (this is latest Windows VM).
> >
> > This crash is extremely frustrating because the crash happens _during saving_, which means I am losing code all the time and have to constantly recover them, which really makes my blood boil.
> >
> > Without crash dump I don't even know where and how to start investigating. Windows kindly opens Visual Studio when Pharo crashes, but all it shows to me is assembly, which is not very useful to me. I can somewhat orient myself in the C code of the VM (as I've alredy fixed some other issues in the VM), but here I am lost.
> >
> > <SNIP>
> > 00407D7F  ret  
> > 00407D80  mov         eax,dword ptr ds:[00596E44h]  
> > 00407D85  test        al,3  
> > 00407D87  jne         00407DB2  
> > 00407D89  mov         ecx,dword ptr ds:[596E08h]  
> > 00407D8F  lea         edx,[ecx-1]  
> > 00407D92  cmp         eax,edx  
> > 00407D94  jb          00407DB2  
> > 00407D96  cmp         dword ptr ds:[596E10h],eax  
> > 00407D9C  jb          00407DB2  
> > 00407D9E  mov         ecx,dword ptr ds:[5B89D0h]  
> > 00407DA4  cmp         dword ptr [eax-4],ecx  
> > 00407DA7  jb          00407DB2  
> > 00407DA9  mov         eax,dword ptr ds:[00596E28h]  
> > ---> 00407DAE  movzx       eax,byte ptr [eax]  
> > 00407DB1  ret  
> >
> > eax = 00000000
> > ecx = 04700000
> > </SNIP>
>
> Can you post a link to the exact VM you're using?  I may be able to make sense of this.

I'm using the latest pharo-win-latest.zip from here http://files.pharo.org/get-files/60/
(2017-03-10 17:16 ... alternative link https://drive.google.com/open?id=0B6_V7NFdYhuWNkdnLTZPRjFlTEk )


Win32 built on Mar 10 2017 16:11:13 CUT Compiler: 4.9.2
VMMaker versionString VM: 201703101446 https://github.com/pharo-project/pharo-vm.git $ Date: Fri Mar 10 15:46:52 2017 +0100 $ Plugins: 201703101446 https://github.com/pharo-project/pharo-vm.git $
CoInterpreter * VMMaker.oscog-eem.2146 uuid: da6aee6e-4738-4161-b221-3c8135ac1bf7 Mar 10 2017
StackToRegisterMappingCogit * VMMaker.oscog-eem.2146 uuid: da6aee6e-4738-4161-b221-3c8135ac1bf7 Mar 10 2017

Pharo5.0
Latest update: #50770
Unnamed


>
> First, I'm very sorry you're experiencing these crashes.  They're likely a bug in the new compactor.  It makes multiple compaction passes when there are lots of objects, only on snapshot.  In a normal GC it only ever makes one pass.
>
> Therefore, one thing that may allow you to work around the bug is to modify the snapshot:andQuit: method to explicitly do a Smalltalk garbageCollect before invoking the snapshot primitive.  If the bug is only on subsequent passes then the explicit GC should not crash and will make it much less likely that the GC done as part of the snapshot will require a second pass, and hence less likely to crash.
>

Thanks, I will try this and see if it improves.

> > So my question I guess is:
> > How can we fix this? How can I help fix this? Because I've lost countless days of time and energy on this f***ing problem...
>
> Helping me match the code above with the VM will be really helpful.  I want to know the values in registers, and the value of nilObj too.

registers:
EAX=00000000 EBX=00ACCA44 ECX=04700000 EDX=00ACD250 ESI=00ACCA44 EDI=00000000 EIP=00407DAE ESP=00ACC0F4 EBP=00ACCA14 EFL=00010202

In the meantime I will try to figure out how to build the VM myself on Windows ...


The disassembly doesn't seem to know what is on the upper addresses, but here's a bit longer snippet

<SNIP>
... lots of ?? ?? ...
00407C67  ?? ??
00407C68  ?? ??
00407C69  add         al,0  
00407C6B  add         byte ptr [eax],al  
00407C6D  add         byte ptr [ebx+596E4C15h],cl  
00407C73  add         byte ptr [ebx+24048902h],cl  
00407C79  call        00483351  
00407C7E  add         esp,1Ch  
00407C81  ret  
00407C82  mov         eax,596E44h  
00407C87  ret  
00407C88  push        esi  
00407C89  push        ebx  
00407C8A  sub         esp,14h  
00407C8D  mov         eax,dword ptr [esp+24h]  
00407C91  xor         ebx,ebx  
00407C93  cmp         eax,23Fh  
00407C98  jg          00407CB1  
00407C9A  mov         ebx,dword ptr [eax*4+575580h]  
00407CA1  cmp         ebx,406866h  
00407CA7  je          00407CB9  
00407CA9  cmp         ebx,43DF10h  
00407CAF  je          00407CD1  
00407CB1  mov         eax,ebx  
00407CB3  add         esp,14h  
00407CB6  pop         ebx  
00407CB7  pop         esi  
00407CB8  ret  
00407CB9  mov         eax,dword ptr ds:[00565084h]  
00407CBE  cmp         eax,0FFFFFFFFh  
00407CC1  je          00407D50  
00407CC7  mov         ebx,eax  
00407CC9  mov         eax,ebx  
00407CCB  add         esp,14h  
00407CCE  pop         ebx  
00407CCF  pop         esi  
00407CD0  ret  
00407CD1  mov         dword ptr [esp],484C4Ah  
00407CD8  call        00480E39  
00407CDD  mov         edx,dword ptr [esp+20h]  
00407CE1  mov         ecx,dword ptr [edx+8]  
00407CE4  mov         eax,ecx  
00407CE6  and         ecx,1  
00407CE9  je          00407D4B  
00407CEB  test        eax,0FFFEh  
00407CF0  je          00407D70  
00407CF2  mov         ebx,dword ptr [esp+20h]  
00407CF6  mov         esi,dword ptr [ebx+0Ch]  
00407CF9  mov         ebx,43DF10h  
00407CFE  test        esi,3  
00407D04  jne         00407CB1  
00407D06  movzx       edx,byte ptr [esi+3]  
00407D0A  and         edx,1Fh  
00407D0D  cmp         edx,2  
00407D10  jne         00407CB1  
00407D12  mov         edx,2  
00407D17  mov         eax,esi  
00407D19  call        004024F8  
00407D1E  cmp         eax,4  
00407D21  jne         00407CB1  
00407D23  mov         eax,dword ptr [esi+14h]  
00407D26  test        al,1  
00407D28  je          00407CB1  
00407D2A  sar         eax,1  
00407D2C  sub         eax,1  
00407D2F  cmp         eax,0FFFh  
00407D34  ja          00407CB1  
00407D3A  mov         ecx,dword ptr [eax*4+58B080h]  
00407D41  test        ecx,ecx  
00407D43  cmovne      ebx,ecx  
00407D46  jmp         00407CB1  
00407D4B  mov         eax,dword ptr [eax+14h]  
00407D4E  jmp         00407CEB  
00407D50  mov         dword ptr [esp+4],5721E5h  
00407D58  mov         dword ptr [esp],5721F4h  
00407D5F  call        004978A4  
00407D64  mov         dword ptr ds:[00565084h],eax  
00407D69  mov         ebx,eax  
00407D6B  jmp         00407CC9  
00407D70  mov         ebx,43DF10h  
00407D75  jmp         00407CB1  
00407D7A  mov         eax,dword ptr ds:[005B8A00h]  
00407D7F  ret  
00407D80  mov         eax,dword ptr ds:[00596E44h]  
00407D85  test        al,3  
00407D87  jne         00407DB2  
00407D89  mov         ecx,dword ptr ds:[596E08h]  
00407D8F  lea         edx,[ecx-1]  
00407D92  cmp         eax,edx  
00407D94  jb          00407DB2  
00407D96  cmp         dword ptr ds:[596E10h],eax  
00407D9C  jb          00407DB2  
00407D9E  mov         ecx,dword ptr ds:[5B89D0h]  
00407DA4  cmp         dword ptr [eax-4],ecx  
00407DA7  jb          00407DB2  
00407DA9  mov         eax,dword ptr ds:[00596E28h]  
----> 00407DAE  movzx       eax,byte ptr [eax]   <-----
00407DB1  ret  
</SNIP>

Peter

Reply | Threaded
Open this post in threaded view
|

Re: More Windows VM Crashing -_-

EstebanLM
In reply to this post by Peter Uhnak
Hi,

I do not think this is a windows problem.
At least, I had some crashes when performing compaction last days too… this is a crash I collected 10 mins ago, in a mac.

cheers,
Esteban




> On 22 Mar 2017, at 10:31, Peter Uhnak <[hidden email]> wrote:
>
> Hi,
>
> Originally I thought my VM is crashing because of FT, so I applied some fix from FogBugz (that is now part of Pharo 6), but that didn't help.
>
> So I disabled FT in settings and changed fonts to non-FT ones; didn't help.
> So I completely removed FT2Plugin.dll (now settings just states that FT fonts are not available), but that didn't help either.
>
> And obviously there is no crash.dmp whatsoever (this is latest Windows VM).
>
> This crash is extremely frustrating because the crash happens _during saving_, which means I am losing code all the time and have to constantly recover them, which really makes my blood boil.
>
> Without crash dump I don't even know where and how to start investigating. Windows kindly opens Visual Studio when Pharo crashes, but all it shows to me is assembly, which is not very useful to me. I can somewhat orient myself in the C code of the VM (as I've alredy fixed some other issues in the VM), but here I am lost.
>
> <SNIP>
> 00407D7F  ret  
> 00407D80  mov         eax,dword ptr ds:[00596E44h]  
> 00407D85  test        al,3  
> 00407D87  jne         00407DB2  
> 00407D89  mov         ecx,dword ptr ds:[596E08h]  
> 00407D8F  lea         edx,[ecx-1]  
> 00407D92  cmp         eax,edx  
> 00407D94  jb          00407DB2  
> 00407D96  cmp         dword ptr ds:[596E10h],eax  
> 00407D9C  jb          00407DB2  
> 00407D9E  mov         ecx,dword ptr ds:[5B89D0h]  
> 00407DA4  cmp         dword ptr [eax-4],ecx  
> 00407DA7  jb          00407DB2  
> 00407DA9  mov         eax,dword ptr ds:[00596E28h]  
> ---> 00407DAE  movzx       eax,byte ptr [eax]  
> 00407DB1  ret  
>
> eax = 00000000
> ecx = 04700000
> </SNIP>
>
> So my question I guess is:
> How can we fix this? How can I help fix this? Because I've lost countless days of time and energy on this f***ing problem...
>
> Peter
>


crash.dmp (96K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More Windows VM Crashing -_-

Aliaksei Syrel
Hi all,

I got a strange crash on Mac, never saw any similar crash dump.

Image: 6.0 #60447

5.0  Mac OS X built on Mar 10 2017 13:46:32 UTC Compiler: 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31) [Production Spur VM]
CoInterpreter * VMMaker.oscog-eem.2146 uuid: da6aee6e-4738-4161-b221-3c8135ac1bf7 Mar 10 2017
StackToRegisterMappingCogit * VMMaker.oscog-eem.2146 uuid: da6aee6e-4738-4161-b221-3c8135ac1bf7 Mar 10 2017
VM: 201703101242 https://github.com/pharo-project/pharo-vm.git $ Date: Fri Mar 10 13:42:34 2017 +0100 $
Plugins: 201703101242 https://github.com/pharo-project/pharo-vm.git $


First few lines of crash (the whole dump is in attachment):

C stack backtrace & registers:
eax 0x849671d0 ebx 0x7c6e3820 ecx 0x90301065 edx 0xe0000000
edi 0x7c6e3820 esi 0x7c6e3820 ebp 0xbff40708 esp 0xbff404c8
eip 0x9ef29013
0   libobjc.A.dylib                     0x9ef29013 objc_msgSend + 19
1   Pharo                               0x0010227a reportStackState + 706
2   Pharo                               0x001025d2 sigsegv + 113
3   libsystem_platform.dylib            0x9f98bebb _sigtramp + 43
4   ???                                 0xffffffff 0x0 + 4294967295
5   CoreFoundation                      0x94671888 _CFAutoreleasePoolPop + 24
6   HIToolbox                           0x9401fa35 IMKInputSessionProcessEventRefWithCompletionHandler + 125
7   HIToolbox                           0x9401ebd2 InputMethodInstanceProcessEventRef_WithCompletionHandler + 135
8   HIToolbox                           0x94006c9c __TSMEventToInputMethod_WithCompletionHandler_block_invoke + 121
9   HIToolbox                           0x9400b3fa __TrySendLockEvent_BeforeEventToInputMethod_WithContinuationHandler_block_invoke + 27
10  HIToolbox                           0x9400b496 __SendTSMDocumentLockEvent_WithCompletionHandler_block_invoke + 90
11  HIToolbox                           0x93ddc6a3 __SendTSMEvent_WithCompletionHandler_block_invoke + 66
12  HIToolbox                           0x93ddf651 __SendEventToEventTargetWithCompletionHandler_block_invoke + 22


Cheers,
Alex

On 22 March 2017 at 21:26, Esteban Lorenzano <[hidden email]> wrote:
Hi,

I do not think this is a windows problem.
At least, I had some crashes when performing compaction last days too… this is a crash I collected 10 mins ago, in a mac.

cheers,
Esteban




> On 22 Mar 2017, at 10:31, Peter Uhnak <[hidden email]> wrote:
>
> Hi,
>
> Originally I thought my VM is crashing because of FT, so I applied some fix from FogBugz (that is now part of Pharo 6), but that didn't help.
>
> So I disabled FT in settings and changed fonts to non-FT ones; didn't help.
> So I completely removed FT2Plugin.dll (now settings just states that FT fonts are not available), but that didn't help either.
>
> And obviously there is no crash.dmp whatsoever (this is latest Windows VM).
>
> This crash is extremely frustrating because the crash happens _during saving_, which means I am losing code all the time and have to constantly recover them, which really makes my blood boil.
>
> Without crash dump I don't even know where and how to start investigating. Windows kindly opens Visual Studio when Pharo crashes, but all it shows to me is assembly, which is not very useful to me. I can somewhat orient myself in the C code of the VM (as I've alredy fixed some other issues in the VM), but here I am lost.
>
> <SNIP>
> 00407D7F  ret
> 00407D80  mov         eax,dword ptr ds:[00596E44h]
> 00407D85  test        al,3
> 00407D87  jne         00407DB2
> 00407D89  mov         ecx,dword ptr ds:[596E08h]
> 00407D8F  lea         edx,[ecx-1]
> 00407D92  cmp         eax,edx
> 00407D94  jb          00407DB2
> 00407D96  cmp         dword ptr ds:[596E10h],eax
> 00407D9C  jb          00407DB2
> 00407D9E  mov         ecx,dword ptr ds:[5B89D0h]
> 00407DA4  cmp         dword ptr [eax-4],ecx
> 00407DA7  jb          00407DB2
> 00407DA9  mov         eax,dword ptr ds:[00596E28h]
> ---> 00407DAE  movzx       eax,byte ptr [eax]
> 00407DB1  ret
>
> eax = 00000000
> ecx = 04700000
> </SNIP>
>
> So my question I guess is:
> How can we fix this? How can I help fix this? Because I've lost countless days of time and energy on this f***ing problem...
>
> Peter
>




crash.dmp (15K) Download Attachment