corruption of PC in context objects or not (?)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

corruption of PC in context objects or not (?)

Andrei Chis
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption? 


 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei



Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Eliot Miranda-2
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
Special-case non-single contexts (because of context-to-stack mapping).
Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
 or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Andrei Chis
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.


On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue:
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue:
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue:
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
Special-case non-single contexts (because of context-to-stack mapping).
Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
 or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)

    frame #0: 0x00000001100b1004

->  0x1100b1004: inl    $0x4c, %eax

    0x1100b1006: leal   -0x5c(%rip), %eax

    0x1100b100c: pushq  %r8

    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 

Target 0: (GlamorousToolkit) stopped.



Process 29929 stopped

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)

    frame #0: 0x00000001100fe7ed

->  0x1100fe7ed: int3   

    0x1100fe7ee: int3   

    0x1100fe7ef: int3   

    0x1100fe7f0: int3   

Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Eliot Miranda-2
 
Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue:
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue:
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue:
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
Special-case non-single contexts (because of context-to-stack mapping).
Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
 or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)

    frame #0: 0x00000001100b1004

->  0x1100b1004: inl    $0x4c, %eax

    0x1100b1006: leal   -0x5c(%rip), %eax

    0x1100b100c: pushq  %r8

    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 

Target 0: (GlamorousToolkit) stopped.



Process 29929 stopped

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)

    frame #0: 0x00000001100fe7ed

->  0x1100fe7ed: int3   

    0x1100fe7ee: int3   

    0x1100fe7ef: int3   

    0x1100fe7f0: int3   

Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





--
_,,,^..^,,,_
best, Eliot


--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Andrei Chis
 
Hi Eliot,

On 12 Sep 2020, at 01:42, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

I don’t think there is something special about that process. It is the process that we start to run tests [1]. The exception happens in the running process and the crash is when copying the stack of that running process.

Checked some previous logs and we get these kinds of crashes on the CI server since at least two years. So it does not look like a new bug (but who knows).


(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Nice!

I also tried the non-recursive implementation of Context>>#copyTo: from Squeak and it also crashes.

Not sure if related but now in the same image as before I got a different crash and printing the stack does not work. But this time the error seems to come from handleStackOverflow

(lldb) call (void *)printCallStack()
invalid frame pointer
invalid frame pointer
invalid frame pointer
error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=EXC_I386_GPFLT).
The process has been returned to the state before expression evaluation.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x121e00000)
  * frame #0: 0x0000000100162258 libGlamorousToolkitVMCore.dylib`marryFrameSP + 584
    frame #1: 0x0000000100172982 libGlamorousToolkitVMCore.dylib`handleStackOverflow + 354
    frame #2: 0x000000010016b025 libGlamorousToolkitVMCore.dylib`ceStackOverflow + 149
    frame #3: 0x00000001100005b3
    frame #4: 0x0000000100174d99 libGlamorousToolkitVMCore.dylib`ptEnterInterpreterFromCallback + 73


Cheers,
Andrei

[1] ./GlamorousToolkit.app/Contents/MacOS/GlamorousToolkit  Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'



Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue: 
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue: 
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue: 
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
 Special-case non-single contexts (because of context-to-stack mapping).
 Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
  or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001100b1004
->  0x1100b1004: inl    $0x4c, %eax
    0x1100b1006: leal   -0x5c(%rip), %eax
    0x1100b100c: pushq  %r8
    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 
Target 0: (GlamorousToolkit) stopped.


Process 29929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001100fe7ed
->  0x1100fe7ed: int3   
    0x1100fe7ee: int3   
    0x1100fe7ef: int3   
    0x1100fe7f0: int3   
Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





-- 
_,,,^..^,,,_
best, Eliot


-- 
_,,,^..^,,,_
best, Eliot

Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Eliot Miranda-2
 
Hi Andrei,


On Sep 14, 2020, at 7:15 AM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

On 12 Sep 2020, at 01:42, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

I don’t think there is something special about that process. It is the process that we start to run tests [1]. The exception happens in the running process and the crash is when copying the stack of that running process.

Ok, cool.  What I’d like to do is get a copy of your test setup and run it in an assert vm to try and get more information.  AFAICT the vm code is good do the bug is not obvious.  An assert vm may give more information before the crash.  Have you tried running the system on an assert vm yet?

Checked some previous logs and we get these kinds of crashes on the CI server since at least two years. So it does not look like a new bug (but who knows).


(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Nice!

I also tried the non-recursive implementation of Context>>#copyTo: from Squeak and it also crashes.

Not sure if related but now in the same image as before I got a different crash and printing the stack does not work. But this time the error seems to come from handleStackOverflow

(lldb) call (void *)printCallStack()
invalid frame pointer
invalid frame pointer
invalid frame pointer
error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=EXC_I386_GPFLT).
The process has been returned to the state before expression evaluation.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x121e00000)
  * frame #0: 0x0000000100162258 libGlamorousToolkitVMCore.dylib`marryFrameSP + 584
    frame #1: 0x0000000100172982 libGlamorousToolkitVMCore.dylib`handleStackOverflow + 354
    frame #2: 0x000000010016b025 libGlamorousToolkitVMCore.dylib`ceStackOverflow + 149
    frame #3: 0x00000001100005b3
    frame #4: 0x0000000100174d99 libGlamorousToolkitVMCore.dylib`ptEnterInterpreterFromCallback + 73


Cheers,
Andrei

[1] ./GlamorousToolkit.app/Contents/MacOS/GlamorousToolkit  Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'



Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue: 
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue: 
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue: 
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
 Special-case non-single contexts (because of context-to-stack mapping).
 Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
  or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001100b1004
->  0x1100b1004: inl    $0x4c, %eax
    0x1100b1006: leal   -0x5c(%rip), %eax
    0x1100b100c: pushq  %r8
    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 
Target 0: (GlamorousToolkit) stopped.


Process 29929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001100fe7ed
->  0x1100fe7ed: int3   
    0x1100fe7ee: int3   
    0x1100fe7ef: int3   
    0x1100fe7f0: int3   
Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





-- 
_,,,^..^,,,_
best, Eliot


-- 
_,,,^..^,,,_
best, Eliot

Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Andrei Chis
 
Hi Eliot,

The setup in GT is a bit customised (some changes in the headless vm, some custom plugins, custom rendering) so I first thought it will be impossible to reproduce the bug in a more standard manner. 
However turns out it is possible. If I use the following script after running the tests a few times in lldb I get the crash starting from a plain Pharo 8 image.

$ ./pharo Pharo.image st --quit

$ lldb ./pharo-vm/Pharo.app/Contents/MacOS/Pharo
(lldb) run --headless Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'


I also tried to compile the vm myself on Mac (Catalina 10.15.6). I build a normal and assert for https://github.com/OpenSmalltalk/opensmalltalk-vm and https://github.com/pharo-project/opensmalltalk-vm from the cog branch.
In both cases I get an issue related to pixman 0.34.0 [1] but that’s easy to workaround. For https://github.com/OpenSmalltalk/opensmalltalk-vm I got an extra problem related to Cairo [2] and had to change libpng from libpng16 to libpng12 to get it to work.

With both the normal VMs I could reproduce the bug and got stacks with the Context>copyTo: messages. 

With the assert VMs I only got a crash for now with the assert vm from https://github.com/pharo-project/opensmalltalk-vm. However there is no Context>copyTo: and the memory seems quite corrupted. 
I suspect the crash also appears in https://github.com/OpenSmalltalk/opensmalltalk-vm  but seems that with the assert vm it is much harder to reproduce. Had to run the tests 20 times and got one crash; running the tests once take 20-30 minutes. 


This is from only crash until now with the assert vm. Not sure if they are helpful or not, or actually related to the problem.

validInstructionPointerinFrame(GIV(instructionPointer), GIV(framePointer)) 18471
Pharo was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 73731 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
    frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
   139   static inline sqInt intAtPointer(char *ptr) { return (sqInt)(*((int *)ptr)); }
   140   static inline sqInt intAtPointerput(char *ptr, int val) { return (sqInt)(*((int *)ptr)= val); }
   141   static inline sqInt longAtPointer(char *ptr) { return *(sqInt *)ptr; }
-> 142   static inline sqInt longAtPointerput(char *ptr, sqInt val) { return *(sqInt *)ptr= val; }
   143   static inline sqLong long64AtPointer(char *ptr) { return *(sqLong *)ptr; }
   144   static inline sqLong long64AtPointerput(char *ptr, sqLong val) { return *(sqLong *)ptr= val; }
   145   static inline float singleFloatAtPointer(char *ptr) { return *(float *)ptr; }
Target 0: (Pharo) stopped.


(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
  * frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
    frame #1: 0x00000001000161cf Pharo`marryFrameSP(theFP=<unavailable>, theSP=0x0000000000000000) at gcc3x-cointerp.c:68120:3 [opt]
    frame #2: 0x000000010001f5ac Pharo`ceContextinstVar(maybeContext=5510359872, slotIndex=0) at gcc3x-cointerp.c:15221:12 [opt]
    frame #3: 0x00000001480017d6
    frame #4: 0x00000001000022be Pharo`interpret at gcc3x-cointerp.c:2755:3 [opt]
    frame #5: 0x00000001000bc244 Pharo`-[sqSqueakMainApplication runSqueak](self=0x0000000101c76dc0, _cmd=<unavailable>) at sqSqueakMainApplication.m:201:2 [opt]
    frame #6: 0x00007fff3326729b Foundation`__NSFirePerformWithOrder + 360
    frame #7: 0x00007fff30ad3335 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    frame #8: 0x00007fff30ad3267 CoreFoundation`__CFRunLoopDoObservers + 457
    frame #9: 0x00007fff30ad2805 CoreFoundation`__CFRunLoopRun + 874
    frame #10: 0x00007fff30ad1e3e CoreFoundation`CFRunLoopRunSpecific + 462
    frame #11: 0x00007fff2f6feabd HIToolbox`RunCurrentEventLoopInMode + 292
    frame #12: 0x00007fff2f6fe6f4 HIToolbox`ReceiveNextEventCommon + 359
    frame #13: 0x00007fff2f6fe579 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 64
    frame #14: 0x00007fff2dd44039 AppKit`_DPSNextEvent + 883
    frame #15: 0x00007fff2dd42880 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
    frame #16: 0x00007fff2dd3458e AppKit`-[NSApplication run] + 658
    frame #17: 0x00007fff2dd06396 AppKit`NSApplicationMain + 777
    frame #18: 0x00007fff6ab3ecc9 libdyld.dylib`start + 1


(lldb) call printCallStack()
    0x7ffeefbe3920 M INVALID RECEIVER>(nil) 0x148716b40: a(n) bad class
    0x7ffeefbe3968 M [] in INVALID RECEIVER>(nil) Context(Object)>>doesNotUnderstand: #bounds
 0x194648118: a(n) bad class
    0x7ffeefbe39a8 M INVALID RECEIVER>(nil) 0x1489fcec0: a(n) bad class
    0x7ffeefbe39e8 M INVALID RECEIVER>(nil)  0x1489fcec0: a(n) bad class
    0x7ffeefbe3a30 I INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3a80 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3ab8 M INVALID RECEIVER>(nil)  0x148163cd0: a(n) bad class
    0x7ffeefbe3b08 I INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b40 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b78 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe3bc0 I INVALID RECEIVER>(nil)  0x148716a38: a(n) bad class
    0x7ffeefbe3c10 I INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c40 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c78 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3cc0 M INVALID RECEIVER>(nil)  0x14d0337f0: a(n) bad class
    0x7ffeefbe3d08 M INVALID RECEIVER>(nil)  0x14d033738: a(n) bad class
    0x7ffeefbe3d50 M INVALID RECEIVER>(nil)  0x14d033680: a(n) bad class
    0x7ffeefbe3d98 M INVALID RECEIVER>(nil)  0x1946493f0: a(n) bad class
    0x7ffeefbe3de0 M INVALID RECEIVER>(nil)  0x194649338: a(n) bad class
    0x7ffeefbe3e28 M INVALID RECEIVER>(nil)  0x194649280: a(n) bad class
    0x7ffeefbe3e70 M INVALID RECEIVER>(nil)  0x1946491c8: a(n) bad class
    0x7ffeefbe3eb8 M INVALID RECEIVER>(nil)  0x194649110: a(n) bad class
    0x7ffeefbec768 M INVALID RECEIVER>(nil)  0x194649038: a(n) bad class
    0x7ffeefbec7b0 M INVALID RECEIVER>(nil)  0x194648f60: a(n) bad class
    0x7ffeefbec7f8 M INVALID RECEIVER>(nil)  0x194648e88: a(n) bad class
    0x7ffeefbec840 M INVALID RECEIVER>(nil)  0x194648dd0: a(n) bad class
    0x7ffeefbec888 M INVALID RECEIVER>(nil)  0x194648d18: a(n) bad class
    0x7ffeefbec8d0 M INVALID RECEIVER>(nil)  0x194648c60: a(n) bad class
    0x7ffeefbec918 M INVALID RECEIVER>(nil)  0x194648b88: a(n) bad class
    0x7ffeefbec960 M INVALID RECEIVER>(nil)  0x194648ad0: a(n) bad class
    0x7ffeefbec9a8 M INVALID RECEIVER>(nil)  0x194648a18: a(n) bad class
    0x7ffeefbec9f0 M INVALID RECEIVER>(nil)  0x194648960: a(n) bad class
    0x7ffeefbeca38 M INVALID RECEIVER>(nil)  0x1946488a8: a(n) bad class
    0x7ffeefbeca80 M INVALID RECEIVER>(nil)  0x1946487f0: a(n) bad class
    0x7ffeefbecac8 M INVALID RECEIVER>(nil)  0x194648708: a(n) bad class
    0x7ffeefbecb10 M INVALID RECEIVER>(nil)  0x194648620: a(n) bad class
    0x7ffeefbecb58 M INVALID RECEIVER>(nil)  0x194648508: a(n) bad class
    0x7ffeefbecba0 M INVALID RECEIVER>(nil)  0x194648450: a(n) bad class
    0x7ffeefbecbe8 M INVALID RECEIVER>(nil)  0x1481641a8: a(n) bad class
    0x7ffeefbecc30 M INVALID RECEIVER>(nil)  0x1481640f0: a(n) bad class
    0x7ffeefbecc78 M INVALID RECEIVER>(nil)  0x148164038: a(n) bad class
    0x7ffeefbeccc0 M INVALID RECEIVER>(nil)  0x148163f80: a(n) bad class
    0x7ffeefbecd08 M INVALID RECEIVER>(nil)  0x148163ec8: a(n) bad class
    0x7ffeefbecd50 M INVALID RECEIVER>(nil)  0x148163e10: a(n) bad class
    0x7ffeefbecd98 M INVALID RECEIVER>(nil)  0x148163d28: a(n) bad class
    0x7ffeefbecde0 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbece28 M INVALID RECEIVER>(nil)  0x148163b38: a(n) bad class
    0x7ffeefbece70 M INVALID RECEIVER>(nil)  0x148163a80: a(n) bad class
    0x7ffeefbeceb8 M INVALID RECEIVER>(nil)  0x1481639c8: a(n) bad class
    0x7ffeefbe7758 M INVALID RECEIVER>(nil)  0x1481638e0: a(n) bad class
    0x7ffeefbe77a0 M INVALID RECEIVER>(nil)  0x148163808: a(n) bad class
    0x7ffeefbe77e8 M INVALID RECEIVER>(nil)  0x148163750: a(n) bad class
    0x7ffeefbe7830 M INVALID RECEIVER>(nil)  0x148163698: a(n) bad class
    0x7ffeefbe7878 M INVALID RECEIVER>(nil)  0x1481635c0: a(n) bad class
    0x7ffeefbe78c0 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe7908 M INVALID RECEIVER>(nil)  0x148163408: a(n) bad class
    0x7ffeefbe7950 M INVALID RECEIVER>(nil)  0x148163350: a(n) bad class
    0x7ffeefbe7998 M INVALID RECEIVER>(nil)  0x148163298: a(n) bad class
    0x7ffeefbe79e0 M INVALID RECEIVER>(nil)  0x148163188: a(n) bad class
    0x7ffeefbe7a28 M INVALID RECEIVER>(nil)  0x148163098: a(n) bad class
    0x7ffeefbe7a70 M INVALID RECEIVER>(nil)  0x148162fa0: a(n) bad class
    0x7ffeefbe7ab8 M INVALID RECEIVER>(nil)  0x148162ec8: a(n) bad class
    0x7ffeefbe7b00 M INVALID RECEIVER>(nil)  0x148162e10: a(n) bad class
    0x7ffeefbe7b48 M INVALID RECEIVER>(nil)  0x148712a08: a(n) bad class
    0x7ffeefbe7b90 M INVALID RECEIVER>(nil)  0x148712950: a(n) bad class
    0x7ffeefbe7bd8 M INVALID RECEIVER>(nil)  0x148712898: a(n) bad class
    0x7ffeefbe7c20 M INVALID RECEIVER>(nil)  0x148713cc0: a(n) bad class
    0x7ffeefbe7c68 M INVALID RECEIVER>(nil)  0x148713018: a(n) bad class
    0x7ffeefbe7cb0 M INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbe7cf8 M INVALID RECEIVER>(nil)  0x148713140: a(n) bad class
    0x7ffeefbe7d40 M INVALID RECEIVER>(nil)  0x148713928: a(n) bad class
    0x7ffeefbe7d88 M INVALID RECEIVER>(nil)  0x1487133c8: a(n) bad class
    0x7ffeefbe7de0 I INVALID RECEIVER>(nil)  0x148713238: a(n) bad class
    0x7ffeefbe7e28 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7e70 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7eb8 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeea68 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeac8 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeb00 M INVALID RECEIVER>(nil)  0x148713108: a(n) bad class
    0x7ffeefbeeb50 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeeb98 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeebe0 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbeec10 M INVALID RECEIVER>(nil) 0x9=1
    0x7ffeefbeec48 M INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeeca0 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeecd0 M INVALID RECEIVER>(nil)  0x1487130d0: a(n) bad class
    0x7ffeefbeed10 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeed58 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeedb0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeedf0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeee20 M [] in INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbeee78 M INVALID RECEIVER>(nil)  0x148162df0: a(n) bad class
    0x7ffeefbeeec0 M INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbe5978 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe59d8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5a18 M INVALID RECEIVER>(nil)  0x148163150: a(n) bad class
    0x7ffeefbe5a68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5aa0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5ad8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5b08 M INVALID RECEIVER>(nil)  0x1481634c0: a(n) bad class
    0x7ffeefbe5b48 M INVALID RECEIVER>(nil)  0x14c403ca8: a(n) bad class
    0x7ffeefbe5b88 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bc0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bf0 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c20 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c98 M INVALID RECEIVER>(nil)  0x194656468: a(n) bad class
    0x7ffeefbe5cd0 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d00 M INVALID RECEIVER>(nil)  0x148163bf0: a(n) bad class
    0x7ffeefbe5d50 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d88 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5dc0 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5e00 M INVALID RECEIVER>(nil)  0x148163de0: a(n) bad class
    0x7ffeefbe5e38 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5e80 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5eb8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdf9d8 M INVALID RECEIVER>(nil)  0x194648430: a(n) bad class
    0x7ffeefbdfa10 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfa50 M [] in INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfa90 M INVALID RECEIVER>(nil)  0x1946486d8: a(n) bad class
    0x7ffeefbdfad0 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfb10 M INVALID RECEIVER>(nil)  0x1946485e0: a(n) bad class
    0x7ffeefbdfb48 M INVALID RECEIVER>(nil)  0x1489f7da8: a(n) bad class
    0x7ffeefbdfb80 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfbb8 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfbe8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfc20 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc58 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc98 M INVALID RECEIVER>(nil)  0x194648c40: a(n) bad class
    0x7ffeefbdfcc8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd08 M INVALID RECEIVER>(nil)  0x194648f40: a(n) bad class
    0x7ffeefbdfd40 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd70 M INVALID RECEIVER>(nil)  0x14902a730: a(n) bad class
    0x7ffeefbdfdb0 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfde8 M INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe20 M [] in INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe70 M [] in INVALID RECEIVER>(nil)  0x14d032f98: a(n) bad class
    0x7ffeefbdfeb8 M INVALID RECEIVER>(nil)  0x14d032fe0: a(n) bad class

(callerContextOrNil == (nilObject())) || (isContext(callerContextOrNil)) 72783
       0x14d033738 is not a context


A more realistic setup would be to run GT with an assert headless vm. But until now I did not figure out how to build an assert vm for the gt-headless branch from https://github.com/feenkcom/opensmalltalk-vm

Cheers,
Andrei



[2] checking for cairo's PNG functions feature... 
configure: WARNING: Could not find libpng in the pkg-config search path
checking whether cairo's PNG functions feature could be enabled... no
configure: error: recommended PNG functions feature could not be enabled

On 14 Sep 2020, at 17:32, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 7:15 AM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

On 12 Sep 2020, at 01:42, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

I don’t think there is something special about that process. It is the process that we start to run tests [1]. The exception happens in the running process and the crash is when copying the stack of that running process.

Ok, cool.  What I’d like to do is get a copy of your test setup and run it in an assert vm to try and get more information.  AFAICT the vm code is good do the bug is not obvious.  An assert vm may give more information before the crash.  Have you tried running the system on an assert vm yet?

Checked some previous logs and we get these kinds of crashes on the CI server since at least two years. So it does not look like a new bug (but who knows).


(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Nice!

I also tried the non-recursive implementation of Context>>#copyTo: from Squeak and it also crashes.

Not sure if related but now in the same image as before I got a different crash and printing the stack does not work. But this time the error seems to come from handleStackOverflow

(lldb) call (void *)printCallStack()
invalid frame pointer
invalid frame pointer
invalid frame pointer
error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=EXC_I386_GPFLT).
The process has been returned to the state before expression evaluation.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x121e00000)
  * frame #0: 0x0000000100162258 libGlamorousToolkitVMCore.dylib`marryFrameSP + 584
    frame #1: 0x0000000100172982 libGlamorousToolkitVMCore.dylib`handleStackOverflow + 354
    frame #2: 0x000000010016b025 libGlamorousToolkitVMCore.dylib`ceStackOverflow + 149
    frame #3: 0x00000001100005b3
    frame #4: 0x0000000100174d99 libGlamorousToolkitVMCore.dylib`ptEnterInterpreterFromCallback + 73


Cheers,
Andrei

[1] ./GlamorousToolkit.app/Contents/MacOS/GlamorousToolkit  Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'



Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue: 
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue: 
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue: 
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
 Special-case non-single contexts (because of context-to-stack mapping).
 Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
  or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001100b1004
->  0x1100b1004: inl    $0x4c, %eax
    0x1100b1006: leal   -0x5c(%rip), %eax
    0x1100b100c: pushq  %r8
    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 
Target 0: (GlamorousToolkit) stopped.


Process 29929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001100fe7ed
->  0x1100fe7ed: int3   
    0x1100fe7ee: int3   
    0x1100fe7ef: int3   
    0x1100fe7f0: int3   
Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





-- 
_,,,^..^,,,_
best, Eliot


-- 
_,,,^..^,,,_
best, Eliot


Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Eliot Miranda-2
 
Hi Andrei,


On Sep 14, 2020, at 3:22 PM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

The setup in GT is a bit customised (some changes in the headless vm, some custom plugins, custom rendering) so I first thought it will be impossible to reproduce the bug in a more standard manner. 
However turns out it is possible. If I use the following script after running the tests a few times in lldb I get the crash starting from a plain Pharo 8 image.

$ ./pharo Pharo.image st --quit

$ lldb ./pharo-vm/Pharo.app/Contents/MacOS/Pharo
(lldb) run --headless Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'


I also tried to compile the vm myself on Mac (Catalina 10.15.6). I build a normal and assert for https://github.com/OpenSmalltalk/opensmalltalk-vm and https://github.com/pharo-project/opensmalltalk-vm from the cog branch.
In both cases I get an issue related to pixman 0.34.0 [1] but that’s easy to workaround. For https://github.com/OpenSmalltalk/opensmalltalk-vm I got an extra problem related to Cairo [2] and had to change libpng from libpng16 to libpng12 to get it to work.

With both the normal VMs I could reproduce the bug and got stacks with the Context>copyTo: messages. 

With the assert VMs I only got a crash for now with the assert vm from https://github.com/pharo-project/opensmalltalk-vm. However there is no Context>copyTo: and the memory seems quite corrupted. 
I suspect the crash also appears in https://github.com/OpenSmalltalk/opensmalltalk-vm  but seems that with the assert vm it is much harder to reproduce. Had to run the tests 20 times and got one crash; running the tests once take 20-30 minutes. 


This is from only crash until now with the assert vm. Not sure if they are helpful or not, or actually related to the problem.

validInstructionPointerinFrame(GIV(instructionPointer), GIV(framePointer)) 18471
Pharo was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 73731 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
    frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
   139   static inline sqInt intAtPointer(char *ptr) { return (sqInt)(*((int *)ptr)); }
   140   static inline sqInt intAtPointerput(char *ptr, int val) { return (sqInt)(*((int *)ptr)= val); }
   141   static inline sqInt longAtPointer(char *ptr) { return *(sqInt *)ptr; }
-> 142   static inline sqInt longAtPointerput(char *ptr, sqInt val) { return *(sqInt *)ptr= val; }
   143   static inline sqLong long64AtPointer(char *ptr) { return *(sqLong *)ptr; }
   144   static inline sqLong long64AtPointerput(char *ptr, sqLong val) { return *(sqLong *)ptr= val; }
   145   static inline float singleFloatAtPointer(char *ptr) { return *(float *)ptr; }
Target 0: (Pharo) stopped.


(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
  * frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
    frame #1: 0x00000001000161cf Pharo`marryFrameSP(theFP=<unavailable>, theSP=0x0000000000000000) at gcc3x-cointerp.c:68120:3 [opt]
    frame #2: 0x000000010001f5ac Pharo`ceContextinstVar(maybeContext=5510359872, slotIndex=0) at gcc3x-cointerp.c:15221:12 [opt]
    frame #3: 0x00000001480017d6
    frame #4: 0x00000001000022be Pharo`interpret at gcc3x-cointerp.c:2755:3 [opt]
    frame #5: 0x00000001000bc244 Pharo`-[sqSqueakMainApplication runSqueak](self=0x0000000101c76dc0, _cmd=<unavailable>) at sqSqueakMainApplication.m:201:2 [opt]
    frame #6: 0x00007fff3326729b Foundation`__NSFirePerformWithOrder + 360
    frame #7: 0x00007fff30ad3335 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    frame #8: 0x00007fff30ad3267 CoreFoundation`__CFRunLoopDoObservers + 457
    frame #9: 0x00007fff30ad2805 CoreFoundation`__CFRunLoopRun + 874
    frame #10: 0x00007fff30ad1e3e CoreFoundation`CFRunLoopRunSpecific + 462
    frame #11: 0x00007fff2f6feabd HIToolbox`RunCurrentEventLoopInMode + 292
    frame #12: 0x00007fff2f6fe6f4 HIToolbox`ReceiveNextEventCommon + 359
    frame #13: 0x00007fff2f6fe579 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 64
    frame #14: 0x00007fff2dd44039 AppKit`_DPSNextEvent + 883
    frame #15: 0x00007fff2dd42880 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
    frame #16: 0x00007fff2dd3458e AppKit`-[NSApplication run] + 658
    frame #17: 0x00007fff2dd06396 AppKit`NSApplicationMain + 777
    frame #18: 0x00007fff6ab3ecc9 libdyld.dylib`start + 1


(lldb) call printCallStack()
    0x7ffeefbe3920 M INVALID RECEIVER>(nil) 0x148716b40: a(n) bad class
    0x7ffeefbe3968 M [] in INVALID RECEIVER>(nil) Context(Object)>>doesNotUnderstand: #bounds
 0x194648118: a(n) bad class
    0x7ffeefbe39a8 M INVALID RECEIVER>(nil) 0x1489fcec0: a(n) bad class
    0x7ffeefbe39e8 M INVALID RECEIVER>(nil)  0x1489fcec0: a(n) bad class
    0x7ffeefbe3a30 I INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3a80 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3ab8 M INVALID RECEIVER>(nil)  0x148163cd0: a(n) bad class
    0x7ffeefbe3b08 I INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b40 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b78 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe3bc0 I INVALID RECEIVER>(nil)  0x148716a38: a(n) bad class
    0x7ffeefbe3c10 I INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c40 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c78 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3cc0 M INVALID RECEIVER>(nil)  0x14d0337f0: a(n) bad class
    0x7ffeefbe3d08 M INVALID RECEIVER>(nil)  0x14d033738: a(n) bad class
    0x7ffeefbe3d50 M INVALID RECEIVER>(nil)  0x14d033680: a(n) bad class
    0x7ffeefbe3d98 M INVALID RECEIVER>(nil)  0x1946493f0: a(n) bad class
    0x7ffeefbe3de0 M INVALID RECEIVER>(nil)  0x194649338: a(n) bad class
    0x7ffeefbe3e28 M INVALID RECEIVER>(nil)  0x194649280: a(n) bad class
    0x7ffeefbe3e70 M INVALID RECEIVER>(nil)  0x1946491c8: a(n) bad class
    0x7ffeefbe3eb8 M INVALID RECEIVER>(nil)  0x194649110: a(n) bad class
    0x7ffeefbec768 M INVALID RECEIVER>(nil)  0x194649038: a(n) bad class
    0x7ffeefbec7b0 M INVALID RECEIVER>(nil)  0x194648f60: a(n) bad class
    0x7ffeefbec7f8 M INVALID RECEIVER>(nil)  0x194648e88: a(n) bad class
    0x7ffeefbec840 M INVALID RECEIVER>(nil)  0x194648dd0: a(n) bad class
    0x7ffeefbec888 M INVALID RECEIVER>(nil)  0x194648d18: a(n) bad class
    0x7ffeefbec8d0 M INVALID RECEIVER>(nil)  0x194648c60: a(n) bad class
    0x7ffeefbec918 M INVALID RECEIVER>(nil)  0x194648b88: a(n) bad class
    0x7ffeefbec960 M INVALID RECEIVER>(nil)  0x194648ad0: a(n) bad class
    0x7ffeefbec9a8 M INVALID RECEIVER>(nil)  0x194648a18: a(n) bad class
    0x7ffeefbec9f0 M INVALID RECEIVER>(nil)  0x194648960: a(n) bad class
    0x7ffeefbeca38 M INVALID RECEIVER>(nil)  0x1946488a8: a(n) bad class
    0x7ffeefbeca80 M INVALID RECEIVER>(nil)  0x1946487f0: a(n) bad class
    0x7ffeefbecac8 M INVALID RECEIVER>(nil)  0x194648708: a(n) bad class
    0x7ffeefbecb10 M INVALID RECEIVER>(nil)  0x194648620: a(n) bad class
    0x7ffeefbecb58 M INVALID RECEIVER>(nil)  0x194648508: a(n) bad class
    0x7ffeefbecba0 M INVALID RECEIVER>(nil)  0x194648450: a(n) bad class
    0x7ffeefbecbe8 M INVALID RECEIVER>(nil)  0x1481641a8: a(n) bad class
    0x7ffeefbecc30 M INVALID RECEIVER>(nil)  0x1481640f0: a(n) bad class
    0x7ffeefbecc78 M INVALID RECEIVER>(nil)  0x148164038: a(n) bad class
    0x7ffeefbeccc0 M INVALID RECEIVER>(nil)  0x148163f80: a(n) bad class
    0x7ffeefbecd08 M INVALID RECEIVER>(nil)  0x148163ec8: a(n) bad class
    0x7ffeefbecd50 M INVALID RECEIVER>(nil)  0x148163e10: a(n) bad class
    0x7ffeefbecd98 M INVALID RECEIVER>(nil)  0x148163d28: a(n) bad class
    0x7ffeefbecde0 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbece28 M INVALID RECEIVER>(nil)  0x148163b38: a(n) bad class
    0x7ffeefbece70 M INVALID RECEIVER>(nil)  0x148163a80: a(n) bad class
    0x7ffeefbeceb8 M INVALID RECEIVER>(nil)  0x1481639c8: a(n) bad class
    0x7ffeefbe7758 M INVALID RECEIVER>(nil)  0x1481638e0: a(n) bad class
    0x7ffeefbe77a0 M INVALID RECEIVER>(nil)  0x148163808: a(n) bad class
    0x7ffeefbe77e8 M INVALID RECEIVER>(nil)  0x148163750: a(n) bad class
    0x7ffeefbe7830 M INVALID RECEIVER>(nil)  0x148163698: a(n) bad class
    0x7ffeefbe7878 M INVALID RECEIVER>(nil)  0x1481635c0: a(n) bad class
    0x7ffeefbe78c0 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe7908 M INVALID RECEIVER>(nil)  0x148163408: a(n) bad class
    0x7ffeefbe7950 M INVALID RECEIVER>(nil)  0x148163350: a(n) bad class
    0x7ffeefbe7998 M INVALID RECEIVER>(nil)  0x148163298: a(n) bad class
    0x7ffeefbe79e0 M INVALID RECEIVER>(nil)  0x148163188: a(n) bad class
    0x7ffeefbe7a28 M INVALID RECEIVER>(nil)  0x148163098: a(n) bad class
    0x7ffeefbe7a70 M INVALID RECEIVER>(nil)  0x148162fa0: a(n) bad class
    0x7ffeefbe7ab8 M INVALID RECEIVER>(nil)  0x148162ec8: a(n) bad class
    0x7ffeefbe7b00 M INVALID RECEIVER>(nil)  0x148162e10: a(n) bad class
    0x7ffeefbe7b48 M INVALID RECEIVER>(nil)  0x148712a08: a(n) bad class
    0x7ffeefbe7b90 M INVALID RECEIVER>(nil)  0x148712950: a(n) bad class
    0x7ffeefbe7bd8 M INVALID RECEIVER>(nil)  0x148712898: a(n) bad class
    0x7ffeefbe7c20 M INVALID RECEIVER>(nil)  0x148713cc0: a(n) bad class
    0x7ffeefbe7c68 M INVALID RECEIVER>(nil)  0x148713018: a(n) bad class
    0x7ffeefbe7cb0 M INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbe7cf8 M INVALID RECEIVER>(nil)  0x148713140: a(n) bad class
    0x7ffeefbe7d40 M INVALID RECEIVER>(nil)  0x148713928: a(n) bad class
    0x7ffeefbe7d88 M INVALID RECEIVER>(nil)  0x1487133c8: a(n) bad class
    0x7ffeefbe7de0 I INVALID RECEIVER>(nil)  0x148713238: a(n) bad class
    0x7ffeefbe7e28 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7e70 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7eb8 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeea68 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeac8 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeb00 M INVALID RECEIVER>(nil)  0x148713108: a(n) bad class
    0x7ffeefbeeb50 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeeb98 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeebe0 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbeec10 M INVALID RECEIVER>(nil) 0x9=1
    0x7ffeefbeec48 M INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeeca0 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeecd0 M INVALID RECEIVER>(nil)  0x1487130d0: a(n) bad class
    0x7ffeefbeed10 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeed58 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeedb0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeedf0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeee20 M [] in INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbeee78 M INVALID RECEIVER>(nil)  0x148162df0: a(n) bad class
    0x7ffeefbeeec0 M INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbe5978 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe59d8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5a18 M INVALID RECEIVER>(nil)  0x148163150: a(n) bad class
    0x7ffeefbe5a68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5aa0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5ad8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5b08 M INVALID RECEIVER>(nil)  0x1481634c0: a(n) bad class
    0x7ffeefbe5b48 M INVALID RECEIVER>(nil)  0x14c403ca8: a(n) bad class
    0x7ffeefbe5b88 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bc0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bf0 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c20 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c98 M INVALID RECEIVER>(nil)  0x194656468: a(n) bad class
    0x7ffeefbe5cd0 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d00 M INVALID RECEIVER>(nil)  0x148163bf0: a(n) bad class
    0x7ffeefbe5d50 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d88 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5dc0 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5e00 M INVALID RECEIVER>(nil)  0x148163de0: a(n) bad class
    0x7ffeefbe5e38 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5e80 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5eb8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdf9d8 M INVALID RECEIVER>(nil)  0x194648430: a(n) bad class
    0x7ffeefbdfa10 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfa50 M [] in INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfa90 M INVALID RECEIVER>(nil)  0x1946486d8: a(n) bad class
    0x7ffeefbdfad0 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfb10 M INVALID RECEIVER>(nil)  0x1946485e0: a(n) bad class
    0x7ffeefbdfb48 M INVALID RECEIVER>(nil)  0x1489f7da8: a(n) bad class
    0x7ffeefbdfb80 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfbb8 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfbe8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfc20 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc58 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc98 M INVALID RECEIVER>(nil)  0x194648c40: a(n) bad class
    0x7ffeefbdfcc8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd08 M INVALID RECEIVER>(nil)  0x194648f40: a(n) bad class
    0x7ffeefbdfd40 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd70 M INVALID RECEIVER>(nil)  0x14902a730: a(n) bad class
    0x7ffeefbdfdb0 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfde8 M INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe20 M [] in INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe70 M [] in INVALID RECEIVER>(nil)  0x14d032f98: a(n) bad class
    0x7ffeefbdfeb8 M INVALID RECEIVER>(nil)  0x14d032fe0: a(n) bad class

(callerContextOrNil == (nilObject())) || (isContext(callerContextOrNil)) 72783
       0x14d033738 is not a context

OK, interesting.  Both the assert failure and the badly corrupted stack trace lead me to believe that the issue happens long before the crash and is probably a stack corruption, either by a primitive cutting back the stack incorrectly, or some other hot riot ion (for example are all those nils in INVALID RECEIVER>(nil) real or an artifact of attempting to print an invalid value?).

So the next step is to run the asset vm with leak checking turned on.  Use

myvm —leakcheck 3 to check after every GC

We can add, eg leak checking after an FFI call, in an afternoon

A more realistic setup would be to run GT with an assert headless vm. But until now I did not figure out how to build an assert vm for the gt-headless branch from https://github.com/feenkcom/opensmalltalk-vm

Cheers,
Andrei



[2] checking for cairo's PNG functions feature... 
configure: WARNING: Could not find libpng in the pkg-config search path
checking whether cairo's PNG functions feature could be enabled... no
configure: error: recommended PNG functions feature could not be enabled

On 14 Sep 2020, at 17:32, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 7:15 AM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

On 12 Sep 2020, at 01:42, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

I don’t think there is something special about that process. It is the process that we start to run tests [1]. The exception happens in the running process and the crash is when copying the stack of that running process.

Ok, cool.  What I’d like to do is get a copy of your test setup and run it in an assert vm to try and get more information.  AFAICT the vm code is good do the bug is not obvious.  An assert vm may give more information before the crash.  Have you tried running the system on an assert vm yet?

Checked some previous logs and we get these kinds of crashes on the CI server since at least two years. So it does not look like a new bug (but who knows).


(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Nice!

I also tried the non-recursive implementation of Context>>#copyTo: from Squeak and it also crashes.

Not sure if related but now in the same image as before I got a different crash and printing the stack does not work. But this time the error seems to come from handleStackOverflow

(lldb) call (void *)printCallStack()
invalid frame pointer
invalid frame pointer
invalid frame pointer
error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=EXC_I386_GPFLT).
The process has been returned to the state before expression evaluation.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x121e00000)
  * frame #0: 0x0000000100162258 libGlamorousToolkitVMCore.dylib`marryFrameSP + 584
    frame #1: 0x0000000100172982 libGlamorousToolkitVMCore.dylib`handleStackOverflow + 354
    frame #2: 0x000000010016b025 libGlamorousToolkitVMCore.dylib`ceStackOverflow + 149
    frame #3: 0x00000001100005b3
    frame #4: 0x0000000100174d99 libGlamorousToolkitVMCore.dylib`ptEnterInterpreterFromCallback + 73


Cheers,
Andrei

[1] ./GlamorousToolkit.app/Contents/MacOS/GlamorousToolkit  Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'



Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue: 
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue: 
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue: 
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
 Special-case non-single contexts (because of context-to-stack mapping).
 Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
  or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001100b1004
->  0x1100b1004: inl    $0x4c, %eax
    0x1100b1006: leal   -0x5c(%rip), %eax
    0x1100b100c: pushq  %r8
    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 
Target 0: (GlamorousToolkit) stopped.


Process 29929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001100fe7ed
->  0x1100fe7ed: int3   
    0x1100fe7ee: int3   
    0x1100fe7ef: int3   
    0x1100fe7f0: int3   
Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





-- 
_,,,^..^,,,_
best, Eliot


-- 
_,,,^..^,,,_
best, Eliot

_,,,^..^,,,_ (phone)
Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Andrei Chis
 
Hi Eliot,

Unfortunately with the assert vm with memory leak checks it seems I cannot reproduce the problem. 
I’ll try to run it a few more times, maybe I get lucky and get a crash (a run takes a few hours). 

For our case, calling `self pc` before copying a context really helped. Before almost 50% of our builds were failing; now we get no more failures at all.
Thanks for your insights!

Are there other ways to get relevant/helpful details from a crash with a normal vm?

Cheers,
Andrei


On 15 Sep 2020, at 01:27, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 3:22 PM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

The setup in GT is a bit customised (some changes in the headless vm, some custom plugins, custom rendering) so I first thought it will be impossible to reproduce the bug in a more standard manner. 
However turns out it is possible. If I use the following script after running the tests a few times in lldb I get the crash starting from a plain Pharo 8 image.

$ ./pharo Pharo.image st --quit 

$ lldb ./pharo-vm/Pharo.app/Contents/MacOS/Pharo
(lldb) run --headless Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'


I also tried to compile the vm myself on Mac (Catalina 10.15.6). I build a normal and assert for https://github.com/OpenSmalltalk/opensmalltalk-vm and https://github.com/pharo-project/opensmalltalk-vm from the cog branch.
In both cases I get an issue related to pixman 0.34.0 [1] but that’s easy to workaround. For https://github.com/OpenSmalltalk/opensmalltalk-vm I got an extra problem related to Cairo [2] and had to change libpng from libpng16 to libpng12 to get it to work.

With both the normal VMs I could reproduce the bug and got stacks with the Context>copyTo: messages. 

With the assert VMs I only got a crash for now with the assert vm from https://github.com/pharo-project/opensmalltalk-vm. However there is no Context>copyTo: and the memory seems quite corrupted. 
I suspect the crash also appears in https://github.com/OpenSmalltalk/opensmalltalk-vm  but seems that with the assert vm it is much harder to reproduce. Had to run the tests 20 times and got one crash; running the tests once take 20-30 minutes. 


This is from only crash until now with the assert vm. Not sure if they are helpful or not, or actually related to the problem.

validInstructionPointerinFrame(GIV(instructionPointer), GIV(framePointer)) 18471
Pharo was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 73731 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
    frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
   139    static inline sqInt intAtPointer(char *ptr) { return (sqInt)(*((int *)ptr)); }
   140    static inline sqInt intAtPointerput(char *ptr, int val) { return (sqInt)(*((int *)ptr)= val); }
   141    static inline sqInt longAtPointer(char *ptr) { return *(sqInt *)ptr; }
-> 142    static inline sqInt longAtPointerput(char *ptr, sqInt val) { return *(sqInt *)ptr= val; }
   143    static inline sqLong long64AtPointer(char *ptr) { return *(sqLong *)ptr; }
   144    static inline sqLong long64AtPointerput(char *ptr, sqLong val) { return *(sqLong *)ptr= val; }
   145    static inline float singleFloatAtPointer(char *ptr) { return *(float *)ptr; }
Target 0: (Pharo) stopped.


(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
  * frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
    frame #1: 0x00000001000161cf Pharo`marryFrameSP(theFP=<unavailable>, theSP=0x0000000000000000) at gcc3x-cointerp.c:68120:3[opt]
    frame #2: 0x000000010001f5ac Pharo`ceContextinstVar(maybeContext=5510359872, slotIndex=0) at gcc3x-cointerp.c:15221:12 [opt]
    frame #3: 0x00000001480017d6
    frame #4: 0x00000001000022be Pharo`interpret at gcc3x-cointerp.c:2755:3 [opt]
    frame #5: 0x00000001000bc244 Pharo`-[sqSqueakMainApplication runSqueak](self=0x0000000101c76dc0, _cmd=<unavailable>) at sqSqueakMainApplication.m:201:2 [opt]
    frame #6: 0x00007fff3326729b Foundation`__NSFirePerformWithOrder + 360
    frame #7: 0x00007fff30ad3335 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    frame #8: 0x00007fff30ad3267 CoreFoundation`__CFRunLoopDoObservers + 457
    frame #9: 0x00007fff30ad2805 CoreFoundation`__CFRunLoopRun + 874
    frame #10: 0x00007fff30ad1e3e CoreFoundation`CFRunLoopRunSpecific + 462
    frame #11: 0x00007fff2f6feabd HIToolbox`RunCurrentEventLoopInMode + 292
    frame #12: 0x00007fff2f6fe6f4 HIToolbox`ReceiveNextEventCommon + 359
    frame #13: 0x00007fff2f6fe579 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 64
    frame #14: 0x00007fff2dd44039 AppKit`_DPSNextEvent + 883
    frame #15: 0x00007fff2dd42880 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
    frame #16: 0x00007fff2dd3458e AppKit`-[NSApplication run] + 658
    frame #17: 0x00007fff2dd06396 AppKit`NSApplicationMain + 777
    frame #18: 0x00007fff6ab3ecc9 libdyld.dylib`start + 1


(lldb) call printCallStack()
    0x7ffeefbe3920 M INVALID RECEIVER>(nil) 0x148716b40: a(n) bad class
    0x7ffeefbe3968 M [] in INVALID RECEIVER>(nil) Context(Object)>>doesNotUnderstand: #bounds
 0x194648118: a(n) bad class
    0x7ffeefbe39a8 M INVALID RECEIVER>(nil) 0x1489fcec0: a(n) bad class
    0x7ffeefbe39e8 M INVALID RECEIVER>(nil)  0x1489fcec0: a(n) bad class
    0x7ffeefbe3a30 I INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3a80 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3ab8 M INVALID RECEIVER>(nil)  0x148163cd0: a(n) bad class
    0x7ffeefbe3b08 I INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b40 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b78 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe3bc0 I INVALID RECEIVER>(nil)  0x148716a38: a(n) bad class
    0x7ffeefbe3c10 I INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c40 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c78 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3cc0 M INVALID RECEIVER>(nil)  0x14d0337f0: a(n) bad class
    0x7ffeefbe3d08 M INVALID RECEIVER>(nil)  0x14d033738: a(n) bad class
    0x7ffeefbe3d50 M INVALID RECEIVER>(nil)  0x14d033680: a(n) bad class
    0x7ffeefbe3d98 M INVALID RECEIVER>(nil)  0x1946493f0: a(n) bad class
    0x7ffeefbe3de0 M INVALID RECEIVER>(nil)  0x194649338: a(n) bad class
    0x7ffeefbe3e28 M INVALID RECEIVER>(nil)  0x194649280: a(n) bad class
    0x7ffeefbe3e70 M INVALID RECEIVER>(nil)  0x1946491c8: a(n) bad class
    0x7ffeefbe3eb8 M INVALID RECEIVER>(nil)  0x194649110: a(n) bad class
    0x7ffeefbec768 M INVALID RECEIVER>(nil)  0x194649038: a(n) bad class
    0x7ffeefbec7b0 M INVALID RECEIVER>(nil)  0x194648f60: a(n) bad class
    0x7ffeefbec7f8 M INVALID RECEIVER>(nil)  0x194648e88: a(n) bad class
    0x7ffeefbec840 M INVALID RECEIVER>(nil)  0x194648dd0: a(n) bad class
    0x7ffeefbec888 M INVALID RECEIVER>(nil)  0x194648d18: a(n) bad class
    0x7ffeefbec8d0 M INVALID RECEIVER>(nil)  0x194648c60: a(n) bad class
    0x7ffeefbec918 M INVALID RECEIVER>(nil)  0x194648b88: a(n) bad class
    0x7ffeefbec960 M INVALID RECEIVER>(nil)  0x194648ad0: a(n) bad class
    0x7ffeefbec9a8 M INVALID RECEIVER>(nil)  0x194648a18: a(n) bad class
    0x7ffeefbec9f0 M INVALID RECEIVER>(nil)  0x194648960: a(n) bad class
    0x7ffeefbeca38 M INVALID RECEIVER>(nil)  0x1946488a8: a(n) bad class
    0x7ffeefbeca80 M INVALID RECEIVER>(nil)  0x1946487f0: a(n) bad class
    0x7ffeefbecac8 M INVALID RECEIVER>(nil)  0x194648708: a(n) bad class
    0x7ffeefbecb10 M INVALID RECEIVER>(nil)  0x194648620: a(n) bad class
    0x7ffeefbecb58 M INVALID RECEIVER>(nil)  0x194648508: a(n) bad class
    0x7ffeefbecba0 M INVALID RECEIVER>(nil)  0x194648450: a(n) bad class
    0x7ffeefbecbe8 M INVALID RECEIVER>(nil)  0x1481641a8: a(n) bad class
    0x7ffeefbecc30 M INVALID RECEIVER>(nil)  0x1481640f0: a(n) bad class
    0x7ffeefbecc78 M INVALID RECEIVER>(nil)  0x148164038: a(n) bad class
    0x7ffeefbeccc0 M INVALID RECEIVER>(nil)  0x148163f80: a(n) bad class
    0x7ffeefbecd08 M INVALID RECEIVER>(nil)  0x148163ec8: a(n) bad class
    0x7ffeefbecd50 M INVALID RECEIVER>(nil)  0x148163e10: a(n) bad class
    0x7ffeefbecd98 M INVALID RECEIVER>(nil)  0x148163d28: a(n) bad class
    0x7ffeefbecde0 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbece28 M INVALID RECEIVER>(nil)  0x148163b38: a(n) bad class
    0x7ffeefbece70 M INVALID RECEIVER>(nil)  0x148163a80: a(n) bad class
    0x7ffeefbeceb8 M INVALID RECEIVER>(nil)  0x1481639c8: a(n) bad class
    0x7ffeefbe7758 M INVALID RECEIVER>(nil)  0x1481638e0: a(n) bad class
    0x7ffeefbe77a0 M INVALID RECEIVER>(nil)  0x148163808: a(n) bad class
    0x7ffeefbe77e8 M INVALID RECEIVER>(nil)  0x148163750: a(n) bad class
    0x7ffeefbe7830 M INVALID RECEIVER>(nil)  0x148163698: a(n) bad class
    0x7ffeefbe7878 M INVALID RECEIVER>(nil)  0x1481635c0: a(n) bad class
    0x7ffeefbe78c0 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe7908 M INVALID RECEIVER>(nil)  0x148163408: a(n) bad class
    0x7ffeefbe7950 M INVALID RECEIVER>(nil)  0x148163350: a(n) bad class
    0x7ffeefbe7998 M INVALID RECEIVER>(nil)  0x148163298: a(n) bad class
    0x7ffeefbe79e0 M INVALID RECEIVER>(nil)  0x148163188: a(n) bad class
    0x7ffeefbe7a28 M INVALID RECEIVER>(nil)  0x148163098: a(n) bad class
    0x7ffeefbe7a70 M INVALID RECEIVER>(nil)  0x148162fa0: a(n) bad class
    0x7ffeefbe7ab8 M INVALID RECEIVER>(nil)  0x148162ec8: a(n) bad class
    0x7ffeefbe7b00 M INVALID RECEIVER>(nil)  0x148162e10: a(n) bad class
    0x7ffeefbe7b48 M INVALID RECEIVER>(nil)  0x148712a08: a(n) bad class
    0x7ffeefbe7b90 M INVALID RECEIVER>(nil)  0x148712950: a(n) bad class
    0x7ffeefbe7bd8 M INVALID RECEIVER>(nil)  0x148712898: a(n) bad class
    0x7ffeefbe7c20 M INVALID RECEIVER>(nil)  0x148713cc0: a(n) bad class
    0x7ffeefbe7c68 M INVALID RECEIVER>(nil)  0x148713018: a(n) bad class
    0x7ffeefbe7cb0 M INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbe7cf8 M INVALID RECEIVER>(nil)  0x148713140: a(n) bad class
    0x7ffeefbe7d40 M INVALID RECEIVER>(nil)  0x148713928: a(n) bad class
    0x7ffeefbe7d88 M INVALID RECEIVER>(nil)  0x1487133c8: a(n) bad class
    0x7ffeefbe7de0 I INVALID RECEIVER>(nil)  0x148713238: a(n) bad class
    0x7ffeefbe7e28 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7e70 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7eb8 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeea68 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeac8 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeb00 M INVALID RECEIVER>(nil)  0x148713108: a(n) bad class
    0x7ffeefbeeb50 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeeb98 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeebe0 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbeec10 M INVALID RECEIVER>(nil) 0x9=1
    0x7ffeefbeec48 M INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeeca0 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeecd0 M INVALID RECEIVER>(nil)  0x1487130d0: a(n) bad class
    0x7ffeefbeed10 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeed58 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeedb0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeedf0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeee20 M [] in INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbeee78 M INVALID RECEIVER>(nil)  0x148162df0: a(n) bad class
    0x7ffeefbeeec0 M INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbe5978 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe59d8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5a18 M INVALID RECEIVER>(nil)  0x148163150: a(n) bad class
    0x7ffeefbe5a68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5aa0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5ad8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5b08 M INVALID RECEIVER>(nil)  0x1481634c0: a(n) bad class
    0x7ffeefbe5b48 M INVALID RECEIVER>(nil)  0x14c403ca8: a(n) bad class
    0x7ffeefbe5b88 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bc0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bf0 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c20 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c98 M INVALID RECEIVER>(nil)  0x194656468: a(n) bad class
    0x7ffeefbe5cd0 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d00 M INVALID RECEIVER>(nil)  0x148163bf0: a(n) bad class
    0x7ffeefbe5d50 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d88 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5dc0 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5e00 M INVALID RECEIVER>(nil)  0x148163de0: a(n) bad class
    0x7ffeefbe5e38 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5e80 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5eb8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdf9d8 M INVALID RECEIVER>(nil)  0x194648430: a(n) bad class
    0x7ffeefbdfa10 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfa50 M [] in INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfa90 M INVALID RECEIVER>(nil)  0x1946486d8: a(n) bad class
    0x7ffeefbdfad0 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfb10 M INVALID RECEIVER>(nil)  0x1946485e0: a(n) bad class
    0x7ffeefbdfb48 M INVALID RECEIVER>(nil)  0x1489f7da8: a(n) bad class
    0x7ffeefbdfb80 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfbb8 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfbe8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfc20 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc58 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc98 M INVALID RECEIVER>(nil)  0x194648c40: a(n) bad class
    0x7ffeefbdfcc8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd08 M INVALID RECEIVER>(nil)  0x194648f40: a(n) bad class
    0x7ffeefbdfd40 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd70 M INVALID RECEIVER>(nil)  0x14902a730: a(n) bad class
    0x7ffeefbdfdb0 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfde8 M INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe20 M [] in INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe70 M [] in INVALID RECEIVER>(nil)  0x14d032f98: a(n) bad class
    0x7ffeefbdfeb8 M INVALID RECEIVER>(nil)  0x14d032fe0: a(n) bad class

(callerContextOrNil == (nilObject())) || (isContext(callerContextOrNil)) 72783
       0x14d033738 is not a context

OK, interesting.  Both the assert failure and the badly corrupted stack trace lead me to believe that the issue happens long before the crash and is probably a stack corruption, either by a primitive cutting back the stack incorrectly, or some other hot riot ion (for example are all those nils in INVALID RECEIVER>(nil) real or an artifact of attempting to print an invalid value?).

So the next step is to run the asset vm with leak checking turned on.  Use

myvm —leakcheck 3 to check after every GC

We can add, eg leak checking after an FFI call, in an afternoon

A more realistic setup would be to run GT with an assert headless vm. But until now I did not figure out how to build an assert vm for the gt-headless branch from https://github.com/feenkcom/opensmalltalk-vm

Cheers,
Andrei



[2] checking for cairo's PNG functions feature... 
configure: WARNING: Could not find libpng in the pkg-config search path
checking whether cairo's PNG functions feature could be enabled... no
configure: error: recommended PNG functions feature could not be enabled

On 14 Sep 2020, at 17:32, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 7:15 AM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

On 12 Sep 2020, at 01:42, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

I don’t think there is something special about that process. It is the process that we start to run tests [1]. The exception happens in the running process and the crash is when copying the stack of that running process.

Ok, cool.  What I’d like to do is get a copy of your test setup and run it in an assert vm to try and get more information.  AFAICT the vm code is good do the bug is not obvious.  An assert vm may give more information before the crash.  Have you tried running the system on an assert vm yet?

Checked some previous logs and we get these kinds of crashes on the CI server since at least two years. So it does not look like a new bug (but who knows).


(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Nice!

I also tried the non-recursive implementation of Context>>#copyTo: from Squeak and it also crashes.

Not sure if related but now in the same image as before I got a different crash and printing the stack does not work. But this time the error seems to come from handleStackOverflow

(lldb) call (void *)printCallStack()
invalid frame pointer
invalid frame pointer
invalid frame pointer
error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=EXC_I386_GPFLT).
The process has been returned to the state before expression evaluation.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x121e00000)
  * frame #0: 0x0000000100162258 libGlamorousToolkitVMCore.dylib`marryFrameSP + 584
    frame #1: 0x0000000100172982 libGlamorousToolkitVMCore.dylib`handleStackOverflow + 354
    frame #2: 0x000000010016b025 libGlamorousToolkitVMCore.dylib`ceStackOverflow + 149
    frame #3: 0x00000001100005b3
    frame #4: 0x0000000100174d99 libGlamorousToolkitVMCore.dylib`ptEnterInterpreterFromCallback + 73


Cheers,
Andrei

[1] ./GlamorousToolkit.app/Contents/MacOS/GlamorousToolkit  Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'



Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue: 
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue: 
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue: 
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
 Special-case non-single contexts (because of context-to-stack mapping).
 Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
  or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001100b1004
->  0x1100b1004: inl    $0x4c, %eax
    0x1100b1006: leal   -0x5c(%rip), %eax
    0x1100b100c: pushq  %r8
    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 
Target 0: (GlamorousToolkit) stopped.


Process 29929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001100fe7ed
->  0x1100fe7ed: int3   
    0x1100fe7ee: int3   
    0x1100fe7ef: int3   
    0x1100fe7f0: int3   
Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





-- 
_,,,^..^,,,_
best, Eliot


-- 
_,,,^..^,,,_
best, Eliot

_,,,^..^,,,_ (phone)

Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Nicolas Cellier
 
If you are on linux, you could try using the recording debugger (rr)... Though, debugging a -O2 or -O3 is hard... No obvious mapping of source code with assembly instructions. It's better if you can reproduce on debug version...

Le mar. 22 sept. 2020 à 14:21, Andrei Chis <[hidden email]> a écrit :
 
Hi Eliot,

Unfortunately with the assert vm with memory leak checks it seems I cannot reproduce the problem. 
I’ll try to run it a few more times, maybe I get lucky and get a crash (a run takes a few hours). 

For our case, calling `self pc` before copying a context really helped. Before almost 50% of our builds were failing; now we get no more failures at all.
Thanks for your insights!

Are there other ways to get relevant/helpful details from a crash with a normal vm?

Cheers,
Andrei


On 15 Sep 2020, at 01:27, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 3:22 PM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

The setup in GT is a bit customised (some changes in the headless vm, some custom plugins, custom rendering) so I first thought it will be impossible to reproduce the bug in a more standard manner. 
However turns out it is possible. If I use the following script after running the tests a few times in lldb I get the crash starting from a plain Pharo 8 image.

$ ./pharo Pharo.image st --quit 

$ lldb ./pharo-vm/Pharo.app/Contents/MacOS/Pharo
(lldb) run --headless Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'


I also tried to compile the vm myself on Mac (Catalina 10.15.6). I build a normal and assert for https://github.com/OpenSmalltalk/opensmalltalk-vm and https://github.com/pharo-project/opensmalltalk-vm from the cog branch.
In both cases I get an issue related to pixman 0.34.0 [1] but that’s easy to workaround. For https://github.com/OpenSmalltalk/opensmalltalk-vm I got an extra problem related to Cairo [2] and had to change libpng from libpng16 to libpng12 to get it to work.

With both the normal VMs I could reproduce the bug and got stacks with the Context>copyTo: messages. 

With the assert VMs I only got a crash for now with the assert vm from https://github.com/pharo-project/opensmalltalk-vm. However there is no Context>copyTo: and the memory seems quite corrupted. 
I suspect the crash also appears in https://github.com/OpenSmalltalk/opensmalltalk-vm  but seems that with the assert vm it is much harder to reproduce. Had to run the tests 20 times and got one crash; running the tests once take 20-30 minutes. 


This is from only crash until now with the assert vm. Not sure if they are helpful or not, or actually related to the problem.

validInstructionPointerinFrame(GIV(instructionPointer), GIV(framePointer)) 18471
Pharo was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 73731 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
    frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
   139    static inline sqInt intAtPointer(char *ptr) { return (sqInt)(*((int *)ptr)); }
   140    static inline sqInt intAtPointerput(char *ptr, int val) { return (sqInt)(*((int *)ptr)= val); }
   141    static inline sqInt longAtPointer(char *ptr) { return *(sqInt *)ptr; }
-> 142    static inline sqInt longAtPointerput(char *ptr, sqInt val) { return *(sqInt *)ptr= val; }
   143    static inline sqLong long64AtPointer(char *ptr) { return *(sqLong *)ptr; }
   144    static inline sqLong long64AtPointerput(char *ptr, sqLong val) { return *(sqLong *)ptr= val; }
   145    static inline float singleFloatAtPointer(char *ptr) { return *(float *)ptr; }
Target 0: (Pharo) stopped.


(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
  * frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
    frame #1: 0x00000001000161cf Pharo`marryFrameSP(theFP=<unavailable>, theSP=0x0000000000000000) at gcc3x-cointerp.c:68120:3[opt]
    frame #2: 0x000000010001f5ac Pharo`ceContextinstVar(maybeContext=5510359872, slotIndex=0) at gcc3x-cointerp.c:15221:12 [opt]
    frame #3: 0x00000001480017d6
    frame #4: 0x00000001000022be Pharo`interpret at gcc3x-cointerp.c:2755:3 [opt]
    frame #5: 0x00000001000bc244 Pharo`-[sqSqueakMainApplication runSqueak](self=0x0000000101c76dc0, _cmd=<unavailable>) at sqSqueakMainApplication.m:201:2 [opt]
    frame #6: 0x00007fff3326729b Foundation`__NSFirePerformWithOrder + 360
    frame #7: 0x00007fff30ad3335 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    frame #8: 0x00007fff30ad3267 CoreFoundation`__CFRunLoopDoObservers + 457
    frame #9: 0x00007fff30ad2805 CoreFoundation`__CFRunLoopRun + 874
    frame #10: 0x00007fff30ad1e3e CoreFoundation`CFRunLoopRunSpecific + 462
    frame #11: 0x00007fff2f6feabd HIToolbox`RunCurrentEventLoopInMode + 292
    frame #12: 0x00007fff2f6fe6f4 HIToolbox`ReceiveNextEventCommon + 359
    frame #13: 0x00007fff2f6fe579 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 64
    frame #14: 0x00007fff2dd44039 AppKit`_DPSNextEvent + 883
    frame #15: 0x00007fff2dd42880 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
    frame #16: 0x00007fff2dd3458e AppKit`-[NSApplication run] + 658
    frame #17: 0x00007fff2dd06396 AppKit`NSApplicationMain + 777
    frame #18: 0x00007fff6ab3ecc9 libdyld.dylib`start + 1


(lldb) call printCallStack()
    0x7ffeefbe3920 M INVALID RECEIVER>(nil) 0x148716b40: a(n) bad class
    0x7ffeefbe3968 M [] in INVALID RECEIVER>(nil) Context(Object)>>doesNotUnderstand: #bounds
 0x194648118: a(n) bad class
    0x7ffeefbe39a8 M INVALID RECEIVER>(nil) 0x1489fcec0: a(n) bad class
    0x7ffeefbe39e8 M INVALID RECEIVER>(nil)  0x1489fcec0: a(n) bad class
    0x7ffeefbe3a30 I INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3a80 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3ab8 M INVALID RECEIVER>(nil)  0x148163cd0: a(n) bad class
    0x7ffeefbe3b08 I INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b40 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b78 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe3bc0 I INVALID RECEIVER>(nil)  0x148716a38: a(n) bad class
    0x7ffeefbe3c10 I INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c40 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c78 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3cc0 M INVALID RECEIVER>(nil)  0x14d0337f0: a(n) bad class
    0x7ffeefbe3d08 M INVALID RECEIVER>(nil)  0x14d033738: a(n) bad class
    0x7ffeefbe3d50 M INVALID RECEIVER>(nil)  0x14d033680: a(n) bad class
    0x7ffeefbe3d98 M INVALID RECEIVER>(nil)  0x1946493f0: a(n) bad class
    0x7ffeefbe3de0 M INVALID RECEIVER>(nil)  0x194649338: a(n) bad class
    0x7ffeefbe3e28 M INVALID RECEIVER>(nil)  0x194649280: a(n) bad class
    0x7ffeefbe3e70 M INVALID RECEIVER>(nil)  0x1946491c8: a(n) bad class
    0x7ffeefbe3eb8 M INVALID RECEIVER>(nil)  0x194649110: a(n) bad class
    0x7ffeefbec768 M INVALID RECEIVER>(nil)  0x194649038: a(n) bad class
    0x7ffeefbec7b0 M INVALID RECEIVER>(nil)  0x194648f60: a(n) bad class
    0x7ffeefbec7f8 M INVALID RECEIVER>(nil)  0x194648e88: a(n) bad class
    0x7ffeefbec840 M INVALID RECEIVER>(nil)  0x194648dd0: a(n) bad class
    0x7ffeefbec888 M INVALID RECEIVER>(nil)  0x194648d18: a(n) bad class
    0x7ffeefbec8d0 M INVALID RECEIVER>(nil)  0x194648c60: a(n) bad class
    0x7ffeefbec918 M INVALID RECEIVER>(nil)  0x194648b88: a(n) bad class
    0x7ffeefbec960 M INVALID RECEIVER>(nil)  0x194648ad0: a(n) bad class
    0x7ffeefbec9a8 M INVALID RECEIVER>(nil)  0x194648a18: a(n) bad class
    0x7ffeefbec9f0 M INVALID RECEIVER>(nil)  0x194648960: a(n) bad class
    0x7ffeefbeca38 M INVALID RECEIVER>(nil)  0x1946488a8: a(n) bad class
    0x7ffeefbeca80 M INVALID RECEIVER>(nil)  0x1946487f0: a(n) bad class
    0x7ffeefbecac8 M INVALID RECEIVER>(nil)  0x194648708: a(n) bad class
    0x7ffeefbecb10 M INVALID RECEIVER>(nil)  0x194648620: a(n) bad class
    0x7ffeefbecb58 M INVALID RECEIVER>(nil)  0x194648508: a(n) bad class
    0x7ffeefbecba0 M INVALID RECEIVER>(nil)  0x194648450: a(n) bad class
    0x7ffeefbecbe8 M INVALID RECEIVER>(nil)  0x1481641a8: a(n) bad class
    0x7ffeefbecc30 M INVALID RECEIVER>(nil)  0x1481640f0: a(n) bad class
    0x7ffeefbecc78 M INVALID RECEIVER>(nil)  0x148164038: a(n) bad class
    0x7ffeefbeccc0 M INVALID RECEIVER>(nil)  0x148163f80: a(n) bad class
    0x7ffeefbecd08 M INVALID RECEIVER>(nil)  0x148163ec8: a(n) bad class
    0x7ffeefbecd50 M INVALID RECEIVER>(nil)  0x148163e10: a(n) bad class
    0x7ffeefbecd98 M INVALID RECEIVER>(nil)  0x148163d28: a(n) bad class
    0x7ffeefbecde0 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbece28 M INVALID RECEIVER>(nil)  0x148163b38: a(n) bad class
    0x7ffeefbece70 M INVALID RECEIVER>(nil)  0x148163a80: a(n) bad class
    0x7ffeefbeceb8 M INVALID RECEIVER>(nil)  0x1481639c8: a(n) bad class
    0x7ffeefbe7758 M INVALID RECEIVER>(nil)  0x1481638e0: a(n) bad class
    0x7ffeefbe77a0 M INVALID RECEIVER>(nil)  0x148163808: a(n) bad class
    0x7ffeefbe77e8 M INVALID RECEIVER>(nil)  0x148163750: a(n) bad class
    0x7ffeefbe7830 M INVALID RECEIVER>(nil)  0x148163698: a(n) bad class
    0x7ffeefbe7878 M INVALID RECEIVER>(nil)  0x1481635c0: a(n) bad class
    0x7ffeefbe78c0 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe7908 M INVALID RECEIVER>(nil)  0x148163408: a(n) bad class
    0x7ffeefbe7950 M INVALID RECEIVER>(nil)  0x148163350: a(n) bad class
    0x7ffeefbe7998 M INVALID RECEIVER>(nil)  0x148163298: a(n) bad class
    0x7ffeefbe79e0 M INVALID RECEIVER>(nil)  0x148163188: a(n) bad class
    0x7ffeefbe7a28 M INVALID RECEIVER>(nil)  0x148163098: a(n) bad class
    0x7ffeefbe7a70 M INVALID RECEIVER>(nil)  0x148162fa0: a(n) bad class
    0x7ffeefbe7ab8 M INVALID RECEIVER>(nil)  0x148162ec8: a(n) bad class
    0x7ffeefbe7b00 M INVALID RECEIVER>(nil)  0x148162e10: a(n) bad class
    0x7ffeefbe7b48 M INVALID RECEIVER>(nil)  0x148712a08: a(n) bad class
    0x7ffeefbe7b90 M INVALID RECEIVER>(nil)  0x148712950: a(n) bad class
    0x7ffeefbe7bd8 M INVALID RECEIVER>(nil)  0x148712898: a(n) bad class
    0x7ffeefbe7c20 M INVALID RECEIVER>(nil)  0x148713cc0: a(n) bad class
    0x7ffeefbe7c68 M INVALID RECEIVER>(nil)  0x148713018: a(n) bad class
    0x7ffeefbe7cb0 M INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbe7cf8 M INVALID RECEIVER>(nil)  0x148713140: a(n) bad class
    0x7ffeefbe7d40 M INVALID RECEIVER>(nil)  0x148713928: a(n) bad class
    0x7ffeefbe7d88 M INVALID RECEIVER>(nil)  0x1487133c8: a(n) bad class
    0x7ffeefbe7de0 I INVALID RECEIVER>(nil)  0x148713238: a(n) bad class
    0x7ffeefbe7e28 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7e70 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7eb8 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeea68 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeac8 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeb00 M INVALID RECEIVER>(nil)  0x148713108: a(n) bad class
    0x7ffeefbeeb50 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeeb98 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeebe0 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbeec10 M INVALID RECEIVER>(nil) 0x9=1
    0x7ffeefbeec48 M INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeeca0 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeecd0 M INVALID RECEIVER>(nil)  0x1487130d0: a(n) bad class
    0x7ffeefbeed10 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeed58 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeedb0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeedf0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeee20 M [] in INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbeee78 M INVALID RECEIVER>(nil)  0x148162df0: a(n) bad class
    0x7ffeefbeeec0 M INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbe5978 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe59d8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5a18 M INVALID RECEIVER>(nil)  0x148163150: a(n) bad class
    0x7ffeefbe5a68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5aa0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5ad8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5b08 M INVALID RECEIVER>(nil)  0x1481634c0: a(n) bad class
    0x7ffeefbe5b48 M INVALID RECEIVER>(nil)  0x14c403ca8: a(n) bad class
    0x7ffeefbe5b88 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bc0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bf0 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c20 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c98 M INVALID RECEIVER>(nil)  0x194656468: a(n) bad class
    0x7ffeefbe5cd0 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d00 M INVALID RECEIVER>(nil)  0x148163bf0: a(n) bad class
    0x7ffeefbe5d50 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d88 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5dc0 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5e00 M INVALID RECEIVER>(nil)  0x148163de0: a(n) bad class
    0x7ffeefbe5e38 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5e80 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5eb8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdf9d8 M INVALID RECEIVER>(nil)  0x194648430: a(n) bad class
    0x7ffeefbdfa10 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfa50 M [] in INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfa90 M INVALID RECEIVER>(nil)  0x1946486d8: a(n) bad class
    0x7ffeefbdfad0 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfb10 M INVALID RECEIVER>(nil)  0x1946485e0: a(n) bad class
    0x7ffeefbdfb48 M INVALID RECEIVER>(nil)  0x1489f7da8: a(n) bad class
    0x7ffeefbdfb80 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfbb8 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfbe8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfc20 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc58 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc98 M INVALID RECEIVER>(nil)  0x194648c40: a(n) bad class
    0x7ffeefbdfcc8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd08 M INVALID RECEIVER>(nil)  0x194648f40: a(n) bad class
    0x7ffeefbdfd40 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd70 M INVALID RECEIVER>(nil)  0x14902a730: a(n) bad class
    0x7ffeefbdfdb0 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfde8 M INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe20 M [] in INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe70 M [] in INVALID RECEIVER>(nil)  0x14d032f98: a(n) bad class
    0x7ffeefbdfeb8 M INVALID RECEIVER>(nil)  0x14d032fe0: a(n) bad class

(callerContextOrNil == (nilObject())) || (isContext(callerContextOrNil)) 72783
       0x14d033738 is not a context

OK, interesting.  Both the assert failure and the badly corrupted stack trace lead me to believe that the issue happens long before the crash and is probably a stack corruption, either by a primitive cutting back the stack incorrectly, or some other hot riot ion (for example are all those nils in INVALID RECEIVER>(nil) real or an artifact of attempting to print an invalid value?).

So the next step is to run the asset vm with leak checking turned on.  Use

myvm —leakcheck 3 to check after every GC

We can add, eg leak checking after an FFI call, in an afternoon

A more realistic setup would be to run GT with an assert headless vm. But until now I did not figure out how to build an assert vm for the gt-headless branch from https://github.com/feenkcom/opensmalltalk-vm

Cheers,
Andrei



[2] checking for cairo's PNG functions feature... 
configure: WARNING: Could not find libpng in the pkg-config search path
checking whether cairo's PNG functions feature could be enabled... no
configure: error: recommended PNG functions feature could not be enabled

On 14 Sep 2020, at 17:32, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 7:15 AM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

On 12 Sep 2020, at 01:42, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

I don’t think there is something special about that process. It is the process that we start to run tests [1]. The exception happens in the running process and the crash is when copying the stack of that running process.

Ok, cool.  What I’d like to do is get a copy of your test setup and run it in an assert vm to try and get more information.  AFAICT the vm code is good do the bug is not obvious.  An assert vm may give more information before the crash.  Have you tried running the system on an assert vm yet?

Checked some previous logs and we get these kinds of crashes on the CI server since at least two years. So it does not look like a new bug (but who knows).


(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Nice!

I also tried the non-recursive implementation of Context>>#copyTo: from Squeak and it also crashes.

Not sure if related but now in the same image as before I got a different crash and printing the stack does not work. But this time the error seems to come from handleStackOverflow

(lldb) call (void *)printCallStack()
invalid frame pointer
invalid frame pointer
invalid frame pointer
error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=EXC_I386_GPFLT).
The process has been returned to the state before expression evaluation.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x121e00000)
  * frame #0: 0x0000000100162258 libGlamorousToolkitVMCore.dylib`marryFrameSP + 584
    frame #1: 0x0000000100172982 libGlamorousToolkitVMCore.dylib`handleStackOverflow + 354
    frame #2: 0x000000010016b025 libGlamorousToolkitVMCore.dylib`ceStackOverflow + 149
    frame #3: 0x00000001100005b3
    frame #4: 0x0000000100174d99 libGlamorousToolkitVMCore.dylib`ptEnterInterpreterFromCallback + 73


Cheers,
Andrei

[1] ./GlamorousToolkit.app/Contents/MacOS/GlamorousToolkit  Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'



Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue: 
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue: 
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue: 
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
 Special-case non-single contexts (because of context-to-stack mapping).
 Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
  or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001100b1004
->  0x1100b1004: inl    $0x4c, %eax
    0x1100b1006: leal   -0x5c(%rip), %eax
    0x1100b100c: pushq  %r8
    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 
Target 0: (GlamorousToolkit) stopped.


Process 29929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001100fe7ed
->  0x1100fe7ed: int3   
    0x1100fe7ee: int3   
    0x1100fe7ef: int3   
    0x1100fe7f0: int3   
Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





-- 
_,,,^..^,,,_
best, Eliot


-- 
_,,,^..^,,,_
best, Eliot

_,,,^..^,,,_ (phone)

Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Eliot Miranda-2
In reply to this post by Andrei Chis
 
Hi Andrei,

On Tue, Sep 22, 2020 at 5:21 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Unfortunately with the assert vm with memory leak checks it seems I cannot reproduce the problem. 
I’ll try to run it a few more times, maybe I get lucky and get a crash (a run takes a few hours). 

For our case, calling `self pc` before copying a context really helped. Before almost 50% of our builds were failing; now we get no more failures at all.
Thanks for your insights!

Are there other ways to get relevant/helpful details from a crash with a normal vm?

The crash.log should include the primitive trace log, which is the last 256 external primitives the VM has called. Can you post that part of the crash log?

The memory leak checking can be turned on in a production VM, it's just that the output is a bit harder to parse.  So try that also.
Cheers,
Andrei


On 15 Sep 2020, at 01:27, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 3:22 PM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

The setup in GT is a bit customised (some changes in the headless vm, some custom plugins, custom rendering) so I first thought it will be impossible to reproduce the bug in a more standard manner. 
However turns out it is possible. If I use the following script after running the tests a few times in lldb I get the crash starting from a plain Pharo 8 image.

$ ./pharo Pharo.image st --quit 

$ lldb ./pharo-vm/Pharo.app/Contents/MacOS/Pharo
(lldb) run --headless Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'


I also tried to compile the vm myself on Mac (Catalina 10.15.6). I build a normal and assert for https://github.com/OpenSmalltalk/opensmalltalk-vm and https://github.com/pharo-project/opensmalltalk-vm from the cog branch.
In both cases I get an issue related to pixman 0.34.0 [1] but that’s easy to workaround. For https://github.com/OpenSmalltalk/opensmalltalk-vm I got an extra problem related to Cairo [2] and had to change libpng from libpng16 to libpng12 to get it to work.

With both the normal VMs I could reproduce the bug and got stacks with the Context>copyTo: messages. 

With the assert VMs I only got a crash for now with the assert vm from https://github.com/pharo-project/opensmalltalk-vm. However there is no Context>copyTo: and the memory seems quite corrupted. 
I suspect the crash also appears in https://github.com/OpenSmalltalk/opensmalltalk-vm  but seems that with the assert vm it is much harder to reproduce. Had to run the tests 20 times and got one crash; running the tests once take 20-30 minutes. 


This is from only crash until now with the assert vm. Not sure if they are helpful or not, or actually related to the problem.

validInstructionPointerinFrame(GIV(instructionPointer), GIV(framePointer)) 18471
Pharo was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 73731 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
    frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
   139    static inline sqInt intAtPointer(char *ptr) { return (sqInt)(*((int *)ptr)); }
   140    static inline sqInt intAtPointerput(char *ptr, int val) { return (sqInt)(*((int *)ptr)= val); }
   141    static inline sqInt longAtPointer(char *ptr) { return *(sqInt *)ptr; }
-> 142    static inline sqInt longAtPointerput(char *ptr, sqInt val) { return *(sqInt *)ptr= val; }
   143    static inline sqLong long64AtPointer(char *ptr) { return *(sqLong *)ptr; }
   144    static inline sqLong long64AtPointerput(char *ptr, sqLong val) { return *(sqLong *)ptr= val; }
   145    static inline float singleFloatAtPointer(char *ptr) { return *(float *)ptr; }
Target 0: (Pharo) stopped.


(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
  * frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
    frame #1: 0x00000001000161cf Pharo`marryFrameSP(theFP=<unavailable>, theSP=0x0000000000000000) at gcc3x-cointerp.c:68120:3[opt]
    frame #2: 0x000000010001f5ac Pharo`ceContextinstVar(maybeContext=5510359872, slotIndex=0) at gcc3x-cointerp.c:15221:12 [opt]
    frame #3: 0x00000001480017d6
    frame #4: 0x00000001000022be Pharo`interpret at gcc3x-cointerp.c:2755:3 [opt]
    frame #5: 0x00000001000bc244 Pharo`-[sqSqueakMainApplication runSqueak](self=0x0000000101c76dc0, _cmd=<unavailable>) at sqSqueakMainApplication.m:201:2 [opt]
    frame #6: 0x00007fff3326729b Foundation`__NSFirePerformWithOrder + 360
    frame #7: 0x00007fff30ad3335 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    frame #8: 0x00007fff30ad3267 CoreFoundation`__CFRunLoopDoObservers + 457
    frame #9: 0x00007fff30ad2805 CoreFoundation`__CFRunLoopRun + 874
    frame #10: 0x00007fff30ad1e3e CoreFoundation`CFRunLoopRunSpecific + 462
    frame #11: 0x00007fff2f6feabd HIToolbox`RunCurrentEventLoopInMode + 292
    frame #12: 0x00007fff2f6fe6f4 HIToolbox`ReceiveNextEventCommon + 359
    frame #13: 0x00007fff2f6fe579 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 64
    frame #14: 0x00007fff2dd44039 AppKit`_DPSNextEvent + 883
    frame #15: 0x00007fff2dd42880 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
    frame #16: 0x00007fff2dd3458e AppKit`-[NSApplication run] + 658
    frame #17: 0x00007fff2dd06396 AppKit`NSApplicationMain + 777
    frame #18: 0x00007fff6ab3ecc9 libdyld.dylib`start + 1


(lldb) call printCallStack()
    0x7ffeefbe3920 M INVALID RECEIVER>(nil) 0x148716b40: a(n) bad class
    0x7ffeefbe3968 M [] in INVALID RECEIVER>(nil) Context(Object)>>doesNotUnderstand: #bounds
 0x194648118: a(n) bad class
    0x7ffeefbe39a8 M INVALID RECEIVER>(nil) 0x1489fcec0: a(n) bad class
    0x7ffeefbe39e8 M INVALID RECEIVER>(nil)  0x1489fcec0: a(n) bad class
    0x7ffeefbe3a30 I INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3a80 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3ab8 M INVALID RECEIVER>(nil)  0x148163cd0: a(n) bad class
    0x7ffeefbe3b08 I INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b40 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b78 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe3bc0 I INVALID RECEIVER>(nil)  0x148716a38: a(n) bad class
    0x7ffeefbe3c10 I INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c40 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c78 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3cc0 M INVALID RECEIVER>(nil)  0x14d0337f0: a(n) bad class
    0x7ffeefbe3d08 M INVALID RECEIVER>(nil)  0x14d033738: a(n) bad class
    0x7ffeefbe3d50 M INVALID RECEIVER>(nil)  0x14d033680: a(n) bad class
    0x7ffeefbe3d98 M INVALID RECEIVER>(nil)  0x1946493f0: a(n) bad class
    0x7ffeefbe3de0 M INVALID RECEIVER>(nil)  0x194649338: a(n) bad class
    0x7ffeefbe3e28 M INVALID RECEIVER>(nil)  0x194649280: a(n) bad class
    0x7ffeefbe3e70 M INVALID RECEIVER>(nil)  0x1946491c8: a(n) bad class
    0x7ffeefbe3eb8 M INVALID RECEIVER>(nil)  0x194649110: a(n) bad class
    0x7ffeefbec768 M INVALID RECEIVER>(nil)  0x194649038: a(n) bad class
    0x7ffeefbec7b0 M INVALID RECEIVER>(nil)  0x194648f60: a(n) bad class
    0x7ffeefbec7f8 M INVALID RECEIVER>(nil)  0x194648e88: a(n) bad class
    0x7ffeefbec840 M INVALID RECEIVER>(nil)  0x194648dd0: a(n) bad class
    0x7ffeefbec888 M INVALID RECEIVER>(nil)  0x194648d18: a(n) bad class
    0x7ffeefbec8d0 M INVALID RECEIVER>(nil)  0x194648c60: a(n) bad class
    0x7ffeefbec918 M INVALID RECEIVER>(nil)  0x194648b88: a(n) bad class
    0x7ffeefbec960 M INVALID RECEIVER>(nil)  0x194648ad0: a(n) bad class
    0x7ffeefbec9a8 M INVALID RECEIVER>(nil)  0x194648a18: a(n) bad class
    0x7ffeefbec9f0 M INVALID RECEIVER>(nil)  0x194648960: a(n) bad class
    0x7ffeefbeca38 M INVALID RECEIVER>(nil)  0x1946488a8: a(n) bad class
    0x7ffeefbeca80 M INVALID RECEIVER>(nil)  0x1946487f0: a(n) bad class
    0x7ffeefbecac8 M INVALID RECEIVER>(nil)  0x194648708: a(n) bad class
    0x7ffeefbecb10 M INVALID RECEIVER>(nil)  0x194648620: a(n) bad class
    0x7ffeefbecb58 M INVALID RECEIVER>(nil)  0x194648508: a(n) bad class
    0x7ffeefbecba0 M INVALID RECEIVER>(nil)  0x194648450: a(n) bad class
    0x7ffeefbecbe8 M INVALID RECEIVER>(nil)  0x1481641a8: a(n) bad class
    0x7ffeefbecc30 M INVALID RECEIVER>(nil)  0x1481640f0: a(n) bad class
    0x7ffeefbecc78 M INVALID RECEIVER>(nil)  0x148164038: a(n) bad class
    0x7ffeefbeccc0 M INVALID RECEIVER>(nil)  0x148163f80: a(n) bad class
    0x7ffeefbecd08 M INVALID RECEIVER>(nil)  0x148163ec8: a(n) bad class
    0x7ffeefbecd50 M INVALID RECEIVER>(nil)  0x148163e10: a(n) bad class
    0x7ffeefbecd98 M INVALID RECEIVER>(nil)  0x148163d28: a(n) bad class
    0x7ffeefbecde0 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbece28 M INVALID RECEIVER>(nil)  0x148163b38: a(n) bad class
    0x7ffeefbece70 M INVALID RECEIVER>(nil)  0x148163a80: a(n) bad class
    0x7ffeefbeceb8 M INVALID RECEIVER>(nil)  0x1481639c8: a(n) bad class
    0x7ffeefbe7758 M INVALID RECEIVER>(nil)  0x1481638e0: a(n) bad class
    0x7ffeefbe77a0 M INVALID RECEIVER>(nil)  0x148163808: a(n) bad class
    0x7ffeefbe77e8 M INVALID RECEIVER>(nil)  0x148163750: a(n) bad class
    0x7ffeefbe7830 M INVALID RECEIVER>(nil)  0x148163698: a(n) bad class
    0x7ffeefbe7878 M INVALID RECEIVER>(nil)  0x1481635c0: a(n) bad class
    0x7ffeefbe78c0 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe7908 M INVALID RECEIVER>(nil)  0x148163408: a(n) bad class
    0x7ffeefbe7950 M INVALID RECEIVER>(nil)  0x148163350: a(n) bad class
    0x7ffeefbe7998 M INVALID RECEIVER>(nil)  0x148163298: a(n) bad class
    0x7ffeefbe79e0 M INVALID RECEIVER>(nil)  0x148163188: a(n) bad class
    0x7ffeefbe7a28 M INVALID RECEIVER>(nil)  0x148163098: a(n) bad class
    0x7ffeefbe7a70 M INVALID RECEIVER>(nil)  0x148162fa0: a(n) bad class
    0x7ffeefbe7ab8 M INVALID RECEIVER>(nil)  0x148162ec8: a(n) bad class
    0x7ffeefbe7b00 M INVALID RECEIVER>(nil)  0x148162e10: a(n) bad class
    0x7ffeefbe7b48 M INVALID RECEIVER>(nil)  0x148712a08: a(n) bad class
    0x7ffeefbe7b90 M INVALID RECEIVER>(nil)  0x148712950: a(n) bad class
    0x7ffeefbe7bd8 M INVALID RECEIVER>(nil)  0x148712898: a(n) bad class
    0x7ffeefbe7c20 M INVALID RECEIVER>(nil)  0x148713cc0: a(n) bad class
    0x7ffeefbe7c68 M INVALID RECEIVER>(nil)  0x148713018: a(n) bad class
    0x7ffeefbe7cb0 M INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbe7cf8 M INVALID RECEIVER>(nil)  0x148713140: a(n) bad class
    0x7ffeefbe7d40 M INVALID RECEIVER>(nil)  0x148713928: a(n) bad class
    0x7ffeefbe7d88 M INVALID RECEIVER>(nil)  0x1487133c8: a(n) bad class
    0x7ffeefbe7de0 I INVALID RECEIVER>(nil)  0x148713238: a(n) bad class
    0x7ffeefbe7e28 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7e70 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7eb8 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeea68 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeac8 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeb00 M INVALID RECEIVER>(nil)  0x148713108: a(n) bad class
    0x7ffeefbeeb50 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeeb98 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeebe0 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbeec10 M INVALID RECEIVER>(nil) 0x9=1
    0x7ffeefbeec48 M INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeeca0 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeecd0 M INVALID RECEIVER>(nil)  0x1487130d0: a(n) bad class
    0x7ffeefbeed10 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeed58 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeedb0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeedf0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeee20 M [] in INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbeee78 M INVALID RECEIVER>(nil)  0x148162df0: a(n) bad class
    0x7ffeefbeeec0 M INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbe5978 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe59d8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5a18 M INVALID RECEIVER>(nil)  0x148163150: a(n) bad class
    0x7ffeefbe5a68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5aa0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5ad8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5b08 M INVALID RECEIVER>(nil)  0x1481634c0: a(n) bad class
    0x7ffeefbe5b48 M INVALID RECEIVER>(nil)  0x14c403ca8: a(n) bad class
    0x7ffeefbe5b88 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bc0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bf0 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c20 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c98 M INVALID RECEIVER>(nil)  0x194656468: a(n) bad class
    0x7ffeefbe5cd0 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d00 M INVALID RECEIVER>(nil)  0x148163bf0: a(n) bad class
    0x7ffeefbe5d50 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d88 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5dc0 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5e00 M INVALID RECEIVER>(nil)  0x148163de0: a(n) bad class
    0x7ffeefbe5e38 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5e80 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5eb8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdf9d8 M INVALID RECEIVER>(nil)  0x194648430: a(n) bad class
    0x7ffeefbdfa10 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfa50 M [] in INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfa90 M INVALID RECEIVER>(nil)  0x1946486d8: a(n) bad class
    0x7ffeefbdfad0 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfb10 M INVALID RECEIVER>(nil)  0x1946485e0: a(n) bad class
    0x7ffeefbdfb48 M INVALID RECEIVER>(nil)  0x1489f7da8: a(n) bad class
    0x7ffeefbdfb80 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfbb8 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfbe8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfc20 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc58 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc98 M INVALID RECEIVER>(nil)  0x194648c40: a(n) bad class
    0x7ffeefbdfcc8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd08 M INVALID RECEIVER>(nil)  0x194648f40: a(n) bad class
    0x7ffeefbdfd40 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd70 M INVALID RECEIVER>(nil)  0x14902a730: a(n) bad class
    0x7ffeefbdfdb0 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfde8 M INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe20 M [] in INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe70 M [] in INVALID RECEIVER>(nil)  0x14d032f98: a(n) bad class
    0x7ffeefbdfeb8 M INVALID RECEIVER>(nil)  0x14d032fe0: a(n) bad class

(callerContextOrNil == (nilObject())) || (isContext(callerContextOrNil)) 72783
       0x14d033738 is not a context

OK, interesting.  Both the assert failure and the badly corrupted stack trace lead me to believe that the issue happens long before the crash and is probably a stack corruption, either by a primitive cutting back the stack incorrectly, or some other hot riot ion (for example are all those nils in INVALID RECEIVER>(nil) real or an artifact of attempting to print an invalid value?).

So the next step is to run the asset vm with leak checking turned on.  Use

myvm —leakcheck 3 to check after every GC

We can add, eg leak checking after an FFI call, in an afternoon

A more realistic setup would be to run GT with an assert headless vm. But until now I did not figure out how to build an assert vm for the gt-headless branch from https://github.com/feenkcom/opensmalltalk-vm

Cheers,
Andrei



[2] checking for cairo's PNG functions feature... 
configure: WARNING: Could not find libpng in the pkg-config search path
checking whether cairo's PNG functions feature could be enabled... no
configure: error: recommended PNG functions feature could not be enabled

On 14 Sep 2020, at 17:32, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 7:15 AM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

On 12 Sep 2020, at 01:42, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

I don’t think there is something special about that process. It is the process that we start to run tests [1]. The exception happens in the running process and the crash is when copying the stack of that running process.

Ok, cool.  What I’d like to do is get a copy of your test setup and run it in an assert vm to try and get more information.  AFAICT the vm code is good do the bug is not obvious.  An assert vm may give more information before the crash.  Have you tried running the system on an assert vm yet?

Checked some previous logs and we get these kinds of crashes on the CI server since at least two years. So it does not look like a new bug (but who knows).


(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Nice!

I also tried the non-recursive implementation of Context>>#copyTo: from Squeak and it also crashes.

Not sure if related but now in the same image as before I got a different crash and printing the stack does not work. But this time the error seems to come from handleStackOverflow

(lldb) call (void *)printCallStack()
invalid frame pointer
invalid frame pointer
invalid frame pointer
error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=EXC_I386_GPFLT).
The process has been returned to the state before expression evaluation.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x121e00000)
  * frame #0: 0x0000000100162258 libGlamorousToolkitVMCore.dylib`marryFrameSP + 584
    frame #1: 0x0000000100172982 libGlamorousToolkitVMCore.dylib`handleStackOverflow + 354
    frame #2: 0x000000010016b025 libGlamorousToolkitVMCore.dylib`ceStackOverflow + 149
    frame #3: 0x00000001100005b3
    frame #4: 0x0000000100174d99 libGlamorousToolkitVMCore.dylib`ptEnterInterpreterFromCallback + 73


Cheers,
Andrei

[1] ./GlamorousToolkit.app/Contents/MacOS/GlamorousToolkit  Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'



Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue: 
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue: 
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue: 
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
 Special-case non-single contexts (because of context-to-stack mapping).
 Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
  or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001100b1004
->  0x1100b1004: inl    $0x4c, %eax
    0x1100b1006: leal   -0x5c(%rip), %eax
    0x1100b100c: pushq  %r8
    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 
Target 0: (GlamorousToolkit) stopped.


Process 29929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001100fe7ed
->  0x1100fe7ed: int3   
    0x1100fe7ee: int3   
    0x1100fe7ef: int3   
    0x1100fe7f0: int3   
Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





-- 
_,,,^..^,,,_
best, Eliot


-- 
_,,,^..^,,,_
best, Eliot

_,,,^..^,,,_ (phone)



--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Andrei Chis
 
Hi Eliot,

Below are some examples of primitive traces: first two are with the headless vm and the third with the normal vm.
Did not manage yet to get a crash while running the production vm with leak checks enabled.

Cheers,
Andrei


—headless vm—
Most recent primitives
wait
class
signal
class
basicIdentityHash
class
basicIdentityHash
**StackOverflow**
basicIdentityHash
**StackOverflow**
basicNew
basicNew
basicNew
signal
basicNew
wait
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
wait
signal
basicNew
wait
**StackOverflow**
basicIdentityHash
signal
wait
**StackOverflow**
signal
wait
**StackOverflow**
signal
wait
**StackOverflow**
signal
wait
**StackOverflow**
signal
wait
**StackOverflow**
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
**StackOverflow**
wait
signal
basicNew
basicNew
class
**StackOverflow**
class
basicNew
**StackOverflow**
wait
basicIdentityHash
signal
**StackOverflow**
basicIdentityHash
**StackOverflow**
basicIdentityHash
class
**StackOverflow**
basicNew
basicNew
basicNew
basicNew
class
**StackOverflow**
basicNew
wait
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
wait
signal
wait
**StackOverflow**
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
**StackOverflow**
wait
signal
**StackOverflow**
basicNew
class
**StackOverflow**
**StackOverflow**
basicNew
**StackOverflow**
**StackOverflow**
**StackOverflow**
wait
class
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
wait
signal
wait
**StackOverflow**
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
wait
**StackOverflow**
signal
**StackOverflow**
basicNew
class
**StackOverflow**
**StackOverflow**
**StackOverflow**
wait
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
**StackOverflow**
wait
signal
wait
**StackOverflow**
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
wait
**StackOverflow**
signal
**StackOverflow**
**StackOverflow**
**StackOverflow**
value
>
value:
value
/
**PrimitiveFailure**
basicNew
value:
class
findNextHandlerOrSignalingContext
at:
at:
tempAt:
tempAt:
basicNew
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**CompactCode**

stack page bytes 8192 available headroom 5576 minimum unused headroom 4920


—headless vm—
Most recent primitives
wait
signal
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
wait
signal
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
wait
signal
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
class
wait
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
signal
**StackOverflow**
class
wait
**StackOverflow**
signal
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
wait
signal
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
signal
basicNew
wait
basicIdentityHash
signal
basicIdentityHash
basicIdentityHash
basicNew
basicNew
basicNew
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
replaceFrom:to:with:startingAt:
valueWithArguments:
valueWithArguments:
valueWithArguments:
valueWithArguments:
basicNew
class
findNextHandlerOrSignalingContext
at:
at:
tempAt:
tempAt:
basicNew
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**CompactCode**
**StackOverflow**


—non-headless vm—
Most recent primitives
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
decimalDigitLength
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
basicNew
basicNew
stringHash:initialHash:
at:
add:
stringHash:initialHash:
add:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
class
stringHash:initialHash:
class
size
compare:with:collated:
primitiveChangeClassTo:
**StackOverflow**
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
isNil
compare:with:collated:
isNil
compare:with:collated:
primitiveChangeClassTo:
stringHash:initialHash:
compare:with:collated:
isString
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
basicIdentityHash
basicNew:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
class
class
class
size
class
stringHash:initialHash:
compare:with:collated:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
compare:with:collated:
class
shallowCopy
shallowCopy
class
**StackOverflow**
**StackOverflow**
**StackOverflow**
class
**StackOverflow**
**StackOverflow**
class
**StackOverflow**
basicIdentityHash
basicNew
basicNew
class
basicNew
perform:withArguments:
basicNew
basicNew
basicNew
basicNew
basicNew
signal
signal
basicNew
basicNew
**StackOverflow**
**StackOverflow**
signal
signal
**StackOverflow**
**StackOverflow**
class
**StackOverflow**
**StackOverflow**
basicNew
**StackOverflow**
class
signal
signal
basicNew
signal
signal
basicNew
signal
signal
basicNew
**StackOverflow**
class
class
class
basicNew
basicNew
**StackOverflow**
class
class
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
signal
signal
class
findSubstring:in:startingAt:matchTable:
indexOfAscii:inString:startingAt:
indexOfAscii:inString:startingAt:
findSubstring:in:startingAt:matchTable:
indexOfAscii:inString:startingAt:
indexOfAscii:inString:startingAt:
object:perform:withArguments:inClass:
basicNew
findNextHandlerOrSignalingContext
at:
at:
tempAt:
class
tempAt:
basicNew
class
size
class
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**CompactCode**



On 22 Sep 2020, at 21:09, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Tue, Sep 22, 2020 at 5:21 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Unfortunately with the assert vm with memory leak checks it seems I cannot reproduce the problem. 
I’ll try to run it a few more times, maybe I get lucky and get a crash (a run takes a few hours). 

For our case, calling `self pc` before copying a context really helped. Before almost 50% of our builds were failing; now we get no more failures at all.
Thanks for your insights!

Are there other ways to get relevant/helpful details from a crash with a normal vm?

The crash.log should include the primitive trace log, which is the last 256 external primitives the VM has called. Can you post that part of the crash log?

The memory leak checking can be turned on in a production VM, it's just that the output is a bit harder to parse.  So try that also.
Cheers,
Andrei


On 15 Sep 2020, at 01:27, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 3:22 PM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

The setup in GT is a bit customised (some changes in the headless vm, some custom plugins, custom rendering) so I first thought it will be impossible to reproduce the bug in a more standard manner. 
However turns out it is possible. If I use the following script after running the tests a few times in lldb I get the crash starting from a plain Pharo 8 image.

$ ./pharo Pharo.image st --quit 

$ lldb ./pharo-vm/Pharo.app/Contents/MacOS/Pharo
(lldb) run --headless Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'


I also tried to compile the vm myself on Mac (Catalina 10.15.6). I build a normal and assert for https://github.com/OpenSmalltalk/opensmalltalk-vm and https://github.com/pharo-project/opensmalltalk-vm from the cog branch.
In both cases I get an issue related to pixman 0.34.0 [1] but that’s easy to workaround. For https://github.com/OpenSmalltalk/opensmalltalk-vm I got an extra problem related to Cairo [2] and had to change libpng from libpng16 to libpng12 to get it to work.

With both the normal VMs I could reproduce the bug and got stacks with the Context>copyTo: messages. 

With the assert VMs I only got a crash for now with the assert vm from https://github.com/pharo-project/opensmalltalk-vm. However there is no Context>copyTo: and the memory seems quite corrupted. 
I suspect the crash also appears in https://github.com/OpenSmalltalk/opensmalltalk-vm  but seems that with the assert vm it is much harder to reproduce. Had to run the tests 20 times and got one crash; running the tests once take 20-30 minutes. 


This is from only crash until now with the assert vm. Not sure if they are helpful or not, or actually related to the problem.

validInstructionPointerinFrame(GIV(instructionPointer), GIV(framePointer)) 18471
Pharo was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 73731 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
    frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
   139    static inline sqInt intAtPointer(char *ptr) { return (sqInt)(*((int *)ptr)); }
   140    static inline sqInt intAtPointerput(char *ptr, int val) { return (sqInt)(*((int *)ptr)= val); }
   141    static inline sqInt longAtPointer(char *ptr) { return *(sqInt *)ptr; }
-> 142    static inline sqInt longAtPointerput(char *ptr, sqInt val) { return *(sqInt *)ptr= val; }
   143    static inline sqLong long64AtPointer(char *ptr) { return *(sqLong *)ptr; }
   144    static inline sqLong long64AtPointerput(char *ptr, sqLong val) { return *(sqLong *)ptr= val; }
   145    static inline float singleFloatAtPointer(char *ptr) { return *(float *)ptr; }
Target 0: (Pharo) stopped.


(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
  * frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
    frame #1: 0x00000001000161cf Pharo`marryFrameSP(theFP=<unavailable>, theSP=0x0000000000000000) at gcc3x-cointerp.c:68120:3[opt]
    frame #2: 0x000000010001f5ac Pharo`ceContextinstVar(maybeContext=5510359872, slotIndex=0) at gcc3x-cointerp.c:15221:12 [opt]
    frame #3: 0x00000001480017d6
    frame #4: 0x00000001000022be Pharo`interpret at gcc3x-cointerp.c:2755:3 [opt]
    frame #5: 0x00000001000bc244 Pharo`-[sqSqueakMainApplication runSqueak](self=0x0000000101c76dc0, _cmd=<unavailable>) at sqSqueakMainApplication.m:201:2 [opt]
    frame #6: 0x00007fff3326729b Foundation`__NSFirePerformWithOrder + 360
    frame #7: 0x00007fff30ad3335 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    frame #8: 0x00007fff30ad3267 CoreFoundation`__CFRunLoopDoObservers + 457
    frame #9: 0x00007fff30ad2805 CoreFoundation`__CFRunLoopRun + 874
    frame #10: 0x00007fff30ad1e3e CoreFoundation`CFRunLoopRunSpecific + 462
    frame #11: 0x00007fff2f6feabd HIToolbox`RunCurrentEventLoopInMode + 292
    frame #12: 0x00007fff2f6fe6f4 HIToolbox`ReceiveNextEventCommon + 359
    frame #13: 0x00007fff2f6fe579 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 64
    frame #14: 0x00007fff2dd44039 AppKit`_DPSNextEvent + 883
    frame #15: 0x00007fff2dd42880 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
    frame #16: 0x00007fff2dd3458e AppKit`-[NSApplication run] + 658
    frame #17: 0x00007fff2dd06396 AppKit`NSApplicationMain + 777
    frame #18: 0x00007fff6ab3ecc9 libdyld.dylib`start + 1


(lldb) call printCallStack()
    0x7ffeefbe3920 M INVALID RECEIVER>(nil) 0x148716b40: a(n) bad class
    0x7ffeefbe3968 M [] in INVALID RECEIVER>(nil) Context(Object)>>doesNotUnderstand: #bounds
 0x194648118: a(n) bad class
    0x7ffeefbe39a8 M INVALID RECEIVER>(nil) 0x1489fcec0: a(n) bad class
    0x7ffeefbe39e8 M INVALID RECEIVER>(nil)  0x1489fcec0: a(n) bad class
    0x7ffeefbe3a30 I INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3a80 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3ab8 M INVALID RECEIVER>(nil)  0x148163cd0: a(n) bad class
    0x7ffeefbe3b08 I INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b40 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b78 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe3bc0 I INVALID RECEIVER>(nil)  0x148716a38: a(n) bad class
    0x7ffeefbe3c10 I INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c40 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c78 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3cc0 M INVALID RECEIVER>(nil)  0x14d0337f0: a(n) bad class
    0x7ffeefbe3d08 M INVALID RECEIVER>(nil)  0x14d033738: a(n) bad class
    0x7ffeefbe3d50 M INVALID RECEIVER>(nil)  0x14d033680: a(n) bad class
    0x7ffeefbe3d98 M INVALID RECEIVER>(nil)  0x1946493f0: a(n) bad class
    0x7ffeefbe3de0 M INVALID RECEIVER>(nil)  0x194649338: a(n) bad class
    0x7ffeefbe3e28 M INVALID RECEIVER>(nil)  0x194649280: a(n) bad class
    0x7ffeefbe3e70 M INVALID RECEIVER>(nil)  0x1946491c8: a(n) bad class
    0x7ffeefbe3eb8 M INVALID RECEIVER>(nil)  0x194649110: a(n) bad class
    0x7ffeefbec768 M INVALID RECEIVER>(nil)  0x194649038: a(n) bad class
    0x7ffeefbec7b0 M INVALID RECEIVER>(nil)  0x194648f60: a(n) bad class
    0x7ffeefbec7f8 M INVALID RECEIVER>(nil)  0x194648e88: a(n) bad class
    0x7ffeefbec840 M INVALID RECEIVER>(nil)  0x194648dd0: a(n) bad class
    0x7ffeefbec888 M INVALID RECEIVER>(nil)  0x194648d18: a(n) bad class
    0x7ffeefbec8d0 M INVALID RECEIVER>(nil)  0x194648c60: a(n) bad class
    0x7ffeefbec918 M INVALID RECEIVER>(nil)  0x194648b88: a(n) bad class
    0x7ffeefbec960 M INVALID RECEIVER>(nil)  0x194648ad0: a(n) bad class
    0x7ffeefbec9a8 M INVALID RECEIVER>(nil)  0x194648a18: a(n) bad class
    0x7ffeefbec9f0 M INVALID RECEIVER>(nil)  0x194648960: a(n) bad class
    0x7ffeefbeca38 M INVALID RECEIVER>(nil)  0x1946488a8: a(n) bad class
    0x7ffeefbeca80 M INVALID RECEIVER>(nil)  0x1946487f0: a(n) bad class
    0x7ffeefbecac8 M INVALID RECEIVER>(nil)  0x194648708: a(n) bad class
    0x7ffeefbecb10 M INVALID RECEIVER>(nil)  0x194648620: a(n) bad class
    0x7ffeefbecb58 M INVALID RECEIVER>(nil)  0x194648508: a(n) bad class
    0x7ffeefbecba0 M INVALID RECEIVER>(nil)  0x194648450: a(n) bad class
    0x7ffeefbecbe8 M INVALID RECEIVER>(nil)  0x1481641a8: a(n) bad class
    0x7ffeefbecc30 M INVALID RECEIVER>(nil)  0x1481640f0: a(n) bad class
    0x7ffeefbecc78 M INVALID RECEIVER>(nil)  0x148164038: a(n) bad class
    0x7ffeefbeccc0 M INVALID RECEIVER>(nil)  0x148163f80: a(n) bad class
    0x7ffeefbecd08 M INVALID RECEIVER>(nil)  0x148163ec8: a(n) bad class
    0x7ffeefbecd50 M INVALID RECEIVER>(nil)  0x148163e10: a(n) bad class
    0x7ffeefbecd98 M INVALID RECEIVER>(nil)  0x148163d28: a(n) bad class
    0x7ffeefbecde0 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbece28 M INVALID RECEIVER>(nil)  0x148163b38: a(n) bad class
    0x7ffeefbece70 M INVALID RECEIVER>(nil)  0x148163a80: a(n) bad class
    0x7ffeefbeceb8 M INVALID RECEIVER>(nil)  0x1481639c8: a(n) bad class
    0x7ffeefbe7758 M INVALID RECEIVER>(nil)  0x1481638e0: a(n) bad class
    0x7ffeefbe77a0 M INVALID RECEIVER>(nil)  0x148163808: a(n) bad class
    0x7ffeefbe77e8 M INVALID RECEIVER>(nil)  0x148163750: a(n) bad class
    0x7ffeefbe7830 M INVALID RECEIVER>(nil)  0x148163698: a(n) bad class
    0x7ffeefbe7878 M INVALID RECEIVER>(nil)  0x1481635c0: a(n) bad class
    0x7ffeefbe78c0 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe7908 M INVALID RECEIVER>(nil)  0x148163408: a(n) bad class
    0x7ffeefbe7950 M INVALID RECEIVER>(nil)  0x148163350: a(n) bad class
    0x7ffeefbe7998 M INVALID RECEIVER>(nil)  0x148163298: a(n) bad class
    0x7ffeefbe79e0 M INVALID RECEIVER>(nil)  0x148163188: a(n) bad class
    0x7ffeefbe7a28 M INVALID RECEIVER>(nil)  0x148163098: a(n) bad class
    0x7ffeefbe7a70 M INVALID RECEIVER>(nil)  0x148162fa0: a(n) bad class
    0x7ffeefbe7ab8 M INVALID RECEIVER>(nil)  0x148162ec8: a(n) bad class
    0x7ffeefbe7b00 M INVALID RECEIVER>(nil)  0x148162e10: a(n) bad class
    0x7ffeefbe7b48 M INVALID RECEIVER>(nil)  0x148712a08: a(n) bad class
    0x7ffeefbe7b90 M INVALID RECEIVER>(nil)  0x148712950: a(n) bad class
    0x7ffeefbe7bd8 M INVALID RECEIVER>(nil)  0x148712898: a(n) bad class
    0x7ffeefbe7c20 M INVALID RECEIVER>(nil)  0x148713cc0: a(n) bad class
    0x7ffeefbe7c68 M INVALID RECEIVER>(nil)  0x148713018: a(n) bad class
    0x7ffeefbe7cb0 M INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbe7cf8 M INVALID RECEIVER>(nil)  0x148713140: a(n) bad class
    0x7ffeefbe7d40 M INVALID RECEIVER>(nil)  0x148713928: a(n) bad class
    0x7ffeefbe7d88 M INVALID RECEIVER>(nil)  0x1487133c8: a(n) bad class
    0x7ffeefbe7de0 I INVALID RECEIVER>(nil)  0x148713238: a(n) bad class
    0x7ffeefbe7e28 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7e70 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7eb8 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeea68 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeac8 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeb00 M INVALID RECEIVER>(nil)  0x148713108: a(n) bad class
    0x7ffeefbeeb50 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeeb98 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeebe0 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbeec10 M INVALID RECEIVER>(nil) 0x9=1
    0x7ffeefbeec48 M INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeeca0 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeecd0 M INVALID RECEIVER>(nil)  0x1487130d0: a(n) bad class
    0x7ffeefbeed10 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeed58 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeedb0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeedf0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeee20 M [] in INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbeee78 M INVALID RECEIVER>(nil)  0x148162df0: a(n) bad class
    0x7ffeefbeeec0 M INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbe5978 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe59d8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5a18 M INVALID RECEIVER>(nil)  0x148163150: a(n) bad class
    0x7ffeefbe5a68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5aa0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5ad8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5b08 M INVALID RECEIVER>(nil)  0x1481634c0: a(n) bad class
    0x7ffeefbe5b48 M INVALID RECEIVER>(nil)  0x14c403ca8: a(n) bad class
    0x7ffeefbe5b88 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bc0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bf0 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c20 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c98 M INVALID RECEIVER>(nil)  0x194656468: a(n) bad class
    0x7ffeefbe5cd0 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d00 M INVALID RECEIVER>(nil)  0x148163bf0: a(n) bad class
    0x7ffeefbe5d50 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d88 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5dc0 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5e00 M INVALID RECEIVER>(nil)  0x148163de0: a(n) bad class
    0x7ffeefbe5e38 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5e80 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5eb8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdf9d8 M INVALID RECEIVER>(nil)  0x194648430: a(n) bad class
    0x7ffeefbdfa10 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfa50 M [] in INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfa90 M INVALID RECEIVER>(nil)  0x1946486d8: a(n) bad class
    0x7ffeefbdfad0 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfb10 M INVALID RECEIVER>(nil)  0x1946485e0: a(n) bad class
    0x7ffeefbdfb48 M INVALID RECEIVER>(nil)  0x1489f7da8: a(n) bad class
    0x7ffeefbdfb80 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfbb8 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfbe8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfc20 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc58 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc98 M INVALID RECEIVER>(nil)  0x194648c40: a(n) bad class
    0x7ffeefbdfcc8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd08 M INVALID RECEIVER>(nil)  0x194648f40: a(n) bad class
    0x7ffeefbdfd40 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd70 M INVALID RECEIVER>(nil)  0x14902a730: a(n) bad class
    0x7ffeefbdfdb0 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfde8 M INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe20 M [] in INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe70 M [] in INVALID RECEIVER>(nil)  0x14d032f98: a(n) bad class
    0x7ffeefbdfeb8 M INVALID RECEIVER>(nil)  0x14d032fe0: a(n) bad class

(callerContextOrNil == (nilObject())) || (isContext(callerContextOrNil)) 72783
       0x14d033738 is not a context

OK, interesting.  Both the assert failure and the badly corrupted stack trace lead me to believe that the issue happens long before the crash and is probably a stack corruption, either by a primitive cutting back the stack incorrectly, or some other hot riot ion (for example are all those nils in INVALID RECEIVER>(nil) real or an artifact of attempting to print an invalid value?).

So the next step is to run the asset vm with leak checking turned on.  Use

myvm —leakcheck 3 to check after every GC

We can add, eg leak checking after an FFI call, in an afternoon

A more realistic setup would be to run GT with an assert headless vm. But until now I did not figure out how to build an assert vm for the gt-headless branch from https://github.com/feenkcom/opensmalltalk-vm

Cheers,
Andrei



[2] checking for cairo's PNG functions feature... 
configure: WARNING: Could not find libpng in the pkg-config search path
checking whether cairo's PNG functions feature could be enabled... no
configure: error: recommended PNG functions feature could not be enabled

On 14 Sep 2020, at 17:32, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 7:15 AM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

On 12 Sep 2020, at 01:42, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

I don’t think there is something special about that process. It is the process that we start to run tests [1]. The exception happens in the running process and the crash is when copying the stack of that running process.

Ok, cool.  What I’d like to do is get a copy of your test setup and run it in an assert vm to try and get more information.  AFAICT the vm code is good do the bug is not obvious.  An assert vm may give more information before the crash.  Have you tried running the system on an assert vm yet?

Checked some previous logs and we get these kinds of crashes on the CI server since at least two years. So it does not look like a new bug (but who knows).


(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Nice!

I also tried the non-recursive implementation of Context>>#copyTo: from Squeak and it also crashes.

Not sure if related but now in the same image as before I got a different crash and printing the stack does not work. But this time the error seems to come from handleStackOverflow

(lldb) call (void *)printCallStack()
invalid frame pointer
invalid frame pointer
invalid frame pointer
error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=EXC_I386_GPFLT).
The process has been returned to the state before expression evaluation.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x121e00000)
  * frame #0: 0x0000000100162258 libGlamorousToolkitVMCore.dylib`marryFrameSP + 584
    frame #1: 0x0000000100172982 libGlamorousToolkitVMCore.dylib`handleStackOverflow + 354
    frame #2: 0x000000010016b025 libGlamorousToolkitVMCore.dylib`ceStackOverflow + 149
    frame #3: 0x00000001100005b3
    frame #4: 0x0000000100174d99 libGlamorousToolkitVMCore.dylib`ptEnterInterpreterFromCallback + 73


Cheers,
Andrei

[1] ./GlamorousToolkit.app/Contents/MacOS/GlamorousToolkit  Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'



Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue: 
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue: 
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue: 
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
 Special-case non-single contexts (because of context-to-stack mapping).
 Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
  or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001100b1004
->  0x1100b1004: inl    $0x4c, %eax
    0x1100b1006: leal   -0x5c(%rip), %eax
    0x1100b100c: pushq  %r8
    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 
Target 0: (GlamorousToolkit) stopped.


Process 29929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001100fe7ed
->  0x1100fe7ed: int3   
    0x1100fe7ee: int3   
    0x1100fe7ef: int3   
    0x1100fe7f0: int3   
Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





-- 
_,,,^..^,,,_
best, Eliot


-- 
_,,,^..^,,,_
best, Eliot

_,,,^..^,,,_ (phone)



-- 
_,,,^..^,,,_
best, Eliot

Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Andrei Chis
In reply to this post by Nicolas Cellier
 


On 22 Sep 2020, at 19:18, Nicolas Cellier <[hidden email]> wrote:

If you are on linux, you could try using the recording debugger (rr)... Though, debugging a -O2 or -O3 is hard... No obvious mapping of source code with assembly instructions. It's better if you can reproduce on debug version...

Mostly I’m on mac but might try this if all else fails.


Le mar. 22 sept. 2020 à 14:21, Andrei Chis <[hidden email]> a écrit :
 
Hi Eliot,

Unfortunately with the assert vm with memory leak checks it seems I cannot reproduce the problem. 
I’ll try to run it a few more times, maybe I get lucky and get a crash (a run takes a few hours). 

For our case, calling `self pc` before copying a context really helped. Before almost 50% of our builds were failing; now we get no more failures at all.
Thanks for your insights!

Are there other ways to get relevant/helpful details from a crash with a normal vm?

Cheers,
Andrei


On 15 Sep 2020, at 01:27, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 3:22 PM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

The setup in GT is a bit customised (some changes in the headless vm, some custom plugins, custom rendering) so I first thought it will be impossible to reproduce the bug in a more standard manner. 
However turns out it is possible. If I use the following script after running the tests a few times in lldb I get the crash starting from a plain Pharo 8 image.

$ ./pharo Pharo.image st --quit 

$ lldb ./pharo-vm/Pharo.app/Contents/MacOS/Pharo
(lldb) run --headless Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'


I also tried to compile the vm myself on Mac (Catalina 10.15.6). I build a normal and assert for https://github.com/OpenSmalltalk/opensmalltalk-vm and https://github.com/pharo-project/opensmalltalk-vm from the cog branch.
In both cases I get an issue related to pixman 0.34.0 [1] but that’s easy to workaround. For https://github.com/OpenSmalltalk/opensmalltalk-vm I got an extra problem related to Cairo [2] and had to change libpng from libpng16 to libpng12 to get it to work.

With both the normal VMs I could reproduce the bug and got stacks with the Context>copyTo: messages. 

With the assert VMs I only got a crash for now with the assert vm from https://github.com/pharo-project/opensmalltalk-vm. However there is no Context>copyTo: and the memory seems quite corrupted. 
I suspect the crash also appears in https://github.com/OpenSmalltalk/opensmalltalk-vm  but seems that with the assert vm it is much harder to reproduce. Had to run the tests 20 times and got one crash; running the tests once take 20-30 minutes. 


This is from only crash until now with the assert vm. Not sure if they are helpful or not, or actually related to the problem.

validInstructionPointerinFrame(GIV(instructionPointer), GIV(framePointer)) 18471
Pharo was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 73731 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
    frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
   139    static inline sqInt intAtPointer(char *ptr) { return (sqInt)(*((int *)ptr)); }
   140    static inline sqInt intAtPointerput(char *ptr, int val) { return (sqInt)(*((int *)ptr)= val); }
   141    static inline sqInt longAtPointer(char *ptr) { return *(sqInt *)ptr; }
-> 142    static inline sqInt longAtPointerput(char *ptr, sqInt val) { return *(sqInt *)ptr= val; }
   143    static inline sqLong long64AtPointer(char *ptr) { return *(sqLong *)ptr; }
   144    static inline sqLong long64AtPointerput(char *ptr, sqLong val) { return *(sqLong *)ptr= val; }
   145    static inline float singleFloatAtPointer(char *ptr) { return *(float *)ptr; }
Target 0: (Pharo) stopped.


(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
  * frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
    frame #1: 0x00000001000161cf Pharo`marryFrameSP(theFP=<unavailable>, theSP=0x0000000000000000) at gcc3x-cointerp.c:68120:3[opt]
    frame #2: 0x000000010001f5ac Pharo`ceContextinstVar(maybeContext=5510359872, slotIndex=0) at gcc3x-cointerp.c:15221:12 [opt]
    frame #3: 0x00000001480017d6
    frame #4: 0x00000001000022be Pharo`interpret at gcc3x-cointerp.c:2755:3 [opt]
    frame #5: 0x00000001000bc244 Pharo`-[sqSqueakMainApplication runSqueak](self=0x0000000101c76dc0, _cmd=<unavailable>) at sqSqueakMainApplication.m:201:2 [opt]
    frame #6: 0x00007fff3326729b Foundation`__NSFirePerformWithOrder + 360
    frame #7: 0x00007fff30ad3335 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    frame #8: 0x00007fff30ad3267 CoreFoundation`__CFRunLoopDoObservers + 457
    frame #9: 0x00007fff30ad2805 CoreFoundation`__CFRunLoopRun + 874
    frame #10: 0x00007fff30ad1e3e CoreFoundation`CFRunLoopRunSpecific + 462
    frame #11: 0x00007fff2f6feabd HIToolbox`RunCurrentEventLoopInMode + 292
    frame #12: 0x00007fff2f6fe6f4 HIToolbox`ReceiveNextEventCommon + 359
    frame #13: 0x00007fff2f6fe579 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 64
    frame #14: 0x00007fff2dd44039 AppKit`_DPSNextEvent + 883
    frame #15: 0x00007fff2dd42880 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
    frame #16: 0x00007fff2dd3458e AppKit`-[NSApplication run] + 658
    frame #17: 0x00007fff2dd06396 AppKit`NSApplicationMain + 777
    frame #18: 0x00007fff6ab3ecc9 libdyld.dylib`start + 1


(lldb) call printCallStack()
    0x7ffeefbe3920 M INVALID RECEIVER>(nil) 0x148716b40: a(n) bad class
    0x7ffeefbe3968 M [] in INVALID RECEIVER>(nil) Context(Object)>>doesNotUnderstand: #bounds
 0x194648118: a(n) bad class
    0x7ffeefbe39a8 M INVALID RECEIVER>(nil) 0x1489fcec0: a(n) bad class
    0x7ffeefbe39e8 M INVALID RECEIVER>(nil)  0x1489fcec0: a(n) bad class
    0x7ffeefbe3a30 I INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3a80 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3ab8 M INVALID RECEIVER>(nil)  0x148163cd0: a(n) bad class
    0x7ffeefbe3b08 I INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b40 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b78 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe3bc0 I INVALID RECEIVER>(nil)  0x148716a38: a(n) bad class
    0x7ffeefbe3c10 I INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c40 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c78 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3cc0 M INVALID RECEIVER>(nil)  0x14d0337f0: a(n) bad class
    0x7ffeefbe3d08 M INVALID RECEIVER>(nil)  0x14d033738: a(n) bad class
    0x7ffeefbe3d50 M INVALID RECEIVER>(nil)  0x14d033680: a(n) bad class
    0x7ffeefbe3d98 M INVALID RECEIVER>(nil)  0x1946493f0: a(n) bad class
    0x7ffeefbe3de0 M INVALID RECEIVER>(nil)  0x194649338: a(n) bad class
    0x7ffeefbe3e28 M INVALID RECEIVER>(nil)  0x194649280: a(n) bad class
    0x7ffeefbe3e70 M INVALID RECEIVER>(nil)  0x1946491c8: a(n) bad class
    0x7ffeefbe3eb8 M INVALID RECEIVER>(nil)  0x194649110: a(n) bad class
    0x7ffeefbec768 M INVALID RECEIVER>(nil)  0x194649038: a(n) bad class
    0x7ffeefbec7b0 M INVALID RECEIVER>(nil)  0x194648f60: a(n) bad class
    0x7ffeefbec7f8 M INVALID RECEIVER>(nil)  0x194648e88: a(n) bad class
    0x7ffeefbec840 M INVALID RECEIVER>(nil)  0x194648dd0: a(n) bad class
    0x7ffeefbec888 M INVALID RECEIVER>(nil)  0x194648d18: a(n) bad class
    0x7ffeefbec8d0 M INVALID RECEIVER>(nil)  0x194648c60: a(n) bad class
    0x7ffeefbec918 M INVALID RECEIVER>(nil)  0x194648b88: a(n) bad class
    0x7ffeefbec960 M INVALID RECEIVER>(nil)  0x194648ad0: a(n) bad class
    0x7ffeefbec9a8 M INVALID RECEIVER>(nil)  0x194648a18: a(n) bad class
    0x7ffeefbec9f0 M INVALID RECEIVER>(nil)  0x194648960: a(n) bad class
    0x7ffeefbeca38 M INVALID RECEIVER>(nil)  0x1946488a8: a(n) bad class
    0x7ffeefbeca80 M INVALID RECEIVER>(nil)  0x1946487f0: a(n) bad class
    0x7ffeefbecac8 M INVALID RECEIVER>(nil)  0x194648708: a(n) bad class
    0x7ffeefbecb10 M INVALID RECEIVER>(nil)  0x194648620: a(n) bad class
    0x7ffeefbecb58 M INVALID RECEIVER>(nil)  0x194648508: a(n) bad class
    0x7ffeefbecba0 M INVALID RECEIVER>(nil)  0x194648450: a(n) bad class
    0x7ffeefbecbe8 M INVALID RECEIVER>(nil)  0x1481641a8: a(n) bad class
    0x7ffeefbecc30 M INVALID RECEIVER>(nil)  0x1481640f0: a(n) bad class
    0x7ffeefbecc78 M INVALID RECEIVER>(nil)  0x148164038: a(n) bad class
    0x7ffeefbeccc0 M INVALID RECEIVER>(nil)  0x148163f80: a(n) bad class
    0x7ffeefbecd08 M INVALID RECEIVER>(nil)  0x148163ec8: a(n) bad class
    0x7ffeefbecd50 M INVALID RECEIVER>(nil)  0x148163e10: a(n) bad class
    0x7ffeefbecd98 M INVALID RECEIVER>(nil)  0x148163d28: a(n) bad class
    0x7ffeefbecde0 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbece28 M INVALID RECEIVER>(nil)  0x148163b38: a(n) bad class
    0x7ffeefbece70 M INVALID RECEIVER>(nil)  0x148163a80: a(n) bad class
    0x7ffeefbeceb8 M INVALID RECEIVER>(nil)  0x1481639c8: a(n) bad class
    0x7ffeefbe7758 M INVALID RECEIVER>(nil)  0x1481638e0: a(n) bad class
    0x7ffeefbe77a0 M INVALID RECEIVER>(nil)  0x148163808: a(n) bad class
    0x7ffeefbe77e8 M INVALID RECEIVER>(nil)  0x148163750: a(n) bad class
    0x7ffeefbe7830 M INVALID RECEIVER>(nil)  0x148163698: a(n) bad class
    0x7ffeefbe7878 M INVALID RECEIVER>(nil)  0x1481635c0: a(n) bad class
    0x7ffeefbe78c0 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe7908 M INVALID RECEIVER>(nil)  0x148163408: a(n) bad class
    0x7ffeefbe7950 M INVALID RECEIVER>(nil)  0x148163350: a(n) bad class
    0x7ffeefbe7998 M INVALID RECEIVER>(nil)  0x148163298: a(n) bad class
    0x7ffeefbe79e0 M INVALID RECEIVER>(nil)  0x148163188: a(n) bad class
    0x7ffeefbe7a28 M INVALID RECEIVER>(nil)  0x148163098: a(n) bad class
    0x7ffeefbe7a70 M INVALID RECEIVER>(nil)  0x148162fa0: a(n) bad class
    0x7ffeefbe7ab8 M INVALID RECEIVER>(nil)  0x148162ec8: a(n) bad class
    0x7ffeefbe7b00 M INVALID RECEIVER>(nil)  0x148162e10: a(n) bad class
    0x7ffeefbe7b48 M INVALID RECEIVER>(nil)  0x148712a08: a(n) bad class
    0x7ffeefbe7b90 M INVALID RECEIVER>(nil)  0x148712950: a(n) bad class
    0x7ffeefbe7bd8 M INVALID RECEIVER>(nil)  0x148712898: a(n) bad class
    0x7ffeefbe7c20 M INVALID RECEIVER>(nil)  0x148713cc0: a(n) bad class
    0x7ffeefbe7c68 M INVALID RECEIVER>(nil)  0x148713018: a(n) bad class
    0x7ffeefbe7cb0 M INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbe7cf8 M INVALID RECEIVER>(nil)  0x148713140: a(n) bad class
    0x7ffeefbe7d40 M INVALID RECEIVER>(nil)  0x148713928: a(n) bad class
    0x7ffeefbe7d88 M INVALID RECEIVER>(nil)  0x1487133c8: a(n) bad class
    0x7ffeefbe7de0 I INVALID RECEIVER>(nil)  0x148713238: a(n) bad class
    0x7ffeefbe7e28 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7e70 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7eb8 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeea68 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeac8 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeb00 M INVALID RECEIVER>(nil)  0x148713108: a(n) bad class
    0x7ffeefbeeb50 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeeb98 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeebe0 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbeec10 M INVALID RECEIVER>(nil) 0x9=1
    0x7ffeefbeec48 M INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeeca0 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeecd0 M INVALID RECEIVER>(nil)  0x1487130d0: a(n) bad class
    0x7ffeefbeed10 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeed58 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeedb0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeedf0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeee20 M [] in INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbeee78 M INVALID RECEIVER>(nil)  0x148162df0: a(n) bad class
    0x7ffeefbeeec0 M INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbe5978 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe59d8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5a18 M INVALID RECEIVER>(nil)  0x148163150: a(n) bad class
    0x7ffeefbe5a68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5aa0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5ad8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5b08 M INVALID RECEIVER>(nil)  0x1481634c0: a(n) bad class
    0x7ffeefbe5b48 M INVALID RECEIVER>(nil)  0x14c403ca8: a(n) bad class
    0x7ffeefbe5b88 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bc0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bf0 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c20 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c98 M INVALID RECEIVER>(nil)  0x194656468: a(n) bad class
    0x7ffeefbe5cd0 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d00 M INVALID RECEIVER>(nil)  0x148163bf0: a(n) bad class
    0x7ffeefbe5d50 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d88 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5dc0 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5e00 M INVALID RECEIVER>(nil)  0x148163de0: a(n) bad class
    0x7ffeefbe5e38 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5e80 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5eb8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdf9d8 M INVALID RECEIVER>(nil)  0x194648430: a(n) bad class
    0x7ffeefbdfa10 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfa50 M [] in INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfa90 M INVALID RECEIVER>(nil)  0x1946486d8: a(n) bad class
    0x7ffeefbdfad0 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfb10 M INVALID RECEIVER>(nil)  0x1946485e0: a(n) bad class
    0x7ffeefbdfb48 M INVALID RECEIVER>(nil)  0x1489f7da8: a(n) bad class
    0x7ffeefbdfb80 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfbb8 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfbe8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfc20 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc58 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc98 M INVALID RECEIVER>(nil)  0x194648c40: a(n) bad class
    0x7ffeefbdfcc8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd08 M INVALID RECEIVER>(nil)  0x194648f40: a(n) bad class
    0x7ffeefbdfd40 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd70 M INVALID RECEIVER>(nil)  0x14902a730: a(n) bad class
    0x7ffeefbdfdb0 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfde8 M INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe20 M [] in INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe70 M [] in INVALID RECEIVER>(nil)  0x14d032f98: a(n) bad class
    0x7ffeefbdfeb8 M INVALID RECEIVER>(nil)  0x14d032fe0: a(n) bad class

(callerContextOrNil == (nilObject())) || (isContext(callerContextOrNil)) 72783
       0x14d033738 is not a context

OK, interesting.  Both the assert failure and the badly corrupted stack trace lead me to believe that the issue happens long before the crash and is probably a stack corruption, either by a primitive cutting back the stack incorrectly, or some other hot riot ion (for example are all those nils in INVALID RECEIVER>(nil) real or an artifact of attempting to print an invalid value?).

So the next step is to run the asset vm with leak checking turned on.  Use

myvm —leakcheck 3 to check after every GC

We can add, eg leak checking after an FFI call, in an afternoon

A more realistic setup would be to run GT with an assert headless vm. But until now I did not figure out how to build an assert vm for the gt-headless branch from https://github.com/feenkcom/opensmalltalk-vm

Cheers,
Andrei



[2] checking for cairo's PNG functions feature... 
configure: WARNING: Could not find libpng in the pkg-config search path
checking whether cairo's PNG functions feature could be enabled... no
configure: error: recommended PNG functions feature could not be enabled

On 14 Sep 2020, at 17:32, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 7:15 AM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

On 12 Sep 2020, at 01:42, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

I don’t think there is something special about that process. It is the process that we start to run tests [1]. The exception happens in the running process and the crash is when copying the stack of that running process.

Ok, cool.  What I’d like to do is get a copy of your test setup and run it in an assert vm to try and get more information.  AFAICT the vm code is good do the bug is not obvious.  An assert vm may give more information before the crash.  Have you tried running the system on an assert vm yet?

Checked some previous logs and we get these kinds of crashes on the CI server since at least two years. So it does not look like a new bug (but who knows).


(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Nice!

I also tried the non-recursive implementation of Context>>#copyTo: from Squeak and it also crashes.

Not sure if related but now in the same image as before I got a different crash and printing the stack does not work. But this time the error seems to come from handleStackOverflow

(lldb) call (void *)printCallStack()
invalid frame pointer
invalid frame pointer
invalid frame pointer
error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=EXC_I386_GPFLT).
The process has been returned to the state before expression evaluation.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x121e00000)
  * frame #0: 0x0000000100162258 libGlamorousToolkitVMCore.dylib`marryFrameSP + 584
    frame #1: 0x0000000100172982 libGlamorousToolkitVMCore.dylib`handleStackOverflow + 354
    frame #2: 0x000000010016b025 libGlamorousToolkitVMCore.dylib`ceStackOverflow + 149
    frame #3: 0x00000001100005b3
    frame #4: 0x0000000100174d99 libGlamorousToolkitVMCore.dylib`ptEnterInterpreterFromCallback + 73


Cheers,
Andrei

[1] ./GlamorousToolkit.app/Contents/MacOS/GlamorousToolkit  Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'



Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue: 
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue: 
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue: 
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
 Special-case non-single contexts (because of context-to-stack mapping).
 Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
  or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001100b1004
->  0x1100b1004: inl    $0x4c, %eax
    0x1100b1006: leal   -0x5c(%rip), %eax
    0x1100b100c: pushq  %r8
    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 
Target 0: (GlamorousToolkit) stopped.


Process 29929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001100fe7ed
->  0x1100fe7ed: int3   
    0x1100fe7ee: int3   
    0x1100fe7ef: int3   
    0x1100fe7f0: int3   
Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





-- 
_,,,^..^,,,_
best, Eliot


-- 
_,,,^..^,,,_
best, Eliot

_,,,^..^,,,_ (phone)


Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Eliot Miranda-2
 
Hi Andrei,

    I encountered the same bug when running the full test suite in squeak.  It’s something to do with code compaction.  I made a lot of changes earlier this year to accommodate the ARMv8 code generator.  One thing to do them would be to use a late 2019 vm and see if the problem goes away.

I shall be trying to track it down in the background as I gave other priorities.  But my experience is that once a bug is reproducible it didn’t take too much effort to fix it.

_,,,^..^,,,_ (phone)

On Sep 23, 2020, at 5:29 AM, Andrei Chis <[hidden email]> wrote:



On 22 Sep 2020, at 19:18, Nicolas Cellier <[hidden email]> wrote:

If you are on linux, you could try using the recording debugger (rr)... Though, debugging a -O2 or -O3 is hard... No obvious mapping of source code with assembly instructions. It's better if you can reproduce on debug version...

Mostly I’m on mac but might try this if all else fails.


Le mar. 22 sept. 2020 à 14:21, Andrei Chis <[hidden email]> a écrit :
 
Hi Eliot,

Unfortunately with the assert vm with memory leak checks it seems I cannot reproduce the problem. 
I’ll try to run it a few more times, maybe I get lucky and get a crash (a run takes a few hours). 

For our case, calling `self pc` before copying a context really helped. Before almost 50% of our builds were failing; now we get no more failures at all.
Thanks for your insights!

Are there other ways to get relevant/helpful details from a crash with a normal vm?

Cheers,
Andrei


On 15 Sep 2020, at 01:27, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 3:22 PM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

The setup in GT is a bit customised (some changes in the headless vm, some custom plugins, custom rendering) so I first thought it will be impossible to reproduce the bug in a more standard manner. 
However turns out it is possible. If I use the following script after running the tests a few times in lldb I get the crash starting from a plain Pharo 8 image.

$ ./pharo Pharo.image st --quit 

$ lldb ./pharo-vm/Pharo.app/Contents/MacOS/Pharo
(lldb) run --headless Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'


I also tried to compile the vm myself on Mac (Catalina 10.15.6). I build a normal and assert for https://github.com/OpenSmalltalk/opensmalltalk-vm and https://github.com/pharo-project/opensmalltalk-vm from the cog branch.
In both cases I get an issue related to pixman 0.34.0 [1] but that’s easy to workaround. For https://github.com/OpenSmalltalk/opensmalltalk-vm I got an extra problem related to Cairo [2] and had to change libpng from libpng16 to libpng12 to get it to work.

With both the normal VMs I could reproduce the bug and got stacks with the Context>copyTo: messages. 

With the assert VMs I only got a crash for now with the assert vm from https://github.com/pharo-project/opensmalltalk-vm. However there is no Context>copyTo: and the memory seems quite corrupted. 
I suspect the crash also appears in https://github.com/OpenSmalltalk/opensmalltalk-vm  but seems that with the assert vm it is much harder to reproduce. Had to run the tests 20 times and got one crash; running the tests once take 20-30 minutes. 


This is from only crash until now with the assert vm. Not sure if they are helpful or not, or actually related to the problem.

validInstructionPointerinFrame(GIV(instructionPointer), GIV(framePointer)) 18471
Pharo was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 73731 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
    frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
   139    static inline sqInt intAtPointer(char *ptr) { return (sqInt)(*((int *)ptr)); }
   140    static inline sqInt intAtPointerput(char *ptr, int val) { return (sqInt)(*((int *)ptr)= val); }
   141    static inline sqInt longAtPointer(char *ptr) { return *(sqInt *)ptr; }
-> 142    static inline sqInt longAtPointerput(char *ptr, sqInt val) { return *(sqInt *)ptr= val; }
   143    static inline sqLong long64AtPointer(char *ptr) { return *(sqLong *)ptr; }
   144    static inline sqLong long64AtPointerput(char *ptr, sqLong val) { return *(sqLong *)ptr= val; }
   145    static inline float singleFloatAtPointer(char *ptr) { return *(float *)ptr; }
Target 0: (Pharo) stopped.


(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
  * frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
    frame #1: 0x00000001000161cf Pharo`marryFrameSP(theFP=<unavailable>, theSP=0x0000000000000000) at gcc3x-cointerp.c:68120:3[opt]
    frame #2: 0x000000010001f5ac Pharo`ceContextinstVar(maybeContext=5510359872, slotIndex=0) at gcc3x-cointerp.c:15221:12 [opt]
    frame #3: 0x00000001480017d6
    frame #4: 0x00000001000022be Pharo`interpret at gcc3x-cointerp.c:2755:3 [opt]
    frame #5: 0x00000001000bc244 Pharo`-[sqSqueakMainApplication runSqueak](self=0x0000000101c76dc0, _cmd=<unavailable>) at sqSqueakMainApplication.m:201:2 [opt]
    frame #6: 0x00007fff3326729b Foundation`__NSFirePerformWithOrder + 360
    frame #7: 0x00007fff30ad3335 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    frame #8: 0x00007fff30ad3267 CoreFoundation`__CFRunLoopDoObservers + 457
    frame #9: 0x00007fff30ad2805 CoreFoundation`__CFRunLoopRun + 874
    frame #10: 0x00007fff30ad1e3e CoreFoundation`CFRunLoopRunSpecific + 462
    frame #11: 0x00007fff2f6feabd HIToolbox`RunCurrentEventLoopInMode + 292
    frame #12: 0x00007fff2f6fe6f4 HIToolbox`ReceiveNextEventCommon + 359
    frame #13: 0x00007fff2f6fe579 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 64
    frame #14: 0x00007fff2dd44039 AppKit`_DPSNextEvent + 883
    frame #15: 0x00007fff2dd42880 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
    frame #16: 0x00007fff2dd3458e AppKit`-[NSApplication run] + 658
    frame #17: 0x00007fff2dd06396 AppKit`NSApplicationMain + 777
    frame #18: 0x00007fff6ab3ecc9 libdyld.dylib`start + 1


(lldb) call printCallStack()
    0x7ffeefbe3920 M INVALID RECEIVER>(nil) 0x148716b40: a(n) bad class
    0x7ffeefbe3968 M [] in INVALID RECEIVER>(nil) Context(Object)>>doesNotUnderstand: #bounds
 0x194648118: a(n) bad class
    0x7ffeefbe39a8 M INVALID RECEIVER>(nil) 0x1489fcec0: a(n) bad class
    0x7ffeefbe39e8 M INVALID RECEIVER>(nil)  0x1489fcec0: a(n) bad class
    0x7ffeefbe3a30 I INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3a80 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3ab8 M INVALID RECEIVER>(nil)  0x148163cd0: a(n) bad class
    0x7ffeefbe3b08 I INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b40 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b78 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe3bc0 I INVALID RECEIVER>(nil)  0x148716a38: a(n) bad class
    0x7ffeefbe3c10 I INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c40 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c78 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3cc0 M INVALID RECEIVER>(nil)  0x14d0337f0: a(n) bad class
    0x7ffeefbe3d08 M INVALID RECEIVER>(nil)  0x14d033738: a(n) bad class
    0x7ffeefbe3d50 M INVALID RECEIVER>(nil)  0x14d033680: a(n) bad class
    0x7ffeefbe3d98 M INVALID RECEIVER>(nil)  0x1946493f0: a(n) bad class
    0x7ffeefbe3de0 M INVALID RECEIVER>(nil)  0x194649338: a(n) bad class
    0x7ffeefbe3e28 M INVALID RECEIVER>(nil)  0x194649280: a(n) bad class
    0x7ffeefbe3e70 M INVALID RECEIVER>(nil)  0x1946491c8: a(n) bad class
    0x7ffeefbe3eb8 M INVALID RECEIVER>(nil)  0x194649110: a(n) bad class
    0x7ffeefbec768 M INVALID RECEIVER>(nil)  0x194649038: a(n) bad class
    0x7ffeefbec7b0 M INVALID RECEIVER>(nil)  0x194648f60: a(n) bad class
    0x7ffeefbec7f8 M INVALID RECEIVER>(nil)  0x194648e88: a(n) bad class
    0x7ffeefbec840 M INVALID RECEIVER>(nil)  0x194648dd0: a(n) bad class
    0x7ffeefbec888 M INVALID RECEIVER>(nil)  0x194648d18: a(n) bad class
    0x7ffeefbec8d0 M INVALID RECEIVER>(nil)  0x194648c60: a(n) bad class
    0x7ffeefbec918 M INVALID RECEIVER>(nil)  0x194648b88: a(n) bad class
    0x7ffeefbec960 M INVALID RECEIVER>(nil)  0x194648ad0: a(n) bad class
    0x7ffeefbec9a8 M INVALID RECEIVER>(nil)  0x194648a18: a(n) bad class
    0x7ffeefbec9f0 M INVALID RECEIVER>(nil)  0x194648960: a(n) bad class
    0x7ffeefbeca38 M INVALID RECEIVER>(nil)  0x1946488a8: a(n) bad class
    0x7ffeefbeca80 M INVALID RECEIVER>(nil)  0x1946487f0: a(n) bad class
    0x7ffeefbecac8 M INVALID RECEIVER>(nil)  0x194648708: a(n) bad class
    0x7ffeefbecb10 M INVALID RECEIVER>(nil)  0x194648620: a(n) bad class
    0x7ffeefbecb58 M INVALID RECEIVER>(nil)  0x194648508: a(n) bad class
    0x7ffeefbecba0 M INVALID RECEIVER>(nil)  0x194648450: a(n) bad class
    0x7ffeefbecbe8 M INVALID RECEIVER>(nil)  0x1481641a8: a(n) bad class
    0x7ffeefbecc30 M INVALID RECEIVER>(nil)  0x1481640f0: a(n) bad class
    0x7ffeefbecc78 M INVALID RECEIVER>(nil)  0x148164038: a(n) bad class
    0x7ffeefbeccc0 M INVALID RECEIVER>(nil)  0x148163f80: a(n) bad class
    0x7ffeefbecd08 M INVALID RECEIVER>(nil)  0x148163ec8: a(n) bad class
    0x7ffeefbecd50 M INVALID RECEIVER>(nil)  0x148163e10: a(n) bad class
    0x7ffeefbecd98 M INVALID RECEIVER>(nil)  0x148163d28: a(n) bad class
    0x7ffeefbecde0 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbece28 M INVALID RECEIVER>(nil)  0x148163b38: a(n) bad class
    0x7ffeefbece70 M INVALID RECEIVER>(nil)  0x148163a80: a(n) bad class
    0x7ffeefbeceb8 M INVALID RECEIVER>(nil)  0x1481639c8: a(n) bad class
    0x7ffeefbe7758 M INVALID RECEIVER>(nil)  0x1481638e0: a(n) bad class
    0x7ffeefbe77a0 M INVALID RECEIVER>(nil)  0x148163808: a(n) bad class
    0x7ffeefbe77e8 M INVALID RECEIVER>(nil)  0x148163750: a(n) bad class
    0x7ffeefbe7830 M INVALID RECEIVER>(nil)  0x148163698: a(n) bad class
    0x7ffeefbe7878 M INVALID RECEIVER>(nil)  0x1481635c0: a(n) bad class
    0x7ffeefbe78c0 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe7908 M INVALID RECEIVER>(nil)  0x148163408: a(n) bad class
    0x7ffeefbe7950 M INVALID RECEIVER>(nil)  0x148163350: a(n) bad class
    0x7ffeefbe7998 M INVALID RECEIVER>(nil)  0x148163298: a(n) bad class
    0x7ffeefbe79e0 M INVALID RECEIVER>(nil)  0x148163188: a(n) bad class
    0x7ffeefbe7a28 M INVALID RECEIVER>(nil)  0x148163098: a(n) bad class
    0x7ffeefbe7a70 M INVALID RECEIVER>(nil)  0x148162fa0: a(n) bad class
    0x7ffeefbe7ab8 M INVALID RECEIVER>(nil)  0x148162ec8: a(n) bad class
    0x7ffeefbe7b00 M INVALID RECEIVER>(nil)  0x148162e10: a(n) bad class
    0x7ffeefbe7b48 M INVALID RECEIVER>(nil)  0x148712a08: a(n) bad class
    0x7ffeefbe7b90 M INVALID RECEIVER>(nil)  0x148712950: a(n) bad class
    0x7ffeefbe7bd8 M INVALID RECEIVER>(nil)  0x148712898: a(n) bad class
    0x7ffeefbe7c20 M INVALID RECEIVER>(nil)  0x148713cc0: a(n) bad class
    0x7ffeefbe7c68 M INVALID RECEIVER>(nil)  0x148713018: a(n) bad class
    0x7ffeefbe7cb0 M INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbe7cf8 M INVALID RECEIVER>(nil)  0x148713140: a(n) bad class
    0x7ffeefbe7d40 M INVALID RECEIVER>(nil)  0x148713928: a(n) bad class
    0x7ffeefbe7d88 M INVALID RECEIVER>(nil)  0x1487133c8: a(n) bad class
    0x7ffeefbe7de0 I INVALID RECEIVER>(nil)  0x148713238: a(n) bad class
    0x7ffeefbe7e28 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7e70 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7eb8 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeea68 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeac8 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeb00 M INVALID RECEIVER>(nil)  0x148713108: a(n) bad class
    0x7ffeefbeeb50 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeeb98 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeebe0 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbeec10 M INVALID RECEIVER>(nil) 0x9=1
    0x7ffeefbeec48 M INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeeca0 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeecd0 M INVALID RECEIVER>(nil)  0x1487130d0: a(n) bad class
    0x7ffeefbeed10 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeed58 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeedb0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeedf0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeee20 M [] in INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbeee78 M INVALID RECEIVER>(nil)  0x148162df0: a(n) bad class
    0x7ffeefbeeec0 M INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbe5978 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe59d8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5a18 M INVALID RECEIVER>(nil)  0x148163150: a(n) bad class
    0x7ffeefbe5a68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5aa0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5ad8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5b08 M INVALID RECEIVER>(nil)  0x1481634c0: a(n) bad class
    0x7ffeefbe5b48 M INVALID RECEIVER>(nil)  0x14c403ca8: a(n) bad class
    0x7ffeefbe5b88 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bc0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bf0 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c20 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c98 M INVALID RECEIVER>(nil)  0x194656468: a(n) bad class
    0x7ffeefbe5cd0 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d00 M INVALID RECEIVER>(nil)  0x148163bf0: a(n) bad class
    0x7ffeefbe5d50 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d88 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5dc0 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5e00 M INVALID RECEIVER>(nil)  0x148163de0: a(n) bad class
    0x7ffeefbe5e38 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5e80 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5eb8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdf9d8 M INVALID RECEIVER>(nil)  0x194648430: a(n) bad class
    0x7ffeefbdfa10 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfa50 M [] in INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfa90 M INVALID RECEIVER>(nil)  0x1946486d8: a(n) bad class
    0x7ffeefbdfad0 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfb10 M INVALID RECEIVER>(nil)  0x1946485e0: a(n) bad class
    0x7ffeefbdfb48 M INVALID RECEIVER>(nil)  0x1489f7da8: a(n) bad class
    0x7ffeefbdfb80 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfbb8 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfbe8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfc20 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc58 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc98 M INVALID RECEIVER>(nil)  0x194648c40: a(n) bad class
    0x7ffeefbdfcc8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd08 M INVALID RECEIVER>(nil)  0x194648f40: a(n) bad class
    0x7ffeefbdfd40 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd70 M INVALID RECEIVER>(nil)  0x14902a730: a(n) bad class
    0x7ffeefbdfdb0 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfde8 M INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe20 M [] in INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe70 M [] in INVALID RECEIVER>(nil)  0x14d032f98: a(n) bad class
    0x7ffeefbdfeb8 M INVALID RECEIVER>(nil)  0x14d032fe0: a(n) bad class

(callerContextOrNil == (nilObject())) || (isContext(callerContextOrNil)) 72783
       0x14d033738 is not a context

OK, interesting.  Both the assert failure and the badly corrupted stack trace lead me to believe that the issue happens long before the crash and is probably a stack corruption, either by a primitive cutting back the stack incorrectly, or some other hot riot ion (for example are all those nils in INVALID RECEIVER>(nil) real or an artifact of attempting to print an invalid value?).

So the next step is to run the asset vm with leak checking turned on.  Use

myvm —leakcheck 3 to check after every GC

We can add, eg leak checking after an FFI call, in an afternoon

A more realistic setup would be to run GT with an assert headless vm. But until now I did not figure out how to build an assert vm for the gt-headless branch from https://github.com/feenkcom/opensmalltalk-vm

Cheers,
Andrei



[2] checking for cairo's PNG functions feature... 
configure: WARNING: Could not find libpng in the pkg-config search path
checking whether cairo's PNG functions feature could be enabled... no
configure: error: recommended PNG functions feature could not be enabled

On 14 Sep 2020, at 17:32, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 7:15 AM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

On 12 Sep 2020, at 01:42, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

I don’t think there is something special about that process. It is the process that we start to run tests [1]. The exception happens in the running process and the crash is when copying the stack of that running process.

Ok, cool.  What I’d like to do is get a copy of your test setup and run it in an assert vm to try and get more information.  AFAICT the vm code is good do the bug is not obvious.  An assert vm may give more information before the crash.  Have you tried running the system on an assert vm yet?

Checked some previous logs and we get these kinds of crashes on the CI server since at least two years. So it does not look like a new bug (but who knows).


(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Nice!

I also tried the non-recursive implementation of Context>>#copyTo: from Squeak and it also crashes.

Not sure if related but now in the same image as before I got a different crash and printing the stack does not work. But this time the error seems to come from handleStackOverflow

(lldb) call (void *)printCallStack()
invalid frame pointer
invalid frame pointer
invalid frame pointer
error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=EXC_I386_GPFLT).
The process has been returned to the state before expression evaluation.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x121e00000)
  * frame #0: 0x0000000100162258 libGlamorousToolkitVMCore.dylib`marryFrameSP + 584
    frame #1: 0x0000000100172982 libGlamorousToolkitVMCore.dylib`handleStackOverflow + 354
    frame #2: 0x000000010016b025 libGlamorousToolkitVMCore.dylib`ceStackOverflow + 149
    frame #3: 0x00000001100005b3
    frame #4: 0x0000000100174d99 libGlamorousToolkitVMCore.dylib`ptEnterInterpreterFromCallback + 73


Cheers,
Andrei

[1] ./GlamorousToolkit.app/Contents/MacOS/GlamorousToolkit  Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'



Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue: 
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue: 
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue: 
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
 Special-case non-single contexts (because of context-to-stack mapping).
 Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
  or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001100b1004
->  0x1100b1004: inl    $0x4c, %eax
    0x1100b1006: leal   -0x5c(%rip), %eax
    0x1100b100c: pushq  %r8
    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 
Target 0: (GlamorousToolkit) stopped.


Process 29929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001100fe7ed
->  0x1100fe7ed: int3   
    0x1100fe7ee: int3   
    0x1100fe7ef: int3   
    0x1100fe7f0: int3   
Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





-- 
_,,,^..^,,,_
best, Eliot


-- 
_,,,^..^,,,_
best, Eliot

_,,,^..^,,,_ (phone)


Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Eliot Miranda-2
 
Hi Andrei,

   I can confirm that it does not reproduce with the assert VM and that it does reproduce with the production VM.

What I've found so far is that the VM is in machine code executing the Object>>copy/shallowCopy/postCopy code on a Context, and it has very recently done a code compaction (the most recent entry in the prim trace log is **CodeCompaction** preceded by many shallowCopy invocations, i.e. the VM is looping copying contexts, the shallowCopy primitive needing to call an interpreter primitive to copy a Context, and has just done a code compaction).

In the following I use the terms cog method and machine code method interchangeably.  The JIT creates machine code/cog methods for bytecode methods that are executed more than once, and the Vm spends most of its time executing code in these machine code/cog methods.

The VM is in machine code executing an illegal instruction which is **within the header** of an unrelated machine code method (scanLitWord in the Squeak Parser).  The instructionPointer (the variable used to hold instruction pointers at suspension points) is pointing just past the call of shallowCopy in Object>>copy, consistent with the VM having invoked the shallowCopy primitive.  However, the call instruction preceding instructionPointer is an unlinked send, calling the lookup trampoline ceSend0Args, whereas it should be calling a machine code version of Object>>#shallowCopy.  I can see that there *are* machine code methods for #copy (the Object>>copy method itself and two closed PICs for it).

So my working hypothesis is that the method zone compaction code fails to retain the method referred to by the instructionPointer, reclaiming it itself.  At the start of code compaction the VM must determine what set of methods are live, which it does by scanning stack frames.  And then it reference counts methods accessible from the live method set. And then it discards the least marked methods until it has freed about 1/3 of machine code. It then plans the compaction, updating all instruction pointers in stack frames to point to the compacted methods new locations, and then it compacts.  But because shallowCopy is a primitive its machine code method does not occur in any stack frame (it is invoked, returns a copy without building a frame, and returns that copy).  And there is no reference to the shallowCopy cog method in the Object>>copy cog method, because pre-compaction, the call for #copy will be to an Open PIC, because this send is highly polymorphic

At this point I'm confused as to what is happening and how the native pc ends up executing bogus code.  The instructionPointer is correctly pointing after the send of shallowCopy.  It is regrettable that the VM reclaimed the shallowCopy PIC and the shallowCopy method, but not fatal.  Ah, unless code compaction is undertaken *within* the shallowCopy primitive!!  That would break things.

Right, got it.  So the context being copied has a machine code PC, but this PC is referring to an instruction in a machine code method that has been long ago reclaimed.  So to create the copy of the Context with a valid bytecode pc, the innards of the shallowCopy primitive has to JIT the context's method to be able to compute the correct bytecode PC from the bytecode PC.  If there is no room in the code zone for another method then a code compaction must be done.  This code compaction occurs in the middle of the shallowCopy primitive and because the cog method is not noted by the compaction machinery as being live it gets reclaimed from under the shallowCopy interpreter primitive being called from that machine code.  So when the shallowCopy interpreter primitive returns into machine code it lands somewhere bogus (somewhere in the header of the scanLitWord cog method).

Great.  I understand the bug.  I have a way of provoking it (I can force a code compaction in the middle of shallowCopy on contexts).  So I should be able to understand and fix the bug very soon.  The only thing I don't understand yet is why the bug doesn't occur in the assert or debug VMs.

Anyay, thank you for your patient and informative responses Andrei.  I should have it fixed in a few days.

On Fri, Sep 25, 2020 at 6:26 AM Eliot Miranda <[hidden email]> wrote:
Hi Andrei,

    I encountered the same bug when running the full test suite in squeak.  It’s something to do with code compaction.  I made a lot of changes earlier this year to accommodate the ARMv8 code generator.  One thing to do them would be to use a late 2019 vm and see if the problem goes away.

I shall be trying to track it down in the background as I gave other priorities.  But my experience is that once a bug is reproducible it didn’t take too much effort to fix it.

_,,,^..^,,,_ (phone)

On Sep 23, 2020, at 5:29 AM, Andrei Chis <[hidden email]> wrote:



On 22 Sep 2020, at 19:18, Nicolas Cellier <[hidden email]> wrote:

If you are on linux, you could try using the recording debugger (rr)... Though, debugging a -O2 or -O3 is hard... No obvious mapping of source code with assembly instructions. It's better if you can reproduce on debug version...

Mostly I’m on mac but might try this if all else fails.


Le mar. 22 sept. 2020 à 14:21, Andrei Chis <[hidden email]> a écrit :
 
Hi Eliot,

Unfortunately with the assert vm with memory leak checks it seems I cannot reproduce the problem. 
I’ll try to run it a few more times, maybe I get lucky and get a crash (a run takes a few hours). 

For our case, calling `self pc` before copying a context really helped. Before almost 50% of our builds were failing; now we get no more failures at all.
Thanks for your insights!

Are there other ways to get relevant/helpful details from a crash with a normal vm?

Cheers,
Andrei


On 15 Sep 2020, at 01:27, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 3:22 PM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

The setup in GT is a bit customised (some changes in the headless vm, some custom plugins, custom rendering) so I first thought it will be impossible to reproduce the bug in a more standard manner. 
However turns out it is possible. If I use the following script after running the tests a few times in lldb I get the crash starting from a plain Pharo 8 image.

$ ./pharo Pharo.image st --quit 

$ lldb ./pharo-vm/Pharo.app/Contents/MacOS/Pharo
(lldb) run --headless Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'


I also tried to compile the vm myself on Mac (Catalina 10.15.6). I build a normal and assert for https://github.com/OpenSmalltalk/opensmalltalk-vm and https://github.com/pharo-project/opensmalltalk-vm from the cog branch.
In both cases I get an issue related to pixman 0.34.0 [1] but that’s easy to workaround. For https://github.com/OpenSmalltalk/opensmalltalk-vm I got an extra problem related to Cairo [2] and had to change libpng from libpng16 to libpng12 to get it to work.

With both the normal VMs I could reproduce the bug and got stacks with the Context>copyTo: messages. 

With the assert VMs I only got a crash for now with the assert vm from https://github.com/pharo-project/opensmalltalk-vm. However there is no Context>copyTo: and the memory seems quite corrupted. 
I suspect the crash also appears in https://github.com/OpenSmalltalk/opensmalltalk-vm  but seems that with the assert vm it is much harder to reproduce. Had to run the tests 20 times and got one crash; running the tests once take 20-30 minutes. 


This is from only crash until now with the assert vm. Not sure if they are helpful or not, or actually related to the problem.

validInstructionPointerinFrame(GIV(instructionPointer), GIV(framePointer)) 18471
Pharo was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 73731 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
    frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
   139    static inline sqInt intAtPointer(char *ptr) { return (sqInt)(*((int *)ptr)); }
   140    static inline sqInt intAtPointerput(char *ptr, int val) { return (sqInt)(*((int *)ptr)= val); }
   141    static inline sqInt longAtPointer(char *ptr) { return *(sqInt *)ptr; }
-> 142    static inline sqInt longAtPointerput(char *ptr, sqInt val) { return *(sqInt *)ptr= val; }
   143    static inline sqLong long64AtPointer(char *ptr) { return *(sqLong *)ptr; }
   144    static inline sqLong long64AtPointerput(char *ptr, sqLong val) { return *(sqLong *)ptr= val; }
   145    static inline float singleFloatAtPointer(char *ptr) { return *(float *)ptr; }
Target 0: (Pharo) stopped.


(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x157800000)
  * frame #0: 0x0000000100015837 Pharo`longAtPointerput(ptr="????", val=5513312480) at sqMemoryAccess.h:142:84 [opt]
    frame #1: 0x00000001000161cf Pharo`marryFrameSP(theFP=<unavailable>, theSP=0x0000000000000000) at gcc3x-cointerp.c:68120:3[opt]
    frame #2: 0x000000010001f5ac Pharo`ceContextinstVar(maybeContext=5510359872, slotIndex=0) at gcc3x-cointerp.c:15221:12 [opt]
    frame #3: 0x00000001480017d6
    frame #4: 0x00000001000022be Pharo`interpret at gcc3x-cointerp.c:2755:3 [opt]
    frame #5: 0x00000001000bc244 Pharo`-[sqSqueakMainApplication runSqueak](self=0x0000000101c76dc0, _cmd=<unavailable>) at sqSqueakMainApplication.m:201:2 [opt]
    frame #6: 0x00007fff3326729b Foundation`__NSFirePerformWithOrder + 360
    frame #7: 0x00007fff30ad3335 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    frame #8: 0x00007fff30ad3267 CoreFoundation`__CFRunLoopDoObservers + 457
    frame #9: 0x00007fff30ad2805 CoreFoundation`__CFRunLoopRun + 874
    frame #10: 0x00007fff30ad1e3e CoreFoundation`CFRunLoopRunSpecific + 462
    frame #11: 0x00007fff2f6feabd HIToolbox`RunCurrentEventLoopInMode + 292
    frame #12: 0x00007fff2f6fe6f4 HIToolbox`ReceiveNextEventCommon + 359
    frame #13: 0x00007fff2f6fe579 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 64
    frame #14: 0x00007fff2dd44039 AppKit`_DPSNextEvent + 883
    frame #15: 0x00007fff2dd42880 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
    frame #16: 0x00007fff2dd3458e AppKit`-[NSApplication run] + 658
    frame #17: 0x00007fff2dd06396 AppKit`NSApplicationMain + 777
    frame #18: 0x00007fff6ab3ecc9 libdyld.dylib`start + 1


(lldb) call printCallStack()
    0x7ffeefbe3920 M INVALID RECEIVER>(nil) 0x148716b40: a(n) bad class
    0x7ffeefbe3968 M [] in INVALID RECEIVER>(nil) Context(Object)>>doesNotUnderstand: #bounds
 0x194648118: a(n) bad class
    0x7ffeefbe39a8 M INVALID RECEIVER>(nil) 0x1489fcec0: a(n) bad class
    0x7ffeefbe39e8 M INVALID RECEIVER>(nil)  0x1489fcec0: a(n) bad class
    0x7ffeefbe3a30 I INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3a80 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe3ab8 M INVALID RECEIVER>(nil)  0x148163cd0: a(n) bad class
    0x7ffeefbe3b08 I INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b40 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbe3b78 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe3bc0 I INVALID RECEIVER>(nil)  0x148716a38: a(n) bad class
    0x7ffeefbe3c10 I INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c40 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3c78 M INVALID RECEIVER>(nil)  0x14d0338e8: a(n) bad class
    0x7ffeefbe3cc0 M INVALID RECEIVER>(nil)  0x14d0337f0: a(n) bad class
    0x7ffeefbe3d08 M INVALID RECEIVER>(nil)  0x14d033738: a(n) bad class
    0x7ffeefbe3d50 M INVALID RECEIVER>(nil)  0x14d033680: a(n) bad class
    0x7ffeefbe3d98 M INVALID RECEIVER>(nil)  0x1946493f0: a(n) bad class
    0x7ffeefbe3de0 M INVALID RECEIVER>(nil)  0x194649338: a(n) bad class
    0x7ffeefbe3e28 M INVALID RECEIVER>(nil)  0x194649280: a(n) bad class
    0x7ffeefbe3e70 M INVALID RECEIVER>(nil)  0x1946491c8: a(n) bad class
    0x7ffeefbe3eb8 M INVALID RECEIVER>(nil)  0x194649110: a(n) bad class
    0x7ffeefbec768 M INVALID RECEIVER>(nil)  0x194649038: a(n) bad class
    0x7ffeefbec7b0 M INVALID RECEIVER>(nil)  0x194648f60: a(n) bad class
    0x7ffeefbec7f8 M INVALID RECEIVER>(nil)  0x194648e88: a(n) bad class
    0x7ffeefbec840 M INVALID RECEIVER>(nil)  0x194648dd0: a(n) bad class
    0x7ffeefbec888 M INVALID RECEIVER>(nil)  0x194648d18: a(n) bad class
    0x7ffeefbec8d0 M INVALID RECEIVER>(nil)  0x194648c60: a(n) bad class
    0x7ffeefbec918 M INVALID RECEIVER>(nil)  0x194648b88: a(n) bad class
    0x7ffeefbec960 M INVALID RECEIVER>(nil)  0x194648ad0: a(n) bad class
    0x7ffeefbec9a8 M INVALID RECEIVER>(nil)  0x194648a18: a(n) bad class
    0x7ffeefbec9f0 M INVALID RECEIVER>(nil)  0x194648960: a(n) bad class
    0x7ffeefbeca38 M INVALID RECEIVER>(nil)  0x1946488a8: a(n) bad class
    0x7ffeefbeca80 M INVALID RECEIVER>(nil)  0x1946487f0: a(n) bad class
    0x7ffeefbecac8 M INVALID RECEIVER>(nil)  0x194648708: a(n) bad class
    0x7ffeefbecb10 M INVALID RECEIVER>(nil)  0x194648620: a(n) bad class
    0x7ffeefbecb58 M INVALID RECEIVER>(nil)  0x194648508: a(n) bad class
    0x7ffeefbecba0 M INVALID RECEIVER>(nil)  0x194648450: a(n) bad class
    0x7ffeefbecbe8 M INVALID RECEIVER>(nil)  0x1481641a8: a(n) bad class
    0x7ffeefbecc30 M INVALID RECEIVER>(nil)  0x1481640f0: a(n) bad class
    0x7ffeefbecc78 M INVALID RECEIVER>(nil)  0x148164038: a(n) bad class
    0x7ffeefbeccc0 M INVALID RECEIVER>(nil)  0x148163f80: a(n) bad class
    0x7ffeefbecd08 M INVALID RECEIVER>(nil)  0x148163ec8: a(n) bad class
    0x7ffeefbecd50 M INVALID RECEIVER>(nil)  0x148163e10: a(n) bad class
    0x7ffeefbecd98 M INVALID RECEIVER>(nil)  0x148163d28: a(n) bad class
    0x7ffeefbecde0 M INVALID RECEIVER>(nil)  0x148163c18: a(n) bad class
    0x7ffeefbece28 M INVALID RECEIVER>(nil)  0x148163b38: a(n) bad class
    0x7ffeefbece70 M INVALID RECEIVER>(nil)  0x148163a80: a(n) bad class
    0x7ffeefbeceb8 M INVALID RECEIVER>(nil)  0x1481639c8: a(n) bad class
    0x7ffeefbe7758 M INVALID RECEIVER>(nil)  0x1481638e0: a(n) bad class
    0x7ffeefbe77a0 M INVALID RECEIVER>(nil)  0x148163808: a(n) bad class
    0x7ffeefbe77e8 M INVALID RECEIVER>(nil)  0x148163750: a(n) bad class
    0x7ffeefbe7830 M INVALID RECEIVER>(nil)  0x148163698: a(n) bad class
    0x7ffeefbe7878 M INVALID RECEIVER>(nil)  0x1481635c0: a(n) bad class
    0x7ffeefbe78c0 M INVALID RECEIVER>(nil)  0x1481634e0: a(n) bad class
    0x7ffeefbe7908 M INVALID RECEIVER>(nil)  0x148163408: a(n) bad class
    0x7ffeefbe7950 M INVALID RECEIVER>(nil)  0x148163350: a(n) bad class
    0x7ffeefbe7998 M INVALID RECEIVER>(nil)  0x148163298: a(n) bad class
    0x7ffeefbe79e0 M INVALID RECEIVER>(nil)  0x148163188: a(n) bad class
    0x7ffeefbe7a28 M INVALID RECEIVER>(nil)  0x148163098: a(n) bad class
    0x7ffeefbe7a70 M INVALID RECEIVER>(nil)  0x148162fa0: a(n) bad class
    0x7ffeefbe7ab8 M INVALID RECEIVER>(nil)  0x148162ec8: a(n) bad class
    0x7ffeefbe7b00 M INVALID RECEIVER>(nil)  0x148162e10: a(n) bad class
    0x7ffeefbe7b48 M INVALID RECEIVER>(nil)  0x148712a08: a(n) bad class
    0x7ffeefbe7b90 M INVALID RECEIVER>(nil)  0x148712950: a(n) bad class
    0x7ffeefbe7bd8 M INVALID RECEIVER>(nil)  0x148712898: a(n) bad class
    0x7ffeefbe7c20 M INVALID RECEIVER>(nil)  0x148713cc0: a(n) bad class
    0x7ffeefbe7c68 M INVALID RECEIVER>(nil)  0x148713018: a(n) bad class
    0x7ffeefbe7cb0 M INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbe7cf8 M INVALID RECEIVER>(nil)  0x148713140: a(n) bad class
    0x7ffeefbe7d40 M INVALID RECEIVER>(nil)  0x148713928: a(n) bad class
    0x7ffeefbe7d88 M INVALID RECEIVER>(nil)  0x1487133c8: a(n) bad class
    0x7ffeefbe7de0 I INVALID RECEIVER>(nil)  0x148713238: a(n) bad class
    0x7ffeefbe7e28 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7e70 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbe7eb8 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeea68 I INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeac8 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeeb00 M INVALID RECEIVER>(nil)  0x148713108: a(n) bad class
    0x7ffeefbeeb50 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeeb98 I INVALID RECEIVER>(nil)  0x148713480: a(n) bad class
    0x7ffeefbeebe0 I INVALID RECEIVER>(nil)  0x1487131f8: a(n) bad class
    0x7ffeefbeec10 M INVALID RECEIVER>(nil) 0x9=1
    0x7ffeefbeec48 M INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeeca0 M [] in INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeecd0 M INVALID RECEIVER>(nil)  0x1487130d0: a(n) bad class
    0x7ffeefbeed10 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeed58 M INVALID RECEIVER>(nil)  0x1487123d8: a(n) bad class
    0x7ffeefbeedb0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeedf0 I INVALID RECEIVER>(nil)  0x1487123c8: a(n) bad class
    0x7ffeefbeee20 M [] in INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbeee78 M INVALID RECEIVER>(nil)  0x148162df0: a(n) bad class
    0x7ffeefbeeec0 M INVALID RECEIVER>(nil)  0x148162ce8: a(n) bad class
    0x7ffeefbe5978 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe59d8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5a18 M INVALID RECEIVER>(nil)  0x148163150: a(n) bad class
    0x7ffeefbe5a68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5aa0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5ad8 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5b08 M INVALID RECEIVER>(nil)  0x1481634c0: a(n) bad class
    0x7ffeefbe5b48 M INVALID RECEIVER>(nil)  0x14c403ca8: a(n) bad class
    0x7ffeefbe5b88 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bc0 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5bf0 M [] in INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c20 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c68 M INVALID RECEIVER>(nil)  0x148162f80: a(n) bad class
    0x7ffeefbe5c98 M INVALID RECEIVER>(nil)  0x194656468: a(n) bad class
    0x7ffeefbe5cd0 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d00 M INVALID RECEIVER>(nil)  0x148163bf0: a(n) bad class
    0x7ffeefbe5d50 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5d88 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5dc0 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbe5e00 M INVALID RECEIVER>(nil)  0x148163de0: a(n) bad class
    0x7ffeefbe5e38 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5e80 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbe5eb8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdf9d8 M INVALID RECEIVER>(nil)  0x194648430: a(n) bad class
    0x7ffeefbdfa10 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfa50 M [] in INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfa90 M INVALID RECEIVER>(nil)  0x1946486d8: a(n) bad class
    0x7ffeefbdfad0 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfb10 M INVALID RECEIVER>(nil)  0x1946485e0: a(n) bad class
    0x7ffeefbdfb48 M INVALID RECEIVER>(nil)  0x1489f7da8: a(n) bad class
    0x7ffeefbdfb80 M INVALID RECEIVER>(nil)  0x148a02930: a(n) bad class
    0x7ffeefbdfbb8 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfbe8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfc20 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc58 M INVALID RECEIVER>(nil)  0x1489fcff0: a(n) bad class
    0x7ffeefbdfc98 M INVALID RECEIVER>(nil)  0x194648c40: a(n) bad class
    0x7ffeefbdfcc8 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd08 M INVALID RECEIVER>(nil)  0x194648f40: a(n) bad class
    0x7ffeefbdfd40 M [] in INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfd70 M INVALID RECEIVER>(nil)  0x14902a730: a(n) bad class
    0x7ffeefbdfdb0 M INVALID RECEIVER>(nil)  0x194648118: a(n) bad class
    0x7ffeefbdfde8 M INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe20 M [] in INVALID RECEIVER>(nil)  0x14c4033c0: a(n) bad class
    0x7ffeefbdfe70 M [] in INVALID RECEIVER>(nil)  0x14d032f98: a(n) bad class
    0x7ffeefbdfeb8 M INVALID RECEIVER>(nil)  0x14d032fe0: a(n) bad class

(callerContextOrNil == (nilObject())) || (isContext(callerContextOrNil)) 72783
       0x14d033738 is not a context

OK, interesting.  Both the assert failure and the badly corrupted stack trace lead me to believe that the issue happens long before the crash and is probably a stack corruption, either by a primitive cutting back the stack incorrectly, or some other hot riot ion (for example are all those nils in INVALID RECEIVER>(nil) real or an artifact of attempting to print an invalid value?).

So the next step is to run the asset vm with leak checking turned on.  Use

myvm —leakcheck 3 to check after every GC

We can add, eg leak checking after an FFI call, in an afternoon

A more realistic setup would be to run GT with an assert headless vm. But until now I did not figure out how to build an assert vm for the gt-headless branch from https://github.com/feenkcom/opensmalltalk-vm

Cheers,
Andrei



[2] checking for cairo's PNG functions feature... 
configure: WARNING: Could not find libpng in the pkg-config search path
checking whether cairo's PNG functions feature could be enabled... no
configure: error: recommended PNG functions feature could not be enabled

On 14 Sep 2020, at 17:32, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,


On Sep 14, 2020, at 7:15 AM, Andrei Chis <[hidden email]> wrote:

Hi Eliot,

On 12 Sep 2020, at 01:42, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Fri, Sep 11, 2020 at 11:48 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks for the answer. That helps to understand what is going on and it can explain why just adding a call to `self pc` makes the crash disappear. 

Just what was maybe not obvious in my previous email is that we get this problem more or less randomly. We have tests for verifying that tools work when various extensions raise exceptions (these tests copy the stack). Sometimes they work correctly and sometimes they crash. These crashes happen in various tests and until now the only common thing we noticed is that the pc of the contexts where the crash happens looks off. Also the contexts in which this happens are at the beginning of the stack so part of a long computation (it gets copied multiple times).

Initially we suspected that there is some memory corruption somewhere due to external calls/memory. Just the fact that calling `self pc` before seems to fix the issue reduces those chances. But who knows.

Well, it does look like a VM bug.  The VM is somehow failing to intercept some access, perhaps in shallow copy.  Weird.  I shall try and reproduce.   Is there anything special about the process you copy using copyTo: ?

I don’t think there is something special about that process. It is the process that we start to run tests [1]. The exception happens in the running process and the crash is when copying the stack of that running process.

Ok, cool.  What I’d like to do is get a copy of your test setup and run it in an assert vm to try and get more information.  AFAICT the vm code is good do the bug is not obvious.  An assert vm may give more information before the crash.  Have you tried running the system on an assert vm yet?

Checked some previous logs and we get these kinds of crashes on the CI server since at least two years. So it does not look like a new bug (but who knows).


(see below)

On Fri, Sep 11, 2020 at 6:36 PM Eliot Miranda <[hidden email]> wrote:
 
Hi Andrei,

On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

The issue is that that value is expected *inside* the VM.  It is the frame pointer for the context.  But above the Vm this value should be hidden. The VM should intercept all accesses to such fields in contexts and automatically map them back to the appropriate values that the image expects to see.  [The same thing is true for CompiledMethods; inside the VM methods may refer to their JITted code, but this is invisible from the image].  Intercepting access to Context state already happens with inst var access in methods, with the shallowCopy primitive, with instVarAt: et al, etc.

So I expect the issue here is that copyTo: invokes some primitive which does not (yet) check for a context receiver and/or argument, and hence accidentally it reveals the hidden state to the image and a crash results.  What I need to know are the definitions for copyTo: and copy, etc all the way down to primitives.

Here is the source code:

Cool, nothing unusual here.  This should all work perfectly.  Tis a VM bug. However...
 
Context >> copyTo: aContext 
"Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil: [
        copy privSender: (self sender copyTo: aContext)].
    ^ copy

Let me suggest

Context >> copyTo: aContext 
   "Copy self and my sender chain down to, but not including, aContext.  End of copied chain will have nil sender."
    | copy |
    self == aContext ifTrue: [^ nil].
    copy := self copy.
    self sender ifNotNil:
        [:mySender| copy privSender: (mySender copyTo: aContext)].
    ^ copy 

Nice!

I also tried the non-recursive implementation of Context>>#copyTo: from Squeak and it also crashes.

Not sure if related but now in the same image as before I got a different crash and printing the stack does not work. But this time the error seems to come from handleStackOverflow

(lldb) call (void *)printCallStack()
invalid frame pointer
invalid frame pointer
invalid frame pointer
error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=EXC_I386_GPFLT).
The process has been returned to the state before expression evaluation.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x121e00000)
  * frame #0: 0x0000000100162258 libGlamorousToolkitVMCore.dylib`marryFrameSP + 584
    frame #1: 0x0000000100172982 libGlamorousToolkitVMCore.dylib`handleStackOverflow + 354
    frame #2: 0x000000010016b025 libGlamorousToolkitVMCore.dylib`ceStackOverflow + 149
    frame #3: 0x00000001100005b3
    frame #4: 0x0000000100174d99 libGlamorousToolkitVMCore.dylib`ptEnterInterpreterFromCallback + 73


Cheers,
Andrei

[1] ./GlamorousToolkit.app/Contents/MacOS/GlamorousToolkit  Pharo.image examples --junit-xml-output 'GToolkit-.*' 'GT4SmaCC-.*' 'DeepTraverser-.*' Brick 'Brick-.*' Bloc 'Bloc-.*' 'Sparta-.*'



Object>>#copy
     ^self shallowCopy postCopy

Object >> shallowCopy
    | class newObject index |
    <primitive: 148>
    class := self class.
    class isVariable
        ifTrue: 
            [index := self basicSize.
            newObject := class basicNew: index.
            [index > 0]
                whileTrue: 
                    [newObject basicAt: index put: (self basicAt: index).
                    index := index - 1]]
        ifFalse: [newObject := class basicNew].
    index := class instSize.
    [index > 0]
        whileTrue: 
            [newObject instVarAt: index put: (self instVarAt: index).
            index := index - 1].
    ^ newObject

The code of the primitiveClone looks the same [1]


Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption?

This looks like an oversight in some primitive.  Here for example is the implementation of the shallowCopy primitive, a.k.a. clone, and you can see where it explcitly intercepts access to a context.

primitiveClone
"Return a shallow copy of the receiver.
 Special-case non-single contexts (because of context-to-stack mapping).
 Can't fail for contexts cuz of image context instantiation code (sigh)."

| rcvr newCopy |
rcvr := self stackTop.
(objectMemory isImmediate: rcvr)
ifTrue:
[newCopy := rcvr]
ifFalse:
[(objectMemory isContextNonImm: rcvr)
ifTrue:
[newCopy := self cloneContext: rcvr]
ifFalse:
[(argumentCount = 0
  or: [(objectMemory isForwarded: rcvr) not])
ifTrue: [newCopy := objectMemory clone: rcvr]
ifFalse: [newCopy := 0]].
newCopy = 0 ifTrue:
[^self primitiveFailFor: PrimErrNoMemory]].
self pop: argumentCount + 1 thenPush: newCopy

But since Squeak doesn't have copyTo: I have no idea what primitive is being used.  I'm guessing 168 primitiveCopyObject, which seems to check for a Context receiver, but not for a CompiledCode receiver.  What does the primitive failure code look like?  Can you post the copyTo: implementations here please?

The code is above. I also see Context>>#copyTo: in Squeak calling also Object>>copy for contexts.

When a crash happens we don't get the exact same error all the time. For example we get most often on mac:

Process 35690 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001100b1004
->  0x1100b1004: inl    $0x4c, %eax
    0x1100b1006: leal   -0x5c(%rip), %eax
    0x1100b100c: pushq  %r8
    0x1100b100e: movabsq $0x1109e78e0, %r9         ; imm = 0x1109E78E0 
Target 0: (GlamorousToolkit) stopped.


Process 29929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001100fe7ed
->  0x1100fe7ed: int3   
    0x1100fe7ee: int3   
    0x1100fe7ef: int3   
    0x1100fe7f0: int3   
Target 0: (GlamorousToolkit) stopped.



Cheers,
Andrei
 

 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





-- 
_,,,^..^,,,_
best, Eliot


-- 
_,,,^..^,,,_
best, Eliot

_,,,^..^,,,_ (phone)




--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Eliot Miranda-2
In reply to this post by Andrei Chis
 
Hi Andrei,

   fixed in commit 561b06530bbaed5f19e9d7f077a7df9eb3a8d236, VMMaker.oscog-eem.2824


On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption? 


 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Andrei Chis
 
Hi Eliot,

Thanks a lot for your help.

I compiled locally a vm from the commit hash 5a0d21d71223b4d1f4d007ebf2441d530c18c3e2 (CogVM source as per VMMaker.oscog-eem.2826) and I got a Context>>copyTo: crash.
I think I compiled the vm correctly. Are them some Pharo VMs built on travis for Mac to try out instead of building locally?

Below are the stack traces and primitive longs for two crashes with the compiled vm.


Process 19347 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x0000000140103f79
->  0x140103f79: int3   
    0x140103f7a: int3   
    0x140103f7b: int3   
    0x140103f7c: int3   
Target 0: (Pharo) stopped.

(lldb) call dumpPrimTraceLog()
basicNew
**StackOverflow**
basicNew
@
@
@
@
basicNew
basicNew
@
@
@
@
@
@
signal
signal
class
basicIdentityHash
basicNew
class
bitShift:
bitAnd:
asFloat
/
asFloat
fractionPart
truncated
fractionPart
truncated
fractionPart
truncated
setAlpha:
truncated
basicIdentityHash
basicNew
basicNew
class
class
basicNew
basicNew
basicIdentityHash
basicNew
basicNew
basicNew
wait
class
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
wait
**StackOverflow**
signal
replaceFrom:to:with:startingAt:
wait
**StackOverflow**
class
signal
basicNew
class
**StackOverflow**
**StackOverflow**
**StackOverflow**
wait
signal
basicIdentityHash
class
basicNew
basicNew
**StackOverflow**
class
class
**StackOverflow**
wait
signal
**StackOverflow**
**StackOverflow**
wait
signal
basicNew
basicNew
basicNew
signal
wait
signal
basicIdentityHash
basicIdentityHash
basicNew
basicNew
class
value:value:
basicNew
replaceFrom:to:with:startingAt:
valueWithArguments:
**StackOverflow**
basicIdentityHash
valueWithArguments:
basicNew
**StackOverflow**
class
findNextHandlerOrSignalingContext
**StackOverflow**
primitive
~=
~=
tempAt:
**StackOverflow**
tempAt:
basicNew
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy



(lldb) call printCallStack()
    0x7ffeefbe6d78 M Context(Object)>copy 0x145bbce28: a(n) Context
    0x7ffeefbe6db0 M Context>copyTo: 0x145bbce28: a(n) Context
    0x7ffeefbe6df8 M Context>copyTo: 0x145bbcd30: a(n) Context
    0x7ffeefbe6e40 M Context>copyTo: 0x145bbcc78: a(n) Context
    0x7ffeefbe6e88 M Context>copyTo: 0x145bbcbc0: a(n) Context
    0x7ffeefbe6ed0 M Context>copyTo: 0x19709adc8: a(n) Context
    0x7ffeefbe6f18 M Context>copyTo: 0x19709ad10: a(n) Context
    0x7ffeefbe37c8 M Context>copyTo: 0x19709ac58: a(n) Context
    0x7ffeefbe3810 M Context>copyTo: 0x19709aba0: a(n) Context
    0x7ffeefbe3858 M Context>copyTo: 0x19709aae8: a(n) Context
    0x7ffeefbe38a0 M Context>copyTo: 0x19709aa10: a(n) Context
    0x7ffeefbe38e8 M Context>copyTo: 0x19709a938: a(n) Context
    0x7ffeefbe3930 M Context>copyTo: 0x19709a860: a(n) Context
    0x7ffeefbe3978 M Context>copyTo: 0x19709a7a8: a(n) Context
    0x7ffeefbe39c0 M Context>copyTo: 0x19709a6f0: a(n) Context
    0x7ffeefbe3a08 M Context>copyTo: 0x19709a638: a(n) Context
    0x7ffeefbe3a50 M Context>copyTo: 0x19709a560: a(n) Context
    0x7ffeefbe3a98 M Context>copyTo: 0x19709a4a8: a(n) Context
    0x7ffeefbe3ae0 M Context>copyTo: 0x19709a3f0: a(n) Context
    0x7ffeefbe3b28 M Context>copyTo: 0x19709a338: a(n) Context
    0x7ffeefbe3b70 M Context>copyTo: 0x19709a280: a(n) Context
    0x7ffeefbe3bb8 M Context>copyTo: 0x19709a1c8: a(n) Context
    0x7ffeefbe3c00 M Context>copyTo: 0x19709a0e0: a(n) Context
    0x7ffeefbe3c48 M Context>copyTo: 0x197099ff8: a(n) Context
    0x7ffeefbe3c90 M Context>copyTo: 0x197099ee0: a(n) Context
    0x7ffeefbe3cd8 M Context>copyTo: 0x197099e28: a(n) Context
    0x7ffeefbe3d20 M Context>copyTo: 0x140169920: a(n) Context
    0x7ffeefbe3d68 M Context>copyTo: 0x140169868: a(n) Context
    0x7ffeefbe3db0 M Context>copyTo: 0x1401697b0: a(n) Context
    0x7ffeefbe3df8 M Context>copyTo: 0x1401696f8: a(n) Context
    0x7ffeefbe3e40 M Context>copyTo: 0x140169640: a(n) Context
    0x7ffeefbe3e88 M Context>copyTo: 0x140169588: a(n) Context
    0x7ffeefbe3ed0 M Context>copyTo: 0x1401694a0: a(n) Context
    0x7ffeefbe3f18 M Context>copyTo: 0x140169390: a(n) Context
    0x7ffeefbf47c8 M Context>copyTo: 0x1401692b0: a(n) Context
    0x7ffeefbf4810 M Context>copyTo: 0x1401691f8: a(n) Context
    0x7ffeefbf4858 M Context>copyTo: 0x140169140: a(n) Context
    0x7ffeefbf48a0 M Context>copyTo: 0x140169058: a(n) Context
    0x7ffeefbf48e8 M Context>copyTo: 0x140168f80: a(n) Context
    0x7ffeefbf4930 M Context>copyTo: 0x140168ec8: a(n) Context
    0x7ffeefbf4978 M Context>copyTo: 0x140168e10: a(n) Context
    0x7ffeefbf49c0 M Context>copyTo: 0x140168d38: a(n) Context
    0x7ffeefbf4a08 M Context>copyTo: 0x140168c58: a(n) Context
    0x7ffeefbf4a50 M Context>copyTo: 0x140168b80: a(n) Context
    0x7ffeefbf4a98 M Context>copyTo: 0x140168ac8: a(n) Context
    0x7ffeefbf4ae0 M Context>copyTo: 0x140168a10: a(n) Context
    0x7ffeefbf4b28 M Context>copyTo: 0x140168900: a(n) Context
    0x7ffeefbf4b70 M Context>copyTo: 0x140168810: a(n) Context
    0x7ffeefbf4bb8 M Context>copyTo: 0x140168718: a(n) Context
    0x7ffeefbf4c00 M Context>copyTo: 0x140168640: a(n) Context
    0x7ffeefbf4c48 M Context>copyTo: 0x14039c748: a(n) Context
    0x7ffeefbf4c90 M Context>copyTo: 0x14039c5d8: a(n) Context
    0x7ffeefbf4cd8 M Context>copyTo: 0x1401684b8: a(n) Context
    0x7ffeefbf4d20 M Context>copyTo: 0x1401683d8: a(n) Context
    0x7ffeefbf4d68 M Context>copyTo: 0x14039c2f8: a(n) Context
    0x7ffeefbf4db0 M Context>copyTo: 0x140167960: a(n) Context
    0x7ffeefbf4df8 M Context>copyTo: 0x1401678a8: a(n) Context
    0x7ffeefbf4e40 M Context>copyTo: 0x140167788: a(n) Context
    0x7ffeefbf4e88 M Context>copyTo: 0x14039bf60: a(n) Context
    0x7ffeefbf4ed0 M Context>copyTo: 0x14039bd38: a(n) Context
    0x7ffeefbf4f18 M Context>copyTo: 0x14039ba58: a(n) Context
    0x7ffeefbf07c8 M Context>copyTo: 0x14039b8e8: a(n) Context
    0x7ffeefbf0810 M Context>copyTo: 0x140167578: a(n) Context
    0x7ffeefbf0858 M Context>copyTo: 0x140167478: a(n) Context
    0x7ffeefbf08a0 M Context>copyTo: 0x14039b608: a(n) Context
    0x7ffeefbf08e8 M Context>copyTo: 0x14039b498: a(n) Context
    0x7ffeefbf0930 M Context>copyTo: 0x14039b328: a(n) Context
    0x7ffeefbf0978 M Context>copyTo: 0x140167310: a(n) Context
    0x7ffeefbf09c0 M Context>copyTo: 0x1401671e8: a(n) Context
    0x7ffeefbf0a08 M Context>copyTo: 0x140167108: a(n) Context
    0x7ffeefbf0a50 M Context>copyTo: 0x14039af90: a(n) Context
    0x7ffeefbf0a98 M Context>copyTo: 0x14039ae20: a(n) Context
    0x7ffeefbf0ae0 M Context>copyTo: 0x140166fe0: a(n) Context
    0x7ffeefbf0b28 M Context>copyTo: 0x140166ec0: a(n) Context
    0x7ffeefbf0b70 M Context>copyTo: 0x14039ab40: a(n) Context
    0x7ffeefbf0bb8 M Context>copyTo: 0x14039a9d0: a(n) Context
    0x7ffeefbf0c00 M Context>copyTo: 0x14039a860: a(n) Context
    0x7ffeefbf0c48 M Context>copyTo: 0x14039a6f0: a(n) Context
    0x7ffeefbf0c90 M Context>copyTo: 0x140166ca0: a(n) Context
    0x7ffeefbf0cd8 M Context>copyTo: 0x140166bb8: a(n) Context
    0x7ffeefbf0d20 M Context>copyTo: 0x140165f60: a(n) Context
    0x7ffeefbf0d68 M Context>copyTo: 0x140165ea8: a(n) Context
    0x7ffeefbf0db0 M Context>copyTo: 0x14039a2a0: a(n) Context
    0x7ffeefbf0df8 M Context>copyTo: 0x140165d48: a(n) Context
    0x7ffeefbf0e40 M Context>copyTo: 0x140165c20: a(n) Context
    0x7ffeefbf0e88 M Context>copyTo: 0x140165b40: a(n) Context
    0x7ffeefbf0ed0 M Context>copyTo: 0x140399e50: a(n) Context
    0x7ffeefbf0f18 M Context>copyTo: 0x140399b70: a(n) Context
    0x7ffeefbee7c8 M Context>copyTo: 0x140165a18: a(n) Context
    0x7ffeefbee810 M Context>copyTo: 0x1401658f8: a(n) Context
    0x7ffeefbee858 M Context>copyTo: 0x140399890: a(n) Context
    0x7ffeefbee8a0 M Context>copyTo: 0x140399720: a(n) Context
    0x7ffeefbee8e8 M Context>copyTo: 0x1403995b0: a(n) Context
    0x7ffeefbee930 M Context>copyTo: 0x140399440: a(n) Context
    0x7ffeefbee978 M Context>copyTo: 0x1403992d0: a(n) Context
    0x7ffeefbee9c0 M Context>copyTo: 0x140399160: a(n) Context
    0x7ffeefbeea08 M Context>copyTo: 0x140398ff0: a(n) Context
    0x7ffeefbeea50 M Context>copyTo: 0x140398e80: a(n) Context
    0x7ffeefbeea98 M Context>copyTo: 0x140398d10: a(n) Context
    0x7ffeefbeeae0 M Context>copyTo: 0x140165748: a(n) Context
    0x7ffeefbeeb28 M Context>copyTo: 0x140165648: a(n) Context
    0x7ffeefbeeb70 M Context>copyTo: 0x140398a30: a(n) Context
    0x7ffeefbeebb8 M Context>copyTo: 0x1403988c0: a(n) Context
    0x7ffeefbeec00 M Context>copyTo: 0x140165530: a(n) Context
    0x7ffeefbeec48 M Context>copyTo: 0x140165458: a(n) Context
    0x7ffeefbeec90 M Context>copyTo: 0x1403985e0: a(n) Context
    0x7ffeefbeecd8 M Context>copyTo: 0x140398470: a(n) Context
    0x7ffeefbeed20 M Context>copyTo: 0x140165088: a(n) Context
    0x7ffeefbeed68 M Context>copyTo: 0x140166ae0: a(n) Context
    0x7ffeefbeedb0 M Context>copyTo: 0x140398190: a(n) Context
    0x7ffeefbeedf8 M Context>copyTo: 0x140398020: a(n) Context
    0x7ffeefbeee40 M Context>copyTo: 0x140397eb0: a(n) Context
    0x7ffeefbeee88 M Context>copyTo: 0x1401669d8: a(n) Context
    0x7ffeefbeeed0 M Context>copyTo: 0x140397c88: a(n) Context
    0x7ffeefbeef18 M Context>copyTo: 0x140394d08: a(n) Context
    0x7ffeefbea798 M Context>copyTo: 0x140394e20: a(n) Context
    0x7ffeefbea7e0 M Context>copyTo: 0x140397838: a(n) Context
    0x7ffeefbea828 M Context>copyTo: 0x1403976c8: a(n) Context
    0x7ffeefbea870 M Context>copyTo: 0x140397558: a(n) Context
    0x7ffeefbea8b8 M Context>copyTo: 0x140395000: a(n) Context
    0x7ffeefbea900 M Context>copyTo: 0x140397330: a(n) Context
    0x7ffeefbea948 M Context>copyTo: 0x1403971c0: a(n) Context
    0x7ffeefbea990 M Context>copyTo: 0x140397050: a(n) Context
    0x7ffeefbea9d8 M Context>copyTo: 0x140396ee0: a(n) Context
    0x7ffeefbeaa20 M Context>copyTo: 0x140396d70: a(n) Context
    0x7ffeefbeaa68 M Context>copyTo: 0x1403958f8: a(n) Context
    0x7ffeefbeaab0 M Context>copyTo: 0x140396b48: a(n) Context
    0x7ffeefbeaaf8 M Context>copyTo: 0x140396298: a(n) Context
    0x7ffeefbeab40 M Context>copyTo: 0x140396920: a(n) Context
    0x7ffeefbeab88 M Context>copyTo: 0x1403967b0: a(n) Context
    0x7ffeefbeabd0 M Context>copyTo: 0x1403961e0: a(n) Context
    0x7ffeefbeac18 M Context>copyTo: 0x140396128: a(n) Context
    0x7ffeefbeac60 M Context>copyTo: 0x140396070: a(n) Context
    0x7ffeefbeacb8 I Context>copyTo: 0x140395e28: a(n) Context
    0x7ffeefbead00 I ZeroDivide(Exception)>freezeUpTo: 0x140395de8: a(n) ZeroDivide
    0x7ffeefbead48 I ZeroDivide(Exception)>freeze 0x140395de8: a(n) ZeroDivide
    0x7ffeefbead88 I BrDebugglableElementStencil>freeze: 0x140396408: a(n) BrDebugglableElementStencil
    0x7ffeefbeadd8 I ZeroDivide(Exception)>asDebuggableElement 0x140395de8: a(n) ZeroDivide
    0x7ffeefbeae18 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbeae50 M BlockClosure>cull: 0x1403959f0: a(n) BlockClosure
    0x7ffeefbeaea0 I Context>evaluateSignal: 0x140396298: a(n) Context
    0x7ffeefbeaed8 M Context>handleSignal: 0x140396298: a(n) Context
    0x7ffeefbeaf20 I ZeroDivide(Exception)>signal 0x140395de8: a(n) ZeroDivide
    0x7ffeefbe7840 I ZeroDivide(Exception)>signal: 0x140395de8: a(n) ZeroDivide
    0x7ffeefbe7888 I ZeroDivide class(Exception class)>signal: 0x1409f9490: a(n) ZeroDivide class
    0x7ffeefbe78c0 M [] in GtPhlowColumnedListViewExamples>gtErrorInColumnDoDataBinderFor: 0x140168630: a(n) GtPhlowColumnedListViewExamples
    0x7ffeefbe7908 M BlockClosure>glamourValueWithArgs: 0x140174180: a(n) BlockClosure
    0x7ffeefbe7960 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7990 M BlockClosure>on:do: 0x1403959b0: a(n) BlockClosure
    0x7ffeefbe79d0 M GtPhlowColumn>performBlock:onException: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7a18 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7a60 M BlockClosure>glamourValueWithArgs: 0x1401701c0: a(n) BlockClosure
    0x7ffeefbe7a98 M BrStencilValuableExecutor>execute 0x140174230: a(n) BrStencilValuableExecutor
    0x7ffeefbe7ad8 M BrColumnCellDataBinder(BrStencilBuilder)>build 0x140170448: a(n) BrColumnCellDataBinder
    0x7ffeefbe7b38 I [] in BrColumnedListDataSource>onBindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7b98 I OrderedCollection(SequenceableCollection)>with:do: 0x140169ff8: a(n) OrderedCollection
    0x7ffeefbe7bf8 I BrColumnedListDataSource>onBindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7c48 I BrColumnedListDataSource(BlInfiniteDataSource)>onBindHolder:at:payloads: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7ca0 M [] in BrColumnedListDataSource(BlInfiniteDataSource)>bindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7ce0 M BlockClosure>ensure: 0x140394df0: a(n) BlockClosure
    0x7ffeefbe7d18 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe7d58 M BrColumnedListDataSource(BlInfiniteDataSource)>bindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7d98 M BlInfiniteRecyclerController>bindHolder:at: 0x1401662c0: a(n) BlInfiniteRecyclerController
    0x7ffeefbe7e10 M BlInfiniteRecycler>elementFor:dryRun: 0x1401652b8: a(n) BlInfiniteRecycler
    0x7ffeefbe7e50 M BlInfiniteRecycler>elementFor: 0x1401652b8: a(n) BlInfiniteRecycler
    0x7ffeefbe7e98 M [] in BlInfiniteLinearLayoutState>nextElement:in: 0x140165020: a(n) BlInfiniteLinearLayoutState
    0x7ffeefbe7ed8 M BlockClosure>ensure: 0x140166a90: a(n) BlockClosure
    0x7ffeefbe7f10 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe27d0 M BlInfiniteLinearLayoutState>nextElement:in: 0x140165020: a(n) BlInfiniteLinearLayoutState
    0x7ffeefbe2818 M [] in BlInfiniteLinearLayout>layoutChunkAdd 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2858 M BlockClosure>ensure: 0x140165410: a(n) BlockClosure
    0x7ffeefbe2890 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe28d8 M BlInfiniteLinearLayout>layoutChunkAdd 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2910 M [] in BlInfiniteLinearLayout>layoutChunk 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2950 M BlockClosure>ensure: 0x140165600: a(n) BlockClosure
    0x7ffeefbe2988 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe29e0 M BlInfiniteLinearLayout>layoutChunk 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2a48 I BlInfiniteLinearLayout>fillLayout: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2ab8 I BlInfiniteLinearLayout>layoutChildrenFillFromStart: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2b00 I BlInfiniteLinearLayout>layoutChildrenFill: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2b58 I BlInfiniteLinearLayout>layoutChildrenIn:state: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2ba8 I BrInfiniteListElement(BlInfiniteElement)>dispatchLayoutSecondStep 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2be8 I BrInfiniteListElement(BlInfiniteElement)>dispatchLayout 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c18 M BrInfiniteListElement(BlInfiniteElement)>onLayout: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c58 M [] in BrInfiniteListElement(BlElement)>applyLayoutSafelyIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c98 M BlockClosure>ensure: 0x1401658b0: a(n) BlockClosure
    0x7ffeefbe2cd0 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe2d30 M BrInfiniteListElement(BlElement)>applyLayoutSafelyIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2d70 M [] in BrInfiniteListElement(BlElement)>applyLayoutIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2da0 M BlockClosure>on:do: 0x140165ad0: a(n) BlockClosure
    0x7ffeefbe2de0 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x140165b28: a(n) BlCompositeErrorHandler
    0x7ffeefbe2e28 M BrInfiniteListElement(BlElement)>applyLayoutIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2e80 M [] in BlLinearLayoutVerticalOrientation>layout:in: 0x140165d38: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe2ed0 M [] in BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe2f18 M Array(SequenceableCollection)>do: 0x140165e98: a(n) Array
    0x7ffeefbe9868 M BlChildrenAccountedByLayout(BlChildrenSubset)>do: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe98a8 M BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9918 M BlLinearLayoutVerticalOrientation>layout:in: 0x140165d38: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9958 M BlLinearLayout>layout:in: 0x140166e18: a(n) BlLinearLayout
    0x7ffeefbe9998 M BrColumnedList(BlElement)>onLayout: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe99d8 M [] in BrColumnedList(BlElement)>applyLayoutSafelyIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9a18 M BlockClosure>ensure: 0x140166e78: a(n) BlockClosure
    0x7ffeefbe9a50 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe9ab0 M BrColumnedList(BlElement)>applyLayoutSafelyIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9af0 M [] in BrColumnedList(BlElement)>applyLayoutIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9b20 M BlockClosure>on:do: 0x140167098: a(n) BlockClosure
    0x7ffeefbe9b60 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x1401670f0: a(n) BlCompositeErrorHandler
    0x7ffeefbe9ba8 M BrColumnedList(BlElement)>applyLayoutIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9c00 M [] in BlLinearLayoutVerticalOrientation>layout:in: 0x140167300: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9c50 M [] in BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9c98 M Array(SequenceableCollection)>do: 0x140167460: a(n) Array
    0x7ffeefbe9cd0 M BlChildrenAccountedByLayout(BlChildrenSubset)>do: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9d10 M BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9d80 M BlLinearLayoutVerticalOrientation>layout:in: 0x140167300: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9dc0 M BlLinearLayout>layout:in: 0x1401676e0: a(n) BlLinearLayout
    0x7ffeefbe9e00 M BlElement>onLayout: 0x140167648: a(n) BlElement
    0x7ffeefbe9e40 M [] in BlElement>applyLayoutSafelyIn: 0x140167648: a(n) BlElement
    0x7ffeefbe9e80 M BlockClosure>ensure: 0x140167740: a(n) BlockClosure
    0x7ffeefbe9eb8 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe9f18 M BlElement>applyLayoutSafelyIn: 0x140167648: a(n) BlElement
    0x7ffeefbed7e0 M [] in BlElement>applyLayoutIn: 0x140167648: a(n) BlElement
    0x7ffeefbed810 M BlockClosure>on:do: 0x140168368: a(n) BlockClosure
    0x7ffeefbed850 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x1401683c0: a(n) BlCompositeErrorHandler
    0x7ffeefbed898 M BlElement>applyLayoutIn: 0x140167648: a(n) BlElement
    0x7ffeefbed908 I BlElement>computeLayout 0x140167648: a(n) BlElement
    0x7ffeefbed948 I BlElement>forceLayout 0x140167648: a(n) BlElement
    0x7ffeefbed9a8 I GtPhlowColumnedListViewExamples>errorInColumnDoDataBinder 0x140168630: a(n) GtPhlowColumnedListViewExamples
    0x7ffeefbed9d8 M GtExampleDebugger(GtExampleEvaluator)>primitiveProcessExample:withEvaluationContext: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbeda38 M [] in GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbeda78 M BlockClosure>ensure: 0x1401688c8: a(n) BlockClosure
    0x7ffeefbedac8 M GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb00 M GtExampleDebugger(GtExampleProcessor)>process: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb38 M [] in GtExampleDebugger(GtExampleProcessor)>value 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb68 M BlockClosure>on:do: 0x140168c38: a(n) BlockClosure
    0x7ffeefbedba8 M GtCurrentExampleContext class>use:during: 0x1446188f0: a(n) GtCurrentExampleContext class
    0x7ffeefbedbe8 M GtExampleDebugger(GtExampleProcessor)>withContextDo: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc20 M GtExampleDebugger(GtExampleProcessor)>value 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc50 M [] in GtExampleDebugger(GtExampleEvaluator)>result 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc80 M GtExampleDebugger>do:on:do: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedcc8 M GtExampleDebugger(GtExampleEvaluator)>result 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedcf8 M GtExample>debug 0x19709dfc8: a(n) GtExample
    0x7ffeefbedd30 M [] in GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedd60 M BlockClosure>on:do: 0x140169368: a(n) BlockClosure
    0x7ffeefbeddb0 M [] in GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedde8 M Time class>microsecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbede20 M Time class>millisecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbede60 M BlockClosure>timeToRun 0x140169558: a(n) BlockClosure
    0x7ffeefbede98 M GtExamplesHDReport>beginExample:runBlock: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedee0 M GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedf18 M [] in GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdea38 M OrderedCollection>do: 0x197099e08: a(n) OrderedCollection
    0x7ffeefbdea70 M [] in GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdeab0 M [] in CurrentExecutionEnvironment class>activate:for: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdeaf0 M BlockClosure>ensure: 0x19709a0b0: a(n) BlockClosure
    0x7ffeefbdeb30 M CurrentExecutionEnvironment class>activate:for: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdeb70 M TestExecutionEnvironment(ExecutionEnvironment)>beActiveDuring: 0x197099fb8: a(n) TestExecutionEnvironment
    0x7ffeefbdeba8 M DefaultExecutionEnvironment>runTestsBy: 0x1409f7da8: a(n) DefaultExecutionEnvironment
    0x7ffeefbdebe0 M CurrentExecutionEnvironment class>runTestsBy: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdec18 M GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdec48 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdec80 M Time class>microsecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbdecb8 M Time class>millisecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbdecf8 M BlockClosure>timeToRun 0x19709a618: a(n) BlockClosure
    0x7ffeefbded28 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbded68 M BlockClosure>ensure: 0x19709a918: a(n) BlockClosure
    0x7ffeefbdeda0 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdedd0 M Author>ifUnknownAuthorUse:during: 0x14102a740: a(n) Author
    0x7ffeefbdee10 M GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdee48 M GtExamplesHDReport class>runPackage: 0x144618008: a(n) GtExamplesHDReport class
    0x7ffeefbdee80 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x144618008: a(n) GtExamplesHDReport class
    0x7ffeefbdeed0 M [] in Set>collect: 0x145bbc4d8: a(n) Set
    0x7ffeefbdef18 M Array(SequenceableCollection)>do: 0x145bbc520: a(n) Array
       0x145bbcc78 s Set>collect:
       0x145bbcd30 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x145bbce28 s [] in GtExamplesCommandLineHandler>runPackages
       0x145bbcf08 s BlockClosure>ensure:
       0x14e950710 s UIManager class>nonInteractiveDuring:
       0x14e9507c8 s GtExamplesCommandLineHandler>runPackages
       0x14e950880 s GtExamplesCommandLineHandler>activate
       0x14e950938 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x14e9509f0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x14e950aa8 s BlockClosure>on:do:
       0x14e950b60 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x14e950c38 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x14e950cf0 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x14e950db8 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x14e950e90 s BlockClosure>on:do:
       0x14e950f68 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x14e951040 s [] in BlockClosure>newProcess



Process 18798 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001481107d1
->  0x1481107d1: int3   
    0x1481107d2: int3   
    0x1481107d3: int3   
    0x1481107d4: int3   
Target 0: (Pharo) stopped.

(lldb) call printAllStacks()
Process        0x15511b938 priority 40
    0x7ffeefbf0a18 M Context(Object)>copy 0x15691cf68: a(n) Context
    0x7ffeefbf0a50 M Context>copyTo: 0x15691cf68: a(n) Context
    0x7ffeefbf0a98 M Context>copyTo: 0x15691ce78: a(n) Context
    0x7ffeefbf0ae0 M Context>copyTo: 0x15691cdc0: a(n) Context
    0x7ffeefbf0b28 M Context>copyTo: 0x15691cd08: a(n) Context
    0x7ffeefbf0b70 M Context>copyTo: 0x15691cc50: a(n) Context
    0x7ffeefbf0bb8 M Context>copyTo: 0x15691cb70: a(n) Context
    0x7ffeefbf0c00 M Context>copyTo: 0x15691ca90: a(n) Context
    0x7ffeefbf0c48 M Context>copyTo: 0x15691c998: a(n) Context
    0x7ffeefbf0c90 M Context>copyTo: 0x15691c8e0: a(n) Context
    0x7ffeefbf0cd8 M Context>copyTo: 0x15691c828: a(n) Context
    0x7ffeefbf0d20 M Context>copyTo: 0x14874a2c8: a(n) Context
    0x7ffeefbf0d68 M Context>copyTo: 0x14874a158: a(n) Context
    0x7ffeefbf0db0 M Context>copyTo: 0x148749fe8: a(n) Context
    0x7ffeefbf0df8 M Context>copyTo: 0x14816e2e8: a(n) Context
    0x7ffeefbf0e40 M Context>copyTo: 0x148749dc0: a(n) Context
    0x7ffeefbf0e88 M Context>copyTo: 0x14816e210: a(n) Context
    0x7ffeefbf0ed0 M Context>copyTo: 0x148749ae0: a(n) Context
    0x7ffeefbf0f18 M Context>copyTo: 0x14816e118: a(n) Context
    0x7ffeefbdc7c8 M Context>copyTo: 0x148749748: a(n) Context
    0x7ffeefbdc810 M Context>copyTo: 0x1487495d8: a(n) Context
    0x7ffeefbdc858 M Context>copyTo: 0x148749468: a(n) Context
    0x7ffeefbdc8a0 M Context>copyTo: 0x1487492f8: a(n) Context
    0x7ffeefbdc8e8 M Context>copyTo: 0x14816e040: a(n) Context
    0x7ffeefbdc930 M Context>copyTo: 0x1487490d0: a(n) Context
    0x7ffeefbdc978 M Context>copyTo: 0x148748f60: a(n) Context
    0x7ffeefbdc9c0 M Context>copyTo: 0x148748df0: a(n) Context
    0x7ffeefbdca08 M Context>copyTo: 0x14816df38: a(n) Context
    0x7ffeefbdca50 M Context>copyTo: 0x148748bc8: a(n) Context
    0x7ffeefbdca98 M Context>copyTo: 0x14816de20: a(n) Context
    0x7ffeefbdcae0 M Context>copyTo: 0x14816eeb8: a(n) Context
    0x7ffeefbdcb28 M Context>copyTo: 0x1487488e8: a(n) Context
    0x7ffeefbdcb70 M Context>copyTo: 0x148748778: a(n) Context
    0x7ffeefbdcbb8 M Context>copyTo: 0x14866e858: a(n) Context
    0x7ffeefbdcc00 M Context>copyTo: 0x148748550: a(n) Context
    0x7ffeefbdcc48 M Context>copyTo: 0x1487483e0: a(n) Context
    0x7ffeefbdcc90 M Context>copyTo: 0x148748270: a(n) Context
    0x7ffeefbdccd8 M Context>copyTo: 0x148748100: a(n) Context
    0x7ffeefbdcd20 M Context>copyTo: 0x148676078: a(n) Context
    0x7ffeefbdcd68 M Context>copyTo: 0x148747ed8: a(n) Context
    0x7ffeefbdcdb0 M Context>copyTo: 0x148747d68: a(n) Context
    0x7ffeefbdcdf8 M Context>copyTo: 0x148747bf8: a(n) Context
    0x7ffeefbdce40 M Context>copyTo: 0x148676280: a(n) Context
    0x7ffeefbdce88 M Context>copyTo: 0x1487479d0: a(n) Context
    0x7ffeefbdced0 M Context>copyTo: 0x1487477a8: a(n) Context
    0x7ffeefbdcf18 M Context>copyTo: 0x148676400: a(n) Context
    0x7ffeefbda7c8 M Context>copyTo: 0x148747410: a(n) Context
    0x7ffeefbda810 M Context>copyTo: 0x1486764d8: a(n) Context
    0x7ffeefbda858 M Context>copyTo: 0x1487471e8: a(n) Context
    0x7ffeefbda8a0 M Context>copyTo: 0x148747078: a(n) Context
    0x7ffeefbda8e8 M Context>copyTo: 0x148746f08: a(n) Context
    0x7ffeefbda930 M Context>copyTo: 0x14873f918: a(n) Context
    0x7ffeefbda978 M Context>copyTo: 0x148746ce0: a(n) Context
    0x7ffeefbda9c0 M Context>copyTo: 0x148746b70: a(n) Context
    0x7ffeefbdaa08 M Context>copyTo: 0x148746a00: a(n) Context
    0x7ffeefbdaa50 M Context>copyTo: 0x148746890: a(n) Context
    0x7ffeefbdaa98 M Context>copyTo: 0x148746720: a(n) Context
    0x7ffeefbdaae0 M Context>copyTo: 0x1487415b8: a(n) Context
    0x7ffeefbdab28 M Context>copyTo: 0x1487412a8: a(n) Context
    0x7ffeefbdab70 M Context>copyTo: 0x148746440: a(n) Context
    0x7ffeefbdabb8 M Context>copyTo: 0x148745088: a(n) Context
    0x7ffeefbdac00 M Context>copyTo: 0x148746218: a(n) Context
    0x7ffeefbdac48 M Context>copyTo: 0x1487414e0: a(n) Context
    0x7ffeefbdac90 M Context>copyTo: 0x148745ff0: a(n) Context
    0x7ffeefbdacd8 M Context>copyTo: 0x148741670: a(n) Context
    0x7ffeefbdad20 M Context>copyTo: 0x148744fd0: a(n) Context
    0x7ffeefbdad68 M Context>copyTo: 0x148745ba0: a(n) Context
    0x7ffeefbdadb0 M Context>copyTo: 0x148745a30: a(n) Context
    0x7ffeefbdadf8 M Context>copyTo: 0x148744bd8: a(n) Context
    0x7ffeefbdae40 M Context>copyTo: 0x148745808: a(n) Context
    0x7ffeefbdae88 M Context>copyTo: 0x148745698: a(n) Context
    0x7ffeefbdaed0 M Context>copyTo: 0x148745528: a(n) Context
    0x7ffeefbdaf18 M Context>copyTo: 0x148744f18: a(n) Context
    0x7ffeefbe0988 I Context>copyTo: 0x148744e60: a(n) Context
    0x7ffeefbe09d0 I MessageNotUnderstood(Exception)>freezeUpTo: 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0a18 I MessageNotUnderstood(Exception)>freeze 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0a48 M [] in GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0a80 M BlockClosure>cull: 0x1487414c0: a(n) BlockClosure
    0x7ffeefbe0ac0 M Context>evaluateSignal: 0x148745088: a(n) Context
    0x7ffeefbe0af8 M Context>handleSignal: 0x148745088: a(n) Context
    0x7ffeefbe0b30 M Context>handleSignal: 0x148744fd0: a(n) Context
    0x7ffeefbe0b68 M MessageNotUnderstood(Exception)>signal 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0bb8 I GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x148744bc8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefbe0bf0 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0c50 M [] in GtExampleEvaluator>basicProcess: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0c90 M BlockClosure>ensure: 0x148744c90: a(n) BlockClosure
    0x7ffeefbe0ce0 M GtExampleEvaluator>basicProcess: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d18 M GtExampleEvaluator(GtExampleProcessor)>process: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d50 M [] in GtExampleEvaluator(GtExampleProcessor)>value 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d80 M BlockClosure>on:do: 0x148741598: a(n) BlockClosure
    0x7ffeefbe0dc0 M GtCurrentExampleContext class>use:during: 0x14c618920: a(n) GtCurrentExampleContext class
    0x7ffeefbe0e00 M GtExampleEvaluator(GtExampleProcessor)>withContextDo: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e38 M GtExampleEvaluator(GtExampleProcessor)>value 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e68 M [] in GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e98 M BlockClosure>on:do: 0x148741360: a(n) BlockClosure
    0x7ffeefbe0ed8 M GtExampleEvaluator>do:on:do: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0f20 M GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbde8b8 M GtExample>run 0x148740050: a(n) GtExample
    0x7ffeefbde8e8 M GtExample>result 0x148740050: a(n) GtExample
    0x7ffeefbde930 I GtExamplesExamplesWithInheritance>resultOfInvalidExampleWithInvalidSuperclassProvider 0x14873f908: a(n) GtExamplesExamplesWithInheritance
    0x7ffeefbde960 M GtExampleDebugger(GtExampleEvaluator)>primitiveProcessExample:withEvaluationContext: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbde9c0 M [] in GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdea00 M BlockClosure>ensure: 0x14873f9d0: a(n) BlockClosure
    0x7ffeefbdea50 M GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdea88 M GtExampleDebugger(GtExampleProcessor)>process: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeac0 M [] in GtExampleDebugger(GtExampleProcessor)>value 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeaf0 M BlockClosure>on:do: 0x1486764b8: a(n) BlockClosure
    0x7ffeefbdeb30 M GtCurrentExampleContext class>use:during: 0x14c618920: a(n) GtCurrentExampleContext class
    0x7ffeefbdeb70 M GtExampleDebugger(GtExampleProcessor)>withContextDo: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeba8 M GtExampleDebugger(GtExampleProcessor)>value 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdebd8 M [] in GtExampleDebugger(GtExampleEvaluator)>result 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec08 M GtExampleDebugger>do:on:do: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec50 M GtExampleDebugger(GtExampleEvaluator)>result 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec80 M GtExample>debug 0x148188088: a(n) GtExample
    0x7ffeefbdecb8 M [] in GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdece8 M BlockClosure>on:do: 0x148676130: a(n) BlockClosure
    0x7ffeefbded38 M [] in GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbded70 M Time class>microsecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbdeda8 M Time class>millisecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbdede8 M BlockClosure>timeToRun 0x14866e910: a(n) BlockClosure
    0x7ffeefbdee20 M GtExamplesHDReport>beginExample:runBlock: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdee68 M GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdeea0 M [] in GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdeee8 M OrderedCollection>do: 0x14816ee98: a(n) OrderedCollection
    0x7ffeefbdef20 M [] in GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8ab0 M [] in CurrentExecutionEnvironment class>activate:for: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8af0 M BlockClosure>ensure: 0x14816ded8: a(n) BlockClosure
    0x7ffeefbd8b30 M CurrentExecutionEnvironment class>activate:for: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8b70 M TestExecutionEnvironment(ExecutionEnvironment)>beActiveDuring: 0x14816ddc0: a(n) TestExecutionEnvironment
    0x7ffeefbd8ba8 M DefaultExecutionEnvironment>runTestsBy: 0x1489f7da8: a(n) DefaultExecutionEnvironment
    0x7ffeefbd8be0 M CurrentExecutionEnvironment class>runTestsBy: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8c18 M GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8c48 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8c80 M Time class>microsecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbd8cb8 M Time class>millisecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbd8cf8 M BlockClosure>timeToRun 0x14816e0f8: a(n) BlockClosure
    0x7ffeefbd8d28 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8d68 M BlockClosure>ensure: 0x14816e1d0: a(n) BlockClosure
    0x7ffeefbd8da0 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8dd0 M Author>ifUnknownAuthorUse:during: 0x14902a740: a(n) Author
    0x7ffeefbd8e10 M GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8e48 M GtExamplesHDReport class>runPackage: 0x14c618038: a(n) GtExamplesHDReport class
    0x7ffeefbd8e80 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x14c618038: a(n) GtExamplesHDReport class
    0x7ffeefbd8ed0 M [] in Set>collect: 0x15691c088: a(n) Set
    0x7ffeefbd8f18 M Array(SequenceableCollection)>do: 0x15691c188: a(n) Array
       0x15691c8e0 s Set>collect:
       0x15691c998 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x15691ca90 s [] in GtExamplesCommandLineHandler>runPackages
       0x15691cb70 s BlockClosure>ensure:
       0x15691cc50 s UIManager class>nonInteractiveDuring:
       0x15691cd08 s GtExamplesCommandLineHandler>runPackages
       0x15691cdc0 s GtExamplesCommandLineHandler>activate
       0x15691ce78 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x15691cf68 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x15691d048 s BlockClosure>on:do:
       0x15691d128 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x15691d200 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x15691d2b8 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x15691d380 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x15691d458 s BlockClosure>on:do:
       0x15691d530 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x15691d608 s [] in BlockClosure>newProcess


(lldb) call dumpPrimTraceLog()
stringHash:initialHash:
compare:with:collated:
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
primitiveChangeClassTo:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
primitiveChangeClassTo:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
stringHash:initialHash:
compare:with:collated:
primitiveChangeClassTo:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
basicNew:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
size
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
**StackOverflow**
perform:withArguments:
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
basicNew
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
**StackOverflow**
**StackOverflow**
class
perform:withArguments:
value:
first
basicNew
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
perform:withArguments:
basicNew
findNextHandlerOrSignalingContext
tempAt:
class
findNextHandlerOrSignalingContext
tempAt:
tempAt:
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**CompactCode**



On 26 Sep 2020, at 23:24, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

   fixed in commit 561b06530bbaed5f19e9d7f077a7df9eb3a8d236, VMMaker.oscog-eem.2824


On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption? 


 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





--
_,,,^..^,,,_
best, Eliot

Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Eliot Miranda-2
 
Hi Andrei,

On Sun, Sep 27, 2020 at 10:29 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks a lot for your help.

I compiled locally a vm from the commit hash 5a0d21d71223b4d1f4d007ebf2441d530c18c3e2 (CogVM source as per VMMaker.oscog-eem.2826) and I got a Context>>copyTo: crash.
I think I compiled the vm correctly. Are them some Pharo VMs built on travis for Mac to try out instead of building locally?

There may be.  The PharoVM team are not maintaining our CI builds so you may not be able to find a Pharo VM.  


Below are the stack traces and primitive longs for two crashes with the compiled vm.

OK, that's disappointing, but important.  Maybe you can put an image/changes, support dylibs package up somewhere (Dropbox?) and I can try and run the test case locally. I tested the fix for some time and it clearly fixed one bug.  But it does seem that there's another in a similar area.  Thanks for your patience.

One question, you're sure you're running the new VM?


Process 19347 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x0000000140103f79
->  0x140103f79: int3   
    0x140103f7a: int3   
    0x140103f7b: int3   
    0x140103f7c: int3   
Target 0: (Pharo) stopped.

(lldb) call dumpPrimTraceLog()
basicNew
**StackOverflow**
basicNew
@
@
@
@
basicNew
basicNew
@
@
@
@
@
@
signal
signal
class
basicIdentityHash
basicNew
class
bitShift:
bitAnd:
asFloat
/
asFloat
fractionPart
truncated
fractionPart
truncated
fractionPart
truncated
setAlpha:
truncated
basicIdentityHash
basicNew
basicNew
class
class
basicNew
basicNew
basicIdentityHash
basicNew
basicNew
basicNew
wait
class
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
wait
**StackOverflow**
signal
replaceFrom:to:with:startingAt:
wait
**StackOverflow**
class
signal
basicNew
class
**StackOverflow**
**StackOverflow**
**StackOverflow**
wait
signal
basicIdentityHash
class
basicNew
basicNew
**StackOverflow**
class
class
**StackOverflow**
wait
signal
**StackOverflow**
**StackOverflow**
wait
signal
basicNew
basicNew
basicNew
signal
wait
signal
basicIdentityHash
basicIdentityHash
basicNew
basicNew
class
value:value:
basicNew
replaceFrom:to:with:startingAt:
valueWithArguments:
**StackOverflow**
basicIdentityHash
valueWithArguments:
basicNew
**StackOverflow**
class
findNextHandlerOrSignalingContext
**StackOverflow**
primitive
~=
~=
tempAt:
**StackOverflow**
tempAt:
basicNew
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy



(lldb) call printCallStack()
    0x7ffeefbe6d78 M Context(Object)>copy 0x145bbce28: a(n) Context
    0x7ffeefbe6db0 M Context>copyTo: 0x145bbce28: a(n) Context
    0x7ffeefbe6df8 M Context>copyTo: 0x145bbcd30: a(n) Context
    0x7ffeefbe6e40 M Context>copyTo: 0x145bbcc78: a(n) Context
    0x7ffeefbe6e88 M Context>copyTo: 0x145bbcbc0: a(n) Context
    0x7ffeefbe6ed0 M Context>copyTo: 0x19709adc8: a(n) Context
    0x7ffeefbe6f18 M Context>copyTo: 0x19709ad10: a(n) Context
    0x7ffeefbe37c8 M Context>copyTo: 0x19709ac58: a(n) Context
    0x7ffeefbe3810 M Context>copyTo: 0x19709aba0: a(n) Context
    0x7ffeefbe3858 M Context>copyTo: 0x19709aae8: a(n) Context
    0x7ffeefbe38a0 M Context>copyTo: 0x19709aa10: a(n) Context
    0x7ffeefbe38e8 M Context>copyTo: 0x19709a938: a(n) Context
    0x7ffeefbe3930 M Context>copyTo: 0x19709a860: a(n) Context
    0x7ffeefbe3978 M Context>copyTo: 0x19709a7a8: a(n) Context
    0x7ffeefbe39c0 M Context>copyTo: 0x19709a6f0: a(n) Context
    0x7ffeefbe3a08 M Context>copyTo: 0x19709a638: a(n) Context
    0x7ffeefbe3a50 M Context>copyTo: 0x19709a560: a(n) Context
    0x7ffeefbe3a98 M Context>copyTo: 0x19709a4a8: a(n) Context
    0x7ffeefbe3ae0 M Context>copyTo: 0x19709a3f0: a(n) Context
    0x7ffeefbe3b28 M Context>copyTo: 0x19709a338: a(n) Context
    0x7ffeefbe3b70 M Context>copyTo: 0x19709a280: a(n) Context
    0x7ffeefbe3bb8 M Context>copyTo: 0x19709a1c8: a(n) Context
    0x7ffeefbe3c00 M Context>copyTo: 0x19709a0e0: a(n) Context
    0x7ffeefbe3c48 M Context>copyTo: 0x197099ff8: a(n) Context
    0x7ffeefbe3c90 M Context>copyTo: 0x197099ee0: a(n) Context
    0x7ffeefbe3cd8 M Context>copyTo: 0x197099e28: a(n) Context
    0x7ffeefbe3d20 M Context>copyTo: 0x140169920: a(n) Context
    0x7ffeefbe3d68 M Context>copyTo: 0x140169868: a(n) Context
    0x7ffeefbe3db0 M Context>copyTo: 0x1401697b0: a(n) Context
    0x7ffeefbe3df8 M Context>copyTo: 0x1401696f8: a(n) Context
    0x7ffeefbe3e40 M Context>copyTo: 0x140169640: a(n) Context
    0x7ffeefbe3e88 M Context>copyTo: 0x140169588: a(n) Context
    0x7ffeefbe3ed0 M Context>copyTo: 0x1401694a0: a(n) Context
    0x7ffeefbe3f18 M Context>copyTo: 0x140169390: a(n) Context
    0x7ffeefbf47c8 M Context>copyTo: 0x1401692b0: a(n) Context
    0x7ffeefbf4810 M Context>copyTo: 0x1401691f8: a(n) Context
    0x7ffeefbf4858 M Context>copyTo: 0x140169140: a(n) Context
    0x7ffeefbf48a0 M Context>copyTo: 0x140169058: a(n) Context
    0x7ffeefbf48e8 M Context>copyTo: 0x140168f80: a(n) Context
    0x7ffeefbf4930 M Context>copyTo: 0x140168ec8: a(n) Context
    0x7ffeefbf4978 M Context>copyTo: 0x140168e10: a(n) Context
    0x7ffeefbf49c0 M Context>copyTo: 0x140168d38: a(n) Context
    0x7ffeefbf4a08 M Context>copyTo: 0x140168c58: a(n) Context
    0x7ffeefbf4a50 M Context>copyTo: 0x140168b80: a(n) Context
    0x7ffeefbf4a98 M Context>copyTo: 0x140168ac8: a(n) Context
    0x7ffeefbf4ae0 M Context>copyTo: 0x140168a10: a(n) Context
    0x7ffeefbf4b28 M Context>copyTo: 0x140168900: a(n) Context
    0x7ffeefbf4b70 M Context>copyTo: 0x140168810: a(n) Context
    0x7ffeefbf4bb8 M Context>copyTo: 0x140168718: a(n) Context
    0x7ffeefbf4c00 M Context>copyTo: 0x140168640: a(n) Context
    0x7ffeefbf4c48 M Context>copyTo: 0x14039c748: a(n) Context
    0x7ffeefbf4c90 M Context>copyTo: 0x14039c5d8: a(n) Context
    0x7ffeefbf4cd8 M Context>copyTo: 0x1401684b8: a(n) Context
    0x7ffeefbf4d20 M Context>copyTo: 0x1401683d8: a(n) Context
    0x7ffeefbf4d68 M Context>copyTo: 0x14039c2f8: a(n) Context
    0x7ffeefbf4db0 M Context>copyTo: 0x140167960: a(n) Context
    0x7ffeefbf4df8 M Context>copyTo: 0x1401678a8: a(n) Context
    0x7ffeefbf4e40 M Context>copyTo: 0x140167788: a(n) Context
    0x7ffeefbf4e88 M Context>copyTo: 0x14039bf60: a(n) Context
    0x7ffeefbf4ed0 M Context>copyTo: 0x14039bd38: a(n) Context
    0x7ffeefbf4f18 M Context>copyTo: 0x14039ba58: a(n) Context
    0x7ffeefbf07c8 M Context>copyTo: 0x14039b8e8: a(n) Context
    0x7ffeefbf0810 M Context>copyTo: 0x140167578: a(n) Context
    0x7ffeefbf0858 M Context>copyTo: 0x140167478: a(n) Context
    0x7ffeefbf08a0 M Context>copyTo: 0x14039b608: a(n) Context
    0x7ffeefbf08e8 M Context>copyTo: 0x14039b498: a(n) Context
    0x7ffeefbf0930 M Context>copyTo: 0x14039b328: a(n) Context
    0x7ffeefbf0978 M Context>copyTo: 0x140167310: a(n) Context
    0x7ffeefbf09c0 M Context>copyTo: 0x1401671e8: a(n) Context
    0x7ffeefbf0a08 M Context>copyTo: 0x140167108: a(n) Context
    0x7ffeefbf0a50 M Context>copyTo: 0x14039af90: a(n) Context
    0x7ffeefbf0a98 M Context>copyTo: 0x14039ae20: a(n) Context
    0x7ffeefbf0ae0 M Context>copyTo: 0x140166fe0: a(n) Context
    0x7ffeefbf0b28 M Context>copyTo: 0x140166ec0: a(n) Context
    0x7ffeefbf0b70 M Context>copyTo: 0x14039ab40: a(n) Context
    0x7ffeefbf0bb8 M Context>copyTo: 0x14039a9d0: a(n) Context
    0x7ffeefbf0c00 M Context>copyTo: 0x14039a860: a(n) Context
    0x7ffeefbf0c48 M Context>copyTo: 0x14039a6f0: a(n) Context
    0x7ffeefbf0c90 M Context>copyTo: 0x140166ca0: a(n) Context
    0x7ffeefbf0cd8 M Context>copyTo: 0x140166bb8: a(n) Context
    0x7ffeefbf0d20 M Context>copyTo: 0x140165f60: a(n) Context
    0x7ffeefbf0d68 M Context>copyTo: 0x140165ea8: a(n) Context
    0x7ffeefbf0db0 M Context>copyTo: 0x14039a2a0: a(n) Context
    0x7ffeefbf0df8 M Context>copyTo: 0x140165d48: a(n) Context
    0x7ffeefbf0e40 M Context>copyTo: 0x140165c20: a(n) Context
    0x7ffeefbf0e88 M Context>copyTo: 0x140165b40: a(n) Context
    0x7ffeefbf0ed0 M Context>copyTo: 0x140399e50: a(n) Context
    0x7ffeefbf0f18 M Context>copyTo: 0x140399b70: a(n) Context
    0x7ffeefbee7c8 M Context>copyTo: 0x140165a18: a(n) Context
    0x7ffeefbee810 M Context>copyTo: 0x1401658f8: a(n) Context
    0x7ffeefbee858 M Context>copyTo: 0x140399890: a(n) Context
    0x7ffeefbee8a0 M Context>copyTo: 0x140399720: a(n) Context
    0x7ffeefbee8e8 M Context>copyTo: 0x1403995b0: a(n) Context
    0x7ffeefbee930 M Context>copyTo: 0x140399440: a(n) Context
    0x7ffeefbee978 M Context>copyTo: 0x1403992d0: a(n) Context
    0x7ffeefbee9c0 M Context>copyTo: 0x140399160: a(n) Context
    0x7ffeefbeea08 M Context>copyTo: 0x140398ff0: a(n) Context
    0x7ffeefbeea50 M Context>copyTo: 0x140398e80: a(n) Context
    0x7ffeefbeea98 M Context>copyTo: 0x140398d10: a(n) Context
    0x7ffeefbeeae0 M Context>copyTo: 0x140165748: a(n) Context
    0x7ffeefbeeb28 M Context>copyTo: 0x140165648: a(n) Context
    0x7ffeefbeeb70 M Context>copyTo: 0x140398a30: a(n) Context
    0x7ffeefbeebb8 M Context>copyTo: 0x1403988c0: a(n) Context
    0x7ffeefbeec00 M Context>copyTo: 0x140165530: a(n) Context
    0x7ffeefbeec48 M Context>copyTo: 0x140165458: a(n) Context
    0x7ffeefbeec90 M Context>copyTo: 0x1403985e0: a(n) Context
    0x7ffeefbeecd8 M Context>copyTo: 0x140398470: a(n) Context
    0x7ffeefbeed20 M Context>copyTo: 0x140165088: a(n) Context
    0x7ffeefbeed68 M Context>copyTo: 0x140166ae0: a(n) Context
    0x7ffeefbeedb0 M Context>copyTo: 0x140398190: a(n) Context
    0x7ffeefbeedf8 M Context>copyTo: 0x140398020: a(n) Context
    0x7ffeefbeee40 M Context>copyTo: 0x140397eb0: a(n) Context
    0x7ffeefbeee88 M Context>copyTo: 0x1401669d8: a(n) Context
    0x7ffeefbeeed0 M Context>copyTo: 0x140397c88: a(n) Context
    0x7ffeefbeef18 M Context>copyTo: 0x140394d08: a(n) Context
    0x7ffeefbea798 M Context>copyTo: 0x140394e20: a(n) Context
    0x7ffeefbea7e0 M Context>copyTo: 0x140397838: a(n) Context
    0x7ffeefbea828 M Context>copyTo: 0x1403976c8: a(n) Context
    0x7ffeefbea870 M Context>copyTo: 0x140397558: a(n) Context
    0x7ffeefbea8b8 M Context>copyTo: 0x140395000: a(n) Context
    0x7ffeefbea900 M Context>copyTo: 0x140397330: a(n) Context
    0x7ffeefbea948 M Context>copyTo: 0x1403971c0: a(n) Context
    0x7ffeefbea990 M Context>copyTo: 0x140397050: a(n) Context
    0x7ffeefbea9d8 M Context>copyTo: 0x140396ee0: a(n) Context
    0x7ffeefbeaa20 M Context>copyTo: 0x140396d70: a(n) Context
    0x7ffeefbeaa68 M Context>copyTo: 0x1403958f8: a(n) Context
    0x7ffeefbeaab0 M Context>copyTo: 0x140396b48: a(n) Context
    0x7ffeefbeaaf8 M Context>copyTo: 0x140396298: a(n) Context
    0x7ffeefbeab40 M Context>copyTo: 0x140396920: a(n) Context
    0x7ffeefbeab88 M Context>copyTo: 0x1403967b0: a(n) Context
    0x7ffeefbeabd0 M Context>copyTo: 0x1403961e0: a(n) Context
    0x7ffeefbeac18 M Context>copyTo: 0x140396128: a(n) Context
    0x7ffeefbeac60 M Context>copyTo: 0x140396070: a(n) Context
    0x7ffeefbeacb8 I Context>copyTo: 0x140395e28: a(n) Context
    0x7ffeefbead00 I ZeroDivide(Exception)>freezeUpTo: 0x140395de8: a(n) ZeroDivide
    0x7ffeefbead48 I ZeroDivide(Exception)>freeze 0x140395de8: a(n) ZeroDivide
    0x7ffeefbead88 I BrDebugglableElementStencil>freeze: 0x140396408: a(n) BrDebugglableElementStencil
    0x7ffeefbeadd8 I ZeroDivide(Exception)>asDebuggableElement 0x140395de8: a(n) ZeroDivide
    0x7ffeefbeae18 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbeae50 M BlockClosure>cull: 0x1403959f0: a(n) BlockClosure
    0x7ffeefbeaea0 I Context>evaluateSignal: 0x140396298: a(n) Context
    0x7ffeefbeaed8 M Context>handleSignal: 0x140396298: a(n) Context
    0x7ffeefbeaf20 I ZeroDivide(Exception)>signal 0x140395de8: a(n) ZeroDivide
    0x7ffeefbe7840 I ZeroDivide(Exception)>signal: 0x140395de8: a(n) ZeroDivide
    0x7ffeefbe7888 I ZeroDivide class(Exception class)>signal: 0x1409f9490: a(n) ZeroDivide class
    0x7ffeefbe78c0 M [] in GtPhlowColumnedListViewExamples>gtErrorInColumnDoDataBinderFor: 0x140168630: a(n) GtPhlowColumnedListViewExamples
    0x7ffeefbe7908 M BlockClosure>glamourValueWithArgs: 0x140174180: a(n) BlockClosure
    0x7ffeefbe7960 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7990 M BlockClosure>on:do: 0x1403959b0: a(n) BlockClosure
    0x7ffeefbe79d0 M GtPhlowColumn>performBlock:onException: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7a18 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7a60 M BlockClosure>glamourValueWithArgs: 0x1401701c0: a(n) BlockClosure
    0x7ffeefbe7a98 M BrStencilValuableExecutor>execute 0x140174230: a(n) BrStencilValuableExecutor
    0x7ffeefbe7ad8 M BrColumnCellDataBinder(BrStencilBuilder)>build 0x140170448: a(n) BrColumnCellDataBinder
    0x7ffeefbe7b38 I [] in BrColumnedListDataSource>onBindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7b98 I OrderedCollection(SequenceableCollection)>with:do: 0x140169ff8: a(n) OrderedCollection
    0x7ffeefbe7bf8 I BrColumnedListDataSource>onBindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7c48 I BrColumnedListDataSource(BlInfiniteDataSource)>onBindHolder:at:payloads: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7ca0 M [] in BrColumnedListDataSource(BlInfiniteDataSource)>bindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7ce0 M BlockClosure>ensure: 0x140394df0: a(n) BlockClosure
    0x7ffeefbe7d18 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe7d58 M BrColumnedListDataSource(BlInfiniteDataSource)>bindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7d98 M BlInfiniteRecyclerController>bindHolder:at: 0x1401662c0: a(n) BlInfiniteRecyclerController
    0x7ffeefbe7e10 M BlInfiniteRecycler>elementFor:dryRun: 0x1401652b8: a(n) BlInfiniteRecycler
    0x7ffeefbe7e50 M BlInfiniteRecycler>elementFor: 0x1401652b8: a(n) BlInfiniteRecycler
    0x7ffeefbe7e98 M [] in BlInfiniteLinearLayoutState>nextElement:in: 0x140165020: a(n) BlInfiniteLinearLayoutState
    0x7ffeefbe7ed8 M BlockClosure>ensure: 0x140166a90: a(n) BlockClosure
    0x7ffeefbe7f10 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe27d0 M BlInfiniteLinearLayoutState>nextElement:in: 0x140165020: a(n) BlInfiniteLinearLayoutState
    0x7ffeefbe2818 M [] in BlInfiniteLinearLayout>layoutChunkAdd 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2858 M BlockClosure>ensure: 0x140165410: a(n) BlockClosure
    0x7ffeefbe2890 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe28d8 M BlInfiniteLinearLayout>layoutChunkAdd 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2910 M [] in BlInfiniteLinearLayout>layoutChunk 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2950 M BlockClosure>ensure: 0x140165600: a(n) BlockClosure
    0x7ffeefbe2988 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe29e0 M BlInfiniteLinearLayout>layoutChunk 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2a48 I BlInfiniteLinearLayout>fillLayout: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2ab8 I BlInfiniteLinearLayout>layoutChildrenFillFromStart: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2b00 I BlInfiniteLinearLayout>layoutChildrenFill: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2b58 I BlInfiniteLinearLayout>layoutChildrenIn:state: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2ba8 I BrInfiniteListElement(BlInfiniteElement)>dispatchLayoutSecondStep 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2be8 I BrInfiniteListElement(BlInfiniteElement)>dispatchLayout 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c18 M BrInfiniteListElement(BlInfiniteElement)>onLayout: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c58 M [] in BrInfiniteListElement(BlElement)>applyLayoutSafelyIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c98 M BlockClosure>ensure: 0x1401658b0: a(n) BlockClosure
    0x7ffeefbe2cd0 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe2d30 M BrInfiniteListElement(BlElement)>applyLayoutSafelyIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2d70 M [] in BrInfiniteListElement(BlElement)>applyLayoutIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2da0 M BlockClosure>on:do: 0x140165ad0: a(n) BlockClosure
    0x7ffeefbe2de0 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x140165b28: a(n) BlCompositeErrorHandler
    0x7ffeefbe2e28 M BrInfiniteListElement(BlElement)>applyLayoutIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2e80 M [] in BlLinearLayoutVerticalOrientation>layout:in: 0x140165d38: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe2ed0 M [] in BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe2f18 M Array(SequenceableCollection)>do: 0x140165e98: a(n) Array
    0x7ffeefbe9868 M BlChildrenAccountedByLayout(BlChildrenSubset)>do: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe98a8 M BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9918 M BlLinearLayoutVerticalOrientation>layout:in: 0x140165d38: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9958 M BlLinearLayout>layout:in: 0x140166e18: a(n) BlLinearLayout
    0x7ffeefbe9998 M BrColumnedList(BlElement)>onLayout: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe99d8 M [] in BrColumnedList(BlElement)>applyLayoutSafelyIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9a18 M BlockClosure>ensure: 0x140166e78: a(n) BlockClosure
    0x7ffeefbe9a50 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe9ab0 M BrColumnedList(BlElement)>applyLayoutSafelyIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9af0 M [] in BrColumnedList(BlElement)>applyLayoutIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9b20 M BlockClosure>on:do: 0x140167098: a(n) BlockClosure
    0x7ffeefbe9b60 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x1401670f0: a(n) BlCompositeErrorHandler
    0x7ffeefbe9ba8 M BrColumnedList(BlElement)>applyLayoutIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9c00 M [] in BlLinearLayoutVerticalOrientation>layout:in: 0x140167300: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9c50 M [] in BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9c98 M Array(SequenceableCollection)>do: 0x140167460: a(n) Array
    0x7ffeefbe9cd0 M BlChildrenAccountedByLayout(BlChildrenSubset)>do: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9d10 M BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9d80 M BlLinearLayoutVerticalOrientation>layout:in: 0x140167300: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9dc0 M BlLinearLayout>layout:in: 0x1401676e0: a(n) BlLinearLayout
    0x7ffeefbe9e00 M BlElement>onLayout: 0x140167648: a(n) BlElement
    0x7ffeefbe9e40 M [] in BlElement>applyLayoutSafelyIn: 0x140167648: a(n) BlElement
    0x7ffeefbe9e80 M BlockClosure>ensure: 0x140167740: a(n) BlockClosure
    0x7ffeefbe9eb8 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe9f18 M BlElement>applyLayoutSafelyIn: 0x140167648: a(n) BlElement
    0x7ffeefbed7e0 M [] in BlElement>applyLayoutIn: 0x140167648: a(n) BlElement
    0x7ffeefbed810 M BlockClosure>on:do: 0x140168368: a(n) BlockClosure
    0x7ffeefbed850 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x1401683c0: a(n) BlCompositeErrorHandler
    0x7ffeefbed898 M BlElement>applyLayoutIn: 0x140167648: a(n) BlElement
    0x7ffeefbed908 I BlElement>computeLayout 0x140167648: a(n) BlElement
    0x7ffeefbed948 I BlElement>forceLayout 0x140167648: a(n) BlElement
    0x7ffeefbed9a8 I GtPhlowColumnedListViewExamples>errorInColumnDoDataBinder 0x140168630: a(n) GtPhlowColumnedListViewExamples
    0x7ffeefbed9d8 M GtExampleDebugger(GtExampleEvaluator)>primitiveProcessExample:withEvaluationContext: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbeda38 M [] in GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbeda78 M BlockClosure>ensure: 0x1401688c8: a(n) BlockClosure
    0x7ffeefbedac8 M GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb00 M GtExampleDebugger(GtExampleProcessor)>process: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb38 M [] in GtExampleDebugger(GtExampleProcessor)>value 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb68 M BlockClosure>on:do: 0x140168c38: a(n) BlockClosure
    0x7ffeefbedba8 M GtCurrentExampleContext class>use:during: 0x1446188f0: a(n) GtCurrentExampleContext class
    0x7ffeefbedbe8 M GtExampleDebugger(GtExampleProcessor)>withContextDo: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc20 M GtExampleDebugger(GtExampleProcessor)>value 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc50 M [] in GtExampleDebugger(GtExampleEvaluator)>result 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc80 M GtExampleDebugger>do:on:do: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedcc8 M GtExampleDebugger(GtExampleEvaluator)>result 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedcf8 M GtExample>debug 0x19709dfc8: a(n) GtExample
    0x7ffeefbedd30 M [] in GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedd60 M BlockClosure>on:do: 0x140169368: a(n) BlockClosure
    0x7ffeefbeddb0 M [] in GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedde8 M Time class>microsecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbede20 M Time class>millisecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbede60 M BlockClosure>timeToRun 0x140169558: a(n) BlockClosure
    0x7ffeefbede98 M GtExamplesHDReport>beginExample:runBlock: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedee0 M GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedf18 M [] in GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdea38 M OrderedCollection>do: 0x197099e08: a(n) OrderedCollection
    0x7ffeefbdea70 M [] in GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdeab0 M [] in CurrentExecutionEnvironment class>activate:for: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdeaf0 M BlockClosure>ensure: 0x19709a0b0: a(n) BlockClosure
    0x7ffeefbdeb30 M CurrentExecutionEnvironment class>activate:for: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdeb70 M TestExecutionEnvironment(ExecutionEnvironment)>beActiveDuring: 0x197099fb8: a(n) TestExecutionEnvironment
    0x7ffeefbdeba8 M DefaultExecutionEnvironment>runTestsBy: 0x1409f7da8: a(n) DefaultExecutionEnvironment
    0x7ffeefbdebe0 M CurrentExecutionEnvironment class>runTestsBy: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdec18 M GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdec48 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdec80 M Time class>microsecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbdecb8 M Time class>millisecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbdecf8 M BlockClosure>timeToRun 0x19709a618: a(n) BlockClosure
    0x7ffeefbded28 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbded68 M BlockClosure>ensure: 0x19709a918: a(n) BlockClosure
    0x7ffeefbdeda0 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdedd0 M Author>ifUnknownAuthorUse:during: 0x14102a740: a(n) Author
    0x7ffeefbdee10 M GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdee48 M GtExamplesHDReport class>runPackage: 0x144618008: a(n) GtExamplesHDReport class
    0x7ffeefbdee80 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x144618008: a(n) GtExamplesHDReport class
    0x7ffeefbdeed0 M [] in Set>collect: 0x145bbc4d8: a(n) Set
    0x7ffeefbdef18 M Array(SequenceableCollection)>do: 0x145bbc520: a(n) Array
       0x145bbcc78 s Set>collect:
       0x145bbcd30 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x145bbce28 s [] in GtExamplesCommandLineHandler>runPackages
       0x145bbcf08 s BlockClosure>ensure:
       0x14e950710 s UIManager class>nonInteractiveDuring:
       0x14e9507c8 s GtExamplesCommandLineHandler>runPackages
       0x14e950880 s GtExamplesCommandLineHandler>activate
       0x14e950938 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x14e9509f0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x14e950aa8 s BlockClosure>on:do:
       0x14e950b60 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x14e950c38 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x14e950cf0 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x14e950db8 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x14e950e90 s BlockClosure>on:do:
       0x14e950f68 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x14e951040 s [] in BlockClosure>newProcess



Process 18798 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001481107d1
->  0x1481107d1: int3   
    0x1481107d2: int3   
    0x1481107d3: int3   
    0x1481107d4: int3   
Target 0: (Pharo) stopped.

(lldb) call printAllStacks()
Process        0x15511b938 priority 40
    0x7ffeefbf0a18 M Context(Object)>copy 0x15691cf68: a(n) Context
    0x7ffeefbf0a50 M Context>copyTo: 0x15691cf68: a(n) Context
    0x7ffeefbf0a98 M Context>copyTo: 0x15691ce78: a(n) Context
    0x7ffeefbf0ae0 M Context>copyTo: 0x15691cdc0: a(n) Context
    0x7ffeefbf0b28 M Context>copyTo: 0x15691cd08: a(n) Context
    0x7ffeefbf0b70 M Context>copyTo: 0x15691cc50: a(n) Context
    0x7ffeefbf0bb8 M Context>copyTo: 0x15691cb70: a(n) Context
    0x7ffeefbf0c00 M Context>copyTo: 0x15691ca90: a(n) Context
    0x7ffeefbf0c48 M Context>copyTo: 0x15691c998: a(n) Context
    0x7ffeefbf0c90 M Context>copyTo: 0x15691c8e0: a(n) Context
    0x7ffeefbf0cd8 M Context>copyTo: 0x15691c828: a(n) Context
    0x7ffeefbf0d20 M Context>copyTo: 0x14874a2c8: a(n) Context
    0x7ffeefbf0d68 M Context>copyTo: 0x14874a158: a(n) Context
    0x7ffeefbf0db0 M Context>copyTo: 0x148749fe8: a(n) Context
    0x7ffeefbf0df8 M Context>copyTo: 0x14816e2e8: a(n) Context
    0x7ffeefbf0e40 M Context>copyTo: 0x148749dc0: a(n) Context
    0x7ffeefbf0e88 M Context>copyTo: 0x14816e210: a(n) Context
    0x7ffeefbf0ed0 M Context>copyTo: 0x148749ae0: a(n) Context
    0x7ffeefbf0f18 M Context>copyTo: 0x14816e118: a(n) Context
    0x7ffeefbdc7c8 M Context>copyTo: 0x148749748: a(n) Context
    0x7ffeefbdc810 M Context>copyTo: 0x1487495d8: a(n) Context
    0x7ffeefbdc858 M Context>copyTo: 0x148749468: a(n) Context
    0x7ffeefbdc8a0 M Context>copyTo: 0x1487492f8: a(n) Context
    0x7ffeefbdc8e8 M Context>copyTo: 0x14816e040: a(n) Context
    0x7ffeefbdc930 M Context>copyTo: 0x1487490d0: a(n) Context
    0x7ffeefbdc978 M Context>copyTo: 0x148748f60: a(n) Context
    0x7ffeefbdc9c0 M Context>copyTo: 0x148748df0: a(n) Context
    0x7ffeefbdca08 M Context>copyTo: 0x14816df38: a(n) Context
    0x7ffeefbdca50 M Context>copyTo: 0x148748bc8: a(n) Context
    0x7ffeefbdca98 M Context>copyTo: 0x14816de20: a(n) Context
    0x7ffeefbdcae0 M Context>copyTo: 0x14816eeb8: a(n) Context
    0x7ffeefbdcb28 M Context>copyTo: 0x1487488e8: a(n) Context
    0x7ffeefbdcb70 M Context>copyTo: 0x148748778: a(n) Context
    0x7ffeefbdcbb8 M Context>copyTo: 0x14866e858: a(n) Context
    0x7ffeefbdcc00 M Context>copyTo: 0x148748550: a(n) Context
    0x7ffeefbdcc48 M Context>copyTo: 0x1487483e0: a(n) Context
    0x7ffeefbdcc90 M Context>copyTo: 0x148748270: a(n) Context
    0x7ffeefbdccd8 M Context>copyTo: 0x148748100: a(n) Context
    0x7ffeefbdcd20 M Context>copyTo: 0x148676078: a(n) Context
    0x7ffeefbdcd68 M Context>copyTo: 0x148747ed8: a(n) Context
    0x7ffeefbdcdb0 M Context>copyTo: 0x148747d68: a(n) Context
    0x7ffeefbdcdf8 M Context>copyTo: 0x148747bf8: a(n) Context
    0x7ffeefbdce40 M Context>copyTo: 0x148676280: a(n) Context
    0x7ffeefbdce88 M Context>copyTo: 0x1487479d0: a(n) Context
    0x7ffeefbdced0 M Context>copyTo: 0x1487477a8: a(n) Context
    0x7ffeefbdcf18 M Context>copyTo: 0x148676400: a(n) Context
    0x7ffeefbda7c8 M Context>copyTo: 0x148747410: a(n) Context
    0x7ffeefbda810 M Context>copyTo: 0x1486764d8: a(n) Context
    0x7ffeefbda858 M Context>copyTo: 0x1487471e8: a(n) Context
    0x7ffeefbda8a0 M Context>copyTo: 0x148747078: a(n) Context
    0x7ffeefbda8e8 M Context>copyTo: 0x148746f08: a(n) Context
    0x7ffeefbda930 M Context>copyTo: 0x14873f918: a(n) Context
    0x7ffeefbda978 M Context>copyTo: 0x148746ce0: a(n) Context
    0x7ffeefbda9c0 M Context>copyTo: 0x148746b70: a(n) Context
    0x7ffeefbdaa08 M Context>copyTo: 0x148746a00: a(n) Context
    0x7ffeefbdaa50 M Context>copyTo: 0x148746890: a(n) Context
    0x7ffeefbdaa98 M Context>copyTo: 0x148746720: a(n) Context
    0x7ffeefbdaae0 M Context>copyTo: 0x1487415b8: a(n) Context
    0x7ffeefbdab28 M Context>copyTo: 0x1487412a8: a(n) Context
    0x7ffeefbdab70 M Context>copyTo: 0x148746440: a(n) Context
    0x7ffeefbdabb8 M Context>copyTo: 0x148745088: a(n) Context
    0x7ffeefbdac00 M Context>copyTo: 0x148746218: a(n) Context
    0x7ffeefbdac48 M Context>copyTo: 0x1487414e0: a(n) Context
    0x7ffeefbdac90 M Context>copyTo: 0x148745ff0: a(n) Context
    0x7ffeefbdacd8 M Context>copyTo: 0x148741670: a(n) Context
    0x7ffeefbdad20 M Context>copyTo: 0x148744fd0: a(n) Context
    0x7ffeefbdad68 M Context>copyTo: 0x148745ba0: a(n) Context
    0x7ffeefbdadb0 M Context>copyTo: 0x148745a30: a(n) Context
    0x7ffeefbdadf8 M Context>copyTo: 0x148744bd8: a(n) Context
    0x7ffeefbdae40 M Context>copyTo: 0x148745808: a(n) Context
    0x7ffeefbdae88 M Context>copyTo: 0x148745698: a(n) Context
    0x7ffeefbdaed0 M Context>copyTo: 0x148745528: a(n) Context
    0x7ffeefbdaf18 M Context>copyTo: 0x148744f18: a(n) Context
    0x7ffeefbe0988 I Context>copyTo: 0x148744e60: a(n) Context
    0x7ffeefbe09d0 I MessageNotUnderstood(Exception)>freezeUpTo: 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0a18 I MessageNotUnderstood(Exception)>freeze 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0a48 M [] in GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0a80 M BlockClosure>cull: 0x1487414c0: a(n) BlockClosure
    0x7ffeefbe0ac0 M Context>evaluateSignal: 0x148745088: a(n) Context
    0x7ffeefbe0af8 M Context>handleSignal: 0x148745088: a(n) Context
    0x7ffeefbe0b30 M Context>handleSignal: 0x148744fd0: a(n) Context
    0x7ffeefbe0b68 M MessageNotUnderstood(Exception)>signal 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0bb8 I GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x148744bc8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefbe0bf0 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0c50 M [] in GtExampleEvaluator>basicProcess: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0c90 M BlockClosure>ensure: 0x148744c90: a(n) BlockClosure
    0x7ffeefbe0ce0 M GtExampleEvaluator>basicProcess: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d18 M GtExampleEvaluator(GtExampleProcessor)>process: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d50 M [] in GtExampleEvaluator(GtExampleProcessor)>value 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d80 M BlockClosure>on:do: 0x148741598: a(n) BlockClosure
    0x7ffeefbe0dc0 M GtCurrentExampleContext class>use:during: 0x14c618920: a(n) GtCurrentExampleContext class
    0x7ffeefbe0e00 M GtExampleEvaluator(GtExampleProcessor)>withContextDo: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e38 M GtExampleEvaluator(GtExampleProcessor)>value 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e68 M [] in GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e98 M BlockClosure>on:do: 0x148741360: a(n) BlockClosure
    0x7ffeefbe0ed8 M GtExampleEvaluator>do:on:do: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0f20 M GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbde8b8 M GtExample>run 0x148740050: a(n) GtExample
    0x7ffeefbde8e8 M GtExample>result 0x148740050: a(n) GtExample
    0x7ffeefbde930 I GtExamplesExamplesWithInheritance>resultOfInvalidExampleWithInvalidSuperclassProvider 0x14873f908: a(n) GtExamplesExamplesWithInheritance
    0x7ffeefbde960 M GtExampleDebugger(GtExampleEvaluator)>primitiveProcessExample:withEvaluationContext: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbde9c0 M [] in GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdea00 M BlockClosure>ensure: 0x14873f9d0: a(n) BlockClosure
    0x7ffeefbdea50 M GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdea88 M GtExampleDebugger(GtExampleProcessor)>process: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeac0 M [] in GtExampleDebugger(GtExampleProcessor)>value 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeaf0 M BlockClosure>on:do: 0x1486764b8: a(n) BlockClosure
    0x7ffeefbdeb30 M GtCurrentExampleContext class>use:during: 0x14c618920: a(n) GtCurrentExampleContext class
    0x7ffeefbdeb70 M GtExampleDebugger(GtExampleProcessor)>withContextDo: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeba8 M GtExampleDebugger(GtExampleProcessor)>value 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdebd8 M [] in GtExampleDebugger(GtExampleEvaluator)>result 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec08 M GtExampleDebugger>do:on:do: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec50 M GtExampleDebugger(GtExampleEvaluator)>result 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec80 M GtExample>debug 0x148188088: a(n) GtExample
    0x7ffeefbdecb8 M [] in GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdece8 M BlockClosure>on:do: 0x148676130: a(n) BlockClosure
    0x7ffeefbded38 M [] in GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbded70 M Time class>microsecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbdeda8 M Time class>millisecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbdede8 M BlockClosure>timeToRun 0x14866e910: a(n) BlockClosure
    0x7ffeefbdee20 M GtExamplesHDReport>beginExample:runBlock: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdee68 M GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdeea0 M [] in GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdeee8 M OrderedCollection>do: 0x14816ee98: a(n) OrderedCollection
    0x7ffeefbdef20 M [] in GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8ab0 M [] in CurrentExecutionEnvironment class>activate:for: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8af0 M BlockClosure>ensure: 0x14816ded8: a(n) BlockClosure
    0x7ffeefbd8b30 M CurrentExecutionEnvironment class>activate:for: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8b70 M TestExecutionEnvironment(ExecutionEnvironment)>beActiveDuring: 0x14816ddc0: a(n) TestExecutionEnvironment
    0x7ffeefbd8ba8 M DefaultExecutionEnvironment>runTestsBy: 0x1489f7da8: a(n) DefaultExecutionEnvironment
    0x7ffeefbd8be0 M CurrentExecutionEnvironment class>runTestsBy: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8c18 M GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8c48 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8c80 M Time class>microsecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbd8cb8 M Time class>millisecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbd8cf8 M BlockClosure>timeToRun 0x14816e0f8: a(n) BlockClosure
    0x7ffeefbd8d28 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8d68 M BlockClosure>ensure: 0x14816e1d0: a(n) BlockClosure
    0x7ffeefbd8da0 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8dd0 M Author>ifUnknownAuthorUse:during: 0x14902a740: a(n) Author
    0x7ffeefbd8e10 M GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8e48 M GtExamplesHDReport class>runPackage: 0x14c618038: a(n) GtExamplesHDReport class
    0x7ffeefbd8e80 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x14c618038: a(n) GtExamplesHDReport class
    0x7ffeefbd8ed0 M [] in Set>collect: 0x15691c088: a(n) Set
    0x7ffeefbd8f18 M Array(SequenceableCollection)>do: 0x15691c188: a(n) Array
       0x15691c8e0 s Set>collect:
       0x15691c998 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x15691ca90 s [] in GtExamplesCommandLineHandler>runPackages
       0x15691cb70 s BlockClosure>ensure:
       0x15691cc50 s UIManager class>nonInteractiveDuring:
       0x15691cd08 s GtExamplesCommandLineHandler>runPackages
       0x15691cdc0 s GtExamplesCommandLineHandler>activate
       0x15691ce78 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x15691cf68 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x15691d048 s BlockClosure>on:do:
       0x15691d128 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x15691d200 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x15691d2b8 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x15691d380 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x15691d458 s BlockClosure>on:do:
       0x15691d530 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x15691d608 s [] in BlockClosure>newProcess


(lldb) call dumpPrimTraceLog()
stringHash:initialHash:
compare:with:collated:
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
primitiveChangeClassTo:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
primitiveChangeClassTo:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
stringHash:initialHash:
compare:with:collated:
primitiveChangeClassTo:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
basicNew:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
size
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
**StackOverflow**
perform:withArguments:
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
basicNew
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
**StackOverflow**
**StackOverflow**
class
perform:withArguments:
value:
first
basicNew
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
perform:withArguments:
basicNew
findNextHandlerOrSignalingContext
tempAt:
class
findNextHandlerOrSignalingContext
tempAt:
tempAt:
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**CompactCode**



On 26 Sep 2020, at 23:24, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

   fixed in commit 561b06530bbaed5f19e9d7f077a7df9eb3a8d236, VMMaker.oscog-eem.2824


On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption? 


 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Andrei Chis
 
Hi Eliot,

I get the following details for the VM that I compiled locally:

CoInterpreter VMMaker.oscog-eem.2824 uuid: 8f091e5b-fc0f-4b4b-ab5e-e90e598f75ee Sep 27 2020
StackToRegisterMappingCogit VMMaker.oscog-eem.2824 uuid: 8f091e5b-fc0f-4b4b-ab5e-e90e598f75ee Sep 27 2020
VM: 202009270258 [hidden email]:Documents/Pharo/vm-compilation/opensmalltalk-vm Date: Sat Sep 26 19:58:41 2020 CommitHash: 5a0d21d71 Plugins: 202009270258 [hidden email]:Documents/Pharo/vm-compilation/opensmalltalk-vm

Mac OS X built on Sep 27 2020 14:27:59 CEST Compiler: 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.32.62)
VMMaker versionString VM: 202009270258 [hidden email]:Documents/Pharo/vm-compilation/opensmalltalk-vm Date: Sat Sep 26 19:58:41 2020 CommitHash: 5a0d21d71 Plugins: 202009270258 [hidden email]:Documents/Pharo/vm-compilation/opensmalltalk-vm
CoInterpreter VMMaker.oscog-eem.2824 uuid: 8f091e5b-fc0f-4b4b-ab5e-e90e598f75ee Sep 27 2020
StackToRegisterMappingCogit VMMaker.oscog-eem.2824 uuid: 8f091e5b-fc0f-4b4b-ab5e-e90e598f75ee Sep 27 2020


I can package together the image/changes file and libs and send them privately. 

Cheers,
Andrei

On 27 Sep 2020, at 21:23, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Sun, Sep 27, 2020 at 10:29 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks a lot for your help.

I compiled locally a vm from the commit hash 5a0d21d71223b4d1f4d007ebf2441d530c18c3e2 (CogVM source as per VMMaker.oscog-eem.2826) and I got a Context>>copyTo: crash.
I think I compiled the vm correctly. Are them some Pharo VMs built on travis for Mac to try out instead of building locally?

There may be.  The PharoVM team are not maintaining our CI builds so you may not be able to find a Pharo VM.  


Below are the stack traces and primitive longs for two crashes with the compiled vm.

OK, that's disappointing, but important.  Maybe you can put an image/changes, support dylibs package up somewhere (Dropbox?) and I can try and run the test case locally. I tested the fix for some time and it clearly fixed one bug.  But it does seem that there's another in a similar area.  Thanks for your patience.

One question, you're sure you're running the new VM?


Process 19347 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x0000000140103f79
->  0x140103f79: int3   
    0x140103f7a: int3   
    0x140103f7b: int3   
    0x140103f7c: int3   
Target 0: (Pharo) stopped.

(lldb) call dumpPrimTraceLog()
basicNew
**StackOverflow**
basicNew
@
@
@
@
basicNew
basicNew
@
@
@
@
@
@
signal
signal
class
basicIdentityHash
basicNew
class
bitShift:
bitAnd:
asFloat
/
asFloat
fractionPart
truncated
fractionPart
truncated
fractionPart
truncated
setAlpha:
truncated
basicIdentityHash
basicNew
basicNew
class
class
basicNew
basicNew
basicIdentityHash
basicNew
basicNew
basicNew
wait
class
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
wait
**StackOverflow**
signal
replaceFrom:to:with:startingAt:
wait
**StackOverflow**
class
signal
basicNew
class
**StackOverflow**
**StackOverflow**
**StackOverflow**
wait
signal
basicIdentityHash
class
basicNew
basicNew
**StackOverflow**
class
class
**StackOverflow**
wait
signal
**StackOverflow**
**StackOverflow**
wait
signal
basicNew
basicNew
basicNew
signal
wait
signal
basicIdentityHash
basicIdentityHash
basicNew
basicNew
class
value:value:
basicNew
replaceFrom:to:with:startingAt:
valueWithArguments:
**StackOverflow**
basicIdentityHash
valueWithArguments:
basicNew
**StackOverflow**
class
findNextHandlerOrSignalingContext
**StackOverflow**
primitive
~=
~=
tempAt:
**StackOverflow**
tempAt:
basicNew
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy



(lldb) call printCallStack()
    0x7ffeefbe6d78 M Context(Object)>copy 0x145bbce28: a(n) Context
    0x7ffeefbe6db0 M Context>copyTo: 0x145bbce28: a(n) Context
    0x7ffeefbe6df8 M Context>copyTo: 0x145bbcd30: a(n) Context
    0x7ffeefbe6e40 M Context>copyTo: 0x145bbcc78: a(n) Context
    0x7ffeefbe6e88 M Context>copyTo: 0x145bbcbc0: a(n) Context
    0x7ffeefbe6ed0 M Context>copyTo: 0x19709adc8: a(n) Context
    0x7ffeefbe6f18 M Context>copyTo: 0x19709ad10: a(n) Context
    0x7ffeefbe37c8 M Context>copyTo: 0x19709ac58: a(n) Context
    0x7ffeefbe3810 M Context>copyTo: 0x19709aba0: a(n) Context
    0x7ffeefbe3858 M Context>copyTo: 0x19709aae8: a(n) Context
    0x7ffeefbe38a0 M Context>copyTo: 0x19709aa10: a(n) Context
    0x7ffeefbe38e8 M Context>copyTo: 0x19709a938: a(n) Context
    0x7ffeefbe3930 M Context>copyTo: 0x19709a860: a(n) Context
    0x7ffeefbe3978 M Context>copyTo: 0x19709a7a8: a(n) Context
    0x7ffeefbe39c0 M Context>copyTo: 0x19709a6f0: a(n) Context
    0x7ffeefbe3a08 M Context>copyTo: 0x19709a638: a(n) Context
    0x7ffeefbe3a50 M Context>copyTo: 0x19709a560: a(n) Context
    0x7ffeefbe3a98 M Context>copyTo: 0x19709a4a8: a(n) Context
    0x7ffeefbe3ae0 M Context>copyTo: 0x19709a3f0: a(n) Context
    0x7ffeefbe3b28 M Context>copyTo: 0x19709a338: a(n) Context
    0x7ffeefbe3b70 M Context>copyTo: 0x19709a280: a(n) Context
    0x7ffeefbe3bb8 M Context>copyTo: 0x19709a1c8: a(n) Context
    0x7ffeefbe3c00 M Context>copyTo: 0x19709a0e0: a(n) Context
    0x7ffeefbe3c48 M Context>copyTo: 0x197099ff8: a(n) Context
    0x7ffeefbe3c90 M Context>copyTo: 0x197099ee0: a(n) Context
    0x7ffeefbe3cd8 M Context>copyTo: 0x197099e28: a(n) Context
    0x7ffeefbe3d20 M Context>copyTo: 0x140169920: a(n) Context
    0x7ffeefbe3d68 M Context>copyTo: 0x140169868: a(n) Context
    0x7ffeefbe3db0 M Context>copyTo: 0x1401697b0: a(n) Context
    0x7ffeefbe3df8 M Context>copyTo: 0x1401696f8: a(n) Context
    0x7ffeefbe3e40 M Context>copyTo: 0x140169640: a(n) Context
    0x7ffeefbe3e88 M Context>copyTo: 0x140169588: a(n) Context
    0x7ffeefbe3ed0 M Context>copyTo: 0x1401694a0: a(n) Context
    0x7ffeefbe3f18 M Context>copyTo: 0x140169390: a(n) Context
    0x7ffeefbf47c8 M Context>copyTo: 0x1401692b0: a(n) Context
    0x7ffeefbf4810 M Context>copyTo: 0x1401691f8: a(n) Context
    0x7ffeefbf4858 M Context>copyTo: 0x140169140: a(n) Context
    0x7ffeefbf48a0 M Context>copyTo: 0x140169058: a(n) Context
    0x7ffeefbf48e8 M Context>copyTo: 0x140168f80: a(n) Context
    0x7ffeefbf4930 M Context>copyTo: 0x140168ec8: a(n) Context
    0x7ffeefbf4978 M Context>copyTo: 0x140168e10: a(n) Context
    0x7ffeefbf49c0 M Context>copyTo: 0x140168d38: a(n) Context
    0x7ffeefbf4a08 M Context>copyTo: 0x140168c58: a(n) Context
    0x7ffeefbf4a50 M Context>copyTo: 0x140168b80: a(n) Context
    0x7ffeefbf4a98 M Context>copyTo: 0x140168ac8: a(n) Context
    0x7ffeefbf4ae0 M Context>copyTo: 0x140168a10: a(n) Context
    0x7ffeefbf4b28 M Context>copyTo: 0x140168900: a(n) Context
    0x7ffeefbf4b70 M Context>copyTo: 0x140168810: a(n) Context
    0x7ffeefbf4bb8 M Context>copyTo: 0x140168718: a(n) Context
    0x7ffeefbf4c00 M Context>copyTo: 0x140168640: a(n) Context
    0x7ffeefbf4c48 M Context>copyTo: 0x14039c748: a(n) Context
    0x7ffeefbf4c90 M Context>copyTo: 0x14039c5d8: a(n) Context
    0x7ffeefbf4cd8 M Context>copyTo: 0x1401684b8: a(n) Context
    0x7ffeefbf4d20 M Context>copyTo: 0x1401683d8: a(n) Context
    0x7ffeefbf4d68 M Context>copyTo: 0x14039c2f8: a(n) Context
    0x7ffeefbf4db0 M Context>copyTo: 0x140167960: a(n) Context
    0x7ffeefbf4df8 M Context>copyTo: 0x1401678a8: a(n) Context
    0x7ffeefbf4e40 M Context>copyTo: 0x140167788: a(n) Context
    0x7ffeefbf4e88 M Context>copyTo: 0x14039bf60: a(n) Context
    0x7ffeefbf4ed0 M Context>copyTo: 0x14039bd38: a(n) Context
    0x7ffeefbf4f18 M Context>copyTo: 0x14039ba58: a(n) Context
    0x7ffeefbf07c8 M Context>copyTo: 0x14039b8e8: a(n) Context
    0x7ffeefbf0810 M Context>copyTo: 0x140167578: a(n) Context
    0x7ffeefbf0858 M Context>copyTo: 0x140167478: a(n) Context
    0x7ffeefbf08a0 M Context>copyTo: 0x14039b608: a(n) Context
    0x7ffeefbf08e8 M Context>copyTo: 0x14039b498: a(n) Context
    0x7ffeefbf0930 M Context>copyTo: 0x14039b328: a(n) Context
    0x7ffeefbf0978 M Context>copyTo: 0x140167310: a(n) Context
    0x7ffeefbf09c0 M Context>copyTo: 0x1401671e8: a(n) Context
    0x7ffeefbf0a08 M Context>copyTo: 0x140167108: a(n) Context
    0x7ffeefbf0a50 M Context>copyTo: 0x14039af90: a(n) Context
    0x7ffeefbf0a98 M Context>copyTo: 0x14039ae20: a(n) Context
    0x7ffeefbf0ae0 M Context>copyTo: 0x140166fe0: a(n) Context
    0x7ffeefbf0b28 M Context>copyTo: 0x140166ec0: a(n) Context
    0x7ffeefbf0b70 M Context>copyTo: 0x14039ab40: a(n) Context
    0x7ffeefbf0bb8 M Context>copyTo: 0x14039a9d0: a(n) Context
    0x7ffeefbf0c00 M Context>copyTo: 0x14039a860: a(n) Context
    0x7ffeefbf0c48 M Context>copyTo: 0x14039a6f0: a(n) Context
    0x7ffeefbf0c90 M Context>copyTo: 0x140166ca0: a(n) Context
    0x7ffeefbf0cd8 M Context>copyTo: 0x140166bb8: a(n) Context
    0x7ffeefbf0d20 M Context>copyTo: 0x140165f60: a(n) Context
    0x7ffeefbf0d68 M Context>copyTo: 0x140165ea8: a(n) Context
    0x7ffeefbf0db0 M Context>copyTo: 0x14039a2a0: a(n) Context
    0x7ffeefbf0df8 M Context>copyTo: 0x140165d48: a(n) Context
    0x7ffeefbf0e40 M Context>copyTo: 0x140165c20: a(n) Context
    0x7ffeefbf0e88 M Context>copyTo: 0x140165b40: a(n) Context
    0x7ffeefbf0ed0 M Context>copyTo: 0x140399e50: a(n) Context
    0x7ffeefbf0f18 M Context>copyTo: 0x140399b70: a(n) Context
    0x7ffeefbee7c8 M Context>copyTo: 0x140165a18: a(n) Context
    0x7ffeefbee810 M Context>copyTo: 0x1401658f8: a(n) Context
    0x7ffeefbee858 M Context>copyTo: 0x140399890: a(n) Context
    0x7ffeefbee8a0 M Context>copyTo: 0x140399720: a(n) Context
    0x7ffeefbee8e8 M Context>copyTo: 0x1403995b0: a(n) Context
    0x7ffeefbee930 M Context>copyTo: 0x140399440: a(n) Context
    0x7ffeefbee978 M Context>copyTo: 0x1403992d0: a(n) Context
    0x7ffeefbee9c0 M Context>copyTo: 0x140399160: a(n) Context
    0x7ffeefbeea08 M Context>copyTo: 0x140398ff0: a(n) Context
    0x7ffeefbeea50 M Context>copyTo: 0x140398e80: a(n) Context
    0x7ffeefbeea98 M Context>copyTo: 0x140398d10: a(n) Context
    0x7ffeefbeeae0 M Context>copyTo: 0x140165748: a(n) Context
    0x7ffeefbeeb28 M Context>copyTo: 0x140165648: a(n) Context
    0x7ffeefbeeb70 M Context>copyTo: 0x140398a30: a(n) Context
    0x7ffeefbeebb8 M Context>copyTo: 0x1403988c0: a(n) Context
    0x7ffeefbeec00 M Context>copyTo: 0x140165530: a(n) Context
    0x7ffeefbeec48 M Context>copyTo: 0x140165458: a(n) Context
    0x7ffeefbeec90 M Context>copyTo: 0x1403985e0: a(n) Context
    0x7ffeefbeecd8 M Context>copyTo: 0x140398470: a(n) Context
    0x7ffeefbeed20 M Context>copyTo: 0x140165088: a(n) Context
    0x7ffeefbeed68 M Context>copyTo: 0x140166ae0: a(n) Context
    0x7ffeefbeedb0 M Context>copyTo: 0x140398190: a(n) Context
    0x7ffeefbeedf8 M Context>copyTo: 0x140398020: a(n) Context
    0x7ffeefbeee40 M Context>copyTo: 0x140397eb0: a(n) Context
    0x7ffeefbeee88 M Context>copyTo: 0x1401669d8: a(n) Context
    0x7ffeefbeeed0 M Context>copyTo: 0x140397c88: a(n) Context
    0x7ffeefbeef18 M Context>copyTo: 0x140394d08: a(n) Context
    0x7ffeefbea798 M Context>copyTo: 0x140394e20: a(n) Context
    0x7ffeefbea7e0 M Context>copyTo: 0x140397838: a(n) Context
    0x7ffeefbea828 M Context>copyTo: 0x1403976c8: a(n) Context
    0x7ffeefbea870 M Context>copyTo: 0x140397558: a(n) Context
    0x7ffeefbea8b8 M Context>copyTo: 0x140395000: a(n) Context
    0x7ffeefbea900 M Context>copyTo: 0x140397330: a(n) Context
    0x7ffeefbea948 M Context>copyTo: 0x1403971c0: a(n) Context
    0x7ffeefbea990 M Context>copyTo: 0x140397050: a(n) Context
    0x7ffeefbea9d8 M Context>copyTo: 0x140396ee0: a(n) Context
    0x7ffeefbeaa20 M Context>copyTo: 0x140396d70: a(n) Context
    0x7ffeefbeaa68 M Context>copyTo: 0x1403958f8: a(n) Context
    0x7ffeefbeaab0 M Context>copyTo: 0x140396b48: a(n) Context
    0x7ffeefbeaaf8 M Context>copyTo: 0x140396298: a(n) Context
    0x7ffeefbeab40 M Context>copyTo: 0x140396920: a(n) Context
    0x7ffeefbeab88 M Context>copyTo: 0x1403967b0: a(n) Context
    0x7ffeefbeabd0 M Context>copyTo: 0x1403961e0: a(n) Context
    0x7ffeefbeac18 M Context>copyTo: 0x140396128: a(n) Context
    0x7ffeefbeac60 M Context>copyTo: 0x140396070: a(n) Context
    0x7ffeefbeacb8 I Context>copyTo: 0x140395e28: a(n) Context
    0x7ffeefbead00 I ZeroDivide(Exception)>freezeUpTo: 0x140395de8: a(n) ZeroDivide
    0x7ffeefbead48 I ZeroDivide(Exception)>freeze 0x140395de8: a(n) ZeroDivide
    0x7ffeefbead88 I BrDebugglableElementStencil>freeze: 0x140396408: a(n) BrDebugglableElementStencil
    0x7ffeefbeadd8 I ZeroDivide(Exception)>asDebuggableElement 0x140395de8: a(n) ZeroDivide
    0x7ffeefbeae18 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbeae50 M BlockClosure>cull: 0x1403959f0: a(n) BlockClosure
    0x7ffeefbeaea0 I Context>evaluateSignal: 0x140396298: a(n) Context
    0x7ffeefbeaed8 M Context>handleSignal: 0x140396298: a(n) Context
    0x7ffeefbeaf20 I ZeroDivide(Exception)>signal 0x140395de8: a(n) ZeroDivide
    0x7ffeefbe7840 I ZeroDivide(Exception)>signal: 0x140395de8: a(n) ZeroDivide
    0x7ffeefbe7888 I ZeroDivide class(Exception class)>signal: 0x1409f9490: a(n) ZeroDivide class
    0x7ffeefbe78c0 M [] in GtPhlowColumnedListViewExamples>gtErrorInColumnDoDataBinderFor: 0x140168630: a(n) GtPhlowColumnedListViewExamples
    0x7ffeefbe7908 M BlockClosure>glamourValueWithArgs: 0x140174180: a(n) BlockClosure
    0x7ffeefbe7960 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7990 M BlockClosure>on:do: 0x1403959b0: a(n) BlockClosure
    0x7ffeefbe79d0 M GtPhlowColumn>performBlock:onException: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7a18 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7a60 M BlockClosure>glamourValueWithArgs: 0x1401701c0: a(n) BlockClosure
    0x7ffeefbe7a98 M BrStencilValuableExecutor>execute 0x140174230: a(n) BrStencilValuableExecutor
    0x7ffeefbe7ad8 M BrColumnCellDataBinder(BrStencilBuilder)>build 0x140170448: a(n) BrColumnCellDataBinder
    0x7ffeefbe7b38 I [] in BrColumnedListDataSource>onBindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7b98 I OrderedCollection(SequenceableCollection)>with:do: 0x140169ff8: a(n) OrderedCollection
    0x7ffeefbe7bf8 I BrColumnedListDataSource>onBindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7c48 I BrColumnedListDataSource(BlInfiniteDataSource)>onBindHolder:at:payloads: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7ca0 M [] in BrColumnedListDataSource(BlInfiniteDataSource)>bindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7ce0 M BlockClosure>ensure: 0x140394df0: a(n) BlockClosure
    0x7ffeefbe7d18 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe7d58 M BrColumnedListDataSource(BlInfiniteDataSource)>bindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7d98 M BlInfiniteRecyclerController>bindHolder:at: 0x1401662c0: a(n) BlInfiniteRecyclerController
    0x7ffeefbe7e10 M BlInfiniteRecycler>elementFor:dryRun: 0x1401652b8: a(n) BlInfiniteRecycler
    0x7ffeefbe7e50 M BlInfiniteRecycler>elementFor: 0x1401652b8: a(n) BlInfiniteRecycler
    0x7ffeefbe7e98 M [] in BlInfiniteLinearLayoutState>nextElement:in: 0x140165020: a(n) BlInfiniteLinearLayoutState
    0x7ffeefbe7ed8 M BlockClosure>ensure: 0x140166a90: a(n) BlockClosure
    0x7ffeefbe7f10 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe27d0 M BlInfiniteLinearLayoutState>nextElement:in: 0x140165020: a(n) BlInfiniteLinearLayoutState
    0x7ffeefbe2818 M [] in BlInfiniteLinearLayout>layoutChunkAdd 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2858 M BlockClosure>ensure: 0x140165410: a(n) BlockClosure
    0x7ffeefbe2890 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe28d8 M BlInfiniteLinearLayout>layoutChunkAdd 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2910 M [] in BlInfiniteLinearLayout>layoutChunk 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2950 M BlockClosure>ensure: 0x140165600: a(n) BlockClosure
    0x7ffeefbe2988 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe29e0 M BlInfiniteLinearLayout>layoutChunk 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2a48 I BlInfiniteLinearLayout>fillLayout: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2ab8 I BlInfiniteLinearLayout>layoutChildrenFillFromStart: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2b00 I BlInfiniteLinearLayout>layoutChildrenFill: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2b58 I BlInfiniteLinearLayout>layoutChildrenIn:state: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2ba8 I BrInfiniteListElement(BlInfiniteElement)>dispatchLayoutSecondStep 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2be8 I BrInfiniteListElement(BlInfiniteElement)>dispatchLayout 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c18 M BrInfiniteListElement(BlInfiniteElement)>onLayout: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c58 M [] in BrInfiniteListElement(BlElement)>applyLayoutSafelyIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c98 M BlockClosure>ensure: 0x1401658b0: a(n) BlockClosure
    0x7ffeefbe2cd0 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe2d30 M BrInfiniteListElement(BlElement)>applyLayoutSafelyIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2d70 M [] in BrInfiniteListElement(BlElement)>applyLayoutIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2da0 M BlockClosure>on:do: 0x140165ad0: a(n) BlockClosure
    0x7ffeefbe2de0 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x140165b28: a(n) BlCompositeErrorHandler
    0x7ffeefbe2e28 M BrInfiniteListElement(BlElement)>applyLayoutIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2e80 M [] in BlLinearLayoutVerticalOrientation>layout:in: 0x140165d38: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe2ed0 M [] in BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe2f18 M Array(SequenceableCollection)>do: 0x140165e98: a(n) Array
    0x7ffeefbe9868 M BlChildrenAccountedByLayout(BlChildrenSubset)>do: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe98a8 M BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9918 M BlLinearLayoutVerticalOrientation>layout:in: 0x140165d38: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9958 M BlLinearLayout>layout:in: 0x140166e18: a(n) BlLinearLayout
    0x7ffeefbe9998 M BrColumnedList(BlElement)>onLayout: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe99d8 M [] in BrColumnedList(BlElement)>applyLayoutSafelyIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9a18 M BlockClosure>ensure: 0x140166e78: a(n) BlockClosure
    0x7ffeefbe9a50 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe9ab0 M BrColumnedList(BlElement)>applyLayoutSafelyIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9af0 M [] in BrColumnedList(BlElement)>applyLayoutIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9b20 M BlockClosure>on:do: 0x140167098: a(n) BlockClosure
    0x7ffeefbe9b60 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x1401670f0: a(n) BlCompositeErrorHandler
    0x7ffeefbe9ba8 M BrColumnedList(BlElement)>applyLayoutIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9c00 M [] in BlLinearLayoutVerticalOrientation>layout:in: 0x140167300: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9c50 M [] in BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9c98 M Array(SequenceableCollection)>do: 0x140167460: a(n) Array
    0x7ffeefbe9cd0 M BlChildrenAccountedByLayout(BlChildrenSubset)>do: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9d10 M BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9d80 M BlLinearLayoutVerticalOrientation>layout:in: 0x140167300: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9dc0 M BlLinearLayout>layout:in: 0x1401676e0: a(n) BlLinearLayout
    0x7ffeefbe9e00 M BlElement>onLayout: 0x140167648: a(n) BlElement
    0x7ffeefbe9e40 M [] in BlElement>applyLayoutSafelyIn: 0x140167648: a(n) BlElement
    0x7ffeefbe9e80 M BlockClosure>ensure: 0x140167740: a(n) BlockClosure
    0x7ffeefbe9eb8 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe9f18 M BlElement>applyLayoutSafelyIn: 0x140167648: a(n) BlElement
    0x7ffeefbed7e0 M [] in BlElement>applyLayoutIn: 0x140167648: a(n) BlElement
    0x7ffeefbed810 M BlockClosure>on:do: 0x140168368: a(n) BlockClosure
    0x7ffeefbed850 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x1401683c0: a(n) BlCompositeErrorHandler
    0x7ffeefbed898 M BlElement>applyLayoutIn: 0x140167648: a(n) BlElement
    0x7ffeefbed908 I BlElement>computeLayout 0x140167648: a(n) BlElement
    0x7ffeefbed948 I BlElement>forceLayout 0x140167648: a(n) BlElement
    0x7ffeefbed9a8 I GtPhlowColumnedListViewExamples>errorInColumnDoDataBinder 0x140168630: a(n) GtPhlowColumnedListViewExamples
    0x7ffeefbed9d8 M GtExampleDebugger(GtExampleEvaluator)>primitiveProcessExample:withEvaluationContext: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbeda38 M [] in GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbeda78 M BlockClosure>ensure: 0x1401688c8: a(n) BlockClosure
    0x7ffeefbedac8 M GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb00 M GtExampleDebugger(GtExampleProcessor)>process: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb38 M [] in GtExampleDebugger(GtExampleProcessor)>value 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb68 M BlockClosure>on:do: 0x140168c38: a(n) BlockClosure
    0x7ffeefbedba8 M GtCurrentExampleContext class>use:during: 0x1446188f0: a(n) GtCurrentExampleContext class
    0x7ffeefbedbe8 M GtExampleDebugger(GtExampleProcessor)>withContextDo: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc20 M GtExampleDebugger(GtExampleProcessor)>value 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc50 M [] in GtExampleDebugger(GtExampleEvaluator)>result 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc80 M GtExampleDebugger>do:on:do: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedcc8 M GtExampleDebugger(GtExampleEvaluator)>result 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedcf8 M GtExample>debug 0x19709dfc8: a(n) GtExample
    0x7ffeefbedd30 M [] in GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedd60 M BlockClosure>on:do: 0x140169368: a(n) BlockClosure
    0x7ffeefbeddb0 M [] in GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedde8 M Time class>microsecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbede20 M Time class>millisecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbede60 M BlockClosure>timeToRun 0x140169558: a(n) BlockClosure
    0x7ffeefbede98 M GtExamplesHDReport>beginExample:runBlock: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedee0 M GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedf18 M [] in GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdea38 M OrderedCollection>do: 0x197099e08: a(n) OrderedCollection
    0x7ffeefbdea70 M [] in GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdeab0 M [] in CurrentExecutionEnvironment class>activate:for: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdeaf0 M BlockClosure>ensure: 0x19709a0b0: a(n) BlockClosure
    0x7ffeefbdeb30 M CurrentExecutionEnvironment class>activate:for: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdeb70 M TestExecutionEnvironment(ExecutionEnvironment)>beActiveDuring: 0x197099fb8: a(n) TestExecutionEnvironment
    0x7ffeefbdeba8 M DefaultExecutionEnvironment>runTestsBy: 0x1409f7da8: a(n) DefaultExecutionEnvironment
    0x7ffeefbdebe0 M CurrentExecutionEnvironment class>runTestsBy: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdec18 M GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdec48 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdec80 M Time class>microsecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbdecb8 M Time class>millisecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbdecf8 M BlockClosure>timeToRun 0x19709a618: a(n) BlockClosure
    0x7ffeefbded28 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbded68 M BlockClosure>ensure: 0x19709a918: a(n) BlockClosure
    0x7ffeefbdeda0 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdedd0 M Author>ifUnknownAuthorUse:during: 0x14102a740: a(n) Author
    0x7ffeefbdee10 M GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdee48 M GtExamplesHDReport class>runPackage: 0x144618008: a(n) GtExamplesHDReport class
    0x7ffeefbdee80 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x144618008: a(n) GtExamplesHDReport class
    0x7ffeefbdeed0 M [] in Set>collect: 0x145bbc4d8: a(n) Set
    0x7ffeefbdef18 M Array(SequenceableCollection)>do: 0x145bbc520: a(n) Array
       0x145bbcc78 s Set>collect:
       0x145bbcd30 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x145bbce28 s [] in GtExamplesCommandLineHandler>runPackages
       0x145bbcf08 s BlockClosure>ensure:
       0x14e950710 s UIManager class>nonInteractiveDuring:
       0x14e9507c8 s GtExamplesCommandLineHandler>runPackages
       0x14e950880 s GtExamplesCommandLineHandler>activate
       0x14e950938 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x14e9509f0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x14e950aa8 s BlockClosure>on:do:
       0x14e950b60 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x14e950c38 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x14e950cf0 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x14e950db8 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x14e950e90 s BlockClosure>on:do:
       0x14e950f68 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x14e951040 s [] in BlockClosure>newProcess



Process 18798 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001481107d1
->  0x1481107d1: int3   
    0x1481107d2: int3   
    0x1481107d3: int3   
    0x1481107d4: int3   
Target 0: (Pharo) stopped.

(lldb) call printAllStacks()
Process        0x15511b938 priority 40
    0x7ffeefbf0a18 M Context(Object)>copy 0x15691cf68: a(n) Context
    0x7ffeefbf0a50 M Context>copyTo: 0x15691cf68: a(n) Context
    0x7ffeefbf0a98 M Context>copyTo: 0x15691ce78: a(n) Context
    0x7ffeefbf0ae0 M Context>copyTo: 0x15691cdc0: a(n) Context
    0x7ffeefbf0b28 M Context>copyTo: 0x15691cd08: a(n) Context
    0x7ffeefbf0b70 M Context>copyTo: 0x15691cc50: a(n) Context
    0x7ffeefbf0bb8 M Context>copyTo: 0x15691cb70: a(n) Context
    0x7ffeefbf0c00 M Context>copyTo: 0x15691ca90: a(n) Context
    0x7ffeefbf0c48 M Context>copyTo: 0x15691c998: a(n) Context
    0x7ffeefbf0c90 M Context>copyTo: 0x15691c8e0: a(n) Context
    0x7ffeefbf0cd8 M Context>copyTo: 0x15691c828: a(n) Context
    0x7ffeefbf0d20 M Context>copyTo: 0x14874a2c8: a(n) Context
    0x7ffeefbf0d68 M Context>copyTo: 0x14874a158: a(n) Context
    0x7ffeefbf0db0 M Context>copyTo: 0x148749fe8: a(n) Context
    0x7ffeefbf0df8 M Context>copyTo: 0x14816e2e8: a(n) Context
    0x7ffeefbf0e40 M Context>copyTo: 0x148749dc0: a(n) Context
    0x7ffeefbf0e88 M Context>copyTo: 0x14816e210: a(n) Context
    0x7ffeefbf0ed0 M Context>copyTo: 0x148749ae0: a(n) Context
    0x7ffeefbf0f18 M Context>copyTo: 0x14816e118: a(n) Context
    0x7ffeefbdc7c8 M Context>copyTo: 0x148749748: a(n) Context
    0x7ffeefbdc810 M Context>copyTo: 0x1487495d8: a(n) Context
    0x7ffeefbdc858 M Context>copyTo: 0x148749468: a(n) Context
    0x7ffeefbdc8a0 M Context>copyTo: 0x1487492f8: a(n) Context
    0x7ffeefbdc8e8 M Context>copyTo: 0x14816e040: a(n) Context
    0x7ffeefbdc930 M Context>copyTo: 0x1487490d0: a(n) Context
    0x7ffeefbdc978 M Context>copyTo: 0x148748f60: a(n) Context
    0x7ffeefbdc9c0 M Context>copyTo: 0x148748df0: a(n) Context
    0x7ffeefbdca08 M Context>copyTo: 0x14816df38: a(n) Context
    0x7ffeefbdca50 M Context>copyTo: 0x148748bc8: a(n) Context
    0x7ffeefbdca98 M Context>copyTo: 0x14816de20: a(n) Context
    0x7ffeefbdcae0 M Context>copyTo: 0x14816eeb8: a(n) Context
    0x7ffeefbdcb28 M Context>copyTo: 0x1487488e8: a(n) Context
    0x7ffeefbdcb70 M Context>copyTo: 0x148748778: a(n) Context
    0x7ffeefbdcbb8 M Context>copyTo: 0x14866e858: a(n) Context
    0x7ffeefbdcc00 M Context>copyTo: 0x148748550: a(n) Context
    0x7ffeefbdcc48 M Context>copyTo: 0x1487483e0: a(n) Context
    0x7ffeefbdcc90 M Context>copyTo: 0x148748270: a(n) Context
    0x7ffeefbdccd8 M Context>copyTo: 0x148748100: a(n) Context
    0x7ffeefbdcd20 M Context>copyTo: 0x148676078: a(n) Context
    0x7ffeefbdcd68 M Context>copyTo: 0x148747ed8: a(n) Context
    0x7ffeefbdcdb0 M Context>copyTo: 0x148747d68: a(n) Context
    0x7ffeefbdcdf8 M Context>copyTo: 0x148747bf8: a(n) Context
    0x7ffeefbdce40 M Context>copyTo: 0x148676280: a(n) Context
    0x7ffeefbdce88 M Context>copyTo: 0x1487479d0: a(n) Context
    0x7ffeefbdced0 M Context>copyTo: 0x1487477a8: a(n) Context
    0x7ffeefbdcf18 M Context>copyTo: 0x148676400: a(n) Context
    0x7ffeefbda7c8 M Context>copyTo: 0x148747410: a(n) Context
    0x7ffeefbda810 M Context>copyTo: 0x1486764d8: a(n) Context
    0x7ffeefbda858 M Context>copyTo: 0x1487471e8: a(n) Context
    0x7ffeefbda8a0 M Context>copyTo: 0x148747078: a(n) Context
    0x7ffeefbda8e8 M Context>copyTo: 0x148746f08: a(n) Context
    0x7ffeefbda930 M Context>copyTo: 0x14873f918: a(n) Context
    0x7ffeefbda978 M Context>copyTo: 0x148746ce0: a(n) Context
    0x7ffeefbda9c0 M Context>copyTo: 0x148746b70: a(n) Context
    0x7ffeefbdaa08 M Context>copyTo: 0x148746a00: a(n) Context
    0x7ffeefbdaa50 M Context>copyTo: 0x148746890: a(n) Context
    0x7ffeefbdaa98 M Context>copyTo: 0x148746720: a(n) Context
    0x7ffeefbdaae0 M Context>copyTo: 0x1487415b8: a(n) Context
    0x7ffeefbdab28 M Context>copyTo: 0x1487412a8: a(n) Context
    0x7ffeefbdab70 M Context>copyTo: 0x148746440: a(n) Context
    0x7ffeefbdabb8 M Context>copyTo: 0x148745088: a(n) Context
    0x7ffeefbdac00 M Context>copyTo: 0x148746218: a(n) Context
    0x7ffeefbdac48 M Context>copyTo: 0x1487414e0: a(n) Context
    0x7ffeefbdac90 M Context>copyTo: 0x148745ff0: a(n) Context
    0x7ffeefbdacd8 M Context>copyTo: 0x148741670: a(n) Context
    0x7ffeefbdad20 M Context>copyTo: 0x148744fd0: a(n) Context
    0x7ffeefbdad68 M Context>copyTo: 0x148745ba0: a(n) Context
    0x7ffeefbdadb0 M Context>copyTo: 0x148745a30: a(n) Context
    0x7ffeefbdadf8 M Context>copyTo: 0x148744bd8: a(n) Context
    0x7ffeefbdae40 M Context>copyTo: 0x148745808: a(n) Context
    0x7ffeefbdae88 M Context>copyTo: 0x148745698: a(n) Context
    0x7ffeefbdaed0 M Context>copyTo: 0x148745528: a(n) Context
    0x7ffeefbdaf18 M Context>copyTo: 0x148744f18: a(n) Context
    0x7ffeefbe0988 I Context>copyTo: 0x148744e60: a(n) Context
    0x7ffeefbe09d0 I MessageNotUnderstood(Exception)>freezeUpTo: 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0a18 I MessageNotUnderstood(Exception)>freeze 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0a48 M [] in GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0a80 M BlockClosure>cull: 0x1487414c0: a(n) BlockClosure
    0x7ffeefbe0ac0 M Context>evaluateSignal: 0x148745088: a(n) Context
    0x7ffeefbe0af8 M Context>handleSignal: 0x148745088: a(n) Context
    0x7ffeefbe0b30 M Context>handleSignal: 0x148744fd0: a(n) Context
    0x7ffeefbe0b68 M MessageNotUnderstood(Exception)>signal 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0bb8 I GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x148744bc8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefbe0bf0 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0c50 M [] in GtExampleEvaluator>basicProcess: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0c90 M BlockClosure>ensure: 0x148744c90: a(n) BlockClosure
    0x7ffeefbe0ce0 M GtExampleEvaluator>basicProcess: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d18 M GtExampleEvaluator(GtExampleProcessor)>process: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d50 M [] in GtExampleEvaluator(GtExampleProcessor)>value 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d80 M BlockClosure>on:do: 0x148741598: a(n) BlockClosure
    0x7ffeefbe0dc0 M GtCurrentExampleContext class>use:during: 0x14c618920: a(n) GtCurrentExampleContext class
    0x7ffeefbe0e00 M GtExampleEvaluator(GtExampleProcessor)>withContextDo: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e38 M GtExampleEvaluator(GtExampleProcessor)>value 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e68 M [] in GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e98 M BlockClosure>on:do: 0x148741360: a(n) BlockClosure
    0x7ffeefbe0ed8 M GtExampleEvaluator>do:on:do: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0f20 M GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbde8b8 M GtExample>run 0x148740050: a(n) GtExample
    0x7ffeefbde8e8 M GtExample>result 0x148740050: a(n) GtExample
    0x7ffeefbde930 I GtExamplesExamplesWithInheritance>resultOfInvalidExampleWithInvalidSuperclassProvider 0x14873f908: a(n) GtExamplesExamplesWithInheritance
    0x7ffeefbde960 M GtExampleDebugger(GtExampleEvaluator)>primitiveProcessExample:withEvaluationContext: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbde9c0 M [] in GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdea00 M BlockClosure>ensure: 0x14873f9d0: a(n) BlockClosure
    0x7ffeefbdea50 M GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdea88 M GtExampleDebugger(GtExampleProcessor)>process: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeac0 M [] in GtExampleDebugger(GtExampleProcessor)>value 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeaf0 M BlockClosure>on:do: 0x1486764b8: a(n) BlockClosure
    0x7ffeefbdeb30 M GtCurrentExampleContext class>use:during: 0x14c618920: a(n) GtCurrentExampleContext class
    0x7ffeefbdeb70 M GtExampleDebugger(GtExampleProcessor)>withContextDo: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeba8 M GtExampleDebugger(GtExampleProcessor)>value 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdebd8 M [] in GtExampleDebugger(GtExampleEvaluator)>result 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec08 M GtExampleDebugger>do:on:do: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec50 M GtExampleDebugger(GtExampleEvaluator)>result 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec80 M GtExample>debug 0x148188088: a(n) GtExample
    0x7ffeefbdecb8 M [] in GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdece8 M BlockClosure>on:do: 0x148676130: a(n) BlockClosure
    0x7ffeefbded38 M [] in GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbded70 M Time class>microsecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbdeda8 M Time class>millisecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbdede8 M BlockClosure>timeToRun 0x14866e910: a(n) BlockClosure
    0x7ffeefbdee20 M GtExamplesHDReport>beginExample:runBlock: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdee68 M GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdeea0 M [] in GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdeee8 M OrderedCollection>do: 0x14816ee98: a(n) OrderedCollection
    0x7ffeefbdef20 M [] in GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8ab0 M [] in CurrentExecutionEnvironment class>activate:for: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8af0 M BlockClosure>ensure: 0x14816ded8: a(n) BlockClosure
    0x7ffeefbd8b30 M CurrentExecutionEnvironment class>activate:for: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8b70 M TestExecutionEnvironment(ExecutionEnvironment)>beActiveDuring: 0x14816ddc0: a(n) TestExecutionEnvironment
    0x7ffeefbd8ba8 M DefaultExecutionEnvironment>runTestsBy: 0x1489f7da8: a(n) DefaultExecutionEnvironment
    0x7ffeefbd8be0 M CurrentExecutionEnvironment class>runTestsBy: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8c18 M GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8c48 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8c80 M Time class>microsecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbd8cb8 M Time class>millisecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbd8cf8 M BlockClosure>timeToRun 0x14816e0f8: a(n) BlockClosure
    0x7ffeefbd8d28 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8d68 M BlockClosure>ensure: 0x14816e1d0: a(n) BlockClosure
    0x7ffeefbd8da0 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8dd0 M Author>ifUnknownAuthorUse:during: 0x14902a740: a(n) Author
    0x7ffeefbd8e10 M GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8e48 M GtExamplesHDReport class>runPackage: 0x14c618038: a(n) GtExamplesHDReport class
    0x7ffeefbd8e80 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x14c618038: a(n) GtExamplesHDReport class
    0x7ffeefbd8ed0 M [] in Set>collect: 0x15691c088: a(n) Set
    0x7ffeefbd8f18 M Array(SequenceableCollection)>do: 0x15691c188: a(n) Array
       0x15691c8e0 s Set>collect:
       0x15691c998 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x15691ca90 s [] in GtExamplesCommandLineHandler>runPackages
       0x15691cb70 s BlockClosure>ensure:
       0x15691cc50 s UIManager class>nonInteractiveDuring:
       0x15691cd08 s GtExamplesCommandLineHandler>runPackages
       0x15691cdc0 s GtExamplesCommandLineHandler>activate
       0x15691ce78 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x15691cf68 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x15691d048 s BlockClosure>on:do:
       0x15691d128 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x15691d200 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x15691d2b8 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x15691d380 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x15691d458 s BlockClosure>on:do:
       0x15691d530 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x15691d608 s [] in BlockClosure>newProcess


(lldb) call dumpPrimTraceLog()
stringHash:initialHash:
compare:with:collated:
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
primitiveChangeClassTo:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
primitiveChangeClassTo:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
stringHash:initialHash:
compare:with:collated:
primitiveChangeClassTo:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
basicNew:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
size
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
**StackOverflow**
perform:withArguments:
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
basicNew
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
**StackOverflow**
**StackOverflow**
class
perform:withArguments:
value:
first
basicNew
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
perform:withArguments:
basicNew
findNextHandlerOrSignalingContext
tempAt:
class
findNextHandlerOrSignalingContext
tempAt:
tempAt:
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**CompactCode**



On 26 Sep 2020, at 23:24, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

   fixed in commit 561b06530bbaed5f19e9d7f077a7df9eb3a8d236, VMMaker.oscog-eem.2824


On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption? 


 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot

Reply | Threaded
Open this post in threaded view
|

Re: corruption of PC in context objects or not (?)

Eliot Miranda-2
 
Hi Andrei,

On Sun, Sep 27, 2020 at 1:01 PM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

I get the following details for the VM that I compiled locally:

CoInterpreter VMMaker.oscog-eem.2824 uuid: 8f091e5b-fc0f-4b4b-ab5e-e90e598f75ee Sep 27 2020
StackToRegisterMappingCogit VMMaker.oscog-eem.2824 uuid: 8f091e5b-fc0f-4b4b-ab5e-e90e598f75ee Sep 27 2020
VM: 202009270258 [hidden email]:Documents/Pharo/vm-compilation/opensmalltalk-vm Date: Sat Sep 26 19:58:41 2020 CommitHash: 5a0d21d71 Plugins: 202009270258 [hidden email]:Documents/Pharo/vm-compilation/opensmalltalk-vm

Mac OS X built on Sep 27 2020 14:27:59 CEST Compiler: 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.32.62)
VMMaker versionString VM: 202009270258 [hidden email]:Documents/Pharo/vm-compilation/opensmalltalk-vm Date: Sat Sep 26 19:58:41 2020 CommitHash: 5a0d21d71 Plugins: 202009270258 [hidden email]:Documents/Pharo/vm-compilation/opensmalltalk-vm
CoInterpreter VMMaker.oscog-eem.2824 uuid: 8f091e5b-fc0f-4b4b-ab5e-e90e598f75ee Sep 27 2020
StackToRegisterMappingCogit VMMaker.oscog-eem.2824 uuid: 8f091e5b-fc0f-4b4b-ab5e-e90e598f75ee Sep 27 2020


I can package together the image/changes file and libs and send them privately. 

Good, let's do that. Thanks.

Cheers,
Andrei

On 27 Sep 2020, at 21:23, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

On Sun, Sep 27, 2020 at 10:29 AM Andrei Chis <[hidden email]> wrote:
 
Hi Eliot,

Thanks a lot for your help.

I compiled locally a vm from the commit hash 5a0d21d71223b4d1f4d007ebf2441d530c18c3e2 (CogVM source as per VMMaker.oscog-eem.2826) and I got a Context>>copyTo: crash.
I think I compiled the vm correctly. Are them some Pharo VMs built on travis for Mac to try out instead of building locally?

There may be.  The PharoVM team are not maintaining our CI builds so you may not be able to find a Pharo VM.  


Below are the stack traces and primitive longs for two crashes with the compiled vm.

OK, that's disappointing, but important.  Maybe you can put an image/changes, support dylibs package up somewhere (Dropbox?) and I can try and run the test case locally. I tested the fix for some time and it clearly fixed one bug.  But it does seem that there's another in a similar area.  Thanks for your patience.

One question, you're sure you're running the new VM?


Process 19347 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x0000000140103f79
->  0x140103f79: int3   
    0x140103f7a: int3   
    0x140103f7b: int3   
    0x140103f7c: int3   
Target 0: (Pharo) stopped.

(lldb) call dumpPrimTraceLog()
basicNew
**StackOverflow**
basicNew
@
@
@
@
basicNew
basicNew
@
@
@
@
@
@
signal
signal
class
basicIdentityHash
basicNew
class
bitShift:
bitAnd:
asFloat
/
asFloat
fractionPart
truncated
fractionPart
truncated
fractionPart
truncated
setAlpha:
truncated
basicIdentityHash
basicNew
basicNew
class
class
basicNew
basicNew
basicIdentityHash
basicNew
basicNew
basicNew
wait
class
tempAt:
tempAt:put:
tempAt:
terminateTo:
signal
findNextUnwindContextUpTo:
terminateTo:
wait
**StackOverflow**
signal
replaceFrom:to:with:startingAt:
wait
**StackOverflow**
class
signal
basicNew
class
**StackOverflow**
**StackOverflow**
**StackOverflow**
wait
signal
basicIdentityHash
class
basicNew
basicNew
**StackOverflow**
class
class
**StackOverflow**
wait
signal
**StackOverflow**
**StackOverflow**
wait
signal
basicNew
basicNew
basicNew
signal
wait
signal
basicIdentityHash
basicIdentityHash
basicNew
basicNew
class
value:value:
basicNew
replaceFrom:to:with:startingAt:
valueWithArguments:
**StackOverflow**
basicIdentityHash
valueWithArguments:
basicNew
**StackOverflow**
class
findNextHandlerOrSignalingContext
**StackOverflow**
primitive
~=
~=
tempAt:
**StackOverflow**
tempAt:
basicNew
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy



(lldb) call printCallStack()
    0x7ffeefbe6d78 M Context(Object)>copy 0x145bbce28: a(n) Context
    0x7ffeefbe6db0 M Context>copyTo: 0x145bbce28: a(n) Context
    0x7ffeefbe6df8 M Context>copyTo: 0x145bbcd30: a(n) Context
    0x7ffeefbe6e40 M Context>copyTo: 0x145bbcc78: a(n) Context
    0x7ffeefbe6e88 M Context>copyTo: 0x145bbcbc0: a(n) Context
    0x7ffeefbe6ed0 M Context>copyTo: 0x19709adc8: a(n) Context
    0x7ffeefbe6f18 M Context>copyTo: 0x19709ad10: a(n) Context
    0x7ffeefbe37c8 M Context>copyTo: 0x19709ac58: a(n) Context
    0x7ffeefbe3810 M Context>copyTo: 0x19709aba0: a(n) Context
    0x7ffeefbe3858 M Context>copyTo: 0x19709aae8: a(n) Context
    0x7ffeefbe38a0 M Context>copyTo: 0x19709aa10: a(n) Context
    0x7ffeefbe38e8 M Context>copyTo: 0x19709a938: a(n) Context
    0x7ffeefbe3930 M Context>copyTo: 0x19709a860: a(n) Context
    0x7ffeefbe3978 M Context>copyTo: 0x19709a7a8: a(n) Context
    0x7ffeefbe39c0 M Context>copyTo: 0x19709a6f0: a(n) Context
    0x7ffeefbe3a08 M Context>copyTo: 0x19709a638: a(n) Context
    0x7ffeefbe3a50 M Context>copyTo: 0x19709a560: a(n) Context
    0x7ffeefbe3a98 M Context>copyTo: 0x19709a4a8: a(n) Context
    0x7ffeefbe3ae0 M Context>copyTo: 0x19709a3f0: a(n) Context
    0x7ffeefbe3b28 M Context>copyTo: 0x19709a338: a(n) Context
    0x7ffeefbe3b70 M Context>copyTo: 0x19709a280: a(n) Context
    0x7ffeefbe3bb8 M Context>copyTo: 0x19709a1c8: a(n) Context
    0x7ffeefbe3c00 M Context>copyTo: 0x19709a0e0: a(n) Context
    0x7ffeefbe3c48 M Context>copyTo: 0x197099ff8: a(n) Context
    0x7ffeefbe3c90 M Context>copyTo: 0x197099ee0: a(n) Context
    0x7ffeefbe3cd8 M Context>copyTo: 0x197099e28: a(n) Context
    0x7ffeefbe3d20 M Context>copyTo: 0x140169920: a(n) Context
    0x7ffeefbe3d68 M Context>copyTo: 0x140169868: a(n) Context
    0x7ffeefbe3db0 M Context>copyTo: 0x1401697b0: a(n) Context
    0x7ffeefbe3df8 M Context>copyTo: 0x1401696f8: a(n) Context
    0x7ffeefbe3e40 M Context>copyTo: 0x140169640: a(n) Context
    0x7ffeefbe3e88 M Context>copyTo: 0x140169588: a(n) Context
    0x7ffeefbe3ed0 M Context>copyTo: 0x1401694a0: a(n) Context
    0x7ffeefbe3f18 M Context>copyTo: 0x140169390: a(n) Context
    0x7ffeefbf47c8 M Context>copyTo: 0x1401692b0: a(n) Context
    0x7ffeefbf4810 M Context>copyTo: 0x1401691f8: a(n) Context
    0x7ffeefbf4858 M Context>copyTo: 0x140169140: a(n) Context
    0x7ffeefbf48a0 M Context>copyTo: 0x140169058: a(n) Context
    0x7ffeefbf48e8 M Context>copyTo: 0x140168f80: a(n) Context
    0x7ffeefbf4930 M Context>copyTo: 0x140168ec8: a(n) Context
    0x7ffeefbf4978 M Context>copyTo: 0x140168e10: a(n) Context
    0x7ffeefbf49c0 M Context>copyTo: 0x140168d38: a(n) Context
    0x7ffeefbf4a08 M Context>copyTo: 0x140168c58: a(n) Context
    0x7ffeefbf4a50 M Context>copyTo: 0x140168b80: a(n) Context
    0x7ffeefbf4a98 M Context>copyTo: 0x140168ac8: a(n) Context
    0x7ffeefbf4ae0 M Context>copyTo: 0x140168a10: a(n) Context
    0x7ffeefbf4b28 M Context>copyTo: 0x140168900: a(n) Context
    0x7ffeefbf4b70 M Context>copyTo: 0x140168810: a(n) Context
    0x7ffeefbf4bb8 M Context>copyTo: 0x140168718: a(n) Context
    0x7ffeefbf4c00 M Context>copyTo: 0x140168640: a(n) Context
    0x7ffeefbf4c48 M Context>copyTo: 0x14039c748: a(n) Context
    0x7ffeefbf4c90 M Context>copyTo: 0x14039c5d8: a(n) Context
    0x7ffeefbf4cd8 M Context>copyTo: 0x1401684b8: a(n) Context
    0x7ffeefbf4d20 M Context>copyTo: 0x1401683d8: a(n) Context
    0x7ffeefbf4d68 M Context>copyTo: 0x14039c2f8: a(n) Context
    0x7ffeefbf4db0 M Context>copyTo: 0x140167960: a(n) Context
    0x7ffeefbf4df8 M Context>copyTo: 0x1401678a8: a(n) Context
    0x7ffeefbf4e40 M Context>copyTo: 0x140167788: a(n) Context
    0x7ffeefbf4e88 M Context>copyTo: 0x14039bf60: a(n) Context
    0x7ffeefbf4ed0 M Context>copyTo: 0x14039bd38: a(n) Context
    0x7ffeefbf4f18 M Context>copyTo: 0x14039ba58: a(n) Context
    0x7ffeefbf07c8 M Context>copyTo: 0x14039b8e8: a(n) Context
    0x7ffeefbf0810 M Context>copyTo: 0x140167578: a(n) Context
    0x7ffeefbf0858 M Context>copyTo: 0x140167478: a(n) Context
    0x7ffeefbf08a0 M Context>copyTo: 0x14039b608: a(n) Context
    0x7ffeefbf08e8 M Context>copyTo: 0x14039b498: a(n) Context
    0x7ffeefbf0930 M Context>copyTo: 0x14039b328: a(n) Context
    0x7ffeefbf0978 M Context>copyTo: 0x140167310: a(n) Context
    0x7ffeefbf09c0 M Context>copyTo: 0x1401671e8: a(n) Context
    0x7ffeefbf0a08 M Context>copyTo: 0x140167108: a(n) Context
    0x7ffeefbf0a50 M Context>copyTo: 0x14039af90: a(n) Context
    0x7ffeefbf0a98 M Context>copyTo: 0x14039ae20: a(n) Context
    0x7ffeefbf0ae0 M Context>copyTo: 0x140166fe0: a(n) Context
    0x7ffeefbf0b28 M Context>copyTo: 0x140166ec0: a(n) Context
    0x7ffeefbf0b70 M Context>copyTo: 0x14039ab40: a(n) Context
    0x7ffeefbf0bb8 M Context>copyTo: 0x14039a9d0: a(n) Context
    0x7ffeefbf0c00 M Context>copyTo: 0x14039a860: a(n) Context
    0x7ffeefbf0c48 M Context>copyTo: 0x14039a6f0: a(n) Context
    0x7ffeefbf0c90 M Context>copyTo: 0x140166ca0: a(n) Context
    0x7ffeefbf0cd8 M Context>copyTo: 0x140166bb8: a(n) Context
    0x7ffeefbf0d20 M Context>copyTo: 0x140165f60: a(n) Context
    0x7ffeefbf0d68 M Context>copyTo: 0x140165ea8: a(n) Context
    0x7ffeefbf0db0 M Context>copyTo: 0x14039a2a0: a(n) Context
    0x7ffeefbf0df8 M Context>copyTo: 0x140165d48: a(n) Context
    0x7ffeefbf0e40 M Context>copyTo: 0x140165c20: a(n) Context
    0x7ffeefbf0e88 M Context>copyTo: 0x140165b40: a(n) Context
    0x7ffeefbf0ed0 M Context>copyTo: 0x140399e50: a(n) Context
    0x7ffeefbf0f18 M Context>copyTo: 0x140399b70: a(n) Context
    0x7ffeefbee7c8 M Context>copyTo: 0x140165a18: a(n) Context
    0x7ffeefbee810 M Context>copyTo: 0x1401658f8: a(n) Context
    0x7ffeefbee858 M Context>copyTo: 0x140399890: a(n) Context
    0x7ffeefbee8a0 M Context>copyTo: 0x140399720: a(n) Context
    0x7ffeefbee8e8 M Context>copyTo: 0x1403995b0: a(n) Context
    0x7ffeefbee930 M Context>copyTo: 0x140399440: a(n) Context
    0x7ffeefbee978 M Context>copyTo: 0x1403992d0: a(n) Context
    0x7ffeefbee9c0 M Context>copyTo: 0x140399160: a(n) Context
    0x7ffeefbeea08 M Context>copyTo: 0x140398ff0: a(n) Context
    0x7ffeefbeea50 M Context>copyTo: 0x140398e80: a(n) Context
    0x7ffeefbeea98 M Context>copyTo: 0x140398d10: a(n) Context
    0x7ffeefbeeae0 M Context>copyTo: 0x140165748: a(n) Context
    0x7ffeefbeeb28 M Context>copyTo: 0x140165648: a(n) Context
    0x7ffeefbeeb70 M Context>copyTo: 0x140398a30: a(n) Context
    0x7ffeefbeebb8 M Context>copyTo: 0x1403988c0: a(n) Context
    0x7ffeefbeec00 M Context>copyTo: 0x140165530: a(n) Context
    0x7ffeefbeec48 M Context>copyTo: 0x140165458: a(n) Context
    0x7ffeefbeec90 M Context>copyTo: 0x1403985e0: a(n) Context
    0x7ffeefbeecd8 M Context>copyTo: 0x140398470: a(n) Context
    0x7ffeefbeed20 M Context>copyTo: 0x140165088: a(n) Context
    0x7ffeefbeed68 M Context>copyTo: 0x140166ae0: a(n) Context
    0x7ffeefbeedb0 M Context>copyTo: 0x140398190: a(n) Context
    0x7ffeefbeedf8 M Context>copyTo: 0x140398020: a(n) Context
    0x7ffeefbeee40 M Context>copyTo: 0x140397eb0: a(n) Context
    0x7ffeefbeee88 M Context>copyTo: 0x1401669d8: a(n) Context
    0x7ffeefbeeed0 M Context>copyTo: 0x140397c88: a(n) Context
    0x7ffeefbeef18 M Context>copyTo: 0x140394d08: a(n) Context
    0x7ffeefbea798 M Context>copyTo: 0x140394e20: a(n) Context
    0x7ffeefbea7e0 M Context>copyTo: 0x140397838: a(n) Context
    0x7ffeefbea828 M Context>copyTo: 0x1403976c8: a(n) Context
    0x7ffeefbea870 M Context>copyTo: 0x140397558: a(n) Context
    0x7ffeefbea8b8 M Context>copyTo: 0x140395000: a(n) Context
    0x7ffeefbea900 M Context>copyTo: 0x140397330: a(n) Context
    0x7ffeefbea948 M Context>copyTo: 0x1403971c0: a(n) Context
    0x7ffeefbea990 M Context>copyTo: 0x140397050: a(n) Context
    0x7ffeefbea9d8 M Context>copyTo: 0x140396ee0: a(n) Context
    0x7ffeefbeaa20 M Context>copyTo: 0x140396d70: a(n) Context
    0x7ffeefbeaa68 M Context>copyTo: 0x1403958f8: a(n) Context
    0x7ffeefbeaab0 M Context>copyTo: 0x140396b48: a(n) Context
    0x7ffeefbeaaf8 M Context>copyTo: 0x140396298: a(n) Context
    0x7ffeefbeab40 M Context>copyTo: 0x140396920: a(n) Context
    0x7ffeefbeab88 M Context>copyTo: 0x1403967b0: a(n) Context
    0x7ffeefbeabd0 M Context>copyTo: 0x1403961e0: a(n) Context
    0x7ffeefbeac18 M Context>copyTo: 0x140396128: a(n) Context
    0x7ffeefbeac60 M Context>copyTo: 0x140396070: a(n) Context
    0x7ffeefbeacb8 I Context>copyTo: 0x140395e28: a(n) Context
    0x7ffeefbead00 I ZeroDivide(Exception)>freezeUpTo: 0x140395de8: a(n) ZeroDivide
    0x7ffeefbead48 I ZeroDivide(Exception)>freeze 0x140395de8: a(n) ZeroDivide
    0x7ffeefbead88 I BrDebugglableElementStencil>freeze: 0x140396408: a(n) BrDebugglableElementStencil
    0x7ffeefbeadd8 I ZeroDivide(Exception)>asDebuggableElement 0x140395de8: a(n) ZeroDivide
    0x7ffeefbeae18 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbeae50 M BlockClosure>cull: 0x1403959f0: a(n) BlockClosure
    0x7ffeefbeaea0 I Context>evaluateSignal: 0x140396298: a(n) Context
    0x7ffeefbeaed8 M Context>handleSignal: 0x140396298: a(n) Context
    0x7ffeefbeaf20 I ZeroDivide(Exception)>signal 0x140395de8: a(n) ZeroDivide
    0x7ffeefbe7840 I ZeroDivide(Exception)>signal: 0x140395de8: a(n) ZeroDivide
    0x7ffeefbe7888 I ZeroDivide class(Exception class)>signal: 0x1409f9490: a(n) ZeroDivide class
    0x7ffeefbe78c0 M [] in GtPhlowColumnedListViewExamples>gtErrorInColumnDoDataBinderFor: 0x140168630: a(n) GtPhlowColumnedListViewExamples
    0x7ffeefbe7908 M BlockClosure>glamourValueWithArgs: 0x140174180: a(n) BlockClosure
    0x7ffeefbe7960 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7990 M BlockClosure>on:do: 0x1403959b0: a(n) BlockClosure
    0x7ffeefbe79d0 M GtPhlowColumn>performBlock:onException: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7a18 M [] in GtPhlowColumn>errorTreatedCellDataBinderFor: 0x140169e48: a(n) GtPhlowColumn
    0x7ffeefbe7a60 M BlockClosure>glamourValueWithArgs: 0x1401701c0: a(n) BlockClosure
    0x7ffeefbe7a98 M BrStencilValuableExecutor>execute 0x140174230: a(n) BrStencilValuableExecutor
    0x7ffeefbe7ad8 M BrColumnCellDataBinder(BrStencilBuilder)>build 0x140170448: a(n) BrColumnCellDataBinder
    0x7ffeefbe7b38 I [] in BrColumnedListDataSource>onBindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7b98 I OrderedCollection(SequenceableCollection)>with:do: 0x140169ff8: a(n) OrderedCollection
    0x7ffeefbe7bf8 I BrColumnedListDataSource>onBindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7c48 I BrColumnedListDataSource(BlInfiniteDataSource)>onBindHolder:at:payloads: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7ca0 M [] in BrColumnedListDataSource(BlInfiniteDataSource)>bindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7ce0 M BlockClosure>ensure: 0x140394df0: a(n) BlockClosure
    0x7ffeefbe7d18 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe7d58 M BrColumnedListDataSource(BlInfiniteDataSource)>bindHolder:at: 0x140169f38: a(n) BrColumnedListDataSource
    0x7ffeefbe7d98 M BlInfiniteRecyclerController>bindHolder:at: 0x1401662c0: a(n) BlInfiniteRecyclerController
    0x7ffeefbe7e10 M BlInfiniteRecycler>elementFor:dryRun: 0x1401652b8: a(n) BlInfiniteRecycler
    0x7ffeefbe7e50 M BlInfiniteRecycler>elementFor: 0x1401652b8: a(n) BlInfiniteRecycler
    0x7ffeefbe7e98 M [] in BlInfiniteLinearLayoutState>nextElement:in: 0x140165020: a(n) BlInfiniteLinearLayoutState
    0x7ffeefbe7ed8 M BlockClosure>ensure: 0x140166a90: a(n) BlockClosure
    0x7ffeefbe7f10 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe27d0 M BlInfiniteLinearLayoutState>nextElement:in: 0x140165020: a(n) BlInfiniteLinearLayoutState
    0x7ffeefbe2818 M [] in BlInfiniteLinearLayout>layoutChunkAdd 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2858 M BlockClosure>ensure: 0x140165410: a(n) BlockClosure
    0x7ffeefbe2890 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe28d8 M BlInfiniteLinearLayout>layoutChunkAdd 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2910 M [] in BlInfiniteLinearLayout>layoutChunk 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2950 M BlockClosure>ensure: 0x140165600: a(n) BlockClosure
    0x7ffeefbe2988 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe29e0 M BlInfiniteLinearLayout>layoutChunk 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2a48 I BlInfiniteLinearLayout>fillLayout: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2ab8 I BlInfiniteLinearLayout>layoutChildrenFillFromStart: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2b00 I BlInfiniteLinearLayout>layoutChildrenFill: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2b58 I BlInfiniteLinearLayout>layoutChildrenIn:state: 0x140165320: a(n) BlInfiniteLinearLayout
    0x7ffeefbe2ba8 I BrInfiniteListElement(BlInfiniteElement)>dispatchLayoutSecondStep 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2be8 I BrInfiniteListElement(BlInfiniteElement)>dispatchLayout 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c18 M BrInfiniteListElement(BlInfiniteElement)>onLayout: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c58 M [] in BrInfiniteListElement(BlElement)>applyLayoutSafelyIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2c98 M BlockClosure>ensure: 0x1401658b0: a(n) BlockClosure
    0x7ffeefbe2cd0 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe2d30 M BrInfiniteListElement(BlElement)>applyLayoutSafelyIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2d70 M [] in BrInfiniteListElement(BlElement)>applyLayoutIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2da0 M BlockClosure>on:do: 0x140165ad0: a(n) BlockClosure
    0x7ffeefbe2de0 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x140165b28: a(n) BlCompositeErrorHandler
    0x7ffeefbe2e28 M BrInfiniteListElement(BlElement)>applyLayoutIn: 0x140165140: a(n) BrInfiniteListElement
    0x7ffeefbe2e80 M [] in BlLinearLayoutVerticalOrientation>layout:in: 0x140165d38: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe2ed0 M [] in BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe2f18 M Array(SequenceableCollection)>do: 0x140165e98: a(n) Array
    0x7ffeefbe9868 M BlChildrenAccountedByLayout(BlChildrenSubset)>do: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe98a8 M BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140165e50: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9918 M BlLinearLayoutVerticalOrientation>layout:in: 0x140165d38: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9958 M BlLinearLayout>layout:in: 0x140166e18: a(n) BlLinearLayout
    0x7ffeefbe9998 M BrColumnedList(BlElement)>onLayout: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe99d8 M [] in BrColumnedList(BlElement)>applyLayoutSafelyIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9a18 M BlockClosure>ensure: 0x140166e78: a(n) BlockClosure
    0x7ffeefbe9a50 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe9ab0 M BrColumnedList(BlElement)>applyLayoutSafelyIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9af0 M [] in BrColumnedList(BlElement)>applyLayoutIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9b20 M BlockClosure>on:do: 0x140167098: a(n) BlockClosure
    0x7ffeefbe9b60 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x1401670f0: a(n) BlCompositeErrorHandler
    0x7ffeefbe9ba8 M BrColumnedList(BlElement)>applyLayoutIn: 0x140166d70: a(n) BrColumnedList
    0x7ffeefbe9c00 M [] in BlLinearLayoutVerticalOrientation>layout:in: 0x140167300: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9c50 M [] in BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9c98 M Array(SequenceableCollection)>do: 0x140167460: a(n) Array
    0x7ffeefbe9cd0 M BlChildrenAccountedByLayout(BlChildrenSubset)>do: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9d10 M BlChildrenAccountedByLayout(BlChildren)>inject:into: 0x140167418: a(n) BlChildrenAccountedByLayout
    0x7ffeefbe9d80 M BlLinearLayoutVerticalOrientation>layout:in: 0x140167300: a(n) BlLinearLayoutVerticalOrientation
    0x7ffeefbe9dc0 M BlLinearLayout>layout:in: 0x1401676e0: a(n) BlLinearLayout
    0x7ffeefbe9e00 M BlElement>onLayout: 0x140167648: a(n) BlElement
    0x7ffeefbe9e40 M [] in BlElement>applyLayoutSafelyIn: 0x140167648: a(n) BlElement
    0x7ffeefbe9e80 M BlockClosure>ensure: 0x140167740: a(n) BlockClosure
    0x7ffeefbe9eb8 M BlNullTelemetry(BlTelemetry)>timeSync:during: 0x14f0f8ce0: a(n) BlNullTelemetry
    0x7ffeefbe9f18 M BlElement>applyLayoutSafelyIn: 0x140167648: a(n) BlElement
    0x7ffeefbed7e0 M [] in BlElement>applyLayoutIn: 0x140167648: a(n) BlElement
    0x7ffeefbed810 M BlockClosure>on:do: 0x140168368: a(n) BlockClosure
    0x7ffeefbed850 M BlCompositeErrorHandler(BlErrorHandler)>with:do:failed: 0x1401683c0: a(n) BlCompositeErrorHandler
    0x7ffeefbed898 M BlElement>applyLayoutIn: 0x140167648: a(n) BlElement
    0x7ffeefbed908 I BlElement>computeLayout 0x140167648: a(n) BlElement
    0x7ffeefbed948 I BlElement>forceLayout 0x140167648: a(n) BlElement
    0x7ffeefbed9a8 I GtPhlowColumnedListViewExamples>errorInColumnDoDataBinder 0x140168630: a(n) GtPhlowColumnedListViewExamples
    0x7ffeefbed9d8 M GtExampleDebugger(GtExampleEvaluator)>primitiveProcessExample:withEvaluationContext: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbeda38 M [] in GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbeda78 M BlockClosure>ensure: 0x1401688c8: a(n) BlockClosure
    0x7ffeefbedac8 M GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb00 M GtExampleDebugger(GtExampleProcessor)>process: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb38 M [] in GtExampleDebugger(GtExampleProcessor)>value 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedb68 M BlockClosure>on:do: 0x140168c38: a(n) BlockClosure
    0x7ffeefbedba8 M GtCurrentExampleContext class>use:during: 0x1446188f0: a(n) GtCurrentExampleContext class
    0x7ffeefbedbe8 M GtExampleDebugger(GtExampleProcessor)>withContextDo: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc20 M GtExampleDebugger(GtExampleProcessor)>value 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc50 M [] in GtExampleDebugger(GtExampleEvaluator)>result 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedc80 M GtExampleDebugger>do:on:do: 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedcc8 M GtExampleDebugger(GtExampleEvaluator)>result 0x1401686f8: a(n) GtExampleDebugger
    0x7ffeefbedcf8 M GtExample>debug 0x19709dfc8: a(n) GtExample
    0x7ffeefbedd30 M [] in GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedd60 M BlockClosure>on:do: 0x140169368: a(n) BlockClosure
    0x7ffeefbeddb0 M [] in GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedde8 M Time class>microsecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbede20 M Time class>millisecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbede60 M BlockClosure>timeToRun 0x140169558: a(n) BlockClosure
    0x7ffeefbede98 M GtExamplesHDReport>beginExample:runBlock: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedee0 M GtExamplesHDReport>runExample: 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbedf18 M [] in GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdea38 M OrderedCollection>do: 0x197099e08: a(n) OrderedCollection
    0x7ffeefbdea70 M [] in GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdeab0 M [] in CurrentExecutionEnvironment class>activate:for: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdeaf0 M BlockClosure>ensure: 0x19709a0b0: a(n) BlockClosure
    0x7ffeefbdeb30 M CurrentExecutionEnvironment class>activate:for: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdeb70 M TestExecutionEnvironment(ExecutionEnvironment)>beActiveDuring: 0x197099fb8: a(n) TestExecutionEnvironment
    0x7ffeefbdeba8 M DefaultExecutionEnvironment>runTestsBy: 0x1409f7da8: a(n) DefaultExecutionEnvironment
    0x7ffeefbdebe0 M CurrentExecutionEnvironment class>runTestsBy: 0x140a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbdec18 M GtExamplesHDReport>runAll 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdec48 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdec80 M Time class>microsecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbdecb8 M Time class>millisecondsToRun: 0x1409fcff0: a(n) Time class
    0x7ffeefbdecf8 M BlockClosure>timeToRun 0x19709a618: a(n) BlockClosure
    0x7ffeefbded28 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbded68 M BlockClosure>ensure: 0x19709a918: a(n) BlockClosure
    0x7ffeefbdeda0 M [] in GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdedd0 M Author>ifUnknownAuthorUse:during: 0x14102a740: a(n) Author
    0x7ffeefbdee10 M GtExamplesHDReport>run 0x197099d30: a(n) GtExamplesHDReport
    0x7ffeefbdee48 M GtExamplesHDReport class>runPackage: 0x144618008: a(n) GtExamplesHDReport class
    0x7ffeefbdee80 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x144618008: a(n) GtExamplesHDReport class
    0x7ffeefbdeed0 M [] in Set>collect: 0x145bbc4d8: a(n) Set
    0x7ffeefbdef18 M Array(SequenceableCollection)>do: 0x145bbc520: a(n) Array
       0x145bbcc78 s Set>collect:
       0x145bbcd30 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x145bbce28 s [] in GtExamplesCommandLineHandler>runPackages
       0x145bbcf08 s BlockClosure>ensure:
       0x14e950710 s UIManager class>nonInteractiveDuring:
       0x14e9507c8 s GtExamplesCommandLineHandler>runPackages
       0x14e950880 s GtExamplesCommandLineHandler>activate
       0x14e950938 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x14e9509f0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x14e950aa8 s BlockClosure>on:do:
       0x14e950b60 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x14e950c38 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x14e950cf0 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x14e950db8 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x14e950e90 s BlockClosure>on:do:
       0x14e950f68 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x14e951040 s [] in BlockClosure>newProcess



Process 18798 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000001481107d1
->  0x1481107d1: int3   
    0x1481107d2: int3   
    0x1481107d3: int3   
    0x1481107d4: int3   
Target 0: (Pharo) stopped.

(lldb) call printAllStacks()
Process        0x15511b938 priority 40
    0x7ffeefbf0a18 M Context(Object)>copy 0x15691cf68: a(n) Context
    0x7ffeefbf0a50 M Context>copyTo: 0x15691cf68: a(n) Context
    0x7ffeefbf0a98 M Context>copyTo: 0x15691ce78: a(n) Context
    0x7ffeefbf0ae0 M Context>copyTo: 0x15691cdc0: a(n) Context
    0x7ffeefbf0b28 M Context>copyTo: 0x15691cd08: a(n) Context
    0x7ffeefbf0b70 M Context>copyTo: 0x15691cc50: a(n) Context
    0x7ffeefbf0bb8 M Context>copyTo: 0x15691cb70: a(n) Context
    0x7ffeefbf0c00 M Context>copyTo: 0x15691ca90: a(n) Context
    0x7ffeefbf0c48 M Context>copyTo: 0x15691c998: a(n) Context
    0x7ffeefbf0c90 M Context>copyTo: 0x15691c8e0: a(n) Context
    0x7ffeefbf0cd8 M Context>copyTo: 0x15691c828: a(n) Context
    0x7ffeefbf0d20 M Context>copyTo: 0x14874a2c8: a(n) Context
    0x7ffeefbf0d68 M Context>copyTo: 0x14874a158: a(n) Context
    0x7ffeefbf0db0 M Context>copyTo: 0x148749fe8: a(n) Context
    0x7ffeefbf0df8 M Context>copyTo: 0x14816e2e8: a(n) Context
    0x7ffeefbf0e40 M Context>copyTo: 0x148749dc0: a(n) Context
    0x7ffeefbf0e88 M Context>copyTo: 0x14816e210: a(n) Context
    0x7ffeefbf0ed0 M Context>copyTo: 0x148749ae0: a(n) Context
    0x7ffeefbf0f18 M Context>copyTo: 0x14816e118: a(n) Context
    0x7ffeefbdc7c8 M Context>copyTo: 0x148749748: a(n) Context
    0x7ffeefbdc810 M Context>copyTo: 0x1487495d8: a(n) Context
    0x7ffeefbdc858 M Context>copyTo: 0x148749468: a(n) Context
    0x7ffeefbdc8a0 M Context>copyTo: 0x1487492f8: a(n) Context
    0x7ffeefbdc8e8 M Context>copyTo: 0x14816e040: a(n) Context
    0x7ffeefbdc930 M Context>copyTo: 0x1487490d0: a(n) Context
    0x7ffeefbdc978 M Context>copyTo: 0x148748f60: a(n) Context
    0x7ffeefbdc9c0 M Context>copyTo: 0x148748df0: a(n) Context
    0x7ffeefbdca08 M Context>copyTo: 0x14816df38: a(n) Context
    0x7ffeefbdca50 M Context>copyTo: 0x148748bc8: a(n) Context
    0x7ffeefbdca98 M Context>copyTo: 0x14816de20: a(n) Context
    0x7ffeefbdcae0 M Context>copyTo: 0x14816eeb8: a(n) Context
    0x7ffeefbdcb28 M Context>copyTo: 0x1487488e8: a(n) Context
    0x7ffeefbdcb70 M Context>copyTo: 0x148748778: a(n) Context
    0x7ffeefbdcbb8 M Context>copyTo: 0x14866e858: a(n) Context
    0x7ffeefbdcc00 M Context>copyTo: 0x148748550: a(n) Context
    0x7ffeefbdcc48 M Context>copyTo: 0x1487483e0: a(n) Context
    0x7ffeefbdcc90 M Context>copyTo: 0x148748270: a(n) Context
    0x7ffeefbdccd8 M Context>copyTo: 0x148748100: a(n) Context
    0x7ffeefbdcd20 M Context>copyTo: 0x148676078: a(n) Context
    0x7ffeefbdcd68 M Context>copyTo: 0x148747ed8: a(n) Context
    0x7ffeefbdcdb0 M Context>copyTo: 0x148747d68: a(n) Context
    0x7ffeefbdcdf8 M Context>copyTo: 0x148747bf8: a(n) Context
    0x7ffeefbdce40 M Context>copyTo: 0x148676280: a(n) Context
    0x7ffeefbdce88 M Context>copyTo: 0x1487479d0: a(n) Context
    0x7ffeefbdced0 M Context>copyTo: 0x1487477a8: a(n) Context
    0x7ffeefbdcf18 M Context>copyTo: 0x148676400: a(n) Context
    0x7ffeefbda7c8 M Context>copyTo: 0x148747410: a(n) Context
    0x7ffeefbda810 M Context>copyTo: 0x1486764d8: a(n) Context
    0x7ffeefbda858 M Context>copyTo: 0x1487471e8: a(n) Context
    0x7ffeefbda8a0 M Context>copyTo: 0x148747078: a(n) Context
    0x7ffeefbda8e8 M Context>copyTo: 0x148746f08: a(n) Context
    0x7ffeefbda930 M Context>copyTo: 0x14873f918: a(n) Context
    0x7ffeefbda978 M Context>copyTo: 0x148746ce0: a(n) Context
    0x7ffeefbda9c0 M Context>copyTo: 0x148746b70: a(n) Context
    0x7ffeefbdaa08 M Context>copyTo: 0x148746a00: a(n) Context
    0x7ffeefbdaa50 M Context>copyTo: 0x148746890: a(n) Context
    0x7ffeefbdaa98 M Context>copyTo: 0x148746720: a(n) Context
    0x7ffeefbdaae0 M Context>copyTo: 0x1487415b8: a(n) Context
    0x7ffeefbdab28 M Context>copyTo: 0x1487412a8: a(n) Context
    0x7ffeefbdab70 M Context>copyTo: 0x148746440: a(n) Context
    0x7ffeefbdabb8 M Context>copyTo: 0x148745088: a(n) Context
    0x7ffeefbdac00 M Context>copyTo: 0x148746218: a(n) Context
    0x7ffeefbdac48 M Context>copyTo: 0x1487414e0: a(n) Context
    0x7ffeefbdac90 M Context>copyTo: 0x148745ff0: a(n) Context
    0x7ffeefbdacd8 M Context>copyTo: 0x148741670: a(n) Context
    0x7ffeefbdad20 M Context>copyTo: 0x148744fd0: a(n) Context
    0x7ffeefbdad68 M Context>copyTo: 0x148745ba0: a(n) Context
    0x7ffeefbdadb0 M Context>copyTo: 0x148745a30: a(n) Context
    0x7ffeefbdadf8 M Context>copyTo: 0x148744bd8: a(n) Context
    0x7ffeefbdae40 M Context>copyTo: 0x148745808: a(n) Context
    0x7ffeefbdae88 M Context>copyTo: 0x148745698: a(n) Context
    0x7ffeefbdaed0 M Context>copyTo: 0x148745528: a(n) Context
    0x7ffeefbdaf18 M Context>copyTo: 0x148744f18: a(n) Context
    0x7ffeefbe0988 I Context>copyTo: 0x148744e60: a(n) Context
    0x7ffeefbe09d0 I MessageNotUnderstood(Exception)>freezeUpTo: 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0a18 I MessageNotUnderstood(Exception)>freeze 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0a48 M [] in GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0a80 M BlockClosure>cull: 0x1487414c0: a(n) BlockClosure
    0x7ffeefbe0ac0 M Context>evaluateSignal: 0x148745088: a(n) Context
    0x7ffeefbe0af8 M Context>handleSignal: 0x148745088: a(n) Context
    0x7ffeefbe0b30 M Context>handleSignal: 0x148744fd0: a(n) Context
    0x7ffeefbe0b68 M MessageNotUnderstood(Exception)>signal 0x148744e10: a(n) MessageNotUnderstood
    0x7ffeefbe0bb8 I GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x148744bc8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefbe0bf0 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0c50 M [] in GtExampleEvaluator>basicProcess: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0c90 M BlockClosure>ensure: 0x148744c90: a(n) BlockClosure
    0x7ffeefbe0ce0 M GtExampleEvaluator>basicProcess: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d18 M GtExampleEvaluator(GtExampleProcessor)>process: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d50 M [] in GtExampleEvaluator(GtExampleProcessor)>value 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0d80 M BlockClosure>on:do: 0x148741598: a(n) BlockClosure
    0x7ffeefbe0dc0 M GtCurrentExampleContext class>use:during: 0x14c618920: a(n) GtCurrentExampleContext class
    0x7ffeefbe0e00 M GtExampleEvaluator(GtExampleProcessor)>withContextDo: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e38 M GtExampleEvaluator(GtExampleProcessor)>value 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e68 M [] in GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0e98 M BlockClosure>on:do: 0x148741360: a(n) BlockClosure
    0x7ffeefbe0ed8 M GtExampleEvaluator>do:on:do: 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbe0f20 M GtExampleEvaluator>result 0x148741238: a(n) GtExampleEvaluator
    0x7ffeefbde8b8 M GtExample>run 0x148740050: a(n) GtExample
    0x7ffeefbde8e8 M GtExample>result 0x148740050: a(n) GtExample
    0x7ffeefbde930 I GtExamplesExamplesWithInheritance>resultOfInvalidExampleWithInvalidSuperclassProvider 0x14873f908: a(n) GtExamplesExamplesWithInheritance
    0x7ffeefbde960 M GtExampleDebugger(GtExampleEvaluator)>primitiveProcessExample:withEvaluationContext: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbde9c0 M [] in GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdea00 M BlockClosure>ensure: 0x14873f9d0: a(n) BlockClosure
    0x7ffeefbdea50 M GtExampleDebugger(GtExampleEvaluator)>basicProcess: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdea88 M GtExampleDebugger(GtExampleProcessor)>process: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeac0 M [] in GtExampleDebugger(GtExampleProcessor)>value 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeaf0 M BlockClosure>on:do: 0x1486764b8: a(n) BlockClosure
    0x7ffeefbdeb30 M GtCurrentExampleContext class>use:during: 0x14c618920: a(n) GtCurrentExampleContext class
    0x7ffeefbdeb70 M GtExampleDebugger(GtExampleProcessor)>withContextDo: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdeba8 M GtExampleDebugger(GtExampleProcessor)>value 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdebd8 M [] in GtExampleDebugger(GtExampleEvaluator)>result 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec08 M GtExampleDebugger>do:on:do: 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec50 M GtExampleDebugger(GtExampleEvaluator)>result 0x148676210: a(n) GtExampleDebugger
    0x7ffeefbdec80 M GtExample>debug 0x148188088: a(n) GtExample
    0x7ffeefbdecb8 M [] in GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdece8 M BlockClosure>on:do: 0x148676130: a(n) BlockClosure
    0x7ffeefbded38 M [] in GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbded70 M Time class>microsecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbdeda8 M Time class>millisecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbdede8 M BlockClosure>timeToRun 0x14866e910: a(n) BlockClosure
    0x7ffeefbdee20 M GtExamplesHDReport>beginExample:runBlock: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdee68 M GtExamplesHDReport>runExample: 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdeea0 M [] in GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbdeee8 M OrderedCollection>do: 0x14816ee98: a(n) OrderedCollection
    0x7ffeefbdef20 M [] in GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8ab0 M [] in CurrentExecutionEnvironment class>activate:for: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8af0 M BlockClosure>ensure: 0x14816ded8: a(n) BlockClosure
    0x7ffeefbd8b30 M CurrentExecutionEnvironment class>activate:for: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8b70 M TestExecutionEnvironment(ExecutionEnvironment)>beActiveDuring: 0x14816ddc0: a(n) TestExecutionEnvironment
    0x7ffeefbd8ba8 M DefaultExecutionEnvironment>runTestsBy: 0x1489f7da8: a(n) DefaultExecutionEnvironment
    0x7ffeefbd8be0 M CurrentExecutionEnvironment class>runTestsBy: 0x148a02930: a(n) CurrentExecutionEnvironment class
    0x7ffeefbd8c18 M GtExamplesHDReport>runAll 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8c48 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8c80 M Time class>microsecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbd8cb8 M Time class>millisecondsToRun: 0x1489fcff0: a(n) Time class
    0x7ffeefbd8cf8 M BlockClosure>timeToRun 0x14816e0f8: a(n) BlockClosure
    0x7ffeefbd8d28 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8d68 M BlockClosure>ensure: 0x14816e1d0: a(n) BlockClosure
    0x7ffeefbd8da0 M [] in GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8dd0 M Author>ifUnknownAuthorUse:during: 0x14902a740: a(n) Author
    0x7ffeefbd8e10 M GtExamplesHDReport>run 0x14816dff0: a(n) GtExamplesHDReport
    0x7ffeefbd8e48 M GtExamplesHDReport class>runPackage: 0x14c618038: a(n) GtExamplesHDReport class
    0x7ffeefbd8e80 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x14c618038: a(n) GtExamplesHDReport class
    0x7ffeefbd8ed0 M [] in Set>collect: 0x15691c088: a(n) Set
    0x7ffeefbd8f18 M Array(SequenceableCollection)>do: 0x15691c188: a(n) Array
       0x15691c8e0 s Set>collect:
       0x15691c998 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x15691ca90 s [] in GtExamplesCommandLineHandler>runPackages
       0x15691cb70 s BlockClosure>ensure:
       0x15691cc50 s UIManager class>nonInteractiveDuring:
       0x15691cd08 s GtExamplesCommandLineHandler>runPackages
       0x15691cdc0 s GtExamplesCommandLineHandler>activate
       0x15691ce78 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x15691cf68 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x15691d048 s BlockClosure>on:do:
       0x15691d128 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x15691d200 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x15691d2b8 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x15691d380 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x15691d458 s BlockClosure>on:do:
       0x15691d530 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x15691d608 s [] in BlockClosure>newProcess


(lldb) call dumpPrimTraceLog()
stringHash:initialHash:
compare:with:collated:
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
primitiveChangeClassTo:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
primitiveChangeClassTo:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
stringHash:initialHash:
compare:with:collated:
primitiveChangeClassTo:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
primitiveChangeClassTo:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
compare:with:collated:
compare:with:collated:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
stringHash:initialHash:
basicNew:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
size
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
**StackOverflow**
perform:withArguments:
**StackOverflow**
stringHash:initialHash:
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
**StackOverflow**
**StackOverflow**
stringHash:initialHash:
basicNew
perform:withArguments:
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
**StackOverflow**
**StackOverflow**
class
perform:withArguments:
value:
first
basicNew
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
**StackOverflow**
basicNew
perform:withArguments:
basicNew
findNextHandlerOrSignalingContext
tempAt:
class
findNextHandlerOrSignalingContext
tempAt:
tempAt:
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**StackOverflow**
shallowCopy
**StackOverflow**
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
shallowCopy
**CompactCode**



On 26 Sep 2020, at 23:24, Eliot Miranda <[hidden email]> wrote:

Hi Andrei,

   fixed in commit 561b06530bbaed5f19e9d7f077a7df9eb3a8d236, VMMaker.oscog-eem.2824


On Fri, Sep 11, 2020 at 8:58 AM Andrei Chis <[hidden email]> wrote:
 
Hi,

We are getting often crashes on our CI when calling `Context>copyTo:` in a GT image and a vm build from https://github.com/feenkcom/opensmalltalk-vm.

To sum up during `Context>copyTo:`, `Object>>#copy` is called on a context leading to a segmentation fault crash. Looking at that context in lldb the pc looks off.  It has the value `0xfffffffffea7f6e1`.

 (lldb) call (void *) printOop(0x1206b6990)
    0x1206b6990: a(n) Context
     0x1206b6a48 0xfffffffffea7f6e1                0x9        0x1146b2e08        0x1206b6b00 
     0x1206b6b28        0x1206b6b50 

Can this indicate some corruption or is it expected to have such values? `CoInterpreter>>ensureContextHasBytecodePC:` has code that also handles negative values for the pc which suggests that this might be expected.

Changing `Context>copyTo:` by adding a `self pc` before calling `self copy` leads to no more crashes. Not sure if there is a reason for that or just plain luck.

A simple reduced stack is below (more details in this issue [1]). The crash happens always with contexts reified as objects (in this case 0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages). 
Could this suggest some kind of issue in the vm when reifying contexts, or just some other problem with memory corruption? 


 0x7ffeefbb4380 M Context(Object)>copy 0x1206b6990: a(n) Context
    0x7ffeefbb43b8 M Context>copyTo: 0x1206b6990: a(n) Context
    0x7ffeefbb4400 M Context>copyTo: 0x1206b5ae0: a(n) Context
  ...
    0x7ffeefba6078 M Context>copyTo: 0x110548b28: a(n) Context
    0x7ffeefba60d0 I Context>copyTo: 0x110548a70: a(n) Context
    0x7ffeefba6118 I MessageNotUnderstood(Exception)>freezeUpTo: 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6160 I MessageNotUnderstood(Exception)>freeze 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba6190 M [] in GtExampleEvaluator>result 0x110544fb8: a(n) GtExampleEvaluator
    0x7ffeefba61c8 M BlockClosure>cull: 0x110545188: a(n) BlockClosure
    0x7ffeefba6208 M Context>evaluateSignal: 0x110548c98: a(n) Context
    0x7ffeefba6240 M Context>handleSignal: 0x110548c98: a(n) Context
    0x7ffeefba6278 M Context>handleSignal: 0x110548be0: a(n) Context
    0x7ffeefba62b0 M MessageNotUnderstood(Exception)>signal 0x110548a20: a(n) MessageNotUnderstood
    0x7ffeefba62f0 M GtDummyExamplesWithInheritanceSubclassB(Object)>doesNotUnderstand: exampleH 0x1105487d8: a(n) GtDummyExamplesWithInheritanceSubclassB
    0x7ffeefba6328 M GtExampleEvaluator>primitiveProcessExample:withEvaluationContext: 0x110544fb8: a(n) GtExampleEvaluator
 ...
    0x7ffeefbe64d0 M [] in GtExamplesHDReport class(HDReport class)>runPackages: 0x1145e41c8: a(n) GtExamplesHDReport class
    0x7ffeefbe6520 M [] in Set>collect: 0x1206b5ab0: a(n) Set
    0x7ffeefbe6568 M Array(SequenceableCollection)>do: 0x1206b5c50: a(n) Array
       0x1206b5b98 s Set>collect:
       0x1206b5ae0 s GtExamplesHDReport class(HDReport class)>runPackages:
       0x1206b6990 s [] in GtExamplesCommandLineHandler>runPackages
       0x1206b6a48 s BlockClosure>ensure:
       0x1206b6b68 s UIManager class>nonInteractiveDuring:
       0x1206b6c48 s GtExamplesCommandLineHandler>runPackages
       0x1206b6d98 s GtExamplesCommandLineHandler>activate
       0x1206b75d0 s GtExamplesCommandLineHandler class(CommandLineHandler class)>activateWith:
       0x1207d2f00 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x1207e6620 s BlockClosure>on:do:
       0x1207f7ab8 s PharoCommandLineHandler(BasicCommandLineHandler)>activateSubCommand:
       0x120809d40 s PharoCommandLineHandler(BasicCommandLineHandler)>handleSubcommand
       0x12082ca60 s PharoCommandLineHandler(BasicCommandLineHandler)>handleArgument:
       0x120789938 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207a83e0 s BlockClosure>on:do:
       0x1207b57a0 s [] in PharoCommandLineHandler(BasicCommandLineHandler)>activate
       0x1207bf830 s [] in BlockClosure>newProcess
Cheers,
Andrei





--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot
12