Pharo7 and full block closures

Hi,

I kind of missed the open alpha of "Scorch" and to make up for it I tried to follow the instructions on Pharo7 but I do crash the VM. Is that to be expected?

"Add special selector for trap instruction"
Smalltalk specialObjectsArray at: 60 put: #trapTripped.
"Disable hot spot detection (to load the Scorch code)"
Smalltalk specialObjectsArray at: 59 put: nil.
"Recompile the fetch mourner primitive which has strange side-effect with alternate bytecode set and closures"
WeakArray class compile: 'primitiveFetchMourner ^ nil' classified: #patch.
"Enable FullBlockClosure and alternate bytecode set"
CompilationContext bytecodeBackend: OpalEncoderForSistaV1.
CompilationContext usesFullBlockClosure: true.
OpalCompiler recompileAll.


If I only execute the last three lines it is crashing as well but at a different point in the VM?


(lldb) bt
* thread #1: tid = 0x5636d2, 0xa165e496 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0xa165e496 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0xa1744c63 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0xa15bea34 libsystem_c.dylib`abort + 141
    frame #3: 0x00087128 Pharo`error + 55
    frame #4: 0x000132e6 Pharo`lookupMethodInClass + 601
    frame #5: 0x000085c1 Pharo`interpret + 25756
    frame #6: 0x00011b9a Pharo`enterSmalltalkExecutiveImplementation + 135
    frame #7: 0x000023ab Pharo`interpret + 646
    frame #8: 0x000889d9 Pharo`-[sqSqueakMainApplication runSqueak] + 476
    frame #9: 0x95ac4a1b Foundation`__NSFirePerformWithOrder + 419
    frame #10: 0x942811fe CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 30
    frame #11: 0x94281157 CoreFoundation`__CFRunLoopDoObservers + 391
    frame #12: 0x942614a2 CoreFoundation`__CFRunLoopRun + 1058
    frame #13: 0x94260e1a CoreFoundation`CFRunLoopRunSpecific + 506
    frame #14: 0x94260c0b CoreFoundation`CFRunLoopRunInMode + 123
    frame #15: 0x93954396 HIToolbox`RunCurrentEventLoopInMode + 268
    frame #16: 0x93954057 HIToolbox`ReceiveNextEventCommon + 195
    frame #17: 0x93953f7b HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 83
    frame #18: 0x91f86015 AppKit`_DPSNextEvent + 1227
    frame #19: 0x926ebbbb AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1742
    frame #20: 0x926eb4e5 AppKit`-[NSApplication(NSEvent) nextEventMatchingMask:untilDate:inMode:dequeue:] + 132
    frame #21: 0x91f7acb6 AppKit`-[NSApplication run] + 943
    frame #22: 0x91f47d1e AppKit`NSApplicationMain + 1368
    frame #23: 0xa152f395 libdyld.dylib`start + 1
(lldb) p (int) activeProcess()
(int) $0 = 115378352
(lldb) p/x (int) activeProcess()
(int) $1 = 0x06e088b0
(lldb) p (void) printAllStacks()
Process  0x6e088b0 priority 40
0xbfffc124 I FileReference(AbstractFileReference)>writeStreamDo: 0x9a303d8: a(n) FileReference
0xbfffc144 M [] in OmBlockFileStore(OmFileStore)>flushEntryBuffer 0x9a2c3c0: a(n) OmBlockFileStore
0xbfffc164 M [] in Semaphore>critical: 0x6e08898: a(n) Semaphore
0xbfffc184 M BlockClosure>ensure: 0x6f2d1f8: a(n) BlockClosure
0xbfffc1a4 M Semaphore>critical: 0x6e08898: a(n) Semaphore
0xbfffc1c8 I OmBlockFileStore(OmFileStore)>critical: 0x9a2c3c0: a(n) OmBlockFileStore
0xbfffc1e4 M OmBlockFileStore(OmFileStore)>flushEntryBuffer 0x9a2c3c0: a(n) OmBlockFileStore
0xbfffc204 I OmDeferrer>sendMessage 0x9a303f8: a(n) OmDeferrer
0xbfffc21c M [] in OmDeferrer>flush 0x9a303f8: a(n) OmDeferrer
0xbfffc23c M BlockClosure>ensure: 0x6f2d0e0: a(n) BlockClosure
0xbfffc260 I OmDeferrer>flush 0x9a303f8: a(n) OmDeferrer
0xbfffc280 I [] in OmDeferrer>runMessageProcess 0x9a303f8: a(n) OmDeferrer
0xbfffc2a0 I [] in BlockClosure>newProcess 0x6e12fb8: a(n) BlockClosure

processes at priority 40
Process  0x7f4ae88 priority 40
0xbfff7208 M IdentityDictionary(Dictionary)>at:ifAbsent: 0x6e11fa8: a(n) IdentityDictionary
0xbfff7228 M IdentityDictionary(Dictionary)>at:ifAbsentPut: 0x6e11fa8: a(n) IdentityDictionary
0xbfff7248 M IRBytecodeGenerator>label: 0x6e12050: a(n) IRBytecodeGenerator
0xbfff7264 M IRTranslatorV2>label: 0x6e12118: a(n) IRTranslatorV2
0xbfff7280 M IRTranslatorV2>visitSequence: 0x6e12118: a(n) IRTranslatorV2
0xbfff729c M IRSequence>accept: 0x6e12138: a(n) IRSequence
0xbfff8fa4 M IRTranslatorV2(IRVisitor)>visitNode: 0x6e12118: a(n) IRTranslatorV2
0xbfff8fc0 M [] in IRTranslatorV2(IRVisitor)>visitNodes: 0x6e12118: a(n) IRTranslatorV2
0xbfff8fe4 M SortedCollection(OrderedCollection)>do: 0x6e12248: a(n) SortedCollection
0xbfff9000 M IRTranslatorV2(IRVisitor)>visitNodes: 0x6e12118: a(n) IRTranslatorV2
0xbfff901c M IRTranslatorV2>visitSequences: 0x6e12118: a(n) IRTranslatorV2
0xbfff9038 M IRTranslatorV2>visitMethod: 0x6e12118: a(n) IRTranslatorV2
0xbfff9054 M IRMethod>accept: 0x6e12320: a(n) IRMethod
0xbfff9070 M IRTranslatorV2(IRVisitor)>visitNode: 0x6e12118: a(n) IRTranslatorV2
0xbfff9094 M IRMethod>generateBlock:withScope: 0x6e12320: a(n) IRMethod
0xbfff90b4 M IRMethod>compiledBlock: 0x6e12320: a(n) IRMethod
0xbfff90d4 M OCASTTranslator>translateFullBlock: 0x6e123a8: a(n) OCASTTranslator
0xbfff90f8 M OCASTTranslatorForValue(OCASTTranslator)>visitFullBlockNode: 0x6e12450: a(n) OCASTTranslatorForValue
0xbfff9120 M OCASTTranslatorForValue(OCASTTranslator)>visitBlockNode: 0x6e12450: a(n) OCASTTranslatorForValue
0xbfff913c M RBBlockNode>acceptVisitor: 0x6e12420: a(n) RBBlockNode
0xbfff9158 M OCASTTranslatorForValue(RBProgramNodeVisitor)>visitNode: 0x6e12450: a(n) OCASTTranslatorForValue
0xbfff9174 M OCASTTranslatorForValue(OCASTTranslator)>visitNode: 0x6e12450: a(n) OCASTTranslatorForValue
0xbfff9190 M [] in OCASTTranslatorForValue(OCASTTranslator)>emitMessageNode: 0x6e12450: a(n) OCASTTranslatorForValue
0xbfff91b4 M OrderedCollection>do: 0x6e12480: a(n) OrderedCollection
0xbfff91d0 M OCASTTranslatorForValue(OCASTTranslator)>emitMessageNode: 0x6e12450: a(n) OCASTTranslatorForValue
0xbfff91ec M OCASTTranslatorForValue(OCASTTranslator)>visitMessageNode: 0x6e12450: a(n) OCASTTranslatorForValue
0xbfff9208 M RBMessageNode>acceptVisitor: 0x6e124f8: a(n) RBMessageNode
0xbfff9224 M OCASTTranslatorForValue(RBProgramNodeVisitor)>visitNode: 0x6e12450: a(n) OCASTTranslatorForValue
0xbfff9240 M OCASTTranslatorForValue(OCASTTranslator)>visitNode: 0x6e12450: a(n) OCASTTranslatorForValue
0xbfff925c M [] in OCASTTranslatorForEffect(OCASTTranslator)>emitMessageNode: 0x6e12520: a(n) OCASTTranslatorForEffect
0xbfff9280 M OrderedCollection>do: 0x6e12550: a(n) OrderedCollection
0xbfff929c M OCASTTranslatorForEffect(OCASTTranslator)>emitMessageNode: 0x6e12520: a(n) OCASTTranslatorForEffect
0xbfffa00c M OCASTTranslatorForEffect>emitMessageNode: 0x6e12520: a(n) OCASTTranslatorForEffect
0xbfffa028 M OCASTTranslatorForEffect(OCASTTranslator)>visitMessageNode: 0x6e12520: a(n) OCASTTranslatorForEffect
0xbfffa044 M RBMessageNode>acceptVisitor: 0x6e125c8: a(n) RBMessageNode
0xbfffa060 M OCASTTranslatorForEffect(RBProgramNodeVisitor)>visitNode: 0x6e12520: a(n) OCASTTranslatorForEffect
0xbfffa07c M OCASTTranslatorForEffect(OCASTTranslator)>visitNode: 0x6e12520: a(n) OCASTTranslatorForEffect
0xbfffa098 M [] in OCASTTranslatorForEffect>visitSequenceNode: 0x6e12520: a(n) OCASTTranslatorForEffect
0xbfffa0bc M OrderedCollection>do: 0x6e12668: a(n) OrderedCollection
0xbfffa0d8 M OCASTTranslatorForEffect>visitSequenceNode: 0x6e12520: a(n) OCASTTranslatorForEffect
0xbfffa0f4 M RBSequenceNode>acceptVisitor: 0x6e126e0: a(n) RBSequenceNode
0xbfffa110 M OCASTTranslatorForEffect(RBProgramNodeVisitor)>visitNode: 0x6e12520: a(n) OCASTTranslatorForEffect
0xbfffa12c M OCASTTranslatorForEffect(OCASTTranslator)>visitNode: 0x6e12520: a(n) OCASTTranslatorForEffect
0xbfffa148 M OCASTTranslator>visitMethodNode: 0x6e12708: a(n) OCASTTranslator
0xbfffa164 M RBMethodNode>acceptVisitor: 0x6e12780: a(n) RBMethodNode
0xbfffa180 M OCASTTranslator(RBProgramNodeVisitor)>visitNode: 0x6e12708: a(n) OCASTTranslator
0xbfffa19c M OCASTTranslator>visitNode: 0x6e12708: a(n) OCASTTranslator
0xbfffa1bc M RBMethodNode>generateIR 0x6e12780: a(n) RBMethodNode
0xbfffa1d4 M RBMethodNode>generate: 0x6e12780: a(n) RBMethodNode
0xbfffa1f4 M [] in OpalCompiler>compile 0x6e127e8: a(n) OpalCompiler
0xbfffa20c M BlockClosure>on:do: 0x6e12868: a(n) BlockClosure
0xbfffa230 M OpalCompiler>compile 0x6e127e8: a(n) OpalCompiler
0xbfffa250 M AbstractNautilusUI class(Behavior)>recompile:from: 0x7fb08f8: a(n) AbstractNautilusUI class
0xbfffa274 M [] in AbstractNautilusUI class(Behavior)>compileAllFrom: 0x7fb08f8: a(n) AbstractNautilusUI class
0xbfffa29c M Set>do: 0x6e12970: a(n) Set
0xbfffaf34 M AbstractNautilusUI class(Behavior)>compileAllFrom: 0x7fb08f8: a(n) AbstractNautilusUI class
0xbfffaf50 M AbstractNautilusUI class(Class)>compileAllFrom: 0x7fb08f8: a(n) AbstractNautilusUI class
0xbfffaf6c M AbstractNautilusUI class(Behavior)>compileAll 0x7fb08f8: a(n) AbstractNautilusUI class
0xbfffaf8c I AbstractNautilusUI class(Class)>compileAll 0x7fb08f8: a(n) AbstractNautilusUI class
0xbfffafac I AbstractNautilusUI class(Behavior)>recompile 0x7fb08f8: a(n) AbstractNautilusUI class
0xbfffafc4 M [] in SmalltalkImage>recompile 0x74ca9b8: a(n) SmalltalkImage
0xbfffaff8 M [] in OrderedCollection(Collection)>do:displayingProgress:every: 0x6e12aa0: a(n) OrderedCollection
0xbfffb01c M OrderedCollection>do: 0x6e12aa0: a(n) OrderedCollection
0xbfffb054 I [] in OrderedCollection(Collection)>do:displayingProgress:every: 0x6e12aa0: a(n) OrderedCollection
0xbfffb078 I BlockClosure>cull: 0x6e12b40: a(n) BlockClosure
0xbfffb094 M [] in Job>run 0x6e12a58: a(n) Job
0xbfffb0b8 M [] in CurrentJob(DynamicVariable)>value:during: 0x72f6dc0: a(n) CurrentJob
0xbfffb0d8 M BlockClosure>ensure: 0x6e12b80: a(n) BlockClosure
0xbfffb104 I CurrentJob(DynamicVariable)>value:during: 0x72f6dc0: a(n) CurrentJob
0xbfffb12c I CurrentJob class(DynamicVariable class)>value:during: 0x7233b28: a(n) CurrentJob class
0xbfffb14c M [] in Job>run 0x6e12a58: a(n) Job
0xbfffb16c M BlockClosure>ensure: 0x6e12c80: a(n) BlockClosure
0xbfffb190 I Job>run 0x6e12a58: a(n) Job
0xbfffb1b0 I MorphicUIManager(UIManager)>displayProgress:from:to:during: 0x9a2c2d8: a(n) MorphicUIManager
0xbfffb1e0 I ByteString(String)>displayProgressFrom:to:during: 0x748c040: a(n) ByteString
0xbfffb214 I OrderedCollection(Collection)>do:displayingProgress:every: 0x6e12aa0: a(n) OrderedCollection
0xbfffb240 I OrderedCollection(Collection)>do:displayingProgress: 0x6e12aa0: a(n) OrderedCollection
0xbfffb268 I SmalltalkImage>recompile 0x74ca9b8: a(n) SmalltalkImage
0xbfffb288 I OpalCompiler class>recompileAll 0x7310170: a(n) OpalCompiler class

> On 22. Sep 2017, at 13:33, Clément Bera <[hidden email]> wrote:


Hi Clement,


> Sista will work for the release of Pharo 7 as it did for Pharo 6 and 6.1 but it won't work every day during the alpha of Pharo 7 and no sane framework developer can guarantee that. Sista does not work currently in Pharo 7 due to some changes breaking the full block closures, I am going to fix it but even then in the incoming months there is a high chance that another change in Pharo 7 breaks it again, so I would never recommend to use it with the alpha.


thank you for your response. I didn't expect it to work (after all you label it "Alpha" and I ran into issues already) but my intention for the mail was a bit different. Let me try to briefly explain my motivation. In my experience:

(1) Software not executed stops to work (or never "worked")
(2) The earlier breakage is detected the less expensive it is to fix.

My assumption is that Opal is part of Pharo and that we look at a regression of the Opal compiler and/or how it interacts with the rest of the image. As it might be easier to understand/fix it now than later I brought it up. From my point of view I wonder if we want to add ~50s to the CI process to execute:

CompilationContext bytecodeBackend: OpalEncoderForSistaV1.
CompilationContext usesFullBlockClosure: true.
OpalCompiler recompileAll.


Is there any value in doing so?


cheers

        holger


PS

(2) The intuitive explanation is that the cost of analysis is a lot lower the earlier the problem is detected. There are fewer potential changes that lead to the breakage and the people involved will remember what/why they did reducing the cost of fixing. Ideally things are found to be broken and fixed before integration.

Maybe an analogy is old-school aviation navigation.. the earlier a mistake is found in navigation the closer to the original course...

Hi clement

let us know how we can help because he all want the new closures.

Stef

On Fri, Sep 22, 2017 at 11:11 AM, Clément Bera <[hidden email]> wrote:
> I totally agree with you on this.
>
> For a while I had a CI job that was trying to execute Sista on the latest
> version of Pharo and VM, and sent me a mail when something had gone wrong. I
> disabled it when there was problems on the CI, I will make it work again.
> This way I am notified and I can update the Sista code as Pharo 7 is
> developed. That helps a lot.
>
> On the Pharo CI the time to run tests matters hence I will try to find
> another solution to test full block and the other bytecode set. I did not do
> that because I hope I can change Pharo to use full block by default soon,
> but since they're broken...
>
> Regards
>
>
>
> On Fri, Sep 22, 2017 at 10:58 AM, Holger Freyther <[hidden email]>
> wrote:
>>
>>
>> > On 22. Sep 2017, at 13:33, Clément Bera <[hidden email]> wrote:
>>
>>
>> Hi Clement,
>>
>>
>> > Sista will work for the release of Pharo 7 as it did for Pharo 6 and 6.1
>> > but it won't work every day during the alpha of Pharo 7 and no sane
>> > framework developer can guarantee that. Sista does not work currently in
>> > Pharo 7 due to some changes breaking the full block closures, I am going to
>> > fix it but even then in the incoming months there is a high chance that
>> > another change in Pharo 7 breaks it again, so I would never recommend to use
>> > it with the alpha.
>>
>>
>> thank you for your response. I didn't expect it to work (after all you
>> label it "Alpha" and I ran into issues already) but my intention for the
>> mail was a bit different. Let me try to briefly explain my motivation. In my
>> experience:
>>
>> (1) Software not executed stops to work (or never "worked")
>> (2) The earlier breakage is detected the less expensive it is to fix.
>>
>> My assumption is that Opal is part of Pharo and that we look at a
>> regression of the Opal compiler and/or how it interacts with the rest of the
>> image. As it might be easier to understand/fix it now than later I brought
>> it up. From my point of view I wonder if we want to add ~50s to the CI
>> process to execute:
>>
>> CompilationContext bytecodeBackend: OpalEncoderForSistaV1.
>> CompilationContext usesFullBlockClosure: true.
>> OpalCompiler recompileAll.
>>
>>
>> Is there any value in doing so?
>>
>>
>> cheers
>>
>>         holger
>>
>>
>> PS
>>
>> (2) The intuitive explanation is that the cost of analysis is a lot lower
>> the earlier the problem is detected. There are fewer potential changes that
>> lead to the breakage and the people involved will remember what/why they did
>> reducing the cost of fixing. Ideally things are found to be broken and fixed
>> before integration.
>>
>> Maybe an analogy is old-school aviation navigation.. the earlier a mistake
>> is found in navigation the closer to the original course...
>
>

Hi Clément, 

   If you need help with the CI, I can also help you. We can create it as a another parallel stage in the PR validation. So once it is stable we can detect asap changes that break it. 

Cheers, 

On 22 Sep 2017 18:45, "Clément Bera" <[hidden email]> wrote:

Well we need to use them but the integration process is difficult to deal with: we need to change the bootstrap to cross compile to the other bytecode set. I need to have some time to focus on it amd I need the help of someone understanding the bootstrap like Guille

On Sep 22, 2017 18:31, "Stephane Ducasse" <[hidden email]> wrote:

Hi clement

let us know how we can help because he all want the new closures.

Stef

On Fri, Sep 22, 2017 at 11:11 AM, Clément Bera <[hidden email]> wrote:
> I totally agree with you on this.
>
> For a while I had a CI job that was trying to execute Sista on the latest
> version of Pharo and VM, and sent me a mail when something had gone wrong. I
> disabled it when there was problems on the CI, I will make it work again.
> This way I am notified and I can update the Sista code as Pharo 7 is
> developed. That helps a lot.
>
> On the Pharo CI the time to run tests matters hence I will try to find
> another solution to test full block and the other bytecode set. I did not do
> that because I hope I can change Pharo to use full block by default soon,
> but since they're broken...
>
> Regards
>
>
>
> On Fri, Sep 22, 2017 at 10:58 AM, Holger Freyther <[hidden email]>
> wrote:
>>
>>
>> > On 22. Sep 2017, at 13:33, Clément Bera <[hidden email]> wrote:
>>
>>
>> Hi Clement,
>>
>>
>> > Sista will work for the release of Pharo 7 as it did for Pharo 6 and 6.1
>> > but it won't work every day during the alpha of Pharo 7 and no sane
>> > framework developer can guarantee that. Sista does not work currently in
>> > Pharo 7 due to some changes breaking the full block closures, I am going to
>> > fix it but even then in the incoming months there is a high chance that
>> > another change in Pharo 7 breaks it again, so I would never recommend to use
>> > it with the alpha.
>>
>>
>> thank you for your response. I didn't expect it to work (after all you
>> label it "Alpha" and I ran into issues already) but my intention for the
>> mail was a bit different. Let me try to briefly explain my motivation. In my
>> experience:
>>
>> (1) Software not executed stops to work (or never "worked")
>> (2) The earlier breakage is detected the less expensive it is to fix.
>>
>> My assumption is that Opal is part of Pharo and that we look at a
>> regression of the Opal compiler and/or how it interacts with the rest of the
>> image. As it might be easier to understand/fix it now than later I brought
>> it up. From my point of view I wonder if we want to add ~50s to the CI
>> process to execute:
>>
>> CompilationContext bytecodeBackend: OpalEncoderForSistaV1.
>> CompilationContext usesFullBlockClosure: true.
>> OpalCompiler recompileAll.
>>
>>
>> Is there any value in doing so?
>>
>>
>> cheers
>>
>>         holger
>>
>>
>> PS
>>
>> (2) The intuitive explanation is that the cost of analysis is a lot lower
>> the earlier the problem is detected. There are fewer potential changes that
>> lead to the breakage and the people involved will remember what/why they did
>> reducing the cost of fixing. Ideally things are found to be broken and fixed
>> before integration.
>>
>> Maybe an analogy is old-school aviation navigation.. the earlier a mistake
>> is found in navigation the closer to the original course...
>
>