snafu for e013b6766a05037812d48960702c0910

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

snafu for e013b6766a05037812d48960702c0910

Eliot Miranda-2
 
Hi All,

    mea culpa.  Yesterday I managed to generate completely broken sources and broke CI servers etc.  I've realized what went wrong.  For the past week I've been doing VM development in a 64-bit Squeak image (*) and yesterday for the first time I generated sources from the 64-bit image.  There was old very broken code in VMMaker:

VMMaker>>initialize
logger := Transcript.
inline := true.
forBrowser := false.
internalPlugins := SortedCollection new.
externalPlugins := SortedCollection new.
platformName := self class machinesDirName.
>> is64BitVM := Smalltalk wordSize == 8.
interpreterClassName := Interpreter name.
optionsDictionary := Dictionary new.
>> optionsDictionary at: #BytesPerWord put: Smalltalk wordSize.
VMStructType voidStructTypeCache

which caused mayhem.  In particular, invalid types were inferred for plugin files.  These files ahem to be generated choosing types such that the generated files work on both 64 and 32-bit.  That means /not/ initializing the VMMaker instance as above.  

I'm not sure when things will be back to normal but it should be soon, given that I finally understand what went wrong.  Apologies for all the breakages.


(*) I was looking at Juan Vuletich's 64-bit scavenger crash due to remember set overflow that Clément fixed recently.  I was adding logging to the scavenger so we can better analyse failures like this and better tune the GC.  Simulation of a 64-bit image is somewhat quicker in 64-bits (less work synthesizing 64-bit values from the heap) and so I thought I'd try developing in 64-bits.
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: snafu for e013b6766a05037812d48960702c0910

Holger Freyther
 

> On 16. Nov 2017, at 07:18, Eliot Miranda <[hidden email]> wrote:
>
> Hi All,

Hey!

I was looking at a OpenSSL issue and master with spur seems to be broken, is it because of the same issue? What would it take to have everyone use Pull Requests and benefit from pre-merge testing? These days computing time is so cheap... (think of a reference to the mythical man month)

cheers

        holger




>
>     mea culpa.  Yesterday I managed to generate completely broken sources and broke CI servers etc.  I've realized what went wrong.  For the past week I've been doing VM development in a 64-bit Squeak image (*) and yesterday for the first time I generated sources from the 64-bit image.  There was old very broken code in VMMaker:
>
> VMMaker>>initialize
> logger := Transcript.
> inline := true.
> forBrowser := false.
> internalPlugins := SortedCollection new.
> externalPlugins := SortedCollection new.
> platformName := self class machinesDirName.
> >> is64BitVM := Smalltalk wordSize == 8.
> interpreterClassName := Interpreter name.
> optionsDictionary := Dictionary new.
> >> optionsDictionary at: #BytesPerWord put: Smalltalk wordSize.
> VMStructType voidStructTypeCache
>
> which caused mayhem.  In particular, invalid types were inferred for plugin files.  These files ahem to be generated choosing types such that the generated files work on both 64 and 32-bit.  That means /not/ initializing the VMMaker instance as above.  
>
> I'm not sure when things will be back to normal but it should be soon, given that I finally understand what went wrong.  Apologies for all the breakages.
>
>
> (*) I was looking at Juan Vuletich's 64-bit scavenger crash due to remember set overflow that Clément fixed recently.  I was adding logging to the scavenger so we can better analyse failures like this and better tune the GC.  Simulation of a 64-bit image is somewhat quicker in 64-bits (less work synthesizing 64-bit values from the heap) and so I thought I'd try developing in 64-bits.
> _,,,^..^,,,_
> best, Eliot

Reply | Threaded
Open this post in threaded view
|

Re: snafu for e013b6766a05037812d48960702c0910

Ben Coman
 


On 24 November 2017 at 12:18, Holger Freyther <[hidden email]> wrote:


> On 16. Nov 2017, at 07:18, Eliot Miranda <[hidden email]> wrote:
>
> Hi All,

Hey!

I was looking at a OpenSSL issue and master with spur seems to be broken, is it because of the same issue? What would it take to have everyone use Pull Requests and benefit from pre-merge testing? These days computing time is so cheap... (think of a reference to the mythical man month)

cheers

        holger

I haven't used them, but there seem a few options for issuing PRs from the command-line, which may make this more palatable.
Thus the CI should run without needing to visit github's web-UI.

https://hub.github.com/     (search for "pull")


cheers -ben

 




>
>     mea culpa.  Yesterday I managed to generate completely broken sources and broke CI servers etc.  I've realized what went wrong.  For the past week I've been doing VM development in a 64-bit Squeak image (*) and yesterday for the first time I generated sources from the 64-bit image.  There was old very broken code in VMMaker:
>
> VMMaker>>initialize
>       logger := Transcript.
>       inline := true.
>       forBrowser := false.
>       internalPlugins := SortedCollection new.
>       externalPlugins := SortedCollection new.
>       platformName := self class machinesDirName.
> >>    is64BitVM := Smalltalk wordSize == 8.
>       interpreterClassName := Interpreter name.
>       optionsDictionary := Dictionary new.
> >>    optionsDictionary at: #BytesPerWord put: Smalltalk wordSize.
>       VMStructType voidStructTypeCache
>
> which caused mayhem.  In particular, invalid types were inferred for plugin files.  These files ahem to be generated choosing types such that the generated files work on both 64 and 32-bit.  That means /not/ initializing the VMMaker instance as above.
>
> I'm not sure when things will be back to normal but it should be soon, given that I finally understand what went wrong.  Apologies for all the breakages.
>
>
> (*) I was looking at Juan Vuletich's 64-bit scavenger crash due to remember set overflow that Clément fixed recently.  I was adding logging to the scavenger so we can better analyse failures like this and better tune the GC.  Simulation of a 64-bit image is somewhat quicker in 64-bits (less work synthesizing 64-bit values from the heap) and so I thought I'd try developing in 64-bits.
> _,,,^..^,,,_
> best, Eliot


Reply | Threaded
Open this post in threaded view
|

Re: snafu for e013b6766a05037812d48960702c0910

Holger Freyther
 

> On 24. Nov 2017, at 13:27, Ben Coman <[hidden email]> wrote:
>

Hey!


>
> I haven't used them, but there seem a few options for issuing PRs from the command-line, which may make this more palatable.
> Thus the CI should run without needing to visit github's web-UI.


and one can probably use the github API to auto-merge PRs that have a successful build and come from a trusted developer. It is nice to live in a time computing time is so cheap. ;)

holger
Reply | Threaded
Open this post in threaded view
|

Re: snafu for e013b6766a05037812d48960702c0910

Nicolas Cellier
 
+1 for using generous features of github.
creating branches is a cheap action for developpers

I also see another problem: we use two different paths for developping the VM
- via VMMaker (slang) changes and VM simulator tests
- via code generation from VMMaker, compilation of VM, and non regression testing by passing some image SUnit test cases

Only the latest form is on Opensmalltalk-vm (github) right now.
This can lead to long periods without github/travis tests.
In our case see:
there's been no travis regression tests from VMMaker.oscog-eem.2262 to VMMaker.oscog-eem.2277 for the JIT
and from VMMaker.oscog-eem.2266 to VMMaker.oscog-eem.2277 for the interpreter...
That's a lot of changes at once.

In particular, each and every change to CCodeGeneration SHALL be tested individually on github/travis, because it is precisely what can make a difference between the VM simulator and the C code.

So having the code generated from VMMaker committed into a specific branch as already proposed might help bissecting the problems...



2017-11-24 7:47 GMT+01:00 Holger Freyther <[hidden email]>:


> On 24. Nov 2017, at 13:27, Ben Coman <[hidden email]> wrote:
>

Hey!


>
> I haven't used them, but there seem a few options for issuing PRs from the command-line, which may make this more palatable.
> Thus the CI should run without needing to visit github's web-UI.


and one can probably use the github API to auto-merge PRs that have a successful build and come from a trusted developer. It is nice to live in a time computing time is so cheap. ;)

holger