[IMPORTANT] Starting migration to Spur VM

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[IMPORTANT] Starting migration to Spur VM

EstebanLM
We will start migration to Spur today.
To complete it, we will require some time, specially to adapt the CI and check everything is ok.

Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
- We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
- No more GC stops (noticeable when running large systems)
- We will be able to scale our systems up to 2G memory consumption without loosing performance.

But, this will have some drawbacks in the first times:

1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
        - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.

2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
        - there are some stuff not possible to compatibilise, notably:
                - Structures now need to inherit from FFIExternalStructure
                - Arrays now are now shadowed
        - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
        - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
2.1) ASMJIT will be removed from system and put in a separated packages.
NOTE: There will be a blog post explaining FFI-NB architecture during the week.

3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)

4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.

5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).


Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

EstebanLM

> On 14 Dec 2015, at 11:24, Esteban Lorenzano <[hidden email]> wrote:
>
> We will start migration to Spur today.
> To complete it, we will require some time, specially to adapt the CI and check everything is ok.
>
> Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
> - We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
> - No more GC stops (noticeable when running large systems)
> - We will be able to scale our systems up to 2G memory consumption without loosing performance.
>
> But, this will have some drawbacks in the first times:
>
> 1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
> - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.
>
> 2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
> - there are some stuff not possible to compatibilise, notably:
> - Structures now need to inherit from FFIExternalStructure
> - Arrays now are now shadowed
> - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
> - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
> 2.1) ASMJIT will be removed from system and put in a separated packages.
> NOTE: There will be a blog post explaining FFI-NB architecture during the week.
>
> 3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)
>
> 4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.
>
> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).
>

ah, I forget it!

6) Some changes to the VM accumulated during the time of transition (some months now). So, some changes introduced there will not be present in new VM. I will merge this changes in the next weeks.

cheers!
Esteban
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] [IMPORTANT] Starting migration to Spur VM

Guillermo Polito
Thanks!

> On 14 dic 2015, at 11:32 a.m., Esteban Lorenzano <[hidden email]> wrote:
>
>
>> On 14 Dec 2015, at 11:24, Esteban Lorenzano <[hidden email]> wrote:
>>
>> We will start migration to Spur today.
>> To complete it, we will require some time, specially to adapt the CI and check everything is ok.
>>
>> Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
>> - We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
>> - No more GC stops (noticeable when running large systems)
>> - We will be able to scale our systems up to 2G memory consumption without loosing performance.
>>
>> But, this will have some drawbacks in the first times:
>>
>> 1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
>> - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.
>>
>> 2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
>> - there are some stuff not possible to compatibilise, notably:
>> - Structures now need to inherit from FFIExternalStructure
>> - Arrays now are now shadowed
>> - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
>> - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
>> 2.1) ASMJIT will be removed from system and put in a separated packages.
>> NOTE: There will be a blog post explaining FFI-NB architecture during the week.
>>
>> 3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)
>>
>> 4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.
>>
>> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).
>>
>
> ah, I forget it!
>
> 6) Some changes to the VM accumulated during the time of transition (some months now). So, some changes introduced there will not be present in new VM. I will merge this changes in the next weeks.
>
> cheers!
> Esteban


Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] [IMPORTANT] Starting migration to Spur VM

kilon.alios
30% - 100% wow thats quite a speed boost, well done guys!
And 2GB images are more than welcomed , especially for me that I have to deal with 3d data.

Does that ThreadedFFIPlugin means we can now use C libraries that use threading ?

On Mon, Dec 14, 2015 at 1:01 PM Guillermo Polito <[hidden email]> wrote:
Thanks!

> On 14 dic 2015, at 11:32 a.m., Esteban Lorenzano <[hidden email]> wrote:
>
>
>> On 14 Dec 2015, at 11:24, Esteban Lorenzano <[hidden email]> wrote:
>>
>> We will start migration to Spur today.
>> To complete it, we will require some time, specially to adapt the CI and check everything is ok.
>>
>> Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
>> - We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
>> - No more GC stops (noticeable when running large systems)
>> - We will be able to scale our systems up to 2G memory consumption without loosing performance.
>>
>> But, this will have some drawbacks in the first times:
>>
>> 1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
>>      - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.
>>
>> 2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
>>      - there are some stuff not possible to compatibilise, notably:
>>              - Structures now need to inherit from FFIExternalStructure
>>              - Arrays now are now shadowed
>>      - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
>>      - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
>> 2.1) ASMJIT will be removed from system and put in a separated packages.
>> NOTE: There will be a blog post explaining FFI-NB architecture during the week.
>>
>> 3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)
>>
>> 4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.
>>
>> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).
>>
>
> ah, I forget it!
>
> 6) Some changes to the VM accumulated during the time of transition (some months now). So, some changes introduced there will not be present in new VM. I will merge this changes in the next weeks.
>
> cheers!
> Esteban


Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] [IMPORTANT] Starting migration to Spur VM

EstebanLM

On 14 Dec 2015, at 12:07, Dimitris Chloupis <[hidden email]> wrote:

30% - 100% wow thats quite a speed boost, well done guys!
And 2GB images are more than welcomed , especially for me that I have to deal with 3d data.

Does that ThreadedFFIPlugin means we can now use C libraries that use threading ?

not yet… but that is the path :)


On Mon, Dec 14, 2015 at 1:01 PM Guillermo Polito <[hidden email]> wrote:
Thanks!

> On 14 dic 2015, at 11:32 a.m., Esteban Lorenzano <[hidden email]> wrote:
>
>
>> On 14 Dec 2015, at 11:24, Esteban Lorenzano <[hidden email]> wrote:
>>
>> We will start migration to Spur today.
>> To complete it, we will require some time, specially to adapt the CI and check everything is ok.
>>
>> Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
>> - We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
>> - No more GC stops (noticeable when running large systems)
>> - We will be able to scale our systems up to 2G memory consumption without loosing performance.
>>
>> But, this will have some drawbacks in the first times:
>>
>> 1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
>>      - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.
>>
>> 2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
>>      - there are some stuff not possible to compatibilise, notably:
>>              - Structures now need to inherit from FFIExternalStructure
>>              - Arrays now are now shadowed
>>      - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
>>      - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
>> 2.1) ASMJIT will be removed from system and put in a separated packages.
>> NOTE: There will be a blog post explaining FFI-NB architecture during the week.
>>
>> 3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)
>>
>> 4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.
>>
>> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).
>>
>
> ah, I forget it!
>
> 6) Some changes to the VM accumulated during the time of transition (some months now). So, some changes introduced there will not be present in new VM. I will merge this changes in the next weeks.
>
> cheers!
> Esteban



Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] [IMPORTANT] Starting migration to Spur VM

kilon.alios
even more good news , thank you :)

On Mon, Dec 14, 2015 at 1:15 PM Esteban Lorenzano <[hidden email]> wrote:
On 14 Dec 2015, at 12:07, Dimitris Chloupis <[hidden email]> wrote:

30% - 100% wow thats quite a speed boost, well done guys!
And 2GB images are more than welcomed , especially for me that I have to deal with 3d data.

Does that ThreadedFFIPlugin means we can now use C libraries that use threading ?

not yet… but that is the path :)


On Mon, Dec 14, 2015 at 1:01 PM Guillermo Polito <[hidden email]> wrote:
Thanks!

> On 14 dic 2015, at 11:32 a.m., Esteban Lorenzano <[hidden email]> wrote:
>
>
>> On 14 Dec 2015, at 11:24, Esteban Lorenzano <[hidden email]> wrote:
>>
>> We will start migration to Spur today.
>> To complete it, we will require some time, specially to adapt the CI and check everything is ok.
>>
>> Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
>> - We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
>> - No more GC stops (noticeable when running large systems)
>> - We will be able to scale our systems up to 2G memory consumption without loosing performance.
>>
>> But, this will have some drawbacks in the first times:
>>
>> 1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
>>      - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.
>>
>> 2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
>>      - there are some stuff not possible to compatibilise, notably:
>>              - Structures now need to inherit from FFIExternalStructure
>>              - Arrays now are now shadowed
>>      - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
>>      - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
>> 2.1) ASMJIT will be removed from system and put in a separated packages.
>> NOTE: There will be a blog post explaining FFI-NB architecture during the week.
>>
>> 3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)
>>
>> 4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.
>>
>> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).
>>
>
> ah, I forget it!
>
> 6) Some changes to the VM accumulated during the time of transition (some months now). So, some changes introduced there will not be present in new VM. I will merge this changes in the next weeks.
>
> cheers!
> Esteban


Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

Ben Coman
In reply to this post by EstebanLM
This is great news. Thanks Eliot and Esteban.  From the 2GB limit I
take this to be 32-bit Spur?

Is it feasible to jump the image build numbers to make that first
image requiring a new VM align on a 100s build numbers.  I see we are
currently 50496.    50500 is nice and distinctive, but even even 50600
would be fine if the last few 400s are consumed in the meantime.


On Mon, Dec 14, 2015 at 6:24 PM, Esteban Lorenzano <[hidden email]> wrote:

> We will start migration to Spur today.
> To complete it, we will require some time, specially to adapt the CI and check everything is ok.
>
> Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
> - We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
> - No more GC stops (noticeable when running large systems)
> - We will be able to scale our systems up to 2G memory consumption without loosing performance.
>
> But, this will have some drawbacks in the first times:
>
> 1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
>         - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.
>
> 2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
>         - there are some stuff not possible to compatibilise, notably:
>                 - Structures now need to inherit from FFIExternalStructure
>                 - Arrays now are now shadowed
>         - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
>         - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
> 2.1) ASMJIT will be removed from system and put in a separated packages.
> NOTE: There will be a blog post explaining FFI-NB architecture during the week.
>
> 3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)
>
> 4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.
>
> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).

Maybe PharoLauncher can be a special hardcoded rule to have two
template headings...
  * Pharo 5.0 Spur (beta); and
  * Pharo 5.0 Pre-Spur (beta)
Enhanced multi-VM support perhaps based off the template headings would be nice.

cheers -ben

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

EstebanLM

> On 14 Dec 2015, at 12:21, Ben Coman <[hidden email]> wrote:
>
> This is great news. Thanks Eliot and Esteban.  From the 2GB limit I
> take this to be 32-bit Spur?

yes of course.
We are already working on 64bits spur but is not ready yet (it will be some point of 2016)

>
> Is it feasible to jump the image build numbers to make that first
> image requiring a new VM align on a 100s build numbers.  I see we are
> currently 50496.    50500 is nice and distinctive, but even even 50600
> would be fine if the last few 400s are consumed in the meantime.

mmm… that I don’t know… correlation of version numbers is usually important for us (for statistics purposes)… also changing number generation is more annoying than you would think (that’s why we don’t  have 4.1, 4.2, etc.).
but well… everything is possible, we’ll see :)

Esteban

>
>
> On Mon, Dec 14, 2015 at 6:24 PM, Esteban Lorenzano <[hidden email]> wrote:
>> We will start migration to Spur today.
>> To complete it, we will require some time, specially to adapt the CI and check everything is ok.
>>
>> Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
>> - We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
>> - No more GC stops (noticeable when running large systems)
>> - We will be able to scale our systems up to 2G memory consumption without loosing performance.
>>
>> But, this will have some drawbacks in the first times:
>>
>> 1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
>>        - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.
>>
>> 2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
>>        - there are some stuff not possible to compatibilise, notably:
>>                - Structures now need to inherit from FFIExternalStructure
>>                - Arrays now are now shadowed
>>        - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
>>        - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
>> 2.1) ASMJIT will be removed from system and put in a separated packages.
>> NOTE: There will be a blog post explaining FFI-NB architecture during the week.
>>
>> 3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)
>>
>> 4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.
>>
>> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).
>
> Maybe PharoLauncher can be a special hardcoded rule to have two
> template headings...
>  * Pharo 5.0 Spur (beta); and
>  * Pharo 5.0 Pre-Spur (beta)
> Enhanced multi-VM support perhaps based off the template headings would be nice.
>
> cheers -ben
>


Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

Ben Coman
On Mon, Dec 14, 2015 at 7:27 PM, Esteban Lorenzano <[hidden email]> wrote:

>
>> On 14 Dec 2015, at 12:21, Ben Coman <[hidden email]> wrote:
>>
>> This is great news. Thanks Eliot and Esteban.  From the 2GB limit I
>> take this to be 32-bit Spur?
>
> yes of course.
> We are already working on 64bits spur but is not ready yet (it will be some point of 2016)
>
>>
>> Is it feasible to jump the image build numbers to make that first
>> image requiring a new VM align on a 100s build numbers.  I see we are
>> currently 50496.    50500 is nice and distinctive, but even even 50600
>> would be fine if the last few 400s are consumed in the meantime.
>
> mmm… that I don’t know… correlation of version numbers is usually important for us (for statistics purposes)… also changing number generation is more annoying than you would think (that’s why we don’t  have 4.1, 4.2, etc.).
> but well… everything is possible, we’ll see :)

Or do we just append "-spur" to the build number - but there will be
no new builds without the "-spur" tag, and continue this up to Pharo 5
Release.  I am just thinking of making it easy to distinguish the
required VM when I want to open old builds to bisect which build
instorduced a bug.

cheer -ben

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

EstebanLM

On 14 Dec 2015, at 13:03, Ben Coman <[hidden email]> wrote:

On Mon, Dec 14, 2015 at 7:27 PM, Esteban Lorenzano <[hidden email]> wrote:

On 14 Dec 2015, at 12:21, Ben Coman <[hidden email]> wrote:

This is great news. Thanks Eliot and Esteban.  From the 2GB limit I
take this to be 32-bit Spur?

yes of course.
We are already working on 64bits spur but is not ready yet (it will be some point of 2016)


Is it feasible to jump the image build numbers to make that first
image requiring a new VM align on a 100s build numbers.  I see we are
currently 50496.    50500 is nice and distinctive, but even even 50600
would be fine if the last few 400s are consumed in the meantime.

mmm… that I don’t know… correlation of version numbers is usually important for us (for statistics purposes)… also changing number generation is more annoying than you would think (that’s why we don’t  have 4.1, 4.2, etc.).
but well… everything is possible, we’ll see :)

Or do we just append "-spur" to the build number - but there will be
no new builds without the "-spur" tag, and continue this up to Pharo 5
Release.  I am just thinking of making it easy to distinguish the
required VM when I want to open old builds to bisect which build
instorduced a bug.

again, harder than it seems (because releasing a version is stupidly-hard)… I’m thinking on move all 50 pre-spur into a 49 directory (4.9), then all is correctly separated in file server. 
This is the easiest I can do it… I think :)

Esteban


cheer -ben

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

Ben Coman
On Mon, Dec 14, 2015 at 8:08 PM, Esteban Lorenzano <[hidden email]> wrote:
> again, harder than it seems (because releasing a version is stupidly-hard)…
> I’m thinking on move all 50 pre-spur into a 49 directory (4.9), then all is
> correctly separated in file server.
> This is the easiest I can do it… I think :)

That sounds fine, but maybe the directory can be "50preSpur" (if possible) ?
cheers -ben

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

EstebanLM

> On 14 Dec 2015, at 13:15, Ben Coman <[hidden email]> wrote:
>
> On Mon, Dec 14, 2015 at 8:08 PM, Esteban Lorenzano <[hidden email]> wrote:
>> again, harder than it seems (because releasing a version is stupidly-hard)…
>> I’m thinking on move all 50 pre-spur into a 49 directory (4.9), then all is
>> correctly separated in file server.
>> This is the easiest I can do it… I think :)
>
> That sounds fine, but maybe the directory can be "50preSpur" (if possible) ?

yeah, it could.

> cheers -ben
>


Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] [IMPORTANT] Starting migration to Spur VM

Tudor Girba-2
In reply to this post by EstebanLM
Thank you Eliot & Esteban for making this happen!

Exciting times :)

Cheers,
Doru



> On Dec 14, 2015, at 11:24 AM, Esteban Lorenzano <[hidden email]> wrote:
>
> We will start migration to Spur today.
> To complete it, we will require some time, specially to adapt the CI and check everything is ok.
>
> Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
> - We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
> - No more GC stops (noticeable when running large systems)
> - We will be able to scale our systems up to 2G memory consumption without loosing performance.
>
> But, this will have some drawbacks in the first times:
>
> 1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
> - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.
>
> 2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
> - there are some stuff not possible to compatibilise, notably:
> - Structures now need to inherit from FFIExternalStructure
> - Arrays now are now shadowed
> - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
> - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
> 2.1) ASMJIT will be removed from system and put in a separated packages.
> NOTE: There will be a blog post explaining FFI-NB architecture during the week.
>
> 3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)
>
> 4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.
>
> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).
>
>

--
www.tudorgirba.com
www.feenk.com

"Presenting is storytelling."


Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

jfabry
In reply to this post by EstebanLM

Excellent work, thanks a lot !!

> On Dec 14, 2015, at 07:24, Esteban Lorenzano <[hidden email]> wrote:
>
> We will start migration to Spur today.
> To complete it, we will require some time, specially to adapt the CI and check everything is ok.
>
> Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
> - We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
> - No more GC stops (noticeable when running large systems)
> - We will be able to scale our systems up to 2G memory consumption without loosing performance.
>
> But, this will have some drawbacks in the first times:
>
> 1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
> - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.
>
> 2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
> - there are some stuff not possible to compatibilise, notably:
> - Structures now need to inherit from FFIExternalStructure
> - Arrays now are now shadowed
> - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
> - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
> 2.1) ASMJIT will be removed from system and put in a separated packages.
> NOTE: There will be a blog post explaining FFI-NB architecture during the week.
>
> 3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)
>
> 4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.
>
> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).
>
>
>



---> Save our in-boxes! http://emailcharter.org <---

Johan Fabry   -   http://pleiad.cl/~jfabry
PLEIAD and RyCh labs  -  Computer Science Department (DCC)  -  University of Chile


Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

kilon.alios
Pharo 5 your mission should you choose to accept it is to integrate with Spur

this image will self destruct in

5

4

3

2

1

JAVA !

On Mon, Dec 14, 2015 at 4:15 PM Johan Fabry <[hidden email]> wrote:

Excellent work, thanks a lot !!

> On Dec 14, 2015, at 07:24, Esteban Lorenzano <[hidden email]> wrote:
>
> We will start migration to Spur today.
> To complete it, we will require some time, specially to adapt the CI and check everything is ok.
>
> Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
> - We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
> - No more GC stops (noticeable when running large systems)
> - We will be able to scale our systems up to 2G memory consumption without loosing performance.
>
> But, this will have some drawbacks in the first times:
>
> 1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
>       - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.
>
> 2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
>       - there are some stuff not possible to compatibilise, notably:
>               - Structures now need to inherit from FFIExternalStructure
>               - Arrays now are now shadowed
>       - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
>       - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
> 2.1) ASMJIT will be removed from system and put in a separated packages.
> NOTE: There will be a blog post explaining FFI-NB architecture during the week.
>
> 3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)
>
> 4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.
>
> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).
>
>
>



---> Save our in-boxes! http://emailcharter.org <---

Johan Fabry   -   http://pleiad.cl/~jfabry
PLEIAD and RyCh labs  -  Computer Science Department (DCC)  -  University of Chile


Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

Damien Cassou-2
In reply to this post by Ben Coman
Ben Coman <[hidden email]> writes:
>> 5) You will need to adapt your Pharo 5.0 related apps and CI
>> processes (to take care about new VM). Some programs will stop work
>> at all (for example, I think Pharo-launcher will need to be adapted).
>
> Maybe PharoLauncher can be a special hardcoded rule to have two
> template headings...
>   * Pharo 5.0 Spur (beta); and
>   * Pharo 5.0 Pre-Spur (beta)
> Enhanced multi-VM support perhaps based off the template headings would be nice.


what happens when executing the VM on an incompatible image (new VM on
old image, old VM on new image)? If the launcher can detect the problem,
it can also try to open the selected image with a different VM.

Launcher2 will be more flexible, but I need someone to help.

--
Damien Cassou
http://damiencassou.seasidehosting.st

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

Martin Bähr
Excerpts from Damien Cassou's message of 2015-12-14 16:25:15 +0100:
> Launcher2 will be more flexible, but I need someone to help.

what kind of help do you need?

greetings, martin.

--
eKita                   -   the online platform for your entire academic life
--
chief engineer                                                       eKita.co
pike programmer      pike.lysator.liu.se    caudium.net     societyserver.org
secretary                                                      beijinglug.org
mentor                                                           fossasia.org
foresight developer  foresightlinux.org                            realss.com
unix sysadmin
Martin Bähr          working in china        http://societyserver.org/mbaehr/

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

Damien Cassou-2
Martin Bähr <[hidden email]> writes:

> Excerpts from Damien Cassou's message of 2015-12-14 16:25:15 +0100:
>> Launcher2 will be more flexible, but I need someone to help.
>
> what kind of help do you need?

- I need people to help me think about the new GUI design and
  interactions (I have some mockups but they need some love)

- I need people to extract pieces of current launcher into external
  packages (I already created a Jenkins Pharo project
  https://github.com/DamienCassou/pharo-jenkins)

- I need someone to leverage Cocoon to specify all launcher settings
  users might want to tweak (proxy configuration, visible jenkins
  servers, location of images and cache...)
  http://www.smalltalkhub.com/#!/PharoExtras/Cocoon

- I need documentation about Brick

- I need people to design a CLI interface to the launcher

- ...

--
Damien Cassou
http://damiencassou.seasidehosting.st

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] [IMPORTANT] Starting migration to Spur VM

Sven Van Caekenberghe-2
In reply to this post by Tudor Girba-2

> On 14 Dec 2015, at 14:14, Tudor Girba <[hidden email]> wrote:
>
> Thank you Eliot & Esteban for making this happen!

Yes!

> Exciting times :)

Yes!

> Cheers,
> Doru
>
>
>
>> On Dec 14, 2015, at 11:24 AM, Esteban Lorenzano <[hidden email]> wrote:
>>
>> We will start migration to Spur today.
>> To complete it, we will require some time, specially to adapt the CI and check everything is ok.
>>
>> Spur will allow us to do a big step forward in Pharo development, in the concrete you will see it immediately for this:
>> - We have noticed a speed increment of 100% in tiny benchmarks (and according to Eliot, it will be at least 35% in general on the system).
>> - No more GC stops (noticeable when running large systems)
>> - We will be able to scale our systems up to 2G memory consumption without loosing performance.
>>
>> But, this will have some drawbacks in the first times:
>>
>> 1) VM will not be compatible between versions anymore: Pharo 5.0 will have a Pharo Spur VM associated (and they are not compatible).
>> - There WILL NOT be a "non-spur" version of Pharo 5.0. Once completed the transitions, this will be the only one.
>>
>> 2) NativeBoost-FFI implementation has been replaced with a new implementation who relies in ThreadedFFIPlugin and IA32Plugin. While we worked a lot to do this transition as painless as possible and we achieve a good level of backward compatibility (most uses of #nbCall: should work out of the box), there are some problems we cannot solve:
>> - there are some stuff not possible to compatibilise, notably:
>> - Structures now need to inherit from FFIExternalStructure
>> - Arrays now are now shadowed
>> - in general is a bit slower (impossible to compete with ASM) but in general is not perceptible.
>> - THERE WILL BE BUGS AND NON-IMPLEMENTED FEATURES: Current implementation is validated with Athens and even Roassal was working, but of course that does not covers all cases.
>> 2.1) ASMJIT will be removed from system and put in a separated packages.
>> NOTE: There will be a blog post explaining FFI-NB architecture during the week.
>>
>> 3) There are more or less 70 new failing tests, some of them important than we need to fix as soon as possible. Please, please, please, help us with them :)
>>
>> 4) In general we foresee the system will became unstable some weeks, before it gets back to normal. Please be patient.
>>
>> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM). Some programs will stop work at all (for example, I think Pharo-launcher will need to be adapted).
>>
>>
>
> --
> www.tudorgirba.com
> www.feenk.com
>
> "Presenting is storytelling."
>
>


Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Starting migration to Spur VM

jfabry
In reply to this post by EstebanLM

Sorry for the stupid question, but how should I adapt my CI job? I saw that the last build failed with no tests being run, so I guess this was due to the VM being changed.

> On Dec 14, 2015, at 07:24, Esteban Lorenzano <[hidden email]> wrote:
>
> 5) You will need to adapt your Pharo 5.0 related apps and CI processes (to take care about new VM).



---> Save our in-boxes! http://emailcharter.org <---

Johan Fabry   -   http://pleiad.cl/~jfabry
PLEIAD and RyCh labs  -  Computer Science Department (DCC)  -  University of Chile


12