pharo bootstrap is in inconsistent state

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

pharo bootstrap is in inconsistent state

alistairgrant
Hi Pablo & Everyone,

The bootstrap process seems to be in an inconsistent state at the
moment as the scripts in:

- Jenkinsfile   and
- bootstrap/scripts/bootstrap.sh

don't match.

bootstrap/scripts/bootstrap.sh has an error in it:

./pharo Pharo.image eval "PBBootstrap fromCommandLine bootstrap" --quit

fails because the "--quit" option is valid for the st handler
(executing a smalltalk file), but isn't recognised by the eval
handler.

Can someone, Pablo?, confirm what the status is, what it should be and
what the next steps are? :-)

Thanks,
Alistair

Reply | Threaded
Open this post in threaded view
|

Re: pharo bootstrap is in inconsistent state

tesonep@gmail.com
Hi, 
     sorry my mistake when updating the scripts. As the bootstrap.sh is not executed during the Jenkins process it has not been validated (locally I'm not using that script, because is trying to download each time the VM and the bootstraping process image) 
The correct line is the one from the JenkinsFile, it should be something like:

./pharo ./Pharo.image bootstrap/scripts/bootstrap.st --ARCH=${architecture} --BUILD_NUMBER=${env.BUILD_ID} --quit

I will submit a PR to fix it, as soon as the CI infrastructure is less on fire. 
Until the fix, if you are building the image locally you can change the error line with:

./pharo ./Pharo.image bootstrap/scripts/bootstrap.st --quit

And it will work. 

The idea is to migrate more parts of the process to pure Pharo code, everything is implemented in a project PharoBootstrapProcess, which is located in bootstrap/src in the same Pharo repository.
The best way to check what is there is to do the same as the process. 

1) Take an image with VMMaker, Espell and Ficus (we have them already made in https://github.com/guillep/PharoBootstrap/releases/download/v1.4/bootstrapImage.zip).
2) Execute bootstrap/scripts/prepare_image.st

That is the same image used to bootstrap Pharo. 
I will do another pass on this first part of the process and generate some documentation about the changes and how to modify the whole process.
We decided to migrate the process by parts to be able to review it easily than waiting to have a whole migration of the process.

The next step is to do the same with the build.sh file, as you can see it, is a little complex and have lots of steps. 
The idea is again convert them into Pharo code.

Again sorry if the partial change made some problems in your local execution of the bootstrap.

Cheers. 


On Fri, Mar 16, 2018 at 2:02 PM, Alistair Grant <[hidden email]> wrote:
Hi Pablo & Everyone,

The bootstrap process seems to be in an inconsistent state at the
moment as the scripts in:

- Jenkinsfile   and
- bootstrap/scripts/bootstrap.sh

don't match.

bootstrap/scripts/bootstrap.sh has an error in it:

./pharo Pharo.image eval "PBBootstrap fromCommandLine bootstrap" --quit

fails because the "--quit" option is valid for the st handler
(executing a smalltalk file), but isn't recognised by the eval
handler.

Can someone, Pablo?, confirm what the status is, what it should be and
what the next steps are? :-)

Thanks,
Alistair




--
Pablo Tesone.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: pharo bootstrap is in inconsistent state

alistairgrant
Hi Pablo,

Thanks for your quick reply.

On 16 March 2018 at 14:38, [hidden email] <[hidden email]> wrote:
> Hi,
>      sorry my mistake when updating the scripts. As the bootstrap.sh is not
> executed during the Jenkins process it has not been validated (locally I'm
> not using that script, because is trying to download each time the VM and
> the bootstraping process image)

Right.  Can we make one of the goals of the refactoring to avoid any
code duplication?

e.g. at the moment (ignoring changes proposed below), Jenkins should
be calling bootstrap.sh.


> The correct line is the one from the JenkinsFile, it should be something
> like:
>
> ./pharo ./Pharo.image bootstrap/scripts/bootstrap.st --ARCH=${architecture}
> --BUILD_NUMBER=${env.BUILD_ID} --quit
>
> I will submit a PR to fix it, as soon as the CI infrastructure is less on
> fire.
> Until the fix, if you are building the image locally you can change the
> error line with:
>
> ./pharo ./Pharo.image bootstrap/scripts/bootstrap.st --quit
>
> And it will work.

Shall do.


> The idea is to migrate more parts of the process to pure Pharo code,

+1


> everything is implemented in a project PharoBootstrapProcess, which is
> located in bootstrap/src in the same Pharo repository.
> The best way to check what is there is to do the same as the process.
> 1) Take an image with VMMaker, Espell and Ficus (we have them already made
> in
> https://github.com/guillep/PharoBootstrap/releases/download/v1.4/bootstrapImage.zip).
> 2) Execute bootstrap/scripts/prepare_image.st
>
> That is the same image used to bootstrap Pharo.
> I will do another pass on this first part of the process and generate some
> documentation about the changes and how to modify the whole process.
> We decided to migrate the process by parts to be able to review it easily
> than waiting to have a whole migration of the process.
>
> The next step is to do the same with the build.sh file, as you can see it,
> is a little complex and have lots of steps.
> The idea is again convert them into Pharo code.

Can we also re-order the bootstrap and break it into several distinct phases.

1. Clean up (this ensures a clean environment for Jenkins)
2. Download all pre-requisites.
    The main change here is moving the call to download_vm.sh out of build.sh.
    This would allow a particular VM to be installed by hand for step
4, see below.
3. Initial preparation, i.e. all the execution prior to build.sh, e.g.
prepare_image.st.
   Currently this is done with the Pharo 6 VM and image, which makes sense as it
   provides stability.
4. Main image bootstrap.  Which is done with the Pharo 7 VM (currently
what is in build.sh).

That will allow the first three steps to be run once and then skipped
when doing personal development / testing.

I want to be able to manually load the VM to be used in step 4 as part
of some stability testing so we can promote a new Pharo 7 VM to
stable.

Are you open to me submitting some PRs that do this restructure to the
phases listed above and removing duplicate code from Jenkinsfile?


> Again sorry if the partial change made some problems in your local execution
> of the bootstrap.

No problem.

Thanks,
Alistair



> Cheers.
>
>
> On Fri, Mar 16, 2018 at 2:02 PM, Alistair Grant <[hidden email]>
> wrote:
>>
>> Hi Pablo & Everyone,
>>
>> The bootstrap process seems to be in an inconsistent state at the
>> moment as the scripts in:
>>
>> - Jenkinsfile   and
>> - bootstrap/scripts/bootstrap.sh
>>
>> don't match.
>>
>> bootstrap/scripts/bootstrap.sh has an error in it:
>>
>> ./pharo Pharo.image eval "PBBootstrap fromCommandLine bootstrap" --quit
>>
>> fails because the "--quit" option is valid for the st handler
>> (executing a smalltalk file), but isn't recognised by the eval
>> handler.
>>
>> Can someone, Pablo?, confirm what the status is, what it should be and
>> what the next steps are? :-)
>>
>> Thanks,
>> Alistair
>>
>
>
>
> --
> Pablo Tesone.
> [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: pharo bootstrap is in inconsistent state

tesonep@gmail.com


On Fri, Mar 16, 2018 at 3:26 PM, Alistair Grant <[hidden email]> wrote:
Hi Pablo,

Thanks for your quick reply.

On 16 March 2018 at 14:38, [hidden email] <[hidden email]> wrote:
> Hi,
>      sorry my mistake when updating the scripts. As the bootstrap.sh is not
> executed during the Jenkins process it has not been validated (locally I'm
> not using that script, because is trying to download each time the VM and
> the bootstraping process image)

Right.  Can we make one of the goals of the refactoring to avoid any
code duplication?

e.g. at the moment (ignoring changes proposed below), Jenkins should
be calling bootstrap.sh.


Yes that is part of the plan.
 

> The correct line is the one from the JenkinsFile, it should be something
> like:
>
> ./pharo ./Pharo.image bootstrap/scripts/bootstrap.st --ARCH=${architecture}
> --BUILD_NUMBER=${env.BUILD_ID} --quit
>
> I will submit a PR to fix it, as soon as the CI infrastructure is less on
> fire.
> Until the fix, if you are building the image locally you can change the
> error line with:
>
> ./pharo ./Pharo.image bootstrap/scripts/bootstrap.st --quit
>
> And it will work.

Shall do.


> The idea is to migrate more parts of the process to pure Pharo code,

+1


> everything is implemented in a project PharoBootstrapProcess, which is
> located in bootstrap/src in the same Pharo repository.
> The best way to check what is there is to do the same as the process.
> 1) Take an image with VMMaker, Espell and Ficus (we have them already made
> in
> https://github.com/guillep/PharoBootstrap/releases/download/v1.4/bootstrapImage.zip).
> 2) Execute bootstrap/scripts/prepare_image.st
>
> That is the same image used to bootstrap Pharo.
> I will do another pass on this first part of the process and generate some
> documentation about the changes and how to modify the whole process.
> We decided to migrate the process by parts to be able to review it easily
> than waiting to have a whole migration of the process.
>
> The next step is to do the same with the build.sh file, as you can see it,
> is a little complex and have lots of steps.
> The idea is again convert them into Pharo code.

Can we also re-order the bootstrap and break it into several distinct phases.

1. Clean up (this ensures a clean environment for Jenkins)
2. Download all pre-requisites.
    The main change here is moving the call to download_vm.sh out of build.sh.
    This would allow a particular VM to be installed by hand for step
4, see below.
3. Initial preparation, i.e. all the execution prior to build.sh, e.g.
prepare_image.st.
   Currently this is done with the Pharo 6 VM and image, which makes sense as it
   provides stability.
4. Main image bootstrap.  Which is done with the Pharo 7 VM (currently
what is in build.sh).

That will allow the first three steps to be run once and then skipped
when doing personal development / testing.

I want to be able to manually load the VM to be used in step 4 as part
of some stability testing so we can promote a new Pharo 7 VM to
stable.

Are you open to me submitting some PRs that do this restructure to the
phases listed above and removing duplicate code from Jenkinsfile?

Yes of course. Thanks 

> Again sorry if the partial change made some problems in your local execution
> of the bootstrap.

No problem.

Thanks,
Alistair



> Cheers.
>
>
> On Fri, Mar 16, 2018 at 2:02 PM, Alistair Grant <[hidden email]>
> wrote:
>>
>> Hi Pablo & Everyone,
>>
>> The bootstrap process seems to be in an inconsistent state at the
>> moment as the scripts in:
>>
>> - Jenkinsfile   and
>> - bootstrap/scripts/bootstrap.sh
>>
>> don't match.
>>
>> bootstrap/scripts/bootstrap.sh has an error in it:
>>
>> ./pharo Pharo.image eval "PBBootstrap fromCommandLine bootstrap" --quit
>>
>> fails because the "--quit" option is valid for the st handler
>> (executing a smalltalk file), but isn't recognised by the eval
>> handler.
>>
>> Can someone, Pablo?, confirm what the status is, what it should be and
>> what the next steps are? :-)
>>
>> Thanks,
>> Alistair
>>
>
>
>
> --
> Pablo Tesone.
> [hidden email]




--
Pablo Tesone.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: pharo bootstrap is in inconsistent state

alistairgrant
Hi Pablo,

On 16 March 2018 at 15:32, [hidden email] <[hidden email]> wrote:

>
>
> On Fri, Mar 16, 2018 at 3:26 PM, Alistair Grant <[hidden email]>
> wrote:
>>
>> Hi Pablo,
>>
>> Thanks for your quick reply.
>>
>> On 16 March 2018 at 14:38, [hidden email] <[hidden email]> wrote:
>> > Hi,
>> >      sorry my mistake when updating the scripts. As the bootstrap.sh is
>> > not
>> > executed during the Jenkins process it has not been validated (locally
>> > I'm
>> > not using that script, because is trying to download each time the VM
>> > and
>> > the bootstraping process image)
>>
>> Right.  Can we make one of the goals of the refactoring to avoid any
>> code duplication?
>>
>> e.g. at the moment (ignoring changes proposed below), Jenkins should
>> be calling bootstrap.sh.
>>
>
> Yes that is part of the plan.
>
>>
>>
>> > The correct line is the one from the JenkinsFile, it should be something
>> > like:
>> >
>> > ./pharo ./Pharo.image bootstrap/scripts/bootstrap.st
>> > --ARCH=${architecture}
>> > --BUILD_NUMBER=${env.BUILD_ID} --quit
>> >
>> > I will submit a PR to fix it, as soon as the CI infrastructure is less
>> > on
>> > fire.
>> > Until the fix, if you are building the image locally you can change the
>> > error line with:
>> >
>> > ./pharo ./Pharo.image bootstrap/scripts/bootstrap.st --quit
>> >
>> > And it will work.
>>
>> Shall do.
>>
>>
>> > The idea is to migrate more parts of the process to pure Pharo code,
>>
>> +1
>>
>>
>> > everything is implemented in a project PharoBootstrapProcess, which is
>> > located in bootstrap/src in the same Pharo repository.
>> > The best way to check what is there is to do the same as the process.
>> > 1) Take an image with VMMaker, Espell and Ficus (we have them already
>> > made
>> > in
>> >
>> > https://github.com/guillep/PharoBootstrap/releases/download/v1.4/bootstrapImage.zip).
>> > 2) Execute bootstrap/scripts/prepare_image.st
>> >
>> > That is the same image used to bootstrap Pharo.
>> > I will do another pass on this first part of the process and generate
>> > some
>> > documentation about the changes and how to modify the whole process.
>> > We decided to migrate the process by parts to be able to review it
>> > easily
>> > than waiting to have a whole migration of the process.
>> >
>> > The next step is to do the same with the build.sh file, as you can see
>> > it,
>> > is a little complex and have lots of steps.
>> > The idea is again convert them into Pharo code.
>>
>> Can we also re-order the bootstrap and break it into several distinct
>> phases.
>>
>> 1. Clean up (this ensures a clean environment for Jenkins)
>> 2. Download all pre-requisites.
>>     The main change here is moving the call to download_vm.sh out of
>> build.sh.
>>     This would allow a particular VM to be installed by hand for step
>> 4, see below.
>> 3. Initial preparation, i.e. all the execution prior to build.sh, e.g.
>> prepare_image.st.
>>    Currently this is done with the Pharo 6 VM and image, which makes sense
>> as it
>>    provides stability.
>> 4. Main image bootstrap.  Which is done with the Pharo 7 VM (currently
>> what is in build.sh).
>>
>> That will allow the first three steps to be run once and then skipped
>> when doing personal development / testing.
>>
>> I want to be able to manually load the VM to be used in step 4 as part
>> of some stability testing so we can promote a new Pharo 7 VM to
>> stable.
>>
>> Are you open to me submitting some PRs that do this restructure to the
>> phases listed above and removing duplicate code from Jenkinsfile?
>>
> Yes of course. Thanks

Could I ask you to have a look at:

https://pharo.fogbugz.com/f/cases/21615/Separate-bootstrap-process-in-to-distinct-stages

https://github.com/pharo-project/pharo/pull/1127

This does the first step of breaking the bootstrap process in to
distinct phases and moving all the downloads to the start (e.g.
allowing a test VM to be substituted for the build stage).

Once this is integrated I can submit a PR that modifies Jenkinsfile to
use the new shell scripts, removing code duplication.


On 16 March 2018 at 14:38, [hidden email] <[hidden email]> wrote:
> Hi,
>      sorry my mistake when updating the scripts. As the bootstrap.sh is not
> executed during the Jenkins process it has not been validated (locally I'm
> not using that script, because is trying to download each time the VM and
> the bootstraping process image)

There are also scripts to snapshot and restore the bootstrap state,
avoiding the need to do the multiple downloads.

Just to be clear again: This is intended to complement and help with
the goal of moving the bootstrap process to pure Pharo code.  Each of
the shell scripts can eventually be replaced by a call to the
equivalent Pharo code.


Thanks!
Alistair

Reply | Threaded
Open this post in threaded view
|

Re: pharo bootstrap is in inconsistent state

Guillermo Polito
Hi!

On Mon, Mar 19, 2018 at 9:17 AM, Alistair Grant <[hidden email]> wrote:

[SNIP]

https://pharo.fogbugz.com/f/cases/21615/Separate-bootstrap-process-in-to-distinct-stages

https://github.com/pharo-project/pharo/pull/1127

This does the first step of breaking the bootstrap process in to
distinct phases and moving all the downloads to the start (e.g.
allowing a test VM to be substituted for the build stage).
 
Nice! thanks!
 

Once this is integrated I can submit a PR that modifies Jenkinsfile to
use the new shell scripts, removing code duplication.

Question: Why not in this same PR?
 


On 16 March 2018 at 14:38, [hidden email] <[hidden email]> wrote:
> Hi,
>      sorry my mistake when updating the scripts. As the bootstrap.sh is not
> executed during the Jenkins process it has not been validated (locally I'm
> not using that script, because is trying to download each time the VM and
> the bootstraping process image)

There are also scripts to snapshot and restore the bootstrap state,
avoiding the need to do the multiple downloads.

Just to be clear again: This is intended to complement and help with
the goal of moving the bootstrap process to pure Pharo code.  Each of
the shell scripts can eventually be replaced by a call to the
equivalent Pharo code.


:)
 

Thanks!
Alistair




--

   

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr


Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: pharo bootstrap is in inconsistent state

alistairgrant
Hi Guille,

On 19 March 2018 at 13:59, Guillermo Polito <[hidden email]> wrote:

> Hi!
>
> On Mon, Mar 19, 2018 at 9:17 AM, Alistair Grant <[hidden email]>
> wrote:
>
> [SNIP]
>>
>>
>>
>> https://pharo.fogbugz.com/f/cases/21615/Separate-bootstrap-process-in-to-distinct-stages
>>
>> https://github.com/pharo-project/pharo/pull/1127
>>
>> This does the first step of breaking the bootstrap process in to
>> distinct phases and moving all the downloads to the start (e.g.
>> allowing a test VM to be substituted for the build stage).
>
>
> Nice! thanks!
>
>>
>>
>> Once this is integrated I can submit a PR that modifies Jenkinsfile to
>> use the new shell scripts, removing code duplication.
>
>
> Question: Why not in this same PR?

Because I'm not at all familiar with Jenkins, so am a bit nervous
about changing it.  Smaller changes will be easier to work on if
something goes wrong. :-)

Cheers,
Alistair



>> On 16 March 2018 at 14:38, [hidden email] <[hidden email]> wrote:
>> > Hi,
>> >      sorry my mistake when updating the scripts. As the bootstrap.sh is
>> > not
>> > executed during the Jenkins process it has not been validated (locally
>> > I'm
>> > not using that script, because is trying to download each time the VM
>> > and
>> > the bootstraping process image)
>>
>> There are also scripts to snapshot and restore the bootstrap state,
>> avoiding the need to do the multiple downloads.
>>
>> Just to be clear again: This is intended to complement and help with
>> the goal of moving the bootstrap process to pure Pharo code.  Each of
>> the shell scripts can eventually be replaced by a call to the
>> equivalent Pharo code.
>>
>
> :)
>
>>
>>
>> Thanks!
>> Alistair
>>
>
>
>
> --
>
>
>
> Guille Polito
>
> Research Engineer
>
> Centre de Recherche en Informatique, Signal et Automatique de Lille
>
> CRIStAL - UMR 9189
>
> French National Center for Scientific Research - http://www.cnrs.fr
>
>
> Web: http://guillep.github.io
>
> Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: pharo bootstrap is in inconsistent state

Guillermo Polito


On Mon, Mar 19, 2018 at 2:13 PM, Alistair Grant <[hidden email]> wrote:
Hi Guille,

On 19 March 2018 at 13:59, Guillermo Polito <[hidden email]> wrote:
> Hi!
>
> On Mon, Mar 19, 2018 at 9:17 AM, Alistair Grant <[hidden email]>
> wrote:
>
> [SNIP]
>>
>>
>>
>> https://pharo.fogbugz.com/f/cases/21615/Separate-bootstrap-process-in-to-distinct-stages
>>
>> https://github.com/pharo-project/pharo/pull/1127
>>
>> This does the first step of breaking the bootstrap process in to
>> distinct phases and moving all the downloads to the start (e.g.
>> allowing a test VM to be substituted for the build stage).
>
>
> Nice! thanks!
>
>>
>>
>> Once this is integrated I can submit a PR that modifies Jenkinsfile to
>> use the new shell scripts, removing code duplication.
>
>
> Question: Why not in this same PR?

Because I'm not at all familiar with Jenkins, so am a bit nervous
about changing it.  Smaller changes will be easier to work on if
something goes wrong. :-)


Ah, but if you do something wrong the build will fail.

In any case, as soon as you want to dive into changing the JenkinsFile, send me a private mail. I probably have to give you permissions in Jenkins (otherwise jenkins ignores changes in the JenkinsFile for security reasons).
 
Cheers,
Alistair



>> On 16 March 2018 at 14:38, [hidden email] <[hidden email]> wrote:
>> > Hi,
>> >      sorry my mistake when updating the scripts. As the bootstrap.sh is
>> > not
>> > executed during the Jenkins process it has not been validated (locally
>> > I'm
>> > not using that script, because is trying to download each time the VM
>> > and
>> > the bootstraping process image)
>>
>> There are also scripts to snapshot and restore the bootstrap state,
>> avoiding the need to do the multiple downloads.
>>
>> Just to be clear again: This is intended to complement and help with
>> the goal of moving the bootstrap process to pure Pharo code.  Each of
>> the shell scripts can eventually be replaced by a call to the
>> equivalent Pharo code.
>>
>
> :)
>
>>
>>
>> Thanks!
>> Alistair
>>
>
>
>
> --
>
>
>
> Guille Polito
>
> Research Engineer
>
> Centre de Recherche en Informatique, Signal et Automatique de Lille
>
> CRIStAL - UMR 9189
>
> French National Center for Scientific Research - http://www.cnrs.fr
>
>
> Web: http://guillep.github.io
>
> Phone: <a href="tel:%2B33%2006%2052%2070%2066%2013" value="+33652706613">+33 06 52 70 66 13




--

   

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr


Web: http://guillep.github.io

Phone: +33 06 52 70 66 13