Pharo Bootstrap familiarity

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Pharo Bootstrap familiarity

Ben Coman
I found it hard to find info on the Pharo Bootstrap, so I went looking to understand the it by identifying very first creation on disk of the smallest  "seed".image  file that will later become the final build artifact.  I'm sharing here to have a record I can refer to later, and maybe others find interesting.

Looking at the log from the latest build...

Searching on "BUILD_NUMBER=" indicates the first major CI step is...

+ BUILD_NUMBER=822 BOOTSTRAP_ARCH=32 bash ./bootstrap/scripts/2-download.sh  [1]
which downloads 24MB " bootstrapImage.zip" [2]
which I presume is an almost full image to run the bootstrap process.


The next major CI step is...

+ BUILD_NUMBER=822 BOOTSTRAP_ARCH=32 bash ./bootstrap/scripts/3-prepare.sh  [3]
which uses the downloaded "Pharo.image" to (I presume) produce the seed "bootstrap.image" file.  
(This is not indicated explicitly in the logs, but step 4 copies that file
with a shell command, so my presumption. It would be nice for this to be explicitly logged"

prepare.sh executes two commands...

./pharo Pharo.image ..../bootstrap/scripts/prepare_image.st --save --quit  [4]
which seems to modify the downloaded "Pharo.image", 
not create bootstrap.image

./pharo Pharo.image ..../bootstrap/scripts/bootstrap.st --ARCH=32 --BUILD_NUMBER=822 --quit   [5]
which does... "PBBootstrap fromCommandLine bootstrap"  [6] [7]
which does...
    PBBootstrap>>#bootstrap 
        PBStepStartedSignal log: 'Starting  Process.'.
        self prepareBootstrap.
        [self createImage ] on:AssertionFailure do: [ :e | e resume ].
        self initializeImage.
        PBStepFinishedSignal log: 'Process ended.'
and digging a bit...

    PBBootstrap>>#createImage
        | builder |
        builder := PBImageBuilderSpur50 forArchitecture: architecture.
        builder definitionFetcher: versionFetcher.
        builder buildNumber: buildNumber.
        builder imageFileReference: imageReference.
        builder systemDefinition: ringEnvironment.
        builder espellBackend instanceVariableMapping: (PBInstanceVariableMapping onEnvironment: ringEnvironment ).
        builder bootstrap


    PBImageBuilderSpur50-class>>#forArchitecture: architecture
        | candidates |
        candidates := self subclasses select: [ :sc | sc name endsWith: architecture, 'bit' ].
        candidates 
            ifEmpty: [ self error: 'No candidates for architecture: ', architecture ].
        candidates size > 1
            ifTrue: [ self error: 'More than one candidate for architecture: ', architecture ].
        ^ candidates anyOne new


    PBImageBuilderSpur50 >>#initialize
        super initialize.
        statistics := PBBootstrapStatistics new.
        logger := PBBootstrapLogger new.
        imageFileReference := 'bootstrap.image' asFileReference.

        "the default espell backend for the bootstrap"
        self initializeBackend.   
        self instantiateClassLoader. 

ahh, there is the reference to "bootstrap.image"

So just to round out a high level overview, the last two messages sent are defined as...

    PBImageBuilderSpur5032bit>>#initializeBackend
        espellBackend := EPSimulatorBackend for32Bit forBootstrap    

which I won't dig further but the class seems available at [8]
and Espell is described briefly at [9] (but what is a "MOP"? the term is not defined there)


    PBImageBuilderSpur50>>#instantiateClassLoader
        classLoader := PBSpurClassLoader new.   

which is defined at [10], and there is an interesting example at [11].




So now I guess that bootstrap.image has been generated?
Just curious, what is the size of this file right now, before the next step?
Perhaps an `ls -l` could echo this to the build log.


The penultimate major CI step seems to then start adding packages to bootstrap.image...

+ BUILD_NUMBER=822 BOOTSTRAP_ARCH=32 bash ./bootstrap/scripts/4-build.sh  
located at [12] and searching on "${VM}" is enlightening.
Its help information says...
Pharo Build Script
  ==================
  This script assumes the existence of a new uninitialized image. Then it proceeds to its initialization and \"growing\".
  * Step 1:
    - Initialize the image
    - output: core.image
  * Step 2:
    - Bootstrap Monticello local repositories
    - output: monticello_bootstrap.image and changes file
  * Step 3:
    - Load Monticello remote repositories
    - output: monticello.image and changes file
  * Step 4:
    - Load Metacello
    - output: metacello.image and changes file
  * Step 5:
    - Load the rest of the image using BaselineOfIDE
    - output: Pharo.image and changes file

Unfortunately those steps are labelled in the script,
but the first action shown in the log is...
    cp bootstrap.image Pharo7.0-compiler-32bit-9912d7c.image 
Initializing Bootstraped Image
    pharo --headless Pharo7.0-compiler-32bit-9912d7c.image




The final major CI step was a fair way past what I was interested in right now. ...

+ BUILD_NUMBER=822 bash ../bootstrap/scripts/prepare_for_upload.sh



cheers -ben

Reply | Threaded
Open this post in threaded view
|

Re: Pharo Bootstrap familiarity

Nicolas Cellier

2018-05-02 20:53 GMT+02:00 Ben Coman <[hidden email]>:
I found it hard to find info on the Pharo Bootstrap, so I went looking to understand the it by identifying very first creation on disk of the smallest  "seed".image  file that will later become the final build artifact.  I'm sharing here to have a record I can refer to later, and maybe others find interesting.

Looking at the log from the latest build...

Searching on "BUILD_NUMBER=" indicates the first major CI step is...

+ BUILD_NUMBER=822 BOOTSTRAP_ARCH=32 bash ./bootstrap/scripts/2-download.sh  [1]
which downloads 24MB " bootstrapImage.zip" [2]
which I presume is an almost full image to run the bootstrap process.


The next major CI step is...

+ BUILD_NUMBER=822 BOOTSTRAP_ARCH=32 bash ./bootstrap/scripts/3-prepare.sh  [3]
which uses the downloaded "Pharo.image" to (I presume) produce the seed "bootstrap.image" file.  
(This is not indicated explicitly in the logs, but step 4 copies that file
with a shell command, so my presumption. It would be nice for this to be explicitly logged"

prepare.sh executes two commands...

./pharo Pharo.image ..../bootstrap/scripts/prepare_image.st --save --quit  [4]
which seems to modify the downloaded "Pharo.image", 
not create bootstrap.image

./pharo Pharo.image ..../bootstrap/scripts/bootstrap.st --ARCH=32 --BUILD_NUMBER=822 --quit   [5]
which does... "PBBootstrap fromCommandLine bootstrap"  [6] [7]
which does...
    PBBootstrap>>#bootstrap 
        PBStepStartedSignal log: 'Starting  Process.'.
        self prepareBootstrap.
        [self createImage ] on:AssertionFailure do: [ :e | e resume ].
        self initializeImage.
        PBStepFinishedSignal log: 'Process ended.'
and digging a bit...

    PBBootstrap>>#createImage
        | builder |
        builder := PBImageBuilderSpur50 forArchitecture: architecture.
        builder definitionFetcher: versionFetcher.
        builder buildNumber: buildNumber.
        builder imageFileReference: imageReference.
        builder systemDefinition: ringEnvironment.
        builder espellBackend instanceVariableMapping: (PBInstanceVariableMapping onEnvironment: ringEnvironment ).
        builder bootstrap


    PBImageBuilderSpur50-class>>#forArchitecture: architecture
        | candidates |
        candidates := self subclasses select: [ :sc | sc name endsWith: architecture, 'bit' ].
        candidates 
            ifEmpty: [ self error: 'No candidates for architecture: ', architecture ].
        candidates size > 1
            ifTrue: [ self error: 'More than one candidate for architecture: ', architecture ].
        ^ candidates anyOne new


    PBImageBuilderSpur50 >>#initialize
        super initialize.
        statistics := PBBootstrapStatistics new.
        logger := PBBootstrapLogger new.
        imageFileReference := 'bootstrap.image' asFileReference.

        "the default espell backend for the bootstrap"
        self initializeBackend.   
        self instantiateClassLoader. 

ahh, there is the reference to "bootstrap.image"

So just to round out a high level overview, the last two messages sent are defined as...

    PBImageBuilderSpur5032bit>>#initializeBackend
        espellBackend := EPSimulatorBackend for32Bit forBootstrap    

which I won't dig further but the class seems available at [8]
and Espell is described briefly at [9] (but what is a "MOP"? the term is not defined there)


    PBImageBuilderSpur50>>#instantiateClassLoader
        classLoader := PBSpurClassLoader new.   

which is defined at [10], and there is an interesting example at [11].




So now I guess that bootstrap.image has been generated?
Just curious, what is the size of this file right now, before the next step?
Perhaps an `ls -l` could echo this to the build log.


The penultimate major CI step seems to then start adding packages to bootstrap.image...

+ BUILD_NUMBER=822 BOOTSTRAP_ARCH=32 bash ./bootstrap/scripts/4-build.sh  
located at [12] and searching on "${VM}" is enlightening.
Its help information says...
Pharo Build Script
  ==================
  This script assumes the existence of a new uninitialized image. Then it proceeds to its initialization and \"growing\".
  * Step 1:
    - Initialize the image
    - output: core.image
  * Step 2:
    - Bootstrap Monticello local repositories
    - output: monticello_bootstrap.image and changes file
  * Step 3:
    - Load Monticello remote repositories
    - output: monticello.image and changes file
  * Step 4:
    - Load Metacello
    - output: metacello.image and changes file
  * Step 5:
    - Load the rest of the image using BaselineOfIDE
    - output: Pharo.image and changes file

Unfortunately those steps are labelled in the script,
but the first action shown in the log is...
    cp bootstrap.image Pharo7.0-compiler-32bit-9912d7c.image 
Initializing Bootstraped Image
    pharo --headless Pharo7.0-compiler-32bit-9912d7c.image




The final major CI step was a fair way past what I was interested in right now. ...

+ BUILD_NUMBER=822 bash ../bootstrap/scripts/prepare_for_upload.sh



cheers -ben


Reply | Threaded
Open this post in threaded view
|

Re: Pharo Bootstrap familiarity

Guillermo Polito
Hi Ben,

as Nicolas said, yes, MOP is for Meta Object Protocol (really nice book btw https://www.amazon.com/Art-Metaobject-Protocol-Gregor-Kiczales/dp/0262610744).

Ben, the process is not carved in stone and we are fully aware that it can be enhanced :). Any enhancement, even the littlest one, is important. If it is important for your workflows to add more logging information, let's add them!

Nicolas,

Actually the special interpreter is not the simulator. We do not use the bytecode interpreter in the simulator, we use an AST interpreter that has "lazy class creation" semantics plus some other little tricks to initialize the circular dependencies of metaclasses and so on... Behind, we use the simulator only to store the objects and decouple ourselves from the object format (this removed for example a lot of work when migrating from V3 to Spur).

On Wed, May 2, 2018 at 10:04 PM, Nicolas Cellier <[hidden email]> wrote:

2018-05-02 20:53 GMT+02:00 Ben Coman <[hidden email]>:
I found it hard to find info on the Pharo Bootstrap, so I went looking to understand the it by identifying very first creation on disk of the smallest  "seed".image  file that will later become the final build artifact.  I'm sharing here to have a record I can refer to later, and maybe others find interesting.

Looking at the log from the latest build...

Searching on "BUILD_NUMBER=" indicates the first major CI step is...

+ BUILD_NUMBER=822 BOOTSTRAP_ARCH=32 bash ./bootstrap/scripts/2-download.sh  [1]
which downloads 24MB " bootstrapImage.zip" [2]
which I presume is an almost full image to run the bootstrap process.


The next major CI step is...

+ BUILD_NUMBER=822 BOOTSTRAP_ARCH=32 bash ./bootstrap/scripts/3-prepare.sh  [3]
which uses the downloaded "Pharo.image" to (I presume) produce the seed "bootstrap.image" file.  
(This is not indicated explicitly in the logs, but step 4 copies that file
with a shell command, so my presumption. It would be nice for this to be explicitly logged"

prepare.sh executes two commands...

./pharo Pharo.image ..../bootstrap/scripts/prepare_image.st --save --quit  [4]
which seems to modify the downloaded "Pharo.image", 
not create bootstrap.image

./pharo Pharo.image ..../bootstrap/scripts/bootstrap.st --ARCH=32 --BUILD_NUMBER=822 --quit   [5]
which does... "PBBootstrap fromCommandLine bootstrap"  [6] [7]
which does...
    PBBootstrap>>#bootstrap 
        PBStepStartedSignal log: 'Starting  Process.'.
        self prepareBootstrap.
        [self createImage ] on:AssertionFailure do: [ :e | e resume ].
        self initializeImage.
        PBStepFinishedSignal log: 'Process ended.'
and digging a bit...

    PBBootstrap>>#createImage
        | builder |
        builder := PBImageBuilderSpur50 forArchitecture: architecture.
        builder definitionFetcher: versionFetcher.
        builder buildNumber: buildNumber.
        builder imageFileReference: imageReference.
        builder systemDefinition: ringEnvironment.
        builder espellBackend instanceVariableMapping: (PBInstanceVariableMapping onEnvironment: ringEnvironment ).
        builder bootstrap


    PBImageBuilderSpur50-class>>#forArchitecture: architecture
        | candidates |
        candidates := self subclasses select: [ :sc | sc name endsWith: architecture, 'bit' ].
        candidates 
            ifEmpty: [ self error: 'No candidates for architecture: ', architecture ].
        candidates size > 1
            ifTrue: [ self error: 'More than one candidate for architecture: ', architecture ].
        ^ candidates anyOne new


    PBImageBuilderSpur50 >>#initialize
        super initialize.
        statistics := PBBootstrapStatistics new.
        logger := PBBootstrapLogger new.
        imageFileReference := 'bootstrap.image' asFileReference.

        "the default espell backend for the bootstrap"
        self initializeBackend.   
        self instantiateClassLoader. 

ahh, there is the reference to "bootstrap.image"

So just to round out a high level overview, the last two messages sent are defined as...

    PBImageBuilderSpur5032bit>>#initializeBackend
        espellBackend := EPSimulatorBackend for32Bit forBootstrap    

which I won't dig further but the class seems available at [8]
and Espell is described briefly at [9] (but what is a "MOP"? the term is not defined there)


    PBImageBuilderSpur50>>#instantiateClassLoader
        classLoader := PBSpurClassLoader new.   

which is defined at [10], and there is an interesting example at [11].




So now I guess that bootstrap.image has been generated?
Just curious, what is the size of this file right now, before the next step?
Perhaps an `ls -l` could echo this to the build log.


The penultimate major CI step seems to then start adding packages to bootstrap.image...

+ BUILD_NUMBER=822 BOOTSTRAP_ARCH=32 bash ./bootstrap/scripts/4-build.sh  
located at [12] and searching on "${VM}" is enlightening.
Its help information says...
Pharo Build Script
  ==================
  This script assumes the existence of a new uninitialized image. Then it proceeds to its initialization and \"growing\".
  * Step 1:
    - Initialize the image
    - output: core.image
  * Step 2:
    - Bootstrap Monticello local repositories
    - output: monticello_bootstrap.image and changes file
  * Step 3:
    - Load Monticello remote repositories
    - output: monticello.image and changes file
  * Step 4:
    - Load Metacello
    - output: metacello.image and changes file
  * Step 5:
    - Load the rest of the image using BaselineOfIDE
    - output: Pharo.image and changes file

Unfortunately those steps are labelled in the script,
but the first action shown in the log is...
    cp bootstrap.image Pharo7.0-compiler-32bit-9912d7c.image 
Initializing Bootstraped Image
    pharo --headless Pharo7.0-compiler-32bit-9912d7c.image




The final major CI step was a fair way past what I was interested in right now. ...

+ BUILD_NUMBER=822 bash ../bootstrap/scripts/prepare_for_upload.sh



cheers -ben





--

   

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr


Web: http://guillep.github.io

Phone: +33 06 52 70 66 13