[IMPORTANT] Following changes in the bootstrapping process

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[IMPORTANT] Following changes in the bootstrapping process

Pavel Krivanek-3
Hello, 

we are checking a huge pull request #177 (https://github.com/pharo-project/pharo/pull/177) that will change some basics of the bootstrap process:

Now we will bootstrap a smaller image that will not include compiler/parser. Compiler and related packages are exported and loaded using a binary exporter named Hermes. 
The compiler is then used to load FileSystem and Monticello. The rest of the bootstrap process will be the same as before.
As the result we will have faster bootstrap and better system modularization and possibilities.

It required some modularization efforts:

- simplified initialization scripts
- Use Zinc converters and encoders instead of FilePathEncoder and old TextConverter
- Use Stdio instead of FileStream
- Using File instead of FileSystem
- Deprecated FileStream & childs (Moved to Deprecated70)
- Extracted Path classes to their on package: FileSystem-Path
- Moved OpalEncoders to their own package. They are required by the runtime (not only for compilation)
- Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase and so on without requiring full Unicode tables from the beginning
- Cleaning up a bit the full exception logging infrastructure (streams, transcript, files, stack printing...)
- Split Ring methods required for system navigation to the Ring-Navigation package
- Remove usages of #translated in the kernel
- Refactored the bootstrapping classes to remove duplications
- Cleaning up dependencies in CompiledMethod>>printOn:
- fix path printing

We need to merge these changes at once and of course it can cause some conflicts with the existing pull requests or external code. Anyway, we need to merge it as soon as possible.

So please, try to look at the PR and test the resultant image [1] to avoid some major problems.


Cheers,
-- Pavel

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Stephane Ducasse-3
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:

> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

philippe.back@highoctane.be
Massive.

What is in the super small image?
Is Hermes going to be a generic binary content loader?

Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

philippeback
In reply to this post by Pavel Krivanek-3
Massive.

What is in the super small image?
Is Hermes going to be a generic binary content loader?

Phil

On Aug 1, 2017 11:58, "Pavel Krivanek" <[hidden email]> wrote:
Hello, 

we are checking a huge pull request #177 (https://github.com/pharo-project/pharo/pull/177) that will change some basics of the bootstrap process:

Now we will bootstrap a smaller image that will not include compiler/parser. Compiler and related packages are exported and loaded using a binary exporter named Hermes. 
The compiler is then used to load FileSystem and Monticello. The rest of the bootstrap process will be the same as before.
As the result we will have faster bootstrap and better system modularization and possibilities.

It required some modularization efforts:

- simplified initialization scripts
- Use Zinc converters and encoders instead of FilePathEncoder and old TextConverter
- Use Stdio instead of FileStream
- Using File instead of FileSystem
- Deprecated FileStream & childs (Moved to Deprecated70)
- Extracted Path classes to their on package: FileSystem-Path
- Moved OpalEncoders to their own package. They are required by the runtime (not only for compilation)
- Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase and so on without requiring full Unicode tables from the beginning
- Cleaning up a bit the full exception logging infrastructure (streams, transcript, files, stack printing...)
- Split Ring methods required for system navigation to the Ring-Navigation package
- Remove usages of #translated in the kernel
- Refactored the bootstrapping classes to remove duplications
- Cleaning up dependencies in CompiledMethod>>printOn:
- fix path printing

We need to merge these changes at once and of course it can cause some conflicts with the existing pull requests or external code. Anyway, we need to merge it as soon as possible.

So please, try to look at the PR and test the resultant image [1] to avoid some major problems.


Cheers,
-- Pavel


Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Pavel Krivanek-3
In reply to this post by philippe.back@highoctane.be


2017-08-01 12:57 GMT+02:00 [hidden email] <[hidden email]>:
Massive.

What is in the super small image?

It is not super small. Currently it has about 5.3MB but it can be probably less if we will do some cleaning before the image saving. The size was not the priority but we are getting there.
We already have an experimental reduced image with Hermes that is able to be bootstrapped under 2 minutes. The minimal system based on the current code-base that is able to be bootstrapped has about 260kB.

-- Pavel

 
Is Hermes going to be a generic binary content loader?

Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>


Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Pavel Krivanek-3


2017-08-01 13:13 GMT+02:00 Pavel Krivanek <[hidden email]>:


2017-08-01 12:57 GMT+02:00 [hidden email] <[hidden email]>:
Massive.

What is in the super small image?

It is not super small. Currently it has about 5.3MB but it can be probably less if we will do some cleaning before the image saving. The size was not the priority but we are getting there.

the raw bootstrapped image has 3.6MB.
 
We already have an experimental reduced image with Hermes that is able to be bootstrapped under 2 minutes. The minimal system based on the current code-base that is able to be bootstrapped has about 260kB.

-- Pavel

 
Is Hermes going to be a generic binary content loader?

Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>



Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Pavel Krivanek-3
In reply to this post by Stephane Ducasse-3
The main credits go to Guille and Pablo. My role in this was very tiny.

-- Pavel

2017-08-01 12:35 GMT+02:00 Stephane Ducasse <[hidden email]>:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>


Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Guillermo Polito
In reply to this post by philippe.back@highoctane.be


On Tue, Aug 1, 2017 at 12:57 PM, [hidden email] <[hidden email]> wrote:
Massive.

What is in the super small image?
Is Hermes going to be a generic binary content loader?

There is still work to do in this front. 
 - Hermes only works for 32bits format. We should adapt it to 64 bits (immediate floats and so on...)
 - There is no format validation. We should add one for safety.
 

Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>




--

   

Guille Polito


Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr



Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Stephane Ducasse-3
In reply to this post by Pavel Krivanek-3
260k is really nice for a small kernel!


On Tue, Aug 1, 2017 at 1:13 PM, Pavel Krivanek <[hidden email]> wrote:

>
>
> 2017-08-01 12:57 GMT+02:00 [hidden email]
> <[hidden email]>:
>>
>> Massive.
>>
>> What is in the super small image?
>
>
> It is not super small. Currently it has about 5.3MB but it can be probably
> less if we will do some cleaning before the image saving. The size was not
> the priority but we are getting there.
> We already have an experimental reduced image with Hermes that is able to be
> bootstrapped under 2 minutes. The minimal system based on the current
> code-base that is able to be bootstrapped has about 260kB.
>
> -- Pavel
>
>
>>
>> Is Hermes going to be a generic binary content loader?
>>
>> Phil
>>
>> On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
>>>
>>> Hi Pavel
>>>
>>> This is super excellent! IMPRESSIVE. An image without the compiler and
>>> a reloadable compiler.
>>> Super cool.
>>>
>>> Stef
>>>
>>>
>>> On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
>>> <[hidden email]> wrote:
>>> > Hello,
>>> >
>>> > we are checking a huge pull request #177
>>> > (https://github.com/pharo-project/pharo/pull/177) that will change some
>>> > basics of the bootstrap process:
>>> >
>>> > Now we will bootstrap a smaller image that will not include
>>> > compiler/parser.
>>> > Compiler and related packages are exported and loaded using a binary
>>> > exporter named Hermes.
>>> > The compiler is then used to load FileSystem and Monticello. The rest
>>> > of the
>>> > bootstrap process will be the same as before.
>>> > As the result we will have faster bootstrap and better system
>>> > modularization
>>> > and possibilities.
>>> >
>>> > It required some modularization efforts:
>>> >
>>> > - simplified initialization scripts
>>> > - Use Zinc converters and encoders instead of FilePathEncoder and old
>>> > TextConverter
>>> > - Use Stdio instead of FileStream
>>> > - Using File instead of FileSystem
>>> > - Deprecated FileStream & childs (Moved to Deprecated70)
>>> > - Extracted Path classes to their on package: FileSystem-Path
>>> > - Moved OpalEncoders to their own package. They are required by the
>>> > runtime
>>> > (not only for compilation)
>>> > - Introduced AsciiCharset in the kernel to answer to #isLetter
>>> > #isUppercase
>>> > and so on without requiring full Unicode tables from the beginning
>>> > - Cleaning up a bit the full exception logging infrastructure (streams,
>>> > transcript, files, stack printing...)
>>> > - Split Ring methods required for system navigation to the
>>> > Ring-Navigation
>>> > package
>>> > - Remove usages of #translated in the kernel
>>> > - Refactored the bootstrapping classes to remove duplications
>>> > - Cleaning up dependencies in CompiledMethod>>printOn:
>>> > - fix path printing
>>> >
>>> > We need to merge these changes at once and of course it can cause some
>>> > conflicts with the existing pull requests or external code. Anyway, we
>>> > need
>>> > to merge it as soon as possible.
>>> >
>>> > So please, try to look at the PR and test the resultant image [1] to
>>> > avoid
>>> > some major problems.
>>> >
>>> > [1]
>>> >
>>> > https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>>> >
>>> > Cheers,
>>> > -- Pavel
>>> >
>>>
>

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Stephane Ducasse-3
In reply to this post by philippe.back@highoctane.be
Hi phil

We will see for hermes. Because we have to pay attention when loading
binary and changing the code we may end up with desynchronised
situation.
But this opens a lot of possibilities.
We could deploy a PharoApp as a mini image and a set of binary files.
I want a process where we can write an application (not just a bunch
of classes) and get a process until a full deploy.
Esteban confirmed to me that he wants to push the desktop deploy front
in this iteration.

Stef


On Tue, Aug 1, 2017 at 12:57 PM, [hidden email]
<[hidden email]> wrote:

> Massive.
>
> What is in the super small image?
> Is Hermes going to be a generic binary content loader?
>
> Phil
>
> On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
>>
>> Hi Pavel
>>
>> This is super excellent! IMPRESSIVE. An image without the compiler and
>> a reloadable compiler.
>> Super cool.
>>
>> Stef
>>
>>
>> On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
>> <[hidden email]> wrote:
>> > Hello,
>> >
>> > we are checking a huge pull request #177
>> > (https://github.com/pharo-project/pharo/pull/177) that will change some
>> > basics of the bootstrap process:
>> >
>> > Now we will bootstrap a smaller image that will not include
>> > compiler/parser.
>> > Compiler and related packages are exported and loaded using a binary
>> > exporter named Hermes.
>> > The compiler is then used to load FileSystem and Monticello. The rest of
>> > the
>> > bootstrap process will be the same as before.
>> > As the result we will have faster bootstrap and better system
>> > modularization
>> > and possibilities.
>> >
>> > It required some modularization efforts:
>> >
>> > - simplified initialization scripts
>> > - Use Zinc converters and encoders instead of FilePathEncoder and old
>> > TextConverter
>> > - Use Stdio instead of FileStream
>> > - Using File instead of FileSystem
>> > - Deprecated FileStream & childs (Moved to Deprecated70)
>> > - Extracted Path classes to their on package: FileSystem-Path
>> > - Moved OpalEncoders to their own package. They are required by the
>> > runtime
>> > (not only for compilation)
>> > - Introduced AsciiCharset in the kernel to answer to #isLetter
>> > #isUppercase
>> > and so on without requiring full Unicode tables from the beginning
>> > - Cleaning up a bit the full exception logging infrastructure (streams,
>> > transcript, files, stack printing...)
>> > - Split Ring methods required for system navigation to the
>> > Ring-Navigation
>> > package
>> > - Remove usages of #translated in the kernel
>> > - Refactored the bootstrapping classes to remove duplications
>> > - Cleaning up dependencies in CompiledMethod>>printOn:
>> > - fix path printing
>> >
>> > We need to merge these changes at once and of course it can cause some
>> > conflicts with the existing pull requests or external code. Anyway, we
>> > need
>> > to merge it as soon as possible.
>> >
>> > So please, try to look at the PR and test the resultant image [1] to
>> > avoid
>> > some major problems.
>> >
>> > [1]
>> >
>> > https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>> >
>> > Cheers,
>> > -- Pavel
>> >
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Eliot Miranda-2
In reply to this post by Guillermo Polito
Hi Guille,

On Aug 1, 2017, at 4:29 AM, Guillermo Polito <[hidden email]> wrote:



On Tue, Aug 1, 2017 at 12:57 PM, [hidden email] <[hidden email]> wrote:
Massive.

What is in the super small image?
Is Hermes going to be a generic binary content loader?

There is still work to do in this front. 
 - Hermes only works for 32bits format. We should adapt it to 64 bits (immediate floats and so on...)
 - There is no format validation. We should add one for safety.

Will Hermes be able to save on 32-bits and load on 64-bits and vice verse?

Does Hermes use the Parcels/Fuel architecture of saving nodes, grouped by class, followed by the arcs?

If yes to both of these, are you willing to keep it in Monticello and collaborate with the Squeak & Cuis communities in developing Hermes?
 
Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>


--

   

Guille Polito


Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr



Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

tesonep@gmail.com
Hi Eliot, 
 
The last version of Hermes, that I have generated today and have to be tested for real yet, is able to load and save from and to 32 and 64 bits, handling the conversion. Because the Hermes representation (if we can call it like that, because is really simple, does not care about the architecture). 

Hermes is using a custom format completely separated from Fuel or any other tool. 

This is a consequence of the design of Hermes. Hermes has been designed to only work as light way of loading code during the bootstrap. So that, it is heavily coupled with Pharo (it can be changed, for example implementing another loader / installer) and uses only really core classes. 

Of course it can be extended and used in another Smalltalk dialects, but I am not sure if the tool can be useful for them, or even for the Pharo community outside the Pharo bootstrap process. 

Today to be useful outside the bootstrap process, some work has to be done (perhaps a lot), but mainly new use cases have to be thinked. Once the Compiler, Monticello or Metacello is available, there is no need for Hermes.  It is only used to generate some binary loadable packages for the compiler from the source code. 

Again, it was only thinked as a tool to improve the speed during the bootstrap. However, if somebody has ideas to use it or want to collaborate are  of course welcomed. 

Hermes is actually in Github, because to me is more comfortable to have it there,  but if somebody is really wanting to use it.... we can manage to have an export process to Smalltalkhub. 

I hope I have clarified a little the idea behind Hermes.

Maybe you have better ideas of what can we do with it. 


On Tue, Aug 1, 2017 at 7:59 PM, Eliot Miranda <[hidden email]> wrote:
Hi Guille,

On Aug 1, 2017, at 4:29 AM, Guillermo Polito <[hidden email]> wrote:



On Tue, Aug 1, 2017 at 12:57 PM, [hidden email] <[hidden email]> wrote:
Massive.

What is in the super small image?
Is Hermes going to be a generic binary content loader?

There is still work to do in this front. 
 - Hermes only works for 32bits format. We should adapt it to 64 bits (immediate floats and so on...)
 - There is no format validation. We should add one for safety.

Will Hermes be able to save on 32-bits and load on 64-bits and vice verse?

Does Hermes use the Parcels/Fuel architecture of saving nodes, grouped by class, followed by the arcs?

If yes to both of these, are you willing to keep it in Monticello and collaborate with the Squeak & Cuis communities in developing Hermes?

 
Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>


--

   

Guille Polito


Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr



Web: http://guillep.github.io

Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13




--
Pablo Tesone.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Eliot Miranda-2
Hi Pablo,

    I understand that Hermes has its own format.  My question was about architecture. The Parcels/Fuel architecture is, AFAIA, the best architecture for binary storage because it is virtual, is fast to load and cleanly separates loading from initialization.  What do I mean?

- concrete formats like ImageSegment are tied to internal representations in the VM and do are difficult to read in forever FB systems and very hard to write in foreign systems.  So a virtual format (like BOSS, Fuel, Parcels, etc) is better but unless done well can be slow

- a first generation system like BOSS is a flattening of a graph traversal using a simple grammar.  Each phrase is either an object reference (object id) or an object definition (object is followed by type info and contents, contents being a sequence of phrases.  This is slow to parse (because each phrase must be decoded), and has invalid intermediate states (e.g. a Set whose contents are not all present, hence whose external hash may change during loading, etc).  A second generation system like Parcels and Fuel separates objects from references (nodes from arcs) and batches object creation, grouping all instances of a given class for bulk, and hence fast, instantiation.  Following my the objects are the references.  So loading has three phases:
- instantiate the objects
- connect the graph by filling in object references
- initialize the objects that require it (e.g. rehash sets)

Consequently the writer must be two pass, the first pass to collect the objects in the graph and sort them by class, the second to write the nodes followed by the references 

So let me ask again, does Hermes use the Parcels/Fuel architecture?

_,,,^..^,,,_ (phone)

On Aug 1, 2017, at 11:52 AM, "[hidden email]" <[hidden email]> wrote:

Hi Eliot, 
 
The last version of Hermes, that I have generated today and have to be tested for real yet, is able to load and save from and to 32 and 64 bits, handling the conversion. Because the Hermes representation (if we can call it like that, because is really simple, does not care about the architecture). 

Hermes is using a custom format completely separated from Fuel or any other tool. 

This is a consequence of the design of Hermes. Hermes has been designed to only work as light way of loading code during the bootstrap. So that, it is heavily coupled with Pharo (it can be changed, for example implementing another loader / installer) and uses only really core classes. 

Of course it can be extended and used in another Smalltalk dialects, but I am not sure if the tool can be useful for them, or even for the Pharo community outside the Pharo bootstrap process. 

Today to be useful outside the bootstrap process, some work has to be done (perhaps a lot), but mainly new use cases have to be thinked. Once the Compiler, Monticello or Metacello is available, there is no need for Hermes.  It is only used to generate some binary loadable packages for the compiler from the source code. 

Again, it was only thinked as a tool to improve the speed during the bootstrap. However, if somebody has ideas to use it or want to collaborate are  of course welcomed. 

Hermes is actually in Github, because to me is more comfortable to have it there,  but if somebody is really wanting to use it.... we can manage to have an export process to Smalltalkhub. 

I hope I have clarified a little the idea behind Hermes.

Maybe you have better ideas of what can we do with it. 


On Tue, Aug 1, 2017 at 7:59 PM, Eliot Miranda <[hidden email]> wrote:
Hi Guille,

On Aug 1, 2017, at 4:29 AM, Guillermo Polito <[hidden email]> wrote:



On Tue, Aug 1, 2017 at 12:57 PM, [hidden email] <[hidden email]> wrote:
Massive.

What is in the super small image?
Is Hermes going to be a generic binary content loader?

There is still work to do in this front. 
 - Hermes only works for 32bits format. We should adapt it to 64 bits (immediate floats and so on...)
 - There is no format validation. We should add one for safety.

Will Hermes be able to save on 32-bits and load on 64-bits and vice verse?

Does Hermes use the Parcels/Fuel architecture of saving nodes, grouped by class, followed by the arcs?

If yes to both of these, are you willing to keep it in Monticello and collaborate with the Squeak & Cuis communities in developing Hermes?

 
Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>


--

   

Guille Polito


Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr



Web: http://guillep.github.io

Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13




--
Pablo Tesone.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

tesonep@gmail.com
Hi again, 
   thanks for the clarification, I haven't understood the question initially, but now I think I can answer it.

Hermes is only exporting the classes, not objects. so it does not have to handle complex graphs of objects. Basically it serializes the definition of the classes, traits and methods. Then they are loaded in sequence. The only caring during the generation is to serialize first the superclasses, and then the subclasses. There is no way of serializing objects outside the classes, methods, traits and literals in a method. 

So the answer is that is not using Parcels or Fuel architecture because they are intended for different uses. Also the format itself has no special design or considerations to be fast to generate or fast to read (as It happens with Fuel)

Cheers, 
Pablo

On Tue, Aug 1, 2017 at 9:16 PM, Eliot Miranda <[hidden email]> wrote:
Hi Pablo,

    I understand that Hermes has its own format.  My question was about architecture. The Parcels/Fuel architecture is, AFAIA, the best architecture for binary storage because it is virtual, is fast to load and cleanly separates loading from initialization.  What do I mean?

- concrete formats like ImageSegment are tied to internal representations in the VM and do are difficult to read in forever FB systems and very hard to write in foreign systems.  So a virtual format (like BOSS, Fuel, Parcels, etc) is better but unless done well can be slow

- a first generation system like BOSS is a flattening of a graph traversal using a simple grammar.  Each phrase is either an object reference (object id) or an object definition (object is followed by type info and contents, contents being a sequence of phrases.  This is slow to parse (because each phrase must be decoded), and has invalid intermediate states (e.g. a Set whose contents are not all present, hence whose external hash may change during loading, etc).  A second generation system like Parcels and Fuel separates objects from references (nodes from arcs) and batches object creation, grouping all instances of a given class for bulk, and hence fast, instantiation.  Following my the objects are the references.  So loading has three phases:
- instantiate the objects
- connect the graph by filling in object references
- initialize the objects that require it (e.g. rehash sets)

Consequently the writer must be two pass, the first pass to collect the objects in the graph and sort them by class, the second to write the nodes followed by the references 

So let me ask again, does Hermes use the Parcels/Fuel architecture?

_,,,^..^,,,_ (phone)

On Aug 1, 2017, at 11:52 AM, "[hidden email]" <[hidden email]> wrote:

Hi Eliot, 
 
The last version of Hermes, that I have generated today and have to be tested for real yet, is able to load and save from and to 32 and 64 bits, handling the conversion. Because the Hermes representation (if we can call it like that, because is really simple, does not care about the architecture). 

Hermes is using a custom format completely separated from Fuel or any other tool. 

This is a consequence of the design of Hermes. Hermes has been designed to only work as light way of loading code during the bootstrap. So that, it is heavily coupled with Pharo (it can be changed, for example implementing another loader / installer) and uses only really core classes. 

Of course it can be extended and used in another Smalltalk dialects, but I am not sure if the tool can be useful for them, or even for the Pharo community outside the Pharo bootstrap process. 

Today to be useful outside the bootstrap process, some work has to be done (perhaps a lot), but mainly new use cases have to be thinked. Once the Compiler, Monticello or Metacello is available, there is no need for Hermes.  It is only used to generate some binary loadable packages for the compiler from the source code. 

Again, it was only thinked as a tool to improve the speed during the bootstrap. However, if somebody has ideas to use it or want to collaborate are  of course welcomed. 

Hermes is actually in Github, because to me is more comfortable to have it there,  but if somebody is really wanting to use it.... we can manage to have an export process to Smalltalkhub. 

I hope I have clarified a little the idea behind Hermes.

Maybe you have better ideas of what can we do with it. 


On Tue, Aug 1, 2017 at 7:59 PM, Eliot Miranda <[hidden email]> wrote:
Hi Guille,

On Aug 1, 2017, at 4:29 AM, Guillermo Polito <[hidden email]> wrote:



On Tue, Aug 1, 2017 at 12:57 PM, [hidden email] <[hidden email]> wrote:
Massive.

What is in the super small image?
Is Hermes going to be a generic binary content loader?

There is still work to do in this front. 
 - Hermes only works for 32bits format. We should adapt it to 64 bits (immediate floats and so on...)
 - There is no format validation. We should add one for safety.

Will Hermes be able to save on 32-bits and load on 64-bits and vice verse?

Does Hermes use the Parcels/Fuel architecture of saving nodes, grouped by class, followed by the arcs?

If yes to both of these, are you willing to keep it in Monticello and collaborate with the Squeak & Cuis communities in developing Hermes?

 
Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>


--

   

Guille Polito


Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr



Web: http://guillep.github.io

Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13




--
Pablo Tesone.
[hidden email]



--
Pablo Tesone.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Eliot Miranda-2
Hi Pablo,

On Tue, Aug 1, 2017 at 12:36 PM, [hidden email] <[hidden email]> wrote:
Hi again, 
   thanks for the clarification, I haven't understood the question initially, but now I think I can answer it.

Hermes is only exporting the classes, not objects. so it does not have to handle complex graphs of objects. Basically it serializes the definition of the classes, traits and methods. Then they are loaded in sequence. The only caring during the generation is to serialize first the superclasses, and then the subclasses. There is no way of serializing objects outside the classes, methods, traits and literals in a method. 

Classes, traits, methods, literals *are* a graph of objects :-). The Parcel architecture, from which Fuel derived its architecture, was designed for loading code in VisualWorks.  In fact, last time I looked Parcels were used only to store code and not as a general purpose (de)serializer.

So the answer is that is not using Parcels or Fuel architecture because they are intended for different uses.

Well, I don't think that's the reason ;-), but fine.  I understand that it doesn't use this architecture.  Thanks.
 
Also the format itself has no special design or considerations to be fast to generate or fast to read (as It happens with Fuel)

Cheers, 
Pablo

On Tue, Aug 1, 2017 at 9:16 PM, Eliot Miranda <[hidden email]> wrote:
Hi Pablo,

    I understand that Hermes has its own format.  My question was about architecture. The Parcels/Fuel architecture is, AFAIA, the best architecture for binary storage because it is virtual, is fast to load and cleanly separates loading from initialization.  What do I mean?

- concrete formats like ImageSegment are tied to internal representations in the VM and do are difficult to read in forever FB systems and very hard to write in foreign systems.  So a virtual format (like BOSS, Fuel, Parcels, etc) is better but unless done well can be slow

- a first generation system like BOSS is a flattening of a graph traversal using a simple grammar.  Each phrase is either an object reference (object id) or an object definition (object is followed by type info and contents, contents being a sequence of phrases.  This is slow to parse (because each phrase must be decoded), and has invalid intermediate states (e.g. a Set whose contents are not all present, hence whose external hash may change during loading, etc).  A second generation system like Parcels and Fuel separates objects from references (nodes from arcs) and batches object creation, grouping all instances of a given class for bulk, and hence fast, instantiation.  Following my the objects are the references.  So loading has three phases:
- instantiate the objects
- connect the graph by filling in object references
- initialize the objects that require it (e.g. rehash sets)

Consequently the writer must be two pass, the first pass to collect the objects in the graph and sort them by class, the second to write the nodes followed by the references 

So let me ask again, does Hermes use the Parcels/Fuel architecture?

_,,,^..^,,,_ (phone)

On Aug 1, 2017, at 11:52 AM, "[hidden email]" <[hidden email]> wrote:

Hi Eliot, 
 
The last version of Hermes, that I have generated today and have to be tested for real yet, is able to load and save from and to 32 and 64 bits, handling the conversion. Because the Hermes representation (if we can call it like that, because is really simple, does not care about the architecture). 

Hermes is using a custom format completely separated from Fuel or any other tool. 

This is a consequence of the design of Hermes. Hermes has been designed to only work as light way of loading code during the bootstrap. So that, it is heavily coupled with Pharo (it can be changed, for example implementing another loader / installer) and uses only really core classes. 

Of course it can be extended and used in another Smalltalk dialects, but I am not sure if the tool can be useful for them, or even for the Pharo community outside the Pharo bootstrap process. 

Today to be useful outside the bootstrap process, some work has to be done (perhaps a lot), but mainly new use cases have to be thinked. Once the Compiler, Monticello or Metacello is available, there is no need for Hermes.  It is only used to generate some binary loadable packages for the compiler from the source code. 

Again, it was only thinked as a tool to improve the speed during the bootstrap. However, if somebody has ideas to use it or want to collaborate are  of course welcomed. 

Hermes is actually in Github, because to me is more comfortable to have it there,  but if somebody is really wanting to use it.... we can manage to have an export process to Smalltalkhub. 

I hope I have clarified a little the idea behind Hermes.

Maybe you have better ideas of what can we do with it. 


On Tue, Aug 1, 2017 at 7:59 PM, Eliot Miranda <[hidden email]> wrote:
Hi Guille,

On Aug 1, 2017, at 4:29 AM, Guillermo Polito <[hidden email]> wrote:



On Tue, Aug 1, 2017 at 12:57 PM, [hidden email] <[hidden email]> wrote:
Massive.

What is in the super small image?
Is Hermes going to be a generic binary content loader?

There is still work to do in this front. 
 - Hermes only works for 32bits format. We should adapt it to 64 bits (immediate floats and so on...)
 - There is no format validation. We should add one for safety.

Will Hermes be able to save on 32-bits and load on 64-bits and vice verse?

Does Hermes use the Parcels/Fuel architecture of saving nodes, grouped by class, followed by the arcs?

If yes to both of these, are you willing to keep it in Monticello and collaborate with the Squeak & Cuis communities in developing Hermes?

 
Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>


--

   

Guille Polito


Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr



Web: http://guillep.github.io

Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13




--
Pablo Tesone.
[hidden email]



--
Pablo Tesone.
[hidden email]



--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

tesonep@gmail.com
Hi, 
    Yes for sure there are objects, and that is very powerful, but comparing the algorithm of serialization of Hermes with Fuel (I don't really know Parcels) is not possible. It is like comparing Filetree with Fuel, The problems they have to take care are completely different. 

Maybe I am not seeing the point.

Cheers. 

On Tue, Aug 1, 2017 at 10:25 PM, Eliot Miranda <[hidden email]> wrote:
Hi Pablo,

On Tue, Aug 1, 2017 at 12:36 PM, [hidden email] <[hidden email]> wrote:
Hi again, 
   thanks for the clarification, I haven't understood the question initially, but now I think I can answer it.

Hermes is only exporting the classes, not objects. so it does not have to handle complex graphs of objects. Basically it serializes the definition of the classes, traits and methods. Then they are loaded in sequence. The only caring during the generation is to serialize first the superclasses, and then the subclasses. There is no way of serializing objects outside the classes, methods, traits and literals in a method. 

Classes, traits, methods, literals *are* a graph of objects :-). The Parcel architecture, from which Fuel derived its architecture, was designed for loading code in VisualWorks.  In fact, last time I looked Parcels were used only to store code and not as a general purpose (de)serializer.

So the answer is that is not using Parcels or Fuel architecture because they are intended for different uses.

Well, I don't think that's the reason ;-), but fine.  I understand that it doesn't use this architecture.  Thanks.
 
Also the format itself has no special design or considerations to be fast to generate or fast to read (as It happens with Fuel)

Cheers, 
Pablo

On Tue, Aug 1, 2017 at 9:16 PM, Eliot Miranda <[hidden email]> wrote:
Hi Pablo,

    I understand that Hermes has its own format.  My question was about architecture. The Parcels/Fuel architecture is, AFAIA, the best architecture for binary storage because it is virtual, is fast to load and cleanly separates loading from initialization.  What do I mean?

- concrete formats like ImageSegment are tied to internal representations in the VM and do are difficult to read in forever FB systems and very hard to write in foreign systems.  So a virtual format (like BOSS, Fuel, Parcels, etc) is better but unless done well can be slow

- a first generation system like BOSS is a flattening of a graph traversal using a simple grammar.  Each phrase is either an object reference (object id) or an object definition (object is followed by type info and contents, contents being a sequence of phrases.  This is slow to parse (because each phrase must be decoded), and has invalid intermediate states (e.g. a Set whose contents are not all present, hence whose external hash may change during loading, etc).  A second generation system like Parcels and Fuel separates objects from references (nodes from arcs) and batches object creation, grouping all instances of a given class for bulk, and hence fast, instantiation.  Following my the objects are the references.  So loading has three phases:
- instantiate the objects
- connect the graph by filling in object references
- initialize the objects that require it (e.g. rehash sets)

Consequently the writer must be two pass, the first pass to collect the objects in the graph and sort them by class, the second to write the nodes followed by the references 

So let me ask again, does Hermes use the Parcels/Fuel architecture?

_,,,^..^,,,_ (phone)

On Aug 1, 2017, at 11:52 AM, "[hidden email]" <[hidden email]> wrote:

Hi Eliot, 
 
The last version of Hermes, that I have generated today and have to be tested for real yet, is able to load and save from and to 32 and 64 bits, handling the conversion. Because the Hermes representation (if we can call it like that, because is really simple, does not care about the architecture). 

Hermes is using a custom format completely separated from Fuel or any other tool. 

This is a consequence of the design of Hermes. Hermes has been designed to only work as light way of loading code during the bootstrap. So that, it is heavily coupled with Pharo (it can be changed, for example implementing another loader / installer) and uses only really core classes. 

Of course it can be extended and used in another Smalltalk dialects, but I am not sure if the tool can be useful for them, or even for the Pharo community outside the Pharo bootstrap process. 

Today to be useful outside the bootstrap process, some work has to be done (perhaps a lot), but mainly new use cases have to be thinked. Once the Compiler, Monticello or Metacello is available, there is no need for Hermes.  It is only used to generate some binary loadable packages for the compiler from the source code. 

Again, it was only thinked as a tool to improve the speed during the bootstrap. However, if somebody has ideas to use it or want to collaborate are  of course welcomed. 

Hermes is actually in Github, because to me is more comfortable to have it there,  but if somebody is really wanting to use it.... we can manage to have an export process to Smalltalkhub. 

I hope I have clarified a little the idea behind Hermes.

Maybe you have better ideas of what can we do with it. 


On Tue, Aug 1, 2017 at 7:59 PM, Eliot Miranda <[hidden email]> wrote:
Hi Guille,

On Aug 1, 2017, at 4:29 AM, Guillermo Polito <[hidden email]> wrote:



On Tue, Aug 1, 2017 at 12:57 PM, [hidden email] <[hidden email]> wrote:
Massive.

What is in the super small image?
Is Hermes going to be a generic binary content loader?

There is still work to do in this front. 
 - Hermes only works for 32bits format. We should adapt it to 64 bits (immediate floats and so on...)
 - There is no format validation. We should add one for safety.

Will Hermes be able to save on 32-bits and load on 64-bits and vice verse?

Does Hermes use the Parcels/Fuel architecture of saving nodes, grouped by class, followed by the arcs?

If yes to both of these, are you willing to keep it in Monticello and collaborate with the Squeak & Cuis communities in developing Hermes?

 
Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>


--

   

Guille Polito


Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr



Web: http://guillep.github.io

Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13




--
Pablo Tesone.
[hidden email]



--
Pablo Tesone.
[hidden email]



--
_,,,^..^,,,_
best, Eliot



--
Pablo Tesone.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Mariano Martinez Peck


On Tue, Aug 1, 2017 at 5:31 PM, [hidden email] <[hidden email]> wrote:
Hi, 
    Yes for sure there are objects, and that is very powerful, but comparing the algorithm of serialization of Hermes with Fuel (I don't really know Parcels) is not possible. It is like comparing Filetree with Fuel, The problems they have to take care are completely different. 

Maybe I am not seeing the point.


Let me add one sentence that may explain Eliot questions... if you serialize the class definition, then it means you would need the compiler. I guess Eliot may be interested in NOT having the compiler at all and instead bring classes directly from binary format (like Fuel / Tanker )


 
Cheers. 

On Tue, Aug 1, 2017 at 10:25 PM, Eliot Miranda <[hidden email]> wrote:
Hi Pablo,

On Tue, Aug 1, 2017 at 12:36 PM, [hidden email] <[hidden email]> wrote:
Hi again, 
   thanks for the clarification, I haven't understood the question initially, but now I think I can answer it.

Hermes is only exporting the classes, not objects. so it does not have to handle complex graphs of objects. Basically it serializes the definition of the classes, traits and methods. Then they are loaded in sequence. The only caring during the generation is to serialize first the superclasses, and then the subclasses. There is no way of serializing objects outside the classes, methods, traits and literals in a method. 

Classes, traits, methods, literals *are* a graph of objects :-). The Parcel architecture, from which Fuel derived its architecture, was designed for loading code in VisualWorks.  In fact, last time I looked Parcels were used only to store code and not as a general purpose (de)serializer.

So the answer is that is not using Parcels or Fuel architecture because they are intended for different uses.

Well, I don't think that's the reason ;-), but fine.  I understand that it doesn't use this architecture.  Thanks.
 
Also the format itself has no special design or considerations to be fast to generate or fast to read (as It happens with Fuel)

Cheers, 
Pablo

On Tue, Aug 1, 2017 at 9:16 PM, Eliot Miranda <[hidden email]> wrote:
Hi Pablo,

    I understand that Hermes has its own format.  My question was about architecture. The Parcels/Fuel architecture is, AFAIA, the best architecture for binary storage because it is virtual, is fast to load and cleanly separates loading from initialization.  What do I mean?

- concrete formats like ImageSegment are tied to internal representations in the VM and do are difficult to read in forever FB systems and very hard to write in foreign systems.  So a virtual format (like BOSS, Fuel, Parcels, etc) is better but unless done well can be slow

- a first generation system like BOSS is a flattening of a graph traversal using a simple grammar.  Each phrase is either an object reference (object id) or an object definition (object is followed by type info and contents, contents being a sequence of phrases.  This is slow to parse (because each phrase must be decoded), and has invalid intermediate states (e.g. a Set whose contents are not all present, hence whose external hash may change during loading, etc).  A second generation system like Parcels and Fuel separates objects from references (nodes from arcs) and batches object creation, grouping all instances of a given class for bulk, and hence fast, instantiation.  Following my the objects are the references.  So loading has three phases:
- instantiate the objects
- connect the graph by filling in object references
- initialize the objects that require it (e.g. rehash sets)

Consequently the writer must be two pass, the first pass to collect the objects in the graph and sort them by class, the second to write the nodes followed by the references 

So let me ask again, does Hermes use the Parcels/Fuel architecture?

_,,,^..^,,,_ (phone)

On Aug 1, 2017, at 11:52 AM, "[hidden email]" <[hidden email]> wrote:

Hi Eliot, 
 
The last version of Hermes, that I have generated today and have to be tested for real yet, is able to load and save from and to 32 and 64 bits, handling the conversion. Because the Hermes representation (if we can call it like that, because is really simple, does not care about the architecture). 

Hermes is using a custom format completely separated from Fuel or any other tool. 

This is a consequence of the design of Hermes. Hermes has been designed to only work as light way of loading code during the bootstrap. So that, it is heavily coupled with Pharo (it can be changed, for example implementing another loader / installer) and uses only really core classes. 

Of course it can be extended and used in another Smalltalk dialects, but I am not sure if the tool can be useful for them, or even for the Pharo community outside the Pharo bootstrap process. 

Today to be useful outside the bootstrap process, some work has to be done (perhaps a lot), but mainly new use cases have to be thinked. Once the Compiler, Monticello or Metacello is available, there is no need for Hermes.  It is only used to generate some binary loadable packages for the compiler from the source code. 

Again, it was only thinked as a tool to improve the speed during the bootstrap. However, if somebody has ideas to use it or want to collaborate are  of course welcomed. 

Hermes is actually in Github, because to me is more comfortable to have it there,  but if somebody is really wanting to use it.... we can manage to have an export process to Smalltalkhub. 

I hope I have clarified a little the idea behind Hermes.

Maybe you have better ideas of what can we do with it. 


On Tue, Aug 1, 2017 at 7:59 PM, Eliot Miranda <[hidden email]> wrote:
Hi Guille,

On Aug 1, 2017, at 4:29 AM, Guillermo Polito <[hidden email]> wrote:



On Tue, Aug 1, 2017 at 12:57 PM, [hidden email] <[hidden email]> wrote:
Massive.

What is in the super small image?
Is Hermes going to be a generic binary content loader?

There is still work to do in this front. 
 - Hermes only works for 32bits format. We should adapt it to 64 bits (immediate floats and so on...)
 - There is no format validation. We should add one for safety.

Will Hermes be able to save on 32-bits and load on 64-bits and vice verse?

Does Hermes use the Parcels/Fuel architecture of saving nodes, grouped by class, followed by the arcs?

If yes to both of these, are you willing to keep it in Monticello and collaborate with the Squeak & Cuis communities in developing Hermes?

 
Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>


--

   

Guille Polito


Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr



Web: http://guillep.github.io

Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13




--
Pablo Tesone.
[hidden email]



--
Pablo Tesone.
[hidden email]



--
_,,,^..^,,,_
best, Eliot



--
Pablo Tesone.
[hidden email]



--
Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

tesonep@gmail.com
Hi Mariano, 
    The definition is not serialized. What is serialized is all the information to regenerate the class (eg: superclass name, instance variable names, class variables names, share pool names, category, layout class, and so on). This information is later used to regenerate the class as it is created initially by the class builder and installer. The same happens with the methods, they are built up from scratch without using the compiler. The bytecode is stored, the literals are serialized and the information in the header is serialized, as it has to be rebuilt when install. 

The same is applicable to the traits and extension methods. 

Basically it can be seen as a class builder / installer, that get all the information from the serialized version and not from the text representation of the class or the methods. 

That's why I said Hermes is so couple with Pharo. 

Thanks for clarifying, I am starting to understand where yours doubts came from. 

Cheers,
Pablo

On Tue, Aug 1, 2017 at 10:52 PM, Mariano Martinez Peck <[hidden email]> wrote:


On Tue, Aug 1, 2017 at 5:31 PM, [hidden email] <[hidden email]> wrote:
Hi, 
    Yes for sure there are objects, and that is very powerful, but comparing the algorithm of serialization of Hermes with Fuel (I don't really know Parcels) is not possible. It is like comparing Filetree with Fuel, The problems they have to take care are completely different. 

Maybe I am not seeing the point.


Let me add one sentence that may explain Eliot questions... if you serialize the class definition, then it means you would need the compiler. I guess Eliot may be interested in NOT having the compiler at all and instead bring classes directly from binary format (like Fuel / Tanker )


 
Cheers. 

On Tue, Aug 1, 2017 at 10:25 PM, Eliot Miranda <[hidden email]> wrote:
Hi Pablo,

On Tue, Aug 1, 2017 at 12:36 PM, [hidden email] <[hidden email]> wrote:
Hi again, 
   thanks for the clarification, I haven't understood the question initially, but now I think I can answer it.

Hermes is only exporting the classes, not objects. so it does not have to handle complex graphs of objects. Basically it serializes the definition of the classes, traits and methods. Then they are loaded in sequence. The only caring during the generation is to serialize first the superclasses, and then the subclasses. There is no way of serializing objects outside the classes, methods, traits and literals in a method. 

Classes, traits, methods, literals *are* a graph of objects :-). The Parcel architecture, from which Fuel derived its architecture, was designed for loading code in VisualWorks.  In fact, last time I looked Parcels were used only to store code and not as a general purpose (de)serializer.

So the answer is that is not using Parcels or Fuel architecture because they are intended for different uses.

Well, I don't think that's the reason ;-), but fine.  I understand that it doesn't use this architecture.  Thanks.
 
Also the format itself has no special design or considerations to be fast to generate or fast to read (as It happens with Fuel)

Cheers, 
Pablo

On Tue, Aug 1, 2017 at 9:16 PM, Eliot Miranda <[hidden email]> wrote:
Hi Pablo,

    I understand that Hermes has its own format.  My question was about architecture. The Parcels/Fuel architecture is, AFAIA, the best architecture for binary storage because it is virtual, is fast to load and cleanly separates loading from initialization.  What do I mean?

- concrete formats like ImageSegment are tied to internal representations in the VM and do are difficult to read in forever FB systems and very hard to write in foreign systems.  So a virtual format (like BOSS, Fuel, Parcels, etc) is better but unless done well can be slow

- a first generation system like BOSS is a flattening of a graph traversal using a simple grammar.  Each phrase is either an object reference (object id) or an object definition (object is followed by type info and contents, contents being a sequence of phrases.  This is slow to parse (because each phrase must be decoded), and has invalid intermediate states (e.g. a Set whose contents are not all present, hence whose external hash may change during loading, etc).  A second generation system like Parcels and Fuel separates objects from references (nodes from arcs) and batches object creation, grouping all instances of a given class for bulk, and hence fast, instantiation.  Following my the objects are the references.  So loading has three phases:
- instantiate the objects
- connect the graph by filling in object references
- initialize the objects that require it (e.g. rehash sets)

Consequently the writer must be two pass, the first pass to collect the objects in the graph and sort them by class, the second to write the nodes followed by the references 

So let me ask again, does Hermes use the Parcels/Fuel architecture?

_,,,^..^,,,_ (phone)

On Aug 1, 2017, at 11:52 AM, "[hidden email]" <[hidden email]> wrote:

Hi Eliot, 
 
The last version of Hermes, that I have generated today and have to be tested for real yet, is able to load and save from and to 32 and 64 bits, handling the conversion. Because the Hermes representation (if we can call it like that, because is really simple, does not care about the architecture). 

Hermes is using a custom format completely separated from Fuel or any other tool. 

This is a consequence of the design of Hermes. Hermes has been designed to only work as light way of loading code during the bootstrap. So that, it is heavily coupled with Pharo (it can be changed, for example implementing another loader / installer) and uses only really core classes. 

Of course it can be extended and used in another Smalltalk dialects, but I am not sure if the tool can be useful for them, or even for the Pharo community outside the Pharo bootstrap process. 

Today to be useful outside the bootstrap process, some work has to be done (perhaps a lot), but mainly new use cases have to be thinked. Once the Compiler, Monticello or Metacello is available, there is no need for Hermes.  It is only used to generate some binary loadable packages for the compiler from the source code. 

Again, it was only thinked as a tool to improve the speed during the bootstrap. However, if somebody has ideas to use it or want to collaborate are  of course welcomed. 

Hermes is actually in Github, because to me is more comfortable to have it there,  but if somebody is really wanting to use it.... we can manage to have an export process to Smalltalkhub. 

I hope I have clarified a little the idea behind Hermes.

Maybe you have better ideas of what can we do with it. 


On Tue, Aug 1, 2017 at 7:59 PM, Eliot Miranda <[hidden email]> wrote:
Hi Guille,

On Aug 1, 2017, at 4:29 AM, Guillermo Polito <[hidden email]> wrote:



On Tue, Aug 1, 2017 at 12:57 PM, [hidden email] <[hidden email]> wrote:
Massive.

What is in the super small image?
Is Hermes going to be a generic binary content loader?

There is still work to do in this front. 
 - Hermes only works for 32bits format. We should adapt it to 64 bits (immediate floats and so on...)
 - There is no format validation. We should add one for safety.

Will Hermes be able to save on 32-bits and load on 64-bits and vice verse?

Does Hermes use the Parcels/Fuel architecture of saving nodes, grouped by class, followed by the arcs?

If yes to both of these, are you willing to keep it in Monticello and collaborate with the Squeak & Cuis communities in developing Hermes?

 
Phil

On Aug 1, 2017 12:36, "Stephane Ducasse" <[hidden email]> wrote:
Hi Pavel

This is super excellent! IMPRESSIVE. An image without the compiler and
a reloadable compiler.
Super cool.

Stef


On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
<[hidden email]> wrote:
> Hello,
>
> we are checking a huge pull request #177
> (https://github.com/pharo-project/pharo/pull/177) that will change some
> basics of the bootstrap process:
>
> Now we will bootstrap a smaller image that will not include compiler/parser.
> Compiler and related packages are exported and loaded using a binary
> exporter named Hermes.
> The compiler is then used to load FileSystem and Monticello. The rest of the
> bootstrap process will be the same as before.
> As the result we will have faster bootstrap and better system modularization
> and possibilities.
>
> It required some modularization efforts:
>
> - simplified initialization scripts
> - Use Zinc converters and encoders instead of FilePathEncoder and old
> TextConverter
> - Use Stdio instead of FileStream
> - Using File instead of FileSystem
> - Deprecated FileStream & childs (Moved to Deprecated70)
> - Extracted Path classes to their on package: FileSystem-Path
> - Moved OpalEncoders to their own package. They are required by the runtime
> (not only for compilation)
> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
> and so on without requiring full Unicode tables from the beginning
> - Cleaning up a bit the full exception logging infrastructure (streams,
> transcript, files, stack printing...)
> - Split Ring methods required for system navigation to the Ring-Navigation
> package
> - Remove usages of #translated in the kernel
> - Refactored the bootstrapping classes to remove duplications
> - Cleaning up dependencies in CompiledMethod>>printOn:
> - fix path printing
>
> We need to merge these changes at once and of course it can cause some
> conflicts with the existing pull requests or external code. Anyway, we need
> to merge it as soon as possible.
>
> So please, try to look at the PR and test the resultant image [1] to avoid
> some major problems.
>
> [1]
> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>
> Cheers,
> -- Pavel
>


--

   

Guille Polito


Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr



Web: http://guillep.github.io

Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13




--
Pablo Tesone.
[hidden email]



--
Pablo Tesone.
[hidden email]



--
_,,,^..^,,,_
best, Eliot



--
Pablo Tesone.
[hidden email]



--



--
Pablo Tesone.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

abergel
In reply to this post by Pavel Krivanek-3
Impressive!!!
--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.



> On Aug 1, 2017, at 7:13 AM, Pavel Krivanek <[hidden email]> wrote:
>
> We already have an experimental reduced image with Hermes that is able to be bootstrapped under 2 minutes. The minimal system based on the current code-base that is able to be bootstrapped has about 260kB.
>


Reply | Threaded
Open this post in threaded view
|

Re: [IMPORTANT] Following changes in the bootstrapping process

Tudor Girba-2
In reply to this post by Stephane Ducasse-3
Wow!

This is impressive indeed.

Doru


> On Aug 1, 2017, at 12:35 PM, Stephane Ducasse <[hidden email]> wrote:
>
> Hi Pavel
>
> This is super excellent! IMPRESSIVE. An image without the compiler and
> a reloadable compiler.
> Super cool.
>
> Stef
>
>
> On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
> <[hidden email]> wrote:
>> Hello,
>>
>> we are checking a huge pull request #177
>> (https://github.com/pharo-project/pharo/pull/177) that will change some
>> basics of the bootstrap process:
>>
>> Now we will bootstrap a smaller image that will not include compiler/parser.
>> Compiler and related packages are exported and loaded using a binary
>> exporter named Hermes.
>> The compiler is then used to load FileSystem and Monticello. The rest of the
>> bootstrap process will be the same as before.
>> As the result we will have faster bootstrap and better system modularization
>> and possibilities.
>>
>> It required some modularization efforts:
>>
>> - simplified initialization scripts
>> - Use Zinc converters and encoders instead of FilePathEncoder and old
>> TextConverter
>> - Use Stdio instead of FileStream
>> - Using File instead of FileSystem
>> - Deprecated FileStream & childs (Moved to Deprecated70)
>> - Extracted Path classes to their on package: FileSystem-Path
>> - Moved OpalEncoders to their own package. They are required by the runtime
>> (not only for compilation)
>> - Introduced AsciiCharset in the kernel to answer to #isLetter #isUppercase
>> and so on without requiring full Unicode tables from the beginning
>> - Cleaning up a bit the full exception logging infrastructure (streams,
>> transcript, files, stack printing...)
>> - Split Ring methods required for system navigation to the Ring-Navigation
>> package
>> - Remove usages of #translated in the kernel
>> - Refactored the bootstrapping classes to remove duplications
>> - Cleaning up dependencies in CompiledMethod>>printOn:
>> - fix path printing
>>
>> We need to merge these changes at once and of course it can cause some
>> conflicts with the existing pull requests or external code. Anyway, we need
>> to merge it as soon as possible.
>>
>> So please, try to look at the PR and test the resultant image [1] to avoid
>> some major problems.
>>
>> [1]
>> https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>>
>> Cheers,
>> -- Pavel
>>
>

--
www.tudorgirba.com
www.feenk.com

"Speaking louder won't make the point worthier."


12