Smalltalk › Pharo › Pharo Smalltalk Developers

Pharo 7 and changes/sources files

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

12 messages Options

Pavel Krivanek-3

Pharo 7 and changes/sources files

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image

Pharo7.0-117934d.changes

Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.

You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-117934d.zip

Cheers,

-- Pavel

Ben Coman

Re: Pharo 7 and changes/sources files

On Fri, Jun 9, 2017 at 4:01 PM, Pavel Krivanek <[hidden email]> wrote:

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image
Pharo7.0-117934d.changes
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.

So what we need is for CompiledMethod>>#sourceCode to be based on git-hashes and directly access git, with one git repository shared by multiple Images.

Debugging into...

(Object>>#printString) sourceCode

I see the trailer has "kind == #SourcePointer"

and in #getSourceFromFile method #decodeSourcePointer does some interesting magic to calculate its "data" that later is returned by #sourcePointer in "SourceFiles sourceCodeAt: self sourcePointer"

where SourceFiles is ...PharoV60.sources and ...PharoV60.changes.

So I guess we'd need something like "kind == #ContentHash"

and then in CompiledMethod>>sourceCode ahead of getSourceFromFile

we add ...

[ self getSourceFromUserLocalGit: contentHash ] ifNotNil: [:s | ^s].

[ self getSourceFromUserGitServer: contentHash ] ifNotNil: [:s | ^s].

[ self getSourceFromPharoGitServer: contentHash ] ifNotNil: [:s | ^s].

[ self getSourceFromPrivateEncryptedRepository: contentHash ] ifNotNil: [:s | ^s].

[ self getSourceFromIPFS: contentHash ] ifNotNil: [:s | ^s].

and such like. Perhaps some would need separate "kind"s,

and this is oversimplistic since I haven't considered saving sources,

but you get the idea.

cheers -ben

Uko2

Re: Pharo 7 and changes/sources files

In reply to this post by Pavel Krivanek-3

Why cannot we have the sources in the image?

When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I don’t care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?

Uko

On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image
Pharo7.0-117934d.changes
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.

You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-117934d.zip

Cheers,
-- Pavel

Pavel Krivanek-3

Re: Pharo 7 and changes/sources files

We plan to change the sources management. But not immediately.

-- Pavel

2017-06-09 14:51 GMT+02:00 Yuriy Tymchuk <[hidden email]>:

Why cannot we have the sources in the image?

When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I don’t care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?

Uko

On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image
Pharo7.0-117934d.changes
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.

You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-117934d.zip

Cheers,
-- Pavel

Guillermo Polito

Re: Pharo 7 and changes/sources files

In reply to this post by Uko2

On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <[hidden email]> wrote:

Why cannot we have the sources in the image?

When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I don’t care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?

We can, but that's something that requires changing lot's of things inside the image and it requires probably lot of testing.

So I'm not against it, but we cannot just make a change like this as we are right now facing lots of key changes at the same time:

- migration to git, iceberg

- bootstrap based process

Uko

On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image
Pharo7.0-117934d.changes
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.

You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-117934d.zip

Cheers,
-- Pavel

Guille Polito

Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr

Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Uko2

Re: Pharo 7 and changes/sources files

In reply to this post by Pavel Krivanek-3

Aha, good to know!

On 9 Jun 2017, at 14:57, Pavel Krivanek <[hidden email]> wrote:

We plan to change the sources management. But not immediately.

-- Pavel

2017-06-09 14:51 GMT+02:00 Yuriy Tymchuk <[hidden email]>:
Why cannot we have the sources in the image?

When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I don’t care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?

Uko

On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image
Pharo7.0-117934d.changes
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.

You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-117934d.zip

Cheers,
-- Pavel

K K Subbu

Re: Pharo 7 and changes/sources files

In reply to this post by Guillermo Polito

On Friday 09 June 2017 06:38 PM, Guillermo Polito wrote:
> On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <[hidden email]
> When I asked it 2 years ago people were like: "oh, but you can just
> download 1 sources file and use it by all the images". I don’t care
> about that micro space savings, but fine, I said: have it your way.
> Now as we have to download sources all the time, why cannot we just
> have it IN the image?
>
> We can, but that's something that requires changing lot's of things
> inside the image and it requires probably lot of testing.

What if the sources is appended to the image file with a special header.
New code can relocate this pointer on the fly. The same technique can be
applied to changes file too. A small script can split files for older
code or combine them for new code.

The resulting image file will be backward compatible. New code can
preserve these files across saves.

Just a thought .. Subbu

Yanni Chiu-2

Re: Pharo 7 and changes/sources files

In reply to this post by Guillermo Polito

On Fri, Jun 9, 2017 at 9:08 AM, Guillermo Polito <[hidden email]> wrote:

On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <[hidden email]> wrote:
Why cannot we have the sources in the image?

We can, but that's something that requires changing lot's of things inside the image and it requires probably lot of testing.

So I'm not against it, but we cannot just make a change like this as we are right now facing lots of key changes at the same time:
- migration to git, iceberg
- bootstrap based process

How about saving the sources in-memory via an instance of MemoryStore, so it should still look just like a file.

Ben Coman

Re: Pharo 7 and changes/sources files

In reply to this post by K K Subbu

On Fri, Jun 9, 2017 at 10:01 PM, K K Subbu <[hidden email]> wrote:

On Friday 09 June 2017 06:38 PM, Guillermo Polito wrote:

On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <[hidden email]
When I asked it 2 years ago people were like: "oh, but you can just
download 1 sources file and use it by all the images". I don’t care
about that micro space savings, but fine, I said: have it your way.
Now as we have to download sources all the time, why cannot we just
have it IN the image?

We can, but that's something that requires changing lot's of things
inside the image and it requires probably lot of testing.

What if the sources is appended to the image file with a special header. New code can relocate this pointer on the fly. The same technique can be applied to changes file too. A small script can split files for older code or combine them for new code.

The resulting image file will be backward compatible. New code can preserve these files across saves.

Just a thought .. Subbu

Or perhaps MemoryFileSystem could be effective?

But no rush.

cheers -ben

Eliot Miranda-2

Re: Pharo 7 and changes/sources files

In reply to this post by Uko2

Hi Yiury,

On Fri, Jun 9, 2017 at 5:51 AM, Yuriy Tymchuk <[hidden email]> wrote:

Why cannot we have the sources in the image?

When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I don’t care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?

It's not a "micro saving". A Pharo6 image is about 48Mb and a Pharo sources file is about 32Mb. So you'd increase the size of the image by 75%. Further, that space has to be dealt with by the GC. The compactor may have to move this objects around; so it's certainly overhead. It might make sense for development, especially if you use Igor's include-source-in-the-method-trailer approach. But it doesn't make sense for deployment where the application isn't going to look at the sources anyway. So why put a heavy burden on deploying applications when the current system works, and off loads a substantial amount of memory from the system?

Uko

On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image
Pharo7.0-117934d.changes
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.

You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-117934d.zip

Cheers,
-- Pavel

_,,,^..^,,,_

best, Eliot

Stephane Ducasse-3

Re: Pharo 7 and changes/sources files

+ 1 :)
It is just a file and this is only for the development of Pharo and
for users of Pharo 70 it will be exactly the same.
Let us focus on the place where we can get a good impact/ratio/plus
for business/plus for innovation.
I have a lot of dreams for Pharo: from the coffee machine to the big
data servers and this story with source files is not interesting.

We download thousands of files in any application.
So let us focus our energy on something else.

stef

On Sat, Jun 10, 2017 at 6:13 AM, Eliot Miranda <[hidden email]> wrote:

> Hi Yiury,
>
> On Fri, Jun 9, 2017 at 5:51 AM, Yuriy Tymchuk <[hidden email]> wrote:
>>
>> Why cannot we have the sources in the image?
>>
>> When I asked it 2 years ago people were like: "oh, but you can just
>> download 1 sources file and use it by all the images". I don’t care about
>> that micro space savings, but fine, I said: have it your way. Now as we have
>> to download sources all the time, why cannot we just have it IN the image?
>
>
> It's not a "micro saving". A Pharo6 image is about 48Mb and a Pharo sources
> file is about 32Mb. So you'd increase the size of the image by 75%.
> Further, that space has to be dealt with by the GC. The compactor may have
> to move this objects around; so it's certainly overhead. It might make
> sense for development, especially if you use Igor's
> include-source-in-the-method-trailer approach. But it doesn't make sense
> for deployment where the application isn't going to look at the sources
> anyway. So why put a heavy burden on deploying applications when the
> current system works, and off loads a substantial amount of memory from the
> system?
>
>>
>>
>> Uko
>>
>>
>> On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:
>>
>> Hi,
>>
>> because during the image bootstrapping all source code is reloaded, it is
>> all placed in the changes file and the sources file is useless. It would be
>> strange to deliver Pharo with empty sources file and big changes so we
>> decided to do sources compacting for every build.
>>
>> So Pharo 7 development versions will be delivered in an archive with three
>> files named like:
>>
>> Pharo7.0-117934d.image
>> Pharo7.0-117934d.changes
>> Pharo7.0-117934d.sources
>>
>> where "117934d" is an abbreviated hash of the Git commit from which the
>> Pharo image was bootstrapped. The sources file will be build-specific and of
>> course you will be able to share it between images based on the same build.
>> In changes file you will have only you own changes.
>>
>> You can test it here:
>> https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-117934d.zip
>>
>> Cheers,
>> -- Pavel
>>
>>
>
>
>
> --
> _,,,^..^,,,_
> best, Eliot

Stephane Ducasse-3

Re: Pharo 7 and changes/sources files

In reply to this post by Pavel Krivanek-3

This is excellent to get the bootstrap in
production.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

On Fri, Jun 9, 2017 at 10:01 AM, Pavel Krivanek
<[hidden email]> wrote:

> Hi,
>
> because during the image bootstrapping all source code is reloaded, it is
> all placed in the changes file and the sources file is useless. It would be
> strange to deliver Pharo with empty sources file and big changes so we
> decided to do sources compacting for every build.
>
> So Pharo 7 development versions will be delivered in an archive with three
> files named like:
>
> Pharo7.0-117934d.image
> Pharo7.0-117934d.changes
> Pharo7.0-117934d.sources
>
> where "117934d" is an abbreviated hash of the Git commit from which the
> Pharo image was bootstrapped. The sources file will be build-specific and of
> course you will be able to share it between images based on the same build.
> In changes file you will have only you own changes.
>
> You can test it here:
> https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-117934d.zip
>
> Cheers,
> -- Pavel
>