Pharo 7 and changes/sources files

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Pharo 7 and changes/sources files

Pavel Krivanek-3
Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image    
Pharo7.0-117934d.changes  
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.


Cheers,
-- Pavel

Reply | Threaded
Open this post in threaded view
|

Re: Pharo 7 and changes/sources files

Ben Coman


On Fri, Jun 9, 2017 at 4:01 PM, Pavel Krivanek <[hidden email]> wrote:
Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image    
Pharo7.0-117934d.changes  
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.



So what we need is for CompiledMethod>>#sourceCode to be based on git-hashes and directly access git, with one git repository shared by multiple Images.

Debugging into...   
     (Object>>#printString) sourceCode
I see the trailer has "kind == #SourcePointer" 
and in #getSourceFromFile method #decodeSourcePointer does some interesting magic to calculate its "data" that later is returned by #sourcePointer in  "SourceFiles sourceCodeAt: self sourcePointer"
where SourceFiles is ...PharoV60.sources and ...PharoV60.changes.

So I guess we'd need something like "kind == #ContentHash"
and then in CompiledMethod>>sourceCode ahead of getSourceFromFile 
we add ...
   [ self getSourceFromUserLocalGit: contentHash ] ifNotNil: [:s | ^s].
   [ self getSourceFromUserGitServer: contentHash ] ifNotNil: [:s | ^s].
   [ self getSourceFromPharoGitServer: contentHash ] ifNotNil: [:s | ^s].
   [ self getSourceFromPrivateEncryptedRepository: contentHash ] ifNotNil: [:s | ^s].
   [ self getSourceFromIPFS: contentHash ] ifNotNil: [:s | ^s].
and such like.  Perhaps some would need separate "kind"s, 
and this is oversimplistic since I haven't considered saving sources,
but you get the idea.

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: Pharo 7 and changes/sources files

Uko2
In reply to this post by Pavel Krivanek-3
Why cannot we have the sources in the image?

When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I don’t care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?

Uko

On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image    
Pharo7.0-117934d.changes  
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.


Cheers,
-- Pavel


Reply | Threaded
Open this post in threaded view
|

Re: Pharo 7 and changes/sources files

Pavel Krivanek-3
We plan to change the sources management. But not immediately.

-- Pavel

2017-06-09 14:51 GMT+02:00 Yuriy Tymchuk <[hidden email]>:
Why cannot we have the sources in the image?

When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I don’t care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?

Uko


On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image    
Pharo7.0-117934d.changes  
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.


Cheers,
-- Pavel



Reply | Threaded
Open this post in threaded view
|

Re: Pharo 7 and changes/sources files

Guillermo Polito
In reply to this post by Uko2


On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <[hidden email]> wrote:
Why cannot we have the sources in the image? 

When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I don’t care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?

We can, but that's something that requires changing lot's of things inside the image and it requires probably lot of testing.

So I'm not against it, but we cannot just make a change like this as we are right now facing lots of key changes at the same time:
  - migration to git, iceberg
  - bootstrap based process
 

Uko


On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image    
Pharo7.0-117934d.changes  
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.


Cheers,
-- Pavel





--

   

Guille Polito


Research Engineer

French National Center for Scientific Research - http://www.cnrs.fr



Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Reply | Threaded
Open this post in threaded view
|

Re: Pharo 7 and changes/sources files

Uko2
In reply to this post by Pavel Krivanek-3
Aha, good to know!

On 9 Jun 2017, at 14:57, Pavel Krivanek <[hidden email]> wrote:

We plan to change the sources management. But not immediately.

-- Pavel

2017-06-09 14:51 GMT+02:00 Yuriy Tymchuk <[hidden email]>:
Why cannot we have the sources in the image?

When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I don’t care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?

Uko


On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image    
Pharo7.0-117934d.changes  
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.


Cheers,
-- Pavel




Reply | Threaded
Open this post in threaded view
|

Re: Pharo 7 and changes/sources files

K K Subbu
In reply to this post by Guillermo Polito
On Friday 09 June 2017 06:38 PM, Guillermo Polito wrote:
> On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <[hidden email]
>     When I asked it 2 years ago people were like: "oh, but you can just
>     download 1 sources file and use it by all the images". I don’t care
>     about that micro space savings, but fine, I said: have it your way.
>     Now as we have to download sources all the time, why cannot we just
>     have it IN the image?
>
> We can, but that's something that requires changing lot's of things
> inside the image and it requires probably lot of testing.

What if the sources is appended to the image file with a special header.
New code can relocate this pointer on the fly. The same technique can be
applied to changes file too. A small script can split files for older
code or combine them for new code.

The resulting image file will be backward compatible. New code can
preserve these files across saves.

Just a thought .. Subbu

Reply | Threaded
Open this post in threaded view
|

Re: Pharo 7 and changes/sources files

Yanni Chiu-2
In reply to this post by Guillermo Polito
On Fri, Jun 9, 2017 at 9:08 AM, Guillermo Polito <[hidden email]> wrote:

On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <[hidden email]> wrote:
Why cannot we have the sources in the image? 

We can, but that's something that requires changing lot's of things inside the image and it requires probably lot of testing.

So I'm not against it, but we cannot just make a change like this as we are right now facing lots of key changes at the same time:
  - migration to git, iceberg
  - bootstrap based process

 How about saving the sources in-memory via an instance of MemoryStore, so it should still look just like a file.


Reply | Threaded
Open this post in threaded view
|

Re: Pharo 7 and changes/sources files

Ben Coman
In reply to this post by K K Subbu


On Fri, Jun 9, 2017 at 10:01 PM, K K Subbu <[hidden email]> wrote:
On Friday 09 June 2017 06:38 PM, Guillermo Polito wrote:
On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <[hidden email]
    When I asked it 2 years ago people were like: "oh, but you can just
    download 1 sources file and use it by all the images". I don’t care
    about that micro space savings, but fine, I said: have it your way.
    Now as we have to download sources all the time, why cannot we just
    have it IN the image?

We can, but that's something that requires changing lot's of things
inside the image and it requires probably lot of testing.

What if the sources is appended to the image file with a special header. New code can relocate this pointer on the fly. The same technique can be applied to changes file too. A small script can split files for older code or combine them for new code.

The resulting image file will be backward compatible. New code can preserve these files across saves.

Just a thought .. Subbu


Or perhaps MemoryFileSystem could be effective?
But no rush.  

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: Pharo 7 and changes/sources files

Eliot Miranda-2
In reply to this post by Uko2
Hi Yiury,

On Fri, Jun 9, 2017 at 5:51 AM, Yuriy Tymchuk <[hidden email]> wrote:
Why cannot we have the sources in the image?

When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I don’t care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?

It's not a "micro saving".  A Pharo6 image is about 48Mb and a Pharo sources file is about 32Mb.  So you'd increase the size of the image by 75%.  Further, that space has to be dealt with by the GC.  The compactor may have to move this objects around; so it's certainly overhead.  It might make sense for development, especially if you use Igor's include-source-in-the-method-trailer approach.  But it doesn't make sense for deployment where the application isn't going to look at the sources anyway.  So why put a heavy burden on deploying applications when the current system works, and off loads a substantial amount of memory from the system?
 

Uko


On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:

Hi,

because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.

So Pharo 7 development versions will be delivered in an archive with three files named like:

Pharo7.0-117934d.image    
Pharo7.0-117934d.changes  
Pharo7.0-117934d.sources

where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.


Cheers,
-- Pavel





--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Pharo 7 and changes/sources files

Stephane Ducasse-3
+ 1 :)
It is just a file and this is only for the development of Pharo and
for users of Pharo 70 it will be exactly the same.
Let us focus on the place where we can get a good impact/ratio/plus
for business/plus for innovation.
I have a lot of dreams for Pharo: from the coffee machine to the big
data servers and this story with source files is not interesting.

We download thousands of files in any application.
So let us focus our energy on something else.

stef

On Sat, Jun 10, 2017 at 6:13 AM, Eliot Miranda <[hidden email]> wrote:

> Hi Yiury,
>
> On Fri, Jun 9, 2017 at 5:51 AM, Yuriy Tymchuk <[hidden email]> wrote:
>>
>> Why cannot we have the sources in the image?
>>
>> When I asked it 2 years ago people were like: "oh, but you can just
>> download 1 sources file and use it by all the images". I don’t care about
>> that micro space savings, but fine, I said: have it your way. Now as we have
>> to download sources all the time, why cannot we just have it IN the image?
>
>
> It's not a "micro saving".  A Pharo6 image is about 48Mb and a Pharo sources
> file is about 32Mb.  So you'd increase the size of the image by 75%.
> Further, that space has to be dealt with by the GC.  The compactor may have
> to move this objects around; so it's certainly overhead.  It might make
> sense for development, especially if you use Igor's
> include-source-in-the-method-trailer approach.  But it doesn't make sense
> for deployment where the application isn't going to look at the sources
> anyway.  So why put a heavy burden on deploying applications when the
> current system works, and off loads a substantial amount of memory from the
> system?
>
>>
>>
>> Uko
>>
>>
>> On 9 Jun 2017, at 10:01, Pavel Krivanek <[hidden email]> wrote:
>>
>> Hi,
>>
>> because during the image bootstrapping all source code is reloaded, it is
>> all placed in the changes file and the sources file is useless. It would be
>> strange to deliver Pharo with empty sources file and big changes so we
>> decided to do sources compacting for every build.
>>
>> So Pharo 7 development versions will be delivered in an archive with three
>> files named like:
>>
>> Pharo7.0-117934d.image
>> Pharo7.0-117934d.changes
>> Pharo7.0-117934d.sources
>>
>> where "117934d" is an abbreviated hash of the Git commit from which the
>> Pharo image was bootstrapped. The sources file will be build-specific and of
>> course you will be able to share it between images based on the same build.
>> In changes file you will have only you own changes.
>>
>> You can test it here:
>> https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-117934d.zip
>>
>> Cheers,
>> -- Pavel
>>
>>
>
>
>
> --
> _,,,^..^,,,_
> best, Eliot

Reply | Threaded
Open this post in threaded view
|

Re: Pharo 7 and changes/sources files

Stephane Ducasse-3
In reply to this post by Pavel Krivanek-3
This is excellent to get the bootstrap in
production.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

On Fri, Jun 9, 2017 at 10:01 AM, Pavel Krivanek
<[hidden email]> wrote:

> Hi,
>
> because during the image bootstrapping all source code is reloaded, it is
> all placed in the changes file and the sources file is useless. It would be
> strange to deliver Pharo with empty sources file and big changes so we
> decided to do sources compacting for every build.
>
> So Pharo 7 development versions will be delivered in an archive with three
> files named like:
>
> Pharo7.0-117934d.image
> Pharo7.0-117934d.changes
> Pharo7.0-117934d.sources
>
> where "117934d" is an abbreviated hash of the Git commit from which the
> Pharo image was bootstrapped. The sources file will be build-specific and of
> course you will be able to share it between images based on the same build.
> In changes file you will have only you own changes.
>
> You can test it here:
> https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-117934d.zip
>
> Cheers,
> -- Pavel
>