The code in the src folders in the repository

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

The code in the src folders in the repository

Laura Perez Cerrato
 
Hi everyone,

Excuse me if this has been asked already or it's documented somewhere and I missed it, but what's the criteria to update the code in /src, /stacksrc, /spursrc and other similar folders in the repository?
 
-Laura Perez Cerrato
Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

David T. Lewis
 
On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
>  
> Hi everyone,
>
> Excuse me if this has been asked already or it's documented somewhere and I
> missed it, but what's the criteria to update the code in /src, /stacksrc,
> /spursrc and other similar folders in the repository?
>
> -Laura Perez Cerrato

Good question. Traditionally, Eliot generates these sources periodically and
commits them to the repository (and in recent years I have done that chore for
the old trunk interpreter VM). At this point, unless Eliot advises otherwise,
I would suggest that no one other than Eliot should commit to the /src trees.

Dave

Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

fniephaus
 
AFAIK the pharo-vm projects generates the sources from the VMMaker package during a build.
Wouldn't it be better if the OpenSmalltalk vm does the same? Then no one needs to generate source manually anymore and we don't have millions of lines [2] of generated code in the repository.

Fabio


--

On Wed, Jun 22, 2016 at 4:02 AM David T. Lewis <[hidden email]> wrote:

On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
>
> Hi everyone,
>
> Excuse me if this has been asked already or it's documented somewhere and I
> missed it, but what's the criteria to update the code in /src, /stacksrc,
> /spursrc and other similar folders in the repository?
>
> -Laura Perez Cerrato

Good question. Traditionally, Eliot generates these sources periodically and
commits them to the repository (and in recent years I have done that chore for
the old trunk interpreter VM). At this point, unless Eliot advises otherwise,
I would suggest that no one other than Eliot should commit to the /src trees.

Dave

Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

NorbertHartl
 

Am 22.06.2016 um 09:51 schrieb Fabio Niephaus <[hidden email]>:

AFAIK the pharo-vm projects generates the sources from the VMMaker package during a build.
Wouldn't it be better if the OpenSmalltalk vm does the same? Then no one needs to generate source manually anymore and we don't have millions of lines [2] of generated code in the repository.

How would you release a version of the vm? This is only possible of you archive the static artefacts.

Norbert

Fabio


--

On Wed, Jun 22, 2016 at 4:02 AM David T. Lewis <[hidden email]> wrote:

On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
>
> Hi everyone,
>
> Excuse me if this has been asked already or it's documented somewhere and I
> missed it, but what's the criteria to update the code in /src, /stacksrc,
> /spursrc and other similar folders in the repository?
>
> -Laura Perez Cerrato

Good question. Traditionally, Eliot generates these sources periodically and
commits them to the repository (and in recent years I have done that chore for
the old trunk interpreter VM). At this point, unless Eliot advises otherwise,
I would suggest that no one other than Eliot should commit to the /src trees.

Dave


Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

philippeback
 
One could thing of doing a git submodule with them but it is more trouble than it is worth.

I want to be able to clone the VM and compile it right away.

Phil

On Wed, Jun 22, 2016 at 10:11 AM, Norbert Hartl <[hidden email]> wrote:
 

Am 22.06.2016 um 09:51 schrieb Fabio Niephaus <[hidden email]>:

AFAIK the pharo-vm projects generates the sources from the VMMaker package during a build.
Wouldn't it be better if the OpenSmalltalk vm does the same? Then no one needs to generate source manually anymore and we don't have millions of lines [2] of generated code in the repository.

How would you release a version of the vm? This is only possible of you archive the static artefacts.

Norbert

Fabio


--

On Wed, Jun 22, 2016 at 4:02 AM David T. Lewis <[hidden email]> wrote:

On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
>
> Hi everyone,
>
> Excuse me if this has been asked already or it's documented somewhere and I
> missed it, but what's the criteria to update the code in /src, /stacksrc,
> /spursrc and other similar folders in the repository?
>
> -Laura Perez Cerrato

Good question. Traditionally, Eliot generates these sources periodically and
commits them to the repository (and in recent years I have done that chore for
the old trunk interpreter VM). At this point, unless Eliot advises otherwise,
I would suggest that no one other than Eliot should commit to the /src trees.

Dave




Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

EstebanLM
 

On 22 Jun 2016, at 12:52, [hidden email] wrote:

One could thing of doing a git submodule with them but it is more trouble than it is worth.

I want to be able to clone the VM and compile it right away.

Phil

On Wed, Jun 22, 2016 at 10:11 AM, Norbert Hartl <[hidden email]> wrote:
 

Am 22.06.2016 um 09:51 schrieb Fabio Niephaus <[hidden email]>:

AFAIK the pharo-vm projects generates the sources from the VMMaker package during a build.
Wouldn't it be better if the OpenSmalltalk vm does the same? Then no one needs to generate source manually anymore and we don't have millions of lines [2] of generated code in the repository.

How would you release a version of the vm? This is only possible of you archive the static artefacts.

I do not understand this question, so maybe I’m answering anything :)
in Pharo, VMMaker package sources coexist along with C sources (thanks to filetree for now, but we are working on enhance that). This way, each release of VM in github contains *exactly* all the sources needed to reconstruct it… 

Esteban 


Norbert

Fabio


--

On Wed, Jun 22, 2016 at 4:02 AM David T. Lewis <[hidden email]> wrote:

On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
>
> Hi everyone,
>
> Excuse me if this has been asked already or it's documented somewhere and I
> missed it, but what's the criteria to update the code in /src, /stacksrc,
> /spursrc and other similar folders in the repository?
>
> -Laura Perez Cerrato

Good question. Traditionally, Eliot generates these sources periodically and
commits them to the repository (and in recent years I have done that chore for
the old trunk interpreter VM). At this point, unless Eliot advises otherwise,
I would suggest that no one other than Eliot should commit to the /src trees.

Dave





Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

David T. Lewis
In reply to this post by fniephaus
 
On Wed, Jun 22, 2016 at 07:51:11AM +0000, Fabio Niephaus wrote:

>  
> On Wed, Jun 22, 2016 at 4:02 AM David T. Lewis <[hidden email]> wrote:
>
> >
> > On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
> > >
> > > Hi everyone,
> > >
> > > Excuse me if this has been asked already or it's documented somewhere
> > and I
> > > missed it, but what's the criteria to update the code in /src, /stacksrc,
> > > /spursrc and other similar folders in the repository?
> > >
> > > -Laura Perez Cerrato
> >
> > Good question. Traditionally, Eliot generates these sources periodically
> > and
> > commits them to the repository (and in recent years I have done that chore
> > for
> > the old trunk interpreter VM). At this point, unless Eliot advises
> > otherwise,
> > I would suggest that no one other than Eliot should commit to the /src
> > trees.
> >
> >
> AFAIK the pharo-vm projects generates the sources from the VMMaker package
> during a build.
> Wouldn't it be better if the OpenSmalltalk vm does the same? Then no one
> needs to generate source manually anymore and we don't have millions of
> lines [2] of generated code in the repository.
>
> Fabio
>
> [1] https://github.com/pharo-project/pharo-vm
> [2] see Eliot's stats at
> https://github.com/OpenSmalltalk/vm/graphs/contributors
>
> --

Yes of course it should be automated. My suggestion is that at this point,
or at least until Eliot says otherwise, Eliot should be the only person who
commits to the /src directories. One step at a time :-)

And of course we can do whatever is necessary in the various branches.

Dave

Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

Eliot Miranda-2
In reply to this post by fniephaus
 
Hi Fabio,

On Jun 22, 2016, at 12:51 AM, Fabio Niephaus <[hidden email]> wrote:

AFAIK the pharo-vm projects generates the sources from the VMMaker package during a build.
Wouldn't it be better if the OpenSmalltalk vm does the same? Then no one needs to generate source manually anymore and we don't have millions of lines [2] of generated code in the repository.

For me this is a very very bad idea.  There are two main objections.  The main problem is that it means there is no versioning of the key source of the VM.

- one cannot ever guarantee generating exactly the same source that was fed to the compiler in a particular build
-- elements of the type inferrence code are affected by hash computations since they use dictionaries etc, and so there are small variations from run to run.  These shouldn't exist and I work to stabilize the type inferrencer but it isn't yet perfect
-- the VMMaker and Cog packages make use of the underlying system and it's conceivable that differences in the generated source, in inessentials like white space, could bubble up into the generated source, especially considering the differences between Pharo and Squeak, which are only widening.

This is disastrous for VM debugging.  One /must/ have a versioned artifact to be able to attach a debugger and set of sources to an already-built production VM and be able to be debugging the VM that corresponds with that source.

This is disastrous for spitting problems with the type inferrencer.  If source is generated and sent to the computer there's no point at which the VM developer gets to check that the difference between one version of the sources and another are as expected.  Often bugs introduced into the type inferrencer are noticed and understood by looking at the changes to the generated source before checking in that source.  Remove this stage and these bugs will only be found much further down stream at much greater cost and at enormous cost to examine because one will have to set up two different images to generate the two versions and manually duff them.

Note that the type inferrencer is key to being able to write clean Smalltalk code for the VM with minimal duplication that works for both 32- and 64- but Spur, and in general.  It allows one to omit C type declaration pragmas from much of the VM source, which is key to a particular method working in 32- and 64-bit versions.  I'm sure that if we ever wanted to try and merge in SqueakJS it would be even more important.


The second problem is performance.  Generating one of the sources takes a substantial fraction of a minute on a fast machine.  On an ARM it takes more than a minute.  Add to that the time taken to download a VM and image and that's more time.  Add to that the loading of VMMaker, Cog and prerequisite packages and now one is in a position where Travis time limits could prevent a buff on google's ARM infrastructure.

So please, no.  This may seem like a pure approach but it is, in fact, insane.  It means that anyone doing serious VM development has to work a lot harder and wait a lot longer to get their work done.  It destroys their productivity.

Fabio


--

On Wed, Jun 22, 2016 at 4:02 AM David T. Lewis <[hidden email]> wrote:

On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
>
> Hi everyone,
>
> Excuse me if this has been asked already or it's documented somewhere and I
> missed it, but what's the criteria to update the code in /src, /stacksrc,
> /spursrc and other similar folders in the repository?
>
> -Laura Perez Cerrato

Good question. Traditionally, Eliot generates these sources periodically and
commits them to the repository (and in recent years I have done that chore for
the old trunk interpreter VM). At this point, unless Eliot advises otherwise,
I would suggest that no one other than Eliot should commit to the /src trees.

Dave

Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

Eliot Miranda-2
In reply to this post by Laura Perez Cerrato
 
Hi Laura,

On Jun 21, 2016, at 7:11 AM, Laura Perez Cerrato <[hidden email]> wrote:

Hi everyone,

Excuse me if this has been asked already or it's documented somewhere and I missed it, but what's the criteria to update the code in /src, /stacksrc, /spursrc and other similar folders in the repository?

I should document this.  There are three main criteria:

- does the source need to be updated to fix a bug or make available new functionality urgently needed?  Since up until now my VM builds have been manually kicked off on the three platforms it took a while and I'd only do new builds when necessary

- does the source look sane?  One should always look at the differences to the generated source if any part of Slang's translator has been modified to check for bugs.

- is the source generated from versioned input?  The generated C source is marked with the Monticello package version, including UUID, of the packages from which it was created (eg a package providing a plugin) and that created it (i.e. the VMMaker package).  These marks will contain asterisks for modified Monticello packages.  Clearly one should never check in source containing asterisks.  The one exception I've allowed myself is MiscPrimitivesPlugin which gets generated from base image code, and I have a few choice base image modifications (such as a much higher character limit on the Transcript, or different code highlighting) that make it really painful to keep MiscPrimitivesPlugin clean.

I'm happy for others to generate and check in VM source provided:
- you carefully check the generated source before checking it in to verify that the delta is as expected and no new bugs have been introduced
- you generate as little change as possible.
-- There is a script that reverts generated plugins that have changed only in their Monticello markings that allows us to not checkin inessential revisions to plugins and see when I fact they did change
-- wrapping the generation code with [...] valueAnswering: false (or whatever the selector is) avoids regenerating VM headers (cointerp.h cogit.h cogmethod.h interp.h) when these haven't been modified

In short, only do this if you know what you're doing.  But got people to know what they're doing the process has to be documented along with the rationale.  I guess this email and my reply to Fabio's question can serve as input to that documentation.

-Laura Perez Cerrato

_,,,^..^,,,_ (phone)
Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

Eliot Miranda-2
In reply to this post by David T. Lewis



> On Jun 22, 2016, at 5:38 AM, David T. Lewis <[hidden email]> wrote:
>
>
>> On Wed, Jun 22, 2016 at 07:51:11AM +0000, Fabio Niephaus wrote:
>>
>>> On Wed, Jun 22, 2016 at 4:02 AM David T. Lewis <[hidden email]> wrote:
>>>
>>>
>>>> On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> Excuse me if this has been asked already or it's documented somewhere
>>> and I
>>>> missed it, but what's the criteria to update the code in /src, /stacksrc,
>>>> /spursrc and other similar folders in the repository?
>>>>
>>>> -Laura Perez Cerrato
>>>
>>> Good question. Traditionally, Eliot generates these sources periodically
>>> and
>>> commits them to the repository (and in recent years I have done that chore
>>> for
>>> the old trunk interpreter VM). At this point, unless Eliot advises
>>> otherwise,
>>> I would suggest that no one other than Eliot should commit to the /src
>>> trees.
>> AFAIK the pharo-vm projects generates the sources from the VMMaker package
>> during a build.
>> Wouldn't it be better if the OpenSmalltalk vm does the same? Then no one
>> needs to generate source manually anymore and we don't have millions of
>> lines [2] of generated code in the repository.
>>
>> Fabio
>>
>> [1] https://github.com/pharo-project/pharo-vm
>> [2] see Eliot's stats at
>> https://github.com/OpenSmalltalk/vm/graphs/contributors
>>
>> --
>
> Yes of course it should be automated. My suggestion is that at this point,
> or at least until Eliot says otherwise, Eliot should be the only person who
> commits to the /src directories. One step at a time :-)
>
> And of course we can do whatever is necessary in the various branches.

Absolutely not.  For the reasons stated.  Those millions of lines are there for good reason:

Reproducibility
Comprehensibility

Optimising the repository for brevity is completely mistaken. That source needs to exist in a versioned form.  Now please can we stop discussing this madness?  I didn't simply choose to set things up as they are on a whim.

>
> Dave
>
Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

Eliot Miranda-2
In reply to this post by fniephaus
 
Hi Fabio, Hi All,

On Jun 22, 2016, at 12:51 AM, Fabio Niephaus <[hidden email]> wrote:

AFAIK the pharo-vm projects generates the sources from the VMMaker package during a build.
Wouldn't it be better if the OpenSmalltalk vm does the same? Then no one needs to generate source manually anymore and we don't have millions of lines [2] of generated code in the repository.

In short, we think nothing of burning pets cycles every time we make any change in the repository, kicking off many many builds.  We should think nothing of these few tens of megabytes of disc that keep a necessary record of the production VM source.  Reducing disc size here is a) optimising the wrong thing and b) optimising something quite cheap.  Both are cardinal errors.

Fabio


--

On Wed, Jun 22, 2016 at 4:02 AM David T. Lewis <[hidden email]> wrote:

On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
>
> Hi everyone,
>
> Excuse me if this has been asked already or it's documented somewhere and I
> missed it, but what's the criteria to update the code in /src, /stacksrc,
> /spursrc and other similar folders in the repository?
>
> -Laura Perez Cerrato

Good question. Traditionally, Eliot generates these sources periodically and
commits them to the repository (and in recent years I have done that chore for
the old trunk interpreter VM). At this point, unless Eliot advises otherwise,
I would suggest that no one other than Eliot should commit to the /src trees.

Dave

Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

Laura Perez Cerrato
In reply to this post by Eliot Miranda-2
 
Hi Eliot, Hi everyone,

Thanks for the comprehensive answer. May I suggest this, along with your response to Fabio's question, be added to CONTRIBUTING.md? I wouldn't mind doing so myself if you all agree (since I'm not a contributor to the GitHub repository I'd submit it through a pull request though). 

Cheers!

-Laura Perez Cerrato

On 22 June 2016 at 10:27, Eliot Miranda <[hidden email]> wrote:
 
Hi Laura,

On Jun 21, 2016, at 7:11 AM, Laura Perez Cerrato <[hidden email]> wrote:

Hi everyone,

Excuse me if this has been asked already or it's documented somewhere and I missed it, but what's the criteria to update the code in /src, /stacksrc, /spursrc and other similar folders in the repository?

I should document this.  There are three main criteria:

- does the source need to be updated to fix a bug or make available new functionality urgently needed?  Since up until now my VM builds have been manually kicked off on the three platforms it took a while and I'd only do new builds when necessary

- does the source look sane?  One should always look at the differences to the generated source if any part of Slang's translator has been modified to check for bugs.

- is the source generated from versioned input?  The generated C source is marked with the Monticello package version, including UUID, of the packages from which it was created (eg a package providing a plugin) and that created it (i.e. the VMMaker package).  These marks will contain asterisks for modified Monticello packages.  Clearly one should never check in source containing asterisks.  The one exception I've allowed myself is MiscPrimitivesPlugin which gets generated from base image code, and I have a few choice base image modifications (such as a much higher character limit on the Transcript, or different code highlighting) that make it really painful to keep MiscPrimitivesPlugin clean.

I'm happy for others to generate and check in VM source provided:
- you carefully check the generated source before checking it in to verify that the delta is as expected and no new bugs have been introduced
- you generate as little change as possible.
-- There is a script that reverts generated plugins that have changed only in their Monticello markings that allows us to not checkin inessential revisions to plugins and see when I fact they did change
-- wrapping the generation code with [...] valueAnswering: false (or whatever the selector is) avoids regenerating VM headers (cointerp.h cogit.h cogmethod.h interp.h) when these haven't been modified

In short, only do this if you know what you're doing.  But got people to know what they're doing the process has to be documented along with the rationale.  I guess this email and my reply to Fabio's question can serve as input to that documentation.

-Laura Perez Cerrato

_,,,^..^,,,_ (phone)


Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

Ben Coman
In reply to this post by David T. Lewis
 
On Wed, Jun 22, 2016 at 10:02 AM, David T. Lewis <[hidden email]> wrote:

>
> On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
>>
>> Hi everyone,
>>
>> Excuse me if this has been asked already or it's documented somewhere and I
>> missed it, but what's the criteria to update the code in /src, /stacksrc,
>> /spursrc and other similar folders in the repository?
>>
>> -Laura Perez Cerrato
>
> Good question. Traditionally, Eliot generates these sources periodically and
> commits them to the repository (and in recent years I have done that chore for
> the old trunk interpreter VM). At this point, unless Eliot advises otherwise,
> I would suggest that no one other than Eliot should commit to the /src trees.
>
> Dave
>

But...  its all one repository.  Everyone who works on the vm in their
personal space, doing a dozen generate/compile/commit cycles along the
way, and then finally submits a worthwhile change via a pull request
that is accepted, gets *all* their intermediate generated src added to
the repository.

> On 22 Jun 2016, at 12:52, [hidden email] wrote:
> One could thing of doing a git submodule with them but it is more trouble than it is worth.

Actually this was exactly what I wanted to suggest.  That perhaps the
srcs live in a submodule/subtree,  that only gets pushed updated by
Travis upon a successful build.  That is, Travis would generate the
srcs, then commit them, and build from that commit, and only push that
commit back to the official repo/branch upon successful build.  Upon
failure the commit might be pushed to a "failure" branch for manual
review before later that branch is thrown away.

Having the generated sources in a submodule/subtree means when Joe
Public makes a contribution, its easier for only his changes to
non-generated code get submitted. Otherwise he need to manually back
out any generated sources for the last commit being submitted as a PR,
but that doesn't remove any previous commits.  Or \every time he does
a personal commit he has to back out the changes just for those
directories (??).


> I want to be able to clone the VM and compile it right away.

The compile script could pull the srcs subtree before compiling.  No
real difference in elapsed time than if the main repo was pulled.

cheers -ben
Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

Eliot Miranda-2
In reply to this post by Laura Perez Cerrato
 


On Wed, Jun 22, 2016 at 6:52 AM, Laura Perez Cerrato <[hidden email]> wrote:
 
Hi Eliot, Hi everyone,

Thanks for the comprehensive answer. May I suggest this, along with your response to Fabio's question, be added to CONTRIBUTING.md? I wouldn't mind doing so myself if you all agree (since I'm not a contributor to the GitHub repository I'd submit it through a pull request though). 

Yes, please!


Cheers!

-Laura Perez Cerrato

On 22 June 2016 at 10:27, Eliot Miranda <[hidden email]> wrote:
 
Hi Laura,

On Jun 21, 2016, at 7:11 AM, Laura Perez Cerrato <[hidden email]> wrote:

Hi everyone,

Excuse me if this has been asked already or it's documented somewhere and I missed it, but what's the criteria to update the code in /src, /stacksrc, /spursrc and other similar folders in the repository?

I should document this.  There are three main criteria:

- does the source need to be updated to fix a bug or make available new functionality urgently needed?  Since up until now my VM builds have been manually kicked off on the three platforms it took a while and I'd only do new builds when necessary

- does the source look sane?  One should always look at the differences to the generated source if any part of Slang's translator has been modified to check for bugs.

- is the source generated from versioned input?  The generated C source is marked with the Monticello package version, including UUID, of the packages from which it was created (eg a package providing a plugin) and that created it (i.e. the VMMaker package).  These marks will contain asterisks for modified Monticello packages.  Clearly one should never check in source containing asterisks.  The one exception I've allowed myself is MiscPrimitivesPlugin which gets generated from base image code, and I have a few choice base image modifications (such as a much higher character limit on the Transcript, or different code highlighting) that make it really painful to keep MiscPrimitivesPlugin clean.

I'm happy for others to generate and check in VM source provided:
- you carefully check the generated source before checking it in to verify that the delta is as expected and no new bugs have been introduced
- you generate as little change as possible.
-- There is a script that reverts generated plugins that have changed only in their Monticello markings that allows us to not checkin inessential revisions to plugins and see when I fact they did change
-- wrapping the generation code with [...] valueAnswering: false (or whatever the selector is) avoids regenerating VM headers (cointerp.h cogit.h cogmethod.h interp.h) when these haven't been modified

In short, only do this if you know what you're doing.  But got people to know what they're doing the process has to be documented along with the rationale.  I guess this email and my reply to Fabio's question can serve as input to that documentation.

-Laura Perez Cerrato

_,,,^..^,,,_ (phone)






--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

David T. Lewis
In reply to this post by Ben Coman
 
On Thu, Jun 23, 2016 at 01:37:12AM +0800, Ben Coman wrote:

>  
> On Wed, Jun 22, 2016 at 10:02 AM, David T. Lewis <[hidden email]> wrote:
> >
> > On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
> >>
> >> Hi everyone,
> >>
> >> Excuse me if this has been asked already or it's documented somewhere and I
> >> missed it, but what's the criteria to update the code in /src, /stacksrc,
> >> /spursrc and other similar folders in the repository?
> >>
> >> -Laura Perez Cerrato
> >
> > Good question. Traditionally, Eliot generates these sources periodically and
> > commits them to the repository (and in recent years I have done that chore for
> > the old trunk interpreter VM). At this point, unless Eliot advises otherwise,
> > I would suggest that no one other than Eliot should commit to the /src trees.
> >
> > Dave
> >
>
> But...  its all one repository.  Everyone who works on the vm in their
> personal space, doing a dozen generate/compile/commit cycles along the
> way, and then finally submits a worthwhile change via a pull request
> that is accepted, gets *all* their intermediate generated src added to
> the repository.

Don't do that. See Eliot's explanation. In addition to what Eliot said, I
will add that ./src generated from different images (e.g. two different
versions of Squeak, or Squeak versus Pharo) can and will produce different
results, even for the same version of VMMaker. It is important to check
carefully for changes like this, because they can have huge impacts on
performance and functionality.

In a perfect world, we would treat the generated ./src as intermediate
artifacts to be discarded just as we would discard the object files after
compiling a C program. But the world is not perfect, and we need to be
able to preserve and distribute the generated ./src files for various
reasons, and to do so in a controlled manner.

I very strongly advise following Eliot's guidelines.

Dave

Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

Bert Freudenberg
In reply to this post by Ben Coman
 
On Wed, Jun 22, 2016 at 7:37 PM, Ben Coman <[hidden email]> wrote:

But...  its all one repository.  Everyone who works on the vm in their
personal space, doing a dozen generate/compile/commit cycles along the
way, and then finally submits a worthwhile change via a pull request
that is accepted, gets *all* their intermediate generated src added to
the repository.

Why would you commit your generated sources at all? And if you do, why would you push them to the public repo?

If you chose to commit the generated sources, then you should do so in a private branch that never gets published. You would cherry-pick non-src commits from that branch to a feature branch that you then publish for public consumption. That way the generated sources do not end up in the public repo.

This is one advantage of git over svn: "commit" does not mean "publish", you still have to "push". You can have as many branches on your own machine as you like.

- Bert -
Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

EstebanLM
In reply to this post by EstebanLM
 
I want to clarify how we do it in Pharo, and why we think is a good approach. 

Please, note that I’m not asking to change anything, I’m just doing story telling :)

1) In Pharo, since a couple of years, VMMaker *and* plugin sources coexist along with platform sources, in same repository. 

pharovm/
image/
platforms/
mc/ —> This is the important!
VMMaker.package/
Freetype-Plugin.package/
...

(we take the effort of keeping the plugin packages here, as a mirror from his original repositories, but this is more or less automated and also, we think it worths the effort)

2) In Pharo, CI build always rebuild all source code: 
- it builds an image, using a BaselineOf from Metacello
- then it builds ALL sources
- then builds the VM

3) CI also executes all Pharo tests (we have an acceptable number of tests)

Of course, doing all this in development time is losing time, but in CI is very important and produces significant advantages: 

1) we know exactly which version (including Monticello versions) correspond with each build (and if we want to reproduce it, is a simple checkout)
2) we know immediately if a recent change breaks VMMaker image build (some times has happened)
3) we know immediately if a change broke compilation as a side effect (this happens more of less often, specially when working with the C translator)
4) we know immediately if a change broke something image side 

This process will change now, as we adopt OS/vm, but in our plans we will keep parts of this model.  We will include OS/vm as submodule along with the packages: 

pharovm/
@osvm/ —> this is a submodule
mc/
- we will first generate image using our method 
- then we will follow OS/vm method, using this image (a Pharo image… for us is very important to be able to build the VM from Pharo)
- we will generate sources using it
- and then we’ll generate the VM using the OS/vm method. 

at least, this is what we are aiming for. 

Notice that this way everybody has what they want: Eliot and others can continue working how they like, and we have what we want: a complete reconstruction process who verifies everything. 

Once put in place, this system will also benefit non-pharo users, because there will be the feedback available too.

cheers, 
Esteban 

On 22 Jun 2016, at 13:15, Esteban Lorenzano <[hidden email]> wrote:


On 22 Jun 2016, at 12:52, [hidden email] wrote:

One could thing of doing a git submodule with them but it is more trouble than it is worth.

I want to be able to clone the VM and compile it right away.

Phil

On Wed, Jun 22, 2016 at 10:11 AM, Norbert Hartl <[hidden email]> wrote:
 

Am 22.06.2016 um 09:51 schrieb Fabio Niephaus <[hidden email]>:

AFAIK the pharo-vm projects generates the sources from the VMMaker package during a build.
Wouldn't it be better if the OpenSmalltalk vm does the same? Then no one needs to generate source manually anymore and we don't have millions of lines [2] of generated code in the repository.

How would you release a version of the vm? This is only possible of you archive the static artefacts.

I do not understand this question, so maybe I’m answering anything :)
in Pharo, VMMaker package sources coexist along with C sources (thanks to filetree for now, but we are working on enhance that). This way, each release of VM in github contains *exactly* all the sources needed to reconstruct it… 

Esteban 


Norbert

Fabio


--

On Wed, Jun 22, 2016 at 4:02 AM David T. Lewis <[hidden email]> wrote:

On Tue, Jun 21, 2016 at 11:11:28AM -0300, Laura Perez Cerrato wrote:
>
> Hi everyone,
>
> Excuse me if this has been asked already or it's documented somewhere and I
> missed it, but what's the criteria to update the code in /src, /stacksrc,
> /spursrc and other similar folders in the repository?
>
> -Laura Perez Cerrato

Good question. Traditionally, Eliot generates these sources periodically and
commits them to the repository (and in recent years I have done that chore for
the old trunk interpreter VM). At this point, unless Eliot advises otherwise,
I would suggest that no one other than Eliot should commit to the /src trees.

Dave






Reply | Threaded
Open this post in threaded view
|

Re: The code in the src folders in the repository

Ben Coman
In reply to this post by Bert Freudenberg
 


On Thu, Jun 23, 2016 at 5:33 PM, Bert Freudenberg <[hidden email]> wrote:
 
On Wed, Jun 22, 2016 at 7:37 PM, Ben Coman <[hidden email]> wrote:

But...  its all one repository.  Everyone who works on the vm in their
personal space, doing a dozen generate/compile/commit cycles along the
way, and then finally submits a worthwhile change via a pull request
that is accepted, gets *all* their intermediate generated src added to
the repository.

Why would you commit your generated sources at all? And if you do, why would you push them to the public repo?

I  *don't* want to commit my generated sources.  But** there they are, tracked files that have been modified so I need to weed out the changes I do want to commit.

**Likely I'm overthinking this; its an imagined scenario.  I should wait to comment further until I actually hit this scenario so at least I know which foot I'm putting in my mouth. :)

cheers -ben
 

If you chose to commit the generated sources, then you should do so in a private branch that never gets published. You would cherry-pick non-src commits from that branch to a feature branch that you then publish for public consumption. That way the generated sources do not end up in the public repo.

This is one advantage of git over svn: "commit" does not mean "publish", you still have to "push". You can have as many branches on your own machine as you like.

- Bert -