[squeak-dev] Let's push it

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[squeak-dev] Let's push it

Andreas.Raab
Folks -

I truly love this discussion and I think it has generated enough insight
that we can try pushing a few things. As a first step, I would like to
propose a slightly different development model for Squeak, because I
think that is one of the fundamental bits that is wrong. Here we go:

1) Agree on MC as the core development tool/model. This is painful and
consequently the decision has to be made consciously. However, from my
perspective there is really no suitable alternative [1].

2) Decide on the canonical repositories. I would suggest that
http://source.squeakfoundation.org/trunk is a fine starting point [2].

3) Decide on who gets commit rights. I would suggest that the board
decides that (equally for eventual access removals). The "masters" list
from Squeak-people could act as a starting point.

4) Implement an update model for MC. I would suggest to adopt a similar
model to what we've been using for Tweak in the past: Use MC config maps
to mark points in the update process where order is important and
otherwise update the packages in a well-defined order[3].

These four steps are intended to achieve the following: I can propose to
the board that Eliot should be given access to the kernel repository so
that he can commit any fixes to the compiler or other changes required
for Cog directly without the need for an intermediate. I can propose to
the board that Nicolas gets commit rights to the kernel repositories so
that he can commit any fixes for numeric stuff he has or comes up with
in the future. If the board agrees, they can do that, period. If they
screw up or abuse their powers the board can retract commit rights.

The main difference to what we've done in the past is simply that we
have an inclusive list of committers instead of a select few and that we
do understand the image as truly being "work in progress" inbetween the
releases. In other words, be more active, change things faster, make it
possible to report, devise, submit, and deploy a change in a matter of
minutes.

For people who don't have commit rights, there is both Mantis or
commit-by-proxy which reinstates the SqC model of having a champion for
your change. I.e., to commit something you can ask someone else to
commit your change but that person implicitly takes responsibility for
your change and will be the person who gets blamed for it and is
expected to take care of it (and they may loose their commit rights if
the change they commit is complete crap). To me, that is a useful
intermediate state because the proxy acts as a mentor to teach the
original author about how to provide changes, make sure it's okay etc.

For Mantis, Keith just added a bunch of wonderful scripts that allow us
to pull changes from it and build images (and hopefully soon run tests
like the integration tests that I've been working on earlier) which
means that we can build images with those changes included, test them,
and if they pass those tests (and possibly other review), commit them to
trunk. Again, I think having a champion who does the commit would be a
good model. Obviously there is still a manual element involved here but
the idea is that the regular committers would be able to go straight to
the repository without requiring further approval.

For users of Squeak there would be an update process which (although
much slower than the update stream of the days of yore) would allow
people to keep track of the ongoing changes and (for forks) merge those
changes as desired along the way using Monticello.

What do people think about this as a straw-man? Obviously, I want to
enable people to contribute in the most straightforward way possible.
Allow development to be more fluid, make mistakes faster, fix them
faster. It does not address the issue of controversial / strategic
changes but I think we're mostly loosing out on the most basic level and
if we want to fix the processes we must make simple things simple and
complex things possible, not the other way around.

Cheers,
   - Andreas

More detail on some of the issues above:

[1] Why I think update streams don't cut it: When you do distributed
development it is necessary to know if there are conflicts with what you
are working on. If you only get to spend two hours a week on development
you need to be able to quickly find out if anything changed in an area
that you've been working on since last week. If you want to provide
backwards compatibility you need to be able to reflect about what has
changed, etc. Monticello is pretty bad but it's still the best we got.

[2] http://source.squeakfoundation.org/trunk: The idea is to use the
version specific directories (Squeak39, Squeak310, Squeak311 etc) as
branches for released versions. The "current" version is always on
trunk, the specific ones provide updates after a version was released.

[3] MC based updates: The basic idea is that the config maps define load
order and intermediate stages where required. There is an
update-<version>.mcm in every repository that is consulted by the update
loader process. The update process loads the update.mcms in order and
uses the order in the last loaded update.mcm to update all the remaining
packages. In  other words, as long as the packages load with the last
defined order, we change nothing. If Bob finds a problem we issue a
(series of) configurations that get past that update bump. Yes, there is
a certain element of risk to it but if Bob runs test builds of the image
for us we can simply update to the last version which Bob was able to
build for himself (and Bob can notify us that there's a problem).


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Let's push it

Igor Stasenko
2009/6/30 Andreas Raab <[hidden email]>:

> Folks -
>
> I truly love this discussion and I think it has generated enough insight
> that we can try pushing a few things. As a first step, I would like to
> propose a slightly different development model for Squeak, because I think
> that is one of the fundamental bits that is wrong. Here we go:
>
> 1) Agree on MC as the core development tool/model. This is painful and
> consequently the decision has to be made consciously. However, from my
> perspective there is really no suitable alternative [1].
>
> 2) Decide on the canonical repositories. I would suggest that
> http://source.squeakfoundation.org/trunk is a fine starting point [2].
>
> 3) Decide on who gets commit rights. I would suggest that the board decides
> that (equally for eventual access removals). The "masters" list from
> Squeak-people could act as a starting point.
>
> 4) Implement an update model for MC. I would suggest to adopt a similar
> model to what we've been using for Tweak in the past: Use MC config maps to
> mark points in the update process where order is important and otherwise
> update the packages in a well-defined order[3].
>

A side note: at this point , i would consult with Keith on his vision
and how his creation (Bob) and rest of infrastructure he built  could
help us in avoiding doing boring stuff. (but ignore this comment, i
see you mention about it down below).

> These four steps are intended to achieve the following: I can propose to the
> board that Eliot should be given access to the kernel repository so that he
> can commit any fixes to the compiler or other changes required for Cog
> directly without the need for an intermediate. I can propose to the board
> that Nicolas gets commit rights to the kernel repositories so that he can
> commit any fixes for numeric stuff he has or comes up with in the future. If
> the board agrees, they can do that, period. If they screw up or abuse their
> powers the board can retract commit rights.
>

I just thought about the similar thing last night, and even
considering to write a new topic with 'Squeak privatization' title,
when idea will settle down.
+1 on that. Lets privatize different areas of Squeak and let the
people become landlords of those areas. If something went wrong - we
always have a board to revoke the rights and burn heretics :)
But lets not get down to a team model - if owner (responsible person)
don't have time to react on others contributions of his areas of
interest, or sleeps 23 hours a day - this should not become the
bottleneck of contribution process.
A more better will be just to step down the position and declare it free.
Or even do this in a more autocratic fashion: initially the landlord
having 10 points - but for each contribution not revised by him or
which get past his attention, he gets -1. When counter goes to zero -
he's get fired :)

> The main difference to what we've done in the past is simply that we have an
> inclusive list of committers instead of a select few and that we do
> understand the image as truly being "work in progress" inbetween the
> releases. In other words, be more active, change things faster, make it
> possible to report, devise, submit, and deploy a change in a matter of
> minutes.
>
> For people who don't have commit rights, there is both Mantis or
> commit-by-proxy which reinstates the SqC model of having a champion for your
> change. I.e., to commit something you can ask someone else to commit your
> change but that person implicitly takes responsibility for your change and
> will be the person who gets blamed for it and is expected to take care of it
> (and they may loose their commit rights if the change they commit is
> complete crap). To me, that is a useful intermediate state because the proxy
> acts as a mentor to teach the original author about how to provide changes,
> make sure it's okay etc.
>
> For Mantis, Keith just added a bunch of wonderful scripts that allow us to
> pull changes from it and build images (and hopefully soon run tests like the
> integration tests that I've been working on earlier) which means that we can
> build images with those changes included, test them, and if they pass those
> tests (and possibly other review), commit them to trunk. Again, I think
> having a champion who does the commit would be a good model. Obviously there
> is still a manual element involved here but the idea is that the regular
> committers would be able to go straight to the repository without requiring
> further approval.
>
> For users of Squeak there would be an update process which (although much
> slower than the update stream of the days of yore) would allow people to
> keep track of the ongoing changes and (for forks) merge those changes as
> desired along the way using Monticello.
>
> What do people think about this as a straw-man? Obviously, I want to enable
> people to contribute in the most straightforward way possible. Allow
> development to be more fluid, make mistakes faster, fix them faster. It does
> not address the issue of controversial / strategic changes but I think we're
> mostly loosing out on the most basic level and if we want to fix the
> processes we must make simple things simple and complex things possible, not
> the other way around.
>

I think this is not a surprise, that model what is you proposing
having a lot of synergy with 3.11 team vision on how community past
3.11 release could contribute to squeak.


> Cheers,
>  - Andreas
>
> More detail on some of the issues above:
>
> [1] Why I think update streams don't cut it: When you do distributed
> development it is necessary to know if there are conflicts with what you are
> working on. If you only get to spend two hours a week on development you
> need to be able to quickly find out if anything changed in an area that
> you've been working on since last week. If you want to provide backwards
> compatibility you need to be able to reflect about what has changed, etc.
> Monticello is pretty bad but it's still the best we got.
>
> [2] http://source.squeakfoundation.org/trunk: The idea is to use the version
> specific directories (Squeak39, Squeak310, Squeak311 etc) as branches for
> released versions. The "current" version is always on trunk, the specific
> ones provide updates after a version was released.
>
> [3] MC based updates: The basic idea is that the config maps define load
> order and intermediate stages where required. There is an
> update-<version>.mcm in every repository that is consulted by the update
> loader process. The update process loads the update.mcms in order and uses
> the order in the last loaded update.mcm to update all the remaining
> packages. In  other words, as long as the packages load with the last
> defined order, we change nothing. If Bob finds a problem we issue a (series
> of) configurations that get past that update bump. Yes, there is a certain
> element of risk to it but if Bob runs test builds of the image for us we can
> simply update to the last version which Bob was able to build for himself
> (and Bob can notify us that there's a problem).
>
>
>



--
Best regards,
Igor Stasenko AKA sig.

Reply | Threaded
Open this post in threaded view
|

[squeak-dev] Re: Let's push it

Klaus D. Witzel
On Tue, 30 Jun 2009 06:29:05 +0200, Igor Stasenko wrote:

...
> But lets not get down to a team model - if owner (responsible person)
> don't have time to react on others contributions of his areas of
> interest, or sleeps 23 hours a day - this should not become the
> bottleneck of contribution process.

In a "normal" project, I would rule out "master" people who have _not_  
committed a bug fix within the last 12 months (even if that would rule out  
me, of course ;)

If this community restarts, it can equally well restart from where every  
newbee starts, to show how it works, to demonstrate committment, to  
everyone who cares.

But I doubt that my proposal will be implemented.

/Klaus

--
"If at first, the idea is not absurd, then there is no hope for it".  
Albert Einstein


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Let's push it

Ian Trudel-2
In reply to this post by Andreas.Raab
2009/6/29 Andreas Raab <[hidden email]>:
> 1) Agree on MC as the core development tool/model. This is painful and
> consequently the decision has to be made consciously. However, from my
> perspective there is really no suitable alternative [1].

Monticello is an acceptable choice to me. MC sounds like something we
can use right here, right now. And as long as we can easily
contribute, we can bring improvements to the development process and
tools in our way.

> 2) Decide on the canonical repositories. I would suggest that
> http://source.squeakfoundation.org/trunk is a fine starting point [2].

Yes, I support this idea.

> 3) Decide on who gets commit rights. I would suggest that the board decides
> that (equally for eventual access removals). The "masters" list from
> Squeak-people could act as a starting point.

Yes, I support on the board decides who is approved and revoked.
Could you point out the masters list mentioned here?

> 4) Implement an update model for MC. I would suggest to adopt a similar
> model to what we've been using for Tweak in the past: Use MC config maps to
> mark points in the update process where order is important and otherwise
> update the packages in a well-defined order[3].

What would be the incidence of such an update model, with well-defined
order, on the eventual maintenance a component manager (someone who
has commit access) has to carry on? Isn't just plain revision number?

> The main difference to what we've done in the past is simply that we have an
> inclusive list of committers instead of a select few and that we do
> understand the image as truly being "work in progress" inbetween the
> releases. In other words, be more active, change things faster, make it
> possible to report, devise, submit, and deploy a change in a matter of
> minutes.

Woot! Sounds exciting. =)

> What do people think about this as a straw-man? Obviously, I want to enable
> people to contribute in the most straightforward way possible. Allow
> development to be more fluid, make mistakes faster, fix them faster. It does
> not address the issue of controversial / strategic changes but I think we're
> mostly loosing out on the most basic level and if we want to fix the
> processes we must make simple things simple and complex things possible, not
> the other way around.

On the account of controversial changes, I think that Squeak Oversight
Board should clearly state what changes will be acceptable without
thorough review and acceptation process by the board/community. For
example, community could commit quickly tests (any), fixes (only on
what is tested), minor improvements (which kind?) with tests, etc. A
clear statement should avoid misunderstandings, in general at least.

Andreas, I would like some clarification on how someone part of the
community, with no special rights, can contribute using your process,
method and tool. I'd be glad if you could provide a use case/scenario.

Your solution does seem leaving the little man outside the loop, sent
back to mantis or something along this line. That's hardly something
interesting. We as a community need instant feedback on our work and
be well integrated in the process. Do not set us apart! It is
important to renew the excitement of contributing to Squeak within the
community and not just for few selected committers.

Considering continuous integration would really help on that matter, I
think it's possible to push updates for review from any body to the
committers. I would really prefer a community repository rather than
yet another elitist process; but I am always fine with the idea that
the selected elites will review, accept, modify and/or reject
community committed content.

Or did I get it all wrong?

Regards,
Ian

--
http://mecenia.blogspot.com/

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Let's push it

keith1y
In reply to this post by Andreas.Raab
For someone like Elliot, the current proposal is as follows.

Lets say his aim is to develop an image with closures in it. We will
call this image 3.10-closures

He would take a fixed starting point... preferably 3.x-build This image
is 3.x with the latest version of all tools (MC with atomic loading etc)
and a couple of additional packages namely Packages and Tasks.

Packages defines the packages which may be loaded that are managed
external to the kernel, and the relationships between them. This could
include the compiler and other kernel packages, if they are maintained
as a common package that all forks may adopt.

Tasks defines actions that may be applied to the kernel image to acheive
particular aims, such as loading all tests, loading fixes, and preparing
the image for release.

Elliot would then develop a "Task" which may combine the loading of
updated packages, and change sets, and fixes to the image. The task is
essentially a script, though if defined as a task it can define
pre-requisites.
Essentially all I am saying is that I just dont think that an mcm will
cut it, I think that each task needs to be crafted to a known fixed point.

While Elliot is developing 3.10-closures. Bob is also able to make test
builds of 3.10-closures+mantis fixes+tests

So the next release becomes a selection of packages and tasks.

Keith

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Let's push it

Edgar J. De Cleene
In reply to this post by Andreas.Raab



On 6/30/09 12:58 AM, "Andreas Raab" <[hidden email]> wrote:

> 1) Agree on MC as the core development tool/model. This is painful and
> consequently the decision has to be made consciously. However, from my
> perspective there is really no suitable alternative [1].

Seems 3.9 an 3.10 don't teach MC as is now is not suited for a .image
develop.
And which MC ?

Maybe MC2 could solve all MC don't , but it's used for this kind of job ?

Edgar




Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Let's push it

David T. Lewis
In reply to this post by Andreas.Raab
On Mon, Jun 29, 2009 at 08:58:18PM -0700, Andreas Raab wrote:

>
> For people who don't have commit rights, there is both Mantis or
> commit-by-proxy which reinstates the SqC model of having a champion for
> your change. I.e., to commit something you can ask someone else to
> commit your change but that person implicitly takes responsibility for
> your change and will be the person who gets blamed for it and is
> expected to take care of it (and they may loose their commit rights if
> the change they commit is complete crap). To me, that is a useful
> intermediate state because the proxy acts as a mentor to teach the
> original author about how to provide changes, make sure it's okay etc.

This sounds like a good approach. It provides a way to get things done,
and a process for ensuring that things get done right. As someone who
has occasionally submitted both good patches and complete crap, I value
both ;) I also think that it is much easier to put crap in than to take
it out, so controlling this well is important.

> For users of Squeak there would be an update process which (although
> much slower than the update stream of the days of yore) would allow
> people to keep track of the ongoing changes and (for forks) merge those
> changes as desired along the way using Monticello.

As a consumer of the update process, I like the update stream (based
on change sets) a *lot*. As a developer, Monticello is indispensable.
Is there some simple way we could get both, such that each new
commit to a Monticello repository can produce a change set relative
to its predecessor, with the comment text from the Monticello version
forming the preamble of the change set? Perhaps this is something
that already exists (or maybe it's an itch I should go scratch?).

> What do people think about this as a straw-man?

I like it.

Dave


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Let's push it

Juan Vuletich-4
In reply to this post by Klaus D. Witzel
Klaus D. Witzel wrote:

> ...
> In a "normal" project, I would rule out "master" people who have _not_
> committed a bug fix within the last 12 months (even if that would rule
> out me, of course ;)
>
> If this community restarts, it can equally well restart from where
> every newbee starts, to show how it works, to demonstrate committment,
> to everyone who cares.
>
> But I doubt that my proposal will be implemented.
>
> /Klaus

Tried to get to the masters list. www.squeak.org/Community points to
people.squeakfoundation.org But it is no longer there... Can anybody advise?

Anyway, last time I read it, it included a lot of people who hasn't been
in this community for over 5 years, and people who never contributed a
single piece of code. On the other hand, many hard working and valuable
people are at the journeyer level.

We need a better list of valuable contributors.

Cheers,
Juan Vuletich

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Let's push it

Edgar J. De Cleene
In reply to this post by David T. Lewis



On 6/30/09 8:53 AM, "David T. Lewis" <[hidden email]> wrote:

> Is there some simple way we could get both, such that each new
> commit to a Monticello repository can produce a change set relative
> to its predecessor, with the comment text from the Monticello version
> forming the preamble of the change set? Perhaps this is something
> that already exists (or maybe it's an itch I should go scratch?).


Well , this was 3.10 and I have many headaches trying to get the work done.
And learn Monticello , as we know now, could not be used for a very complex
task like building a .image.
Cs can and we don't have yet IMHO some better.
Eras ago I volunteer to test both DeltaStreams and MC2 , but nobody wish
risk and we take the Blue Pill (remember?)

But maybe some found the silver bullet and I was the only disbelieving
that...


Edgar




Reply | Threaded
Open this post in threaded view
|

DeltaStreams status (Re: [squeak-dev] Let's push it)

Göran Krampe
Hi!

Edgar J. De Cleene wrote:
> Eras ago I volunteer to test both DeltaStreams and MC2 , but nobody wish
> risk and we take the Blue Pill (remember?)

Ah! DeltaStreams! Just waited for it to get mentioned. :)

I actually sat last night trying to get it slightly into shape for Igor
to join in. Quick recap and status:

DeltaStreams is an idea (and code) that I came up with a while back when
trying to figure out "new ways" to live in a forked world. The idea
consisted of two parts - Deltas and Streams of them. :)

A Delta is simply a "ChangeSet DeLuxe". Main differences off the top of
my head:

- Is in fact an *ordered* sequence of changes. (A ChangeSet is a *Set* -
no ordering)
- Each change in a Delta is much more fine granular and has *all*
information captured about the change.
- Each change can produce its "anti" change.
- A Delta is a *standalone* object. It does not refer to anything in the
image, each change contains all its information in its own object graph.

This standalone property means that you first *load* a Delta into the
image and then *apply* it if you want to. Thus you can load one from
disk and look at it *without* applying it! You could harvest the
Internet for Deltas, suck hundreds of them into an image, and then
inspect them.

The anti bit above also means that a Delta can be reverted, it just
applies all its anti changes in reverse order! And yes, this shit
actually works. :)

The DeltaStreams code uses SystemEditor from Colin Putney - the new
mechanism to *atomically* apply changes to the image. In fact, our work
on DS contributed several fixes to SE because DS was first to use it
seriously. I thin MC1.6 maybe uses SE too?

Deltas are "recording" just like ChangeSets are, but since they record
all changes in order - they do double duty as a changelog too.

Matthew Fulmer started out implementing a clever hack so that Deltas
could be saved as .cs files and actually "work" as a ChangeSet if filed
in as one. Now that I am diving into the code again I have decoupled
that file format (but still there as an extra package) and instead will
implement a format using Tirade. I have blogged about Tirade in 3
articles so you can read about it there. It is a readable format that
looks like Smalltalk code, but it is very fast to parse and secure.


Status:

The domain code is perhaps 90% covered. There are some small things to
take care of and since I am now hooking in Tirade I am refactoring some
things a bit. A lot of tests are green - but not all. I really want to
move to a fully green state. Experts are more than welcome to help out!

An example of a "problem": If I have a method change that moves a method
to a new method category - and then I *revert* that change, should the
new method category (if now empty) be removed? tricksy.

The Tirade hookup is ongoing but not hard work, it should be doable in a
couple of nights.

Then I wish to implement a DualDeltaSorter :) - simply a "clone" of dual
changesorter that works with Deltas. If anyone would like to start that
work, or implement some other cool thing like:

- ChangeSet -> Delta converter. Filing in a ChangeSet should simply
create a Delta instead of a ChangeSet.
- MC diff -> Delta converter. Given two MCs we can produce a Delta.

Just holler!

Usecase:

Let's say the Pharo people fixes a bug in Pharo. By the look of it the
bug should exist in Squeak.org too. Let's say we produce a Delta either
from the Pharo changeset or from the MC snapshot that contained the fix.

The Delta is filed out from Pharo in Tirade format. It is easily
readable in a text editor just like a ChangeSet is. You fire up
Squeak.org and open up the DualDeltaSorter and *load* the Delta into the
image. At this point *no code* has been touched in the image, we just
deserialized the Tirade file and created the DSDelta object.

We can now look at the actual changes it contains, perhaps 3 new
methods, 4 modified methods, some ivars added - all over the image.

A Delta is like a "semantic patch". Since it knows more about itself it
can reason about its own applicability. For example, it separates
"modified method" and "added method". And it contains both the "before"
and "after" state of each change. For example a modified method contains
both code before and the new code!

We thus ask the Delta if it can be "cleanly applied". Cleanly means that
the DSDelta checks if the state of the image concerning the changes is
the same as it was in the Pharo image. For example, regarding the
modified methods - do they look like they did in Pharo *before* they
were modified? If yes, then we are good to go. Can we add the ivars? Are
all expected classes in the image? Did any of the new methods perhaps
already exist?

Let's say it claims to apply cleanly (there are levels of "cleanliness"
here, for example a method may already be in the "new state", which is
probably fine but not perfectly clean).

We apply it (done by SystemEditor atomically). Then we run tests or
whatever, if it seems to have broken something we could decide to just
back it out with "revert". We could also set the Delta to record and
then try to fix it.

Conclusion:

DeltaStreams are not here yet - but pretty darn close to be able to
replace ChangeSets at least! It just needs a push. This means I agree
with Andreas to use MC currently, but I also would like us to dig in our
heels and get DS ready. It should bring us:

- Speed.
- Lots of flexibility.

regards, Göran


Reply | Threaded
Open this post in threaded view
|

Re: DeltaStreams status (Re: [squeak-dev] Let's push it)

keith1y

> The DeltaStreams code uses SystemEditor from Colin Putney - the new
> mechanism to *atomically* apply changes to the image. In fact, our
> work on DS contributed several fixes to SE because DS was first to use
> it seriously. I thin MC1.6 maybe uses SE too?
Hi Göran,

MC1.6 is pretty much primed to use SE, and its 3x faster than MC1.5
Traits are not yet supported AFAIK, matthew was working on it.

If you manage to recruit some helpers, perhaps you could have a further
look SystemEditor.
Recently a bug was introduced which results in obsolete classes.

Keith

Reply | Threaded
Open this post in threaded view
|

Re: DeltaStreams status (Re: [squeak-dev] Let's push it)

Nicolas Cellier
In reply to this post by Göran Krampe
2009/6/30 Göran Krampe <[hidden email]>:

> Hi!
>
> Edgar J. De Cleene wrote:
>>
>> Eras ago I volunteer to test both DeltaStreams and MC2 , but nobody wish
>> risk and we take the Blue Pill (remember?)
>
> Ah! DeltaStreams! Just waited for it to get mentioned. :)
>
> I actually sat last night trying to get it slightly into shape for Igor to
> join in. Quick recap and status:
>
> DeltaStreams is an idea (and code) that I came up with a while back when
> trying to figure out "new ways" to live in a forked world. The idea
> consisted of two parts - Deltas and Streams of them. :)
>
> A Delta is simply a "ChangeSet DeLuxe". Main differences off the top of my
> head:
>
> - Is in fact an *ordered* sequence of changes. (A ChangeSet is a *Set* - no
> ordering)
> - Each change in a Delta is much more fine granular and has *all*
> information captured about the change.
> - Each change can produce its "anti" change.
> - A Delta is a *standalone* object. It does not refer to anything in the
> image, each change contains all its information in its own object graph.
>
> This standalone property means that you first *load* a Delta into the image
> and then *apply* it if you want to. Thus you can load one from disk and look
> at it *without* applying it! You could harvest the Internet for Deltas, suck
> hundreds of them into an image, and then inspect them.
>
> The anti bit above also means that a Delta can be reverted, it just applies
> all its anti changes in reverse order! And yes, this shit actually works. :)
>

Goran, this is great work I'd like to support.
But I can only believe that it works in most cases...
More exactly, I suspect it works for code, but not on lively instances:
If I remove an inst var, all instances of the class and subclasses
will loose some information.
Do you store that information in the delta so that reversion restore the state?
Otherwise, I know of some anti-refactoring that will kill my image nicely.

If I remove a class, how will orphaned instances be restored?

Do DeltaStream address something like refactoring ScaledDecimal from
something like:
Number subclass: #ScaledDecimal instanceVariables: 'fraction precision'.
to:
Fraction subclass: #ScaledDecimal instanceVariables: 'precision'.

This shit is real (MIT refactoring applied in Pharo) and requires some
other shit like:
Create class ScaledDecimalTemp
Mutate allInstances of ScaledDecimal to ScaledDecimalTemp
remove ScaledDecimal
rename ScaledDecimalTemp to ScaledDecimal

I'm very interested how would you handle such mutation and
anti-mutation in DeltaStream?
It's doable with user crafted pre/post scripts without system
validation... Is that the solution?

Nicolas

> The DeltaStreams code uses SystemEditor from Colin Putney - the new
> mechanism to *atomically* apply changes to the image. In fact, our work on
> DS contributed several fixes to SE because DS was first to use it seriously.
> I thin MC1.6 maybe uses SE too?
>
> Deltas are "recording" just like ChangeSets are, but since they record all
> changes in order - they do double duty as a changelog too.
>
> Matthew Fulmer started out implementing a clever hack so that Deltas could
> be saved as .cs files and actually "work" as a ChangeSet if filed in as one.
> Now that I am diving into the code again I have decoupled that file format
> (but still there as an extra package) and instead will implement a format
> using Tirade. I have blogged about Tirade in 3 articles so you can read
> about it there. It is a readable format that looks like Smalltalk code, but
> it is very fast to parse and secure.
>
>
> Status:
>
> The domain code is perhaps 90% covered. There are some small things to take
> care of and since I am now hooking in Tirade I am refactoring some things a
> bit. A lot of tests are green - but not all. I really want to move to a
> fully green state. Experts are more than welcome to help out!
>
> An example of a "problem": If I have a method change that moves a method to
> a new method category - and then I *revert* that change, should the new
> method category (if now empty) be removed? tricksy.
>
> The Tirade hookup is ongoing but not hard work, it should be doable in a
> couple of nights.
>
> Then I wish to implement a DualDeltaSorter :) - simply a "clone" of dual
> changesorter that works with Deltas. If anyone would like to start that
> work, or implement some other cool thing like:
>
> - ChangeSet -> Delta converter. Filing in a ChangeSet should simply create a
> Delta instead of a ChangeSet.
> - MC diff -> Delta converter. Given two MCs we can produce a Delta.
>
> Just holler!
>
> Usecase:
>
> Let's say the Pharo people fixes a bug in Pharo. By the look of it the bug
> should exist in Squeak.org too. Let's say we produce a Delta either from the
> Pharo changeset or from the MC snapshot that contained the fix.
>
> The Delta is filed out from Pharo in Tirade format. It is easily readable in
> a text editor just like a ChangeSet is. You fire up Squeak.org and open up
> the DualDeltaSorter and *load* the Delta into the image. At this point *no
> code* has been touched in the image, we just deserialized the Tirade file
> and created the DSDelta object.
>
> We can now look at the actual changes it contains, perhaps 3 new methods, 4
> modified methods, some ivars added - all over the image.
>
> A Delta is like a "semantic patch". Since it knows more about itself it can
> reason about its own applicability. For example, it separates "modified
> method" and "added method". And it contains both the "before" and "after"
> state of each change. For example a modified method contains both code
> before and the new code!
>
> We thus ask the Delta if it can be "cleanly applied". Cleanly means that the
> DSDelta checks if the state of the image concerning the changes is the same
> as it was in the Pharo image. For example, regarding the modified methods -
> do they look like they did in Pharo *before* they were modified? If yes,
> then we are good to go. Can we add the ivars? Are all expected classes in
> the image? Did any of the new methods perhaps already exist?
>
> Let's say it claims to apply cleanly (there are levels of "cleanliness"
> here, for example a method may already be in the "new state", which is
> probably fine but not perfectly clean).
>
> We apply it (done by SystemEditor atomically). Then we run tests or
> whatever, if it seems to have broken something we could decide to just back
> it out with "revert". We could also set the Delta to record and then try to
> fix it.
>
> Conclusion:
>
> DeltaStreams are not here yet - but pretty darn close to be able to replace
> ChangeSets at least! It just needs a push. This means I agree with Andreas
> to use MC currently, but I also would like us to dig in our heels and get DS
> ready. It should bring us:
>
> - Speed.
> - Lots of flexibility.
>
> regards, Göran
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: DeltaStreams status (Re: [squeak-dev] Let's push it)

Göran Krampe
Hi!

Nicolas Cellier wrote:
> 2009/6/30 Göran Krampe <[hidden email]>:
> Goran, this is great work I'd like to support.

:) Thanks!

> But I can only believe that it works in most cases...
> More exactly, I suspect it works for code, but not on lively instances:
> If I remove an inst var, all instances of the class and subclasses
> will loose some information.
> Do you store that information in the delta so that reversion restore the state?

Nope, we only deal with source. If you want image level kind of
reversion then I can only recommend... image snapshots :). Or some kind
of Gemstone-ish memory.

> Otherwise, I know of some anti-refactoring that will kill my image nicely.
>
> If I remove a class, how will orphaned instances be restored?

Again, we only deal with code. Like ChangeSets do.

> Do DeltaStream address something like refactoring ScaledDecimal from
> something like:
> Number subclass: #ScaledDecimal instanceVariables: 'fraction precision'.
> to:
> Fraction subclass: #ScaledDecimal instanceVariables: 'precision'.
>
> This shit is real (MIT refactoring applied in Pharo) and requires some
> other shit like:
> Create class ScaledDecimalTemp
> Mutate allInstances of ScaledDecimal to ScaledDecimalTemp
> remove ScaledDecimal
> rename ScaledDecimalTemp to ScaledDecimal
>
> I'm very interested how would you handle such mutation and
> anti-mutation in DeltaStream?

Ouch! :) Again, no, we have not looked at instance migration issues.

> It's doable with user crafted pre/post scripts without system
> validation... Is that the solution?

You mean in Deltas? Well, a Delta is meant to be able to do everything a
Changeset can - so we need to support scripts (it is just another change
class called "DSDoit"). Since the changes are ordered the "doits" can
appear not only before or after but in the middle too :).

So... well, I presume "surgery" gets a tad simpler to do with Deltas
since they simply record all the steps you perform. Thus, if you perform
the steps you describe above - in sequence - then you have a DSDelta
that will work. Slightly simpler than editing the Delta in order to add
pre/post scripts etc.

This also means that we typically want to "prune" doits from Deltas,
since they are normally not really wanted.

regards, Göran


Reply | Threaded
Open this post in threaded view
|

Re: DeltaStreams status (Re: [squeak-dev] Let's push it)

Nicolas Cellier
2009/6/30 Göran Krampe <[hidden email]>:

> Hi!
>
> Nicolas Cellier wrote:
>>
>> 2009/6/30 Göran Krampe <[hidden email]>:
>> Goran, this is great work I'd like to support.
>
> :) Thanks!
>
>> But I can only believe that it works in most cases...
>> More exactly, I suspect it works for code, but not on lively instances:
>> If I remove an inst var, all instances of the class and subclasses
>> will loose some information.
>> Do you store that information in the delta so that reversion restore the
>> state?
>
> Nope, we only deal with source. If you want image level kind of reversion
> then I can only recommend... image snapshots :). Or some kind of
> Gemstone-ish memory.
>
>> Otherwise, I know of some anti-refactoring that will kill my image nicely.
>>
>> If I remove a class, how will orphaned instances be restored?
>
> Again, we only deal with code. Like ChangeSets do.
>
>> Do DeltaStream address something like refactoring ScaledDecimal from
>> something like:
>> Number subclass: #ScaledDecimal instanceVariables: 'fraction precision'.
>> to:
>> Fraction subclass: #ScaledDecimal instanceVariables: 'precision'.
>>
>> This shit is real (MIT refactoring applied in Pharo) and requires some
>> other shit like:
>> Create class ScaledDecimalTemp
>> Mutate allInstances of ScaledDecimal to ScaledDecimalTemp
>> remove ScaledDecimal
>> rename ScaledDecimalTemp to ScaledDecimal
>>
>> I'm very interested how would you handle such mutation and
>> anti-mutation in DeltaStream?
>
> Ouch! :) Again, no, we have not looked at instance migration issues.
>
>> It's doable with user crafted pre/post scripts without system
>> validation... Is that the solution?
>
> You mean in Deltas? Well, a Delta is meant to be able to do everything a
> Changeset can - so we need to support scripts (it is just another change
> class called "DSDoit"). Since the changes are ordered the "doits" can appear
> not only before or after but in the middle too :).
>
> So... well, I presume "surgery" gets a tad simpler to do with Deltas since
> they simply record all the steps you perform. Thus, if you perform the steps
> you describe above - in sequence - then you have a DSDelta that will work.
> Slightly simpler than editing the Delta in order to add pre/post scripts
> etc.
>
> This also means that we typically want to "prune" doits from Deltas, since
> they are normally not really wanted.
>
> regards, Göran
>
>

Thanks, that's what I thought.
I just wanted to point out that in some cases, special doIts are necessary.
In theses cases, anti-delta generally won't work, and should better
not be attempted.
OK for me if tools cover 90%, atomic loading is just another example
not covering 100%.
Just think of providing enough hooks for the remaining 10% in the
overall process.
This kind of refactoring will likely happen to anyone touching
Process, Graphics, user interaction like Sensor, Compiler and other
Kernel beasts... That's not that rare. Ask Igor, Andreas, Eliot,
Michael, Marcus, etc...

Nicolas

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Let's push it

Ken Causey-3
In reply to this post by Andreas.Raab
The SqueakPeople Masters and Journeyers as of 12/1/2008 when the site
was closed:

Masters

 Kazuhiro Abe
 Germán Arduino
 Jecel Assumpção Jr
 Bruce Badger
 Avi Bryant
 Damien Cassou
 Ward Cunningham
 Edgar De Cleene
 Cees de Groot
 Marcus Denker
 Stéphane Ducasse
 Tansel Ersavas
 Brad Fowlow
 Bert Freudenberg
 Diego Gomez Deck
 Dale Henrichs
 Jon Hylands
 Daniel Ingalls
 Ralph Johnson
 Ted Kaehler
 Bryce Kampjes
 Ned Konz
 Göran Krampe
 Pavel Krivanek
 Craig Latta
 Adrian Lienhard
 John M McIntosh
 Janko Mivsek
 Chris Muller
 Luciano Notarfrancesco
 Yoshiki Ohshima
 Stephen Pair
 Ian Piumarta
 Colin Putney
 Andreas Raab
 Lukas Renggli
 Kim Rose
 Tim Rowledge
 Michael Rueger
 Lex Spoon
 Masato Sumi
 Andrew Tween
 Masashi Umezawa
 Daniel Vainsencher
 Scott Wallace
 Doug Way
 Stephan B. Wessels
 Roel Wuyts
 Takashi Yamamiya

Journeyers

 Cedrick Beler
 Alexandre Bergel
 Torsten Bergmann
 Andrew Black
 Todd Blanchard
 Noury Bouraqadi
 Julien Bourdon
 Brian Brown
 Luca Bruno
 Chris Burkert
 Blaine Buxton
 Frank Caggiano
 Gwenael Casaccio
 Andrew Catton
 Ken Causey
 nicolas cellier
 Gary Chambers
 Yanni Chiu
 Giovanni Corriga
 Eddie Cottongim
 Ramiro Diaz Trepat
 Pierre-Andre Dreyfuss
 Jason Dufair
 Dave Faught
 Hilaire Fernandes
 Julian Fitzell
 James Foster
 Rita Freudenberg
 Brad Fuller
 Matthew Fulmer
 Thushar G R
 Markus Gaelli
 Boris Gaertner
 Joshua Gargus
 Giovanni Giorgi
 Sean Glazier
 Matthew Hamrick
 William Harford
 Norbert Hartl
 Michael Haupt
 Tetsuya HAYASHI
 Keith Hodges
 Ragnar Hojland Espinosa
 Simon Holland
 Dwight Hughes
 Eiichiro Ito
 SeungBum Kim
 Simon Kirk
 Tom Koenig
 Jan B. Krejci
 Martin Kuball
 tsuyoshi kuwatani
 Alexander Lazarevic
 Ramon Leon
 Dave Lewis
 Alan Lovejoy
 Adrian Manrique
 Norberto Manzanos
 Mark McCahill
 Martin McClure
 Webb McDonald
 Simon Michael
 Kazuki Minamitani
 Masaaki Mitsuhashi
 Toshio Miyasaka
 Jens Moenig
 Peter Moore
 German Morales
 Hans-Martin Mosner
 Philippe Mougin
 Javier Musa
 Oscar Nierstrasz
 Bruce O'Neel
 Emilio Oca
 Tomohiro Oda
 DongHee Park
 Jerome Peace
 Nicolas Petton
 John Pierce
 Brent Pinkney
 Alain Plantec
 Damien Pollet
 Reza Razavi
 José L. Redrejo Rodríguez
 Trygve Reenskaug
 Aaron Reichow
 Brian Rice
 Gerardo Richarte
 Romain Robbes
 Stéphane Rollandin
 Dan Rozenfarb
 Stephan Rudlof
 Randal Schwartz
 Mark Schwenk
 David Shaffer
 Frank Shearar
 Rik Smoody
 Adam Spitz
 Igor Stasenko
 Serge Stinckwich
 Mathieu Suen
 Gen Suzuki
 Steven Swerling
 Hisashi Tagawa
 Makoto TAKAHASHI
 Toshiyuki Takeda
 Ron Teitelbaum
 Hernán Tylim
 Michael van der Gulik
 Adriaan van Os
 Brent Vukmer
 Juan Vuletich
 Rob Withers
 Klaus D. Witzel
 Koji Yokokawa
 Yakov Zaytsev




signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: DeltaStreams status (Re: [squeak-dev] Let's push it)

Göran Krampe
In reply to this post by Nicolas Cellier
Hi!

Nicolas Cellier wrote:

> 2009/6/30 Göran Krampe <[hidden email]>:
>>> It's doable with user crafted pre/post scripts without system
>>> validation... Is that the solution?
>> You mean in Deltas? Well, a Delta is meant to be able to do everything a
>> Changeset can - so we need to support scripts (it is just another change
>> class called "DSDoit"). Since the changes are ordered the "doits" can appear
>> not only before or after but in the middle too :).
>>
>> So... well, I presume "surgery" gets a tad simpler to do with Deltas since
>> they simply record all the steps you perform. Thus, if you perform the steps
>> you describe above - in sequence - then you have a DSDelta that will work.
>> Slightly simpler than editing the Delta in order to add pre/post scripts
>> etc.
>>
>> This also means that we typically want to "prune" doits from Deltas, since
>> they are normally not really wanted.
>>
>> regards, Göran
>
> Thanks, that's what I thought.
> I just wanted to point out that in some cases, special doIts are necessary.
> In theses cases, anti-delta generally won't work, and should better
> not be attempted.

I gully agree. :)

> OK for me if tools cover 90%, atomic loading is just another example
> not covering 100%.
> Just think of providing enough hooks for the remaining 10% in the
> overall process.
> This kind of refactoring will likely happen to anyone touching
> Process, Graphics, user interaction like Sensor, Compiler and other
> Kernel beasts... That's not that rare. Ask Igor, Andreas, Eliot,
> Michael, Marcus, etc...

I agree. This is actually one of the first inspirations of Deltas - I
wanted a tool that could do relatively nice merging/conflict detection
etc *without* advanced history. We all like the concreteness of
ChangeSets - but we could still have a modern implementation. :)

In turn this was also inspired by git - not git itself, but by the
philosophy of "let the tool be less smart and more predictable", it will
still cover 80-90% and will be enough. And you get fewer surprises.

regards, Göran


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Let's push it

Ian Trudel-2
In reply to this post by Edgar J. De Cleene
2009/6/30 Edgar J. De Cleene <[hidden email]>:
> Seems 3.9 an 3.10 don't teach MC as is now is not suited for a .image
> develop.
> And which MC ?
>
> Maybe MC2 could solve all MC don't , but it's used for this kind of job ?
>
> Edgar

Good question, Edgar. By the way, let's not forget that we could use
Andreas' ConflictFinder as well. Every bit of tool we can have for a
better continuous integration.

Ian.

--
http://mecenia.blogspot.com/

Reply | Threaded
Open this post in threaded view
|

Re: DeltaStreams status (Re: [squeak-dev] Let's push it)

Stéphane Rollandin
In reply to this post by Göran Krampe
> I gully agree. :)

Beware of guano :)

Stef


Reply | Threaded
Open this post in threaded view
|

Re: DeltaStreams status (Re: [squeak-dev] Let's push it)

Colin Putney
In reply to this post by Göran Krampe

On 30-Jun-09, at 6:58 AM, Göran Krampe wrote:

> DeltaStreams are not here yet - but pretty darn close to be able to  
> replace ChangeSets at least! It just needs a push. This means I  
> agree with Andreas to use MC currently, but I also would like us to  
> dig in our heels and get DS ready.

MC2 is in this position as well. It's available and useful, but it  
hasn't had the kind of long-term and widespread usage that MC1 has. As  
the interaction and collaboration patterns that MC2 favors become  
clear, and the code base matures, it will become suitable for a large-
scale project like Squeak. But it's not there yet.

Colin
Reply | Threaded
Open this post in threaded view
|

Re: DeltaStreams status (Re: [squeak-dev] Let's push it)

Edgar J. De Cleene
In reply to this post by Göran Krampe



On 6/30/09 10:58 AM, "Göran Krampe" <[hidden email]> wrote:

> Conclusion:
>
> DeltaStreams are not here yet - but pretty darn close to be able to
> replace ChangeSets at least! It just needs a push. This means I agree
> with Andreas to use MC currently, but I also would like us to dig in our
> heels and get DS ready. It should bring us:
>
> - Speed.
> - Lots of flexibility.
>
> regards, Göran

I confident in you so I wish someone use DeltaStreams instead MC as I know
MC "as is now" don't work for a release.

If needed, I can do the dirty work.

Edgar




12