[ANN] DeltaStreams release 0.1

This is the first alpha release of delta streams, a change management
and distribution tool.

Delta streams aims to be a compelling replacement to change sets
and the update stream. Delta streams are analogous to the
update stream, and deltas are analogous to change sets. A more
complete feature comparison is at
http://wiki.squeak.org/squeak/DeltasVsChangeSets

Please note that this is an alpha release, and not all features we
have been bragging about are implemented. For the rundown, see
http://wiki.squeak.org/squeak/6013

========== Installation ==========

DeltaStreams 0.1 is available for Squeak 3.9, Squeak 3.10, and Croquet.
It is available as a .sar installer for each platform. It is not
available through universes at this time.

- Get the .sar from http://wiki.squeak.org/squeak/6013
- drop it into the image, or load it with the file list
- If you get a popup asking whether to automatically define the pool
  dictionary EditorSubjectPool, answer no. This only seems to happen on
  Croquet
- When you get a popup asking which tool set you would prefer, answer
  #StandardToolSet, not #PlusToolSet

---------------------------------------------------------------------

For more information about this release, especially what features are
not yet implemented, see:
http://wiki.squeak.org/squeak/6013

The delta streams user guide is on the Swiki:
http://wiki.squeak.org/squeak/DeltaStreamsUserGuide

The homepage for DeltaStreams is
http://wiki.squeak.org/squeak/DeltaStreams

--
Matthew Fulmer -- http://mtfulmer.wordpress.com/
Help improve Squeak Documentation: http://wiki.squeak.org/squeak/808

I didn't see anywhere if one can add a comment to a delta.  Is that
planned or present?  I think the ability to add a comment to why a
delta was created is important.

Thanks,
Jason

On Nov 7, 2007 4:31 PM, Matthew Fulmer <[hidden email]> wrote:

On Thu, Nov 08, 2007 at 06:31:30PM +0100, Jason Johnson wrote:
> I didn't see anywhere if one can add a comment to a delta.  Is that
> planned or present?  I think the ability to add a comment to why a
> delta was created is important.

Deltas can have arbitrary stuff tacked on to them (they have a
properties dictionary), and we may have implemented a
getter/setter for comments or descriptions (don't remember), but
it is not exposed from the UI, even if it is implemented.

If you want to help, we'll gladly accept. I'd say this as a
trivial feature and could be scheduled for the next release
(which will likely be 0.1.1, with few new features).
DeltaStreams development is stalled at present, due to both
Goran and I being quite busy.

--
Matthew Fulmer -- http://mtfulmer.wordpress.com/
Help improve Squeak Documentation: http://wiki.squeak.org/squeak/808

Hi Matthew -

Can you say a little more about the anticipated workflow with
DeltaStreams? I love what I see so far (it mostly looks like a version
of change sets for the 21st century ;-) but it is unclear to me how this
integrates into the larger picture, with Monticello, images, versions etc.

Any insights are greatly appreciated.

Cheers,
   - Andreas

Matthew Fulmer wrote:

Hi!

(have a few minutes so I am throwing in my thoughts from the hip)

> Hi Matthew -
>
> Can you say a little more about the anticipated workflow with
> DeltaStreams?

Well, a little :). First of all - just like with changesets a Delta has
(at least not currently) no historic information in it.

The basic idea to amend this is to instead make it very fine granular and
specific (for example, modifying a method and creating it is not the same
change) combined with each change containing both the state AFTER and
BEFORE the change.

So... the most trivial way of using Deltas would be much like a ChangeSet
but with the added benefit of being able to have a validation step where
the Delta can check the image and tell if it will apply "clean" (the
BEFORE state of all changes are exactly as expected), "dirty" (the BEFORE
state is not as expected but the end result will still be "correct" in
some sense) or  "partial" (not all changes can be meaningfully applied).

In practice this would mean that a FIX from Croquet, recorded as a Delta,
when imported into say 3.10 could easily tell if the FIX is reasonable to
apply. And when applied and perhaps deemed to not work it can still be
fully reverted. :)

Still, this is not what you may be asking about - I presume you see all
the above - I presume you wonder about the "streams" part. I honestly
don't know yet but I would like to experiment with multiple streams
connected to projects, images, packages and even people - and combine this
with a publish/subscribe system so that you can easily discover streams -
subscribe to them etc.

If we try something like that I guess we immediately will hit upon the
next big NEED and that would be out-of-order apply - or in other words
"cherry picking". And I am hoping we can create

> I love what I see so far (it mostly looks like a version
> of change sets for the 21st century ;-)

Yes, Deltas themselves are intentionally exactly that I would say. This is
redundant I guess but I want them to be ROCK SOLID with full test coverage
(we have quite lots of tests) and covering all details (like for example
deal properly with class renaming). Also, since the default applier uses
SystemEditor it should work quite solidly (atomic etc) - but there are
still bugs in SE to iron out.

> but it is unclear to me how this
> integrates into the larger picture, with Monticello, images, versions etc.

It is definitely unclear to us too - but I intend to take the trivial
approach here: "one thing at a time". I am dead focused on making Deltas
work as intended - then we can tackle streams of them.

But if there are people interested in coming up with something around this
aspect - feel free to join up! For example, Giovanni Corriga is starting
some work on SM3 with Atom feeds etc, and without knowing exactly what he
is doing I would say it definitely relates.

Stephane also asked me about this btw, and I have no clear answers. I just
hope this new tool in our toolbox will be useful. ;)

> Any insights are greatly appreciated.
>
> Cheers,
>    - Andreas

Not sure if this post gave any, but hey. :) I want to keep things open so
that people are more interested in joining up.

And one important aspect here is cross-fork-compatibility - we are totally
committed to support all major Squeak "forks" and while on the subject -
what is your take on ToolBuilder in that respect? :)

regards, Göran

Il giorno gio, 08/11/2007 alle 21.35 +0100, Göran Krampe ha scritto:
At the heart of Squeak Map3 will be Atom Publishing Protocol (APP)
server. APP is a REST-style protocol that uses the Atom syndication
protocol as the interexchange format between the server and the clients.
Since the list of packages, the list of releases of a package, the
details of a package, etc. will be exposed as Atom feeds, it will be
easy for a client to subscribe to different feeds and to mix packages
from different maps, etc.
A generic APP server could be easily extracted from the Squeak Map 3
server and adapted for use with Deltas. This would allow the stream
remixing that Goran would like to have.

        Ciao,

                Giovanni

Göran Krampe wrote:
> Still, this is not what you may be asking about - I presume you see all
> the above - I presume you wonder about the "streams" part.

Well, no, not exactly. What I'm wondering about is how DS address the
issues that I'd like to see addressed when looking at both Monticello
and change sets. Because what I'm hoping DS will do is to integrate the
best of change sets (flexibility, size, speed, cross-package changes)
with the best of Monticello (versioning, compare, diff, merge). The
reason why I want these together is that although I loathe Monticello
for all its shortcomings I pick it in every new project over using
change sets. Because by the end of the day, being able to say what is
version X of package Y is more important than all the great properties
of change sets (at least that is true for my work).

So what I'm wondering is, how can DS be used such that they overcome the
problems with Monticello and how would one apply them to such uses. The
main area that I see relates to "versions of packages". By far the
biggest shortcoming of change sets is that they have no idea of versions
and packages and because of that the use of change sets is currently
mutually exclusive with the use of Monticello (don't get me started
about this Frankenstream that is currently used to deal with MCZs as
updates).

I think a version really is the entire set of transformations (Deltas)
applied to bring the package to a particular state. In other words, a
possible version of a package maintained with Deltas could just be a zip
file with all the deltas included. However, in order to to be able to
stream these changes we'd still need that information somehow inside the
delta.

The best way I can think of dealing with it is if a delta could include
information of the "from/to" versions of packages it applies to. In
other words, it would include enough information to say: This delta
started from package version Foo-bar.1, added method Foo>>bar and the
result was package version Foo-bar.2.

This information allows two very important things: First, subscribers to
the stream know which version their working version is based on (it
doesn't guarantee that it is the exact version but similar to Monticello
one could simply mark the package as dirty and require a full diff to
find out whether it has actually changed) which solves the update stream
problems we have today. With the "before state" that deltas have for
individual changes it seems like a perfect match as one could easily
reason about whether an incoming change conflicts with a local
modification etc.

But secondly it should be possible to have multiple versions described
that way for has cross-package changes. The reason why this is important
is that obviously, when you need an atomic update of package Foo and Bar
together they both need to be at the "right" version (e.g., the version
that was described in the delta) and having that information allows you
to update Foo, and when you hit the update in Foo that applies to both
Foo and Bar, you can update Bar up to the point required by the delta
and then apply it.

So it seems that having "some" version information in the deltas
themselves is pretty important. It allows a number of operations that
are otherwise hard or impossible to achieve and would integrate well
with a "Monticello point of view" of maintaining sets of packages while
at the same time having all the nice properties of change sets.

Thoughts?

Cheers,
   - Andreas

On Thu, Nov 08, 2007 at 10:14:27AM -0800, Andreas Raab wrote:
> Hi Matthew -
>
> Can you say a little more about the anticipated workflow with DeltaStreams?

I wrote a long reply to your question, but I figured a more
concise answer might be more welcome. I copied the long answer to
http://wiki.squeak.org/squeak/DeltaStreamsUseCases

Here are three use cases I have in mind.

I would like to reliably determine how various images are
different and cherry-pick changes between them. The most obvious
use for this is porting changes from squeak.org to croquet. This
may or may not actually involve deltas and delta streams;
SystemEditor and the UI I developed for it may be sufficient.

I would also like to publish an update stream that can be safely
applied on various images. This is what deltas and delta streams
are really about. For instance, I would like to add some comments
to core classes, and be able for everyone to see them with minimal
hassle. This would require some automatic conflict handling to
make it work without much trouble.

I also want the update stream to be very searchable and
debuggable. The browser will automatically show you where in the
update stream each method was changed. I would like that and more
in the future, especially being able to revert changes.

> I love what I see so far (it mostly looks like a version of change sets for
> the 21st century ;-) but it is unclear to me how this integrates into the
> larger picture, with Monticello, images, versions etc.

In 3.9 and 3.10, the update stream lost many of it's capabilities
to both find the source of a bug and undo the change once found.
One idea I have to alleviate this difficulty is to log the saving
of a MC snapshot in the delta as a note:

"This image just changed the version of package Kernel from
version 4 to version 5"

When a receiving image encountered this, it would optionally check
that the SystemEditor's package Kernel now matches the definition
in Kernel-5.mcz. Upon commit, the version of the Working copy
would be updated. This would allow the delta stream to be
searchable and revert-able, while using MC to make unloadable
entities.

A disadvantage of this is that a project would have to decide
whether it wants to use Monticello packages directly, or use
packages in subordination to an update stream. Currently, only the
release team would want to use MC this way, and most projects
would continue using MC, rather than DeltaStreams, as their
primary distribution mechanism.

Thus, using a delta stream would allow maintaining a cluster of
inter-dependent packages, while using MC directly would be better
for projects with well-factored or not very many packages.

> Any insights are greatly appreciated.

Thanks for asking questions. I hope DeltaStreams can be useful
either on their own or as a step to something even better.

--
Matthew Fulmer -- http://mtfulmer.wordpress.com/
Help improve Squeak Documentation: http://wiki.squeak.org/squeak/808

Hi Matthew -

Matthew Fulmer wrote:
> I wrote a long reply to your question, but I figured a more
> concise answer might be more welcome. I copied the long answer to
> http://wiki.squeak.org/squeak/DeltaStreamsUseCases

Thanks, this is pretty interesting.

> Here are three use cases I have in mind.
>
> I would like to reliably determine how various images are
> different and cherry-pick changes between them. The most obvious
> use for this is porting changes from squeak.org to croquet. This
> may or may not actually involve deltas and delta streams;
> SystemEditor and the UI I developed for it may be sufficient.

Yes, and I will point out that the main difficulty today is granularity.
If you've ever tried to look through the 100 or so versions of the
Morphic package in 3.9 you'll know what I mean ;-) Being able to do this
*quickly* would actually be a major advantage for larger packages.

> I would also like to publish an update stream that can be safely
> applied on various images. This is what deltas and delta streams
> are really about. For instance, I would like to add some comments
> to core classes, and be able for everyone to see them with minimal
> hassle. This would require some automatic conflict handling to
> make it work without much trouble.

This goal may be a bit of a red herring, depending on how you think
about it. The way in which it could make sense is if there were a
package delta stream so that (for example) Seaside could have a common
delta stream for all supported Squeak image versions. Outside of that I
have my doubts about the usefulness of this approach - I cannot even
begin to imagine applying changes (even from "harmless" streams) across
the board automatically without some manual review and intervention.

> I also want the update stream to be very searchable and
> debuggable. The browser will automatically show you where in the
> update stream each method was changed. I would like that and more
> in the future, especially being able to revert changes.

Yup. This is all useful information we used to have with change sets.
And it'll be great to have it back.

Right. This is very similar to what I described in my other message. I
think attributing deltas to have that information is a key property to
ensure integration with other kinds of package management.

> When a receiving image encountered this, it would optionally check
> that the SystemEditor's package Kernel now matches the definition
> in Kernel-5.mcz. Upon commit, the version of the Working copy
> would be updated. This would allow the delta stream to be
> searchable and revert-able, while using MC to make unloadable
> entities.

Exactly.

> A disadvantage of this is that a project would have to decide
> whether it wants to use Monticello packages directly, or use
> packages in subordination to an update stream. Currently, only the
> release team would want to use MC this way, and most projects
> would continue using MC, rather than DeltaStreams, as their
> primary distribution mechanism.

Or, make MC represent sequences of deltas.

> Thanks for asking questions. I hope DeltaStreams can be useful
> either on their own or as a step to something even better.

I think it's important to work out a few use cases that we understand
enough to drive development to something useful. The interim step of
producing both, delta as well as Monticello package may be a good way to
address the first round of problems.

Cheers,
   - Andreas

On Thu, Nov 08, 2007 at 10:18:21PM -0800, Andreas Raab wrote:
> Matthew Fulmer wrote:
>
> This goal may be a bit of a red herring, depending on how you think about
> it. The way in which it could make sense is if there were a package delta
> stream so that (for example) Seaside could have a common delta stream for
> all supported Squeak image versions. Outside of that I have my doubts about
> the usefulness of this approach - I cannot even begin to imagine applying
> changes (even from "harmless" streams) across the board automatically
> without some manual review and intervention.

The application I have in mind that would benefit from an
approach like this is documentation. Having comments in the
image is a very good thing, but getting them in is more trouble
than most people are willing to invest, especially in core
packages, which sometimes need it the most.

How I would like to solve this dilemma is to make a delta stream
that only logged comment changes (both class comments and method
comments). This stream would auto-commit regularly (every day or
so) to an unofficial world-writable stream managed by the doc
team (me). The doc team would review deltas published to this
stream and either reject them or publish them to an official
stream, which most images could subscribe to.

I plan on, at least initially, using the update stream format
(lots of chunk files + updates.list) as the delta stream format.
This format allows any change to be marked by which image
versions it is compatible with. Any subsequent format should
have this property as well. (If Giovanni Corriga has his way,
delta streams will one day be atom feeds)

Delta stream publishers should definitely be careful about what
they publish, and make sure it works as advertised. If they
don't, social mechanism's will kick in and most people will just
unsubscribe. This includes image publishers, like the release
team and Damien, who can decide which streams are subscribed to
in their distributions.

--
Matthew Fulmer -- http://mtfulmer.wordpress.com/
Help improve Squeak Documentation: http://wiki.squeak.org/squeak/808

On 9 nov. 07, at 05:00, Andreas Raab wrote:

>
> So what I'm wondering is, how can DS be used such that they  
> overcome the problems with Monticello and how would one apply them  
> to such uses. The main area that I see relates to "versions of  
> packages". By far the biggest shortcoming of change sets is that  
> they have no idea of versions and packages and because of that the  
> use of change sets is currently mutually exclusive with the use of  
> Monticello (don't get me started about this Frankenstream that is  
> currently used to deal with MCZs as updates).

may be I do not understand your english so if I'm off discard my email.
Now if you refer to our attempt to use MC to manage 3.9. We tried and  
this was worth doing it. Now we have a version of MC with 1000 times  
faster
load. But it is not in a sharable state. Having a much faster MC  
would made it a completely other story.
I think that having packages based on MC or not is really important  
even for managing images. Now may be not for shipping updates. :)
what would be nice is to be able to use deltastreams as cs for update  
and mark the result as versioned package on the client.
Or even that we work with packages, then the update computes deltas  
and send them and that the packages get versioned on the updaters.

Stef

On 9 nov. 07, at 07:18, Andreas Raab wrote:

I should be one of few crazy that did that. So unfortunately I know  
what you mean :)
and I have the impression that using deltastreams for update that  
version packages on demand.
We could have a change that says please version package X.
I hope to be able to work on faster MC loading but time time.... and  
it should be really stable and robust so we will see.

yes or having a change element that is version package X after that  
bunch of change elements.
I do not really know if MC representing sequences of deltas will save  
the following problem (but I suspect): we got really problems with  
3.9 when a change was cutting two packages (we mentioned that in our  
postmortem analysis), our problem was that it was not easy to know  
(without doing a diff between the two
config files) which changes have to be removed together (for example  
when we had to roll back and complex changes touching multiple  
packages). This
was an information that we really missed.

> This use case does not actually involve deltas or delta streams, only
> a UI and SystemEditor. However, I mention it because SystemEditor and
> DeltaStreams are evolving together, and my work on the delta streams
> browser has also resulted in a generic system editor browser, which
> allows viewing exactly what changes a particular edit will make once
> committed.
>
> It is not yet supported, as we have no code yet to apply a change set
> or a package to a SystemEditor.
Hi Matthew, MC1.5 has code to apply a package to a system editor.

Keith