Atomic updates & SystemEditor

Hello guys,

A SystemEditor is the only tool , currently existing, which allows us
to perform an atomic updates.
However, its having own shortages.
A SystemEditor mimics the Behavior/ClassDescription/Class protocol,
so any change you going to make to system is recorded , but not
applied immediately.

This is good, but at same time, this is the main source of shortage,
because once you may want to change the Behavior protocol(s), or
extend it (see Traits), you gonna to change a SystemEditor as well to
mimic new behavior and new state, otherwise it wont work as expected.
Needless to say that, keeping two different parts of system
synchronized doubling the amount of work, as well as makes it brittle
and highly error prone.

So, i asking the question, is it possible to implement a framework,
which would be able to:
 a) capture changes to any object(s) of interest.
 b) serialize/deserialize captured changes (to file out/in)
 c) apply changes atomically

One of the ways, i'm thinking of, is to use a 'capture-by-intent'
i.e., whenever you think that a state change should be recorded, you
using a special protocol for recording that change. And framework,
which lies somewhere in system takes responsibility for recording it,
while in code you don't have to care much about details, and it should
look pretty much as usual, like:

Foo>>bar: anObject
   bar := anObject

Now, if we rewrite it like this:

Foo>>bar: anObject
   self captureIvarChange: #bar action: [ bar := anObject ].

we now should be able to capture the change of 'bar' instance variable
in receiver.
A framework could choose to:
 - [very][deep]copy the original value of 'bar'
 - apply the action block
- (optionally) compare the new value with old one and see what changed
based on this analysis , produce a delta, which can be serialized and
then reconstructed in different image/environment.

Also, by making sure, that there is no other assignments to 'bar' ivar
in our code, except in #bar: accessor, we 100% guarantee that any
change to our state
can be captured.
Same scheme, of course, can be used for capturing changes to
collections and variable size receiver's.

The intent is, that we should put such hooks directly in the places
where actual changes is made and at such places, which we consider
worth to capture, so, the code which capturing the change will live
together with code which actually does the change, but not in separate
layer like SystemEditor.

Then we can be sure, that no matter what permutations we making with
core system classes & state they holding, we could be able to track
all changes without need to synchronize it with some 'tracking code'
which lies in separate place.

Also, potentially, such framework should work in three different ways:
- recording changes while someone applying them (same as DeltaStreams does)
- recording changes, but not applying them (same as SystemEditor does)
- replaying changes

--
Best regards,
Igor Stasenko AKA sig.

On 2009-12-24, at 9:42 AM, Igor Stasenko wrote:

Hi Igor,

I agree with your assessment of SystemEditor - the design is inherently fragile, because it relies on one set of objects closely mimicking the behaviour another set of objects, but with a completely different implementation. It's definitely error-prone and the parts of SystemEditor that mimic the class-modification protocol are the ugliest.

However, I think that a generalized framework for capturing state changes and applying them later is a Very Hard Problem. It might be possible if you implemented a new Smalltalk with transactional semantics like Gemstone: commit and abort operations, and a third operation to abort-but-create-a-change-set. Encoding the change set in such away that it was reasonably independent of the starting state and could be applied to a *different* starting state is an exercise I leave to the reader, but one can imagine that being free to play with the semantics of the programming language could make it feasible.

I think that trying to implement this against an existing system with lots of legacy code would be impossible. Every so often somebody runs up against some limitation of Monticello and wonders why we didn't just implement a generalized object versioning system. This is another framing of the same problem. I don't have the math or comp sci chops to prove it, but I bet it's one of those nature-of-computing issues. I suspect that solving this problem would require solving the halting problem, or proving that np=p or some similarly stupendous theoretical achievement.

The approach we took with Monticello and SystemEditor was to cheat by limiting the domain to a certain set of objects with well-understood semantics, ie, the Smalltalk meta-objects: Classes, methods etc. There might be a better way to implement SystemEditor, make different engineering tradeoffs etc. I don't think there's a way to make it robust in the face of changes to the semantics of Smalltalk language, like Traits.

Colin

> However, I think that a generalized framework for capturing state changes and applying them later is a Very Hard Problem. It might be possible if you implemented a new Smalltalk with transactional semantics like Gemstone: commit and abort operations, and a third operation to abort-but-create-a-change-set.

In fact it is quite simple to do at the image level:

    http://scg.unibe.ch/archive/papers/Reng08aTransMemory.pdf

I haven't tested the implementation on installing code though. The
problem is that this involves quite some primitives that need to be
handled differently.

Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

2009/12/25 Lukas Renggli <[hidden email]>:
Thanks for the pointer, Lukas. Very nice alternative :)

This, however is a bit overkill, to my thinking, since inside a
transaction its going to capture all changes to any existing objects
regardless the intended scope.
While, if we talking about classes, our scope is tend to be small
(comparing to whole system) since we have to care about much less
things: method dictionary, organization, ivars etc etc.
For instance, if we would want to change a number of instance
variables for existing class inside transaction,
this will lead to recompiling the whole class and subclasses and
mutating all of existing instances, and inevitably, a transaction will
capture quite big number of objects.

While, if we define the scope: capture changes only to specific
object(s), while rest of the code will run normally (even if inside
transaction), this will be much more efficient.

This is what actually a SystemEditor does, but i want to find out a
simple way how we could make it without the need of creating a
separate layer which needs to mimic all the behavior of observed
object(s).

> Lukas
>
> --
> Lukas Renggli
> http://www.lukas-renggli.ch
>

--
Best regards,
Igor Stasenko AKA sig.

2009/12/24 Colin Putney <[hidden email]>:
Let me clarify couple things: Of course i don't want to make it too
generic. I want to make it generic enough for capturing changes in
existing Class/Metaclass model. This may be hard , but i think it is
not Very Hard (an existance of SystemEditor is living proof of that).

> I think that trying to implement this against an existing system with lots of legacy code would be impossible. Every so often somebody runs up against some limitation of Monticello and wonders why we didn't just implement a generalized object versioning system. This is another framing of the same problem. I don't have the math or comp sci chops to prove it, but I bet it's one of those nature-of-computing issues. I suspect that solving this problem would require solving the halting problem, or proving that np=p or some similarly stupendous theoretical achievement.
>
> The approach we took with Monticello and SystemEditor was to cheat by limiting the domain to a certain set of objects with well-understood semantics, ie, the Smalltalk meta-objects: Classes, methods etc. There might be a better way to implement SystemEditor, make different engineering tradeoffs etc. I don't think there's a way to make it robust in the face of changes to the semantics of Smalltalk language, like Traits.
>

Smalltalk claims to be a self-descriptive language. And, since its
claims so, then it should be possible to observe what changes is going
to happen, if i evaluate some code (a Lukas paper is good illustration
to that).
So, then it is up to a software engineer to make it easy to describe a
system change along with the change itself.

This is what a SystemChangeNotifier, isn't?
And best way to ensure that description is in sync with actual
implementation is to make it live as close as possible (in same
method).
It is not automated (in contrast to transactions), but hand-crafted
observation of already made changes.
So, why we can't employ the same approach, but for not-yet made changes?


> Colin
>



--
Best regards,
Igor Stasenko AKA sig.