Recording in Croquet

Hi,

At the University of Minnesota we are interested in recording all the
activities that occur on a given Croquet island for later playback.  We
want this functionality because:

1) Instructors won't always be present in a space, but still want to be
able to review the activities that occurred.
2) Sometimes Croquet interactions might take place between two student's
computers and they might want to "hand in" what they did to their
instructor.
3) Classes or seminars given in Croquet could be reviewed later by others.

I don't think this is functionality only of interest to educators... I
can imagine Qwaq Forums wanting to record a meeting for later playback.

Anyway, we have hit a brick wall with this and are hoping that some
people would also be interested enough in recording/playback that they
would like to start a dialog about this.  I'll briefly describe our
approach:

We capture all messages at the TSimpleRouter>>send:from: level and
flatten them to a file.  We then read these back in later, reconstitute
and send them just as if they were coming from TSimpleRouter.  This
works absolutely fantastically for most things.  The problem we have
encountered is if anything new is introduced to the island while it is
being recorded.  During playback, when the object is added to the space
(using the addChild: method) it will fail trying to decode the
arguments.  We have thought that perhaps the OID's of things were not
getting set correctly, but I am not 100% sure of that and am even
doubting it because it still happens if we block all other events on the
playback island.

Andreas mentioned that we could perhaps capture at a higher level than
the router.  The thing that calls the router seems to be the
TFarRef>>future call and that seems to be as high as we can capture
unless we want to add code to every object on the island.

I admit we've been staring at this for a long time now and there
probably is something obvious that we can no longer see... but we would
love to work or talk with anyone else interested in this.  I can provide
more detail and code if that is helpful.

Thanks!
Liz Wendland
University of Minnesota

Liz - there is a semantic question here regarding what you want to
happen with respect to new object creation.

Recording the message flow in a computation works correctly to
reconstruct the computation exactly, given a particular starting
state.   You must ensure that the particular starting state is indeed
the same, otherwise, you end up with a huge mess.

Understanding why the mess happens is useful to figuring out a good
way.  Rather than "higher level" (what is "level" anyway?  That is an
arbitrary distinction that does not appear in the code, so it leads to
programming by magic belief... :-( ) the key is to think about groups of
objects.  (once upon a time, this concept of groups was referred to by
me as TeaParties.  Don't confuse with Islands.  That's related but
merely an implementation hack).

Essentially, you can view any collection of objects you can draw a line
around as a source and sink of messages.   Since only messages
communicate information, the messages *into* the collection completely
determine the messages *out*.   The messages prior to a particular
TeaTime (back to the origin) *are* the state of the collection at that
TeaTime - this is what it means to be deterministic.

In a few words, then, this is the entire semantics of TeaTime and
TeaObjects and collections of TeaObjects.

If you want to "record" for later replay, you probably want to record
only a subset of the objects, and only for a particular period of
TeaTime.   And presumably, you want to be able to play the recording
back with new messages (such messages as input events from the replayer,
state requests from the replayer desiring to render from a different
point of view, etc.).   Otherwise, there is no value to a recording.  At
the same time, you want to be able to discard messages that are sent out
in the original execution of the collection, so they don't cause havoc
to collections of objects not interested in that replay.

Note that the "replay" cannot be identical.   But one hopes that its
behavior is not affected much during the replay.   You wouldn't want it
to be completely identical - you want effective identity.

So here are the semantic problems you need to deal with somehow.  
Dealing with means defining what you mean.  Then the implementation
should fall out nicely and directly.

First, objects outside of your collection may be part of causal chains
that link outgoing messages back to incoming messages into the
collection.   In other words, an event inside your collection may have
an effect on future events in your collection because a message goes out
from the collection to outside objects, then comes back in.    If you
think about this, recording all incoming messages into the collection
solves this problem - as long as you don't introduce new messages that
can cause new outgoing messages that weren't there the first time the
simulation was run which then result in causal chains generating
messages back in.

One solution to this first problem is to be careful to select the
"right" set of objects to be in your collection.   The rule needs to
make sure that any of the newly added messages cannot add
causally-linked flows involving objects outside the collection.  This
rule is too strict, because there can be objects outside the collection
(such as the ones used in replaying itself - the user's rendering
environment and input management) that are correct.

But this doesn't completely solve the problem, because some of the input
messages into the collection will be messages that result from the
causal chains arising from output messages.  Those messages should not
be included in the recording.  One way to squeeze them out, perhaps, is
to recognize that causally linked messages have the same TeaTime as the
ones that cause them.   Thus, incoming messages that have the same
TeaTime as a prior (in execution time) outgoing message are special
cases.   They *may* be part of a causal chain.  At replay time, playing
them back can be problematic because they will be essentially
duplicates, if the environment handles the outgoing message the same way
as the first time.  (of course the environment is *different* during the
playback, so the result won't be exactly a duplicate - but the old
causally linked message should not be played back).

Second, the state as defined above is the sequence of messages from the
beginning of time till the recording starts.   That is different from a
collection of object ids.   Replaying those messages will create a new
set of objects, distinct from the original set (but completely
isomorphic to it - it is a replicated computation of the first one).

Here you have to decide what you want.   If you want a new set of
objects, then you have to be careful what *state* you start with.  
Don't reuse the old objects.   The "image" you might save is just a
compressed form of the past history of input messages.  If you an
isomorphic set of object states with new object ids, and translate all
messages to use those new object ids instead of those in the collection,
that works in the sense of recreating history.

If you want to play back the messages against the original set of
objects (in their later-in-time states) then you can do so, but the set
of objects includes objects that have been deleted.  Not clear what that
means.  And more complicatedly, the causal-chain loopback messages (for
want of a better name) that you recorded probably should NOT be played
back if there is an environment that is doing the same thing.

Note that I have not "solved" the problem, but attempted to lay out the
semantic issues you are struggling with.

It's pretty clear that simple recording of low level events won't work.

Recording input device events only might work - but recognize that they
are interpreted relative to user's UI state, which is not available at
playback time (as an example, the input event clicking on a 2D point is
interpreted by projecting a ray into the 3D space, and asking the
objects touched what to do with the event - this assumes a particular 3D
camera view).





Liz Wendland wrote:

[Resend; I first sent this from an account that wasn't subscribed]

Hi Liz -

I think David gave a good overview of the associated issues so I'll
stick to the issues at hand.

I'm pretty sure that the problem you see with new objects being created
is related to a problem with OIDs. This is the only explanation and it's
quite likely depending on how exactly you "feed" the events into the
stream. Basically, the replay must be completely passive; you cannot
send any other messages into the island because they will change the
state that later messages expect to find.

Object names are just the most probable place for these issues to occur
because the naming system uses a pseudo-RNG which diverges almost
instantly if "additional" messages have been sent in the meantime. The
issue with object names could be fixed though; I'm just not sure if it's
worth it ("worth it" as in: whether the problems introduced by extra
events just get a lot harder to find and more obscure).

Depending on what exactly you are trying to do, a perhaps better way to
deal with these replays may be to think of them as one of the little 3D
portals (miniature worlds) and all you can do is watch what is going on
in there but not yourself interact with it. In which case (because you
can't interact with the embedded space) there shouldn't be any problem
at all.

If that (e.g., replaying of messages without any additional events)
doesn't work then there is something deeper broken.

Cheers,
   - Andreas


Liz Wendland wrote:

Hi All,

Thanks so much to David and Andreas for their excellent responses.  
Everything they said made sense and I don't think I am disobeying any of
the basic axioms described.  I decided the easiest thing for me to do to
explain what I am seeing was to create a simple package demonstrating
it.  I have attached it to this posting.

Basically it is a new morph called RecordingTest.  It has two items in
the zoom menu "Record" and "Play".  To see my problem, drag out the
morph, bring up the Zoom Navigation and click on Record.  Quit the
morph.  Two files are created:
cache/liz.c3d  - the saved island
Recordings/liz.caa  - a recording of a TRectangle being added to the space.
If you then drag out RecordingTest again and click Play you will get an
error "No such object".

The approach we have taken is that we capture certain events at the
Router>>send:from: level.  On playback we then emulate the Router and
dispatch the message to the proper receiver.  The problem occurs on the
addChild: call.  The arguments for that message are encoded, but when we
attempt to decode that message it can't find the ID on the island.  I
have turned on logging in the TObjectID>>from: method and I can see that
my TRectangle has an identical ID in both the record and playback
phase.  Why can't the decode find it?

Please note that on playback I am restoring the saved island as a new
island and the only thing that happens on the island is the playback of
the saved events.  I store a TPlaybackCamera on the island before I save
it so I don't have to create any events there on playback.

Thanks again for any ideas!
Liz

Andreas Raab wrote:

Hi Liz -

Liz Wendland wrote:
> Everything they said made sense and I don't think I am disobeying any of
> the basic axioms described.  I decided the easiest thing for me to do to
> explain what I am seeing was to create a simple package demonstrating
> it.  I have attached it to this posting.

Thanks, this is very helpful. The problem you are seeing is caused by
promise-pipelining, e.g., the ability to send messages to objects that
don't even exist yet. In your example you do something like here:

        b := self activeIsland future new: TRectangle.
        b wait.
        pause := self activeSpace future addChild: b.
        pause wait.

Which is at the heart of the problem. First, there is really no need to
add the waits there - promise pipelining ensures that you can simply write:

        b := self activeIsland future new: TRectangle.
        pause := self activeSpace future addChild: b.

This is also the cause of the problem you are seeing. When a client
creates a promise for a future send, it assigns a name to it which
describes "the object resulting from executing some message" and which
is consequently independent of the name of the object it ultimately
results to. It is *that* name which is being looked for when you see the
error (print out the name of the variable "b" above to see it) and it is
that name that isn't being found (not the object's internal name).

This is why in TIsland>>execute: the result of the computation is stored
under the ID associated with that message (which is the name of the
promise that was created when it was send).

Obviously, your code needs to do the same, e.g., register the result of
the computation under the id of the message being executed. Something
like here:

TPlaybackCamera>>performMessage: msg
   "... yaddaya ..."
   result := receiver  perform: msg selector withArguments:  arguments.
   msg id ifNotNil:[
     Processor activeIsland register: result name: msg id.
   ].

Cheers,
   - Andreas

Dear Andreas,

Thanks so much for the excellent advice!  Your fix works like a charm
for my example. We are finally able to move ahead with our recording
code.  :)  I need to polish it a bit and then I'll post it to
Contributions...

Thanks again!
Liz Wendland


Andreas Raab wrote: