Replicated vs. centralized services

Ralph Johnson wrote:
> The orginal question was about having some sort of attached processor to
> do physics calculations.  Could Croquet off-load physics computation to
> these attached processors?  The answer is: only if they produce
> identical results on every machine.

Although this answer is technically correct, it leaves me with a bit of
a bitter taste because it comes across as "you can't do physics in
Croquet unless you have an engine that produces bit-identical results"
which is incorrect.

A more accurate statement is that you can't do REPLICATED physics unless
the physics engine produces bit-identical results on all the platforms.
It is, however, perfectly possible to do non-replicated physics (say on
a big super-computer) and merely broad-cast the results (events) to all
participants.

And even though this may not be a "pure" solution, it is one that works
extremely well and is extremely well understood (since it's nothing but
a central server) and that we *always* anticipated would be of use for
some problems. It is very important to understand that  Croquet is not
limited to replicated computations but can be used in what some might
call "server mode" - all it means is that some machine regularly informs
the other participants about "what happened" and because of the nature
of Croquet you might use dedicated "physics servers" or "game physics
servers" working with the very same islands that your web-browser shows.

Bottom line (and the point I'm trying to make here: Croquet is not
limited to "replicated computation only" - if centralized services are
needed they integrated trivially and smoothly into the system (a fact
often overlooked since we don't usually emphasize it to get the point of
replicated computation across).

Cheers,
   - Andreas

Andreas's amplification of Ralph Johnson's comment regarding physics vs.
replication is exactly correct.   But there is no need for full
centralization, either: Croquet's TeaTime-based computing architecture
*also* makes it possible to replicate computation closer to the user
machines, when (and only when) there are accelerator resources closer to
the users that compute correct answers depending on deterministic
inputs.   So if there is a population of "super machines" floating among
the user machines, replicate the computation to any number of the super
machines, then replicate the results distribution further, to include
machines that don't have the acceleration capability.

This adds to the programming burden, of course.  The "design patterns"
needed to do this simply are not developed enough to bundle them into a
simple abstraction.   But given common situations involving
accelerators, it should be possible to exploit the Croquet programming
abstractions to do this.

Croquet's computation model (virtual machine semantics) was designed the
way it was (mostly by me, with Andreas and David Smith critiquing all
along) *precisely* to make the abstraction of replicated computing work
simply enough enough to be used for coordination when computations are
moved around the network flexibly as needed for the application design.

Trying to do this in languages whose semantics are essentially
randomized (like C or FORTRAN are) would be foolish.  But given the
ability to replicate a computation (a well-defined virtual machine is
the base case), fluid computation is possible.

Andreas Raab wrote:

On 5/1/06, Andreas Raab <[hidden email]> wrote:
On some of the IGDA mailing lists, I pointed out Croquet as a possible
platform for future MMO (massively multiplayer online) game
development.  The response was overwhelmingly negative, because "you
don't want to expose the details of how your game works to any hacker
who comes along."

I tried to explain (I'm learning Smalltalk, so forgive me if I
misspeak here) the concept of a FarRef to a non-replicated (or, if
replicated, replicated between a cluster of back-end machines, any of
which could be routed to by the Router -- which, incidentally, I'd
like to see replicable as well) server backend.  The downside to this
concept is that the details of the server's protocol actions and their
handling are visible to everyone.

However, I'm confused as to how I can use the Croquet engine without
all the rendering, to do that back-end computation in a replicated
(and thus fault-tolerant -- a MAJOR selling point) manner.  (Then
again, I'd like to see the option of just not using a facility that
isn't there, rather than throwing an error that can't be aborted.
Positional updates can be maintained without having to render them,
but not in the current architecture.)

> Bottom line (and the point I'm trying to make here: Croquet is not
> limited to "replicated computation only" - if centralized services are
> needed they integrated trivially and smoothly into the system (a fact
> often overlooked since we don't usually emphasize it to get the point of
> replicated computation across).

For an MMORPG, I'd use replicated computation to update peoples'
positions relative to each other, and a central server to handle the
mobs, physics, and damage calculation.

And, incidentally: Hi, I'm new to this list.  My name is Kyle
Hamilton, and I'm interested (if you couldn't tell) in utilizing
Croquet to create an MMORPG-type system.  I haven't determined all of
my design requirements as yet; however, I'd like to learn how the
system works before I try to break the rules [if I have to].

For example: What happens when someone modifies their local copy of a
Smalltalk object?  According to the introductory videos, it doesn't
automatically propagate to everyone else, which could lead to
inconsistencies in the shared environment?

-Kyle H

Hi David,

   It sounds like TeaTime could provide some sort of distributed  
computation. Could you elaborate on this? Splitting computations to  
different nodes have always been a hard problems but it's actually  
getting important these days as the hardware are scaling out  
horizontally with dual core CPUs becoming mainstream in the next  
couple years.

Thanks.

David Li

On May 2, 2006, at 10:55 PM, David P. Reed wrote:

Hi Kyle -

> On some of the IGDA mailing lists, I pointed out Croquet as a possible
> platform for future MMO (massively multiplayer online) game
> development.  The response was overwhelmingly negative, because "you
> don't want to expose the details of how your game works to any hacker
> who comes along."

Yes, this is a very well-known concern ("the client is in the hands of
the enemy" as I've heard this referred to). And I would agree - if you
want to make a "fair" game you cannot realistically let each participant
decide on their own whether the bullet hit or not. And that's perfectly
fine - Croquet gives you the option to decide what portions of your
application you'd like to replicate where and how they should interact.

> I tried to explain (I'm learning Smalltalk, so forgive me if I
> misspeak here) the concept of a FarRef to a non-replicated (or, if
> replicated, replicated between a cluster of back-end machines, any of
> which could be routed to by the Router -- which, incidentally, I'd
> like to see replicable as well) server backend.  The downside to this
> concept is that the details of the server's protocol actions and their
> handling are visible to everyone.

No, not necessarily. Consider for example a situation in which we have a
group of replicated servers that handle the "backend stuff" and give
each of those a front-end gateway with a router for the users. That
means that the front and the back-ends are completely decoupled;
logically the user only deals with the router (server) "they are on".
That this machine then relays the messages to another set of replicated
("upstream") machines which handle the backend stuff is completely
invisible to the user. This model also has the advantage that it should
be fairly scalable for the number of users since you have a router
network now (something that we talked about but haven't done yet) which
allows you to again decouple the user handling from the back-end
messaging infrastructure.

> However, I'm confused as to how I can use the Croquet engine without
> all the rendering, to do that back-end computation in a replicated
> (and thus fault-tolerant -- a MAJOR selling point) manner.  (Then
> again, I'd like to see the option of just not using a facility that
> isn't there, rather than throwing an error that can't be aborted.
> Positional updates can be maintained without having to render them,
> but not in the current architecture.)

Using Croquet "without the rendering" is quite simple. Assuming you have
a router already running somewhere all you do is something like here
(this -and the router setup- is also in the docs):

   controller := TSimpleController new.
   controller connectTo: address port: port sessionID: id.
   p := controller login: user password: pass.
   p wait. "until connected"
   p := controller join.    "receive messages"
   p wait. "until joined"
   p := controller sync. "request replicated state"
   p wait.    "until sync-ed"

   p result ifNil:[
     island := controller newIsland.
     island future id: id name: 'Test'.
   ] ifNotNil:[
     controller install: p result.
     island := controller island.
   ].

At this point you have a reference to an island that you can perform
arbitrary computations in, say something like:

   "Create a service"
   svc := island future new: MyService.
   "Invoke a (replicated) computation on the service"
   svc future performService.

> For an MMORPG, I'd use replicated computation to update peoples'
> positions relative to each other, and a central server to handle the
> mobs, physics, and damage calculation.

Yes. Something along these lines seems appropriate (since I've never
done one I wouldn't know but it does sound reasonable).

> And, incidentally: Hi, I'm new to this list.  My name is Kyle
> Hamilton, and I'm interested (if you couldn't tell) in utilizing
> Croquet to create an MMORPG-type system.  I haven't determined all of
> my design requirements as yet; however, I'd like to learn how the
> system works before I try to break the rules [if I have to].

Nice meeting you.

> For example: What happens when someone modifies their local copy of a
> Smalltalk object?  According to the introductory videos, it doesn't
> automatically propagate to everyone else, which could lead to
> inconsistencies in the shared environment?

If someone "modifies their local copy" (by which I understanding
performing reflective surgery on the system to deliberately bypass the
regular means of message-based communication) it will get out of sync.
We don't try to prevent that. Generally, our model of security (which
this is a part of) is aimed at what we call "replicated security"
(contrasted with "individual security") by which we mean we care more
about what happens for the group of users rather than the individual.
Put differently, if you want to shoot yourself in the foot be my guest.
But if you want to shoot everyone else in the foot, now that would be
something that I'd indeed think is a problem that we need to deal with.

Cheers,
   - Andreas