Smalltalk › Squeak › Croquet › Croquet - Dev

An idea: allow Croquet Islands to add read-only off island resources to cache in single shot

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

7 messages Options

Baldur Johannsson

An idea: allow Croquet Islands to add read-only off island resources to cache in single shot

H'lo Croquet Developers

An idea came to me in response an observation/half-complaint of one
of the maintainers of
Croquet Collaborative made on his (we)blog http://www.wetmachine.com/item/769

The basic idea is to allow Croquet Islands to write out data to a
file in one transaction.
The file would be named by an cryptographic hash of its contents.
The transaction would return that name.

Pro: allows offloading of rarely changing content and data from the
Island dynamicly.
Con: would require at least one Island server to be also cache
server for the island.

Psuedo code implementation sketch:
TIsland writeOnceOut: data
"data should be an immutable bytearray"
| hash |
hash := sha1 hash: data.
self signal: #writeOnceOut with: data with: hash.
^ hash.

CacheManager runScript: #writeOnceOut:withHash: when: {TIsland.
#writeOnceOut}

CacheManager writeOnceOut: data withHash: hash
"writes data out into a file with the name hash"

please feel free to criticise it as you see fit
-Zarutian

Andreas.Raab

Re: An idea: allow Croquet Islands to add read-only off island resources to cache in single shot

Baldur Johannsson wrote:
> An idea came to me in response an observation/half-complaint of one
> of the maintainers of
> Croquet Collaborative made on his (we)blog http://www.wetmachine.com/item/769

Which observation/complaint?

> The basic idea is to allow Croquet Islands to write out data to a
> file in one transaction.
> The file would be named by an cryptographic hash of its contents.
> The transaction would return that name.
>
> Pro: allows offloading of rarely changing content and data from the
> Island dynamicly.
> Con: would require at least one Island server to be also cache
> server for the island.

But what problem is it solving?

> please feel free to criticise it as you see fit

Before I could realistically do that I would need to understand what
problem you're trying to address. The pseudo-code as given doesn't make
any sense to me since just writing out data once doesn't seem to address
any problem that I can see (shouldn't there at least be something that
makes use of that data somewhere?).

Cheers,
- Andreas

Baldur Johannsson

Re: An idea: allow Croquet Islands to add read-only off island resources to cache in single shot

On 06/11/2007, Andreas Raab <[hidden email]> wrote:
> Baldur Johannsson wrote:
> > An idea came to me in response an observation/half-complaint of one
> > of the maintainers of
> > Croquet Collaborative made on his (we)blog http://www.wetmachine.com/item/769
>
> Which observation/complaint?
>
This one (from that post):
Another issue is that the snapshot still has a lot of stuff in it that
doesn't really need to be there. We've done a lot of work in the
Collaborative code to move media outside of the simulation. We call
that being "off the island." But meshes are still on-island, and the
full-resolution textures created by the free-hand sketch tool are
still on-island. I think this is another factor of four to 10.

> > The basic idea is to allow Croquet Islands to write out data to a
> > file in one transaction.
> > The file would be named by an cryptographic hash of its contents.
> > The transaction would return that name.
> >
> > Pro: allows offloading of rarely changing content and data from the
> > Island dynamicly.
> > Con: would require at least one Island server to be also cache
> > server for the island.
>
> But what problem is it solving?

seems to me (when I did an prelimiary dig through Croquets code) that
there is no way for an code on an Island, for instance, to write out
an objects 3d mesh that doesnt change

>
> > please feel free to criticise it as you see fit
>
> Before I could realistically do that I would need to understand what
> problem you're trying to address. The pseudo-code as given doesn't make
> any sense to me since just writing out data once doesn't seem to address
> any problem that I can see (shouldn't there at least be something that
> makes use of that data somewhere?).

The Island would use that written out data just as it would for any
other off Island resource.

>
> Cheers,
> - Andreas
>
>

Andreas.Raab

Re: An idea: allow Croquet Islands to add read-only off island resources to cache in single shot

Baldur Johannsson wrote:

> On 06/11/2007, Andreas Raab <[hidden email]> wrote:
>> Baldur Johannsson wrote:
>>> An idea came to me in response an observation/half-complaint of one
>>> of the maintainers of
>>> Croquet Collaborative made on his (we)blog http://www.wetmachine.com/item/769
>> Which observation/complaint?
>>
> This one (from that post):
> Another issue is that the snapshot still has a lot of stuff in it that
> doesn't really need to be there. We've done a lot of work in the
> Collaborative code to move media outside of the simulation. We call
> that being "off the island." But meshes are still on-island, and the
> full-resolution textures created by the free-hand sketch tool are
> still on-island. I think this is another factor of four to 10.

I see.

>>> The basic idea is to allow Croquet Islands to write out data to a
>>> file in one transaction.
>>> The file would be named by an cryptographic hash of its contents.
>>> The transaction would return that name.
>>>
>>> Pro: allows offloading of rarely changing content and data from the
>>> Island dynamicly.
>>> Con: would require at least one Island server to be also cache
>>> server for the island.
>> But what problem is it solving?
>
> seems to me (when I did an prelimiary dig through Croquets code) that
> there is no way for an code on an Island, for instance, to write out
> an objects 3d mesh that doesnt change

Unfortunately, that particular problem is a bit larger than that. You've
solved a tiny part of it - writing out a binary data blob. What you
haven't solved (and this is where the real work lies) is how to utilize
it effectively.

If you look at texture management for example, only a tiny amount of
code is related to writing the texture data; but everything else is how
and where you use it. The other thing that is interesting is that for
textures for example, there isn't even "in-island" code to write it. And
that's because when you deal with these issues you quickly find that
it's easier to never put it into the island in the first place. The same
is true for other chunks of data.

>>> please feel free to criticise it as you see fit
>> Before I could realistically do that I would need to understand what
>> problem you're trying to address. The pseudo-code as given doesn't make
>> any sense to me since just writing out data once doesn't seem to address
>> any problem that I can see (shouldn't there at least be something that
>> makes use of that data somewhere?).
>
> The Island would use that written out data just as it would for any
> other off Island resource.

Yes, and *that* is the way the cookie crumbles (hm ... not sure if I got
my idioms right but hey ;-) It is really necessary to address the
problem from this perspective because writing binary data is by far the
least of your problems and should be done to serve the actual solution
you implement.

In Qwaq Forums (which supports off-island meshes) we deal with mesh data
just as we deal with textures: the data never actually enters the
island, all that enters it is a thumbnail representation of the mesh. So
we don't actually need code that writes binary chunks from the island
because we never let it get there.

So, to summarize: Writing binary data all by itself doesn't solve the
problem Howard was mentioning. In the more realistic usage scenarios it
is often easier to prevent the chunks from being put into the island to
begin with (since someone's got to put it there that code can as well
write it to someplace else) which is why there isn't any such code in
TIsland currently. There may be some use for it for "squeezing out"
existing bits but that code should be part of the larger solution in
this case.

Cheers,
- Andreas

Howard Stearns-3

Re: An idea: allow Croquet Islands to add read-only off island resources to cache in single shot

I'm chalking this one up on the conceptually "done" side of the board.

If I recall correctly, getting meshes off-island is on the roadmap
for Jabberwocky, and David already has a working version of this.
Just as Andreas describes, it never puts the full-resolution meshes
on-island to begin with, and this is also the way textures have long
worked.

In addition, "prims" such as cubes, spheres, and so forth can be done
with procedural geometry so that there really isn't any need for any
immutable mesh data to be part of the island definition that is
sync'd when a new participant joins a collaboration.

Results so far show typical work spaces at around 1/2 MB, with very
little growth as things get complicated.

Both of the these require some changes to rendering so that the
transformation can come from the replicated on-island model object,
so this is not just a small extension to be put top of the Hedgehog
codebase.

-H

On Nov 5, 2007, at 9:35 PM, Andreas Raab wrote:

> Baldur Johannsson wrote:
>> On 06/11/2007, Andreas Raab <[hidden email]> wrote:
>>> Baldur Johannsson wrote:
>>>> An idea came to me in response an observation/half-complaint
>>>> of one
>>>> of the maintainers of
>>>> Croquet Collaborative made on his (we)blog http://
>>>> www.wetmachine.com/item/769
>>> Which observation/complaint?
>>>
>> This one (from that post):
>> Another issue is that the snapshot still has a lot of stuff in it
>> that
>> doesn't really need to be there. We've done a lot of work in the
>> Collaborative code to move media outside of the simulation. We call
>> that being "off the island." But meshes are still on-island, and the
>> full-resolution textures created by the free-hand sketch tool are
>> still on-island. I think this is another factor of four to 10.
>
> I see.
>
>>>> The basic idea is to allow Croquet Islands to write out data to a
>>>> file in one transaction.
>>>> The file would be named by an cryptographic hash of its contents.
>>>> The transaction would return that name.
>>>>
>>>> Pro: allows offloading of rarely changing content and data
>>>> from the
>>>> Island dynamicly.
>>>> Con: would require at least one Island server to be also cache
>>>> server for the island.
>>> But what problem is it solving?
>> seems to me (when I did an prelimiary dig through Croquets code) that
>> there is no way for an code on an Island, for instance, to write out
>> an objects 3d mesh that doesnt change
>
> Unfortunately, that particular problem is a bit larger than that.
> You've solved a tiny part of it - writing out a binary data blob.
> What you haven't solved (and this is where the real work lies) is
> how to utilize it effectively.
>
> If you look at texture management for example, only a tiny amount
> of code is related to writing the texture data; but everything else
> is how and where you use it. The other thing that is interesting is
> that for textures for example, there isn't even "in-island" code to
> write it. And that's because when you deal with these issues you
> quickly find that it's easier to never put it into the island in
> the first place. The same is true for other chunks of data.
>
>>>> please feel free to criticise it as you see fit
>>> Before I could realistically do that I would need to understand what
>>> problem you're trying to address. The pseudo-code as given
>>> doesn't make
>>> any sense to me since just writing out data once doesn't seem to
>>> address
>>> any problem that I can see (shouldn't there at least be something
>>> that
>>> makes use of that data somewhere?).
>> The Island would use that written out data just as it would for any
>> other off Island resource.
>
> Yes, and *that* is the way the cookie crumbles (hm ... not sure if
> I got my idioms right but hey ;-) It is really necessary to address
> the problem from this perspective because writing binary data is by
> far the least of your problems and should be done to serve the
> actual solution you implement.
>
> In Qwaq Forums (which supports off-island meshes) we deal with mesh
> data just as we deal with textures: the data never actually enters
> the island, all that enters it is a thumbnail representation of the
> mesh. So we don't actually need code that writes binary chunks from
> the island because we never let it get there.
>
> So, to summarize: Writing binary data all by itself doesn't solve
> the problem Howard was mentioning. In the more realistic usage
> scenarios it is often easier to prevent the chunks from being put
> into the island to begin with (since someone's got to put it there
> that code can as well write it to someplace else) which is why
> there isn't any such code in TIsland currently. There may be some
> use for it for "squeezing out" existing bits but that code should
> be part of the larger solution in this case.
>
> Cheers,
> - Andreas

Andreas Vox-2

Re: An idea: allow Croquet Islands to add read-only off island resources to cache in single shot

In reply to this post by Andreas.Raab

Hi there,

my name is also Andreas (what a coincidence! ;-) ) and I joined the
ML just a few days ago. I havent done any Smalltalk or 3D development
yet, but I'm interested in the theoretical aspects and might do some
coding in the future.

I've a question about this:

Am 06.11.2007 um 04:35 schrieb Andreas Raab:

> In Qwaq Forums (which supports off-island meshes) we deal with mesh
> data just as we deal with textures: the data never actually enters
> the island, all that enters it is a thumbnail representation of the
> mesh. So we don't actually need code that writes binary chunks from
> the island because we never let it get there.
>
> So, to summarize: Writing binary data all by itself doesn't solve
> the problem Howard was mentioning. In the more realistic usage
> scenarios it is often easier to prevent the chunks from being put
> into the island to begin with (since someone's got to put it there
> that code can as well write it to someplace else) which is why
> there isn't any such code in TIsland currently. There may be some
> use for it for "squeezing out" existing bits but that code should
> be part of the larger solution in this case.

So how is off-island data distributed? Or is it necessary that all
client images already contain the data before they a started?

I read a comment in the roadmap about integrating a DHT powered cache
and that sounds like an interesting solution. Any details about that?

Also I wonder how well the client code is separated from the in-
island code - if at all. Would it be possible to write a Java client
with an embedded Sqeak VM just for interpreting the in-island
messages and do all the UI stuff from Java?

Cheers,
/Andreas

Howard Stearns-3

Re: An idea: allow Croquet Islands to add read-only off island resources to cache in single shot

There are, as Andreas suggested, a lot of issues here about managing
off-island resource.

What is built into all the SDK demos, is a mechanism whereby a high-
resolution texture can be kept in a local cache (in-memory first,
falling back to file system), and if not there, the system asks the
router for the resource, which in turn asks the participating machines.

The KAT demo (also in the SDK) generalizes this in two ways:
- It is used for all immutable media, not just textures.
- If the resource isn't in the local cache, it asks a separate
"meta-island" instead of the one Croquet island that you're
participating in. In this way, off-island immutable media is shared
across worlds. The meta-island is set up so that unlike a regular
Croquet island, you don't get a snapshot of the entire content when
you join.

As implemented in the KAT, the meta-island is routed ("hosted") by a
single Squeak image. That's not going to work at a global scale. The
API is designed so that the whole thing could be replaced by a DHT
designed/tuned for immutable data. Note that such a media DHT would
be useful for other applications. (!)

The KAT shared resource management works by eagerly pushing new
resources into the meta-island. (E.g., when you drag a picture or
movie from your desktop into a space.) There are several different
kinds of policies that one might want for this. The KAT code ducks
this issue.

There are other ways to generalize the off-island texture management.
I imagine that the next Croquet version will have a simple but
general off-island content manager, that can be subclassed to meet
the needs (and politics!) of particular applications.

See also: http://opencroquet.org/index.php/Off-Island_Resources
----------
With regard to Java: you could write parts or even all of Croquet in
Java. You could even write it in C if you limit yourself to a
particular hardware setup. What's required is a means of making sure
that the same execution produces the same results on all participants
(e.g., no floating point drift), and a means of getting an identical
snapshot of the current state of an island. See http://
opencroquet.org/index.php/The_Core_Model

Whether a particular approach makes sense is, of course, dependent on
what problem you are trying to solve.

On Nov 6, 2007, at 10:23 AM, Andreas Vox wrote:

> Hi there,
>
> my name is also Andreas (what a coincidence! ;-) ) and I joined the
> ML just a few days ago. I havent done any Smalltalk or 3D
> development yet, but I'm interested in the theoretical aspects and
> might do some coding in the future.
>
> I've a question about this:
>
> Am 06.11.2007 um 04:35 schrieb Andreas Raab:
>
>> In Qwaq Forums (which supports off-island meshes) we deal with
>> mesh data just as we deal with textures: the data never actually
>> enters the island, all that enters it is a thumbnail
>> representation of the mesh. So we don't actually need code that
>> writes binary chunks from the island because we never let it get
>> there.
>>
>> So, to summarize: Writing binary data all by itself doesn't solve
>> the problem Howard was mentioning. In the more realistic usage
>> scenarios it is often easier to prevent the chunks from being put
>> into the island to begin with (since someone's got to put it there
>> that code can as well write it to someplace else) which is why
>> there isn't any such code in TIsland currently. There may be some
>> use for it for "squeezing out" existing bits but that code should
>> be part of the larger solution in this case.
>
> So how is off-island data distributed? Or is it necessary that all
> client images already contain the data before they a started?
>
> I read a comment in the roadmap about integrating a DHT powered
> cache and that sounds like an interesting solution. Any details
> about that?
>
> Also I wonder how well the client code is separated from the in-
> island code - if at all. Would it be possible to write a Java
> client with an embedded Sqeak VM just for interpreting the in-
> island messages and do all the UI stuff from Java?
>
> Cheers,
> /Andreas
>