[ANN] Hydra VM: A multi-core capable Croquet VM

On 22/02/2008, Klaus D. Witzel <[hidden email]> wrote: I have no read access on that file. I took another
minimalmorphic.7201.zip located in same directory.

>  Recall that these .images have rather incomplete file i/o and minimal
>  error-UI handling, but MC works (only used load so far).
>
>  >
>
>
>
>


--
Best regards,
Igor Stasenko AKA sig.

On Fri, 22 Feb 2008 13:30:13 +0100, Igor Stasenko wrote:

Yes, saw it, chmod+ftpd+Web browser's ftp are not really friends. Checked  
that it works now.

> I took another
> minimalmorphic.7201.zip located in same directory.

Never run that with Croquet.exe, you better delete that one now.

>>  Recall that these .images have rather incomplete file i/o and minimal
>>  error-UI handling, but MC works (only used load so far).
>>
>>  >
>>
>

On Fri, 2008-02-22 at 14:17 +0200, Igor Stasenko wrote: I should have more spare time (and knowledge of course). This is
exactly what would want to do. A hydra vm with a lot of spoon
vms.

Go, go, go :)

Norbert

On 22/02/2008, Klaus D. Witzel <[hidden email]> wrote:
Btw this one works (minimalmorphic.7201) but not minimalmorphic2.

Okay i found the cause of this: simply because window is being hidden
in SetWindowSize() and then never shown later :) I wonder why this bug
pops out depending of what image i'm using.

By disabling hiding i'm also found that there are two main windows
created instead of one.
Oh.. this windowing stuff are quite messy.. i'll try to fix this today.
But i'm not sure that this will fix a 'freezing' bug. Well, lets hope it will :)


--
Best regards,
Igor Stasenko AKA sig.

Igor Stasenko wrote:
> In this respect, i think, best would be to incorporate Spoon features to VM.
> As i understand (Craig, correct me if i'm wrong), a Spoon VM changes
> are incremental and not breaking any compatibility.

I'm not sure which changes you mean. If it's about imprinting, I'm no
fan of that. I don't want methods automatically transferred between
images - I want tools that I can use in a remote image to load and edit
code just like I do in the "main" image. With the end result being a
well-defined set of packages that are designed to work together.

Cheers,
   - Andreas

On 22/02/2008, Andreas Raab <[hidden email]> wrote:
I'm about marking oops while code running, to get a part of image
which required to run your code and scrap the rest.


--
Best regards,
Igor Stasenko AKA sig.

Igor Stasenko wrote:
Yes, and that's exactly what I'm no fan of ;-) In a realistic situation
you'd really want to understand what is being used where - the fact that
a particular method hasn't been touched during a particular run of the
code is just as insignificant as the fact that during a debugging
session you might touch code that isn't even in the product you're
trying to design.

Cheers,
   - Andreas

On 22/02/2008, Andreas Raab <[hidden email]> wrote:
Well, there is people who wish to see such capabilities, and if Spoon
features orthogonal to your approach, then why not include them?
In this case both camps will feel suited with new VM and can use it in
one way or another.


--
Best regards,
Igor Stasenko AKA sig.

On Fri, 22 Feb 2008 17:39:14 +0100, Igor Stasenko wrote:

You know about the difference between, this my code has been executed at  
least once v.s. this my code will be executed? It is as easy to  
mechanically remove a method like #(cannotReturn: mustBeBoolean) as is  
removing any other as-yet-not-exexcuted method ...

Good luck :)

>
>

Igor Stasenko wrote:
> Well, there is people who wish to see such capabilities, and if Spoon
> features orthogonal to your approach, then why not include them?

Because there is cost associated with it. Probably some runtime cost
(which I haven't measured) but definitely cost in terms of added
complexity in the VM.

> In this case both camps will feel suited with new VM and can use it in
> one way or another.

My problem with this is that I cannot see any situation in which someone
would really want to rely on that mechanism. It seems to me that the
approach is (as you said in an earlier email) a way to create smaller
images "without much effort". I'm interested in the *hard* way of doing
this (i.e., by going in and refactoring and removing things manually)
because I think this is the only way in which it can be done in practice.

But be that as it may - the real question is this: Let's assume we have
a tiny kernel image that is about 100k in size and that we can build up
from. What good is the mechanism at this point? It seems to me that it
is only useful to get to a small image but once this is achieved it's
basically obsolete. And I'm definitely not in favor to modify the VM for
something that will only have a very short window of usefulness.

Cheers,
   - Andreas

I can imagine a system where the display for another hydra instance appears like a small icon (akin to a morphic sub project, or even a croquet portal)...jump into it and the bits on the display are being painted by the other hydra instance.  You are now immersed in this other object space and able to freely work with all the tools, etc.  This seems a much easier path than trying to shuffle a bunch of objects between images and build remote enabled tools.

On 22/02/2008, Andreas Raab <[hidden email]> wrote:
Well, for instance, lets consider following use case:
suppose i ran some code in main image, which marked all objects which
this code used.
Then i can create new object memory, containing this memory and spawn
zillion of parallel running interpreters.
It is good, because you don't have to deal with files and preparing
images. A marked code memory footprint can be really small in this
way, which you can never achieve using 'hard' way.

This can look as simple as following:

copies := HydraVM markMemoryWhileDoing: [ .... code...  ] thenSpawnCopies: 100.

Yes, as Klaus noticed, there of course the difference between code
which runned and code which potentially can be run, if you provide
different input to it.
But this, i think, up to ones who using this, not me :)

Oh, and maybe then, if you strongly opposing this to be as part of VM,
then what about doing this at language side? What if let users develop
own ways how to create custom object memory, and make this as simple
as:

array := MyStripper scanMemoryForUsefulObjects.
HydraVM spawnInterpreterWithMemory: array.

.. so this will require a single change in HydraVM, just a simple primitive :)

>  But be that as it may - the real question is this: Let's assume we have
>  a tiny kernel image that is about 100k in size and that we can build up
>  from. What good is the mechanism at this point? It seems to me that it
>  is only useful to get to a small image but once this is achieved it's
>  basically obsolete. And I'm definitely not in favor to modify the VM for
>  something that will only have a very short window of usefulness.
>




--
Best regards,
Igor Stasenko AKA sig.

On 22/02/2008, Stephen Pair <[hidden email]> wrote:
Yes, HydraVM architecture designed in way, that its potentially easy
to shuffle heads.
Early, i'm even thinking about adding some magic keys (like Alt-TAB ..
) to shuffle ownership of screen and input between images.
Or even add support to have each thread having own separate window and input.
But i'm not sure if we really need to push things in that way, because
its would be not better than running multiple processes each with own
image.
In another turn, an absence of ways how you can interact with images
directly pushing you forward to develop a ways how you can control
them remotely. Which is good :)


--
Best regards,
Igor Stasenko AKA sig.

Igor Stasenko wrote:
> Well, for instance, lets consider following use case:
> suppose i ran some code in main image, which marked all objects which
> this code used.
> Then i can create new object memory, containing this memory and spawn
> zillion of parallel running interpreters.
> It is good, because you don't have to deal with files and preparing
> images. A marked code memory footprint can be really small in this
> way, which you can never achieve using 'hard' way.

Let me correct you. Not only can the "hard" way achieve it, it is in
fact the *only* way by which this can be done in general. You're
probably not realizing this but what you describe above is literally how
Croquet replication works. When a participant joins, we create a
snapshot of the current space and we do this by creating an image
segment for all the objects in this space. The only reason we can do
this is because we *know* (by having built it the hard way) which
objects are in what island. Without that knowledge originating from
islands and future sends it would be completely impossible to extract a
self-contained, consistent set of objects.

> Yes, as Klaus noticed, there of course the difference between code
> which runned and code which potentially can be run, if you provide
> different input to it.
> But this, i think, up to ones who using this, not me :)

True. I just happen to think that the mechanism is fundamentally flawed
and cannot possibly work the way in which people naively think it ought
to work. And by now we do have quite some experience in building actual
production systems that use such mechanisms and that's exactly where my
opposition comes from. Note that it's always easy to come up with
trivial little examples where things will work all right. It's even
possible to come up with situations where it works "most" of the time
(Morphic projects are one such example). But *all* the time? No way
unless you're doing it the hard way because it is the only case in which
you can assert the properties that you need.

> Oh, and maybe then, if you strongly opposing this to be as part of VM,
> then what about doing this at language side? What if let users develop
> own ways how to create custom object memory, and make this as simple
> as:
>
> array := MyStripper scanMemoryForUsefulObjects.
> HydraVM spawnInterpreterWithMemory: array.
>
> .. so this will require a single change in HydraVM, just a simple primitive :)

We have this already. It's called an image segment.

Cheers,
   - Andreas

On Feb 22, 2008, at 11:12 AM, Andreas Raab wrote:
>
> We have this already. It's called an image segment.
>
> Cheers,
>  - Andreas


Er if you have any fixes or say a technical note talking about how to  
store/read objects using an image segment that would be good. .

I'll note in Sophie we use image segments to store the font example  
image cache between restarts that has never given us a problem,
and we were storing a collection of Assocations where the key was a  
string and the value a Form. This worked oh 99.99% of the time
but the 0.01% failure rate made us pull the feature and now we write  
the data directly to a file.

Yes I guess I should make a mantis report

http://bugs.squeak.org/view.php?id=6944

1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
Recursive not understood error encountered
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
482693636 ImageSegment>restoreEndianness
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
482692604 ImageSegment>comeFullyUpOnReload:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
473153124 DataStream>next
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
473153032 ReferenceStream>next
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
473148616 SmartRefStream>next
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472929900 SmartRefStream>scanFrom:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472929380 ObjectScanner>scanFrom:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472929160 [] in PositionableStream>fileInAnnouncing:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472929056 BlockContext>on:do:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472676684 [] in PositionableStream>fileInAnnouncing:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472919144 [] in CProgressBar>displayProgress:at:from:to:during:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472918548 BlockContext>on:do:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472918640 [] in CProgressBar>displayProgress:at:from:to:during:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472918456 BlockContext>ensure:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472677148 CProgressBar>displayProgress:at:from:to:during:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472676868 CProgressBar class>displayProgress:at:from:to:during:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472676776 TweakUIManager>displayProgress:at:from:to:during:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472676592 String>displayProgressAt:from:to:during:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472676456 PositionableStream>fileInAnnouncing:
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472676344 FileStream>fileIn
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472676252 MultiByteFileStream>fileIn
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472676160 ReadWriteStream>fileInObjectAndCode
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472676068 FileStream>fileInObjectAndCode
1/26/08 1:46:46 AM [0x0-0xae0ae0].org.sophieproject.sophie[27135]  
472675344 SophieResourceManager  
class>specialGetImageSegmentArrayFromFileURI:

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================

On Fri, 22 Feb 2008 20:12:55 +0100, Andreas Raab wrote:

> Igor Stasenko wrote:
...
And there is Ian's RemoteFrameBuffer,

-  
http://lists.squeakfoundation.org/pipermail/squeak-dev/2007-January/112860.html

a natural way for commanding one .image from the other (don't forget  
setting the display depth on both sides, then it works, here with  
Croquet.exe and a MinimalMorphic.image controlled from Squeak.exe with a  
3.9 final .image :)

And thanks to Norbert for his 3.9 RemoteFrameBuffer .cs !

Cant't wait for the fixed #saveAsHeadlessImage: to work ;-)

/Klaus

> Cheers,
>    - Andreas
>
>

John M McIntosh wrote:
Well *the* ground rule is to not use class ImageSegment ;-) The
primitives are simple enough but the code in ImageSegment is ... err ...
difficult to understand (to put it mildly). If you are curious check out
IslandWriter and TIslandWriter in Croquet which use the primitives but
not the code in ImageSegment (of course, they do rely on islands).

The other thing you need to keep in mind is that for image segments to
work you must not introduce rogue pointers to the objects you are trying
to store. If you do, in *any* shape or form you are doomed. This is most
likely the problem you face (it can happen indirectly by leaving a
debugger in the background; by having a delay being active; by some
random mutex being in use etc).

> I'll note in Sophie we use image segments to store the font example
> image cache between restarts that has never given us a problem,
> and we were storing a collection of Assocations where the key was a
> string and the value a Form. This worked oh 99.99% of the time
> but the 0.01% failure rate made us pull the feature and now we write the
> data directly to a file.

Yes, that's what I mean by situations that work "most" of the time. It's
hard to track these down - in Croquet we have rules that allow us to
catch and trace situations in which this goes wrong (the dreaded
CheckpointFailure ;-) Fortunately this has become an excruciatingly rare
situation (basically only if we make a dumb mistake like I just did
today ;-)

Cheers,
   - Andreas

On Sat, 23 Feb 2008 04:32:28 +0100, Joshua Gargus wrote:

You might also want to check

- http://www.google.com/search?q=ibm+k42+oo-design+per-resource+object

in particular their three techniques they found to be especially important  
in achieving scalability. I leave comparision of Hydra approach with K42  
approach to the reader ;-)

/Klaus

> Thanks,
> Josh
>
>
>>
>> - Stephen
[...]

On Feb 23, 2008, at 12:31 AM, Klaus D. Witzel wrote:

Yay!  Real data!

Thanks for the very interesting link, this is exactly the sort of  
thing that I had in mind.

Josh