Capabilities in Squeak (attn: Lex Spoon and friends)

Hi all.

Does anybody have code (particularly VM modifications) which allow
Capabilities in Squeak?

In particular, I'm referring to code that implements stuff described on
this page: http://minnow.cc.gatech.edu/squeak/uploads/2074/sandbox.html

In the past there have been a couple of ambitious projects that allow
security in Squeak such as Squeak-E. What became of these? Is there code
floating around somewhere? I'm only seeing Squeak-ELib on squeakmap.

I'm particularly interested in a debugger which uses Capabilities to
debug/inspect objects rather than invoke methods such as
Object>>printOn:. I'm working with message capture stuff, and the
debugger falls over badly.

Michael.

I've been using a class called "MessageCapture" which lets you receive
messages sent to an object. It's caused me no end of agony trying to get
it working, so I thought I'd share my agony with the community :-P.

Source code here: http://www.squeaksource.com/DPON.html - click on
"latest", MessageCapture-mvdg.2.mcz.

It's released under the.. umm... MIT license.

Also included is a Future class, which lets you do things like:

result := Future doing: [ some long computation ].

This will return immediately, forking off a process to do the
computation and replacing "result" with the result when it's finished.

Some of the tests fail. Any hints / tips / guidance is very welcome!
Especially as to how to capture #== and #class. Those two messages are
sent by the interpreter even before doing a message lookup.

Michael.

Am 15.10.2006 um 08:56 schrieb Michael van der Gulik:

You might want to look at how this is handled in Croquet (MIT  
licensed, too). As you know (or maybe not) Croquet is not foremost  
about 3D but about distributed computing with capability-based  
security. The messaging infrastructure with future sends, returned  
promises etc. is independent of the UI.

Now the current implementation does not yet provide water-tight  
security, but the principles are there and it's surely worth having a  
look.

- Bert -

Hi Michael,

how would you execute [ some long computation ] on another processor or  
another computer.

On another processor, Squeak does't work because its VM is neither  
reentrant nor multiprocessing-capable (sorry for using terms which are >  
30 years old ;-)

So it would be interesting to learn how you plan to offload [ some long  
computation ] to another computer (or, FWIW, to another process running on  
the same computer).

/Klaus

On Sun, 15 Oct 2006 08:56:48 +0200, Michael wrote:

Actually, you could fork the VM, establish a communication channel  
(I've used named pipes on Unix, works without VM modification), send  
the future message over there, wait for the result. As long as no  
full GC happens in either process, this is even memory efficient on  
systems using copy-on-write memory pages.

Offloading to a different machine is a bit more involved because you  
might have to clone the execution state in the master image first.

- Bert -

Am 15.10.2006 um 14:46 schrieb Klaus D. Witzel:

On Sun, 15 Oct 2006 14:56:38 +0200, Bert Freudenberg wrote:

> Actually, you could fork the VM, establish a communication channel (I've  
> used named pipes on Unix, works without VM modification), send the  
> future message over there, wait for the result. As long as no full GC  
> happens in either process, this is even memory efficient on systems  
> using copy-on-write memory pages.

Aha, this is how to avoid serialization of context(s), method(s) and other  
resistant factors.

> Offloading to a different machine is a bit more involved because you  
> might have to clone the execution state in the master image first.

I agree. So whatever the desired path to the solution, doesn't  
serialization ("clone the execution state") presuppose that at least the  
compiled methods are in sync between any pair of involved systems.

/Klaus

On Sun, Oct 15, 2006 at 02:56:38PM +0200, Bert Freudenberg wrote:
> Actually, you could fork the VM, establish a communication channel  
> (I've used named pipes on Unix, works without VM modification), send  
> the future message over there, wait for the result. As long as no  
> full GC happens in either process, this is even memory efficient on  
> systems using copy-on-write memory pages.

Why the named pipe? An OSPipe should work fine without requiring any
external setup outside of Squeak.

Dave

Am 15.10.2006 um 19:08 schrieb David T. Lewis:

> On Sun, Oct 15, 2006 at 02:56:38PM +0200, Bert Freudenberg wrote:
>> Actually, you could fork the VM, establish a communication channel
>> (I've used named pipes on Unix, works without VM modification), send
>> the future message over there, wait for the result. As long as no
>> full GC happens in either process, this is even memory efficient on
>> systems using copy-on-write memory pages.
>
> Why the named pipe? An OSPipe should work fine without requiring any
> external setup outside of Squeak.

The project I use the named pipes in does not have the  
OSProcessPlugin. But sure, if you use OSProcess for forking, OSPipes  
are available :-)

- Bert -

Bert Freudenberg wrote:
Correct me if I'm wrong, but I think Croquet uses pretty normal objects
for what it does. Objects are grouped together, and inter-group
references are done using FarRef objects. I think it completely avoids
the problem of capturing references - arguably a more stable approach
than what I'm doing.

Can anybody confirm this?

I'm aware that Spoon and Magma also do message capture. I like Spoon's
approach best: modify the VM.

Michael.

Klaus D. Witzel wrote:
> Hi Michael,
>
> how would you execute [ some long computation ] on another processor or  
> another computer.

Simple. To use another processor, just modify the VM to use pthreads.
Easy peasy! (*)

To do it on another computer, use my DPON project. Unfortunately it's
current status is "completely freezes the Squeak image after 5 seconds".
That's a bit hard to debug.

(*) this is sarcasm.

> On another processor, Squeak does't work because its VM is neither  
> reentrant nor multiprocessing-capable (sorry for using terms which are
>  >  30 years old ;-)

One day, after I've finished all my other projects, I'll be looking at
making the Squeak VM use pthreads so it can use multiple CPUs. Hopefully
somebody will beat me to it :-).

Michael.

On Mon, 16 Oct 2006 10:37:52 +0200, Michael van der Gulik wrote:
> Klaus D. Witzel wrote:
>> Hi Michael,
>>  how would you execute [ some long computation ] on another processor  
>> or  another computer.
>
> Simple. To use another processor, just modify the VM to use pthreads.  
> Easy peasy! (*)

:)

> To do it on another computer, use my DPON project. Unfortunately it's  
> current status is "completely freezes the Squeak image after 5 seconds".  
> That's a bit hard to debug.

:(

> (*) this is sarcasm.

Sure 8-)

>> On another processor, Squeak does't work because its VM is neither  
>> reentrant nor multiprocessing-capable (sorry for using terms which are  
>>  >  30 years old ;-)
>
> One day, after I've finished all my other projects, I'll be looking at  
> making the Squeak VM use pthreads so it can use multiple CPUs. Hopefully  
> somebody will beat me to it :-).

Hopefully :)

/Klaus

> Michael.
>
>
>

On Oct 16, 2006, at 4:32 AM, Michael van der Gulik wrote:

That's correct.  Croquet also has a notion of "future", but unlike  
your Future example above, there is no immediately-returned  
placeholder that is later replaced (via #become: ?) by the real value  
when it becomes known.  Instead, the immediately-returned value is  
another FarRef.  To determine whether the computation has finished,  
you can ask it #isResolved, or you can use 'aFarRef wait' to block  
until the FarRef is resolved.

Josh

Am 16.10.2006 um 10:32 schrieb Michael van der Gulik:

In the current implementation, yes. An earlier version used many  
automagic tricks to work transparently. But Magic Is Hard To Debug  
(TM). So I think the plan is to do it with normal objects first, do  
everything explicitly, get it right, and when the system really  
works, then you can add back the magic to make it easier to use. But  
broken magic does more harm than good.

- Bert -

Bert Freudenberg wrote:
They're very smart. I've pushed along with broken magic for a couple of
years. Believe me - it would be much, much easier not using broken magic.

Currently I'm trying to debug an image which freezes at random
intervals. Alt-. is useless and I've already applied my alt-. improving
tweaks. I've inserted print statements everywhere - none of them get
printed. Broken magic is fun :-#.

Michael.

On Oct 14, 2006, at 9:45 PM, Michael van der Gulik wrote:

> In the past there have been a couple of ambitious projects that  
> allow security in Squeak such as Squeak-E. What became of these? Is  
> there code floating around somewhere? I'm only seeing Squeak-ELib  
> on squeakmap.

Squeak-Elib is the only result from Squeak-E.  I also implement a  
Future class (called a Promise) which accepts messages and forwards  
them prior to resolving to a value.   It fails in 2 senses.

First of all there are bugs when resolving the result of a  
computation to a promise, particularly when going inter-vat, which is  
handled differently than an intra-vat promise.

Secondly, FarRefs and promises don't understand all the base protocol  
that a normal object understands so many of the tools in the image  
don't deal well with eventual objects.  Ultimately, this is your  
issue with needing VM changes, I believe.  Primitives cannot handle  
eventual arguments.  One example is when sending a #printString to an  
eventual object returns a promise then is used as the argument for  
rendering in Morphic.   Another example is an eventual ref to true is  
sent the message #ifTrue:.  What I think you would want to have is to  
protect the primitives.   In a primitive, if an argument is eventual,  
send an "invokePrimitive" message to that ref, such that the  
primitive will be invoked eventually.  That changes all primitives,  
unfortunately.

I have a vm changes file that changes #== and #class out there.

I have stopped development of Squeak-Elib.

I hope this helps,
Robert

Robert Withers wrote:
> Secondly, FarRefs and promises don't understand all the base protocol
> that a normal object understands so many of the tools in the image don't
> deal well with eventual objects.

Actually, I consider this a fatal bug of FarRefs which I finally solved
in the Croquet version (TFarRefs). In Croquet, we do not hide the
difference between "near" and "far" objects simply because we cannot
hide the difference in the transport protocol utilized. Since all
replicated messages require a roundtrip to the router before they are
executed you can kiss the idea of a "transparent" handling goodbye, even
with promise pipelining and all the other tricks. It's simply not
possible and trying to hide the difference makes the problems much worse
than admitting the differences and giving people a way to deal with it.
One of the things that gets relevant in a hurry for example is that when
you're behind a DSL link your upstream bandwidth is *extremely* limited
and you simply need to know which messages you send where and when and
why upstream. And that means the programmer needs to see when things go
over the wire, these messages simply must be explicit. [My apologies go
to both MarkM and MarcS who consistently told me that this would be the
case but until I had the chance of looking at "other people's code" I
hadn't realized how big both the conceptual as well as the practical
mess is if you try to "make those differences go away"].

So, today when you use Croquet and have a far ref you *must* send
explicit future messages. TFarRef itself is a plain old Object subclass
which means printing, inspecting and debugging works just fine. And I am
in the process of doing precisely the same for Tweak which will
dramatically simplify a lot of stuff that was just unbelievably awkward
before.

Cheers,
   - Andreas

On Oct 16, 2006, at 8:42 PM, Andreas Raab wrote:

> Robert Withers wrote:
>> Secondly, FarRefs and promises don't understand all the base  
>> protocol that a normal object understands so many of the tools in  
>> the image don't deal well with eventual objects.
>
> Actually, I consider this a fatal bug of FarRefs which I finally  
> solved in the Croquet version (TFarRefs).

I think that's pretty smart.  My description of changing the  
primitives to be eventual aware are intended to describe my concept  
of going the other way and that means that any object could possibly  
be eventual.  Whether they are remote or not doesn't matter - it  
truly is a change in the execution semantics of the VM and it's best  
to make that change rather than doing what I was doing.  Of course,  
this doesn't address issues that may arise due to latency or ordering  
which could still affect all those tools.  I agree that you may still  
desire to be more explicit when dealing with remote objects, but to  
my way of thinking that is secondary to the idea of making the VM  
eventual.  Anyway, that is how you would need to do it to get  
SqueakElib truly working in the image.

I think it's smart to have done what you did.

Cheers,
Robert

 > I think it's smart to have done what you did.

Thanks. Now if I could only claim that to be my idea... ;-)

Cheers,
   - Andreas

Robert Withers wrote:

On Oct 16, 2006, at 9:47 PM, Andreas Raab wrote:

> > I think it's smart to have done what you did.
>
> Thanks. Now if I could only claim that to be my idea... ;-)

ditto.  :-)  I can't recall where those ideas I expressed originated,  
perhaps yourself, but I was expressing them as opinions.

Cheers,
Robert

Robert Withers wrote:
Yea, I've discovered that. My next project will be modifying the
debugger to not send *any* messages to the objects it's debugging, but
rather use a capability with primitives to peek at each object's state.

 > Ultimately, this is your  issue
> with needing VM changes, I believe.  Primitives cannot handle  eventual
> arguments.  One example is when sending a #printString to an  eventual
> object returns a promise then is used as the argument for  rendering in
> Morphic.   Another example is an eventual ref to true is  sent the
> message #ifTrue:.  What I think you would want to have is to  protect
> the primitives.   In a primitive, if an argument is eventual,  send an
> "invokePrimitive" message to that ref, such that the  primitive will be
> invoked eventually.  That changes all primitives,  unfortunately.

I've solved this (with many bugs and instabilities) by making Futures
block on any message sends until the result returns.

The way I've done it is that MessageCapture is a subclass of ProtoObject
that overrides #doesNotUnderstand. I added a few methods like
#printString to make it work with the debugger and added a
MethodCaptureInspecter that will let you inspect a MessageCapture.

The Future itself is a normal object which uses MessageCapture.

This works for primitive methods, but there are issues with some sends
which I think I can solve. I think Craig Latta's approach in Spoon is
the best: create a special class (MessageCapture) which the VM checks
for on every message send, including #== and #class. If it is that
class, package up a Message and send it to a method on that class.

> I have a vm changes file that changes #== and #class out there.

Here?: http://minnow.cc.gatech.edu/squeak/2410

I'll be making changes to the VM eventually, so this code is much
appreciated. Do you mind if I consider it released under the MIT license?

Michael.