HydraTools and minimal images

On 14/02/2008, Klaus D. Witzel <[hidden email]> wrote:
Hehe, i assuming you realize that #doit's channel is just a system
default channel, which should be available for anyone. You can always
create own, pervasive channels, handling a data in pervasive manner
without polluting any files :)

It's as easy as doing:

myOtherImage := HydraVM loadAndStartNewImage: 'myOther.image'.

HydraVM doIt: 'HydraChannel listen: #myPervasiveChannel onReceive: [:data |
        <place your code here> ]' on: myOtherImage

now your just loaded image listening a #myPervasiveChannel
and you free to send anything to it :)

ermmrmrm.. can't quite understand what given example suppose to do.

suppose you having a method in HydraSMS class
foo: aString

and now, if you wish that by invoking a #foo: in your 'home' image,
string should be passed to 'work' image, you write there:
mychannel send: aString.

At other side, when you received a string, it may be collected
somewhere, so on next call to #nextString, it will return the first
available string in FIFO manner.

I don't see a reason in transferring a blocks (compiled bytecode),
since you stating, that both images containing very same code (and if
not, you can always perform a pervasive doit to make it appear :).

>
>  /Klaus
>

--
Best regards,
Igor Stasenko AKA sig.

On Thu, 14 Feb 2008 11:30:12 +0100, Igor Stasenko wrote:

Sure, you wrote that some time ago.

Nobody said that this wouldn't be possible; I don't see this compares to  
HydraSMS. BTW you forgot to write the things in the above which make your  
"counter" example synchronous ;-)

Demonstrate the synchronous transfer of c-o-n-t-r-o-l and a snippet on  
how/when/where to pass an argument. Have this solved once so that the  
zillions of asynchronous computational tasks (I mean, future projects  
which want to benefit from Hydra's multi-processor capabilities) can be  
c-o-n-t-r-o-l-l-e-d predictable and reliable.

> suppose you having a method in HydraSMS class
>
> foo: aString
>
> and now, if you wish that by invoking a #foo: in your 'home' image,

No, I didn't attempt to have an asynchronous string sent to another  
.image, do you pretend to not understand what synchronous is about :)

> string should be passed to 'work' image, you write there:
> mychannel send: aString.

But why the heck use that low level stuff, why not abstract away each and  
everything?

HydraSMS can do the same work over Sockets and (possibly, we have as yet  
not gone that far) over channels, without one bit of change. Not that is  
it therefore "better"; it is using the low level constructs you always  
give in your "counter" example, no doubt.

But whenever *you* write "myChannel send: aString" like in the above, you  
are obliged to also write all the checks before+after, that everything is  
still O.K. Otherwise that's not comparable to what HydraSMS is about.

Don't misunderstand; it's O.K. that "myChannel send: aString" exists, many  
(if not all) things will be built on top of it ;-)

> At other side, when you received a string, it may be collected
> somewhere, so on next call to #nextString, it will return the first
> available string in FIFO manner.

There's no doubt about the possibility of sending and receiving things  
through Hydra's channel.

> I don't see a reason in transferring a blocks (compiled bytecode),

Transfer of c-o-n-t-r-o-l is not the same as transfering compiled  
bytecode. HydraSMS doesn't transfer the compiled method, I wrote that  
earlier.

> since you stating, that both images containing very same code

Sure.

> (and if
> not, you can always perform a pervasive doit to make it appear :).

I think the divergence that you attempt here is because I try to abstract  
the nitty gritty details away?

Please: let's not waste our time discussing your asynchronous arguments  
v.s. my synchronous attempt with HydraSMS.

>>
>>  /Klaus
>>
>

On 14/02/2008, Klaus D. Witzel <[hidden email]> wrote:

[waste of time]

>  Please: let's not waste our time discussing your asynchronous arguments
>  v.s. my synchronous attempt with HydraSMS.
>

I'm not trying, what i tried is to understand what you putting in
'passing control' and in 'synchronous'.
Is this means, that you want do following:

- initiator image sends message to another one (a message payload is
not really relevant)
- then waits during another image performs requested operation
- then resuming operations (by receiving notification or return value).


--
Best regards,
Igor Stasenko AKA sig.

On Tue, Feb 12, 2008 at 02:16:25PM -0800, Craig Latta wrote:
I'm not actively involved with either HydraVM or Spoon (not enough hours
in the day), but the synergy seems clear. It would be very interesting
if some of Spoon's features could be available in a HydraVM and vice
versa. Also, any cooperation would expand the audience for both projects
considerably.

Dave

On 14/02/2008, David T. Lewis <[hidden email]> wrote:
With some guidance of Craig, i would try to run Spoon images on HydraVM.
The things i need to know:
- from where i can download latest Spoon image
- where i should put my dirty hands to prepare two Spoon images for
interacting via Hydra channels.

I'm really want to test-drive Spoon. Since last time i tried it, it
was not fascinating at all :)
I was able to run host image, but unable to connect it with child
image because of some bugs in windows version.

>  Dave
>


--
Best regards,
Igor Stasenko AKA sig.

On Thu, 14 Feb 2008 13:26:55 +0100, Igor Stasenko wrote:

> On 14/02/2008, Klaus D. Witzel wrote:
>
> [waste of time]

Who knows :) we'll see :)

>
>>  Please: let's not waste our time discussing your asynchronous arguments
>>  v.s. my synchronous attempt with HydraSMS.
>>
>
> I'm not trying, what i tried is to understand what you putting in
> 'passing control' and in 'synchronous'.
> Is this means, that you want do following:

First we need to add this:

0] if any operation on either side fails, then the other side fails as  
well, with proper notification *whatsoever*.

> - initiator image sends message to another one (a message payload is
> not really relevant)

1] this part has more: initiator asks the other .image to establish a  
response channel. If handshaking fails, then 0]

> - then waits during another image performs requested operation

No, no RPC. This is Smalltalk land, we want no stinkin' RPC :) instead,

2] then the other .image sends back sufficient payload so that the  
initiator side has the "next" block evaluated, and so on with roles  
swapped, back and forth. If something fails, then 0]

No other part of the method is "resumed-to", on neither side. This is  
meant when I write, unit of transfer is Smalltalk's block. Apologies if I  
didn't make *block* clearer.

May I repeat it? it's *blocks* :) Each block is evaluated on "its" side  
(#atHomeDo: #atWorkDo:), same with a block's blocks.

> - then resuming operations (by receiving notification or return value).

If you would have meant RPC by that: no. All that is transferred, is which  
Smalltalk block on which side is to be evaluated next (recalling that both  
sides are in the same compiled method and let's forget about  
parameter/argument passing for a moment). And Smalltalkers can put  
*everything* inside their blocks :)

-------------

Both sides intentionally return self (but of course they can persist  
parameters/arguments and do with them whatever they please).

/Klaus

P.S. bbl

On 14/02/2008, Klaus D. Witzel <[hidden email]> wrote:
Of course.
Thanks GOD! :)

Still is little bit confusing. Especially when you saying 'block's blocks' :)
Can you elaborate little more (by suggesting how #atHomeDo: #atWorkDo:
should look like)?

HydraSMS>>atHomeDo: aBlock
 "Nothing sophisticated, just evaluate a block"
 homeBlock := aBlock.
 ^ aBlock value

HydraSMS>>atWorkDo: aBlock
  "Send message to other side to evaluate it's Home block"
  channel reliablySend: (self encodeEvaluateAtHomeDoPacket)
 "what to do with aBlock here?"

HydraSMS>>nextStringPut: aString
 "Send string to be used by #nextString at remote side"
  channel reliablySend: (self encodeNextStringPutPacket: aString)


HydraSMS>>decodePacket: aPacket
"this method executed when you receiving data from channel"
 aPacket isEvaluateAtHomeDo ifTrue: [ self atHomeDo: homeBlock ].
 aPacket isNextStringPut ifTrue: [ nextString := aPacket extractString ].

HydraSMS>>nextString
"simple and stupid :)"
 ^ nextString


--
Best regards,
Igor Stasenko AKA sig.

On Thu, 14 Feb 2008 15:02:57 +0100, Igor Stasenko wrote:

> On 14/02/2008, Klaus D. Witzel wrote:
>> On Thu, 14 Feb 2008 13:26:55 +0100, Igor Stasenko wrote:
...
:)

A block's block is created from bytecodes for which (thisContext  
isMemberOf: BlockContext) holds; let's call them nested blocks.
In the same sense is a non-block's block created from bytecodes for which  
(thisContext isMemberOf: MethodContext) holds.

Both, #atHomeDo:'s and #atWorkDo:'s, blocks can have nested blocks :)

> Can you elaborate little more (by suggesting how #atHomeDo: #atWorkDo:
> should look like)?

Worth to note: instances of HydraSMS (aka NetworkContext) know in which  
context they had been created. At creation time they insert a context for  
#ifCurtailed* and another for fall-through-return into their process'  
stack. But they can also be detached from their initiating process; I have  
subinstances of Model (Browser*) for which that is important and whose  
HydraSMS (aka NetworkContext) instance lives longer than the instance  
creating process.

> HydraSMS>>atHomeDo: aBlock
>  "Nothing sophisticated, just evaluate a block"

This is indeed a *very* sophisticated method; #atHomeDo: can be sent by  
either side, from a nested block or from the method initiated at the other  
side. Recall that we provide support for the-same-method-in-both-images;  
this might incur obstacles at HydraSMS development-time but they are  
offset by very-easy-to-use at library-use-time (only less burden-as-user  
can be more fun ;-)

>  homeBlock := aBlock.
>  ^ aBlock value

Yes, something like this is done on the (respective) other side, "just"  
after gathering+transfering sufficient payload data such that

  ((BlockContext newForMethod: imaginaryContext method)
   home: imaginaryContext
   startpc: startPC
   nargs: 0) value

can be performed (after the corresponding imaginaryContext has been  
determined by the respective side).

>
> HydraSMS>>atWorkDo: aBlock
>   "Send message to other side to evaluate it's Home block"
>   channel reliablySend: (self encodeEvaluateAtHomeDoPacket)
>  "what to do with aBlock here?"

Do the same with it as #atHomeDo: does, except that roles are swapped and  
other out-of-sequence-errors are possible.

> HydraSMS>>nextStringPut: aString
>  "Send string to be used by #nextString at remote side"
>   channel reliablySend: (self encodeNextStringPutPacket: aString)

Precisely, except that because of your Hydra channels implementation (no  
offense), you have two instance variables in HydraSMS instances which  
cannot both be named 'channel' ;-)

>
> HydraSMS>>decodePacket: aPacket
> "this method executed when you receiving data from channel"
>  aPacket isEvaluateAtHomeDo ifTrue: [ self atHomeDo: homeBlock ].
>  aPacket isNextStringPut ifTrue: [ nextString := aPacket extractString ].

Precisely, except that Smalltalk developers believe that we provide them a  
stream (they see #next*) and so you have to have a FIFO mechanism like the  
one you mentioned in a previous message.

>
> HydraSMS>>nextString
> "simple and stupid :)"
>  ^ nextString
  "still simple but not *that* stupid"
  ^ receivedDataBlocks removeFirst

On 14/02/2008, Klaus D. Witzel <[hidden email]> wrote:
What can i say? Only that you are real guru in metaprogramming, and
i'm lacking in expertise to deal with this task.
I never got so deep under the hood to have need in creating a block
contexts, or using #ifCurtailed: methods.

Maybe it would be less confusing for me, from a start, if methods
names would be #remotelyDo: , #locallyDo:..
I'm working at home, about a 2 years till now, so there is no real
difference for me between #atHomeDo: and #atWorkDo: :)

In general, i finally understood your idea. But due to my lack of
practice (and knowledge) in uber-metaprogramming i don't think i will
be capable do that without help :)

What i can currently do, is provide an implementation of reliable
bidirectional communication. The rest, i hope you can do yourself.
Orr.. if you prototype most of meta-crunching stuff and point me out
what places i should add to make it live.

--
Best regards,
Igor Stasenko AKA sig.

Hi Igor--

 > With some guidance of Craig, i would try to run Spoon images on
 > HydraVM.
 >
 > The things i need to know:
 > - from where i can download latest Spoon image

      The latest release is always at [1], it's probably the one you
already have. Making new releases is very labor-intensive currently; my
available time (and the level of interest during this bootstrapping
phase, where the user often needs to do some debugging to make progress)
has made them infrequent so far.

 > - where i should put my dirty hands to prepare two Spoon images for
 > interacting via Hydra channels.

      Let's discuss this on the #spoon IRC channel at irc.freenode.net.
I'm there on Mondays from 1800-0300 GMT (the rest of the week I can only
check the conversation briefly once in a while).

 > I was able to run host image, but unable to connect it with child
 > image because of some bugs in windows version.

      What happened? (Followups should probably go to the Spoon mailing
list.)


      thanks,

-C

[1] http://netjam.org/spoon/releases/current

--
Craig Latta
improvisational musical informaticist
www.netjam.org
Smalltalkers do: [:it | All with: Class, (And love: it)]

On Thu, 14 Feb 2008 19:19:33 +0100, Igor Stasenko wrote:

...
> What can i say? Only that you are real guru in metaprogramming, and
> i'm lacking in expertise to deal with this task.

Thank you, but that's too much for this grey-bearded OM :) Don't forget,  
you made Hydra going :)

> I never got so deep under the hood to have need in creating a block
> contexts, or using #ifCurtailed: methods.

#ifCurtailed: is only needed if you cannot do ([]ensure: []), etc, for  
whatever reason, with HydraSMS. Will try to avoid them if (if!) possible.

...

> Maybe it would be less confusing for me, from a start, if methods
> names would be #remotelyDo: , #locallyDo:..

Now in Huston there's a problem: both #atHomeDo: and #atWorkDo: never  
result in #locallyDo: ...

> I'm working at home, about a 2 years till now, so there is no real
> difference for me between #atHomeDo: and #atWorkDo: :)

LOL :)

> In general, i finally understood your idea. But due to my lack of
> practice (and knowledge) in uber-metaprogramming i don't think i will
> be capable do that without help :)

We can find a meeting point somewhere between us, I'm sure we can :) You  
know why? it's Smalltalk :)

> What i can currently do, is provide an implementation of reliable
> bidirectional communication. The rest, i hope you can do yourself.

If I had a wish free: when you begin with reliable bidirectional  
communication please use, for the public part, the vocabulary which is  
present in Socket's message categories #open, #queries, #receiving,  
#sending and #waiting. Please.

See, SocketStream is a user of Socket, so G"oran would have a wonderful  
SocketStream work-free weekend :-D

> Orr.. if you prototype most of meta-crunching stuff and point me out
> what places i should add to make it live.

I will meet you there, could start by populating subclasses which extend  
your Hydra channel classes, like

  SocketLikeHydraReceivingChannel
  SocketLikeHydraSendingChannel

and then we can see how far this gets us up to a meeting point :)

/Klaus

P.S. please now switch to offline email, unless you don't fear that we get  
moderated :)

"Klaus D. Witzel" <[hidden email]> writes:

> P.S. please now switch to offline email, unless you don't fear that we get
> moderated :)

I've been following this thread with a great deal of interest! If it's
convenient for you, I'd love to be able to carry on reading this thread,
here on squeak-dev!

frank

Hi Frank,

on Thu, 14 Feb 2008 21:06:36 +0100, you wrote:

> "Klaus D. Witzel" writes:
>
>> P.S. please now switch to offline email, unless you don't fear that we  
>> get moderated :)
>
> I've been following this thread with a great deal of interest! If it's
> convenient for you, I'd love to be able to carry on reading this thread,
> here on squeak-dev!

Message understood :)

/Klaus

> frank
>
>
>

On 14/02/2008, Klaus D. Witzel <[hidden email]> wrote: Yes, i read a comments in this method before, and it was exactly
that you just wrote: avoid at all costs :)
No problem, but it needs a time to learn everything (to be precise -
infinite time) :)
channels don't having too much methods requiring that many category types.
As you may noticed they are very basic wrap around corresponding primitives.

You can't have two kinds of 'SocketLike' channels.
If you talking about stream sockets, which is reliable kind of
communication - they are bidirectional by default. So, there can be
only a single class 'SocketLikeHydraChannel'.

If you talking about UDP, or raw sockets, then channels is already
providing comparable functionality (and a little on top).
But there are differencies, that why i chosen (maybe inconvenient)
names of methods.
The intent was to indicate that it's not a sockets (to not put you in
false direction that you can operate with channels as with sockets).
Also, note that is not really a final design, just a preview (message
names and even primitives can be changed, and it's a low-level layer,
so what the deal) :)
It will need some time to clean things a bit and make it look like a
real API for public use.

>
>  /Klaus
>
>
>  P.S. please now switch to offline email, unless you don't fear that we get
>  moderated :)
>



--
Best regards,
Igor Stasenko AKA sig.

Oh, i wasn't attentive when writing this. I meant for current
implementation only.
Of course, a bidirectional protocol will look as much as close to sockets :)

--
Best regards,
Igor Stasenko AKA sig.

On Thu, 14 Feb 2008 22:52:41 +0100, Igor Stasenko wrote:

... ah, and I was just about to think that it wasn't you who wrote ;-)

> Of course, a bidirectional protocol will look as much as close to  
> sockets :)

NB, some fruits with HydraSMS and its same-method principle, a #doit from  
a workspace, just out of the blue and neither any other preparation nor  
other knowledge:

  | sms | sms := HydraSMS doitAt: 2.
  sms atHomeDo: [Transcript cr; show: 'Hydra .image #2 is up and running']

, for which #doitAt: automagically compiles the very-same method in the  
other image :)

/Klaus

On 15/02/2008, Klaus D. Witzel <[hidden email]> wrote: Be warned, that using index instead of handle to identify interpreter
is not a good way.

All primitives are handle-oriented, an #at: methods simply doing:
HydraVM interpreters at: n.

There is a danger, that when in future i add code to shutdown images,
then indexed can be changed, and thus, code based on indexes might
fail, because
if you having 3 interpreters running:

#(x y z)

and you shutting down 2nd one,
then 'HydraVM interpreters' will return array with 2 handles:
#(x z) , so #at: 3 will lead to error, and #at: 2 will return different handle.

>  /Klaus
>
>
>


--
Best regards,
Igor Stasenko AKA sig.

On Fri, 15 Feb 2008 09:47:54 +0100, Igor Stasenko wrote:

> On 15/02/2008, Klaus D. Witzel wrote:
>> On Thu, 14 Feb 2008 22:52:41 +0100, Igor Stasenko wrote:
...
Well, that's exactly what the higher levels are good for: tell the user  
what's up, without demanding from him to know anything about the  
internals. Of course I cannot post the above example so that it rises all  
possible exceptions, so at the moment you have to engage imagination and  
*think* that proper notification is given (NetworkContext does check and  
report whatever problem it finds).

> All primitives are handle-oriented, an #at: methods simply doing:
> HydraVM interpreters at: n.

We cannot really want the user to type in values for your  
interpreterHandle, unless they are used to typing MS$ registry keys and  
*demand* that from us ;-)

> There is a danger, that when in future i add code to shutdown images,
> then indexed can be changed, and thus, code based on indexes might
> fail, because if you having 3 interpreters running:
>
> #(x y z)
>
> and you shutting down 2nd one,
> then 'HydraVM interpreters' will return array with 2 handles:
> #(x z) , so #at: 3 will lead to error,

What error, Smalltalk's Error instance? Image crash? Interpreter crash?  
Executable crash? OS crash? Market melt-down? Blackout of the Internet?

And, BTW, why would you make that undeterministic? I'm used to multi-CPU  
machines for decades and we always index the physical processor (also  
virtual processor) by its constant index. I could send you an indexing  
schema in case you need one.

> and #at: 2 will return different handle.

I don't see much danger with different handles, since I will be carefully  
reading your future Hydra release notes :)

>>  /Klaus
>>
>>
>>
>
>

On 15/02/2008, Klaus D. Witzel <[hidden email]> wrote: Of course not, that's why i added #at: in different places. I'm too
lazy myself for typing:

HydaVM doit: '...' on: (HydraVM mainInterpreter)

when i simply can:

HydaVM doit: '...' at: 1
a simple index out of bounds error.

>  And, BTW, why would you make that undeterministic? I'm used to multi-CPU
>  machines for decades and we always index the physical processor (also
>  virtual processor) by its constant index. I could send you an indexing
>  schema in case you need one.
>

Remember, that interpreters now is the objects, regardless that most
of it's functionality are sitting in VM. But there actually was idea
to make them behave much like real ST objects:
 you can create new instance of HydraInterpreter, load image in it ,
run commands e.t.c.
and then , when you no longer need it, it will simply be freed once GC
will do it's work :)
There can be a another, useful macro operations on interpreters, for
instance creating a copy.

So, you can do:
 myCopy := interpreter copy.

to create new instance of interpreter with exactly same state & object
memory as original.
In that way, it can be useful for forking complex computations without
need in initializing all forked images personally. I can take single
interpreter, load all i needed to it's object memory, simply preparing
it for something dramatical :)
and then fork as many new interpreters as i need to begin rock&roll.
Of course, you may do something similar by snapshotting image, and
then starting it couple of times. But it's much less effective and
forcing to pass startup phase when loading, which you may not want.

There could be another useful things, like suspending, profiling,
inspecting state e.t.c.

I can actually kill the native thread, but keep interpreter's state
and object memory. And then start debugging, using special
interpretStep() function which will evaluate a single bytecode per
call.

For this reasons, i'd like to operate with interpreters using handles
instead of indexes.
This, in particular, can guarantee that if you save your image which
contains references to handles, they will render invalid during next
squeak session. While indexes is not.

>  > and #at: 2 will return different handle.
>
>
> I don't see much danger with different handles, since I will be carefully
>  reading your future Hydra release notes :)
>
>  >>  /Klaus
>  >>


--
Best regards,
Igor Stasenko AKA sig.

 
> Of course not, that's why i added #at: in different places. I'm too
> lazy myself for typing:
>
> HydaVM doit: '...' on: (HydraVM mainInterpreter)
>
> when i simply can:
>
> HydaVM doit: '...' at: 1
>  
But what is this interface doing on the class side at all. Here you have
instances of remote image!

(HydraVM at: 1) doit: '...'.

HydraVM all do: [ :each | each doit: '...' ].

Keith