HydraTools and minimal images
Locked
123
123
 
|
Hi -
Some of you may have been following Igors efforts on HydraVM [1] which is an attempt to enable the use of multi-core CPUs within a single Squeak VM by using multiple images. Having finally had the time to play around with it on the weekend I was *amazed* how usable it already is in terms of speed (10-20% slowdown) as well as productivity. Since the basic development process is in place (load alternative images, send commands, save the result) I was thinking about the next steps and it occurred to me that having a set of tools that can operate on another image would be tremendously useful. Not only for development inside Hydra but also for some of the people out there trying to create smaller images. With a set of HydraTools you should be able to for example, rip out *all* of the UIs and tools that exist in an image, fix up the result and be able to try loading the UI framework from first principles. So my question is basically this: Is there a tool set (browser, senders, implementors, maybe debugger) that has "enough" of an abstraction about its model that it would be easy to fit in the Hydra channel protocol to make them work? E.g., there needs to be a well-understood protocol that is being used to access the underlying classes and that can be put onto a HydraVM channel to communicate with another image. If anyone knows about a good starting point that would be great. [1]http://lists.squeakfoundation.org/pipermail/squeak-dev/2008-February/125150.html Cheers, - Andreas |
Re: HydraTools and minimal images
|
|
On Feb 12, 2008, at 11:09 AM, Andreas Raab wrote: > I was *amazed* how usable it already is in terms of speed (10-20% > slowdown) as well as productivity. I assume you meant speedup as opposed to slowdown? ;-) - Brian |
Re: HydraTools and minimal images
|
|
In reply to this post by Andreas.Raab
> So my question is basically this: Is there a tool set (browser, senders,
> implementors, maybe debugger) that has "enough" of an abstraction about > its model that it would be easy to fit in the Hydra channel protocol to > make them work? OmniBrowser has all this. GemStone is using that do provide a development environment on their headless VMs. The OmniBrowser model runs in GemStone (they ported the code), and the UI runs in Squeak. They have system browser, hierarchy browser, sender browser, implementor browser, variable browser, debugger, inspector, workspace, monticello, etc. Lukas -- Lukas Renggli http://www.lukas-renggli.ch |
Re: HydraTools and minimal images
|
|
In reply to this post by Brian Brown-2
Brian Brown wrote:
> > On Feb 12, 2008, at 11:09 AM, Andreas Raab wrote: > >> I was *amazed* how usable it already is in terms of speed (10-20% >> slowdown) as well as productivity. > > > I assume you meant speedup as opposed to slowdown? No, I meant slowdown. HydraVM is a little slower than your average Squeak VM because it needs to manage the multi-threaded state. But if you are capable of making effective use of it in a multicore environment you obviously get more than you pay for by multiplying it for the number of cores. Cheers, - Andreas |
|
|
In reply to this post by Andreas.Raab
Hi Andreas-- > ...I was *amazed* how usable it already is in terms of speed (10-20% > slowdown) as well as productivity. Great! > Since the basic development process is in place (load alternative > images, send commands, save the result) I was thinking about the next > steps and it occurred to me that having a set of tools that can > operate on another image would be tremendously useful. Not only for > development inside Hydra but also for some of the people out there > trying to create smaller images. > > With a set of HydraTools you should be able to for example, rip out > *all* of the UIs and tools that exist in an image, fix up the result > and be able to try loading the UI framework from first principles. I've done that with Spoon, with separate VMs communicating over a proxy system that uses sockets. It would be interesting to extend the proxy system to use Hydra's channels (some form of inter-thread communication I assume, I haven't looked at the Hydra implementation yet). > So my question is basically this: Is there a tool set (browser, > senders, implementors, maybe debugger) that has "enough" of an > abstraction about its model that it would be easy to fit in the Hydra > channel protocol to make them work? My approach with Spoon was to just have a completely transparent proxy system, so that all the normal tools work without special support. I've had all the tools you mention working remotely for a few years now. I did do some magic to support debugging processes which have a combination of local and remote contexts, but it was surprisingly little. Anyway, the proxy system isn't especially tied to sockets or any particular transport, so I think this is doable. thanks, -C -- Craig Latta improvisational musical informaticist www.netjam.org Smalltalkers do: [:it | All with: Class, (And love: it)] |
Re: HydraTools and minimal images
|
|
In reply to this post by Lukas Renggli
Lukas Renggli wrote:
>> So my question is basically this: Is there a tool set (browser, senders, >> implementors, maybe debugger) that has "enough" of an abstraction about >> its model that it would be easy to fit in the Hydra channel protocol to >> make them work? > > OmniBrowser has all this. GemStone is using that do provide a > development environment on their headless VMs. The OmniBrowser model > runs in GemStone (they ported the code), and the UI runs in Squeak. This sounds exactly like what I was looking for. Is their stuff available to look at for learning how to integrate such a mechanism? I'm not interested in any of their proprietary IP just in learning how you hook this together in OB. Any other examples? > They have system browser, hierarchy browser, sender browser, > implementor browser, variable browser, debugger, inspector, workspace, > monticello, etc. Wow. That's pretty cool. Way more than I expected. Cheers, - Andreas |
Re: HydraTools and minimal images
|
|
On 12-Feb-08, at 8:03 PM, Andreas Raab wrote: > This sounds exactly like what I was looking for. Is their stuff > available to look at for learning how to integrate such a mechanism? > I'm not interested in any of their proprietary IP just in learning > how you hook this together in OB. Any other examples? I believe they use FFI to call out to a C library (called "gci") that implements the low-level communications with Gemstone. The code is publicly available here: http://seaside.gemstone.com/ss/ In particular, see the GemStone Tools project. Another thing to look at is OB-Web. It's less "remote" than the Gemstone or Spoon approach, in that the class browser runs in-image rather than remotely. What *is* remote are the widgets. They're implemented in Javascript and run in Safari or Firefox, communicating with the image via AJAX. It's available from the repository below, but has some dependencies - let me know if you want to play with it. http://source.wiresong.ca/ob Colin |
Re: HydraTools and minimal images
|
|
My personal feelings, when i first implemented a HydraChannels and
tried to play with them, was is how little it is, and yet very powerful :) - a setup code is very little - a data handling code is just before your eyes. For instance a #doit channel implementation consists from few lines of code: (HydraChannel listen: #doIt onReceive: [:data | [ Compiler evaluate: data asString notifying: nil logged: true ] fork ]) resumeAtStartup: true .. once you execute this code, your image now listening #doit channel. And by having doits, you can do virtually anything with it. With channels, you don't need to care about many issues, which are existing with non-reliable communications, like: out-of-order packets arrival, packet slicing, data loss, and many other things which you should care when using sockets, like resolving, buffering e.t.c.. I wasn't expected that HydraVM hides a great potential for playing with images (there was no real intents to make it). Since you don't need to care, if your image is having UI, and all what you need is to keep it listening a #doit channel, with which you can fix virtually anything, if something goes wrong. During my experiments i lost one of my test images (UI stops responding), but when loaded as secondary image, i was still able to send doits to it, thus having a chance to restore things back to normal :) Andreas is right, a next logical step would be to make some tools with which you can explore image contents and debug code in it. But before going deep, i would like to get some feedback on channels implementation. There is some very interesting alternatives, which should be considered. Currently a send mechanism makes following: - allocate memory needed for buffer - copy bytes from given bytearray into allocated buffer - create a 'send' event, and finally put it into receiver's event queue. On receiver's side: - dequeue event - find a corresponding channel - if channel is found, put event into channel's queue , if channel's input queue is full, ignore event - if channel not found, ignore event - by ignoring, its simply deallocates memory of buffer When ST finally gets data from channel: - dequeue packet from channel receiving queue - copy buffer bytes in newly allocated bytearray - deallocate packet buffer. --- Alternatives: - preallocated send queue with fixed packet length. This will eliminate the need in allocating/deallocating memory for packets, making system work a bit faster. - adding some feedback mechanism, when incoming packet can't be delivered (when channel input queue is full, or simply there is no channel with given credentials). I'm not sure, if this is really needed to be implemented by VM. An efficient error/feedback mechanisms can be implemented very easily at language side. - for an ultra fast communication, put packets directly into channel's queue, bypassing interpreter event queue. This will require some thought , how to implement it without endangering VM stability. In this case, a packets can be handled right after few processor cycles passed when channel received packet. You simply can't have such speed when using sockets :) -- Best regards, Igor Stasenko AKA sig. |
Re: HydraTools and minimal images
|
|
Hi Igor,
on Wed, 13 Feb 2008 09:35:09 +0100, you wrote: > My personal feelings, when i first implemented a HydraChannels and > tried to play with them, was is how little it is, and yet very > powerful :) > - a setup code is very little > - a data handling code is just before your eyes. Yes, HydraChannels look easy and straight forward :) ... > And by having doits, you can do virtually anything with it. > ... > Andreas is right, a next logical step would be to make some tools with > which you can explore image contents and debug code in it. I think that at the time being, debugger would be of higher priority than browser, since filing in source code from filedOut material is already solved by using #doit channel :) > But before going deep, i would like to get some feedback on channels > implementation. If I had a wish free then I'd say that Hydra's channels could provide the [a sensible part of] existing vocabulary of Socket (Socket's messages categories #open, #queries, #receiving, #sending and #waiting). This would make porting Socket-based applications easier and can leverage existing knowledge about Sockets. > There is some very interesting alternatives, which should be considered. ... > --- > Alternatives: ... > - adding some feedback mechanism, when incoming packet can't be > delivered (when channel input queue is full, or simply there is no > channel with given credentials). Considering a port one of my projects which currently is using Sockets, what I'm missing is something like #waitForDataFor:ifClosed:ifTimedOut: Is something like that possible with the Hydra channels primitives? > I'm not sure, if this is really needed to be implemented by VM. An > efficient error/feedback mechanisms can be implemented very easily at > language side. It wouldn't need more/other error/feedback than what Socket provides today, IMHO. BTW: what's happening now if the second .image does no longer respond to the first .image, how can/does the other .image get known to this? /Klaus |
Re: HydraTools and minimal images
|
|
On 13/02/2008, Klaus D. Witzel <[hidden email]> wrote:
> Hi Igor, > > on Wed, 13 Feb 2008 09:35:09 +0100, you wrote: > > > My personal feelings, when i first implemented a HydraChannels and > > tried to play with them, was is how little it is, and yet very > > powerful :) > > - a setup code is very little > > - a data handling code is just before your eyes. > > Yes, HydraChannels look easy and straight forward :) > > ... > > And by having doits, you can do virtually anything with it. > > > ... > > Andreas is right, a next logical step would be to make some tools with > > which you can explore image contents and debug code in it. > > I think that at the time being, debugger would be of higher priority than > browser, since filing in source code from filedOut material is already > solved by using #doit channel :) > > > But before going deep, i would like to get some feedback on channels > > implementation. > > If I had a wish free then I'd say that Hydra's channels could provide the > [a sensible part of] existing vocabulary of Socket (Socket's messages > categories #open, #queries, #receiving, #sending and #waiting). > There is, a waiting (hidden from your eyes). You can check the HydraReceivingChannel>>startListening: aBlock VM signals a semaphore, when data arrives on channel, so a process which is waiting on that semaphore is then awaken and can do anything he needs to handle incoming packet. Actually, you can omit using semaphore, and just pull data from channel at any time you need. The readData primitive works pretty straightforward: If there is data waiting on channel, you'll receive a bytearray, if not - you just receive a nil in response, indicating that channel input queue is empty. So, you can implement own listening scheme, which polls channel for data periodically, and if there is no data received after supplied timeout, evaluate a onTimeOut handler.. > This would make porting Socket-based applications easier and can leverage > existing knowledge about Sockets. > > > There is some very interesting alternatives, which should be considered. > ... > > --- > > Alternatives: > ... > > - adding some feedback mechanism, when incoming packet can't be > > delivered (when channel input queue is full, or simply there is no > > channel with given credentials). > > Considering a port one of my projects which currently is using Sockets, > what I'm missing is something like #waitForDataFor:ifClosed:ifTimedOut: > A channels is unidirectional. You sending data and have no feedback. It works in same way as UDP sockets. But on top of UDP you can always build more sophisticated layers (as bidirectional streams), where you can introduce behavior like #waitForDataFor:ifClosed:ifTimedOut:. > Is something like that possible with the Hydra channels primitives? ... and there is no need to make it at VM level. It can be easily done at language level. > > > I'm not sure, if this is really needed to be implemented by VM. An > > efficient error/feedback mechanisms can be implemented very easily at > > language side. > > It wouldn't need more/other error/feedback than what Socket provides > today, IMHO. > > BTW: what's happening now if the second .image does no longer respond to > the first .image, how can/does the other .image get known to this? Again, there is no 'response' in any means. Channels just listening for data. You can send data to channel, but you don't getting any feedback about if it was delivered or handled. To build a system with feedback all you need is to setup two channels (one for each image) and create a feedback protocol, like, when channel receives a message, it sends a message back, that he is actually received it :) As a small exercise of bidirectional communication, you can check HydraPing class. > > /Klaus > > -- Best regards, Igor Stasenko AKA sig. |
Re: HydraTools and minimal images
|
|
Hi!
"Igor Stasenko" <[hidden email]> wrote: > A channels is unidirectional. You sending data and have no feedback. > It works in same way as UDP sockets. > But on top of UDP you can always build more sophisticated layers (as > bidirectional streams), where you can introduce behavior like > #waitForDataFor:ifClosed:ifTimedOut:. I presume you could take SocketStream and rewrite it to use Hydra channels. Would be a fun thing to do a rainy afternoon. regards, Göran |
Re: HydraTools and minimal images
|
|
In reply to this post by Igor Stasenko
Some more words about channels:
channel primitives was designed to provide enough facilities to make inter-image communication. There was no intent in building a bidirectional, with delivery notifications channels, because of speed reasons, and because some communication schemes would not require a guaranteed delivery at all. As analogy, you may consider channels as a low-level network transport, on top of which you place a TCP/IP layer or, of course, anything what you want. Or, you may consider a real world example: - when someone telling something to you, he sending a message. But he can't know if you heard it, understand it and even if so, willing to say something in response. He can _know_ that you received his message only if you say something in response, or do something, which will confirm that message is delivered and understood well :) -- Best regards, Igor Stasenko AKA sig. |
Re: HydraTools and minimal images
|
|
In reply to this post by Igor Stasenko
Hi Igor,
on Wed, 13 Feb 2008 13:24:05 +0100, you wrote: > On 13/02/2008, Klaus D. Witzel wrote: >> Hi Igor, >> >> on Wed, 13 Feb 2008 09:35:09 +0100, you wrote: >> ... >> > But before going deep, i would like to get some feedback on channels >> > implementation. >> >> If I had a wish free then I'd say that Hydra's channels could provide >> the >> [a sensible part of] existing vocabulary of Socket (Socket's messages >> categories #open, #queries, #receiving, #sending and #waiting). >> > > There is, a waiting (hidden from your eyes). > You can check the HydraReceivingChannel>>startListening: aBlock > > VM signals a semaphore, when data arrives on channel, so a process > which is waiting on that semaphore is then awaken and can do anything > he needs to handle incoming packet. If these you lines are related to my #waitForDataFor:ifClosed:ifTimedOut: question (I assume that): a] the timeout can occur while waiting (listening) b] the close can occur while waiting (listening) c] data can arrive while waiting Case c] is easy with the #startListening: but how about capturing the other two cases? > Actually, you can omit using semaphore, and just pull data from > channel at any time you need. It would have to know that there's no data to pull (and why that is so), that is what #waitForDataFor:ifClosed:ifTimedOut: is about (besides pulling data). > The readData primitive works pretty straightforward: If there is data > waiting on channel, you'll receive a bytearray, if not - you just > receive a nil in response, indicating that channel input queue is > empty. > So, you can implement own listening scheme, which polls channel for > data periodically, and if there is no data received after supplied > timeout, evaluate a onTimeOut handler.. IC. >> This would make porting Socket-based applications easier and can >> leverage >> existing knowledge about Sockets. >> >> > There is some very interesting alternatives, which should be >> considered. >> ... >> > --- >> > Alternatives: >> ... >> > - adding some feedback mechanism, when incoming packet can't be >> > delivered (when channel input queue is full, or simply there is no >> > channel with given credentials). >> >> Considering a port one of my projects which currently is using Sockets, >> what I'm missing is something like #waitForDataFor:ifClosed:ifTimedOut: >> > > A channels is unidirectional. You sending data and have no feedback. > It works in same way as UDP sockets. > But on top of UDP you can always build more sophisticated layers (as > bidirectional streams), where you can introduce behavior like > #waitForDataFor:ifClosed:ifTimedOut:. > >> Is something like that possible with the Hydra channels primitives? > > ... and there is no need to make it at VM level. It can be easily done > at language level. > >> >> > I'm not sure, if this is really needed to be implemented by VM. An >> > efficient error/feedback mechanisms can be implemented very easily at >> > language side. >> >> It wouldn't need more/other error/feedback than what Socket provides >> today, IMHO. >> >> BTW: what's happening now if the second .image does no longer respond to >> the first .image, how can/does the other .image get known to this? > > Again, there is no 'response' in any means. So I have to teach the app's users, "there's no response" ;-) > Channels just listening > for data. You can send data to channel, but you don't getting any > feedback about if it was delivered or handled. More seriously, I understand that in order to save the whole app from potentially "freezing on no response", some timeout mechanism needs to be added. But timeout alone cannot find out that the other .image has gone away. Is there something you would suggest for checking the other .image is still alive (or not alive)? NB: this check is independent of using Sockets or Hydra channels between any pair of Hydra .images, but it looks like there can be easier control with Hydra because everything is on the same machine. > To build a system with feedback all you need is to setup two channels > (one for each image) and create a feedback protocol, like, when > channel receives a message, it sends a message back, that he is > actually received it :) Sure, DIY :) > As a small exercise of bidirectional communication, you can check > HydraPing class. Sure. >> /Klaus >> >> > |
Re: HydraTools and minimal images
|
|
On 13/02/2008, Klaus D. Witzel <[hidden email]> wrote:
> Hi Igor, > > on Wed, 13 Feb 2008 13:24:05 +0100, you wrote: > > > On 13/02/2008, Klaus D. Witzel wrote: > >> Hi Igor, > >> > >> on Wed, 13 Feb 2008 09:35:09 +0100, you wrote: > >> > ... > >> > But before going deep, i would like to get some feedback on channels > >> > implementation. > >> > >> If I had a wish free then I'd say that Hydra's channels could provide > >> the > >> [a sensible part of] existing vocabulary of Socket (Socket's messages > >> categories #open, #queries, #receiving, #sending and #waiting). > >> > > > > There is, a waiting (hidden from your eyes). > > You can check the HydraReceivingChannel>>startListening: aBlock > > > > VM signals a semaphore, when data arrives on channel, so a process > > which is waiting on that semaphore is then awaken and can do anything > > he needs to handle incoming packet. > > If these you lines are related to my #waitForDataFor:ifClosed:ifTimedOut: > question (I assume that): > > a] the timeout can occur while waiting (listening) > b] the close can occur while waiting (listening) > c] data can arrive while waiting > > Case c] is easy with the #startListening: but how about capturing the > other two cases? > There is no 'close' action, there is only 'stop listening' action. So, case b is simply impossible with such design. Case a] can be simply implemented if you place #waitTimeoutMsecs: instead of just #wait. Everything in our hands :) > > Actually, you can omit using semaphore, and just pull data from > > channel at any time you need. > > It would have to know that there's no data to pull (and why that is so), > that is what #waitForDataFor:ifClosed:ifTimedOut: is about (besides > pulling data). > > > The readData primitive works pretty straightforward: If there is data > > waiting on channel, you'll receive a bytearray, if not - you just > > receive a nil in response, indicating that channel input queue is > > empty. > > So, you can implement own listening scheme, which polls channel for > > data periodically, and if there is no data received after supplied > > timeout, evaluate a onTimeOut handler.. > > IC. > > >> This would make porting Socket-based applications easier and can > >> leverage > >> existing knowledge about Sockets. > >> > >> > There is some very interesting alternatives, which should be > >> considered. > >> ... > >> > --- > >> > Alternatives: > >> ... > >> > - adding some feedback mechanism, when incoming packet can't be > >> > delivered (when channel input queue is full, or simply there is no > >> > channel with given credentials). > >> > >> Considering a port one of my projects which currently is using Sockets, > >> what I'm missing is something like #waitForDataFor:ifClosed:ifTimedOut: > >> > > > > A channels is unidirectional. You sending data and have no feedback. > > It works in same way as UDP sockets. > > But on top of UDP you can always build more sophisticated layers (as > > bidirectional streams), where you can introduce behavior like > > #waitForDataFor:ifClosed:ifTimedOut:. > > > >> Is something like that possible with the Hydra channels primitives? > > > > ... and there is no need to make it at VM level. It can be easily done > > at language level. > > > >> > >> > I'm not sure, if this is really needed to be implemented by VM. An > >> > efficient error/feedback mechanisms can be implemented very easily at > >> > language side. > >> > >> It wouldn't need more/other error/feedback than what Socket provides > >> today, IMHO. > >> > >> BTW: what's happening now if the second .image does no longer respond to > >> the first .image, how can/does the other .image get known to this? > > > > Again, there is no 'response' in any means. > > So I have to teach the app's users, "there's no response" ;-) Well, i assume there is developer standing between low-level APIs and application, who makes everything he needs to eliminate the need in teaching users about anything :) I would like to hear, is there enough functionality in channel _primitives_ (not classes which currently using them), which would allow you to build more sophisticated communication frameworks. As far, as i can see, from cases you given, there is no need to add/change anything to VM's primitives. > > > Channels just listening > > for data. You can send data to channel, but you don't getting any > > feedback about if it was delivered or handled. > > More seriously, I understand that in order to save the whole app from > potentially "freezing on no response", some timeout mechanism needs to be > added. But timeout alone cannot find out that the other .image has gone > away. Is there something you would suggest for checking the other .image > is still alive (or not alive)? > > NB: this check is independent of using Sockets or Hydra channels between > any pair of Hydra .images, but it looks like there can be easier control > with Hydra because everything is on the same machine. > You can check: - if there is still an interpreter in VM ( HydraVM>>#isValidInterpreterInstance:) - if there is still a channel, listening at given interpreter instance. You may check, it by using discover primitive, and check the newly received id and magic and compare them with previously received. A channel is identified by a pair: interpreter handle and name (by analogy as IP address and port number) A HydraSendingChannel using #primitiveDiscoverChannelIn: intr name: aname to get channel's id and magic numbers. Once id and magic are received, it can use them in primitiveSendDataTo: intr channelId: anId magicNum: aMagic data: aData to send data to channel. If that primitive fails, it means, that given interpreter handle is not valid anymore. Or, aData object is not a bytearray. Note: at this stage it's not checking that given channel exists! So, if you continue sending data to channel with bad id/magic but valid interpreter handle, it will be silently accepted. This is the price we pay in having multiple interpreters running in parallel. I'm using id/magic instead of just name to not search and compare strings in channel list every time you sending data. This is done simply for speed reasons. Also, magic number is unique for all channels instances in image. This means, that if you stop listening existing channel, and then start listening new even with same name, it will receive different magic number, which will prevent it from receiving any packets addressed to previous one. Yes, any senders that was targeted to send data to closed channel are not notified automatically upon channel close. But look at bright side: nothing stops you from implementing such notification :) One could create a #control channel (which will be a part of his reliable communication framework), and ControlledHydraChannel , which upon closing will broadcast to all current interpreters #control channels that it's closing, so they can make appropriate actions by receiving such event. I think, there can be many ways, how you can build communication framework using channels. Some people would like it to behave same as IP stream sockets, some are not. > > To build a system with feedback all you need is to setup two channels > > (one for each image) and create a feedback protocol, like, when > > channel receives a message, it sends a message back, that he is > > actually received it :) > > Sure, DIY :) > > > As a small exercise of bidirectional communication, you can check > > HydraPing class. > > Sure. > > >> /Klaus > >> > >> -- Best regards, Igor Stasenko AKA sig. |
Re: HydraTools and minimal images
|
|
Hi Igor,
O.K. I think I got it :) Some final remarks: o yes I have the impression that functionality is good o it would not hurt to have written test cases for o - a channel which fails before 1st data transfer o - a channel which fails after 1st data transfer o - example timeout cases (other side is busy) o - example unresponsive cases (other side is dead) o - some positive cases (must transfer+verify good example data) o - demonstrate how to obtain+interpret feedback in these cases Thanx for the illustrative conversation :) /Klaus On Wed, 13 Feb 2008 15:49:33 +0100, Igor wrote: > On 13/02/2008, Klaus D. Witzel wrote: >> Hi Igor, >> >> on Wed, 13 Feb 2008 13:24:05 +0100, you wrote: >> >> > On 13/02/2008, Klaus D. Witzel wrote: >> >> Hi Igor, >> >> >> >> on Wed, 13 Feb 2008 09:35:09 +0100, you wrote: >> >> >> ... >> >> > But before going deep, i would like to get some feedback on >> channels >> >> > implementation. >> >> >> >> If I had a wish free then I'd say that Hydra's channels could provide >> >> the >> >> [a sensible part of] existing vocabulary of Socket (Socket's messages >> >> categories #open, #queries, #receiving, #sending and #waiting). >> >> >> > >> > There is, a waiting (hidden from your eyes). >> > You can check the HydraReceivingChannel>>startListening: aBlock >> > >> > VM signals a semaphore, when data arrives on channel, so a process >> > which is waiting on that semaphore is then awaken and can do anything >> > he needs to handle incoming packet. >> >> If these you lines are related to my >> #waitForDataFor:ifClosed:ifTimedOut: >> question (I assume that): >> >> a] the timeout can occur while waiting (listening) >> b] the close can occur while waiting (listening) >> c] data can arrive while waiting >> >> Case c] is easy with the #startListening: but how about capturing the >> other two cases? >> > > There is no 'close' action, there is only 'stop listening' action. So, > case b is simply impossible with such design. > Case a] can be simply implemented if you place #waitTimeoutMsecs: > instead of just #wait. > Everything in our hands :) > >> > Actually, you can omit using semaphore, and just pull data from >> > channel at any time you need. >> >> It would have to know that there's no data to pull (and why that is so), >> that is what #waitForDataFor:ifClosed:ifTimedOut: is about (besides >> pulling data). >> >> > The readData primitive works pretty straightforward: If there is data >> > waiting on channel, you'll receive a bytearray, if not - you just >> > receive a nil in response, indicating that channel input queue is >> > empty. >> > So, you can implement own listening scheme, which polls channel for >> > data periodically, and if there is no data received after supplied >> > timeout, evaluate a onTimeOut handler.. >> >> IC. >> >> >> This would make porting Socket-based applications easier and can >> >> leverage >> >> existing knowledge about Sockets. >> >> >> >> > There is some very interesting alternatives, which should be >> >> considered. >> >> ... >> >> > --- >> >> > Alternatives: >> >> ... >> >> > - adding some feedback mechanism, when incoming packet can't be >> >> > delivered (when channel input queue is full, or simply there is no >> >> > channel with given credentials). >> >> >> >> Considering a port one of my projects which currently is using >> Sockets, >> >> what I'm missing is something like >> #waitForDataFor:ifClosed:ifTimedOut: >> >> >> > >> > A channels is unidirectional. You sending data and have no feedback. >> > It works in same way as UDP sockets. >> > But on top of UDP you can always build more sophisticated layers (as >> > bidirectional streams), where you can introduce behavior like >> > #waitForDataFor:ifClosed:ifTimedOut:. >> > >> >> Is something like that possible with the Hydra channels primitives? >> > >> > ... and there is no need to make it at VM level. It can be easily done >> > at language level. >> > >> >> >> >> > I'm not sure, if this is really needed to be implemented by VM. An >> >> > efficient error/feedback mechanisms can be implemented very easily >> at >> >> > language side. >> >> >> >> It wouldn't need more/other error/feedback than what Socket provides >> >> today, IMHO. >> >> >> >> BTW: what's happening now if the second .image does no longer >> respond to >> >> the first .image, how can/does the other .image get known to this? >> > >> > Again, there is no 'response' in any means. >> >> So I have to teach the app's users, "there's no response" ;-) > > Well, i assume there is developer standing between low-level APIs and > application, who makes everything he needs to eliminate the need in > teaching users about anything :) > > I would like to hear, is there enough functionality in channel > _primitives_ (not classes which currently using them), which would > allow you to build more sophisticated communication frameworks. > As far, as i can see, from cases you given, there is no need to > add/change anything to VM's primitives. > >> >> > Channels just listening >> > for data. You can send data to channel, but you don't getting any >> > feedback about if it was delivered or handled. >> >> More seriously, I understand that in order to save the whole app from >> potentially "freezing on no response", some timeout mechanism needs to >> be >> added. But timeout alone cannot find out that the other .image has gone >> away. Is there something you would suggest for checking the other .image >> is still alive (or not alive)? >> >> NB: this check is independent of using Sockets or Hydra channels between >> any pair of Hydra .images, but it looks like there can be easier control >> with Hydra because everything is on the same machine. >> > > You can check: > - if there is still an interpreter in VM ( > HydraVM>>#isValidInterpreterInstance:) > - if there is still a channel, listening at given interpreter > instance. You may check, it by using discover primitive, and check the > newly received id and magic and compare them with previously received. > > A channel is identified by a pair: interpreter handle and name > (by analogy as IP address and port number) > > A HydraSendingChannel using > #primitiveDiscoverChannelIn: intr name: aname > to get channel's id and magic numbers. > > Once id and magic are received, it can use them in > primitiveSendDataTo: intr channelId: anId magicNum: aMagic data: aData > to send data to channel. > > If that primitive fails, it means, that given interpreter handle is > not valid anymore. > Or, aData object is not a bytearray. > Note: at this stage it's not checking that given channel exists! > So, if you continue sending data to channel with bad id/magic but > valid interpreter handle, it will be silently accepted. This is the > price we pay in having multiple interpreters running in parallel. > > I'm using id/magic instead of just name to not search and compare > strings in channel list every time you sending data. This is done > simply for speed reasons. > Also, magic number is unique for all channels instances in image. This > means, that if you stop listening existing channel, and then start > listening new even with same name, it will receive different magic > number, which will prevent it from receiving any packets addressed to > previous one. > > Yes, any senders that was targeted to send data to closed channel are > not notified automatically upon channel close. > But look at bright side: nothing stops you from implementing such > notification :) > > One could create a #control channel (which will be a part of his > reliable communication framework), and ControlledHydraChannel , which > upon closing will broadcast to all current interpreters #control > channels that it's closing, so they can make appropriate actions by > receiving such event. > > I think, there can be many ways, how you can build communication > framework using channels. Some people would like it to behave same as > IP stream sockets, some are not. > >> > To build a system with feedback all you need is to setup two channels >> > (one for each image) and create a feedback protocol, like, when >> > channel receives a message, it sends a message back, that he is >> > actually received it :) >> >> Sure, DIY :) >> >> > As a small exercise of bidirectional communication, you can check >> > HydraPing class. >> >> Sure. >> >> >> /Klaus >> >> >> >> > |
Re: HydraTools and minimal images
|
|
On 13/02/2008, Klaus D. Witzel <[hidden email]> wrote:
> Hi Igor, > > O.K. I think I got it :) Some final remarks: > > o yes I have the impression that functionality is good > o it would not hurt to have written test cases for > o - a channel which fails before 1st data transfer > o - a channel which fails after 1st data transfer - currently impossible , once 1st send was successful (you got id/magic) the future sends can't fail unless receiver's side interpreter will leave VM (which again is impossible, because not yet implemented :) ) > o - example timeout cases (other side is busy) You mean timeouts when waiting incoming data? It's pointless to introduce timeouts at this stage. But well, this can be easily illustrated. > o - example unresponsive cases (other side is dead) > o - some positive cases (must transfer+verify good example data) > o - demonstrate how to obtain+interpret feedback in these cases > I would add: o - comparable benchmarks sockets vs channels but first, i'd like to hear a design approval from project supervisor ( hello Andreas ;) ). There is no point in writing test cases, if it could be changed/replaced by something else at any moment :) > Thanx for the illustrative conversation :) Thank you for your feedback. > > /Klaus -- Best regards, Igor Stasenko AKA sig. |
Re: HydraTools and minimal images
|
|
In reply to this post by Colin Putney
I'm very interested in OB-Web
Tried loading it yesterday through Monticello, but couldn't figure get it running. I'm going to keep trying, but pointers would be useful. On Feb 13, 2008 12:34 AM, Colin Putney <[hidden email]> wrote: > > On 12-Feb-08, at 8:03 PM, Andreas Raab wrote: > > > This sounds exactly like what I was looking for. Is their stuff > > available to look at for learning how to integrate such a mechanism? > > I'm not interested in any of their proprietary IP just in learning > > how you hook this together in OB. Any other examples? > > I believe they use FFI to call out to a C library (called "gci") that > implements the low-level communications with Gemstone. The code is > publicly available here: > > http://seaside.gemstone.com/ss/ > > In particular, see the GemStone Tools project. > > Another thing to look at is OB-Web. It's less "remote" than the > Gemstone or Spoon approach, in that the class browser runs in-image > rather than remotely. What *is* remote are the widgets. They're > implemented in Javascript and run in Safari or Firefox, communicating > with the image via AJAX. It's available from the repository below, but > has some dependencies - let me know if you want to play with it. > > http://source.wiresong.ca/ob > > Colin > > |
Re: HydraTools and minimal images
|
|
In reply to this post by Göran Krampe
On 13/02/2008, [hidden email] <[hidden email]> wrote:
> Hi! > > "Igor Stasenko" <[hidden email]> wrote: > > A channels is unidirectional. You sending data and have no feedback. > > It works in same way as UDP sockets. > > But on top of UDP you can always build more sophisticated layers (as > > bidirectional streams), where you can introduce behavior like > > #waitForDataFor:ifClosed:ifTimedOut:. > > I presume you could take SocketStream and rewrite it to use Hydra > channels. Would be a fun thing to do a rainy afternoon. > > regards, Göran > -- Best regards, Igor Stasenko AKA sig. |
Re: HydraTools and minimal images
|
|
Updated a HydraVM dev notes.
Some words about legacy plugins and added HydraStream into wish list :) http://squeakvm.org/~sig/hydravm/devnotes.html -- Best regards, Igor Stasenko AKA sig. |
Re: HydraTools and minimal images
|
|
On Thu, 14 Feb 2008 02:20:02 +0100, Igor Stasenko wrote:
> Updated a HydraVM dev notes. > Some words about legacy plugins and added HydraStream into wish list :) > http://squeakvm.org/~sig/hydravm/devnotes.html You might want to add HydraSMS to the wish list (Hydra Synchronous Message Send, just claiming the name for now :) I will attempt a port of my NetworkContext project to HydraSMS, of course after your project supervisor has approved the test cases we discussed earlier :) The main purpose of HydraSMS is to provide a simple, lightweight, easy usable [etc, etc] synchronous mechanism for exchanging messages between two endpoints of a network (between two Hydra .images). This can be used for controlling any form of asynchronous computation in multiple Hydra .images, for example without always having to compile #doit's (imagine 10'000 requests per minute polluting the .changes file ;-) The unit of transfer is Smalltalk's block (i.e. what is transferred is control, back and forth). All that is needed is to compile the very same method in both Hydra .images once before using it. From a security point of view, you are supposed to trust the very same method in both .images ;-) Example with bells and whistles, except exception handling: mySMSexample | sms tmp | sms := HydraSMS with: #myChannel at: 2. "I'm now running in the other image, in which I do, " sms atHomeDo: ["this block is performed back in the initiating .image" sms nextStringPut: 'a parameter'; atWorkDo: ["and this nested block is performed in the other .image" tmp := sms nextString "get the parameter" ] ] "both sides always return self" HydraSMS (aka NetworkContext) provides protocol for #nextString* and I think about supporting SmartRefStream as well. /Klaus |
«
Return to Squeak - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |