[squeak-dev] [ANN] Hydra now can do mitosis

> On Sun, 22 Feb 2009 14:36:30 +0100, Igor Stasenko wrote:
>
>> Update.
>> you can download binaries from here
>>
>> ftp://ftp.squeak.org/HydraVM/latest_builds/win32/HydraVM-bin-22.02.2009.zip
>>
>
> Let me add 2-3 bits: first we also have to thank Alexandre "Alex" Bergel and
> Guillermo "Guille" Adrián Molina for discussing with us earlier approaches
> to running Smalltalk on n-core CPUs in parallel.
>
> 2nd: what expression (HydraClone new cloneIdleProcess) does for you, is it
> runs the computational task which is described by the corresponding object
> subgraph, in a *new* native thread (with own heap and instance of HydraVM).
> Therefore, on your n-core CPU you can expect that computation is spread
> among the cores. The #cloneIdleProcess example does just what its name says:
> it runs Smalltalk's idle process ;) as proof of concept.
>
> You Smalltalkers may have other Smalltalk processes that you want to run in
> parallel on their own native thread; let us know how we can help you with
> that.

> 2009/2/22 Klaus D. Witzel :
>> On Sun, 22 Feb 2009 14:36:30 +0100, Igor Stasenko wrote:
>>
>>> Update.
>>> you can download binaries from here
>>>
>>> ftp://ftp.squeak.org/HydraVM/latest_builds/win32/HydraVM-bin-22.02.2009.zip
>>>
>>
>> Let me add 2-3 bits: ...

>>
>> You Smalltalkers may have other Smalltalk processes that you want to  
>> run in
>> parallel on their own native thread; let us know how we can help you  
>> with
>> that.
>
> Well I have this web framework thingy where each http connection runs
> in it's own process. Request processing is quite fast, sometimes below
> a millisecond. There is potentially a lot sharing going on between
> those connections for configuration, sessions, continuations and
> stuff.

> On Sun, 22 Feb 2009 16:52:58 +0100, Philippe Marschall wrote:
>
>> 2009/2/22 Klaus D. Witzel :
>>>
>>> On Sun, 22 Feb 2009 14:36:30 +0100, Igor Stasenko wrote:
>>>
>>>> Update.
>>>> you can download binaries from here
>>>>
>>>>
>>>> ftp://ftp.squeak.org/HydraVM/latest_builds/win32/HydraVM-bin-22.02.2009.zip
>>>>
>>>
>>> Let me add 2-3 bits: ...
>
> [...]
>>>
>>> You Smalltalkers may have other Smalltalk processes that you want to run
>>> in
>>> parallel on their own native thread; let us know how we can help you with
>>> that.
>>
>> Well I have this web framework thingy where each http connection runs
>> in it's own process. Request processing is quite fast, sometimes below
>> a millisecond. There is potentially a lot sharing going on between
>> those connections for configuration, sessions, continuations and
>> stuff.
>
> Yeah we thought about that, after exchanging some Qs+As with Lukas (and/or
> you?) during last Camp Smalltalk, for the general case for Hydra.
>
> Several things are competing for attention/resolution/prototypical work:
>
> 1) some [accepted] Socket (and its OS dependent bits) can "belong" only to
> one native thread (= independent .image) in Hydra. Is this compatible with
> your web framework's requirements.
>
> 2) it must be possible for one native thread to accept a Socket, for another
> one to read/write it and get rid of it (and of its OS dependent bits). This
> is not a problem for us at the Smalltalk language/heap level but, until now
> I did't find an OS guy who said "this works, with guarantee, on these XYZ
> platforms".

> Hello lists,
>
> (see http://en.wikipedia.org/wiki/Cell_division for explanation what
> mitosis is ;)
>
> i just finished another torturing session in slang land, and finally
> made some of it working :)
>
> I added a two new primitives, which aimed towards creating a new heap
> by cloning a designated set of objects from existing one.
> A first primitive is helps you to create a closed graph by reporting a
> set of objects which is referenced by objects in provided array, but
> not included in this array itself.
>
> A second primitive is actually serves for creating a baby heap.
> I takes two array arguments: objects graph, and replacements (or substitutions).
>
> The rules is simple:
> each object included in first array, should have references to objects
> which included in this array as well i.e.
> (objects identityIncludes: (objects at: x) someIvar ) == true
>
> or if its not , then it can be found in a substitutions array.
> A substitutions array is a flat array of pairs oop -> index, where oop
> can be any oop, and index is a smallint index of item in objects
> array.
> Upon creation of new heap, all refs to oops in 'substitutions' array
> will be replaced by refs from 'objects' array by taking corresponding
> index.
>
> Both array arguments should contain oops only (except from indexes in
> substitutions array) , should not contain duplicates and their
> intersection should be empty , i.e.
>   (objects intersection: substitutions) isEmpty == true.
>
> All of these rules actually serving to ensure that newly created
> object memory will form a fully closed graph (hence its oops can be
> cloned to a standalone object memory).
>
> There are also, an initial example how to create a 'baby' object memory
> load http://www.squeaksource.com/HydraVM/HydraVM-sig.17.mcz
>
> and do:
>
> HydraClone new cloneIdleProcess
>
> Of course you will need new VM.
> Sorry, i didn't published an update of sources yet. I can only give
> you a prebuilt one.
> Once i had a write access to ftp.squeak.org/HydraVM but i forgot
> credentials/key to it. (if Box Admins help me with this, i will upload
> it there).
> Or, you can write me in private and i will send you binaries.
>
> P.S. I want to give a credit to Klaus D. Witzel who helped me to
> finish these new features, and inspired to do it :)
>
>

>>> Well I have this web framework thingy where each http connection runs
>>> in it's own process. Request processing is quite fast, sometimes below
>>> a millisecond. There is potentially a lot sharing going on between
>>> those connections for configuration, sessions, continuations and
>>> stuff.
>>
>> Yeah we thought about that, after exchanging some Qs+As with Lukas  
>> (and/or
>> you?) during last Camp Smalltalk, for the general case for Hydra.
>>
>> Several things are competing for attention/resolution/prototypical work:
>>
>> 1) some [accepted] Socket (and its OS dependent bits) can "belong" only  
>> to
>> one native thread (= independent .image) in Hydra. Is this compatible  
>> with
>> your web framework's requirements.
>>
>> 2) it must be possible for one native thread to accept a Socket, for  
>> another
>> one to read/write it and get rid of it (and of its OS dependent bits).  
>> This
>> is not a problem for us at the Smalltalk language/heap level but, until  
>> now
>> I did't find an OS guy who said "this works, with guarantee, on these  
>> XYZ
>> platforms".
>
> That and 1) do not necessarily need to happen. You could imagine one
> image/native thread doing all the socket IO and handing the actual
> work to other images/native threads.

> Klaus D. Witzel wrote:
>
>> 2nd: what expression (HydraClone new cloneIdleProcess) does for you, is  
>> it runs the computational task which is described by the corresponding  
>> object subgraph, in a *new* native thread (with own heap and instance of
>
> Just trying to understand this...
>
> Does the object subgraph have to be complete (include all objects,  
> classes and methods) to run the task?

> On Sun, 22 Feb 2009 18:19:33 +0100, Andreas Raab wrote:
>
>> Interesting. Is this conceptually different from making a snapshot using
>> image segments and move this over to a different Hydra instance?
>
> In principle: Yes.
>
> Hydra mitosis tranfers all objects required by contractual obligation with
> the VM (special objects array, methods, context, process, scheduler, method
> dictionaries, classes, language level constants, etc).
>

>>
>> Cheers,
>>   - Andreas
>
> [...]
>
> --
> "If at first, the idea is not absurd, then there is no hope for it". Albert
> Einstein
>
>
>

> On Sun, Feb 22, 2009 at 1:38 PM, Klaus D. Witzel <[hidden email]>
> wrote:
>>
>> Yes, "classic" Hydra loads the second .image from file and mitosis Hydra
>> populates (the second) an empty .image from am arrayOfObjects passed in.
>
> Ok...this is just, well, fun, but I can't quite figure out how to get a
> second image loaded with the #doIt channel "working."  I figure it's because
> I don't have the image I created it from set up right?  All I did was go
> down the line of in the Helpful doits workspace:
>
> "Sanity check, are we running HydraVM?"
> HydraVM isHydra  [true]
>
> "To prepare an image for running in background thread (careful, VM will quit
> if you run this)"
> HydraVM saveAsHeadlessImage: 'test.image'  [current image shut down and
> created another image]
>
> "To run previously prepared image"
> HydraVM loadAndRunNewImage: 'test.image'  [seemed to work]
>
> "How many interpreters VM currently running"
> HydraVM interpreterInstanceCount [2...this is good!]
>
> "Test to check if main image listening for #transcript channel ..."
> HydraChannel send: 'Transcript test' to: (HydraInterpreter current) channel:
> #transcript [true after the following line]
>
> "... if not, try doing this first: "
> HydraTranscript install
>
> "Simple test to check if second image listens #doIt channel "
> HydraChannel hasChannel: #doIt at: (HydraInterpreter at: 2)  [false]
>
> Drat!!!
>

> " ... and has Transcript global var replaced to redirect all input to main
> image"
> (HydraInterpreter at: 2) doIt: 'Transcript show: ''Doit Test'''
>
> "Do you like to install something to secondary image?"
> (HydraInterpreter at: 2) doIt: '(HTTPSocket httpGet:
> ''installer.pbwiki.com/f/Installer.st'') fileIn '
>
> "Do you like to save it? "
> (HydraInterpreter at: 2) doIt: 'Smalltalk snapshot: true andQuit: false '
>
>
> Take care,
>
> Rob
>
>
>
>
>

> 2009/2/22 Rob Rothwell <[hidden email]>:
>> On Sun, Feb 22, 2009 at 1:38 PM, Klaus D. Witzel <[hidden email]>
>> wrote:
>>>
>>> Yes, "classic" Hydra loads the second .image from file and mitosis Hydra
>>> populates (the second) an empty .image from am arrayOfObjects passed in.
>>
>> Ok...this is just, well, fun, but I can't quite figure out how to get a
>> second image loaded with the #doIt channel "working."  I figure it's because
>> I don't have the image I created it from set up right?  All I did was go
>> down the line of in the Helpful doits workspace:
>>
>> "Sanity check, are we running HydraVM?"
>> HydraVM isHydra  [true]
>>
>> "To prepare an image for running in background thread (careful, VM will quit
>> if you run this)"
>> HydraVM saveAsHeadlessImage: 'test.image'  [current image shut down and
>> created another image]
>>
>> "To run previously prepared image"
>> HydraVM loadAndRunNewImage: 'test.image'  [seemed to work]
>>
>> "How many interpreters VM currently running"
>> HydraVM interpreterInstanceCount [2...this is good!]
>>
>> "Test to check if main image listening for #transcript channel ..."
>> HydraChannel send: 'Transcript test' to: (HydraInterpreter current) channel:
>> #transcript [true after the following line]
>>
>> "... if not, try doing this first: "
>> HydraTranscript install
>>
>> "Simple test to check if second image listens #doIt channel "
>> HydraChannel hasChannel: #doIt at: (HydraInterpreter at: 2)  [false]
>>
>> Drat!!!
>>
>
> Yep.. in 3.10 this not working..
> It looks like there are some crap in image startup method.
> As a workaround, you can register the #doIt channel in your base image:
>
> HydraVM registerDoItChannel.
> and only then do
> HydraVM saveAsHeadlessImage: 'test.image'
>
> should work ok.
>

>> " ... and has Transcript global var replaced to redirect all input to main
>> image"
>> (HydraInterpreter at: 2) doIt: 'Transcript show: ''Doit Test'''
>>
>> "Do you like to install something to secondary image?"
>> (HydraInterpreter at: 2) doIt: '(HTTPSocket httpGet:
>> ''installer.pbwiki.com/f/Installer.st'') fileIn '
>>
>> "Do you like to save it? "
>> (HydraInterpreter at: 2) doIt: 'Smalltalk snapshot: true andQuit: false '
>>
>>
>> Take care,
>>
>> Rob
>>
>>
>>
>>
>>
>
>
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>

> On Sun, 22 Feb 2009 18:18:20 +0100, Philippe Marschall wrote:
>
>> 2009/2/22 Klaus D. Witzel :
>
> ...
>>>>
>>>> Well I have this web framework thingy where each http connection runs
>>>> in it's own process. Request processing is quite fast, sometimes below
>>>> a millisecond. There is potentially a lot sharing going on between
>>>> those connections for configuration, sessions, continuations and
>>>> stuff.
>>>
>>> Yeah we thought about that, after exchanging some Qs+As with Lukas
>>> (and/or
>>> you?) during last Camp Smalltalk, for the general case for Hydra.
>>>
>>> Several things are competing for attention/resolution/prototypical work:
>>>
>>> 1) some [accepted] Socket (and its OS dependent bits) can "belong" only
>>> to
>>> one native thread (= independent .image) in Hydra. Is this compatible
>>> with
>>> your web framework's requirements.
>>>
>>> 2) it must be possible for one native thread to accept a Socket, for
>>> another
>>> one to read/write it and get rid of it (and of its OS dependent bits).
>>> This
>>> is not a problem for us at the Smalltalk language/heap level but, until
>>> now
>>> I did't find an OS guy who said "this works, with guarantee, on these XYZ
>>> platforms".
>>
>> That and 1) do not necessarily need to happen. You could imagine one
>> image/native thread doing all the socket IO and handing the actual
>> work to other images/native threads.
>
> Yup. And the price tag is a) transfer "task description" to worker thread
> and b) transfer "result string" back in full or in part(s). And when using
> Hydra channels for transfer, add c) c-alloc of strings-to-be-transported.
>
>>> 3) when your web framework+app depends on a database "whatsoever", how is
>>> this to be shared (if sharing is really needed) between multiple native
>>> threads.
>>
>> Yepp, if you interpret database in a more general sense (a bunch of
>> objects).
>
> ... which includes framework/app config update/change. Right.
>
>> There is also the question of how coordination happens
>> between multiple images especially on this shared data.
>
> Right. Looks like a topic for the next Camp Smalltalk, with hands-on.