Recreating live coding in CPython 3.5
Locked
12
12
Recreating live coding in CPython 3.5
 
|
Hey there as it says in the title I am trying to recreat live coding Pharo chracteristics in Python . One of the issues I have is live replacing of a method to an existing instance.
I know how to do this in Python but that shown me I would benefit greatily from a more deeper understanding of Pharo live coding internals than I have at the moment which is only at the user level. This will help me avoid common pitfalls in a field that I am not experienced with. Have you guys written any paper , article or blog post in this area, explaining the internals of the VM to that regard ? I am also interested in time machine livde coding, meaning live coding that does not keeps you in the present but also can send you back in the past restoring the live state of old object instances in a specific time moment. Generally any guidance on the art of implementing live coding will be greatly appreciated. I don't care how this can be done in Python because I have already recreated a very basic live coding enviroment in Python and there is a lot of documentation about Python overall what I am interested only is the Smalltalk way of doing live coding but from the perspective of the implementors not the mere users. |
Re: Recreating live coding in CPython 3.5
|
2017-10-11 12:14 GMT+02:00 Dimitris Chloupis <[hidden email]>:
Now I am confused, I thought you already did this? In the other thread you wrote: "From my experience around 100 lines of code are enough to offer most of Pharo’s basic live coding functionality"
And in the other thread you wrote: "Well live coding is a simple process of reloading code.
In Pharo the VM recompiles a method and replace it’s old instance with a
new one while code executes. Python
you basically reload the module. Because Python does not compile per
method but per module which means executing code lively. This happens
because when you import a module in Python, Python does nothing special
than executing the source code. Not before compilation but during
execution. Hence live coding, the Python VM replaces objects lively.
Python can also compile any size of code including individual methods. That happens with one line of code importlib.reload(mymodule) " So, if python can already reload code and replace the instances, what is missing ?
I remember a project that developed some tools for back-in-time debugging and testing https://www.hpi.uni-potsdam.de/hirschfeld/trac/SqueakCommunityProjects/wiki/pathToolsFramework
|
Re: Recreating live coding in CPython 3.5
|
|
on the subject of " In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes" I was wrong, the rest it correct in the live execution of the code and other objects. Pharo VM will update the class objects (together with other type of objects) but it wont update instance objects.Probably this is done on purpose for not overriding and losing on reload the existing live state and live code (in this case instance methods), because module reloading is not made specific for live coding as I explained in the thread. So a class object will be replaced but not an instance. What led me to this wrong assumption was that the code I am working on is reinitialising my objects for debugging purposes because currently live state is not my concern as I am working on a GUI and some other technical reason. I did not realise that it was my reinisitialisation that was updating my instance objects(method wise, state was lost obviously) and not Python VM. Hence in my code the instance objects were always updated to the latest code as I was changing it. However now I enter the stage that I need to also update the live state without a full reinitialisation. Its possible to update the instance object manually . So far I have found two ways of doing this a) Reinitialisation and take the state from the old one (instance variables and their values are stored as a dicitionary) and copy them to the new one and discard the old instance or b) take the methods (methods in python are objects and so are functions so they can be referenced) from the new one and copy them to the old one and discard the new instance. I have tested both, both work as expected. Now I am in the process of finding how to detect that a class changed in a module so i dont have to reinitialise all classes (I know the overall solution to this as well) in a module and because reloading works per module at least I have avoided the other instances. But that is not my problem. My problem is what other challanges I have to face which I am not aware of live coding wise. Because I never tried implementing live coding enviroment in another language (my experiements are focused only on live execution of code because at the time the idea was to keep using Pharo for live state data) before it would be great if experts give me an insight about Pharo's live coding internals. This way I can "steal" cool ideas that I may not be aware they exists and make my live coding enviroment library progressively closer to Pharo. Hence, I need guidance for the advanced stuff. On Wed, Oct 11, 2017 at 2:09 PM Nicolai Hess <[hidden email]> wrote:
|
Re: Recreating live coding in CPython 3.5
|
|
oops forgot to thank you for link, will give a thorough read. It seems the project you talking about is PathFinder. On Wed, Oct 11, 2017 at 3:11 PM Dimitris Chloupis <[hidden email]> wrote:
|
|
|
In reply to this post by kilon.alios
On Wed, Oct 11, 2017 at 8:11 PM, Dimitris Chloupis <[hidden email]> wrote:
I'm not sure if this matches what you want, but I remember finding these articles enlightening about VM internals... Alternatively, with your background and focus on Python, maybe you can learn something from RSqueak? ... * http://scg.unibe.ch/archive/papers/Bolz08aSpy.pdf cheers -ben
|
Re: Recreating live coding in CPython 3.5
|
|
unfortunately no, those articles focus on JIT compilation and method lookups and not live coding.
I know how to find method objects , thats not my problem. JIT is not my concern either. But none the less thanks for the links they made a very enlighting read after lunch. Well lets first start with the obvious questions 1) What are the live coding capabilities/features of the Cog VM ? 2) What are its live coding limitations ? 3) When a live object is updated is it only the new compiled method that is injected ? 4) Does this injection follows a particular schedule or process ? Or is something happening ASAP after the user accepts the method ? 4) Does class and method renaming also happen at VM level ? Is it related to live coding ? On Wed, Oct 11, 2017 at 3:49 PM Ben Coman <[hidden email]> wrote:
|
Re: Recreating live coding in CPython 3.5
|
|
In reply to this post by Ben Coman
A couple years ago Paul Fernhout did start working on a Morphic environment in Python. He tried the live coding too, but couldn't get it to work. This was prompted by Alan Kay suggesting the need for a live coding environment for OLPC. Guido van Rossum himself stepped in and created a module that lets you replace most things on the fly. I don't recall how it worked exactly, but I'm sure you can find it. But it never found much acceptance in the Python community, AFAIK. Another approach is the one taken by HPI's RSqueak VM which is built on top of PyPy and lets you develop, run and debug Smalltalk and Python code (https://conf.researchr.org/profile/MoreVMs-2017/fabioniephaus). - Bert - On Wed 11. Oct 2017 at 14:50, Ben Coman <[hidden email]> wrote:
|
Re: Recreating live coding in CPython 3.5
|
|
> On 11.10.2017, at 15:50, Bert Freudenberg <[hidden email]> wrote: > > A couple years ago Paul Fernhout did start working on a Morphic environment in Python. He tried the live coding too, but couldn't get it to work. This was prompted by Alan Kay suggesting the need for a live coding environment for OLPC. Guido van Rossum himself stepped in and created a module that lets you replace most things on the fly. I don't recall how it worked exactly, but I'm sure you can find it. But it never found much acceptance in the Python community, AFAIK. > http://www.diva-portal.org/smash/get/diva2:22296/FULLTEXT01.pdf |
Re: Recreating live coding in CPython 3.5
|
|
In reply to this post by Bert Freudenberg
Are you talking about live instances updating or module reloading ? Module reloading is very old , certianly not a couple of years
According to the archives of Python documentation it was available at version 1.5 back in Aprill 1998 About the other features I cannot vouche for back then , but many do exist nowdays. live instance updating is not a problem as I said , python can replace methods lively in a existing instance or inject new methods that exist only in a single instance (you can assign even a fuction as a method to an instance runtime because functions are also objects) , manipulate its live state by changing/renaming/creating instance variables again during execution etc. I can do this with simple python commands. Python also like Cog VM has its own dictionarie for methods and classes, seperate dictionaries for instance and class variables. Each method in an instance is a separate object and so is each variable as pretty much anything in Python is an object, etc etc. No idea about the Guido Van Rossume module you speak of. Those features are not part of a module or library they are an internal part of the language. The one that is a module is the imp library which nowdays is called importlib, which is responsibe for reloading a module and generallly other module functions. I am talking about beyond those obvious features that Python has them covered is there anything special the Cog VM is doing live coding wise that I am not aware of ? On Wed, Oct 11, 2017 at 4:50 PM Bert Freudenberg <[hidden email]> wrote:
|
Re: Recreating live coding in CPython 3.5
|
|
In reply to this post by Tobias Pape
Ah PyMorphic, yeah I actually used it for making a Blender Python GUI API. Really clean code but only a tiny fraction of Morphic abilities. I did not know it tackled live coding, I dont remember seeing anything relevant in the source code. Maybe i should take a nother look thanks.
On Wed, Oct 11, 2017 at 4:56 PM Tobias Pape <[hidden email]> wrote:
|
Re: Recreating live coding in CPython 3.5
|
|
In reply to this post by kilon.alios
Dimitris, > unfortunately no, those articles focus on JIT compilation and > method lookups and not live coding. Actually, these things are at the very core of live coding. Unless we are using the term in very different ways. So let me explain what I mean by "live coding". Imagine that I have two code browsers showing methods like: methA: x ... [...] whileTrue: [ self methB ]. ... and methoB ... ... and I have a third window where I am watching the effect of running #methA. What happens if I change the text in the second window and select "accept"? What happens if I edit the first window and accept? If the system is a simple interpreter and "accept" generated new bytecodes that get installed in the method dictionary in place of the previous version of the method, we would expect an instant change in behavior when changing #methB but nothing would happen when #methA is modified. That is because #methB is repeatedly called and the current execution finishes executing using the old code and then it gets invoked again and starts executing the new code. #methA, on the other hand, is in the middle of a very long loop and will keep executing the old code (pointed to from the stack) even though the method dictionary now has new code. An alternative system could use #become: to replace bytecodes when changing a method, but that would be hard to do (suddenly the stack state would no longer make sense with the new bytecodes) and would be harder for the user to understand. So let's not go there. If we have a JIT and not a simple interpreter, things can be complicated if we want to keep the same semantics. The compiler might have inlined #methB into #methA and in that case changing #methB would no longer have a visible effect in the third window. The technology to make it work as expected is called "dynamic deoptimization" and is pretty sophisticated. -- Jecel |
Re: Recreating live coding in CPython 3.5
|
|
In reply to this post by kilon.alios
On Wed 11. Oct 2017 at 16:27, Dimitris Chloupis <[hidden email]> wrote:
Instance enumeration, the become operation, and reified stack frames are features that set Smalltalk apart and enable many of its live coding capabilities. Cog supports these features. Cog was specifically designed to allow fast stack operations in the normal case while still allowing frame manipulation. Spur was designed to make "become" fast, it used to be very expensive. - Bert - |
Re: Recreating live coding in CPython 3.5
|
|
In reply to this post by Jecel Assumpcao Jr
Jecel , you hit a head on the issues I want to tackle, this exactly the concept I want to understand to implement an efficient live coding enviroment so thank you.
Before starting excuse any misunderstanding I am very fresh into the whole idea of live coding implementation so I may not understand the specific you mention. Now the way I understand this in Python its possible to keep around both the old and new instance , if a method is replaced what happens is that Python treats methods a bit like variables, again I am not talking VM I know nothing about Python VM internals. This is purely python code. So if you have a method my_instance.my_method(arg1, arg2) the statement my_instance.my_method wont call the method because it does not contain the parenthesses needed for method and function calls However it wont throw an error either. What will happen is that like a variable it will return a reference to the method associated. That method is a separate object. Of type MethodObject which has a function __CALL__ which is doing the work of parentheses and of course it has instance variables for arguments, classes that is bound to etc. So if I do what you say and change methodB and inject the updated version of methodB which basically will be its new instance containing the modified code in bytecode format to the place of old instance, i will be replacing a reference and as such the 'variable' now will reference another method . so Any call that will now be issues will be sent directly to the updated version of the method. so basically I would be doing following the previous example updated_method = updated_instance.method my_instance.my_method=updated_method # now points to updated_instance.method my_instace.my_method(arg1,arg2) #this equals to updated_instance.method(arg1,arg2) It wont matter if its a long loop , as matter of fact I am working on a loop right now doing these experiment because the Python I use is in Blender (3d app) running concurent with many other stuff sometimes GBs of data etc etc. Unless we talking here millisecond to microsecond responses ? I have not tested in such speeds Do I understand you correctly or I miss something here. Why you need a JIT to do this ? I see that pharo has CompiledMethod so methods also in pharo are objects that means they are also referenced not tied to the object instances , right ? Should not this work outside the box ? So in my scenario of live replacing methods you should see the results immediately , meaning as soon as the method is called again and both editors will have to display the same exact code if they view the same method. All calls are essentialy done via reference. What am I missing here ? On Wed, Oct 11, 2017 at 6:02 PM Jecel Assumpcao Jr. <[hidden email]> wrote:
|
Re: Recreating live coding in CPython 3.5
|
|
In reply to this post by Bert Freudenberg
does "become" only exchange references ? Or is it more to it than that ?
where I can find information about what insance enumeration and reified stack frames are ? Python has methods in gc module (module for its garbage collector) to find objects that point to an object or objects that an object point to . But it has no convenient method to exchange those references . I assume you could do it via an iteration but I never tried it in practice. Indeed such feature is essential for live coding. I will have to add it to my implementation thanks. Is this book still relevant for my implementation ? On Wed, Oct 11, 2017 at 6:07 PM Bert Freudenberg <[hidden email]> wrote:
|
|
|
> Indeed such feature is essential for live coding. I will have to add > it to my implementation thanks. I don't think become: is essential. Smalltalk/X for instance does not use #become: yet it still provides live coding (in my opinion). HTH, Jan |
Re: Recreating live coding in CPython 3.5
|
|
Do you know how instances are reshaped without become in Smalltalk/X? But indeed it may not be a problem in Python where instances do not have a fixed number of instance variables anyway. You still need to be able to enumerate them, to fix them up. - Bert - |
|
|
On Wed, 2017-10-11 at 21:49 +0200, Bert Freudenberg wrote: > > On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany <[hidden email]> > wrote: > > > > > Indeed such feature is essential for live coding. I will have to > > add > > > it to my implementation thanks. > > > > I don't think become: is essential. Smalltalk/X for instance does > > not > > use #become: yet it still provides live coding (in my opinion). > > Do you know how instances are reshaped without become in Smalltalk/X? Yes, it's very simple: they are not reshaped at all :-) Jan |
Re: Recreating live coding in CPython 3.5
|
|
2017-10-11 21:56 GMT+02:00 Jan Vrany <[hidden email]>:
Then does that mean that - classes are not mutated, but that a new one is created, and old objects are still pointing on old classes, - or that mutation is forbidden as long as instances or sub-instances exist (like in Smalltalk V if I remember) - or that one may expect catastrophic consequences when reshaping? Nicolas |
|
|
On Wed, 2017-10-11 at 22:01 +0200, Nicolas Cellier wrote: > > > > 2017-10-11 21:56 GMT+02:00 Jan Vrany <[hidden email]>: > > On Wed, 2017-10-11 at 21:49 +0200, Bert Freudenberg wrote: > > > > > > On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany <[hidden email] > > > > > > wrote: > > > > > > > > > Indeed such feature is essential for live coding. I will > > have to > > > > add > > > > > it to my implementation thanks. > > > > > > > > I don't think become: is essential. Smalltalk/X for instance > > does > > > > not > > > > use #become: yet it still provides live coding (in my opinion). > > > > > > Do you know how instances are reshaped without become in > > Smalltalk/X? > > > > Yes, it's very simple: they are not reshaped at all :-) > > > > Jan > > > Then does that mean that > - classes are not mutated, but that a new one is created, and old > objects are still pointing on old classes, Exactly. The old class eventually becomes (not #become:s :-) garbage and gets (eventually) collected. > - or that mutation is forbidden as long as instances or sub-instances > exist (like in Smalltalk V if I remember) > - or that one may expect catastrophic consequences when reshaping? Depends what you call catastrophic consequences... Jan |
Re: Recreating live coding in CPython 3.5
|
|
In reply to this post by Jan Vrany
I found actually a way to have become without become :D
what if instead of actually changing the references of one object to the other we changed the object themselves and instead of exchanging references we exchange instance variables and instance methods ? This way objects that pointed to object a will still point to object a instead of object b but a will now have really become object b , and object b will have become object a. is there anything more to an instance besides its instance variables and methods ? I am including of course inherited instance variables and methods. I tried this in Python and it worked but I am not sure if become is doing anything more than that. On Wed, Oct 11, 2017 at 10:32 PM Jan Vrany <[hidden email]> wrote:
|
«
Return to Squeak VM
|
1 view|%1 views
| Free forum by Nabble | Edit this page |