Recreating live coding in CPython 3.5

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Recreating live coding in CPython 3.5

kilon.alios
 
Python allows you to inject new variables to an instance through normal assignement as you can do also with methods. So you replace existing ones as I mentioned before but also extend the instance with new variables and methods after its created.

The one thing I thought Python would not allow me to do was to delete variables and especially methods from an instance but it appears it does that too. Always during run time of course. 

Its safe to assume that the same applies to object classes as well. So technically this means you have complete freedom on how much you can affect a live instance. At least thats how it looks to me so far. 

 

On Wed, Oct 11, 2017 at 10:50 PM Bert Freudenberg <[hidden email]> wrote:
On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany <[hidden email]> wrote:


> Indeed such feature is essential for live coding.  I will have to add
> it to my implementation thanks.

I don't think become: is essential. Smalltalk/X for instance does not
use #become: yet it still provides live coding (in my opinion).

​Do you know how instances are reshaped without become in Smalltalk/X?​

But indeed it may not be a problem in Python where instances do not have a fixed number of instance variables anyway. You still need to be able to enumerate them, to fix them up. 

- Bert -

Reply | Threaded
Open this post in threaded view
|

Re: Recreating live coding in CPython 3.5

Nicolas Cellier
In reply to this post by Jan Vrany
 


2017-10-11 22:11 GMT+02:00 Jan Vrany <[hidden email]>:

On Wed, 2017-10-11 at 22:01 +0200, Nicolas Cellier wrote:
>  
>
>
> 2017-10-11 21:56 GMT+02:00 Jan Vrany <[hidden email]>:
> > On Wed, 2017-10-11 at 21:49 +0200, Bert Freudenberg wrote:
> > >  
> > > On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany <[hidden email]
> > >
> > > wrote:
> > > >
> > > > > Indeed such feature is essential for live coding.  I will
> > have to
> > > > add
> > > > > it to my implementation thanks.
> > > >
> > > > I don't think become: is essential. Smalltalk/X for instance
> > does
> > > > not
> > > > use #become: yet it still provides live coding (in my opinion).
> > >
> > > Do you know how instances are reshaped without become in
> > Smalltalk/X?
> >
> > Yes, it's very simple: they are not reshaped at all :-)
> >
> > Jan
>
>  
> Then does that mean that
> - classes are not mutated, but that a new one is created, and old
> objects are still pointing on old classes,

Exactly. The old class eventually becomes (not #become:s :-) garbage 
and gets (eventually) collected.

> - or that mutation is forbidden as long as instances or sub-instances
> exist (like in Smalltalk V if I remember)
> - or that one may expect catastrophic consequences when reshaping?

Depends what you call catastrophic consequences...

Jan

Catastrophic could be for example a buffer overflow that would bypass VM safety...
Typically when instances and compiled methods idea of inst. var. layout divorces.
That does not seem the case in Smalltalk/X from what you describe...
For example, in Squeak or VW, when there is a Foo>>bar method up in the stack that triggers removal of an ivar, then writes into an ivar, boom! object memory is corrupted... Indeed the methods cannot easily be become'd while still active... So they are replaced - but the instances are become'd
Reply | Threaded
Open this post in threaded view
|

Re: Recreating live coding in CPython 3.5

Jan Vrany
 
On Wed, 2017-10-11 at 22:19 +0200, Nicolas Cellier wrote:

>  
>
>
> 2017-10-11 22:11 GMT+02:00 Jan Vrany <[hidden email]>:
> > On Wed, 2017-10-11 at 22:01 +0200, Nicolas Cellier wrote:
> > >  
> > >
> > >
> > > 2017-10-11 21:56 GMT+02:00 Jan Vrany <[hidden email]>:
> > > > On Wed, 2017-10-11 at 21:49 +0200, Bert Freudenberg wrote:
> > > > >  
> > > > > On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany <[hidden email]
> > t.cz
> > > > >
> > > > > wrote:
> > > > > >
> > > > > > > Indeed such feature is essential for live coding.  I will
> > > > have to
> > > > > > add
> > > > > > > it to my implementation thanks.
> > > > > >
> > > > > > I don't think become: is essential. Smalltalk/X for
> > instance
> > > > does
> > > > > > not
> > > > > > use #become: yet it still provides live coding (in my
> > opinion).
> > > > >
> > > > > Do you know how instances are reshaped without become in
> > > > Smalltalk/X?
> > > >
> > > > Yes, it's very simple: they are not reshaped at all :-)
> > > >
> > > > Jan
> > >
> > >  
> > > Then does that mean that
> > > - classes are not mutated, but that a new one is created, and old
> > > objects are still pointing on old classes,
> >
> > Exactly. The old class eventually becomes (not #become:s :-)
> > garbage 
> > and gets (eventually) collected.
> >
> > > - or that mutation is forbidden as long as instances or sub-
> > instances
> > > exist (like in Smalltalk V if I remember)
> > > - or that one may expect catastrophic consequences when
> > reshaping?
> >
> > Depends what you call catastrophic consequences...
> >
> > Jan
> >
>
> Catastrophic could be for example a buffer overflow that would bypass
> VM safety...
> Typically when instances and compiled methods idea of inst. var.
> layout divorces.
> That does not seem the case in Smalltalk/X from what you describe...
> For example, in Squeak or VW, when there is a Foo>>bar method up in
> the stack that triggers removal of an ivar, then writes into an ivar,
> boom! object memory is corrupted...

I see. No, this should not happen in Smalltalk/X. Indeed all that 
logic implemented on Smalltalk level so one can construct code that
results in this kind of a corruption (simply change offsets in 
method's bytecode array).

To make the system robust against this sort of problems one might need 
to take the approach not unlike the one of Java (JVM) which is
more robust w.r.t. this. This is one of the reasons (not the only one)
why "live coding" in Java is lot more complicated (but not entirely
impossible).


> Indeed the methods cannot easily be become'd while still active... So
> they are replaced - but the instances are become'd

Jan


Reply | Threaded
Open this post in threaded view
|

Re: Recreating live coding in CPython 3.5

Ben Coman
In reply to this post by kilon.alios
 


On Thu, Oct 12, 2017 at 12:38 AM, Dimitris Chloupis <[hidden email]> wrote:
 
does "become" only exchange references ? Or is it more to it than that ?


 

where I can find information about what insance enumeration and reified stack frames are ? 

Lhe last entry here...

Also, does Python do Polymorphic Inline Caching?...

or bytecode level hotspot recompilation...

cheers -ben

 

Python has methods in gc module (module for its garbage collector) to find objects that point to an object or objects that an object point to . But it has no convenient method to exchange those references . I assume you could do it via an iteration but I never tried it in practice. 

Indeed such feature is essential for live coding.  I will have to add it to my implementation thanks.

Is this book still relevant for my implementation ?


On Wed, Oct 11, 2017 at 6:07 PM Bert Freudenberg <[hidden email]> wrote:
On Wed 11. Oct 2017 at 16:27, Dimitris Chloupis <[hidden email]> wrote:

I am talking about beyond those obvious features that Python has them covered is there anything special the Cog VM is doing live coding wise that I am not aware of ? 

Instance enumeration, the become operation, and reified stack frames are features that set Smalltalk apart and enable many of its live coding capabilities. Cog supports these features. Cog was specifically designed to allow fast stack operations in the normal case while still allowing frame manipulation. Spur was designed to make "become" fast, it used to be very expensive.

- Bert -


Reply | Threaded
Open this post in threaded view
|

Re: Recreating live coding in CPython 3.5

kilon.alios
 
Thanks Ben actually your first link was the first thing I read when I googled become. I dont think VM lookup benefits me in my case as this is already handled by the Python VM. 

You other tlinks are JIT VM related, Python is not friendly to JIT because its creator believe that performance enhancement should come from wrapping C libraries and wants to avoid the cost of more complex VM . In the case of Python this is easy to follow because Python has a ton of high performance libraries which by the way almost never are used when comparing Python with other languages speed wise (kinda fare if you just compare language implementations). Pharo does benefit a lot more from JIT VM as it does not have as many libraries targeting high performance. 

On the last link, I do not know if I understand this correctly at sista side but at python side yes you can take any string and compile to bytecode real time and inject that code back to your live execution. If that is what you mean. Live bytecode compilation can happen at module level too of course , and a string can be anything from one line of code to a collection of modules. The cool thing is that you can sandbox the compiling too so you do not cause any security issues, giving the code access only to a fraction of your state and code. So yes it gives you a lot of room to move around in this department but again there is no JIT VM , at least for CPython.

PyPy is a JIT VM for python with Slang like language called rpython that can be used to port any language and I spoken with some of it authoer and seem to follow Cog VM vclosely and of coruse expressed admiration of Eliot's work. Unfortunately PyPy had a long track of issues with python libraries that wrapped C code so it never get a lot of traction, but its non the less used by some python coders 

If anyone does mind I would like some pointers on the subject of 
class enumaration 
and reified stack frames 
that were mentioned earlier on, I googled them but did not find enough to clarify what they are in case of Cog VM. 


On Thu, Oct 12, 2017 at 12:42 AM Ben Coman <[hidden email]> wrote:
On Thu, Oct 12, 2017 at 12:38 AM, Dimitris Chloupis <[hidden email]> wrote:
 
does "become" only exchange references ? Or is it more to it than that ?


 

where I can find information about what insance enumeration and reified stack frames are ? 


 

Python has methods in gc module (module for its garbage collector) to find objects that point to an object or objects that an object point to . But it has no convenient method to exchange those references . I assume you could do it via an iteration but I never tried it in practice. 

Indeed such feature is essential for live coding.  I will have to add it to my implementation thanks.

Is this book still relevant for my implementation ?


On Wed, Oct 11, 2017 at 6:07 PM Bert Freudenberg <[hidden email]> wrote:
On Wed 11. Oct 2017 at 16:27, Dimitris Chloupis <[hidden email]> wrote:

I am talking about beyond those obvious features that Python has them covered is there anything special the Cog VM is doing live coding wise that I am not aware of ? 

Instance enumeration, the become operation, and reified stack frames are features that set Smalltalk apart and enable many of its live coding capabilities. Cog supports these features. Cog was specifically designed to allow fast stack operations in the normal case while still allowing frame manipulation. Spur was designed to make "become" fast, it used to be very expensive.

- Bert -

Reply | Threaded
Open this post in threaded view
|

Re: Recreating live coding in CPython 3.5

Nicolai Hess-3-2
 


2017-10-12 10:50 GMT+02:00 Dimitris Chloupis <[hidden email]>:
 
Thanks Ben actually your first link was the first thing I read when I googled become. I dont think VM lookup benefits me in my case as this is already handled by the Python VM. 

You other tlinks are JIT VM related, Python is not friendly to JIT because its creator believe that performance enhancement should come from wrapping C libraries and wants to avoid the cost of more complex VM . In the case of Python this is easy to follow because Python has a ton of high performance libraries which by the way almost never are used when comparing Python with other languages speed wise (kinda fare if you just compare language implementations). Pharo does benefit a lot more from JIT VM as it does not have as many libraries targeting high performance. 


The interesting parts in this "JIT VM related" links is, if you happen to write a JIT VM , you have to take care of the language features smalltalk provides. For example, you can not just optimize away
smalltalks context chain by replacing it with native stack frames, without losing the features a smallalltak method context provides on the language level.
This language features are already explained very well in the blue book, but the links about the cog vm and other vm and jit vm papers for dynamic langueges are helpful too.

I don't know which language features (late bindng, method lookup , thisContext / execution context, become operation, etc) are *necessary* to provide the "smalltalk art of live coding".
Or to what extent this features exists in other language. But it is one reason why it is "easy" to build within smalltalk itself this smalltalk environment (edit / compile/ run / debug smallalk code from within a running smalltalk environment).



Reply | Threaded
Open this post in threaded view
|

Re: Recreating live coding in CPython 3.5

kilon.alios
 
The problem in my case is that I do not build a VM from scratch neither I am modifying an existing VM 

For me it boils down to 

1) What is this in Smalltalk ?
2) Can Python do this ?
3) Can I make it easier without spending too much time and compromising ?

Python has its own stack frames, they are object too, not native stack frames (they may reference native stack frames though) and if I am not mistaken because unlike my previous statements I have not messed at all with them, they also come with their own context which includes local variables etc 


I can make to a degree such claims because anything I test live coding wise in python I do it from debugger. The Python debugger is called PDB as you would expect and it works from the terminal , beyond coming with its own commands and can do repl suff it also comes with an instance of Python REPL which of course gives you complete freedom in doing what you would do in source code file. This is how I hack objects removing and adding methods to them , mess with local variables etc. talking about live instance objects etc. 

Of course I have to copy the code manually back to the source file , maybe there is a way around this as well or some feature I am not aware of, but the code fragments are usually small so its not a major problem for now. LiveCoding wise, besides the fact that indeed some things do not happen automagically as you would expect in Pharo still the debugger has not stopped me from doing anything live coding wise. 

My only big frustration so far has been the fact that classes do not keep a list of their instances, but that is also easy to amend for my own classes adding their instances when it is created in class variable list or dictionary. Technically you can still get all references to an object (via gc module coming included with python) and ask each one if it is an instance of a class , still I think thats a big flaw and introduces extra works that should not happen when Python already tackles far more complex tasks of live coding. 

But that's your usual language I guess, there is always something to annoy you. 

In any case, I am not dismissing anything and of course I will read and delve deeper live coding related or not, so thanks for any help you guys give even if I do not need it. I may not need it now , but I may need it later on, so learning is always worth it.   But I am also careful not to recreate Python functionality because a) I will be wasting my time b) I will be ignoring Python's native design that can create incompatibilities down the road. 

So gain a good understanding from both sides Cog VM and Python library (so far I had no need to seek information about Python VM because the functionality I want may not be completely automated but is non the less available as a library part of the standard distribution)

So the deeper I get , of course I expect to get more complex, obviously for now I want to keep it simple and then move step by step to something more sophisticated. But yes I have read the material, not extensively but I did gave it a fast read and I will re read again and I will be also reading the Bluebook. 

On Thu, Oct 12, 2017 at 12:58 PM Nicolai Hess <[hidden email]> wrote:
2017-10-12 10:50 GMT+02:00 Dimitris Chloupis <[hidden email]>:
 
Thanks Ben actually your first link was the first thing I read when I googled become. I dont think VM lookup benefits me in my case as this is already handled by the Python VM. 

You other tlinks are JIT VM related, Python is not friendly to JIT because its creator believe that performance enhancement should come from wrapping C libraries and wants to avoid the cost of more complex VM . In the case of Python this is easy to follow because Python has a ton of high performance libraries which by the way almost never are used when comparing Python with other languages speed wise (kinda fare if you just compare language implementations). Pharo does benefit a lot more from JIT VM as it does not have as many libraries targeting high performance. 


The interesting parts in this "JIT VM related" links is, if you happen to write a JIT VM , you have to take care of the language features smalltalk provides. For example, you can not just optimize away
smalltalks context chain by replacing it with native stack frames, without losing the features a smallalltak method context provides on the language level.
This language features are already explained very well in the blue book, but the links about the cog vm and other vm and jit vm papers for dynamic langueges are helpful too.

I don't know which language features (late bindng, method lookup , thisContext / execution context, become operation, etc) are *necessary* to provide the "smalltalk art of live coding".
Or to what extent this features exists in other language. But it is one reason why it is "easy" to build within smalltalk itself this smalltalk environment (edit / compile/ run / debug smallalk code from within a running smalltalk environment).



bpi
Reply | Threaded
Open this post in threaded view
|

Re: Recreating live coding in CPython 3.5

bpi
Reply | Threaded
Open this post in threaded view
|

Re: Recreating live coding in CPython 3.5

kilon.alios
 
Thanks Berhard

I managed to accomplish a full live coding environment in Python and I am now in the process turning this to a library, to automate all the steps required.

It was with your help I realized my mistakes , especially that post that pointed me to the python implementation of become, guided me towards the solution

My conclusion at least from a user perspective that Python is already a live coding environment, that someone has turned off.

So in order for old instances to be updated there is no need to mess with their objects and hence no need to implement a become functionality. Python allows to change the methods of an old instance with methods of a new updated instances. However this is no solution because other old instances of same class are not updated.

Not to bore you with technical details but the actual solution is to access the references to old classes and update their methods to the methods of the new classes. If that happens automagically all old instances get the new methods and there is no reason for extra steps.

This workflow work before compilation, after compilation and of course the most important during execution. Essentially this accomplishes live coding going the opposite direction of become. It applies for any kind of Repl and any kind of debugger. I have tried python repl and python debugger as well as iPython repl and iPython debugger. No problem so far.

This works also for superclasses and subclasses because Python uses Singleton references wherever it can unless of course instructed otherwise by the coder. So I don’t have to worry about inheritance issues.

I am a bit worried about creating dangling references but this would require some stress testing to inspect how well the garbage collector works but I don’t expect major issues because I am generating new objects with empty state while old state is retained in its place. Of course state can be manipulated lively through live coding as one would expect.

I don’t see any other problem and what is left now is to gather all my tests under a single library. The library will detect when a source file got modified and follow the above workflow.

I would like to thank you all for your assistance I am very happy that accomplished this goal.
On Sat, 14 Oct 2017 at 15:29, Bernhard Pieber <[hidden email]> wrote:

Hi Dimitris,

This conference article might be relevant:
https://2017.programming-conference.org/event/morevms-2017-papers-when-a-mouse-eats-a-python-smalltalk-style-development-for-python-and-ruby

Cheers,
Bernhard
12