Smalltalk › Pharo › Pharo Smalltalk Users

Extending live coding beyond Pharo

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

3 messages Options

kilon.alios

Extending live coding beyond Pharo

Live coding on Pharo is great, easy , simple and working out of the box. An extremely useful tool to be able to code an application while it runs, change and improve code with no interruptions.

However there will be cases that Pharo wont be enough, that can be because pharo code is not as fast as you want it to be, or plainly you found a great library in C or C++ you wanna use from Pharo and there is literally a ton of them. However there is a catch , you kiss goodbye to live coding , you say hello to slow compiles and crashes because of bad memory management.

Unfortunately that is the nature of C/C++. However live coding is gaining ground lately and more and more languages use it for super fast development. Some C++ coders have wondered whether they can bring this feature back to C++ and they have succeed.

One example is Unreal game engine I am using .It can compile code on the fly and load it back even though the game is keep running. It calls this process "hot reloading".

But for people that dont want to bother with Unreal there is this project

http://runtimecompiledcplusplus.blogspot.gr/

The nice things about this is that not only it allows for live coding and very fast compiles but also its able to catch crashes with system exceptions, in a case of a crash the application keeps running because the crashy code is exchanged with the previous version of the code that did not crash and the crash is captured with an exception, you can use the debugger to debug it , correct the code and continue like you would with Pharo.

There are disadvantages though, because RCPP makes no compromises when it comes to performance that means that it needs a very specific way of coding to accomodate for live coding, usually a main loop should exist and there is an API that must be used to track changes to the files and reload code on demand. But generally it does work very well and its also used by commercial projects all around the world.

This will be a real nice addition to the toolbox of people who like to extend Pharo with C/C++ but do not want to live the comforts of live coding and have low tolerance for slow compile times and crashes.

Ben Coman

Re: Extending live coding beyond Pharo

On Thu, Sep 15, 2016 at 3:38 AM, Dimitris Chloupis
<[hidden email]> wrote:

> Live coding on Pharo is great, easy , simple and working out of the box. An
> extremely useful tool to be able to code an application while it runs,
> change and improve code with no interruptions.
>
> However there will be cases that Pharo wont be enough, that can be because
> pharo code is not as fast as you want it to be, or plainly you found a great
> library in C or C++ you wanna use from Pharo and there is literally a ton of
> them. However there is a catch , you kiss goodbye to live coding , you say
> hello to slow compiles and crashes because of bad memory management.
>
> Unfortunately that is the nature of C/C++. However live coding is gaining
> ground lately and more and more languages use it for super fast development.
> Some C++ coders have wondered whether they can bring this feature back to
> C++ and they have succeed.
>
> One example is Unreal game engine I am using .It can compile code on the fly
> and load it back even though the game is keep running. It calls this
> process "hot reloading".
>
> But for people that dont want to bother with Unreal there is this project
>
> http://runtimecompiledcplusplus.blogspot.gr/
>
> The nice things about this is that not only it allows for live coding and
> very fast compiles but also its able to catch crashes with system
> exceptions, in a case of a crash the application keeps running because the
> crashy code is exchanged with the previous version of the code that did not
> crash and the crash is captured with an exception, you can use the debugger
> to debug it , correct the code and continue like you would with Pharo.
>
> There are disadvantages though, because RCPP makes no compromises when it
> comes to performance that means that it needs a very specific way of coding
> to accomodate for live coding, usually a main loop should exist and there is
> an API that must be used to track changes to the files and reload code on
> demand. But generally it does work very well and its also used by commercial
> projects all around the world.
>
> This will be a real nice addition to the toolbox of people who like to
> extend Pharo with C/C++ but do not want to live the comforts of live coding
> and have low tolerance for slow compile times and crashes.

Nice idea. I've been wondering if there might be some way of having a
SafeFFI for use during development and by newbies. After all, gdb
doesn't crash when the program its hosting crashes I guess there is
*some* trick could be used. It could even be a *lot* slower and still
be useful, to be turned off at deployment.

Actually just last night I was contemplating what a Pharo system might
be like with FFI-libclang dynamically compiling in-Image C code,
(re-)linking it into a dynamic library and immediately calling those
functions using FFI again. My very first graduate programming job was
a company developing satellite image processing software that got a
lot of their speed by generating/compiling/linking/loading C functions
for filter-chains to apply to the huge data sets.

Maybe something like...

Behavior subclass: #CLanguageDescription
instanceVariableNames: 'index translationUnit'
classVariableNames: ''
package: 'ExternalLanguage-C'
"Provide compatibility to integrate equivalent of C files in our
System Browser.
- class-pane==>C-file, method-pane==>types and functions"

ProtoObject subclass: #CLanguageRootHierarchy
instanceVariableNames: ''
classVariableNames: ''
package: 'ExternalLanguage-C'
"My subclasses are instances of CLanguageDescription compiled
by a C compiler such that pure C syntax can be used.
i.e. positional function arguments f(a,b)"

At a minimum, it might bring FFI function callout definitions closer
to pure cut and paste. (Although the current UFFI callout format is
pretty good.)

A *lot* further stretch... you might store all the VM platform sources
inside the image, as well as the generated C code from slang
primitives. Updating the slang method immediately re-generates the C
code, compiles it and updates the dynamic library. Dependencies
could be tracked at a finer level that standard file based makefile,
such that an update to a platform C type or function declaration would
be linked to other types and functions using (based on AST analysis
and the minimum set recompiled immediately. Unit tests for platform
sources would be Smalltalk based FFI callouts invoked from TestRunner.

(But I know... its easy to dream. harder to do.)
cheers -ben

kilon.alios

Re: Extending live coding beyond Pharo

It's not a trick or slow , it's actually a standard feature and blazing fast. Its a feature that has existed for several decades now . It's called "exception". There are three types of exceptions.

1) language exception. This type of exception is handled by the language and it can involve both an error or even a normal running code. Usually they are handled by a library or VM.

2) OS system exceptions. This exception is handled by the operation system itself and its what debuggers usually use to capture crashes. In Windows they are called "structured exceptions"

https://msdn.microsoft.com/en-us/library/windows/desktop/ms680657(v=vs.85).aspx

In UNIX type OS they are called "signals and traps"

http://www.tutorialspoint.com/unix/unix-signals-traps.htm

3) hardware exception. This exception is handled directly by the CPU.

http://www.internals.com/articles/protmode/interrupts.htm

If you are interested in the internals of live coding and exception handling this guy made a series of videos how to make a game from scratch using no library and has 3 videos on the subject that show himself coding these features

https://m.youtube.com/watch?v=WMSBRk5WG58

https://m.youtube.com/watch?v=oijEnriqqcs

https://m.youtube.com/watch?v=xrUSrVvB21c

The only trick is not to let the DLL have any of its own memory, instead the main executable keeps its memory in one place, in this case a struct is used, and the memory is passed as reference via a pointer to DLL. The DLL in turn passes pointers to its functions to the executable, and the executable passes also pointers to its functions to the DLL that the DLL may want to use as callbacks. In short code and memory is passed around only as reference.

This is just very basic C/C++ coding using DLLs nothing special about it.

Storing the code inside Pharo image is very easy todo , after all we store graphics files in the image. But it's not really necessary or useful. If you decide to keep everything in image you lose a lot of tools that can make your life a lot easier, like coding tools, debuggers etc that averages C coder uses.

If you want to do trigger those things from inside Pharo you can use OSProcess to do so as we do with git , see gitfiletree.

If you want real time compiling without the need to force the user to install a compiler or increase the size of the Pharo installation you can use CI or a online service that provides compilations of DLLs via the Internet. Pharo of course already has this ability because it uses CI to build its VM and image in order to provide you with a download link.

So basically you have all those tools already available to you.

Tracking changes can happen at file level so you don't need to do that in image either. The videos I linked clearly illustrate the process.

The only catch is that you cannot makes this automatic because live coding and real time compiling depends a lot on the structure of the code and this is not something you can predict. Also it's a good idea the user knows what he is doing because else he will run into trouble fast. For example people who use Slang are VM developers and they both know the VM internals and how to code in C. The same way that in order to code in Pharo you have to know Pharo.

If you don't want to hack the Pharo executable then you can simple use one DLL to load another DLL , this way the first DLL will act as container/sandbox for the second DLL and replicate much of the same workflow I mentioned for the executable. After all DLLs are very similar to executables.

On Thu, 15 Sep 2016 at 06:03, Ben Coman <[hidden email]> wrote:

On Thu, Sep 15, 2016 at 3:38 AM, Dimitris Chloupis
<[hidden email]> wrote:
> Live coding on Pharo is great, easy , simple and working out of the box. An
> extremely useful tool to be able to code an application while it runs,
> change and improve code with no interruptions.
>
> However there will be cases that Pharo wont be enough, that can be because
> pharo code is not as fast as you want it to be, or plainly you found a great
> library in C or C++ you wanna use from Pharo and there is literally a ton of
> them. However there is a catch , you kiss goodbye to live coding , you say
> hello to slow compiles and crashes because of bad memory management.
>
> Unfortunately that is the nature of C/C++. However live coding is gaining
> ground lately and more and more languages use it for super fast development.
> Some C++ coders have wondered whether they can bring this feature back to
> C++ and they have succeed.
>
> One example is Unreal game engine I am using .It can compile code on the fly
> and load it back even though the game is keep running. It calls this
> process "hot reloading".
>
> But for people that dont want to bother with Unreal there is this project
>
> http://runtimecompiledcplusplus.blogspot.gr/
>
> The nice things about this is that not only it allows for live coding and
> very fast compiles but also its able to catch crashes with system
> exceptions, in a case of a crash the application keeps running because the
> crashy code is exchanged with the previous version of the code that did not
> crash and the crash is captured with an exception, you can use the debugger
> to debug it , correct the code and continue like you would with Pharo.
>
> There are disadvantages though, because RCPP makes no compromises when it
> comes to performance that means that it needs a very specific way of coding
> to accomodate for live coding, usually a main loop should exist and there is
> an API that must be used to track changes to the files and reload code on
> demand. But generally it does work very well and its also used by commercial
> projects all around the world.
>
> This will be a real nice addition to the toolbox of people who like to
> extend Pharo with C/C++ but do not want to live the comforts of live coding
> and have low tolerance for slow compile times and crashes.

Nice idea. I've been wondering if there might be some way of having a
SafeFFI for use during development and by newbies. After all, gdb
doesn't crash when the program its hosting crashes I guess there is
*some* trick could be used. It could even be a *lot* slower and still
be useful, to be turned off at deployment.

Actually just last night I was contemplating what a Pharo system might
be like with FFI-libclang dynamically compiling in-Image C code,
(re-)linking it into a dynamic library and immediately calling those
functions using FFI again. My very first graduate programming job was
a company developing satellite image processing software that got a
lot of their speed by generating/compiling/linking/loading C functions
for filter-chains to apply to the huge data sets.

Maybe something like...

Behavior subclass: #CLanguageDescription
instanceVariableNames: 'index translationUnit'
classVariableNames: ''
package: 'ExternalLanguage-C'
"Provide compatibility to integrate equivalent of C files in our
System Browser.
- class-pane==>C-file, method-pane==>types and functions"

ProtoObject subclass: #CLanguageRootHierarchy
instanceVariableNames: ''
classVariableNames: ''
package: 'ExternalLanguage-C'
"My subclasses are instances of CLanguageDescription compiled
by a C compiler such that pure C syntax can be used.
i.e. positional function arguments f(a,b)"

At a minimum, it might bring FFI function callout definitions closer
to pure cut and paste. (Although the current UFFI callout format is
pretty good.)

A *lot* further stretch... you might store all the VM platform sources
inside the image, as well as the generated C code from slang
primitives. Updating the slang method immediately re-generates the C
code, compiles it and updates the dynamic library. Dependencies
could be tracked at a finer level that standard file based makefile,
such that an update to a platform C type or function declaration would
be linked to other types and functions using (based on AST analysis
and the minimum set recompiled immediately. Unit tests for platform
sources would be Smalltalk based FFI callouts invoked from TestRunner.

(But I know... its easy to dream. harder to do.)
cheers -ben