using UFFI in non-crashing way

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

using UFFI in non-crashing way

Peter Uhnak
Hi,

is it possible to use UFFI and avoid crashing the image when the called code segfaults?

In other words, can I somehow wrap the call and throw in-image Exception/Error on failure instead of the whole thing crashing?

Or is the only way to do that have the called thing running as a separate app (e.g. via OSSubProcess) and communicate via pipes or sockets?

Thanks,
Peter

Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

EstebanLM

> On 26 Mar 2017, at 15:40, Peter Uhnak <[hidden email]> wrote:
>
> Hi,
>
> is it possible to use UFFI and avoid crashing the image when the called code segfaults?
>
> In other words, can I somehow wrap the call and throw in-image Exception/Error on failure instead of the whole thing crashing?

that’s not possible, once you segfault, you segfault.

> Or is the only way to do that have the called thing running as a separate app (e.g. via OSSubProcess) and communicate via pipes or sockets?

that’s not done.
no idea how much effort is required to implement, but I imagine is not trivial.

Esteban

>
> Thanks,
> Peter
>


Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

kilon.alios
In reply to this post by Peter Uhnak

Hi,

is it possible to use UFFI and avoid crashing the image when the called code segfaults?


Yes its possible. When I was exploring the means to live code with C++ I came to specific implementations of live coding for C++. One of them explored a feature called "Hardware exceptions" .Hardware exceptions are very similar to regular exception with the big difference that are handled by the CPU and not the language or the OS. 

Crashes is a standard behavior that is imposed by the OS kernel in the case that an app tries to access memory is not allowed to access or access it in the way it is not allowed to access it. An exception will capture this behavior it wont allow the application to crash.

Hardware exceptions are a standard feature for most CPUs and obviously they are not implemented at language level.

For example you can find documentation about Hardware exceptions on Windows at

https://msdn.microsoft.com/en-us/library/windows/desktop/ms680657(v=vs.85).aspx

This means the C code you will be calling with UFFI will have to be wrapped inside a Hardware exception.

When a crash is suppose to happen inside code that is wrapped with a hardware exception, the error (seffault in your example) will be displayed in the console but the application wont crash it will continue to execute depending on how you handle the exception.

Generally hardware exception are to be avoided because they can hide bugs because the OS not always reports and can crash an app without an error in that you will be oblivious to what went wrong, when and how as the crash will not happen.

 
In other words, can I somehow wrap the call and throw in-image Exception/Error on failure instead of the whole thing crashing?


Its possible if you make a C function that contains a hardware exception take as argument a pointer to a specific function and then calls it inside the exception.
 
Or is the only way to do that have the called thing running as a separate app (e.g. via OSSubProcess) and communicate via pipes or sockets?

It wont make a diffirence if the crash is caused by UFFI it will still crash Pharo that applies even if you use hardware exceptions , so you need always to be careful.

Plus Pharo VM at any point can crash for its own reason, that is completely unrelated to what you are doing

Thanks,
Peter

So yes its possible, but you still need to be careful. 
Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

David T. Lewis
In reply to this post by EstebanLM
On Sun, Mar 26, 2017 at 05:45:04PM +0200, Esteban Lorenzano wrote:

>
> > On 26 Mar 2017, at 15:40, Peter Uhnak <[hidden email]> wrote:
> >
> > Hi,
> >
> > is it possible to use UFFI and avoid crashing the image when the called code segfaults?
> >
> > In other words, can I somehow wrap the call and throw in-image Exception/Error on failure instead of the whole thing crashing?
>
> that???s not possible, once you segfault, you segfault.
>
> > Or is the only way to do that have the called thing running as a separate app (e.g. via OSSubProcess) and communicate via pipes or sockets?
>
> that???s not done.
> no idea how much effort is required to implement, but I imagine is not trivial.
>

I have not tried it with FFI calls, but maybe this will do what you want:

  RemoteTask do: [ 2 + 2 ] ==> 4

Instead of [ 2 + 2 ] use some block containing your FFI calls.

It works in Pharo. To get RemoteTask, load OSProcess and CommandShell.
You also need Fuel, but I think that is in the image by default.

An explanation of RemoteTask is at http://wiki.squeak.org/squeak/6176.
You are right, it was not trivial to implement ;-)

HTH,
Dave
 

Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

Peter Uhnak
In reply to this post by EstebanLM
On Sun, Mar 26, 2017 at 05:45:04PM +0200, Esteban Lorenzano wrote:

>
> > On 26 Mar 2017, at 15:40, Peter Uhnak <[hidden email]> wrote:
> >
> > Hi,
> >
> > is it possible to use UFFI and avoid crashing the image when the called code segfaults?
> >
> > In other words, can I somehow wrap the call and throw in-image Exception/Error on failure instead of the whole thing crashing?
>
> that’s not possible, once you segfault, you segfault.

Ah, ok.

>
> > Or is the only way to do that have the called thing running as a separate app (e.g. via OSSubProcess) and communicate via pipes or sockets?
>
> that’s not done.
> no idea how much effort is required to implement, but I imagine is not trivial.

Nono, I didn't mean making UFFI calls via socket (although that would be interesting), but rather making a C app that would make the calls, and that would provide a socket interface. (Which is what I typically do with other langs).

Thanks,
Peter

Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

Thierry Goubier
Le 26/03/2017 à 19:21, Peter Uhnak a écrit :

> On Sun, Mar 26, 2017 at 05:45:04PM +0200, Esteban Lorenzano wrote:
>>
>>> On 26 Mar 2017, at 15:40, Peter Uhnak <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> is it possible to use UFFI and avoid crashing the image when the
>>> called code segfaults?
>>>
>>> In other words, can I somehow wrap the call and throw in-image
>>> Exception/Error on failure instead of the whole thing crashing?
>>
>> that’s not possible, once you segfault, you segfault.
>
> Ah, ok.
>
>>
>>> Or is the only way to do that have the called thing running as a
>>> separate app (e.g. via OSSubProcess) and communicate via pipes or
>>> sockets?
>>
>> that’s not done. no idea how much effort is required to implement,
>> but I imagine is not trivial.
>
> Nono, I didn't mean making UFFI calls via socket (although that would
> be interesting), but rather making a C app that would make the calls,
> and that would provide a socket interface. (Which is what I typically
> do with other langs).

With UFFI and a standard way to marshall types over a socket (ASN.1 or
even xml-rpc), this could be generated automatically.

Of course, you may have a few performance issues, but you could have a
long-lived child to avoid paying the startup time, and mmap / shared
memory to lower the cost of moving memory back and forth.

Thierry

> Thanks, Peter
>
>


Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

Guillermo Polito
In reply to this post by Peter Uhnak


On Sun, Mar 26, 2017 at 7:21 PM, Peter Uhnak <[hidden email]> wrote:
On Sun, Mar 26, 2017 at 05:45:04PM +0200, Esteban Lorenzano wrote:
>
> > On 26 Mar 2017, at 15:40, Peter Uhnak <[hidden email]> wrote:
> >
> > Hi,
> >
> > is it possible to use UFFI and avoid crashing the image when the called code segfaults?
> >
> > In other words, can I somehow wrap the call and throw in-image Exception/Error on failure instead of the whole thing crashing?
>
> that’s not possible, once you segfault, you segfault.

Ah, ok.

Well, you can capture the signal, but the problem is not the signal itself but what caused the problem. The most common case of segfaut is memory corruption (and thus a program that tries to wrongly interpret wrong addresses). If you already corrupted the memory doing something wrong, how can you "fix it"?
 

>
> > Or is the only way to do that have the called thing running as a separate app (e.g. via OSSubProcess) and communicate via pipes or sockets?
>
> that’s not done.
> no idea how much effort is required to implement, but I imagine is not trivial.

Nono, I didn't mean making UFFI calls via socket (although that would be interesting), but rather making a C app that would make the calls, and that would provide a socket interface. (Which is what I typically do with other langs).

Thanks,
Peter


Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

Denis Kudriashov

2017-03-27 11:34 GMT+02:00 Guillermo Polito <[hidden email]>:
Well, you can capture the signal, but the problem is not the signal itself but what caused the problem. The most common case of segfaut is memory corruption (and thus a program that tries to wrongly interpret wrong addresses). If you already corrupted the memory doing something wrong, how can you "fix it"?

I think important case here is creating wrappers for new C libraries. Common mistake is passing wrong structures to foreign function. Then image crashes and you need restart it and recover code - big waste of time and losing smalltalk liveness.
Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

Guillermo Polito


On Mon, Mar 27, 2017 at 12:04 PM, Denis Kudriashov <[hidden email]> wrote:

2017-03-27 11:34 GMT+02:00 Guillermo Polito <[hidden email]>:
Well, you can capture the signal, but the problem is not the signal itself but what caused the problem. The most common case of segfaut is memory corruption (and thus a program that tries to wrongly interpret wrong addresses). If you already corrupted the memory doing something wrong, how can you "fix it"?

I think important case here is creating wrappers for new C libraries. Common mistake is passing wrong structures to foreign function. Then image crashes and you need restart it and recover code - big waste of time and losing smalltalk liveness.

But then the problem is on "how to not lose your code", not in FFI... 

Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

philippeback


On Mon, Mar 27, 2017 at 1:01 PM, Guillermo Polito <[hidden email]> wrote:


On Mon, Mar 27, 2017 at 12:04 PM, Denis Kudriashov <[hidden email]> wrote:

2017-03-27 11:34 GMT+02:00 Guillermo Polito <[hidden email]>:
Well, you can capture the signal, but the problem is not the signal itself but what caused the problem. The most common case of segfaut is memory corruption (and thus a program that tries to wrongly interpret wrong addresses). If you already corrupted the memory doing something wrong, how can you "fix it"?

I think important case here is creating wrappers for new C libraries. Common mistake is passing wrong structures to foreign function. Then image crashes and you need restart it and recover code - big waste of time and losing smalltalk liveness.

But then the problem is on "how to not lose your code", not in FFI... 

Am spending my time restarting my image due to this. Very frustrating to recreate the setup etc.

Havign a way to detect go back one frame would be very handy.

Phil

Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

Denis Kudriashov
In reply to this post by Guillermo Polito

2017-03-27 13:01 GMT+02:00 Guillermo Polito <[hidden email]>:
But then the problem is on "how to not lose your code", not in FFI... 

Restarting image is a problem. For any other "smalltalk" problems we got suitable debugger but not in that case.
Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

Denis Kudriashov

2017-04-06 10:37 GMT+02:00 Denis Kudriashov <[hidden email]>:
But then the problem is on "how to not lose your code", not in FFI... 

Restarting image is a problem. For any other "smalltalk" problems we got suitable debugger but not in that case.

Also when image crashed we need to recover all objects state for the script which produce crash
Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

EstebanLM

On 6 Apr 2017, at 10:39, Denis Kudriashov <[hidden email]> wrote:


2017-04-06 10:37 GMT+02:00 Denis Kudriashov <[hidden email]>:
But then the problem is on "how to not lose your code", not in FFI... 

Restarting image is a problem. For any other "smalltalk" problems we got suitable debugger but not in that case.

Also when image crashed we need to recover all objects state for the script which produce crash

FFI is a border.
Once you start working with it, you are in C world. 
And C world is nasty, hard and cumbersome.

In C world, pointers not correctly alloc’d produce a crash. 
In C world, pointers not correctly freed also produce a crash, eventually (and those are harder to fix). 
In C world everything pointing to wrong location will produce a crash (and you can point anything to anywhere). 

That’s why garbage collectors were invented, after all. 

So my only advice here is: 

- recheck all your allocs/free, rigorously
- save your image before test
- be sure epicea is working 

Finally, I would say the only way to prevent your image to crash is to not execute FFI calls inside it. To do this, you can always use ImageWorker (http://smalltalkhub.com/#!/~PharoExtras/ImageWorker) to actually execute your tests in a different execution. Yes is slow, but it will be safer. 

Esteban
Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

Denis Kudriashov

2017-04-07 8:22 GMT+02:00 Esteban Lorenzano <[hidden email]>:
But then the problem is on "how to not lose your code", not in FFI... 

Restarting image is a problem. For any other "smalltalk" problems we got suitable debugger but not in that case.

Also when image crashed we need to recover all objects state for the script which produce crash

FFI is a border.
Once you start working with it, you are in C world. 
And C world is nasty, hard and cumbersome.

Yes. And my remark was about possibility to catch C error to prevent image crash, why it will make FFI less hard.
Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

kilon.alios
Do you guys received my CPU exceptions solution message ? Because the issue you described is already resolved by hardware. Simple and elegant solution too
On Fri, 7 Apr 2017 at 11:03, Denis Kudriashov <[hidden email]> wrote:

2017-04-07 8:22 GMT+02:00 Esteban Lorenzano <[hidden email]>:
But then the problem is on "how to not lose your code", not in FFI... 

Restarting image is a problem. For any other "smalltalk" problems we got suitable debugger but not in that case.

Also when image crashed we need to recover all objects state for the script which produce crash

FFI is a border.
Once you start working with it, you are in C world. 
And C world is nasty, hard and cumbersome.

Yes. And my remark was about possibility to catch C error to prevent image crash, why it will make FFI less hard.
Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

raffaello.giulietti
On 07/04/17 18:28, Dimitris Chloupis wrote:

> Do you guys received my CPU exceptions solution message ? Because the
> issue you described is already resolved by hardware. Simple and elegant
> solution too
> On Fri, 7 Apr 2017 at 11:03, Denis Kudriashov <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>     2017-04-07 8:22 GMT+02:00 Esteban Lorenzano <[hidden email]
>     <mailto:[hidden email]>>:
>
>>                 But then the problem is on "how to not lose your
>>                 code", not in FFI...
>>
>>
>>             Restarting image is a problem. For any other "smalltalk"
>>             problems we got suitable debugger but not in that case.
>>
>>
>>         Also when image crashed we need to recover all objects state
>>         for the script which produce crash
>
>         FFI is a border.
>         Once you start working with it, you are in C world.
>         And C world is nasty, hard and cumbersome.
>
>
>     Yes. And my remark was about possibility to catch C error to prevent
>     image crash, why it will make FFI less hard.
>


My experience with the UFFI so far is quite positive. Of course, a lot
of care needs to go to the C part, but that's a fact of life with C's
unsafe computational model.

So, if the C code is well written and thoroughly tested, possibly
outside Pharo, then integrating it by means of the UFFI is rather
straightforward. I agree that small errors using the UFFI can lead to
spurious or catastrophic outcomes, so unit (or even smaller scoped)
testing really becomes vital.

From what I can tell so far, the UFFI is fast, well performing and, with
some care, not even so hard as it might seem at first sight.


Greetings
Raffaello

Reply | Threaded
Open this post in threaded view
|

Re: using UFFI in non-crashing way

Ben Coman
On Sun, May 7, 2017 at 7:45 PM,  <[hidden email]> wrote:

> On 07/04/17 18:28, Dimitris Chloupis wrote:
>> Do you guys received my CPU exceptions solution message ? Because the
>> issue you described is already resolved by hardware. Simple and elegant
>> solution too
>> On Fri, 7 Apr 2017 at 11:03, Denis Kudriashov <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>
>>     2017-04-07 8:22 GMT+02:00 Esteban Lorenzano <[hidden email]
>>     <mailto:[hidden email]>>:
>>
>>>                 But then the problem is on "how to not lose your
>>>                 code", not in FFI...
>>>
>>>
>>>             Restarting image is a problem. For any other "smalltalk"
>>>             problems we got suitable debugger but not in that case.
>>>
>>>
>>>         Also when image crashed we need to recover all objects state
>>>         for the script which produce crash
>>
>>         FFI is a border.
>>         Once you start working with it, you are in C world.
>>         And C world is nasty, hard and cumbersome.
>>
>>
>>     Yes. And my remark was about possibility to catch C error to prevent
>>     image crash, why it will make FFI less hard.
>>
>
>
> My experience with the UFFI so far is quite positive. Of course, a lot
> of care needs to go to the C part, but that's a fact of life with C's
> unsafe computational model.
>
> So, if the C code is well written and thoroughly tested, possibly
> outside Pharo, then integrating it by means of the UFFI is rather
> straightforward. I agree that small errors using the UFFI can lead to
> spurious or catastrophic outcomes, so unit (or even smaller scoped)
> testing really becomes vital.
>
> From what I can tell so far, the UFFI is fast, well performing and, with
> some care, not even so hard as it might seem at first sight.
>
>
> Greetings
> Raffaello
>

btw, I started a UFFI / Clang tutorial here (that I need to get back to)
http://blog.openinworld.com/2016/09/pharo-libclang-ffi-part-1-preamble/

cheers -ben