There are something fishy with FFI plugin

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

There are something fishy with FFI plugin

Torsten Bergmann
 
Load latest from ConfigurationOfFFI from MetacellRepository:

For me the FFIPluginTest fails with latest Pharo 1.4 VM
- so at least one has a reproducable case.

Thx
T.
--
NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!                                  
Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
Reply | Threaded
Open this post in threaded view
|

Re: There are something fishy with FFI plugin

Igor Stasenko
 
On 24 April 2012 19:16, Torsten Bergmann <[hidden email]> wrote:
>
> Load latest from ConfigurationOfFFI from MetacellRepository:
>
> For me the FFIPluginTest fails with latest Pharo 1.4 VM
> - so at least one has a reproducable case.
>
Esteban spent whole day today trying to find the offending bug.
Source code match 1:1 with Eliot's code, because there was no any
changes in it..
still the results is that it refuses to work correctly on windoze.

> Thx
> T.
> --
> NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!
> Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a



--
Best regards,
Igor Stasenko.
Reply | Threaded
Open this post in threaded view
|

Re: There are something fishy with FFI plugin

Eliot Miranda-2
 


On Tue, Apr 24, 2012 at 2:53 PM, Igor Stasenko <[hidden email]> wrote:

On 24 April 2012 19:16, Torsten Bergmann <[hidden email]> wrote:
>
> Load latest from ConfigurationOfFFI from MetacellRepository:
>
> For me the FFIPluginTest fails with latest Pharo 1.4 VM
> - so at least one has a reproducable case.
>
Esteban spent whole day today trying to find the offending bug.
Source code match 1:1 with Eliot's code, because there was no any
changes in it..
still the results is that it refuses to work correctly on windoze.

You need to understand what the ThreadedFFIPlugin tries to do with alloca and that various compilers may implement alloca in a number of different ways.  You then need to look at what the compiler actually does and figure out how to subvert it to do what you want to do.

1.  the ThreadedFFIPlugin uses alloca to stack-allocate the space for the outgoing parameters of the FFI call.  Sicne on x86 all parameters are passed on the stack and none in registers alloca is ideal.  Notionally the plugin can call alloca to extend the stack downwards (stacks grow down, the result answered by alloca is a pointer to the start of N bytes, hence it is effectively the stack pointer. If this doesn't make sense to you *draw a picture on a whiteboard* of what you expect alloca(29) to do)

2. some C compilers are nice and implement alloca(N) as effectively stackPointer := stackPointer - roundUpTo(N,StackAlignment).  For these compilers we don't need to do anything.  Some C compilers try and be clever and answer stackPointer + 4, or something else.  In these cases we need to get the actual stack pointer.  So see the code ThreadedFFIPlugin class>>preambleCCode.

3.  look at what the C compiler produces for the alloca call (and/or run under gdb) to see what happens to the stack pointer and what ends up in the allocation var in ffiCall:ArgArrayOrNil:NumArgs:.

4. update ThreadedFFIPlugin class>>preambleCCode. appropriately until you've bent alloca and the C compiler to your will.

You will not make this work without trying to understand what is going on and looking at what your C compiler produces.  This is low-level hackery that aims to avoid having to write any assembler (outside of very few C asm statements).

HTH
Eliot


> Thx
> T.
> --
> NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!
> Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a



--
Best regards,
Igor Stasenko.



--
best,
Eliot

Reply | Threaded
Open this post in threaded view
|

Re: There are something fishy with FFI plugin

EstebanLM
 
Hi Eliot,

Ok, after spending 2.5 days on this (but well... now I know a lot about this stuff, he), I found that the problem with our compilation is that two flags are needed: 

STACK_ALIGN_BYTES=16
ALLOCA_LIES_SO_USE_GETSP=0 //Yep, alloca is not lying here

problem is... while first flag can be set by command line, second one is setted on preamble for ThreadedFFIPlugin, then I need to change it... 

so, here my question: 

Why is working as is for you and not for us? (I think both of us are using mingw, and I installed gcc 3.4.5 to use same version as you) Do I'm missing something?

best,
Esteban

On Apr 25, 2012, at 1:20 AM, Eliot Miranda wrote:



On Tue, Apr 24, 2012 at 2:53 PM, Igor Stasenko <[hidden email]> wrote:

On 24 April 2012 19:16, Torsten Bergmann <[hidden email]> wrote:
>
> Load latest from ConfigurationOfFFI from MetacellRepository:
>
> For me the FFIPluginTest fails with latest Pharo 1.4 VM
> - so at least one has a reproducable case.
>
Esteban spent whole day today trying to find the offending bug.
Source code match 1:1 with Eliot's code, because there was no any
changes in it..
still the results is that it refuses to work correctly on windoze.

You need to understand what the ThreadedFFIPlugin tries to do with alloca and that various compilers may implement alloca in a number of different ways.  You then need to look at what the compiler actually does and figure out how to subvert it to do what you want to do.

1.  the ThreadedFFIPlugin uses alloca to stack-allocate the space for the outgoing parameters of the FFI call.  Sicne on x86 all parameters are passed on the stack and none in registers alloca is ideal.  Notionally the plugin can call alloca to extend the stack downwards (stacks grow down, the result answered by alloca is a pointer to the start of N bytes, hence it is effectively the stack pointer. If this doesn't make sense to you *draw a picture on a whiteboard* of what you expect alloca(29) to do)

2. some C compilers are nice and implement alloca(N) as effectively stackPointer := stackPointer - roundUpTo(N,StackAlignment).  For these compilers we don't need to do anything.  Some C compilers try and be clever and answer stackPointer + 4, or something else.  In these cases we need to get the actual stack pointer.  So see the code ThreadedFFIPlugin class>>preambleCCode.

3.  look at what the C compiler produces for the alloca call (and/or run under gdb) to see what happens to the stack pointer and what ends up in the allocation var in ffiCall:ArgArrayOrNil:NumArgs:.

4. update ThreadedFFIPlugin class>>preambleCCode. appropriately until you've bent alloca and the C compiler to your will.

You will not make this work without trying to understand what is going on and looking at what your C compiler produces.  This is low-level hackery that aims to avoid having to write any assembler (outside of very few C asm statements).

HTH
Eliot


> Thx
> T.
> --
> NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!
> Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a



--
Best regards,
Igor Stasenko.



--
best,
Eliot


Reply | Threaded
Open this post in threaded view
|

Re: There are something fishy with FFI plugin

Igor Stasenko
 
Fix confirmed.
https://ci.lille.inria.fr/pharo/view/Cog/job/Cog-VM/Architecture=32,OS=win/
build >= #29,  FFI works fine..

Thanks, Esteban.



--
Best regards,
Igor Stasenko.
Reply | Threaded
Open this post in threaded view
|

Re: There are something fishy with FFI plugin

Gary Chambers-4
 
I'll test out tomorrow...
Ideally we'd want the multithreaded VM at some point,
investigating direct USB devices on Windows. ReadFile blocks, of course...

Regards, Gary

----- Original Message -----
From: "Igor Stasenko" <[hidden email]>
To: "Squeak Virtual Machine Development Discussion"
<[hidden email]>
Sent: Wednesday, April 25, 2012 4:44 PM
Subject: Re: [Vm-dev] There are something fishy with FFI plugin


>
> Fix confirmed.
> https://ci.lille.inria.fr/pharo/view/Cog/job/Cog-VM/Architecture=32,OS=win/
> build >= #29,  FFI works fine..
>
> Thanks, Esteban.
>
>
>
> --
> Best regards,
> Igor Stasenko.

Reply | Threaded
Open this post in threaded view
|

Re: There are something fishy with FFI plugin

Andreas.Raab
 
On 4/25/2012 17:58, Gary Chambers wrote:
>
> I'll test out tomorrow...
> Ideally we'd want the multithreaded VM at some point,
> investigating direct USB devices on Windows. ReadFile blocks, of
> course...

Of course. Two things you can do:
1) Use overlapped I/O completion routines. This is a bit tricky to get
right so it might be easier to implement option
2) Simply read from within a separate thread. I used this approach for
async file support; you can find prototypical code in
platforms/win32/plugins/AsynchFilePlugin

Cheers,
   -  Andreas

>
> Regards, Gary
>
> ----- Original Message ----- From: "Igor Stasenko" <[hidden email]>
> To: "Squeak Virtual Machine Development Discussion"
> <[hidden email]>
> Sent: Wednesday, April 25, 2012 4:44 PM
> Subject: Re: [Vm-dev] There are something fishy with FFI plugin
>
>
>>
>> Fix confirmed.
>> https://ci.lille.inria.fr/pharo/view/Cog/job/Cog-VM/Architecture=32,OS=win/ 
>>
>> build >= #29,  FFI works fine..
>>
>> Thanks, Esteban.
>>
>>
>>
>> --
>> Best regards,
>> Igor Stasenko.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: There are something fishy with FFI plugin

Gary Chambers-4
 
Was hoping to avoid doing another (trivial) plugin...

Regards, Gary

----- Original Message -----
From: "Andreas Raab" <[hidden email]>
To: <[hidden email]>
Sent: Wednesday, April 25, 2012 6:26 PM
Subject: Re: [Vm-dev] There are something fishy with FFI plugin


>
> On 4/25/2012 17:58, Gary Chambers wrote:
>>
>> I'll test out tomorrow...
>> Ideally we'd want the multithreaded VM at some point,
>> investigating direct USB devices on Windows. ReadFile blocks, of
>> course...
>
> Of course. Two things you can do:
> 1) Use overlapped I/O completion routines. This is a bit tricky to get
> right so it might be easier to implement option
> 2) Simply read from within a separate thread. I used this approach for
> async file support; you can find prototypical code in
> platforms/win32/plugins/AsynchFilePlugin
>
> Cheers,
>   -  Andreas
>
>>
>> Regards, Gary
>>
>> ----- Original Message ----- From: "Igor Stasenko" <[hidden email]>
>> To: "Squeak Virtual Machine Development Discussion"
>> <[hidden email]>
>> Sent: Wednesday, April 25, 2012 4:44 PM
>> Subject: Re: [Vm-dev] There are something fishy with FFI plugin
>>
>>
>>>
>>> Fix confirmed.
>>> https://ci.lille.inria.fr/pharo/view/Cog/job/Cog-VM/Architecture=32,OS=win/
>>> build >= #29,  FFI works fine..
>>>
>>> Thanks, Esteban.
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Igor Stasenko.
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: There are something fishy with FFI plugin

Eliot Miranda-2
In reply to this post by EstebanLM
 


On Wed, Apr 25, 2012 at 4:30 AM, Esteban Lorenzano <[hidden email]> wrote:
 
Hi Eliot,

Ok, after spending 2.5 days on this (but well... now I know a lot about this stuff, he), I found that the problem with our compilation is that two flags are needed: 

STACK_ALIGN_BYTES=16
ALLOCA_LIES_SO_USE_GETSP=0 //Yep, alloca is not lying here

problem is... while first flag can be set by command line, second one is setted on preamble for ThreadedFFIPlugin, then I need to change it... 

so, here my question: 

Why is working as is for you and not for us? (I think both of us are using mingw, and I installed gcc 3.4.5 to use same version as you) Do I'm missing something?

Are you sure you're actually using   gcc 3.4.5?  Otherwise the difference could be because you're probably using CMake and I'm not.  i.e. I use the makefiles <a href="http://www.squeakvm.org/svn/squeak/branches/Cog/cygwinbuild/{Makefile,Makefile.plugin}">http://www.squeakvm.org/svn/squeak/branches/Cog/cygwinbuild/{Makefile,Makefile.plugin}


best,
Esteban

On Apr 25, 2012, at 1:20 AM, Eliot Miranda wrote:



On Tue, Apr 24, 2012 at 2:53 PM, Igor Stasenko <[hidden email]> wrote:

On 24 April 2012 19:16, Torsten Bergmann <[hidden email]> wrote:
>
> Load latest from ConfigurationOfFFI from MetacellRepository:
>
> For me the FFIPluginTest fails with latest Pharo 1.4 VM
> - so at least one has a reproducable case.
>
Esteban spent whole day today trying to find the offending bug.
Source code match 1:1 with Eliot's code, because there was no any
changes in it..
still the results is that it refuses to work correctly on windoze.

You need to understand what the ThreadedFFIPlugin tries to do with alloca and that various compilers may implement alloca in a number of different ways.  You then need to look at what the compiler actually does and figure out how to subvert it to do what you want to do.

1.  the ThreadedFFIPlugin uses alloca to stack-allocate the space for the outgoing parameters of the FFI call.  Sicne on x86 all parameters are passed on the stack and none in registers alloca is ideal.  Notionally the plugin can call alloca to extend the stack downwards (stacks grow down, the result answered by alloca is a pointer to the start of N bytes, hence it is effectively the stack pointer. If this doesn't make sense to you *draw a picture on a whiteboard* of what you expect alloca(29) to do)

2. some C compilers are nice and implement alloca(N) as effectively stackPointer := stackPointer - roundUpTo(N,StackAlignment).  For these compilers we don't need to do anything.  Some C compilers try and be clever and answer stackPointer + 4, or something else.  In these cases we need to get the actual stack pointer.  So see the code ThreadedFFIPlugin class>>preambleCCode.

3.  look at what the C compiler produces for the alloca call (and/or run under gdb) to see what happens to the stack pointer and what ends up in the allocation var in ffiCall:ArgArrayOrNil:NumArgs:.

4. update ThreadedFFIPlugin class>>preambleCCode. appropriately until you've bent alloca and the C compiler to your will.

You will not make this work without trying to understand what is going on and looking at what your C compiler produces.  This is low-level hackery that aims to avoid having to write any assembler (outside of very few C asm statements).

HTH
Eliot


> Thx
> T.
> --
> NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!
> Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a



--
Best regards,
Igor Stasenko.



--
best,
Eliot






--
best,
Eliot