Help with FFI crash in latest Spur (only in Linux)

On Fri, Jan 8, 2016 at 8:41 PM, Nicolai Hess <[hidden email]> wrote:

2016-01-09 0:12 GMT+01:00 Mariano Martinez Peck <[hidden email]>:

On Jan 8, 2016 4:13 PM, "Ben Coman" <[hidden email]> wrote:
>
> On Sat, Jan 9, 2016 at 2:04 AM, Mariano Martinez Peck
> <[hidden email]> wrote:
> > Hi guys,
> >
> > I wonder if someone could give me a hand to find out why a FFI calling I am
> > doing is crashing. In OSX it works correct but I am testing in CentOS and it
> > fails. I wonder if it also crashes in other Linuxes too.
>
> I only had enough time to run it a few time so you know it also
> crashes in Debian Jessie 32-bit. There were no debug logs.  I got
> these sorts of messages...
>

Thanks!

> *** Error in `/home/ben/tst/pharo-vm/pharo': free(): invalid next size
> (fast): 0x08841a70 ***
>
> pharo: malloc.c:3695: _int_malloc: Assertion `(unsigned long) (size)
> >= (unsigned long) (nb)' failed.
> *** Error in `/home/ben/tst/pharo-vm/pharo': malloc(): memory
> corruption: 0x0984c1e0 ***
>
>
> Searching github for "posix_spawn_file_actions_init " (https://git.io/vuSPL)
> I see a lot a function definitions of the form...
>    int posix_spawn_file_actions_init(posix_spawn_file_actions_t *fa)
>   {
>        fa->__actions = 0;
>        return 0;
>    }
>
> ...so it seems you need to first allocate the space for the struct and
> then pass the address of that.

I thought the same. But I also read in glibc that that the init and destroy would free and alloc exactly so that you don't have to do it.

In fact, the structure is known to be opaque. I cannot rely in what I see in internet since each os may have a different.

And I cant get the size of it from ffi so I cannot allocate accurately.  I think I am screw. I don't want to go to plugin side grrr ..

And osx does work.

I think I will try against glibc rather than libc.

Another idea?

> {
>   int __allocated;
>   int __used;
>   struct __spawn_action *__actions;
>   int __pad[16];
> } posix_spawn_file_actions_t;
> // http://linux.die.net/include/spawn.h
>

But that's opaque right? I cannot rely on that

no,  posix_spawn_file_actions_init is not supposed  allocate the structure.

http://linux.die.net/man/3/posix_spawn_file_actions_init:
"The posix_spawn_file_actions_init() function shall *initialize the object referenced*

Yeah, damn. And the following text may explain why it DOES work on OSX:

As an implementation detail, the externally visibily type
 *		posix_spawn_file_actions_t is defined to be a void *, and
 *		initialization involves allocation of a memory object.
 *		Subsequent changes to the spawn file actions may result in
 *		reallocation under the covers.

 

Extracted from http://www.opensource.apple.com/source/Libc/Libc-583/sys/posix_spawn.c

> cheers -ben
>
>
> >
> > I am using latest Pharo 5.0 with Spur. To reproduce:
> >
> > 1) Get latest Pharo 5.0 and Spur via:
> > wget -O- get.pharo.org/alpha+vm | bash
> >
> > 2) Inside Pharo, load my prototype tool:
> >
> > Gofer it
> > package: 'OSSubprocess';
> > url: 'http://smalltalkhub.com/mc/marianopeck/OSSubprocess/main';
> > load.
> >
> > 3) This is the code I am executing and it's crashing:
> >
> > | posixSpawnFileActionsT | posixSpawnFileActionsT := ExternalAddress
> > allocate: 4. OSSUnixSubprocess new primitivePosixSpawnFileActionsInit:
> > posixSpawnFileActionsT. posixSpawnFileActionsT free.
> > 4) The primitive is as simple as:
> >
> > primitivePosixSpawnFileActionsInit: aPosixSpawnFileActionsT
> > ^ self ffiCall: #( int posix_spawn_file_actions_init(void*
> > aPosixSpawnFileActionsT) ) module: LibC
> >
> > I have no idea what I am doing wrong. And again, this works on OSX. The
> > function I am calling is:  int
> > posix_spawn_file_actions_init(posix_spawn_file_actions_t *file_actions); as
> > you can read in [1]
> >
> > Below is the stacktrace I get the Linux terminal.
> >
> > Any hint is greatly appreciated.
> >
> > Thanks,
> >
> >
> > [1]
> > http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_spawn_file_actions_destroy.html
>

--

Mariano
http://marianopeck.wordpress.com

On Sat, Jan 9, 2016 at 12:15 PM, Ben Coman <[hidden email]> wrote:
> On Sat, Jan 9, 2016 at 7:12 AM, Mariano Martinez Peck
> <[hidden email]> wrote:
>>
>> On Jan 8, 2016 4:13 PM, "Ben Coman" <[hidden email]> wrote:
>>>
>>> On Sat, Jan 9, 2016 at 2:04 AM, Mariano Martinez Peck
>>> <[hidden email]> wrote:
>>> > Hi guys,
>>> >
>>> > I wonder if someone could give me a hand to find out why a FFI calling I
>>> > am
>>> > doing is crashing. In OSX it works correct but I am testing in CentOS
>>> > and it
>>> > fails. I wonder if it also crashes in other Linuxes too.
>>>
>>> I only had enough time to run it a few time so you know it also
>>> crashes in Debian Jessie 32-bit. There were no debug logs.  I got
>>> these sorts of messages...
>>>
>>
>> Thanks!
>>
>>> *** Error in `/home/ben/tst/pharo-vm/pharo': free(): invalid next size
>>> (fast): 0x08841a70 ***
>>>
>>> pharo: malloc.c:3695: _int_malloc: Assertion `(unsigned long) (size)
>>> >= (unsigned long) (nb)' failed.
>>> *** Error in `/home/ben/tst/pharo-vm/pharo': malloc(): memory
>>> corruption: 0x0984c1e0 ***
>>>
>>>
>>> Searching github for "posix_spawn_file_actions_init "
>>> (https://git.io/vuSPL)
>>> I see a lot a function definitions of the form...
>>>    int posix_spawn_file_actions_init(posix_spawn_file_actions_t *fa)
>>>   {
>>>        fa->__actions = 0;
>>>        return 0;
>>>    }
>>>
>>> ...so it seems you need to first allocate the space for the struct and
>>> then pass the address of that.
>>
>> I thought the same. But I also read in glibc that that the init and destroy
>> would free and alloc exactly so that you don't have to do it.
>>
>> In fact, the structure is known to be opaque. I cannot rely in what I see in
>> internet since each os may have a different.
>>
>> And I cant get the size of it from ffi so I cannot allocate accurately.  I
>> think I am screw. I don't want to go to plugin side grrr ..
>>
>> And osx does work.
>>
>> I think I will try against glibc rather than libc.
>>
>> Another idea?
>>
>>> {
>>>   int __allocated;
>>>   int __used;
>>>   struct __spawn_action *__actions;
>>>   int __pad[16];
>>> } posix_spawn_file_actions_t;
>>> // http://linux.die.net/include/spawn.h
>>>
>>
>> But that's opaque right? I cannot rely on that
>
> Your ExternalAddress use only allocates 4 bytes for the pointer.  You
> need to also allocate the space for the structure.  I guess these may
> need separate ExternalXXX objects, but just as hack experiment I find
> below that n:=68 crashes and n:=69 its fine.

Whoops, I missed that the space for the struct seems actually
allocated by ExternalAddress.  Doh! there it is staring at me with the
#allocate: message.   So I guess a separate ExternalXXX object isn't
required.

>
> | posixSpawnFileActionsT n |
> n := 68.  "n := 69"
> posixSpawnFileActionsT := ExternalAddress allocate: n.
> OSSUnixSubprocess new primitivePosixSpawnFileActionsInit:
> posixSpawnFileActionsT.
> posixSpawnFileActionsT free.
>
>
> In trying to understand this I found an interesting article...
>      http://www.catb.org/esr/structure-packing/
> What I don't know is for a 32-bit OS on a 64-bit machine, is my
> pointer 4 or 8 bytes?
> Assuming 4 I derive the structure size for my 32-bit system is...
> 4      int __allocated;
> 4      int __used;
> 4      struct __spawn_action *__actions;
> 64    int __pad[16];
> 76    posix_spawn_file_actions_t;
>
> 76 - 68 = 8 is an interesting round number, but I can't quite reason it out.

btw I confirmed *my* structure size by compiling and running...
  #include <stdio.h>
  #include <spawn.h>
  posix_spawn_file_actions_t tst;
  int main()
  {
    printf("-->sizeof=%d\n", sizeof(posix_spawn_file_actions_t));
  }

-->sizeof=76

cheers -ben

> With FFI how to you allocate some external memory and then get a
> pointer to it to pass to primitivePosixSpawnFileActionsInit ?
>
> cheers -ben
>
>>> > I am using latest Pharo 5.0 with Spur. To reproduce:
>>> >
>>> > 1) Get latest Pharo 5.0 and Spur via:
>>> > wget -O- get.pharo.org/alpha+vm | bash
>>> >
>>> > 2) Inside Pharo, load my prototype tool:
>>> >
>>> > Gofer it
>>> > package: 'OSSubprocess';
>>> > url: 'http://smalltalkhub.com/mc/marianopeck/OSSubprocess/main';
>>> > load.
>>> >
>>> > 3) This is the code I am executing and it's crashing:
>>> >
>>> > | posixSpawnFileActionsT | posixSpawnFileActionsT := ExternalAddress
>>> > allocate: 4. OSSUnixSubprocess new primitivePosixSpawnFileActionsInit:
>>> > posixSpawnFileActionsT. posixSpawnFileActionsT free.
>>> > 4) The primitive is as simple as:
>>> >
>>> > primitivePosixSpawnFileActionsInit: aPosixSpawnFileActionsT
>>> > ^ self ffiCall: #( int posix_spawn_file_actions_init(void*
>>> > aPosixSpawnFileActionsT) ) module: LibC
>>> >
>>> > I have no idea what I am doing wrong. And again, this works on OSX. The
>>> > function I am calling is:  int
>>> > posix_spawn_file_actions_init(posix_spawn_file_actions_t *file_actions);
>>> > as
>>> > you can read in [1]
>>> >
>>> > Below is the stacktrace I get the Linux terminal.
>>> >
>>> > Any hint is greatly appreciated.
>>> >
>>> > Thanks,
>>> >
>>> >
>>> > [1]
>>> >
>>> > http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_spawn_file_actions_destroy.html
>>>

Hi Ben, allf

Ok, I can also confirm that doing ExternalAddress allocate: 76 does work in Linux. And yes, since it's opaque I don't have to map it and just allocating the correct size is enough. 

I expect this would work in all GNU libc (glibc) based UNIXses.

Esteban, well, I thought the init function would do both, allocate and init, but in glibc it seems it only init. In OSX it seems it does both. But from the API point of view it's not explicit that they should allocate the space. 

Anyway, I think we get this working this time. Thanks all for the help.

Now the problem I am facing (in Linux) is that my none blocking pipes do not seem to be none blocking or either feof() call is not working as expected. Ok, I have to deal with this one now. 

Thanks!!

On Sat, Jan 9, 2016 at 9:44 AM, Ben Coman <[hidden email]> wrote:

On Sat, Jan 9, 2016 at 12:15 PM, Ben Coman <[hidden email]> wrote:
> On Sat, Jan 9, 2016 at 7:12 AM, Mariano Martinez Peck
> <[hidden email]> wrote:
>>
>> On Jan 8, 2016 4:13 PM, "Ben Coman" <[hidden email]> wrote:
>>>
>>> On Sat, Jan 9, 2016 at 2:04 AM, Mariano Martinez Peck
>>> <[hidden email]> wrote:
>>> > Hi guys,
>>> >
>>> > I wonder if someone could give me a hand to find out why a FFI calling I
>>> > am
>>> > doing is crashing. In OSX it works correct but I am testing in CentOS
>>> > and it
>>> > fails. I wonder if it also crashes in other Linuxes too.
>>>
>>> I only had enough time to run it a few time so you know it also
>>> crashes in Debian Jessie 32-bit. There were no debug logs.  I got
>>> these sorts of messages...
>>>
>>
>> Thanks!
>>
>>> *** Error in `/home/ben/tst/pharo-vm/pharo': free(): invalid next size
>>> (fast): 0x08841a70 ***
>>>
>>> pharo: malloc.c:3695: _int_malloc: Assertion `(unsigned long) (size)
>>> >= (unsigned long) (nb)' failed.
>>> *** Error in `/home/ben/tst/pharo-vm/pharo': malloc(): memory
>>> corruption: 0x0984c1e0 ***
>>>
>>>
>>> Searching github for "posix_spawn_file_actions_init "
>>> (https://git.io/vuSPL)
>>> I see a lot a function definitions of the form...
>>>    int posix_spawn_file_actions_init(posix_spawn_file_actions_t *fa)
>>>   {
>>>        fa->__actions = 0;
>>>        return 0;
>>>    }
>>>
>>> ...so it seems you need to first allocate the space for the struct and
>>> then pass the address of that.
>>
>> I thought the same. But I also read in glibc that that the init and destroy
>> would free and alloc exactly so that you don't have to do it.
>>
>> In fact, the structure is known to be opaque. I cannot rely in what I see in
>> internet since each os may have a different.
>>
>> And I cant get the size of it from ffi so I cannot allocate accurately.  I
>> think I am screw. I don't want to go to plugin side grrr ..
>>
>> And osx does work.
>>
>> I think I will try against glibc rather than libc.
>>
>> Another idea?
>>
>>> {
>>>   int __allocated;
>>>   int __used;
>>>   struct __spawn_action *__actions;
>>>   int __pad[16];
>>> } posix_spawn_file_actions_t;
>>> // http://linux.die.net/include/spawn.h
>>>
>>
>> But that's opaque right? I cannot rely on that
>
> Your ExternalAddress use only allocates 4 bytes for the pointer.  You
> need to also allocate the space for the structure.  I guess these may
> need separate ExternalXXX objects, but just as hack experiment I find
> below that n:=68 crashes and n:=69 its fine.

Whoops, I missed that the space for the struct seems actually
allocated by ExternalAddress.  Doh! there it is staring at me with the
#allocate: message.   So I guess a separate ExternalXXX object isn't
required.

>
> | posixSpawnFileActionsT n |
> n := 68.  "n := 69"
> posixSpawnFileActionsT := ExternalAddress allocate: n.
> OSSUnixSubprocess new primitivePosixSpawnFileActionsInit:
> posixSpawnFileActionsT.
> posixSpawnFileActionsT free.
>
>
> In trying to understand this I found an interesting article...
>      http://www.catb.org/esr/structure-packing/
> What I don't know is for a 32-bit OS on a 64-bit machine, is my
> pointer 4 or 8 bytes?
> Assuming 4 I derive the structure size for my 32-bit system is...
> 4      int __allocated;
> 4      int __used;
> 4      struct __spawn_action *__actions;
> 64    int __pad[16];
> 76    posix_spawn_file_actions_t;
>
> 76 - 68 = 8 is an interesting round number, but I can't quite reason it out.

btw I confirmed *my* structure size by compiling and running...
  #include <stdio.h>
  #include <spawn.h>
  posix_spawn_file_actions_t tst;
  int main()
  {
    printf("-->sizeof=%d\n", sizeof(posix_spawn_file_actions_t));
  }

-->sizeof=76

cheers -ben

> With FFI how to you allocate some external memory and then get a
> pointer to it to pass to primitivePosixSpawnFileActionsInit ?
>
> cheers -ben
>
>>> > I am using latest Pharo 5.0 with Spur. To reproduce:
>>> >
>>> > 1) Get latest Pharo 5.0 and Spur via:
>>> > wget -O- get.pharo.org/alpha+vm | bash
>>> >
>>> > 2) Inside Pharo, load my prototype tool:
>>> >
>>> > Gofer it
>>> > package: 'OSSubprocess';
>>> > url: 'http://smalltalkhub.com/mc/marianopeck/OSSubprocess/main';
>>> > load.
>>> >
>>> > 3) This is the code I am executing and it's crashing:
>>> >
>>> > | posixSpawnFileActionsT | posixSpawnFileActionsT := ExternalAddress
>>> > allocate: 4. OSSUnixSubprocess new primitivePosixSpawnFileActionsInit:
>>> > posixSpawnFileActionsT. posixSpawnFileActionsT free.
>>> > 4) The primitive is as simple as:
>>> >
>>> > primitivePosixSpawnFileActionsInit: aPosixSpawnFileActionsT
>>> > ^ self ffiCall: #( int posix_spawn_file_actions_init(void*
>>> > aPosixSpawnFileActionsT) ) module: LibC
>>> >
>>> > I have no idea what I am doing wrong. And again, this works on OSX. The
>>> > function I am calling is:  int
>>> > posix_spawn_file_actions_init(posix_spawn_file_actions_t *file_actions);
>>> > as
>>> > you can read in [1]
>>> >
>>> > Below is the stacktrace I get the Linux terminal.
>>> >
>>> > Any hint is greatly appreciated.
>>> >
>>> > Thanks,
>>> >
>>> >
>>> > [1]
>>> >
>>> > http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_spawn_file_actions_destroy.html
>>>

--

Mariano
http://marianopeck.wordpress.com