FFI Struct Argument Pass By Value Fails on Mac 64 bit

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

FFI Struct Argument Pass By Value Fails on Mac 64 bit

tblanchard
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

I'm stuck.  The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {
  const void *ptr_data[2];
  unsigned int_data;
} CXSourceLocation;

/**
 * \brief Identifies a half-open character range in the source code.
 *
 * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the
 * starting and end locations from a source range, respectively.
 */
typedef struct {
  const void *ptr_data[2];
  unsigned begin_int_data;
  unsigned end_int_data;
} CXSourceRange;

const char* first = "first_pointer";
const char* second = "second_pointer";

// return a fake range with non zero data
CXSourceRange clang_getArbitraryRange()
{
  CXSourceRange range = {0};
  range.ptr_data[0] = (void*)first;
  range.ptr_data[1] = (void*)second;
  range.begin_int_data = 17;
  range.end_int_data = 24;
  return range;
}

// Actual clang function - range is always zero'd here despite having values in the image
CXSourceLocation clang_getRangeStart(CXSourceRange range) {
  // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1) {
    CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };
    return Result;    
  }

  

  CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },
    range.begin_int_data };
  return Result;
}




LibCClang-FFI-Binding.st (7K) Download Attachment
microclang.c (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

Ben Coman
 
Hi Todd,

Could you advise the the Playground code to use to test this?
And maybe a C program that links to the microclang shared library that does the equivalent test?

cheers -ben

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

I'm stuck.  The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {
  const void *ptr_data[2];
  unsigned int_data;
} CXSourceLocation;

/**
 * \brief Identifies a half-open character range in the source code.
 *
 * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the
 * starting and end locations from a source range, respectively.
 */
typedef struct {
  const void *ptr_data[2];
  unsigned begin_int_data;
  unsigned end_int_data;
} CXSourceRange;

const char* first = "first_pointer";
const char* second = "second_pointer";

// return a fake range with non zero data
CXSourceRange clang_getArbitraryRange()
{
  CXSourceRange range = {0};
  range.ptr_data[0] = (void*)first;
  range.ptr_data[1] = (void*)second;
  range.begin_int_data = 17;
  range.end_int_data = 24;
  return range;
}

// Actual clang function - range is always zero'd here despite having values in the image
CXSourceLocation clang_getRangeStart(CXSourceRange range) {
  // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1) {
    CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };
    return Result;    
  }

  

  CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },
    range.begin_int_data };
  return Result;
}





Reply | Threaded
Open this post in threaded view
|

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

tblanchard
 
Playground code:

LibCClang startFromRange:(LibCClang defaultRange)

it worked if the resulting struct is not all zeroes.

A tiny C program is the equivalent.

int main()
{
  if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) printf("That failed\n"); 
  else printf("That worked\n")
  return 0;
}


On Nov 21, 2017, at 11:18 PM, Ben Coman <[hidden email]> wrote:

Hi Todd,

Could you advise the the Playground code to use to test this?
And maybe a C program that links to the microclang shared library that does the equivalent test?

cheers -ben

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

I'm stuck.  The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {
  const void *ptr_data[2];
  unsigned int_data;
} CXSourceLocation;

/**
 * \brief Identifies a half-open character range in the source code.
 *
 * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the
 * starting and end locations from a source range, respectively.
 */
typedef struct {
  const void *ptr_data[2];
  unsigned begin_int_data;
  unsigned end_int_data;
} CXSourceRange;

const char* first = "first_pointer";
const char* second = "second_pointer";

// return a fake range with non zero data
CXSourceRange clang_getArbitraryRange()
{
  CXSourceRange range = {0};
  range.ptr_data[0] = (void*)first;
  range.ptr_data[1] = (void*)second;
  range.begin_int_data = 17;
  range.end_int_data = 24;
  return range;
}

// Actual clang function - range is always zero'd here despite having values in the image
CXSourceLocation clang_getRangeStart(CXSourceRange range) {
  // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1) {
    CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };
    return Result;    
  }
  
  CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },
    range.begin_int_data };
  return Result;
}






Reply | Threaded
Open this post in threaded view
|

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

Ben Coman
In reply to this post by tblanchard
 


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)



I'm stuck.  The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {
  const void *ptr_data[2];
  unsigned int_data;
} CXSourceLocation;

/**
 * \brief Identifies a half-open character range in the source code.
 *
 * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the
 * starting and end locations from a source range, respectively.
 */
typedef struct {
  const void *ptr_data[2];
  unsigned begin_int_data;
  unsigned end_int_data;
} CXSourceRange;

const char* first = "first_pointer";
const char* second = "second_pointer";

// return a fake range with non zero data
CXSourceRange clang_getArbitraryRange()
{
  CXSourceRange range = {0};
  range.ptr_data[0] = (void*)first;
  range.ptr_data[1] = (void*)second;
  range.begin_int_data = 17;
  range.end_int_data = 24;
  return range;
}

// Actual clang function - range is always zero'd here despite having values in the image
CXSourceLocation clang_getRangeStart(CXSourceRange range) {
  // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1) {
    CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };
    return Result;    
  }

  

  CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },
    range.begin_int_data };
  return Result;
}


clang_ffi_struct_test.zip (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

Ben Coman
 


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...


___microlang.h___
typedef unsigned  uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);



___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{       CXSourceRange_ range = {0};
        range.ptr_data[0] = (void*)libraryString;
        return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{       // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
        if ((uintptr_t)range.ptr_data[0] & 0x1)
        {       return 0;       }
        else
        {       return 1;       }
}




___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString =  "local_pointer";

void test( CXSourceRange_ range, char *note )
{       int result = clang_getRangeEnd_(range);
        if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else
        {       printf("That worked (%s)\n", note); }
}

int main()
{       CXSourceRange_ range1 = clang_getArbitraryRange_();
        test(range1, "library string");

        CXSourceRange_ range2 = {0};
        range2.ptr_data[0] = (void*)localString;
        test(range2, "local string");
}



___Makefile___
default: clean static shared

clean:
        rm -f *so *App
        @echo

shared:
        clang -g -o libmicroclang.so -shared -fPIC microclang.c
        clang -g -o sharedApp test.c -L. -lmicroclang
        LD_LIBRARY_PATH=. ./sharedApp
        @echo

static:
        clang -g -o staticApp test.c microclang.c
        ./staticApp
        @echo



Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)


cheers -ben

clang2.zip (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

Ben Coman
 


On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...


___microlang.h___
typedef unsigned  uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);



___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{       CXSourceRange_ range = {0};
        range.ptr_data[0] = (void*)libraryString;
        return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{       // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
        if ((uintptr_t)range.ptr_data[0] & 0x1)
        {       return 0;       }
        else
        {       return 1;       }
}




___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString =  "local_pointer";

void test( CXSourceRange_ range, char *note )
{       int result = clang_getRangeEnd_(range);
        if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else
        {       printf("That worked (%s)\n", note); }
}

int main()
{       CXSourceRange_ range1 = clang_getArbitraryRange_();
        test(range1, "library string");

        CXSourceRange_ range2 = {0};
        range2.ptr_data[0] = (void*)localString;
        test(range2, "local string");
}



___Makefile___
default: clean static shared

clean:
        rm -f *so *App
        @echo

shared:
        clang -g -o libmicroclang.so -shared -fPIC microclang.c
        clang -g -o sharedApp test.c -L. -lmicroclang
        LD_LIBRARY_PATH=. ./sharedApp
        @echo

static:
        clang -g -o staticApp test.c microclang.c
        ./staticApp
        @echo



Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)


Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned  uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{       return myLibraryString;
}

int lib_testString( const char *aString )
{       unsigned test = (uintptr_t)aString & 0x1;
        printf("\n  test=%d, aString-->%d\n", test, (uintptr_t)aString);
        if (test)
        {       return 0;       }
        else
        {       return 1;       }
}


___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString =  "local_pointer";

void test( int result, char *note )
{       if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else 
        {       printf("That worked (%s)\n", note); }
}

int main()
{       const char * libraryString = lib_getLibraryString();
        test(lib_testString(libraryString), "library string");

        test(lib_testString(localString), "local string");
}


$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

  test=0, aString-->4196150
  That worked (library string)

  test=0, aString-->4196068
  That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

  test=1, aString-->-792512599
  That failed (library string)

  test=0, aString-->4196484
  That worked (local string)


cheers -ben


sharedLibString.zip (2K) Download Attachment
clang3.zip (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

tblanchard
 
i'm getting the idea that we should probably write a test suite/library for FFI

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:



On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...


___microlang.h___
typedef unsigned  uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);



___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{       CXSourceRange_ range = {0};
        range.ptr_data[0] = (void*)libraryString;
        return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{       // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
        if ((uintptr_t)range.ptr_data[0] & 0x1)
        {       return 0;       }
        else
        {       return 1;       }
}




___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString =  "local_pointer";

void test( CXSourceRange_ range, char *note )
{       int result = clang_getRangeEnd_(range);
        if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else
        {       printf("That worked (%s)\n", note); }
}

int main()
{       CXSourceRange_ range1 = clang_getArbitraryRange_();
        test(range1, "library string");

        CXSourceRange_ range2 = {0};
        range2.ptr_data[0] = (void*)localString;
        test(range2, "local string");
}



___Makefile___
default: clean static shared

clean:
        rm -f *so *App
        @echo

shared:
        clang -g -o libmicroclang.so -shared -fPIC microclang.c
        clang -g -o sharedApp test.c -L. -lmicroclang
        LD_LIBRARY_PATH=. ./sharedApp
        @echo

static:
        clang -g -o staticApp test.c microclang.c
        ./staticApp
        @echo



Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)


Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned  uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{       return myLibraryString;
}

int lib_testString( const char *aString )
{       unsigned test = (uintptr_t)aString & 0x1;
        printf("\n  test=%d, aString-->%d\n", test, (uintptr_t)aString);
        if (test)
        {       return 0;       }
        else
        {       return 1;       }
}


___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString =  "local_pointer";

void test( int result, char *note )
{       if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else 
        {       printf("That worked (%s)\n", note); }
}

int main()
{       const char * libraryString = lib_getLibraryString();
        test(lib_testString(libraryString), "library string");

        test(lib_testString(localString), "local string");
}


$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

  test=0, aString-->4196150
  That worked (library string)

  test=0, aString-->4196068
  That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

  test=1, aString-->-792512599
  That failed (library string)

  test=0, aString-->4196484
  That worked (local string)


cheers -ben

<sharedLibString.zip><clang3.zip>

Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

Ben Coman
 


On 27 November 2017 at 00:24, Todd Blanchard <[hidden email]> wrote:
 
i'm getting the idea that we should probably write a test suite/library for FFI

I noticed these...

These look like c-code test frames for FFI.  Could these be built by the OpenSmalltalk CI to be normally shipped with the VM so that Image-side CI can test against them?

And also some general info...

 

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:


On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

My last analysis discovered something interesting about strings defined inside shared libraries being handled differently, but only later realised I had chased the wrong rabbit down the hole.  
 
I've now investigated the premise you actually poses, and I agree, that structs being zero is some cases.  
The attached zipfile containing  libstruct.c  a few comparison cases - three "good" layouts that work fine and one "bad" that mostly gets zeros but sometimes other weird numbers.  There is a Makefile with three main targets:
1. make layout - statically compiles  libstruct.c  and runs produced  a.out  to display structure layouts 

2. make run - downloads Pharo, starts it loading  LibStruct.st  , then you manually run LibStruct>>>LibStructTest>>#testStructs and observe structure values on console

3. make debug - starts LLDB to run Pharo with breakpoints pre-configured for when you run #testStructs.  
   Note Pharo will freeze and you need to move to LLDB.  Try these commands... 
      frame variable
      call print_struct(&GoodStruct1_fmt, &aStruct)
      continue

That works on Ubuntu 16.04 64 bit. 
You will need to tune it for OSX.
clang and lldb are required.

================================
The offsets configured in the class variables of all ExternalStructs 
correctly matches that reported by the C code test frame results here...

$ make layout        
clang -g libstruct.c
./a.out

GoodStruct1:
uint32_t:int1:    01 4 
uint32_t:int2:    05 4 
GoodStruct2:
uint32_t:int1:    01 4 
uint32_t:int2:    05 4 
void*:ptr_data:    09 8 
GoodStruct3:
void*:ptr_data:    01 8 
uint32_t:int1:    09 4 
uint32_t:int2:    13 4 
BadStruct:
void*:ptr_data1:    01 8 
void*:ptr_data2:    09 8 
uint32_t:int1:    17 4 
uint32_t:int2:    21 4 

================================

$ make run   
clang -g -o libstruct.so -shared -fPIC libstruct.c
getpharo/pharo-vm/lib/pharo/5.0-201707201942/pharo getpharo/Pharo.image ../LibStruct.st

Now manually browse to and run  LibStruct>>>LibStructTest>>#testStructs
LibStructTest>>testStructs
|good1Struct good2Struct good3Struct badStruct |
good1Struct    := GoodStruct1    new int1: 2; int2: 3.
good2Struct    := GoodStruct2    new int1: 2; int2: 3; ptr_force_int: 4.
good3Struct    := GoodStruct3    new int1: 2; int2: 3; ptr_force_int: 4.
badStruct      := BadStruct      new int1: 2; int2: 3; ptr_force_int1: 4; ptr_force_int1: 5.
self assert: (LibStruct tryGood1: good1Struct)       equals: 6.
self assert: (LibStruct tryGood2: good2Struct)       equals: 6.
self assert: (LibStruct tryGood3: good3Struct)       equals: 6.
"Problem exposed in next line"
self assert: (LibStruct tryBad: badStruct)           equals: 6.

which on console produces...

GoodStruct1:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
GoodStruct2:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
void*:ptr_data:    09 8 = 04 00 00 00 00 00 00 00
GoodStruct3:
void*:ptr_data:    01 8 = 04 00 00 00 00 00 00 00
uint32_t:int1:    09 4 = 02 00 00 00
uint32_t:int2:    13 4 = 03 00 00 00
BadStruct:
void*:ptr_data1:    01 8 = 00 00 00 00 00 00 00 00
void*:ptr_data2:    09 8 = 00 00 00 00 00 00 00 00
uint32_t:int1:    17 4 = 29 03 03 00
uint32_t:int2:    21 4 = 00 00 00 00

Comparing GoodStruct3 and BadStruct, it seems one pointer is handled fine, but not two.
The first time this is run after Image boots seems like BadStruct gets some random data.

================================
In same image, subsequent runs of  LibStruct>>>LibStructTest>>testStructs     
give only all zeros for BadStruct, as Todd observed.

GoodStruct1:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
GoodStruct2:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
void*:ptr_data:    09 8 = 00 00 00 00 00 00 00 00
GoodStruct3:
void*:ptr_data:    01 8 = 00 00 00 00 00 00 00 00
uint32_t:int1:    09 4 = 02 00 00 00
uint32_t:int2:    13 4 = 03 00 00 00
BadStruct:
void*:ptr_data1:    01 8 = 00 00 00 00 00 00 00 00
void*:ptr_data2:    09 8 = 00 00 00 00 00 00 00 00
uint32_t:int1:    17 4 = 00 00 00 00
uint32_t:int2:    21 4 = 00 00 00 00


cheers -ben


FFI64StructTest.zip (4K) Download Attachment