FFI Struct Argument Pass By Value Fails on Mac 64 bit

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

FFI Struct Argument Pass By Value Fails on Mac 64 bit

tblanchard
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

I'm stuck.  The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {
  const void *ptr_data[2];
  unsigned int_data;
} CXSourceLocation;

/**
 * \brief Identifies a half-open character range in the source code.
 *
 * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the
 * starting and end locations from a source range, respectively.
 */
typedef struct {
  const void *ptr_data[2];
  unsigned begin_int_data;
  unsigned end_int_data;
} CXSourceRange;

const char* first = "first_pointer";
const char* second = "second_pointer";

// return a fake range with non zero data
CXSourceRange clang_getArbitraryRange()
{
  CXSourceRange range = {0};
  range.ptr_data[0] = (void*)first;
  range.ptr_data[1] = (void*)second;
  range.begin_int_data = 17;
  range.end_int_data = 24;
  return range;
}

// Actual clang function - range is always zero'd here despite having values in the image
CXSourceLocation clang_getRangeStart(CXSourceRange range) {
  // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1) {
    CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };
    return Result;    
  }

  

  CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },
    range.begin_int_data };
  return Result;
}




LibCClang-FFI-Binding.st (7K) Download Attachment
microclang.c (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

Ben Coman
 
Hi Todd,

Could you advise the the Playground code to use to test this?
And maybe a C program that links to the microclang shared library that does the equivalent test?

cheers -ben

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

I'm stuck.  The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {
  const void *ptr_data[2];
  unsigned int_data;
} CXSourceLocation;

/**
 * \brief Identifies a half-open character range in the source code.
 *
 * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the
 * starting and end locations from a source range, respectively.
 */
typedef struct {
  const void *ptr_data[2];
  unsigned begin_int_data;
  unsigned end_int_data;
} CXSourceRange;

const char* first = "first_pointer";
const char* second = "second_pointer";

// return a fake range with non zero data
CXSourceRange clang_getArbitraryRange()
{
  CXSourceRange range = {0};
  range.ptr_data[0] = (void*)first;
  range.ptr_data[1] = (void*)second;
  range.begin_int_data = 17;
  range.end_int_data = 24;
  return range;
}

// Actual clang function - range is always zero'd here despite having values in the image
CXSourceLocation clang_getRangeStart(CXSourceRange range) {
  // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1) {
    CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };
    return Result;    
  }

  

  CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },
    range.begin_int_data };
  return Result;
}





Reply | Threaded
Open this post in threaded view
|

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

tblanchard
 
Playground code:

LibCClang startFromRange:(LibCClang defaultRange)

it worked if the resulting struct is not all zeroes.

A tiny C program is the equivalent.

int main()
{
  if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) printf("That failed\n"); 
  else printf("That worked\n")
  return 0;
}


On Nov 21, 2017, at 11:18 PM, Ben Coman <[hidden email]> wrote:

Hi Todd,

Could you advise the the Playground code to use to test this?
And maybe a C program that links to the microclang shared library that does the equivalent test?

cheers -ben

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

I'm stuck.  The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {
  const void *ptr_data[2];
  unsigned int_data;
} CXSourceLocation;

/**
 * \brief Identifies a half-open character range in the source code.
 *
 * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the
 * starting and end locations from a source range, respectively.
 */
typedef struct {
  const void *ptr_data[2];
  unsigned begin_int_data;
  unsigned end_int_data;
} CXSourceRange;

const char* first = "first_pointer";
const char* second = "second_pointer";

// return a fake range with non zero data
CXSourceRange clang_getArbitraryRange()
{
  CXSourceRange range = {0};
  range.ptr_data[0] = (void*)first;
  range.ptr_data[1] = (void*)second;
  range.begin_int_data = 17;
  range.end_int_data = 24;
  return range;
}

// Actual clang function - range is always zero'd here despite having values in the image
CXSourceLocation clang_getRangeStart(CXSourceRange range) {
  // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1) {
    CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };
    return Result;    
  }
  
  CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },
    range.begin_int_data };
  return Result;
}






Reply | Threaded
Open this post in threaded view
|

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

Ben Coman
In reply to this post by tblanchard
 


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)



I'm stuck.  The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {
  const void *ptr_data[2];
  unsigned int_data;
} CXSourceLocation;

/**
 * \brief Identifies a half-open character range in the source code.
 *
 * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the
 * starting and end locations from a source range, respectively.
 */
typedef struct {
  const void *ptr_data[2];
  unsigned begin_int_data;
  unsigned end_int_data;
} CXSourceRange;

const char* first = "first_pointer";
const char* second = "second_pointer";

// return a fake range with non zero data
CXSourceRange clang_getArbitraryRange()
{
  CXSourceRange range = {0};
  range.ptr_data[0] = (void*)first;
  range.ptr_data[1] = (void*)second;
  range.begin_int_data = 17;
  range.end_int_data = 24;
  return range;
}

// Actual clang function - range is always zero'd here despite having values in the image
CXSourceLocation clang_getRangeStart(CXSourceRange range) {
  // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1) {
    CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };
    return Result;    
  }

  

  CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },
    range.begin_int_data };
  return Result;
}


clang_ffi_struct_test.zip (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

Ben Coman
 


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...


___microlang.h___
typedef unsigned  uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);



___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{       CXSourceRange_ range = {0};
        range.ptr_data[0] = (void*)libraryString;
        return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{       // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
        if ((uintptr_t)range.ptr_data[0] & 0x1)
        {       return 0;       }
        else
        {       return 1;       }
}




___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString =  "local_pointer";

void test( CXSourceRange_ range, char *note )
{       int result = clang_getRangeEnd_(range);
        if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else
        {       printf("That worked (%s)\n", note); }
}

int main()
{       CXSourceRange_ range1 = clang_getArbitraryRange_();
        test(range1, "library string");

        CXSourceRange_ range2 = {0};
        range2.ptr_data[0] = (void*)localString;
        test(range2, "local string");
}



___Makefile___
default: clean static shared

clean:
        rm -f *so *App
        @echo

shared:
        clang -g -o libmicroclang.so -shared -fPIC microclang.c
        clang -g -o sharedApp test.c -L. -lmicroclang
        LD_LIBRARY_PATH=. ./sharedApp
        @echo

static:
        clang -g -o staticApp test.c microclang.c
        ./staticApp
        @echo



Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)


cheers -ben

clang2.zip (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

Ben Coman
 


On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...


___microlang.h___
typedef unsigned  uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);



___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{       CXSourceRange_ range = {0};
        range.ptr_data[0] = (void*)libraryString;
        return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{       // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
        if ((uintptr_t)range.ptr_data[0] & 0x1)
        {       return 0;       }
        else
        {       return 1;       }
}




___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString =  "local_pointer";

void test( CXSourceRange_ range, char *note )
{       int result = clang_getRangeEnd_(range);
        if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else
        {       printf("That worked (%s)\n", note); }
}

int main()
{       CXSourceRange_ range1 = clang_getArbitraryRange_();
        test(range1, "library string");

        CXSourceRange_ range2 = {0};
        range2.ptr_data[0] = (void*)localString;
        test(range2, "local string");
}



___Makefile___
default: clean static shared

clean:
        rm -f *so *App
        @echo

shared:
        clang -g -o libmicroclang.so -shared -fPIC microclang.c
        clang -g -o sharedApp test.c -L. -lmicroclang
        LD_LIBRARY_PATH=. ./sharedApp
        @echo

static:
        clang -g -o staticApp test.c microclang.c
        ./staticApp
        @echo



Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)


Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned  uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{       return myLibraryString;
}

int lib_testString( const char *aString )
{       unsigned test = (uintptr_t)aString & 0x1;
        printf("\n  test=%d, aString-->%d\n", test, (uintptr_t)aString);
        if (test)
        {       return 0;       }
        else
        {       return 1;       }
}


___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString =  "local_pointer";

void test( int result, char *note )
{       if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else 
        {       printf("That worked (%s)\n", note); }
}

int main()
{       const char * libraryString = lib_getLibraryString();
        test(lib_testString(libraryString), "library string");

        test(lib_testString(localString), "local string");
}


$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

  test=0, aString-->4196150
  That worked (library string)

  test=0, aString-->4196068
  That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

  test=1, aString-->-792512599
  That failed (library string)

  test=0, aString-->4196484
  That worked (local string)


cheers -ben


sharedLibString.zip (2K) Download Attachment
clang3.zip (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

tblanchard
 
i'm getting the idea that we should probably write a test suite/library for FFI

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:



On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...


___microlang.h___
typedef unsigned  uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);



___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{       CXSourceRange_ range = {0};
        range.ptr_data[0] = (void*)libraryString;
        return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{       // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
        if ((uintptr_t)range.ptr_data[0] & 0x1)
        {       return 0;       }
        else
        {       return 1;       }
}




___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString =  "local_pointer";

void test( CXSourceRange_ range, char *note )
{       int result = clang_getRangeEnd_(range);
        if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else
        {       printf("That worked (%s)\n", note); }
}

int main()
{       CXSourceRange_ range1 = clang_getArbitraryRange_();
        test(range1, "library string");

        CXSourceRange_ range2 = {0};
        range2.ptr_data[0] = (void*)localString;
        test(range2, "local string");
}



___Makefile___
default: clean static shared

clean:
        rm -f *so *App
        @echo

shared:
        clang -g -o libmicroclang.so -shared -fPIC microclang.c
        clang -g -o sharedApp test.c -L. -lmicroclang
        LD_LIBRARY_PATH=. ./sharedApp
        @echo

static:
        clang -g -o staticApp test.c microclang.c
        ./staticApp
        @echo



Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)


Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned  uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{       return myLibraryString;
}

int lib_testString( const char *aString )
{       unsigned test = (uintptr_t)aString & 0x1;
        printf("\n  test=%d, aString-->%d\n", test, (uintptr_t)aString);
        if (test)
        {       return 0;       }
        else
        {       return 1;       }
}


___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString =  "local_pointer";

void test( int result, char *note )
{       if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else 
        {       printf("That worked (%s)\n", note); }
}

int main()
{       const char * libraryString = lib_getLibraryString();
        test(lib_testString(libraryString), "library string");

        test(lib_testString(localString), "local string");
}


$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

  test=0, aString-->4196150
  That worked (library string)

  test=0, aString-->4196068
  That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

  test=1, aString-->-792512599
  That failed (library string)

  test=0, aString-->4196484
  That worked (local string)


cheers -ben

<sharedLibString.zip><clang3.zip>

Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

Ben Coman
 


On 27 November 2017 at 00:24, Todd Blanchard <[hidden email]> wrote:
 
i'm getting the idea that we should probably write a test suite/library for FFI

I noticed these...

These look like c-code test frames for FFI.  Could these be built by the OpenSmalltalk CI to be normally shipped with the VM so that Image-side CI can test against them?

And also some general info...

 

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:


On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

My last analysis discovered something interesting about strings defined inside shared libraries being handled differently, but only later realised I had chased the wrong rabbit down the hole.  
 
I've now investigated the premise you actually poses, and I agree, that structs being zero is some cases.  
The attached zipfile containing  libstruct.c  a few comparison cases - three "good" layouts that work fine and one "bad" that mostly gets zeros but sometimes other weird numbers.  There is a Makefile with three main targets:
1. make layout - statically compiles  libstruct.c  and runs produced  a.out  to display structure layouts 

2. make run - downloads Pharo, starts it loading  LibStruct.st  , then you manually run LibStruct>>>LibStructTest>>#testStructs and observe structure values on console

3. make debug - starts LLDB to run Pharo with breakpoints pre-configured for when you run #testStructs.  
   Note Pharo will freeze and you need to move to LLDB.  Try these commands... 
      frame variable
      call print_struct(&GoodStruct1_fmt, &aStruct)
      continue

That works on Ubuntu 16.04 64 bit. 
You will need to tune it for OSX.
clang and lldb are required.

================================
The offsets configured in the class variables of all ExternalStructs 
correctly matches that reported by the C code test frame results here...

$ make layout        
clang -g libstruct.c
./a.out

GoodStruct1:
uint32_t:int1:    01 4 
uint32_t:int2:    05 4 
GoodStruct2:
uint32_t:int1:    01 4 
uint32_t:int2:    05 4 
void*:ptr_data:    09 8 
GoodStruct3:
void*:ptr_data:    01 8 
uint32_t:int1:    09 4 
uint32_t:int2:    13 4 
BadStruct:
void*:ptr_data1:    01 8 
void*:ptr_data2:    09 8 
uint32_t:int1:    17 4 
uint32_t:int2:    21 4 

================================

$ make run   
clang -g -o libstruct.so -shared -fPIC libstruct.c
getpharo/pharo-vm/lib/pharo/5.0-201707201942/pharo getpharo/Pharo.image ../LibStruct.st

Now manually browse to and run  LibStruct>>>LibStructTest>>#testStructs
LibStructTest>>testStructs
|good1Struct good2Struct good3Struct badStruct |
good1Struct    := GoodStruct1    new int1: 2; int2: 3.
good2Struct    := GoodStruct2    new int1: 2; int2: 3; ptr_force_int: 4.
good3Struct    := GoodStruct3    new int1: 2; int2: 3; ptr_force_int: 4.
badStruct      := BadStruct      new int1: 2; int2: 3; ptr_force_int1: 4; ptr_force_int1: 5.
self assert: (LibStruct tryGood1: good1Struct)       equals: 6.
self assert: (LibStruct tryGood2: good2Struct)       equals: 6.
self assert: (LibStruct tryGood3: good3Struct)       equals: 6.
"Problem exposed in next line"
self assert: (LibStruct tryBad: badStruct)           equals: 6.

which on console produces...

GoodStruct1:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
GoodStruct2:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
void*:ptr_data:    09 8 = 04 00 00 00 00 00 00 00
GoodStruct3:
void*:ptr_data:    01 8 = 04 00 00 00 00 00 00 00
uint32_t:int1:    09 4 = 02 00 00 00
uint32_t:int2:    13 4 = 03 00 00 00
BadStruct:
void*:ptr_data1:    01 8 = 00 00 00 00 00 00 00 00
void*:ptr_data2:    09 8 = 00 00 00 00 00 00 00 00
uint32_t:int1:    17 4 = 29 03 03 00
uint32_t:int2:    21 4 = 00 00 00 00

Comparing GoodStruct3 and BadStruct, it seems one pointer is handled fine, but not two.
The first time this is run after Image boots seems like BadStruct gets some random data.

================================
In same image, subsequent runs of  LibStruct>>>LibStructTest>>testStructs     
give only all zeros for BadStruct, as Todd observed.

GoodStruct1:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
GoodStruct2:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
void*:ptr_data:    09 8 = 00 00 00 00 00 00 00 00
GoodStruct3:
void*:ptr_data:    01 8 = 00 00 00 00 00 00 00 00
uint32_t:int1:    09 4 = 02 00 00 00
uint32_t:int2:    13 4 = 03 00 00 00
BadStruct:
void*:ptr_data1:    01 8 = 00 00 00 00 00 00 00 00
void*:ptr_data2:    09 8 = 00 00 00 00 00 00 00 00
uint32_t:int1:    17 4 = 00 00 00 00
uint32_t:int2:    21 4 = 00 00 00 00


cheers -ben


FFI64StructTest.zip (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

tblanchard
 
I got pulled off to do some $ work and haven't got back to this but I would like to get back to it.

Has there been any progress on this in the publicly available 64bit VM?

I see there is a completely separate FFI plugin for 32 vs 64 and worry 64 is still not quite ready for real work.

On Dec 3, 2017, at 7:18 AM, Ben Coman <[hidden email]> wrote:



On 27 November 2017 at 00:24, Todd Blanchard <[hidden email]> wrote:
 
i'm getting the idea that we should probably write a test suite/library for FFI

I noticed these...

These look like c-code test frames for FFI.  Could these be built by the OpenSmalltalk CI to be normally shipped with the VM so that Image-side CI can test against them?

And also some general info...

 

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:


On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

My last analysis discovered something interesting about strings defined inside shared libraries being handled differently, but only later realised I had chased the wrong rabbit down the hole.  
 
I've now investigated the premise you actually poses, and I agree, that structs being zero is some cases.  
The attached zipfile containing  libstruct.c  a few comparison cases - three "good" layouts that work fine and one "bad" that mostly gets zeros but sometimes other weird numbers.  There is a Makefile with three main targets:
1. make layout - statically compiles  libstruct.c  and runs produced  a.out  to display structure layouts 

2. make run - downloads Pharo, starts it loading  LibStruct.st  , then you manually run LibStruct>>>LibStructTest>>#testStructs and observe structure values on console

3. make debug - starts LLDB to run Pharo with breakpoints pre-configured for when you run #testStructs.  
   Note Pharo will freeze and you need to move to LLDB.  Try these commands... 
      frame variable
      call print_struct(&GoodStruct1_fmt, &aStruct)
      continue

That works on Ubuntu 16.04 64 bit. 
You will need to tune it for OSX.
clang and lldb are required.

================================
The offsets configured in the class variables of all ExternalStructs 
correctly matches that reported by the C code test frame results here...

$ make layout        
clang -g libstruct.c
./a.out

GoodStruct1:
uint32_t:int1:    01 4 
uint32_t:int2:    05 4 
GoodStruct2:
uint32_t:int1:    01 4 
uint32_t:int2:    05 4 
void*:ptr_data:    09 8 
GoodStruct3:
void*:ptr_data:    01 8 
uint32_t:int1:    09 4 
uint32_t:int2:    13 4 
BadStruct:
void*:ptr_data1:    01 8 
void*:ptr_data2:    09 8 
uint32_t:int1:    17 4 
uint32_t:int2:    21 4 

================================

$ make run   
clang -g -o libstruct.so -shared -fPIC libstruct.c
getpharo/pharo-vm/lib/pharo/5.0-201707201942/pharo getpharo/Pharo.image ../LibStruct.st

Now manually browse to and run  LibStruct>>>LibStructTest>>#testStructs
LibStructTest>>testStructs
|good1Struct good2Struct good3Struct badStruct |
good1Struct    := GoodStruct1    new int1: 2; int2: 3.
good2Struct    := GoodStruct2    new int1: 2; int2: 3; ptr_force_int: 4.
good3Struct    := GoodStruct3    new int1: 2; int2: 3; ptr_force_int: 4.
badStruct      := BadStruct      new int1: 2; int2: 3; ptr_force_int1: 4; ptr_force_int1: 5.
self assert: (LibStruct tryGood1: good1Struct)       equals: 6.
self assert: (LibStruct tryGood2: good2Struct)       equals: 6.
self assert: (LibStruct tryGood3: good3Struct)       equals: 6.
"Problem exposed in next line"
self assert: (LibStruct tryBad: badStruct)           equals: 6.

which on console produces...

GoodStruct1:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
GoodStruct2:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
void*:ptr_data:    09 8 = 04 00 00 00 00 00 00 00
GoodStruct3:
void*:ptr_data:    01 8 = 04 00 00 00 00 00 00 00
uint32_t:int1:    09 4 = 02 00 00 00
uint32_t:int2:    13 4 = 03 00 00 00
BadStruct:
void*:ptr_data1:    01 8 = 00 00 00 00 00 00 00 00
void*:ptr_data2:    09 8 = 00 00 00 00 00 00 00 00
uint32_t:int1:    17 4 = 29 03 03 00
uint32_t:int2:    21 4 = 00 00 00 00

Comparing GoodStruct3 and BadStruct, it seems one pointer is handled fine, but not two.
The first time this is run after Image boots seems like BadStruct gets some random data.

================================
In same image, subsequent runs of  LibStruct>>>LibStructTest>>testStructs     
give only all zeros for BadStruct, as Todd observed.

GoodStruct1:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
GoodStruct2:
uint32_t:int1:    01 4 = 02 00 00 00
uint32_t:int2:    05 4 = 03 00 00 00
void*:ptr_data:    09 8 = 00 00 00 00 00 00 00 00
GoodStruct3:
void*:ptr_data:    01 8 = 00 00 00 00 00 00 00 00
uint32_t:int1:    09 4 = 02 00 00 00
uint32_t:int2:    13 4 = 03 00 00 00
BadStruct:
void*:ptr_data1:    01 8 = 00 00 00 00 00 00 00 00
void*:ptr_data2:    09 8 = 00 00 00 00 00 00 00 00
uint32_t:int1:    17 4 = 00 00 00 00
uint32_t:int2:    21 4 = 00 00 00 00


cheers -ben

<FFI64StructTest.zip>

Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

Eliot Miranda-2
In reply to this post by tblanchard
 
Hi Todd,

On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <[hidden email]> wrote:
 
i'm getting the idea that we should probably write a test suite/library for FFI

See the package FFI-Tests at http://source.squeak.org/FFI and the file sqFFITestFuncs.c
 

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:



On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...


___microlang.h___
typedef unsigned  uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);



___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{       CXSourceRange_ range = {0};
        range.ptr_data[0] = (void*)libraryString;
        return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{       // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
        if ((uintptr_t)range.ptr_data[0] & 0x1)
        {       return 0;       }
        else
        {       return 1;       }
}




___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString =  "local_pointer";

void test( CXSourceRange_ range, char *note )
{       int result = clang_getRangeEnd_(range);
        if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else
        {       printf("That worked (%s)\n", note); }
}

int main()
{       CXSourceRange_ range1 = clang_getArbitraryRange_();
        test(range1, "library string");

        CXSourceRange_ range2 = {0};
        range2.ptr_data[0] = (void*)localString;
        test(range2, "local string");
}



___Makefile___
default: clean static shared

clean:
        rm -f *so *App
        @echo

shared:
        clang -g -o libmicroclang.so -shared -fPIC microclang.c
        clang -g -o sharedApp test.c -L. -lmicroclang
        LD_LIBRARY_PATH=. ./sharedApp
        @echo

static:
        clang -g -o staticApp test.c microclang.c
        ./staticApp
        @echo



Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)


Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned  uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{       return myLibraryString;
}

int lib_testString( const char *aString )
{       unsigned test = (uintptr_t)aString & 0x1;
        printf("\n  test=%d, aString-->%d\n", test, (uintptr_t)aString);
        if (test)
        {       return 0;       }
        else
        {       return 1;       }
}


___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString =  "local_pointer";

void test( int result, char *note )
{       if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else 
        {       printf("That worked (%s)\n", note); }
}

int main()
{       const char * libraryString = lib_getLibraryString();
        test(lib_testString(libraryString), "library string");

        test(lib_testString(localString), "local string");
}


$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

  test=0, aString-->4196150
  That worked (library string)

  test=0, aString-->4196068
  That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

  test=1, aString-->-792512599
  That failed (library string)

  test=0, aString-->4196484
  That worked (local string)


cheers -ben

<sharedLibString.zip><clang3.zip>





--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

Eliot Miranda-2
 
Todd,

On Thu, Mar 1, 2018 at 10:01 AM, Eliot Miranda <[hidden email]> wrote:
Hi Todd,

On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <[hidden email]> wrote:
 
i'm getting the idea that we should probably write a test suite/library for FFI

See the package FFI-Tests at http://source.squeak.org/FFI and the file sqFFITestFuncs.c

I suggest you extend these with an example that demonstrates your failing case.  One thing to be very careful about is the platform.  Are you on Mac OS X, linux or Win64?  Mac OS X & linux use the SysV ABI (see e.g. https://software.intel.com/sites/default/files/article/402129/mpx-linux64-abi.pdf) whereas Win64 uses Microsoft's own ABI (see https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147217396).  In particular under the Sys V ABI certain structs (structs having two 64-bit fields) will be passed in registers, if two registers are available at that point in the parameter list.  The ThreadedX64SysVFFIPlugin is written to pass structs in this manner but there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.  Microsoft's own ABI passes structs of size 8, 16, 32, or 64 bits and __m64 in a single register.  The ThreadedX64Win64FFIPlugin is written to pass structs in this manner but again there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.
 

HTH

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:



On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...


___microlang.h___
typedef unsigned  uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);



___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{       CXSourceRange_ range = {0};
        range.ptr_data[0] = (void*)libraryString;
        return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{       // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
        if ((uintptr_t)range.ptr_data[0] & 0x1)
        {       return 0;       }
        else
        {       return 1;       }
}




___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString =  "local_pointer";

void test( CXSourceRange_ range, char *note )
{       int result = clang_getRangeEnd_(range);
        if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else
        {       printf("That worked (%s)\n", note); }
}

int main()
{       CXSourceRange_ range1 = clang_getArbitraryRange_();
        test(range1, "library string");

        CXSourceRange_ range2 = {0};
        range2.ptr_data[0] = (void*)localString;
        test(range2, "local string");
}



___Makefile___
default: clean static shared

clean:
        rm -f *so *App
        @echo

shared:
        clang -g -o libmicroclang.so -shared -fPIC microclang.c
        clang -g -o sharedApp test.c -L. -lmicroclang
        LD_LIBRARY_PATH=. ./sharedApp
        @echo

static:
        clang -g -o staticApp test.c microclang.c
        ./staticApp
        @echo



Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)


Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned  uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{       return myLibraryString;
}

int lib_testString( const char *aString )
{       unsigned test = (uintptr_t)aString & 0x1;
        printf("\n  test=%d, aString-->%d\n", test, (uintptr_t)aString);
        if (test)
        {       return 0;       }
        else
        {       return 1;       }
}


___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString =  "local_pointer";

void test( int result, char *note )
{       if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else 
        {       printf("That worked (%s)\n", note); }
}

int main()
{       const char * libraryString = lib_getLibraryString();
        test(lib_testString(libraryString), "library string");

        test(lib_testString(localString), "local string");
}


$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

  test=0, aString-->4196150
  That worked (library string)

  test=0, aString-->4196068
  That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

  test=1, aString-->-792512599
  That failed (library string)

  test=0, aString-->4196484
  That worked (local string)


cheers -ben

<sharedLibString.zip><clang3.zip>





--
_,,,^..^,,,_
best, Eliot



--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

tblanchard
 
I'm mac osx 

I will update the test library and ping back

On Mar 1, 2018, at 10:11 AM, Eliot Miranda <[hidden email]> wrote:

Todd,

On Thu, Mar 1, 2018 at 10:01 AM, Eliot Miranda <[hidden email]> wrote:
Hi Todd,

On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <[hidden email]> wrote:
 
i'm getting the idea that we should probably write a test suite/library for FFI

See the package FFI-Tests at http://source.squeak.org/FFI and the file sqFFITestFuncs.c

I suggest you extend these with an example that demonstrates your failing case.  One thing to be very careful about is the platform.  Are you on Mac OS X, linux or Win64?  Mac OS X & linux use the SysV ABI (see e.g. https://software.intel.com/sites/default/files/article/402129/mpx-linux64-abi.pdf) whereas Win64 uses Microsoft's own ABI (see https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147217396).  In particular under the Sys V ABI certain structs (structs having two 64-bit fields) will be passed in registers, if two registers are available at that point in the parameter list.  The ThreadedX64SysVFFIPlugin is written to pass structs in this manner but there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.  Microsoft's own ABI passes structs of size 8, 16, 32, or 64 bits and __m64 in a single register.  The ThreadedX64Win64FFIPlugin is written to pass structs in this manner but again there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.
 

HTH

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:



On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...


___microlang.h___
typedef unsigned  uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);



___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{       CXSourceRange_ range = {0};
        range.ptr_data[0] = (void*)libraryString;
        return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{       // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
        if ((uintptr_t)range.ptr_data[0] & 0x1)
        {       return 0;       }
        else
        {       return 1;       }
}




___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString =  "local_pointer";

void test( CXSourceRange_ range, char *note )
{       int result = clang_getRangeEnd_(range);
        if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else
        {       printf("That worked (%s)\n", note); }
}

int main()
{       CXSourceRange_ range1 = clang_getArbitraryRange_();
        test(range1, "library string");

        CXSourceRange_ range2 = {0};
        range2.ptr_data[0] = (void*)localString;
        test(range2, "local string");
}



___Makefile___
default: clean static shared

clean:
        rm -f *so *App
        @echo

shared:
        clang -g -o libmicroclang.so -shared -fPIC microclang.c
        clang -g -o sharedApp test.c -L. -lmicroclang
        LD_LIBRARY_PATH=. ./sharedApp
        @echo

static:
        clang -g -o staticApp test.c microclang.c
        ./staticApp
        @echo



Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)


Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned  uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{       return myLibraryString;
}

int lib_testString( const char *aString )
{       unsigned test = (uintptr_t)aString & 0x1;
        printf("\n  test=%d, aString-->%d\n", test, (uintptr_t)aString);
        if (test)
        {       return 0;       }
        else
        {       return 1;       }
}


___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString =  "local_pointer";

void test( int result, char *note )
{       if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else 
        {       printf("That worked (%s)\n", note); }
}

int main()
{       const char * libraryString = lib_getLibraryString();
        test(lib_testString(libraryString), "library string");

        test(lib_testString(localString), "local string");
}


$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

  test=0, aString-->4196150
  That worked (library string)

  test=0, aString-->4196068
  That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

  test=1, aString-->-792512599
  That failed (library string)

  test=0, aString-->4196484
  That worked (local string)


cheers -ben

<sharedLibString.zip><clang3.zip>





-- 
_,,,^..^,,,_
best, Eliot



-- 
_,,,^..^,,,_
best, Eliot

Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

tblanchard
 
Silly me.

I have no idea how to update this library.  I thought it would be a file.  I've used Monticello years ago in Squeak - however trying to load this repository into Pharo 6.1-64 is impossible.  Invariably I end up with an undefined symbol and a couple of undismissable progress dialogs.

Also, its .c files that compile into a library, right?  Where would such files be in this repository?

I love Smalltalk.

I'm really starting to hate this though.

On Mar 1, 2018, at 10:25 AM, todd blanchard <[hidden email]> wrote:

I'm mac osx 

I will update the test library and ping back

On Mar 1, 2018, at 10:11 AM, Eliot Miranda <[hidden email]> wrote:

Todd,

On Thu, Mar 1, 2018 at 10:01 AM, Eliot Miranda <[hidden email]> wrote:
Hi Todd,

On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <[hidden email]> wrote:
 
i'm getting the idea that we should probably write a test suite/library for FFI

See the package FFI-Tests at http://source.squeak.org/FFI and the file sqFFITestFuncs.c

I suggest you extend these with an example that demonstrates your failing case.  One thing to be very careful about is the platform.  Are you on Mac OS X, linux or Win64?  Mac OS X & linux use the SysV ABI (see e.g. https://software.intel.com/sites/default/files/article/402129/mpx-linux64-abi.pdf) whereas Win64 uses Microsoft's own ABI (see https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147217396).  In particular under the Sys V ABI certain structs (structs having two 64-bit fields) will be passed in registers, if two registers are available at that point in the parameter list.  The ThreadedX64SysVFFIPlugin is written to pass structs in this manner but there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.  Microsoft's own ABI passes structs of size 8, 16, 32, or 64 bits and __m64 in a single register.  The ThreadedX64Win64FFIPlugin is written to pass structs in this manner but again there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.
 

HTH

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:



On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:


On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:
 
I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated.  The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations.  After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included.  Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
   test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
   if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) 

I presume you meant...
    if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) 
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.  
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked


So it seems a similar problem exists outside our FFI. 

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...


___microlang.h___
typedef unsigned  uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);



___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{       CXSourceRange_ range = {0};
        range.ptr_data[0] = (void*)libraryString;
        return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{       // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
        if ((uintptr_t)range.ptr_data[0] & 0x1)
        {       return 0;       }
        else
        {       return 1;       }
}




___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString =  "local_pointer";

void test( CXSourceRange_ range, char *note )
{       int result = clang_getRangeEnd_(range);
        if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else
        {       printf("That worked (%s)\n", note); }
}

int main()
{       CXSourceRange_ range1 = clang_getArbitraryRange_();
        test(range1, "library string");

        CXSourceRange_ range2 = {0};
        range2.ptr_data[0] = (void*)localString;
        test(range2, "local string");
}



___Makefile___
default: clean static shared

clean:
        rm -f *so *App
        @echo

shared:
        clang -g -o libmicroclang.so -shared -fPIC microclang.c
        clang -g -o sharedApp test.c -L. -lmicroclang
        LD_LIBRARY_PATH=. ./sharedApp
        @echo

static:
        clang -g -o staticApp test.c microclang.c
        ./staticApp
        @echo



Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)


Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned  uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{       return myLibraryString;
}

int lib_testString( const char *aString )
{       unsigned test = (uintptr_t)aString & 0x1;
        printf("\n  test=%d, aString-->%d\n", test, (uintptr_t)aString);
        if (test)
        {       return 0;       }
        else
        {       return 1;       }
}


___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString =  "local_pointer";

void test( int result, char *note )
{       if(result == 0)
        {       printf("That failed (%s)\n", note); }
        else 
        {       printf("That worked (%s)\n", note); }
}

int main()
{       const char * libraryString = lib_getLibraryString();
        test(lib_testString(libraryString), "library string");

        test(lib_testString(localString), "local string");
}


$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

  test=0, aString-->4196150
  That worked (library string)

  test=0, aString-->4196068
  That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

  test=1, aString-->-792512599
  That failed (library string)

  test=0, aString-->4196484
  That worked (local string)


cheers -ben

<sharedLibString.zip><clang3.zip>





-- 
_,,,^..^,,,_
best, Eliot



-- 
_,,,^..^,,,_
best, Eliot